Potential Coordinate Convention Inconsistency in Extrinsics (World↔Camera)

[wangst0181]

Hi, I found a possible inconsistency in the coordinate convention for extrinsics used in AnySplat, which might cause confusion, especially when integrating VGGT.

After checking both VGGT’s data and model outputs, as well as how the losses are computed, I confirmed that VGGT uses world-to-camera (w2c) extrinsics consistently throughout the pipeline.

However, in AnySplat, the situation appears reversed: Input data and the encoder outputs both use camera-to-world (c2w) extrinsics. Despite this, one part of the code assumes w2c convention.

Specifically, in the training setup:
if self.model.encoder.pred_pose: self.loss_pose = HuberLoss(alpha=self.train_cfg.pose_loss_alpha, delta=self.train_cfg.pose_loss_delta)

This loss calls:
GT_pose_enc = extri_intri_to_pose_encoding(context_extrinsics, context_intrinsics, image_size_hw)

According to the function’s intended design, extri_intri_to_pose_encoding expects world-to-camera (w2c) extrinsics. However, context_extrinsics in AnySplat are actually camera-to-world (c2w).

Although this pose_loss is currently not actively used in the final training pipeline, this mismatch introduces confusion when interpreting coordinate conventions and potentially misleads users trying to integrate or debug extrinsics-related features.

Thanks for your great work on this project! A clarification of the extrinsics coordinate flow throughout the codebase would be very helpful for users working on pose alignment and dataset integration.

[Giperx]

Hi! According to these use:

AnySplat/src/model/encoder/anysplat.py

Lines 376 to 378 in 57937e4

	distill_pts_all = batchify_unproject_depth_map_to_point_map(
	distill_depth_map, distill_extrinsic, distill_intrinsic
	)

and

AnySplat/src/model/encoder/anysplat.py

Lines 429 to 431 in 57937e4

	pts_all = batchify_unproject_depth_map_to_point_map(
	depth_map, extrinsic, intrinsic
	)

I might have missed some steps in the conversion process, do the outputs of AnySplat and VGGT both represent the world-to-camera extrinsics?

[wangst0181]

@Giperx
The lines you referenced show how the inputs from the VGGT output (i.e., distill_extrinsic) are used directly to unproject the depth map to points:

distill_pts_all = batchify_unproject_depth_map_to_point_map(
    distill_depth_map, distill_extrinsic, distill_intrinsic
)

This indeed suggests that the VGGT output, distill_extrinsic, must be in the World-to-Camera (W2C) convention, consistent with the claim that VGGT uses W2C.

However, I found a critical detail in the AnySplat encoder's prediction for the extrinsics. Right after the encoder generates the raw pose prediction, it applies an inverse (W2C -> C2W):

extrinsic = torch.cat([extrinsic, extrinsic_padding], dim=2).inverse()

Also, context_extrinsics in AnySplat are actually camera-to-world (C2W)) either.

[Giperx]

@Giperx The lines you referenced show how the inputs from the VGGT output (i.e., distill_extrinsic) are used directly to unproject the depth map to points:

distill_pts_all = batchify_unproject_depth_map_to_point_map(
    distill_depth_map, distill_extrinsic, distill_intrinsic
)

This indeed suggests that the VGGT output, distill_extrinsic, must be in the World-to-Camera (W2C) convention, consistent with the claim that VGGT uses W2C.

However, I found a critical detail in the AnySplat encoder's prediction for the extrinsics. Right after the encoder generates the raw pose prediction, it applies an inverse (W2C -> C2W):

extrinsic = torch.cat([extrinsic, extrinsic_padding], dim=2).inverse()

Also, context_extrinsics in AnySplat are actually camera-to-world (C2W)) either.

Thank you for your reply, I understand.

[ricshaw]

@Giperx The lines you referenced show how the inputs from the VGGT output (i.e., distill_extrinsic) are used directly to unproject the depth map to points:

distill_pts_all = batchify_unproject_depth_map_to_point_map(
    distill_depth_map, distill_extrinsic, distill_intrinsic
)

This indeed suggests that the VGGT output, distill_extrinsic, must be in the World-to-Camera (W2C) convention, consistent with the claim that VGGT uses W2C.

However, I found a critical detail in the AnySplat encoder's prediction for the extrinsics. Right after the encoder generates the raw pose prediction, it applies an inverse (W2C -> C2W):

extrinsic = torch.cat([extrinsic, extrinsic_padding], dim=2).inverse()

Also, context_extrinsics in AnySplat are actually camera-to-world (C2W)) either.

Does this mean there is a bug with the distillation loss?

[Giperx]

@Giperx The lines you referenced show how the inputs from the VGGT output (i.e., distill_extrinsic) are used directly to unproject the depth map to points:

distill_pts_all = batchify_unproject_depth_map_to_point_map(
    distill_depth_map, distill_extrinsic, distill_intrinsic
)

This indeed suggests that the VGGT output, distill_extrinsic, must be in the World-to-Camera (W2C) convention, consistent with the claim that VGGT uses W2C.
However, I found a critical detail in the AnySplat encoder's prediction for the extrinsics. Right after the encoder generates the raw pose prediction, it applies an inverse (W2C -> C2W):
extrinsic = torch.cat([extrinsic, extrinsic_padding], dim=2).inverse()
Also, context_extrinsics in AnySplat are actually camera-to-world (C2W)) either.

Does this mean there is a bug with the distillation loss?

There is no problem with distillation loss.

AnySplat/src/model/encoder/anysplat.py

Lines 376 to 378 in 57937e4

	distill_pts_all = batchify_unproject_depth_map_to_point_map(
	distill_depth_map, distill_extrinsic, distill_intrinsic
	)

AnySplat/src/model/encoder/anysplat.py

Lines 429 to 431 in 57937e4

	pts_all = batchify_unproject_depth_map_to_point_map(
	depth_map, extrinsic, intrinsic
	)

Both distill_extrinsic and extrinsic come from the use of the pose_encoding_to_extri_intri function. For the pose_encoding obtained from the camera_head of VGGT and AnySplat, both represent W2C, so the use of the loss below is not problematic:

AnySplat/src/loss/loss_distill.py

Lines 111 to 121 in 57937e4

	def forward(self, distill_infos, pred_pose_enc_list, prediction, batch):
	loss_pose = 0.0
	if pred_pose_enc_list is not None:
	num_predictions = len(pred_pose_enc_list)
	pesudo_gt_pose_enc = distill_infos['pred_pose_enc_list']
	for i in range(num_predictions):
	i_weight = self.gamma ** (num_predictions - i - 1)
	cur_pred_pose_enc = pred_pose_enc_list[i]
	cur_pesudo_gt_pose_enc = pesudo_gt_pose_enc[i]
	loss_pose += i_weight * huber_loss(cur_pred_pose_enc, cur_pesudo_gt_pose_enc).mean()

The conversion from W2C to C2W ends at the encoder part in AnySplat, and when returning, extrinsic it is obtained through .inverse():

AnySplat/src/model/encoder/anysplat.py

Line 575 in 57937e4

extrinsic=torch.cat([extrinsic, extrinsic_padding], dim=2).inverse(),

distill loss uses pred_pose_enc_list, and .inverse() does not affect pred_pose_enc_list:

AnySplat/src/model/encoder/anysplat.py

Line 573 in 57937e4

pred_pose_enc_list=pred_pose_enc_list,