Inconsistency between `pred_context_pose` and `pred_all_context_extrinsic`/`pred_all_context_intrinsic` in `evaluate` function

[tan901q]

Hi, thank you for sharing this amazing work! While reading through the code, I noticed a potential inconsistency in the eval_nvs.py regarding the handling of pred_context_pose and pred_all_context_extrinsic/pred_all_context_intrinsic. I would like to clarify this with you.

In the evaluate function:

• pred_context_pose is obtained from the encoder output and is aligned with the gaussians.
• pred_all_context_extrinsic and pred_all_context_intrinsic are decoded from pose_encoding_to_extri_intri.

Although the code performs scale alignment for the translation part of pred_all_context_extrinsic, the rotation matrices and intrinsic matrices are not explicitly aligned. This raises the following concerns:

1. Training view rendering: If the rotation matrices in pred_context_pose['extrinsic'] and pred_all_context_extrinsic are inconsistent, could this lead to a mismatch between the training views and the Gaussian point cloud? If the intrinsic matrices in pred_context_pose['intrinsic'] and pred_all_context_intrinsic are inconsistent, could this cause the rendered views to deviate from the training views?
2. Output vs. target views: Since pred_context_pose and pred_all_context_extrinsic/pred_all_context_intrinsic are not fully aligned, could the rendered output views differ from the intended target views?

Here is the relevant code snippet from the evaluate function:

scale_factor = pred_context_pose['extrinsic'][:, :, :3, 3].mean() / pred_all_context_extrinsic[:, :, :3, 3].mean()
pred_all_target_extrinsic[..., :3, 3] = pred_all_target_extrinsic[..., :3, 3] * scale_factor
pred_all_context_extrinsic[..., :3, 3] = pred_all_context_extrinsic[..., :3, 3] * scale_factor

The code aligns the translation part of pred_all_context_extrinsic with pred_context_pose['extrinsic'], but the rotation matrices and intrinsic matrices are not aligned.

Questions

1. Should the rotation matrices in pred_all_context_extrinsic be aligned with pred_context_pose['extrinsic']?
2. Should the intrinsic matrices in pred_all_context_intrinsic be aligned with pred_context_pose['intrinsic']?
3. If no alignment is performed, could this affect the accuracy of the rendered training and testing views?

Thank you for your time and assistance! Looking forward to your response. 😊

[richardchen225]

I've encountered the same issue as you. Would it be possible to discuss it further privately?

[tan901q]

I've encountered the same issue as you. Would it be possible to discuss it further privately?
我按照作者的代码，单纯对齐了一下训练视角相机中心的尺度，确实最终render出来的视角会有轻微的差异。另外两次不同的预测，相机中心感觉是不是也不是简单的放大缩小的尺度差异，感觉也没那么鲁棒，feed-forward的方法。

[richardchen225]

是的，我尝试过把内参也对齐，最后的psnr会更高，我不太清楚作者为何没有对齐内参，也可能是我理解错误。

[tan901q]

是的，我尝试过把内参也对齐，最后的psnr会更高，我不太清楚作者为何没有对齐内参，也可能是我理解错误。
对齐以后从视觉上和gt视角是一致的吗？你再对齐一下旋转试一试？主要我感觉两次预测的相对位姿可能不是一个简单的线性关系，就会有点麻烦。他那个外参只线性对齐了外参的平移部分。

[richardchen225]

是的，我尝试过把内参也对齐，最后的psnr会更高，我不太清楚作者为何没有对齐内参，也可能是我理解错误。
对齐以后从视觉上和gt视角是一致的吗？你再对齐一下旋转试一试？主要我感觉两次预测的相对位姿可能不是一个简单的线性关系，就会有点麻烦。他那个外参只线性对齐了外参的平移部分。

我认为旋转的对齐和t,K的对齐不太一样，因为t和K是有scale的。我感觉旋转对齐，可能需要获得一下vggt输出的ctx+tgt的ctx部分的每个相机旋转，和anysplat输出的相机位姿（ctx部分）的每个相机旋转，然后分别计算一下相对的变换矩阵，然后可能再把这个变换矩阵平均一下？最后作用到tgt的相机上去？但感觉这个可能对齐效果也不好。