The research groups in computer vision, graphics, and machine learning have dedicated a substantial amount of attention to the areas of 3D object reconstruction, augmentation, and registration. Deep learning is the predominant method used in artificial intelligence for addressing computer vision challenges. However, deep learning on three-dimensional data presents distinct obstacles and is now in its nascent phase. There have been significant advancements in deep learning specifically for three-dimensional data, offering a range of ways to address these issues. This study offers a comprehensive examination of the latest advancements in deep learning methodologies. We examine many benchmark models for the tasks of 3D object registration, augmentation, and reconstruction. We thoroughly analyse their architectures, advantages, and constraints. In summary, this report provides a comprehensive overview of recent advancements in three-dimensional deep learning and highlights unresolved research areas that will need to be addressed in the future.

In contemporary practice, intraoral scans and cone-beam computed tomography (CBCT) are widely adopted techniques for tooth localization and the acquisition of comprehensive three-dimensional models. Despite their utility, each dataset presents inherent merits and limitations, prompting the pursuit of an amalgamated solution for optimization. Thus, this research introduces a novel 3D registration approach aimed at harmonizing these distinct datasets to offer a holistic perspective. In the pre-processing phase, a retrained Mask-RCNN is deployed on both sagittal and panoramic projections to partition upper and lower teeth from the encompassing CBCT raw data. Simultaneously, a chromatic classification model is proposed for segregating gingival tissue from tooth structures in intraoral scan data. Subsequently, the segregated datasets are aligned based on dental crowns, employing the robust RANSAC and ICP algorithms. To assess the proposed methodology\'s efficacy, the Euclidean distance between corresponding points is statistically evaluated. Additionally, dental experts, including two orthodontists and an experienced general dentist, evaluate the clinical potential by measuring distances between landmarks on tooth surfaces. The computed error in corresponding point distances between intraoral scan data and CBCT data in the automatically registered datasets utilizing the proposed technique is quantified at 0.234 ± 0.019 mm, which is significantly below the 0.3 mm CBCT voxel size. Moreover, the average measurement discrepancy among expert-identified landmarks ranges from 0.368 to 1.079 mm, underscoring the promise of the proposed method.

OBJECTIVE: Arthroplasty surgery can be challenging because of limited exposure of the bones. However, in shoulder arthroplasty for example, a good positioning of the glenoid component is essential to mitigate risks of revision surgeries. To improve the procedure\'s outcomes, mixed reality can be used as a guidance system relying on a fundamental registration step between the patient\'s bone and its corresponding 3D model.
METHODS: We present a complete registration workflow for shoulder arthroplasty using Hololens 2 Head Mounted Display. We rely on acquisitions made thanks to our marker-based tracking system, an improved Iterative Closest Point algorithm and verification steps. Our accuracy targets are 1.5 ± 1.5 mm for the glenoid guidewire entry point on both antero-posterior and supero-inferior axes, and 1.5 ± 1[Formula: see text] for inclination and version. The overall process must last less than 5 min.
RESULTS: We have evaluated our process on a cohort of 13 3D printed glenoid bones of all types, showing an average accuracy of 0.84 ± 0.58 mm on the antero-posterior axis and 0.49 ± 0.41 mm on the supero-inferior one for the entry point. As for inclination and version, we have 0.89 ± 0.6[Formula: see text] and 0.97 ± 0.8[Formula: see text], respectively. The mean process time is about 1 min 24 s.
CONCLUSIONS: We have developed a complete, embedded registration workflow and a verification protocol to evaluate our accuracy. Our results are promising for the improvement of the glenoid guidewire placement. Moreover, everything is performed in the field of view of the surgeon, which allows them to fully concentrate on the surgical site.

With the popularization of the concept of \"metaverse\", Augmented Reality (AR) technology is slowly being applied to people\'s daily life as its underlying technology support. In recent years, rapid 3D reconstruction of interior furniture to meet AR shopping needs has become a new method. In this paper, a virtual home environment system is designed and the related core technologies in the system are studied. Background removal and instance segmentation are performed for furniture images containing complex backgrounds, and a Bayesian Classifier and GrabCut (BCGC) algorithm is proposed to improve on the traditional foreground background separation technique. The reconstruction part takes the classical occupancy network reconstruction algorithm as the network basis and proposes a precise occupancy network (PONet) algorithm, which can reconstruct the structural details of furniture images, and the model accuracy is improved. Because the traditional 3D registration model is prone to the problems of model position shift and inaccurate matching with the scene, the AKAZE-based tracking registration algorithm is improved, and a Multiple Filtering-AKAZE (MF-AKAZE) based on AKAZE is proposed to remove the matching points. The matching accuracy is increased by improving the RANSAC filtering mis-matching algorithm based on further screening of the matching results. Finally, the system is verified to realize the function of the AR visualization furniture model, which can better complete the reconstruction as well as registration effect.

OBJECTIVE: Posttraumatic fibular malunion alters ankle joint biomechanics and may lead to pain, stiffness, and premature osteoarthritis. The accurate restoration is key for success of reconstructive surgeries. The aim of this study was to analyze the accuracy of a novel three-dimensional (3D) registration algorithm using different segments of the contralateral anatomy to restore the distal fibula.
METHODS: Triangular 3D surface models were reconstructed from computed tomographic data of 96 paired lower legs. Four segments were defined: 25% tibia, 50% tibia, 75% fibula, and 75% fibula and tibia. A surface registration algorithm was used to superimpose the mirrored contralateral model on the original model. The accuracy of distal fibula restoration was measured.
RESULTS: The median rotation error, 3D distance (Euclidean distance), and 3D angle (Euler\'s angle) using the distal 25% tibia segment for the registration were 0.8° (- 1.7-4.8), 2.1 mm (1.4-2.9), and 2.9° (1.9-5.4), respectively. The restoration showed the highest errors using the 75% fibula segment (rotation error 3.2° (0.1-8.3); Euclidean distance 4.2 mm (3.1-5.8); Euler\'s angle 5.8° (3.4-9.2)). The translation error did not differ significantly between segments.
CONCLUSIONS: 3D registration of the contralateral tibia and fibula reliably approximated the premorbid anatomy of the distal fibula. Registration of the 25% distal tibia, including distinct anatomical landmarks of the fibular notch and malleolar colliculi, restored the anatomy with increasing accuracy, minimizing both rotational and translational errors. This new method of evaluating malreductions could reduce morbidity in patients with ankle fractures.
METHODS: IV.

This paper proposes a new technique for performing 3D static-point cloud registration after calibrating a multi-view RGB-D camera using a 3D (dimensional) joint set. Consistent feature points are required to calibrate a multi-view camera, and accurate feature points are necessary to obtain high-accuracy calibration results. In general, a special tool, such as a chessboard, is used to calibrate a multi-view camera. However, this paper uses joints on a human skeleton as feature points for calibrating a multi-view camera to perform calibration efficiently without special tools. We propose an RGB-D-based calibration algorithm that uses the joint coordinates of the 3D joint set obtained through pose estimation as feature points. Since human body information captured by the multi-view camera may be incomplete, a joint set predicted based on image information obtained through this may be incomplete. After efficiently integrating a plurality of incomplete joint sets into one joint set, multi-view cameras can be calibrated by using the combined joint set to obtain extrinsic matrices. To increase the accuracy of calibration, multiple joint sets are used for optimization through temporal iteration. We prove through experiments that it is possible to calibrate a multi-view camera using a large number of incomplete joint sets.

We provide a database aimed at real-time quantitative analysis of 3D reconstruction and alignment methods, containing 3140 point clouds from 10 subjects/objects. These scenes are acquired with a high-resolution 3D scanner. It contains depth maps that produce point clouds with more than 500k points on average. This dataset is useful to develop new models and alignment strategies to automatically reconstruct 3D scenes from data acquired with optical scanners or benchmarking purposes.

Intravascular ultrasound (IVUS) imaging offers accurate cross-sectional vessel information. To this end, registering temporal IVUS pullbacks acquired at two time points can assist the clinicians to accurately assess pathophysiological changes in the vessels, disease progression and the effect of the treatment intervention. In this paper, we present a novel two-stage registration framework for aligning pairs of longitudinal and axial IVUS pullbacks. Initially, we use a Dynamic Time Warping (DTW)-based algorithm to align the pullbacks in a temporal fashion. Subsequently, an intensity-based registration method, that utilizes a variant of the Harmony Search optimizer to register each matched pair of the pullbacks by maximizing their Mutual Information, is applied. The presented method is fully automated and only required two single global image-based measurements, unlike other methods that require extraction of morphology-based features. The data used includes 42 synthetically generated pullback pairs, achieving an alignment error of 0.1853 frames per pullback, a rotation error 0.93° and a translation error of 0.0161 mm. In addition, it was also tested on 11 baseline and follow-up, and 10 baseline and post-stent deployment real IVUS pullback pairs from two clinical centres, achieving an alignment error of 4.3±3.9 for the longitudinal registration, and a distance and a rotational error of 0.56±0.323 mm and 12.4°±10.5°, respectively, for the axial registration. Although the performance of the proposed method does not match that of the state-of-the-art, our method relies on computationally lighter steps for its computations, which is crucial in real-time applications. On the other hand, the proposed method performs even or better that the state-of-the-art when considering the axial registration. The results indicate that the proposed method can support clinical decision making and diagnosis based on sequential imaging examinations.

Three-dimensional feature description for a local surface is a core technology in 3D computer vision. Existing descriptors perform poorly in terms of distinctiveness and robustness owing to noise, mesh decimation, clutter, and occlusion in real scenes. In this paper, we propose a 3D local surface descriptor using point-pair transformation feature histograms (PPTFHs) to address these challenges. The generation process of the PPTFH descriptor consists of three steps. First, a simple but efficient strategy is introduced to partition the point-pair sets on the local surface into four subsets. Then, three feature histograms corresponding to each point-pair subset are generated by the point-pair transformation features, which are computed using the proposed Darboux frame. Finally, all the feature histograms of the four subsets are concatenated into a vector to generate the overall PPTFH descriptor. The performance of the PPTFH descriptor is evaluated on several popular benchmark datasets, and the results demonstrate that the PPTFH descriptor achieves superior performance in terms of descriptiveness and robustness compared with state-of-the-art algorithms. The benefits of the PPTFH descriptor for 3D surface matching are demonstrated by the results obtained from five benchmark datasets.

BACKGROUND: In the clinical applications of optical coherence tomography angiography (OCTA), the repeated scanning and averaging method can provide better contrast with reduced speckle noises in the final results, which are useful for visualizing and quantifying vascular components with high accuracy, reproducibility, and reliability. However, the inevitable patient motion presents a challenge to this method. The objective of this study is to meet this challenge by introducing a 3D registration method to register optical coherence tomography (OCT)/OCTA scans for precise volume averaging of multiple scans to improve the signal-to-noise ratio (SNR) and increase quantification accuracy.
METHODS: The proposed method utilized both rigid affine transformation and non-rigid B-spline transformation in which their parameters were optimized and calculated by the average stochastic gradient descent on OCT structural images. In addition, we also introduced a multi-level resolution approach to further improve the robustness and computational speed of our proposed method. The imaging performance was tested on in vivo imaging of human skin and eye and assessed by SNR, peak signal-to-noise ratio (PSNR) and normalized correlation coefficient (NCC).
RESULTS: Five subjects were enrolled in this study for obtaining in vivo images of skin and retina. The proposed registration and averaging method provided substantial improvements of the imaging performance in terms of vessel connectivity and signal to noise ratio. The increase of repeated volume numbers in the averaging improves all the metrics assessed, i.e., SNR, PSNR and NCC. An improvement of the SNR from 10 to 40 dB after 10 repeated volumetric averaging was achieved.
CONCLUSIONS: The proposed 3D registration and averaging method is effective in reducing speckle noises and suppressing motion artifacts, thereby improving SNR, PSNR and NCC metrics for final averaged images. It is expected that the proposed algorithm would be practically useful in better visualization and more reliable quantification of in vivo OCT and OCTA data, which would be beneficial to OCT clinical applications.