Marker-less real-time intra-operative camera and hand-eye calibration procedure for surgical augmented reality

• Main Authors: Megha Kalia, Prateek Mathur, Nassir Navab, Septimiu E. Salcudean
• Format: Article
• Language: English
• Published: Wiley 2019-10-01
• Series: Healthcare Technology Letters
• Subjects: medical robotics; medical image processing; cameras; endoscopes; calibration; robot vision; surgery; phantoms; rendering (computer graphics); augmented reality; image registration; biomedical optical imaging; hand-eye calibration procedure; surgical augmented reality; pre-operative medical data; camera intrinsic matrix estimation; hand-eye transformation; camera calibration; endoscope; high visual error; virtual rendered tool tip; marker-less real-time intra-operative camera; augmented reality rendering; subsequent gradient descent steps; da vinci robot; average target registration error; prostate phantom
• Online Access: https://digital-library.theiet.org/content/journals/10.1049/htl.2019.0094

Accurate medical Augmented Reality (AR) rendering requires two calibrations, a camera intrinsic matrix estimation and a hand-eye transformation. We present a unified, practical, marker-less, real-time system to estimate both these transformations during surgery. For camera calibration we perform calibrations at multiple distances from the endoscope, pre-operatively, to parametrize the camera intrinsic matrix as a function of distance from the endoscope. Then, we retrieve the camera parameters intra-operatively by estimating the distance of the surgical site from the endoscope in less than 1 s. Unlike in prior work, our method does not require the endoscope to be taken out of the patient; for the hand-eye calibration, as opposed to conventional methods that require the identification of a marker, we make use of a rendered tool-tip in 3D. As the surgeon moves the instrument and observes the offset between the actual and the rendered tool-tip, they can select points of high visual error and manually bring the instrument tip to match the virtual rendered tool tip. To evaluate the hand-eye calibration, 5 subjects carried out the hand-eye calibration procedure on a da Vinci robot. Average Target Registration Error of approximately 7mm was achieved with just three data points.