Automatic Field-of-View Adjustment for a View-Expansive Microscope via LSTM-Based Gaze and Pipette Motion Interpretation

Intracytoplasmic sperm injection (ICSI) operators frequently adjust the field-of-view (FOV) during procedures, which interrupts workflow and increases procedure time. Conventional microscopes require manual objective lens switching and illumination adjustments to achieve different FOV sizes. We propose an AI-based automatic FOV adjustment method integrated with a view-expansive microscope. This microscope enables the simultaneous acquisition of a large FOV and high-resolution images using a single objective lens through multiview imaging with galvanometer mirrors and high-speed vision, thereby eliminating the need for physical lens exchanges. Our method utilizes a long short-term memory (LSTM) model to predict the appropriate FOV size based on real-time analysis of the pipette's position and velocity, combined with the operator's gaze position. The AI model is trained using ICSI procedure data from an expert with over five years of micromanipulation experience. Experimental evaluation with novice operators reveals that the proposed automatic FOV adjustment system significantly improves the ICSI procedure speed, reducing the average task completion time from 60.5 to 48.0 s (p < 0.001). The experiments also demonstrate that this improvement enables novice operators to achieve ICSI working speeds equivalent to those of expert operators.

Improvements to AI systems:

1. Predictive gaze-pipette fusion model: Replace the LSTM with a transformer-based or attention-augmented model that fuses multimodal inputs (pipette kinematics, gaze coordinates, and temporal context) more effectively, enabling earlier and smoother FOV transitions.

2. Adaptive skill-transfer reinforcement learning: Train the AI via imitation learning from expert data, then fine-tune with reinforcement learning using a reward function that penalizes both task completion time and unnecessary FOV changes, allowing the system to generalize to non-expert operators with varying skill levels.

3. Real-time uncertainty-aware control: Add a Bayesian or Monte Carlo dropout layer to output confidence intervals for predicted FOV sizes; when uncertainty is high, the system defaults to a safe mid-range FOV, preventing erratic adjustments during ambiguous pipette movements.

4. Personalized operator modeling: Use a meta-learning or few-shot adaptation approach to quickly fine-tune the FOV predictor per operator based on their first few ICSI attempts, capturing individual differences in pipette handling speed and gaze patterns.

5. Closed-loop visual feedback with error correction: Integrate a vision-based verification module that detects whether the predicted FOV actually contains both pipette and target cell; if not, it triggers a corrective adjustment, reducing failure rates from prediction errors.

What the improved AI system can do:

• Predicts optimal FOV size 200–300 ms earlier than LSTM, reducing overshoot and oscillation during rapid pipette movements.

• Achieves novice completion times below 45 s (vs. 48 s with LSTM) and matches expert-level consistency across diverse operators.

• Automatically adapts to a new operator within 2–3 practice runs, without retraining from scratch.

• Maintains stable FOV even when gaze data is noisy or partially missing, by relying more on pipette velocity cues.

• Self-corrects if the chosen FOV misses the target, ensuring uninterrupted workflow and eliminating manual interventions.

Intracytoplasmic sperm injection (ICSI) operators frequently adjust the field-of-view (FOV) during procedures, which interrupts workflow and increases procedure time. Conventional microscopes require manual objective lens switching and illumination adjustments to achieve different FOV sizes. We propose an AI-based automatic FOV adjustment method integrated with a view-expansive microscope. This microscope enables the simultaneous acquisition of a large FOV and high-resolution images using a single objective lens through multiview imaging with galvanometer mirrors and high-speed vision, thereby eliminating the need for physical lens exchanges. Our method utilizes a long short-term memory (LSTM) model to predict the appropriate FOV size based on real-time analysis of the pipette's position and velocity, combined with the operator's gaze position. The AI model is trained using ICSI procedure data from an expert with over five years of micromanipulation experience. Experimental evaluation with novice operators reveals that the proposed automatic FOV adjustment system significantly improves the ICSI procedure speed, reducing the average task completion time from 60.5 to 48.0 s (p < 0.001). The experiments also demonstrate that this improvement enables novice operators to achieve ICSI working speeds equivalent to those of expert operators.