Video2SwimFish: An Automated Pipeline for Reconstructing Controllable Fish Models and Biological Locomotion from Real Fish Videos

arXiv:2609.38966 · cs.RO · Submitted 2026-09-30 · Read on arXiv

cs.RO

Submitted: 2026-09-30

Updated: 2026-09-30

Code: https://github.com/hangongchen/Video2SwimFish

Project page: https://hangongchen.github.io/video2swimfish-web

Terminology

• Actor — 37× in this paper · explained in 3 episodes
• Benchmark — 20× in this paper · explained in 2 episodes
• Fidelity — 14× in this paper · explained in 2 episodes
• Reinforcement Learning — 13× in this paper · explained in 4 episodes
• Alignment — 11× in this paper · explained in 1 episode
• Action Space — 7× in this paper · explained in 1 episode
• Computer Vision and Pattern Recognition — 5× in this paper
• Biological Locomotion Manifold — 4× in this paper
• Calibration — 3× in this paper · explained in 7 episodes
• Embodied AI — 3× in this paper · explained in 2 episodes
• Fine-Tuning — 3× in this paper · explained in 5 episodes
• Generative AI — 3× in this paper · explained in 2 episodes
• Institutional Animal Care and Use Committee — 3× in this paper
• Pseudo-labels — 3× in this paper · explained in 2 episodes
• Regularization — 3× in this paper · explained in 3 episodes
• Deep Reinforcement Learning — 2× in this paper · explained in 2 episodes
• Foundation Model — 2× in this paper · explained in 3 episodes
• International Conference on Robotics and Automation — 2× in this paper
• Wasserstein Distance — 2× in this paper · explained in 4 episodes
• Accuracy — 1× in this paper · explained in 1 episode
• Adam — 1× in this paper · explained in 2 episodes
• Foundation Models — 1× in this paper · explained in 7 episodes
• Principal Component Analysis (PCA) — 1× in this paper · explained in 4 episodes
• Regret — 1× in this paper · explained in 3 episodes
• Robustness — 1× in this paper · explained in 5 episodes
• Twenty-Seventh International Joint Conference on Artificial Intelligence — 1× in this paper
• Verification — 1× in this paper · explained in 2 episodes
• Verifier — 1× in this paper · explained in 2 episodes
• alpha — 1× in this paper · explained in 1 episode

Sources

• Aquaculture field robotics: Applications, lessons learned and future prospects
• Qwen3-VL Technical Report
• URDFormer: A Pipeline for Constructing Articulated Simulation Environments from Real-World Images
• ManiSkill2: A Unified Benchmark for Generalizable Manipulation Skills
• LRM: Large Reconstruction Model for Single Image to 3D
• BEHAVIOR-1K: A Human-Centered, Embodied AI Benchmark with 1,000 Everyday Activities and Realistic Simulation
• Imitation from Observation: Learning to Imitate Behaviors from Raw Video via Context Translation
• Isaac Gym: High Performance GPU-Based Physics Simulation For Robot Learning
• Proximal Policy Optimization Algorithms
• Habitat 2.0: Training Home Assistants to Rearrange their Habitat
• TripoSR: Fast 3D Object Reconstruction from a Single Image
• InstantMesh: Efficient 3D Mesh Generation from a Single Image with Sparse-view Large Reconstruction Models

Related papers

• FMT x: An Efficient and Asymptotically Optimal Extension of the Fast Marching Tree for Dynamic Replanning
• MPCFormer: A physics-informed data-driven approach for explainable socially-aware autonomous driving
• RoboLab: A High-Fidelity Simulation Benchmark for Analysis of Task Generalist Policies
• HRDexDB: A 4D Dexterous Grasping Dataset Across Human and Multiple Robot Embodiments
• APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
• Fine-tuning is Not Enough: A Parallel Framework for Collaborative Imitation and Reinforcement Learning in End-to-end Autonomous Driving