Modeling Fly Social Behavior with Agent-Centric Pose Forecasting
summary
The gist
The gist: This work introduces an agent-centric framework for training autoregressive models of animal behavior by constraining deep networks to receive sensory inputs and produce motor outputs
In short
This work introduces an agent-centric framework to train models of animal behavior by forcing deep networks to mimic a specific animal's sensory inputs and motor outputs. It uses a library called AnimalPoseForecasting to manage complex data representations, allowing researchers to study how animals generate behavior rather than just fitting movement trajectories. The results show this approach captures fine-grained social behaviors accurately.
Key concepts
- Agent-Centric Framework
- This core idea treats each animal as an independent agent that receives sensory information and produces movements in the body's frame. The network is constrained so that it must solve the same problem as the animal itself, aiming to understand how behavior is generated rather than just predicting movement paths.
- AnimalPoseForecasting Library
- This general-purpose library manages complex data representations needed for the framework. It handles invertible operations between different views of pose and sensing—such as world-frame, egocentric, and machine learning friendly formats—using composable sequences to connect these representations.
- Sensory Input/Output Representations
- The input is carefully designed to limit the agent to what the animal can actually perceive. For flies, this includes depth maps for visual information about other flies and summarized tactile data like ceiling height at keypoints. Output choices explore different prediction forms and temporal frames.
- Autoregressive Rollout
- Generation involves iteratively predicting the next step in a dynamic environment. Because the next step depends on all agents' previous actions, this requires inverting predicted movements back into world coordinates at each step to maintain consistency across the simulation.
Terminology used across episodes
This episode discusses
- Modeling Fly Social Behavior with Agent-Centric Pose Forecasting · Paper Radio
- MABe22: A Multi-Species Multi-Task Benchmark for Learned Representations of Behavior
- Review of Pedestrian Trajectory Prediction Methods: Comparing Deep Learning and Knowledge-based Approaches
- DESIRE: Distant Future Prediction in Dynamic Scenes with Interacting Agents
- The Trajectron: Probabilistic Multi-Agent Trajectory Modeling With Dynamic Spatiotemporal Graphs
- AgentFormer: Agent-Aware Transformers for Socio-Temporal Multi-Agent Forecasting
- Learning recurrent representations for hierarchical behavior modeling
- Evaluation metrics for behaviour modeling
- Recurrent Network Models for Human Dynamics
- QuaterNet: A Quaternion-based Recurrent Model for Human Motion
- SoMoFormer: Multi-Person Pose Forecasting with Transformers
- On human motion prediction using recurrent neural networks
- Learning Trajectory Dependencies for Human Motion Prediction
- Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learning
- Offline Reinforcement Learning as One Big Sequence Modeling Problem
- RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control
- OpenVLA: An Open-Source Vision-Language-Action Model
- A Reduction of Imitation Learning and Structured Prediction to No-Regret Online Learning
The paper
Modeling Fly Social Behavior with Agent-Centric Pose Forecasting · Read on arXiv
Eyrun Eyjolfsdottir, Kristin Branson
HHMI Janelia Research Campus
Transcript
Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.
Tom: I'm Tom, and with me are Jane, Lu, senior AI researcher at Tsinghua, Meng, lead engineer at a mysterious AI startup and Lalam, the in-house Large Language Model.
Jane: Today's paper: "Modeling Fly Social Behavior with Agent-Centric Pose Forecasting".
Tom: The gist:
Jane: First, who's behind it and why it matters.
Paper summary: Tom: So, to recap, this paper introduces the "Modeling Fly Social Behavior with Agent-Centric Pose Forecasting" framework. The core thesis is building autoregressive models of animal behavior where every animal is an agent that inputs sensory data and outputs movements in its own body frame.
Jane: They claim this approach allows the network and the animal to solve the same problem, shifting the focus from just fitting trajectories to understanding how an animal generates its behavior.
Lu: The framework requires managing many parallel representations of data alongside ML-specific transformations like discretization, which they support with a library called AnimalPoseForecasting.
Meng: This library handles the necessary transformations between different pose representations—world-frame, egocentric, and ML-friendly ones—through composable and invertible operations.
Lalam: The sensory input is hand-crafted to respect what the animal perceives, like using depth maps for visual information about other flies or summarizing tactile info by looking at ceiling height at keypoints.
Tom: And they also explore output representation choices across three axes: the form of the prediction, whether it's continuous, discrete, or hybrid; the temporal frame of reference, absolute pose versus change in pose; and how you parameterize the pose itself.
Jane: When dealing with dynamic environments where agents interact, generation happens by applying next-step prediction iteratively. This requires inverting every agent’s predicted movement back into a change in world-coordinate pose at each rollout step.
Lu: They show that this library design ensures that the same operations used to map raw pose into the model's egocentric input space during training are inverted at rollout time to map predictions back into world coordinates.
Meng: The evaluation looked at qualitative characterization, quantitatively comparing distributions of behavior features between real and simulated data, and training discriminators to tell real from simulated data.
Lalam: Their key finding is that agent-centric representations enabled by the library are most important for accurate behavior modeling in this work on fly social behavior.
Conclusion: Tom: So, looking at the full paper, "Modeling Fly Social Behavior with Agent-Centric Pose Forecasting," the authors are aiming to use these models as tools for scientific discovery, trying to figure out what internal structures animals implement.
Jane: They show that this agent-centric formulation is essential infrastructure for using data-driven predictive models as actual tools for finding new behaviors.
Lu: The work suggests that mechanistic interpretability, by interrogating a trained model, is a promising path toward this goal, letting scientists extract patterns that they haven't previously articulated.
Meng: From an engineering standpoint, it’s interesting because the framework's adaptability was demonstrated by applying it successfully to RatInABox, which is a synthetic neuro-behavior environment.
Lalam: The paper also highlights that discretization plays a vital role in avoiding the pitfall of forecasting no movement when modeling these systems.
Tom: Ultimately, this work pushes us toward using these models not just as simulators but as ways to understand the computations animals are actually performing, which is a big step for ethology and neuroscience.
More episodes
- 2607.21927-RIS-Kernel: A Model-Agnostic Architecture for Long-Context LLM Inference via Sparse Attention
- 2607.19198-ATLAS: A Foundation Neural Sampler for Amorphous Materials
- 2404.14855-The Geometry of the Set of Equivalent Linear Neural Networks
- 2409.09984-Convergence of Sharpness-Aware Minimization Algorithms using Increasing Batch Size and Decaying Learning Rate
- 2505.24438-Weisfeiler and Leman Follow the Arrow of Time: Expressive Power of Message Passing in Temporal Event Graphs
- 2410.13735-Generative Conformal Prediction with Optimized Coverage Allocation
- 2501.19179-Learning Non-Local Molecular Interactions via Equivariant Local Representations and Charge Equilibration
- 2507.15431-Inexact calculus of variations on the hyperspherical tangent bundle with connections to the attention mechanism
- 2503.07775-Sublinear Algorithms for Wasserstein and Total Variation Distances: Applications to Fairness and Privacy Auditing
- 2503.11147-Asynchronous Sharpness-Aware Minimization For Fast and Accurate Deep Learning