Empowering Children to Create AI-Enabled Augmented Reality Experiences

This paper introduces Capybara, an AR-based and AI-powered visual programming environment that empowers children to create, customize, and program 3D characters overlaid onto the physical world. Capybara enables children to create virtual characters and accessories using text-to-3D generative AI models, and to animate these characters through auto-rigging and body tracking. In addition, our system employs vision-based AI models to recognize physical objects, allowing children to program interactive behaviors between virtual characters and their physical surroundings. We demonstrate the expressiveness of Capybara through a set of novel AR experiences. We conducted user studies with 20 children in the United States and Argentina. Our findings suggest that Capybara can empower children to harness AI in authoring personalized and engaging AR experiences that seamlessly bridge the virtual and physical worlds.

The paper's contribution is threefold: 1) Capybara, an authoring tool for children to create AI-enabled AR experiences, which seamlessly combines AI techniques with children’s content creation process; 2) a set of example experiences demonstrating the expressiveness of the tool; and 3) empirical insights gained from our user studies with children, highlighting its benefits and challenges of using Capybara.

The system has three primary design goals: D1 - Lowering the barrier of entry for programming AI-enabled AR behaviors, D2 - Enabling higher customizability, and D3 - Facilitating interactions between the virtual and physical worlds.

Capybara is based on a block-based programming environment in AR. The code blocks feature an interlocking, puzzle-like design that visually guides users in constructing functional programs. These blocks are categorized into five distinct groups, each with a unique color for easy identification: (1) Event Blocks (Yellow), (2) Motion Blocks (Dark Blue), (3) Looks Blocks (Purple), (4) Control Blocks (Orange), and (5) Sensing Blocks (Light Blue).

The system integrates Generative AI (GenAI) functionalities to allow children to create personalized AR experiences through customized characters and accessories. The character customization process is initiated through the Settings menu, where users can select Custom Character to begin. Accessory customization, on the other hand, is triggered directly within the inventory interface using the start wear code block. Both character and accessory generation workflows follow a similar UI pattern. Upon initiating the generation process, users see a dedicated creation page asking them to describe their desired character or accessory using speech. The system uses the Google Gemini 1.5 Flash LLM to review the prompt and ensure the appropriateness of the request for a young audience. Once the prompt passes the filter, the system calls an external text-to-3D-object generation service.

Capybara also enables users to customize the character’s animation via puppeteering. The system refines and modifies the open-sourced auto-rigging algorithm proposed by Baran and Popović. In Capybara, the animation feature is integrated into the block-based programming environment by providing the code block named start animation. Users can drag the codeblock and place it on the canvas, which then guides them to define the animation clip that can be programmatically replayed.

A key limitation of existing authoring tools for children is that the outcome experiences remain in the digital world. One of the design goals of Capybara is to facilitate children to create interactions between the virtual and the physical world. The system leverages off-the-shelf vision-based AI models to detect objects in the user’s physical surroundings. Users can program interactions with physical objects using the touches object block. The system adapted the YOLOv11s model, a deep neural network for real-time object detection, and curated a set of 24 recognizable objects. The system also introduced the touches zone block as part of the Sensing Blocks. Zones are user-defined regions on a physical surface that also support collision detection, enabling interactions with parts of the physical surroundings that are not covered by object detection.

To demonstrate the complete authoring workflow of combining the above functionalities and showcase the expressiveness of Capybara, the paper offers a set of novel experiences, including: Getting ready for the day, Alphabet learning, and Ping-pong gameplay.

To understand the benefits and challenges of Capybara, the researchers conducted a series of user studies with children in the United States and Argentina. They recruited 20 participants (12 female and 8 male, age 7–16), including nine groups. Each study session lasted approximately 90 minutes. The study session began with an introduction and a walkthrough of the system that lasted approximately 20 minutes. After the walkthrough, participants were asked to complete a warm-up exercise, where they were asked to create the example experience depicted in Figure 7.a from scratch. Then, participants completed a 30-minute open-ended authoring task. After finishing all activities, participants filled out a system usability survey and concluded with an open-ended discussion.

The results show that all participants were able to complete the given warm-up exercise using Capybara. Participants generally felt that Capybara was easy to use (M=3.80, SD=1.01). They also thought that they enjoyed playing Capybara (M=4.35, SD=0.99) and that they felt confident when playing Capybara (M=4.40, SD=0.82).

The findings highlight several creative usage patterns: creating animations of physical activities for storytelling, generating 3D characters for both realistic and fantastical experiences, and building interactions between the virtual and the physical world.

The qualitative feedback revealed several benefits and challenges. Participants thought that Capybara feels unique due to its 3D coding environment and AI-enabled customization and interaction. They also found that Capybara offers creative freedom due to AI-enabled customization. However, the integration of AI also introduced challenges of AI alignments and distractions. Participants reported that Capybara facilitates programming interactions with the physical world. Finally, the findings suggest pedagogical potential for fostering Computational Thinking and AI literacy.

The discussion section highlights opportunities and challenges of the design of Capybara and discusses implications for future research at the intersection of AR, AI, and child-centered programming environments. The paper discusses programming intricate interactions between the virtual and physical worlds, balancing AI-enabled customization and children’s agency, fostering Computational Thinking and AI literacy, and safety, privacy, and equity towards large-scale deployment.

The paper also acknowledges limitations. System limitations include the auto-rigging algorithm's assumptions (closed and connected mesh, proportions roughly matching the predefined rigging skeleton), the speech-to-3D module's generation time, and the object detection's limited set of recognizable objects. Evaluation limitations include the small number of participants (N = 20) and the short interaction time, which may not generalize to all future users.

Based on the paper, here are specific improvements I can make to AI systems and what the improved system can do:

• Improvement: Implement a feedback loop that compares the generated 3D mesh against the user's prompt using a vision-language model (e.g., CLIP-based scoring). If the semantic similarity is low, automatically re-prompt the text-to-3D service with refined keywords or request a regeneration.

• What it can do: Reduces the frustration children experienced when generated characters didn't match their intent (e.g., a capybara that looked like a rat with a tail). The system would proactively detect mismatches and offer alternatives, preserving creative flow.

• Improvement: Before rigging, use a lightweight neural network to estimate the generated mesh's limb proportions relative to the predefined skeleton. If proportions deviate beyond a threshold, automatically apply a non-uniform scaling or a retargeting step to better fit the skeleton, or suggest a different prompt to the user.

• What it can do: Prevents the tortilla stomach artifact during animation (where the character's body folds unnaturally). The system would ensure that even non-standard characters (e.g., long-limbed or squat) animate smoothly, expanding the range of creatable characters.

• Improvement: Monitor user behavior during the authoring process. If a user spends more than a threshold time (e.g., 3 minutes) on prompt refinement without making progress on the overall program, trigger a gentle, non-intrusive suggestion (e.g., You can use the character now and refine it later) or offer a set of pre-validated prompt templates that are more likely to succeed.

• What it can do: Mitigates the observed distraction where children became absorbed in prompting and lost sight of their creative goals. The system would help maintain a balance between exploration and completion, fostering a more structured creative process.

• Improvement: Integrate a monocular depth estimation model (e.g., MiDaS) alongside YOLOv11. When an object is detected, estimate its 3D bounding box using depth cues, rather than relying solely on 2D bounding boxes and raycasting.

• What it can do: Enables more accurate collision detection between the virtual character and physical objects, especially when objects are partially occluded or at varying distances. This supports more intricate interactions, such as the character stepping onto a physical book or avoiding a wall, as requested by participants.

• Improvement: Extend the zone detection to allow users to label zones with semantic meanings (e.g., dance floor, bed, danger zone) using speech. The system then uses a language model to interpret these labels and automatically adjusts the character's behavior (e.g., playing a dance animation when entering a dance floor zone).

• What it can do: Reduces the programming burden for children by allowing them to express intent in natural language rather than explicit conditional logic. For example, a child could say this zone is a pool and the character would automatically perform a swimming animation when touching it, without needing to manually program each behavior.

• Improvement: When replaying a recorded animation (from puppeteering), apply a lightweight physics-based correction (e.g., using inverse kinematics) to prevent self-intersection and unnatural folding, especially for characters with non-standard proportions.

• What it can do: Eliminates the unsettling folding artifacts observed during jumping jacks. The character's animations would look more natural, increasing immersion and reducing the cognitive dissonance that could distract from the storytelling experience.

• Improvement: Based on the user's interaction patterns (e.g., repeated failures with object detection), dynamically inject short, age-appropriate explanations about how the AI works (e.g., The AI looks for patterns it learned from many pictures. If your object looks different, it might not recognize it.). Use a large language model to generate these explanations in real-time.

• What it can do: Addresses the participants' misconceptions about AI (e.g., expecting perfect recognition regardless of brand or appearance). The system would turn errors into teachable moments, fostering AI literacy without requiring external instruction.

• Improvement: Extend the speech-to-3D pipeline to also generate sound effects and ambient environments. For example, a child could say add a rain sound or make a forest background, and the system would use a text-to-audio model (e.g., AudioLDM) and a text-to-scene model to generate these assets.

• What it can do: Directly addresses participants' desires for sound effects, animal noises, and virtual environments (e.g., a TV, a mountain). This would allow children to create richer, more immersive AR experiences without needing separate tools or advanced skills.

• Improvement: Implement a two-tier moderation system: first, use the on-device LLM for immediate screening; if the prompt is flagged as borderline, escalate to a cloud-based LLM with a more comprehensive policy. Additionally, store all generated assets locally by default, with an explicit opt-in for cloud processing.

• What it can do: Balances safety and privacy. The system would reduce the risk of inappropriate content slipping through while minimizing data exposure, making it more suitable for large-scale deployment in schools and homes.

• Improvement: Allow multiple devices to share the same AR space (using ARKit's collaborative sessions). Children can program different characters or zones that interact with each other's creations in real-time.

• What it can do: Supports social and collaborative learning, as seen in the pair-based study sessions. It would enable children to build joint stories or games, fostering teamwork and communication skills, and making the tool more engaging for classroom use.

In summary, the improved AI system would:

• Generate 3D content that more reliably matches children's intent.

• Animate characters more naturally, even with non-standard shapes.

• Guide children to stay focused on their creative goals.

• Understand the physical world more accurately for richer interactions.

• Support natural language for defining complex behaviors.

• Educate children about AI in real-time.

• Expand creative possibilities to sound and environments.

• Ensure safety and privacy at scale.

• Enable collaborative, social AR experiences.

Despite their potential to enhance children's learning experiences, AI-enabled AR technologies are predominantly used in ways that position children as consumers rather than creators. We introduce Capybara, an AR-based and AI-powered visual programming environment that empowers children to create, customize, and program 3D characters overlaid onto the physical world. Capybara enables children to create virtual characters and accessories using text-to-3D generative AI models, and to animate these characters through auto-rigging and body tracking. In addition, our system employs vision-based AI models to recognize physical objects, allowing children to program interactive behaviors between virtual characters and their physical surroundings. We demonstrate the expressiveness of Capybara through a set of novel AR experiences. We conducted user studies with 20 children in the United States and Argentina. Our findings suggest that Capybara can empower children to harness AI in authoring personalized and engaging AR experiences that seamlessly bridge the virtual and physical worlds.