ANEForge: Python for direct computation on the Apple Neural Engine

summary

The gist

The gist The ANEForge package provides Python for direct computation on the Apple Neural Engine, enabling users to program it directly and without CoreML, which bypasses the sanctioned runtime

In short

ANEForge is a Python package that lets users program Apple's Neural Engine directly, bypassing CoreML. It compiles a lazy tensor graph into a single ANE program and dispatches it using Apple's internal framework stack. This allows for direct access to hardware features like native fused attention, enabling high-performance inference and on-device training without relying on sanctioned runtime scheduling.

Key concepts

Apple Neural Engine (ANE): The ANE is a fixed-function neural accelerator found in recent Apple devices. Unlike general GPUs, it's designed specifically for neural network tasks. ANEForge allows Python code to talk directly to this hardware without needing the standard CoreML wrapper.
Lazy Tensor Graph Compilation: Instead of running operations immediately, ANEForge builds a 'lazy tensor graph.' This graph represents the entire sequence of mathematical operations. The package then lowers this complex structure into one single, optimized ANE program that runs on the hardware.
Native Fused Attention: This refers to a highly efficient attention mechanism built directly into the ANE hardware. Instead of breaking down attention into separate steps like matrix multiplication and softmax, ANEForge targets this native layer for maximum speed and efficiency in tasks like those found in Transformers.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 19 episodes
Federated Learning — explained in 17 episodes
Quantum Fisher Information (QFI) — explained in 16 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes

This episode discusses

The paper

ANEForge: Python for direct computation on the Apple Neural Engine · Read on arXiv

Georgia Institute of Technology

ANEForge is a Python package that programs the Apple Neural Engine (ANE), the fixed-function neural accelerator on every recent Apple device, directly and without CoreML. In production the engine is reachable only through CoreML, which treats it as a scheduling option: no configuration requires the ANE, and a model can silently run on the CPU or GPU instead. ANEForge compiles a lazy tensor graph, built from 58 fused operators and 19 native bridge operators, into a single ANE program. The program is dispatched through the same ANE daemon and kernel-driver stack as Apple's internal framework. Beyond inference, the package reaches the engine's native fused attention, streams int8, int4, and sparse weights, keeps decoder and optimizer state resident across steps, and runs the forward pass, backward pass, and optimizer update of training on the engine. A small fused program completes a call in about 90us, near the engine's 70us per-program dispatch floor, and a pretrained ResNet-18 forward runs end-to-end in 0.33ms. ResNet-18, a sentence encoder, and a Vision Transformer run end-to-end against framework references, and a Stable Diffusion U-Net validates its forward pass. ANEForge targets Apple Silicon under macOS 14 and later. Each release is verified against a recorded macOS and ANE-compiler version.

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: I'm Tom.

Jane: And I'm Jane, here with Lu, Meng and our in-house large language model, Lalam.

Tom: Today's paper: "ANEForge: Python for direct computation on the Apple Neural Engine".

Jane: The gist The ANEForge package provides Python for direct computation on the Apple Neural Engine, enabling users to program it directly and without CoreML,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: We started by looking at ANEForge: Python for direct computation on the Apple Neural Engine to see what this package actually claims it can do. The core thesis is that it lets you program the Apple Neural Engine directly and without CoreML, bypassing that runtime scheduling option entirely.

Jane: Exactly. The paper says that ANEForge compiles a lazy tensor graph into a single ANE program built from fifty-eight fused operators and nineteen native bridge operators, and dispatches it through the same private stack as Apple's internal framework <ref:2606.17090#pg1,built from 58 fused operators and 19 native bridge operators>.

Lu: It claims this provides direct access to the engine via private symbols that aren't exposed to application code, which is something previously inaccessible.

Meng: So this isn't just a wrapper; it’s a path that uses the fixed dispatch units rather than relying on heuristic selection for scheduling.

Tom: And they also highlight capabilities beyond just inference, including native fused attention and streaming int8 or int4 sparse weights directly from the engine’s dequantization path.

Jane: That weight compression is significant; they say it can reduce the weight footprint by about four times for int4 and two times for int8.

Lu: They also detail how it keeps decoder and optimizer state resident across steps, which is important for sequential tasks like autoregressive decoding or training.

Meng: I'm thinking about the deployment side here; if you can keep that state resident, it means lower memory pressure during long inference sequences on device.

Tom: The performance claims are concrete too: a small fused program completes a call in about ninety microseconds, near the engine's seventy microsecond per-program dispatch floor.

Jane: And for training, they show it can run the forward pass, backward pass, and optimizer update entirely on the engine using a fused softmax cross-entropy with an analytic half-precision stable gradient.

Lu: They validate this end to end against framework references, showing things like ResNet-eighteen and Stable Diffusion U-Net running successfully <ref:2606.17090#pg1>.

Meng: So the key claim is that this provides fixed dispatch units programmatically, which is a characterization tool for understanding engine speed and power across different workloads.

Tom: It’s about moving from a black box scheduling option to having programmatic control over how the work actually gets dispatched on the hardware.

Jane: And it also mentions that there is a public lineage, with tinygrad dispatching single operators to the engine from macOS in two thousand twenty–two thousand twenty-one but only by using specific methods <ref:2606.17090#pg2,single operators to the engine from macOS in 2020–2021, but only>.

Conclusion: Tom: So looking at ANEForge: Python for direct computation on the Apple Neural Engine, it seems like this is a way to get deep programmatic access to the hardware that’s usually hidden behind CoreML.

Jane: Right. It's about giving users a direct path to program the engine, compiling graphs and dispatching them using private symbols directly into that internal stack without needing any CoreML dependency for execution.

Lu: The paper points out that it's an instrument for characterizing the engine, exposing the dispatch paths and tooling for speed and power measurement.

Meng: It’s about moving beyond just running a model to understanding how fast the hardware really is under different load regimes, which is useful when you're trying to deploy at scale.

Tom: And it also supports on-engine training where all the data and optimizer state stay on the device, which is a substrate for research into private personalization using the lowest-power programmable compute block of the system-on-chip.

Jane: In simple terms, it’s providing Python access to direct computation that lets you build complex models that leverage specific hardware features like native attention or sparse weights in a way that's not available through the standard CoreML interface.

Lu: The main caveat is you have to remember that because it calls private symbols, each release is verified against a recorded macOS and ANE-compiler version, which means those symbols can change without notice.

Meng: That uncertainty around the API contract is a major hurdle for anyone trying to actually ship this into production environments.

Tom: So the title ANEForge: Python for direct computation on the Apple Neural Engine seems to be about giving developers a way to program this specific hardware directly, rather than treating it as just another scheduling option.

More episodes

← Home