FALCON: FPGA-Accelerated Latency-Certifiable Occupancy Networks for Camera-Based Autonomous Driving

summary

Video file (mp4)

The gist

The gist: A perception system that must brake for a pedestrian does not need a low average latency, but must deliver an answer before the deadline on every frame, the worst case

In short

FALCON is a camera-based 3D occupancy front-end built on an AMD Kria KR260 embedded FPGA to guarantee frame deadlines for safety-critical applications like pedestrian braking. It achieves this by layering latency guarantees: analytically bounded programmable logic, measured inference envelopes over 100k frames, and a watchdog enforcing output delivery. The system demonstrates strong accuracy (mIoU 29.88) while achieving deterministic timing under worst-case conditions.

Key concepts

Latency Determinability
This is the ability to guarantee that a computation will finish within a specific time limit, regardless of input changes or system noise. FALCON achieves this by bounding the programmable logic (PL) compute analytically and measuring inference latency over extensive traces, moving beyond just optimizing average performance.
Layered Latency Guarantees
The system enforces timing constraints through multiple mechanisms. Programmable Logic (PL) is bounded mathematically for cycle-level certainty. Inference latency is verified as a measured envelope across 100k frames. Finally, a watchdog ensures output delivery, flagging stale data if the deadline is missed.
Fixed-Trip-Count PL IPs
The system implements two specific operations—view-transform gather and bilinear resize—as fixed hardware IPs. These are numerically matched to their software references and have a known, fixed number of computational trips. This eliminates nondeterminism in these critical steps that commodity accelerators often struggle with.
Worst-Case Deadline Enforcement
Instead of aiming for low average latency, FALCON focuses on meeting the deadline for every single frame, including the worst-case scenario. It uses a fixed-work proxy of six ResNet-50 inferences per frame to ensure that across three 100k-frame runs, all 300k proxy frames delivered an output within the strict 100 ms deadline plus a small buffer.

Terminology used across episodes

This episode discusses

The paper

FALCON: FPGA-Accelerated Latency-Certifiable Occupancy Networks for Camera-Based Autonomous Driving · Read on arXiv

Haohua Que, Mingkai Liu, Handong Yao

College of Engineering, University of Georgia

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: We're looking at the paper: "FALCON: FPGA-Accelerated Latency-Certifiable Occupancy Networks for Camera-Based Autonomous Driving".

Jane: The gist: A perception system that must brake for a pedestrian does not need a low average latency, but must deliver an answer before the deadline on every frame,

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up, the paper "FALCON: FPGA-Accelerated Latency-Certifiable Occupancy Networks for Camera-Based Autonomous Driving" is all about making latency deterministic on edge hardware by explicitly bounding the compute in programmable logic and using a watchdog to enforce delivery deadlines.

Jane: The core implication is that you don't need low average latency when you have a hard deadline, especially when dealing with systems where missing that deadline is dangerous, like braking for a pedestrian one <ref:2610.22143#pg2>.

Lu: They achieved this by decomposing the unpredictability into architecture and platform sharing, and they implemented fixed-trip-count IPs for operations the DPU couldn't handle, which they numerically matched to their software references two <ref:2610.22143#pg2>.

Meng: For those of us shipping models, it shows that locking down parts of the inference graph into static operations can give you a much better guarantee than just relying on dynamic scheduling or hoping for low OS jitter two <ref:2610.22143#pg2>.

Lalam: I'm confident in the result because they show that this deterministic approach, even at a certain compute cost, gives you certifiable timing bounds over one hundred thousand frames one.

Jane: It’s about moving from just measuring performance to proving it meets a specific time constraint under worst-case conditions one <ref:2610.22143#pg2>. That's what makes this paper important for real world deployment.

Conclusion: Tom: So, this paper is about FALCON, which is an FPGA front-end for occupancy networks on camera data one.

Jane: It sounds complicated because of all that hardware talk, but really it's about making sure a perception system doesn't miss a pedestrian when it needs to brake one.

Tom: Exactly. And the authors are focusing on latency guarantees, not just speed. They’re using this AMD Kria KR260 board and turning programmable logic into something cycle-bounded one.

Jane: That’s the big shift from just trying to get a low average latency to actually proving an answer will arrive before a deadline every single time, even in the worst cases one.

Tom: Right. They use this layered approach—analytical bounding for the programmable logic, measured envelopes for inference, and then a watchdog that flags stale data if things run late one.

Jane: It means they’re tackling the problem of nondeterminism by explicitly defining where it happens, whether it's the hardware architecture or how the software runs on top of it one.

Tom: And they do this by adding two fixed-trip-count programmable logic IPs that handle things the main DPU can't manage, like view transforms and resizing one.

Jane: That’s a tangible thing for an engineer to look at—taking those hard software references and making them fixed hardware paths one.

Tom: The results are pretty strong. They got a mIoU of twenty-nine point eighty-eight which is ninety-two point eight percent of what the DPU couldn't even do on paper one.

Jane: And they showed that this setup stays within fifteen times its median performance across different hardware compared to other systems like Jetson Orin or datacenter GPUs one.

Tom: It’s a lot of data showing how you can enforce these hard time constraints on commodity edge accelerators one.

Jane: It really changes the conversation from just "how fast is it?" to "when will I get my answer?" and that's a huge difference for safety systems.

Tom: If we can make this kind of latency guarantee practical on the edge, it opens up real deployment possibilities for critical autonomous features one.

More episodes

← Home