Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators

summary

The gist

The gist: Terastal introduces a soft real-time framework that jointly designs layer variants offline and schedules them online to reduce deadline misses for multi-DNN workloads on heterogeneous

In short

Terastal introduces a framework that combines offline design and online scheduling to improve real-time performance for multi-DNN workloads on different hardware accelerators. It creates 'layer variants' to reduce execution time on less ideal chips, uses virtual budgets to manage layer deadlines, and employs an intelligent scheduler that maps tasks and selects variants simultaneously, significantly cutting deadline misses while controlling accuracy loss.

Key concepts

Layer Variant Architecture: This involves modifying a specific layer within a neural network so it runs faster on a particular accelerator. It achieves this by reshaping the computation using transformations like space-todepth or depth-to-space, effectively changing how data flows across spatial and channel dimensions to better suit the target hardware's architecture.
Virtual Budget Distribution: The system breaks down a model's total deadline into smaller 'virtual budgets' for each individual layer. This allows the framework to assign these budgets during offline planning, ensuring that even if some accelerators are slower, the layer has a realistic time constraint based on its assigned budget.
Online Scheduler Design: The scheduler makes real-time decisions by prioritizing layers with the most urgent remaining slack (best-case slack). It jointly decides which accelerator to use and which layer variant to apply, ensuring that the most critical tasks are scheduled first while respecting their virtual deadlines and accuracy limits.
Latency Reduction on Non-Preferred Accelerators: This is achieved by designing specific layer variants that are trained independently. These variants are optimized so that the execution time of a layer on a non-preferred accelerator becomes comparable to or even lower than its execution time on the preferred accelerator, thus boosting performance where it is needed most.

Terminology used across episodes

Reinforcement Learning (RL) — explained in 25 episodes
Sample Complexity — explained in 18 episodes
Federated Learning — explained in 17 episodes
Retrieval-Augmented Generation (RAG) — explained in 16 episodes
Interpretability — explained in 15 episodes
Quantum Fisher Information (QFI) — explained in 15 episodes

This episode discusses

The paper

Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators · Read on arXiv

Sing-Yao Wu, Fengshuo Song, Eli Bozorgzadeh

Department of Computer Science, University of California, Irvine

Transcript

Introduction to the show: ident: AI Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Today's paper: "Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators".

Jane: The gist: Terastal introduces a soft real-time framework that jointly designs layer variants offline and schedules them online to reduce deadline misses for multi-DNN workloads on heterogeneous accelerators.

Tom: First, who's behind it and why it matters.

Paper summary: Tom: So wrapping up "Terastal: Layer-Variant-based Scheduling for Real-Time Multi-DNN Workloads on Heterogeneous Accelerators", it's a framework that combines offline design, variant construction, and online scheduling to handle multi-DNN workloads on heterogeneous accelerators.

Jane: The authors introduce layer variants as customized layer implementations that reduce latency on non-preferred accelerators and then build a budget-guided framework for constructing those variants.

Lu: And finally, they design an online scheduler that uses layer-wise slack to jointly choose the accelerator mappings and variant usage, making sure to respect accuracy constraints during runtime.

Meng: For practical implementation, this suggests we can design specialized layers for specific hardware targets before deployment and then let the scheduler handle the dynamic runtime decisions based on real-time slack measurements.

Tom: It addresses how large layer-latency differences across accelerators limit scheduling flexibility and increase deadline misses under skewed workloads by optimizing both mapping and variant selection simultaneously.

Jane: The main implication is a more robust approach to achieving soft real-time performance for complex AI inference tasks running on diverse, heterogeneous hardware setups.

Lu: It shows how fine-grained, layer-granularity scheduling can be powerfully augmented by customized computation rather than just relying on static mapping choices.

Tom: We'll keep an eye on how this approach scales when we move from a few models to a much larger collection of concurrent DNN tasks across many devices.

Conclusion: Tom: So, Terastal is about taking multiple deep neural networks that need to run at specific deadlines on different kinds of hardware, like GPUs or NPUs, and making sure they don't miss those deadlines.

Jane: Right. The authors are looking at how to manage that complexity by combining three things: designing specialized layer versions, figuring out how much time each layer gets, and then scheduling everything online.

Lu: What’s really interesting is the idea of these layer variants—basically making a layer run faster on one type of chip by changing how the math flows, like using space-todepth transformations.

Meng: From an engineering side, that means you can tailor the execution path for a specific accelerator without having to rewrite the whole model from scratch every time.

Lalam: And thinking about that customization, it could eventually mean we can deploy AI on devices with much lower power constraints by making the hardware utilization much more precise.

Tom: So why does this matter right now? Because as models get bigger and we have more diverse chips, the old scheduling methods just don't keep up with those huge latency differences between accelerators.

Jane: Exactly. The paper shows that they use virtual budgets assigned offline, which tells the system how much time each layer *should* get to finish its job.

Lu: And then online, the scheduler uses those budgets and checks available hardware to pick the best combination of variant and chip for every layer running right now.

Meng: The results show a significant drop in deadline misses—like forty percent or thirty percent better than just using standard scheduling techniques like FCFS or EDF.

Tom: And they didn't sacrifice too much accuracy; the average loss was quite small, around two to three percent, which is pretty good for real-time systems.

Jane: It’s a solid framework because it tackles the problem at three levels simultaneously: design, budgeting, and scheduling.

Lalam: The vision here is that this isn't just about faster inference; it’s about creating a more resilient AI system capable of meeting strict timing guarantees in unpredictable environments.

Tom: So the big picture is that we can move toward truly reliable, real-time AI deployment on whatever hardware you have available, rather than just hoping the scheduling works out okay.

Jane: It’s a lot of heavy lifting done offline to get a much smoother ride online for those demanding workloads.

Lu: The future work would probably look at how this framework handles even more complex dependencies or maybe extends these layer variants to include different types of neural network operations.

More episodes

← Home