2608.07091-Human-Centered Explainable AI for TinyML Edge Devices: A Pareto-Based Selection Framework with LLM-Guided Design

Introduction to the show: ident: Paper Radio. Generated commentary on the latest Artificial Intelligence papers.

Tom: Next we'll be talking about the paper "Human-Centered Explainable AI for TinyML Edge Devices: A Pareto-Based Selection Framework with LLM-Guided Design".

Jane: The paper was written by Zeinab Dehghani, Dhavalkumar Thakker, Koorosh Aslansefat, Kuniko Paxton, Bhupesh Kumar Mishra et al. from University of Hull.

Tom: Stay tuned as we take you through the paper and discuss its implications.

Paper summary: Tom: So we've got a fascinating one today — a paper from the University of Hull, all about explainable eye on tiny devices.

Jane: TinyML, right? We're talking microcontrollers with kilobytes of memory running eye models, not data centers.

Tom: Exactly. And the paper tackles a real headache: when you put eye on these tiny devices in healthcare, how do you explain what the model is doing?

Jane: Because clinicians need to trust the model before they act on its advice. But most explainability methods are too heavy for a microcontroller.

Lu: Right. The authors frame it as a multi-objective design problem. You've got explanation quality, stability, and deployment cost — and they trade off against each other.

Tom: They use LLMs to help generate candidate explainability methods from stakeholder preferences. Then deterministic filtering and Pareto optimization do the heavy lifting.

Jane: So it's a hybrid approach — LLM for the creative part, math for the rigorous part.

Meng: And they test it on a skin lesion classification task with HAM10000. The model runs on MobileNetV3-Small, which is designed for constrained devices.

Lalam: What I find compelling is the human-centered angle. They define three goal profiles — Clinical, TinyML, Balanced — and each has different priorities.

Meng: Clinical cares about fidelity and stability, TinyML cares about deployment cost, Balanced is in between.

Tom: The key finding? A simple method called CAM — Class Activation Mapping — came out on top across all three profiles.

Jane: Which is kind of beautiful, honestly. The simplest method won.

Tom: But there's nuance — some methods achieved perfect stability scores while being almost useless for explanations.

Lu: That's the classic trap. You optimize for one metric, and you get something that looks good on paper but fails in practice.

Lalam: And the paper makes a strong case that Pareto analysis reveals these trade-offs systematically.

Jane: The final selection was also LLM-constrained — it could only pick from Pareto-optimal configurations, with deterministic validation.

Meng: That's the auditable part. LLM suggestions, but hard constraints enforced by rules.

Tom: One thing to flag — the authors say physical deployment on actual MCU hardware is future work. This is a proof of concept.

Jane: Good. We should keep that caveat in mind as we dig into the pages.

Lalam: The framework is the contribution, not the deployment results. That's where the value is.

Tom: Alright, let's start with page one and see how they set up the problem.

Page 1: Jane: Page one is all about motivation — why should we care about explainable eye on tiny devices?

Tom: And the answer is in healthcare. Edge eye protects patient privacy and enables real-time processing. But you need explanations for trust.

Lu: The authors point out a mismatch — most Xeye methods assume rich computing environments. Activation and gradient methods like Grad-CAM need internal model access.

Jane: And perturbation methods like LIME require multiple rounds of inference. On a microcontroller, that's brutal.

Meng: There's also the stability problem. Hardware heterogeneity and environmental variability can make explanations inconsistent over time.

Tom: The authors use this great phrase — the choice of explanation method often depends on the developer's experience and judgment.

Jane: Which is code for "we're guessing." And that's dangerous in a clinical setting.

Lu: And they mention that Xeye isn't just a model analysis tool — it's a decision-support interface. The quality and presentation of explanations affect user trust and reliance behavior.

Meng: Inappropriate explanations can lead to over-reliance or erroneous judgments. So the stakes are high.

Tom: The key framing on this page: selecting an Xeye method in a TinyML environment means considering hardware constraints, explanation quality, task objectives, and human interpretability requirements — all at once.

Jane: That's a lot of factors. And they say evaluating all of them places a significant burden even on experts.

Lu: That's where LLMs come in. They can act as an interface between human intent and system design — exploring and ranking candidate solutions.

Tom: But the authors are careful — LLMs do the exploration, while deterministic procedures handle optimization and feasibility enforcement.

Meng: So LLMs propose, math disposes. I like that division of labor.

Lalam: The important thing here is they're not just slapping an LLM on the problem. They keep the safety-critical parts deterministic.

Tom: And they formulate selection as a constrained multi-objective optimization problem. That gives you trade-offs between fidelity, stability, and deployment cost.

Jane: Speaking of which — maybe we should take a look at how this relates to prior work.

Page 2: Jane: Page two reviews the literature — and it's a smart way to position the paper.

Lu: They start with Xeye in TinyML. Most prior work used Xeye for design support — pruning, hardware optimization, robustness enhancement.

Tom: But not for explaining predictions to end users. There's a gap between "Xeye as a developer tool" and "Xeye as a patient-facing interface."

Meng: Some studies do generate explanations on edge devices. But they often lack analysis of computational costs and hardware constraints.

Jane: And when they do consider hardware, the selection is developer-driven. Stakeholder preferences barely get a mention.

Lu: Exactly. It's a technical problem in the literature, not a human problem. That's the gap.

Tom: Then there's human-centered explainability. Studies show explanation quality influences trust and reliance — sometimes leading to over-reliance or under-reliance.

Meng: The authors make an important distinction — trust is not synonymous with reliance. You need validated evaluation measures.

Lalam: In medical applications, explanations should align with domain knowledge and support clinically meaningful reasoning. But existing work treats human factors and system constraints separately.

Jane: Which limits real-world applicability. The authors are really hammering that point.

Tom: And the selection frameworks — multi-criteria decision-making, property-oriented approaches, taxonomies. They're mostly for developers or data scientists.

Lu: Manual evaluation, conceptual frameworks, limited empirical involvement of domain experts. The paper says this plainly.

Meng: One recent study used LLMs for Xeye design on small MCUs, but even that was driven by hardware and model performance — not domain-user requirements.

Jane: So the authors' contribution is to fill that void: stakeholder-oriented goal profiles, LLM-guided candidates, deterministic feasibility, Pareto-based selection, and a proposed human review stage.

Tom: And they're honest — the present proof-of-concept evaluates the computational stages. Human-expert review remains future work.

Lalam: That's a strong positioning. They know exactly what they've demonstrated and what's left.

Lu: The four contributions are clear: stakeholder-oriented formulation, LLM-guided candidate generation, Pareto-based selection, and transparent auditable comparison.

Jane: Let's see how those contributions actually get implemented in the method.

Page 3: Tom: Now we get into the architecture of the framework. The problem definition stage starts with an interesting design choice.

Jane: They separate the model specification from stakeholder intent. That's deliberate — it prevents qualitative requirements from being conflated with device-specific constraints.

Lu: The stakeholder intent is encoded through three predefined goal profiles: Clinical, TinyML, and Balanced. Each record includes decision context, intended users, explanation goals, risk tolerance, and safety considerations.

Tom: Then deterministic rule-based mappings translate those profiles into policy constraints. So the profiles aren't just vibes — they become enforceable rules.

Meng: The knowledge source stage has two components: an Xeye method knowledge base and a hardware profile library.

Jane: The method knowledge base records properties like method family, execution scope, forward passes, and analytical resource proxies.

Lu: Proxies, not direct measurements. They're used for relative comparison and feasibility screening. That's an important caveat.

Tom: The hardware library represents the target environment — supported precision, SRAM and Flash capacity, memory reservations. Given that, you can calculate available headroom for XAI.

Meng: Then the LLM-guided design stage. The LLM receives the model spec, the goal profile, and a structured catalog summary.

Jane: But crucially — no measured latency, energy, SRAM, or Flash values. The LLM can't hallucinate resource estimates because it doesn't see them.

Lu: It returns a ranked shortlist of at most five methods, aligned with the qualitative profile.

Tom: Then the feasibility enforcement stage kicks in. Deterministic filtering checks SRAM and Flash limits, execution scope, forward-pass limits.

Jane: Host-only methods are never classified as MCU-feasible. If host fallback is allowed, they can remain as hybrid alternatives. Otherwise, they're rejected.

Lalam: This separation of responsibilities is the core insight. LLMs for design creativity, deterministic rules for safety-critical constraints.

Tom: The knowledge base and hardware library are the structured memory of the system — they ground the whole process.

Meng: And the feasibility indicator is a binary — satisfies all constraints or not. No grey zone.

Jane: The MCU-feasible set is defined as those methods that pass. Then the validation stage begins — measuring fidelity and stability.

Tom: Which we'll get into on the next page. This is where the quantitative evaluation starts.

Page 4: Jane: Now we're at the validation stage. This is where the framework measures explanation quality.

Tom: Two complementary perspectives: attribution fidelity and explanation stability.

Lu: Fidelity is evaluated using deletion AUC, insertion AUC, and AOPC — Area Over the Perturbation Curve.

Meng: For the skin lesion classification task, they fix the evaluation class to the predicted class from the unmodified input. Then they perturb the image.

Jane: Deletion progressively replaces the highest-ranked patches with a blurred baseline. If the model score drops fast, the explanation is good.

Tom: Insertion is the reverse — start from the blurred baseline and restore the highest-ranked regions. If the score rises fast, that's good.

Lu: The heatmaps are min-max normalized and partitioned into 16×16 patches. For a 224×224 input, that gives 196 patches.

Meng: They use 20 perturbation intervals rather than doing every patch individually. That's a computational efficiency win.

Tom: AOPC measures the average reduction in the original predicted class logit across the deletion intervals. Higher is better — removing important regions should hurt the score.

Jane: Then they combine the three metrics into a composite fidelity score with equal weights.

Lu: Deletion AUC is inverted because lower is better. So the composite is one-third normalized insertion, one-third normalized AOPC, one-third inverted normalized deletion.

Tom: Then stability. They generate three mildly perturbed versions of each input — reflection padding, random crop, and Gaussian noise.

Meng: Brightness and contrast transformations are not used. And they don't guarantee label preservation — these are just mild variations.

Jane: SSIM — Structural Similarity Index Measure — compares the heatmaps from original and perturbed inputs. Higher SSIM means more stable explanations.

Tom: But the authors immediately add a warning — strongly compressed or nearly invariant heatmaps can get high SSIM.

Lu: That's a key caveat. Stability alone isn't enough. A blank heatmap is perfectly stable and completely useless.

Meng: So fidelity and stability are always considered jointly. The paper is really careful about this.

Jane: Then deployment overhead. They use runtime and SRAM proxies, not direct energy measurements.

Tom: The runtime proxy includes the shared model forward pass, base CAM construction, and configuration-specific post-processing.

Lu: The SRAM proxy comes from analytical metadata — estimated tensor storage for the base method. Not parameter-specific measurements.

Meng: And the relative deployment cost combines runtime and SRAM with weights 0.85 and 0.15, prioritizing runtime.

Jane: So now we have three objectives: maximize fidelity, maximize stability, minimize deployment cost.

Tom: That sets up the Pareto optimization — which we'll see on the next page. But this validation stage is already rich with technical decisions.

Page 5: Tom: Now we're into the Pareto optimization. This is where the framework finds the non-dominated configurations.

Jane: Let me simplify that — a configuration dominates another if it's better on at least one objective and not worse on any of the rest.

Lu: So the Pareto set contains all configurations where nothing else dominates them. These are the trade-off frontiers.

Tom: The deployment cost proxy combines normalized runtime and SRAM. Lower is better. Fidelity and stability are maximized.

Meng: Within each stakeholder profile, the runtime proxy is normalized over configurations with timing values. SRAM over configurations with analytical metadata.

Jane: Then Pareto filtering is restricted to configurations with complete objective values. You need all three to participate.

Tom: That completeness requirement matters — some methods had missing data in some profiles, so they were excluded.

Lu: After finding the Pareto set, they do a Pareto-constrained final selection. Only Pareto-optimal configurations can be considered.

Meng: They normalize fidelity, stability, and deployment cost again within the Pareto set. Then compute a goal score with profile-specific weights.

Jane: The weights are fascinating. For Clinical, fidelity gets 0.45 and stability 0.40, with cost only 0.15.

Tom: For TinyML, cost gets 0.60. It's the reverse priority.

Lu: Balanced is almost even — 0.34/0.33/0.33.

Meng: Then GPT-4.1 mini receives the ranked Pareto configurations and must select within strict constraints.

Tom: For Clinical, it must select exactly one primary configuration with the highest fidelity in the Pareto set.

Jane: For Balanced and TinyML, it selects two primaries and one fallback. The anchor must be max-fidelity, lowest-cost. The complementary primary needs fidelity ≥0.70 and better stability than the anchor.

Lu: The fallback must provide higher fidelity than the complementary primary and stay within the same cost interval.

Meng: And the LLM's response is validated deterministically. If it fails, it goes back with the errors for correction.

Tom: That's the auditable part — the LLM proposes, but the constraints are checked mechanically.

Jane: If a valid response isn't obtained, the procedure terminates without reporting a selection. That's a safety-critical design choice.

Lu: Then the proposed human review stage — medical experts would review the Pareto-valid explanations for clinical relevance and artifacts.

Tom: But that stage was not implemented in this proof of concept. It's explicitly future work.

Meng: So the system can be fully transparent and auditable, even before the human-in-the-loop stage is added.

Jane: Let's move to the experimental setup — how they actually tested this framework.

Page 6: Jane: So how did they test all this? The experimental setup is on page six.

Tom: HAM10000 dataset — 10,015 dermoscopic images across seven diagnostic categories.

Lu: Stratified split with a fixed seed of 42 — 70 percent train, 10 percent validation, 20 percent test.

Meng: MobileNetV3-Small is the backbone, because it's designed for constrained environments.

Tom: They use a two-stage transfer learning approach. First, frozen backbone with a learning rate of 1e-3. Then the last 40 percent of the backbone layers unfrozen at 3e-5.

Lu: Class weighting and label smoothing handle the class imbalance. Augmentation includes horizontal flip, rotation, zoom, and contrast factor.

Jane: Now the interesting part — 67 parameterized Xeye configurations. Seven families: CAM, Tiny-Saliency, LR-CAM, Micro-CAM, Binary-CAM, TopK-CAM, and TopK×Binary.

Tom: The shared-forward execution model is clever. One model inference returns the final convolutional activation tensor and class logits. Then configuration-specific post-processing generates the variants.

Meng: Tiny-Saliency also uses the shared activation tensor — no additional inference needed.

Jane: The hardware profile is a generic Cortex-M7 with 512 kB SRAM and 2 MB Flash. After reserving memory for the system, stack, and model, they calculate headroom.

Tom: The numbers work out to 184 kB of SRAM and 548 kB of Flash available for XAI.

Lu: Those are used for deterministic screening, not measured on physical hardware. The paper is very clear about that.

Meng: Runtime measurements come from a batch of 16 test images, with two warm-up runs and five repetitions.

Jane: Then they compare the LLM proposals — GPT-4.1 mini and Gemini 2.0 Flash — across the three stakeholder profiles.

Tom: And the results start on the next page, so let's get to them.

Page 7: Tom: Results time! And the first thing that jumps out — both LLMs proposed the same five methods for the TinyML profile.

Jane: Same order too: CAM, LR-CAM, Binary-CAM, TopK-CAM, Micro-CAM. All passed feasibility.

Lu: For Clinical, both ranked LR-CAM first. But GPT-4.1 mini proposed four on-device methods, while Gemini proposed three host-only methods.

Meng: After feasibility filtering, GPT-4.1 mini had four on-device methods left, Gemini had two.

Tom: Balanced profiles were similar but with differences — GPT-4.1 mini proposed Binary-CAM, Gemini proposed Micro-CAM.

Jane: So the LLMs agreed on TinyML but diverged on the other profiles. The deterministic filtering then trimmed accordingly.

Tom: For runtime — the CAM baseline was around 45 milliseconds per batch of 16 images. Binary-CAM added negligible overhead, about 0.005 to 0.011 milliseconds.

Lu: Micro-CAM s=5 was the biggest overhead at 0.203 ms. Interesting — the compressed version took more post-processing time.

Meng: The shared forward pass dominates the total runtime. Post-processing is a small fraction.

Jane: Now the fidelity results. The highest composite fidelity was 0.9397, achieved by CAM, LR-CAM with d=1, and Micro-CAM with sizes 7-10.

Tom: But no single configuration dominated across all fidelity metrics. TopK-CAM with k=0.46 had the lowest deletion AUC and highest AOPC, but lower composite fidelity.

Lu: And the stability numbers reveal the trap. Micro-CAM s=1 hit SSIM of 1.000 — perfect stability — but composite fidelity dropped to 0.0502.

Meng: That's nearly a blank heatmap. Perfectly stable, completely uninformative.

Tom: Binary-CAM and TopK-CAM showed the same pattern — stronger sparsification increases stability while reducing fidelity.

Jane: So the paper's warning is validated: high SSIM should not be interpreted independently as quality.

Lu: Fidelity and stability must be read together. Otherwise you're choosing a model that explains nothing.

Tom: And those trade-offs are exactly what the Pareto analysis is designed to expose.

Page 8: Jane: The Pareto analysis — this is the heart of the paper. Let's walk through the numbers.

Tom: They had 55 eligible configurations under TinyML, 45 each under Clinical and Balanced.

Lu: After three-objective Pareto filtering: 16 retained for TinyML, 12 for Clinical, 13 for Balanced.

Meng: CAM appeared in all three Pareto sets with the lowest deployment cost of 0.150 and the highest fidelity of 0.9397.

Jane: It's the reference configuration. Low cost, high fidelity, decent stability around 0.82.

Tom: Binary-CAM variants occupied the low-cost region, with thresholds trading fidelity for stability.

Lu: TopK-CAM sat in an intermediate cost region. Lower retained ratios gave higher stability but lower fidelity.

Meng: Micro-CAM only appeared in the TinyML Pareto set — its completeness records were only available there.

Tom: And the LR-CAM configurations with SSIM of 1.000 — the paper flags them clearly. They're in the Pareto sets but with fidelity of only 0.0502.

Jane: So Pareto membership alone doesn't guarantee good explanations. The authors really want that point to land.

Tom: Then the final selection. GPT-4.1 mini selected CAM for Clinical — highest fidelity, lowest cost.

Lu: For Balanced and TinyML, it selected CAM as the anchor and Binary-CAM with τ=0.60 as complementary.

Meng: Binary-CAM τ=0.50 was the fallback — higher fidelity than the complementary primary, same cost region.

Jane: All selections passed deterministic validation on the first attempt. No corrective retry needed.

Tom: That demonstrates the LLM respecting the constraints. But it's a single run — sensitivity analysis across repeated runs is future work.

Lu: The paper also notes the absence of Micro-CAM in Clinical and Balanced shouldn't be read as dominance-based rejection — it was a data completeness issue.

Meng: Right. That's a subtle but important distinction.

Jane: The Pareto sets shared a common low-cost structure but weren't identical across profiles.

Tom: And no single configuration simultaneously optimized all three objectives. That's the whole point of Pareto analysis.

Lu: Now we should look at the discussion and conclusion — where the authors step back and reflect.

Conclusion: Tom: Wrapping up. The paper showed how to combine LLMs with deterministic optimization for Xeye selection on TinyML devices.

Jane: And the key lesson — stability-only evaluation is dangerous. Perfect SSIM can hide an explanation that says nothing.

Lu: The Pareto framework captures real trade-offs. CAM was the simple, high-fidelity champion; Binary-CAM added stability at modest cost.

Meng: The LLMs were useful — they proposed candidate methods and made final selections. But every step was subject to deterministic validation.

Tom: The authors were careful about their claims. These are computational results. Physical MCU deployment and clinical evaluation are explicit future work.

Jane: The human-expert review stage — dermatologists evaluating the heatmaps — is proposed but not yet implemented.

Lu: And the runtime measurements came from the experimental environment, not from actual Cortex-M7 hardware.

Meng: So the framework is a proof of concept. Traceable, auditable, and honest about its limits.

Lalam: The significance is in the architecture — a division of labor between generative eye and rule-based enforcement.

Tom: LLMs for creativity, deterministic logic for safety. That's a model for many high-stakes eye applications.

Jane: And the clinical context makes it meaningful. Skin lesion classification on constrained devices is a real-world need.

Lu: The HAM10000 dataset with MobileNetV3-Small — a realistic combination. Not a toy example.

Meng: The Pareto analysis revealed that simple methods can outperform complex ones when you balance all objectives.

Tom: CAM won across all three profiles. Sometimes the classic approach is the right one.

Jane: The paper gives us a systematic way to discover that rather than relying on developer intuition.

Lalam: That's the contribution — making the selection process transparent, reproducible, and stakeholder-aware.

Tom: We'll be thinking about this framework as more work comes out on TinyML explainability.

Jane: Especially once physical deployment results arrive. That's the next big step.

Lu: And the LLM sensitivity analysis across repeated runs — also worth watching.

Meng: For now, the paper stands as a rigorous, honest framework for a hard problem.

Tom: Great discussion, everyone. Let's move on to the next paper.