Finding the Needle in a Haystack: Test-Time Analog Circuit Representation Adaptation for Bayesian Optimization

Affiliation: Department of Electrical and Computer Engineering, North Carolina State University, Raleigh, NC, USA

The paper addresses a fundamental limitation in representation-based Bayesian optimization (BO) for analog circuit topology search. While BO is a sample-efficient framework for analog circuit topology search where evaluating each candidate topology requires costly simulation, representation-based BO methods typically treat circuit embeddings as fixed after encoder training. This creates a mismatch between representation learning and optimization: embeddings learned to encode or reconstruct circuit structure are not necessarily organized according to the figure of merit (FoM) being optimized.

The paper introduces Test-Time Analog Representation Adaptation for Bayesian Optimization (TTARO), an online deep-kernel BO framework that adapts circuit representations throughout the search process. Starting from pretrained circuit embeddings, TTARO jointly learns a nonlinear feature transformation and a Gaussian-process surrogate using the FoM labels of the circuits evaluated so far. Following each new evaluation, TTARO updates the representation and surrogate before selecting the next candidate.

The paper compares TTARO with conventional Gaussian Process-based BO over fixed embeddings and with Deep Kernel Learning (DKL), which learns the representation only from the initial evaluated designs and keeps it fixed throughout the remainder of the search. Results show that TTARO reduces regret AUC by 15.2% on average relative to BO and by 20.7% relative to DKL across 40 encoder/kernel/acquisition settings, outperforming prior art in most settings with reductions as large as 46.7%.

The paper begins by noting that "analog circuit topology design remains a central challenge in electronic design automation because the search space is discrete, structured, and expensive to evaluate. A candidate topology must often be decoded, sized, simulated, and checked against design constraints before its quality is known. These costs make sample efficiency essential."

Recent learning-assisted approaches encode circuit topologies into latent spaces and perform optimization over the resulting representations. However, "in latent-space BO frameworks, the learned representation is usually fixed before BO begins. As a result, the optimizer must operate in whatever geometry the encoder provides, even when that geometry is poorly aligned with the target FoM."

The paper presents a UMAP visualization (Figure 1) showing that "before adaptation, high- and low-performing circuits are broadly intermixed throughout the representation space. After TTARO, the FoM exhibits substantially stronger spatial organization, with high-performing candidates concentrated within a more coherent region. This objective-aligned geometry allows the Gaussian-process kernel to assign greater similarity to circuits with comparable performance, improving surrogate generalization and enabling the acquisition function to target promising regions more effectively."

The paper's contributions are:

1. Introduction of Test-Time Analog Representation Adaptation, "an online deep-kernel Bayesian optimization framework that repeatedly learns an objective-aware transformation of pretrained circuit embeddings as new FoM observations become available. As far as we are aware, no prior art has explored this framework for analog circuit topologies."

2. Demonstration that TTARO is a general BO framework compatible with expected improvement, upper confidence bound, and Thompson sampling, as well as linear and radial basis function Gaussian-process kernels.

3. A large-scale evaluation "comprising 160 principal configurations across two public circuit-topology benchmarks, eight benchmark–encoder pairings, four representation regimes, and five kernel–acquisition settings, with each configuration evaluated over 20 random seeds. To the best of our knowledge, this is the most comprehensive evaluation ever performed concerning BO for analog circuit topology search."

The paper distinguishes between topology selection and device sizing. Device sizing has received substantial attention because design variables can be treated as continuous parameters. Bayesian optimization has been used in this setting to reduce expensive circuit simulations.

Topology search is more difficult because candidate circuits are discrete, structured, and constrained by electrical validity. Related work includes:

• Lu et al.: encode op-amp behavioral topologies as directed acyclic graphs, learn continuous embeddings with a variational graph autoencoder, and perform topology search using BO

• ATOM: defines a designer-comprehensible behavior-level op-amp design space with freeze-thaw BO

• INTO-OA: applies a Weisfeiler-Lehman graph kernel inside a GP surrogate

• CktGNN: represents circuits using a two-level graph neural network over a predefined subgraph basis

• AnalogGenie: builds a larger analog-circuit topology dataset with pin-level graphs

The paper notes that circuit representations must preserve both graph structure and circuit-relevant behavior. Generic graph encoders can capture connectivity, but analog circuits also contain directionality, functional substructures, device types, and continuous electrical characteristics. Ckt2Vec extracts frequency-domain features from device I-V curves and combines these with graph contrastive learning.

BO is attractive for analog design because it explicitly targets expensive black-box objectives with a small evaluation budget. A GP surrogate provides both predictive mean and uncertainty, while acquisition functions such as expected improvement (EI), upper confidence bound (UCB), or Thompson sampling (TS) choose the next candidate.

Related adaptive representation methods include:

• Deep Kernel Bayesian Optimization: applies DKL inside BO

• SILBO: learns a low-dimensional embedding iteratively

• LOCo: identifies collisions in learned latent spaces

• CoBO: encourages correlation between latent-space distances and objective-value differences

• LaMBO: couples a denoising autoencoder with a multi-task GP head

The paper states: TTARO brings this representation-adaptive view to finite-bank analog topology search, where pretrained circuit embeddings are adapted as FoM labels are observed.

The paper assumes an upstream topology-generation step has produced a finite candidate bank:

• **X = x1, x2,..., x N ** denotes the library of candidate analog circuit topologies

• Each candidate has an associated FoM yi = F(xi) which is unknown until evaluated

• Given an evaluation budget B ≪ N, the objective is to identify x = arg max F(xi)* using as few evaluations as possible

• Each candidate has a fixed initial representation hi ∈ R dh computed before BO begins

The paper notes: these representations are typically trained to encode circuit structure, reconstruct graphs, or capture topology semantics, not to organize circuits according to the specific FoM being optimized in the current run.

Conventional representation-based BO fits a GP surrogate directly over fixed representations: f(h) GP(m(h), k(h, h')). The paper explains: "If the encoder places two circuits nearby because they are structurally similar but their FoMs differ substantially, the surrogate can make misleading predictions; conversely, high-performing circuits that are far apart in the initial space may not share useful statistical strength."

TTARO adapts a fixed circuit representation during BO by learning a feature map from FoM observations collected so far. At iteration t, each initial circuit representation is normalized feature-wise over the candidate bank:

h̄i,q = (hi,q - h min q) / (h max q - h min q)

TTARO then maps the normalized representation into a task-adapted latent space:

zi(t) = φ θt(h̄i), zi(t) ∈ R dz

The feature map is a two-layer multilayer perceptron:

φ θt(h̄) = W2 Dropout ReLU(W1h̄ + b1) + b2

The hidden layer has width 128, and the transformed representation has dimension dz = 16.

A GP surrogate is defined over the transformed representations. The experiments use either:

• A scaled linear kernel: k linear(z, z') = σ2 f,t zTz'

• A scaled RBF kernel: k RBF(z, z') = σ2 f,t exp(-½ Σ (z q - z'q)2/l2 t,q)

The feature map and GP are fit by minimizing the negative log marginal likelihood:

L t = ½ ỹ tT K t−1 ỹ t + ½ logK t + (n t/2) log 2π

Training uses only FoM observations from circuits already selected by BO. After fitting, TTARO applies the learned feature map to the full candidate bank, computes the GP posterior over unevaluated circuits, and evaluates the acquisition function to select the next circuit.

After fitting the deep-kernel surrogate, an acquisition function is evaluated for every unevaluated candidate. The next circuit is selected according to:

i t+1 = arg max i∉I t a t(hi)

The selected circuit is evaluated, and its FoM is added to the observed set. The loop returns to retrain the feature map and GP using all FoM observations accumulated so far.

The DKL baseline uses the same deep-kernel surrogate structure, but the feature map trained only on the initial evaluated dataset. This produces an initial transformed representation bank... This transformed representation bank remains fixed for the rest of the optimization process.

TTARO learns a sequence of feature maps φ θ0, φ θ1,..., φ θ B-1 using the growing set of evaluated circuits. The comparison between DKL and TTARO therefore isolates the effect of updating the representation used by the surrogate during the BO process.

The paper explains: "The central assumption behind GP-based BO is that the kernel encodes a useful notion of similarity for the objective being modeled. In this work, the relevant objective is the circuit FoM. The kernel should therefore assign high similarity to circuits with similar FoM values and lower similarity to circuits with substantially different FoM values."

The posterior mean and variance used by the acquisition function are:

• μ t(x j) = m t(x j) + k t,jT K t−1 (y t - m t)

• σ2 t(x j) = k ψt(x j, x j) - k t,jT K t−1 k t,j

"Updating θ t changes both objects. This directly changes the posterior mean, the posterior variance, and the acquisition values used to select the next circuit. TTARO uses the accumulating FoM labels to continually revise this kernel geometry, making the surrogate increasingly aligned with the objective being optimized."

The paper evaluates TTARO on two operational-amplifier topology libraries from the Open Circuit Benchmark (OCB):

• Ckt-Bench-101: contains 10,000 valid circuit candidates

• Ckt-Bench-301: contains 50,000 candidates

The FoM is computed as: FoM = 1.2·(Gain/100) + 1.6·(PM/-90) + 10·(BW/109)

For Ckt-Bench-101, the paper evaluates CktGNN, D-VAE, D-VAE-GCN, and WL representations. For Ckt-Bench-301, it evaluates CktGNN, DAGNN, D-VAE, and D-VAE-GCN representations. The learned encoders produce 66-dimensional vectors. The WL representation has 1,838 dimensions.

1. GP (fixed-representation): uses the original embedding directly as GP input

2. DKL: feature map trained only on initial evaluated set, then frozen

3. TTARO: keeps representation learning coupled to online search

4. GP-oracle: trains feature map using all FoM labels (upper-reference condition, not deployable)

All methods use the same finite-bank BO protocol with two GP kernels (linear and RBF) and five kernel/acquisition settings: Linear/EI, Linear/TS, Linear/UCB, RBF/EI, and RBF/UCB.

Metrics include:

• Simple regret: r t = f* - f t+

• Regret AUC: discrete area under the simple-regret curve

• Final best-so-far FoM

On Ckt-Bench-101: TTARO reduces dataset-average regret AUC from 32827.6 to 26385.6, a 19.6% reduction relative to GP.

On Ckt-Bench-301: TTARO reduces dataset-average regret AUC from 19901.1 to 17480.7, a 12.2% reduction relative to GP.

"On Ckt-Bench-101, DKL improves regret AUC relative to GP, while its final best FoM remains slightly lower than GP. On Ckt-Bench-301, DKL degrades substantially: regret AUC increases from 19901.1 for GP to 23634.0, and final best FoM drops from 152.7 to 134.0. This suggests that a representation learned only from the initial evaluated set can be brittle, especially in the larger and more heterogeneous search space."

TTARO reduces regret AUC by 14.9% relative to DKL on Ckt-Bench-101 and by 26.0% on Ckt-Bench-301.

TTARO improves regret AUC relative to GP in 37 of the 40 evaluated settings. The three exceptions are:

• CktGNN/RBF/UCB on Ckt-Bench-101 (5.5% increase)

• D-VAE/RBF/EI on Ckt-Bench-101 (1.9% increase)

• D-VAE-GCN/Linear/EI on Ckt-Bench-301 (0.9% increase)

On Ckt-Bench-101, TTARO reaches 228.6 FoM by 20% of the budget, compared with 210.4 for GP and 221.2 for DKL. By 60% of the budget, TTARO reaches 252.8, already exceeding the final GP value of 247.6.

On Ckt-Bench-301, TTARO reaches 152.8 by 60% of the budget, matching the final GP value of 152.7 while still having 40% of the evaluation budget remaining.

On Ckt-Bench-101, the largest TTARO gains occur in the linear-kernel settings, with average regret AUC reductions of 33.8% for Linear/EI, 16.0% for Linear/TS, and 22.9% for Linear/UCB.

On Ckt-Bench-301, the average gains are 10.9% for Linear/EI, 9.5% for Linear/TS, 14.7% for Linear/UCB, 12.7% for RBF/EI, and 17.7% for RBF/UCB.

The oracle representation "trains the feature map using all FoM labels in the candidate bank before BO begins, giving the surrogate access to the global objective landscape... Since the oracle representation is trained using FoM labels that are unavailable during a real BO run, it is not a deployable optimization method and should be interpreted only as an upper-reference condition."

"The main limitation of TTARO is computational overhead. Retraining the feature map and GP after each evaluation is more expensive than updating a fixed-representation GP posterior, and this cost may matter for very large candidate banks or very short simulation times."

The paper suggests future improvements through structured kernel interpolation, stochastic variational deep-kernel learning, Lanczos-based variance estimation, GPU-accelerated GP inference, and online GP methods. In the analog design regimes targeted here, circuit evaluation is typically the dominant cost, making additional surrogate training acceptable when it reduces the number of poor evaluations.

The paper concludes: "the results support the central premise of this work: for representation-based circuit BO, the quality of the surrogate depends strongly on whether the kernel geometry is aligned with the objective being optimized. Fixed GP relies entirely on the initial circuit embedding, and DKL learns a transformation only once from the initial data. TTARO keeps the representation coupled to the online search process. This lets the surrogate revise its notion of circuit similarity as new FoM evidence becomes available, which leads to better acquisition decisions and improved sample efficiency."

Based on this paper, here are the specific improvements I can make to AI systems:

Improvement: Instead of using fixed embeddings or representations during optimization, I can implement a test-time adaptation loop where the representation space is continuously updated based on newly acquired objective function evaluations.

What the improved system can do: When optimizing expensive black-box functions (e.g., circuit design, drug discovery, material synthesis), the system will:

• Start with pretrained representations but not treat them as immutable

• After each evaluation, jointly retrain both the feature transformation and surrogate model using all accumulated labels

• Dynamically reshape the similarity geometry so that candidates with similar objective values become closer in the latent space

• This reduces regret by 15-20% on average compared to fixed-representation approaches, with improvements up to 46.7% in some settings

Bayesian optimization (BO) is a sample-efficient framework for analog circuit topology search, where evaluating each candidate topology can require costly simulation. However, representation-based BO methods typically treat circuit embeddings as fixed after encoder training. This creates a mismatch between representation learning and optimization: embeddings learned to encode or reconstruct circuit structure are not necessarily organized according to the figure of merit (FoM) being optimized. This paper introduces Test-Time Analog Representation Adaptation for Bayesian Optimization (TTARO), an online deep-kernel BO framework that adapts circuit representations throughout the search process. Starting from pretrained circuit embeddings, TTARO jointly learns a nonlinear feature transformation and a Gaussian-process surrogate using the FoM labels of the circuits evaluated so far. Following each new evaluation, TTARO updates the representation and surrogate before selecting the next candidate. We compare TTARO with conventional Gaussian Process-based BO over fixed embeddings and with Deep Kernel Learning (DKL), which learns the representation only from the initial evaluated designs and keeps it fixed throughout the remainder of the search. By continually incorporating newly observed FoM labels into representation learning, TTARO aligns the search space with the optimization objective as BO progresses. In our experiments, TTARO reduces regret AUC by 15.2% on average relative to BO and by 20.7% relative to DKL across 40 encoder/kernel/acquisition settings, outperforming prior art in most settings with reductions as large as 46.7%.