Quantum control of solid state qubits via Reinforcement Learning

Introduction to the show: ident: Quantum Radio. Generated commentary on the latest quantum physics and condensed matter papers.

Kai: I'm Kai.

Mira: And I'm Mira, here with Lev.

Kai: Today's paper: "Quantum control of solid state qubits via Reinforcement Learning".

Mira: The gist The use of reinforcement learning (RL) in the design of quantum control strategies for pulse level control of NISQ era quantum devices is attracting growing interest.

Kai: First, who's behind it and why it matters.

Paper summary: Mira: To summarize what this paper is about, it investigates the use of model-free RL to design pulse sequences for controlling nuclear spins in an NV center while accounting for its microscopic spin-bath.

Kai: They set up a system where the NV center register involves the electronic spin and 13C nuclear spins coupled to that electronic one, and they describe the full Hamiltonian as HR plus HB plus HI.

Lev: The modeling of the spin-bath is done using the generalized cluster correlation expansion, or gCCE technique, which lets them go beyond a simple mean field treatment of the bath.

Mira: They simulate this by calculating the n-th order gCCE contribution to the density matrix by tracing out bath spins, where their Hamiltonian entering that free evolution propagator contains H(E0) = HR plus H(E0)B,m plus H(E0)I,m.

Kai: This simulation process is what they use to model the open quantum dynamics and then reward the RL agents based on those simulated outcomes.

Lev: The RL agent itself acts on this environment by applying mw pulses to the NV electronic spin to perform state preparation or gate design tasks.

Mira: The action space for these agents is continuous, parameterized by things like idle time, pulse duration, and pulse phase, which makes the Proximal Policy Optimization algorithm a natural choice.

Kai: They train these agents using sparse reward strategies where rewards are only assigned at the end of an episode to accomplish the task.

Lev: For state engineering, they estimate state fidelity F = Tr(ρρ˜) by giving a single-shot reward based on one/dTrρPine(in)m over Pr(in) <ref:2610.21987#pg1>.

Mira: For gate engineering, they use process fidelity F = Tr(U†V)two/d2 which can be estimated using either full direct fidelity estimation or Hofmann classical fidelity bounds.

Kai: They also formulate the zero-fidelity estimator as a single-shot reward strategy, which is another way to get a measure of performance.

Lev: They use domain randomization at two levels: randomizing the state configuration of the spin-bath within the gCCE and randomizing physical parameters like Rabi frequency or coupling constants.

Mira: Domain randomization helps produce robust policies that should remain effective when transferred to real world experimental setups, which is important for simulation-to-real transfer.

Kai: They pre-train an agent in a simulated environment and then fine-tune it with single-shot fidelity rewards in different environments as a blueprint for how to use it experimentally.

Lev: The benchmarking involves comparing the RL agents against black-box optimization methods like Bayesian Optimization or Evolution strategies such as CMA-ES.

Mira: They often use a five hundred-shot fidelity estimate to emulate the completely mixed state of the spin-bath when comparing agents A(ss) and B(ss).

Kai: The results show that RL agents can discover qualitatively different high-fidelity protocols and reach parameter space regions that other quantum optimization strategies haven't explored.

Lev: They conclude that these RL agents can learn to cope with the presence of a non-Markovian spin-bath and find solutions robust to uncertainty in the theoretical modeling of the environment.

Conclusion: Mira: So looking at the paper, "Quantum control of solid state qubits via Reinforcement Learning," it really centers on showing that RL agents can learn to handle a non-Markovian spin-bath and find solutions that are robust even when the theoretical model has some uncertainty.

Kai: The main point is that combining domain randomization during training with fine-tuning at deployment and using single-shot fidelity rewards creates a blueprint for actually using these agents in real experimental situations.

Lev: What this means practically is that we have a pipeline where you can train an agent in simulation, and then use it to find high fidelity pulse sequences for things like preparing a Bell state between two 13C nuclear spins using the NV center as a bus.

Mira: The paper suggests these RL agents can find high fidelity pulse sequences for this task, which is something that other black-box optimization methods haven't achieved even without a spin-bath present.

Kai: Overall, the implication is that reinforcement learning offers a way to tackle the hard problem of finding control strategies in complex, noisy solid-state quantum systems where traditional methods might struggle.

Lev: It points toward potential applications in real solid-state qubit devices because the agents learn to adapt to physical realizations outside of their exact training distribution.