A Framework for Designing Reward Functions: From Objectives to Features to Human-Aligned Reward Functions

The paper presents a formal, three-step framework for designing human-aligned reward functions in reinforcement learning, aimed at enabling non-experts to instantiate and iterate on reward functions that adhere to a given preference ordering over trajectories. The framework takes a task described in natural language and produces a linear reward function.

The three steps are: (i) distill the task description into a set of fundamental objectives and derive measurable outcome variables that capture those fundamental objectives; (ii) select a causally representative, low-cost subset of outcome variables as the reward terms; and (iii) fit weights to those reward terms via preference elicitation.

The first contribution (Section 3.1) is a guided workflow for distilling fundamental objectives from an initial task description and deriving measurable outcome variables from them. This involves iteratively asking why each objective is important to progress towards more fundamental ones, and then introducing measurable proxies (outcome variables) for objectives that are not directly measurable. The paper provides examples, such as converting minimize time to trip duration (seconds) and minimize passenger discomfort to passenger satisfaction (scalar value). It also addresses potential issues like no measurable proxy existing, proxies not being observable during training, and proxies being gameable, with corresponding remedies.

The second contribution (Section 3.2) formalizes the selection of a low-cost, causally representative subset of outcome variables as a minimum-cost partial cover problem on a causal DAG, solved via reduction to max-flow. The paper defines a valid cover as a set S that covers every demand node, where coverage is recursive: a node is covered if it is in S, or if it is not a source node and every parent is covered. The Minimum Cost Partial Cover (MCPC) problem asks for a valid cover minimizing the sum of node costs. The paper proves (Theorem 2) that MCPC equals the minimum weight vertex cut separating all sources from all demand nodes, and reduces this to a standard minimum s-t cut problem via node splitting. The algorithm (Algorithm 1) constructs a flow network with node splitting, computes maximum flow, and returns the set of nodes corresponding to the minimum cut. Runtime is bounded by max-flow algorithms: Edmonds-Karp with O(V'E'2) and Dinic's with O(V'2E'). A locally greedy alternative is also provided for cases where global cost or causal information is incomplete.

The third contribution (Section 3.3) fits weights to the selected reward terms via preference elicitation, framed as a convex feasibility problem. The paper assumes a linear reward function, justified by Ng & Russell (2000) and the need for convex optimization concepts. The weight vector is identified with a point on the unit sphere S n-1 due to scale-invariance. Each preference query yields a half-space constraint on the weight vector, and the feasible region is iteratively narrowed. The approach uses separation oracle methods, specifically the analytic center cutting plane method (ACCPM) and the volumetric center method, both requiring O(n log κ) oracle calls where κ = nR/ϵ. The paper highlights benefits of synthetic trajectory generation: consistency (guaranteed conflict-free feasible region), complexity guarantees, and controllability (e.g., rounding outcomes for human readability).

The paper concludes that these steps address three recurring failure modes in reward design: redundancy (mitigated by causal cover), reward hacking (mitigated by grounding reward terms in elicited objectives rather than shaped intermediate behaviors), and preference misalignment (mitigated by producing a consistent feasible weight region by construction).

Improvements to AI Systems:

1. Causal Reward Decomposition Engine: An AI system that automatically parses natural-language task descriptions, generates a causal DAG of measurable outcome variables, and applies the minimum-cost partial cover algorithm to select only causally necessary, non-redundant reward terms. This eliminates reward hacking by removing proxy variables that are gameable or causally downstream of true objectives.

2. Interactive Preference-Consistent Reward Tuner: An AI system that, after selecting reward terms, runs a cutting-plane method (ACCPM or volumetric center) to maintain a convex feasible region of weight vectors. It generates synthetic trajectory pairs for preference queries, guarantees a non-empty feasible region by construction, and outputs a weight vector that provably satisfies all elicited preferences within an ε-tolerance. This prevents preference misalignment by ensuring the final reward is consistent with all human feedback.

3. Non-Expert Reward Design Assistant: An AI system that guides users through the three-step framework via a conversational interface. It asks why iteratively to distill fundamental objectives, suggests measurable proxies when objectives are unobservable, flags proxies that are not observable during training or are gameable, and automatically proposes a minimal-cost valid cover. The system then runs preference elicitation with synthetic trajectories, presenting the feasible weight region visually and allowing users to refine preferences until convergence.

4. Self-Auditing Reward Function Validator: An AI system that, given an existing reward function, reconstructs its causal DAG, checks whether the reward terms form a valid cover of all demand nodes, and identifies redundant or missing terms. It then recomputes the minimum-cost cover and re-fits weights via preference elicitation, outputting a corrected reward function with a certificate of causal representativeness and preference consistency.

5. Low-Cost Causal Cover Optimizer for Continuous Learning: An AI system that, in lifelong learning settings, incrementally updates the causal DAG and re-solves the minimum-cost partial cover problem using max-flow re-optimization. It dynamically adds or removes reward terms as new objectives emerge, maintaining a minimal-cost, causally representative reward set without full recomputation, enabling real-time adaptation to changing human preferences.

We present a formal process to enable non-experts to instantiate and iterate on human-aligned reward functions, i.e. reward functions that adhere to a given preference ordering over trajectories. Given a task described in natural language, our process produces a linear reward function in three steps: distill the task's objectives into a set of fundamental objectives and derive measurable outcome variables that capture those fundamental objectives, select a causally representative subset of outcome variables as the reward terms, and fit weights to those reward terms via preference elicitation. Our contributions describe the first step and formalize the latter two steps. The first is a guided workflow for deriving outcome variables. The second is a reduction of reward term selection to minimum-cost partial cover on a causal DAG, solved in polynomial time via max-flow. The third is a geometric framing of weight fitting as a convex feasibility problem iteratively narrowed by preference queries, solved by existing separation oracle methods. To the best of our knowledge, this is the first reward-design method that maintains a deterministically conflict-free feasible weight region, narrowed to a desired tolerance via a separation oracle with O(n log kappa) preference queries.