Training Agents at the Frontier of AI Industry

Master Engineering Task: Build an Evaluation-Driven Enterprise Agent Optimization and RL Pipeline

ROLE
Act as a Staff AI Systems Engineer, Reinforcement Learning Research Engineer, and ML Platform Engineer.

I am building production-grade agentic AI systems for enterprise business use cases. I want to
systematically improve business outcomes by evaluating agents on held-out datasets, analyzing
execution traces, optimizing prompts and workflows, and eventually applying reinforcement
learning to compatible models.

Do not build a toy demonstration. Inspect the existing repository, identify the actual
architecture, and implement an incremental, testable optimization framework that integrates
with the current codebase.

1. EXISTING TECHNOLOGY STACK
My existing stack includes:
- AWS Bedrock Runtime for foundation model inference.
- Python and asyncio for asynchronous execution.
- LangGraph for stateful agent orchestration and multi-step workflows.
- DSPy for prompt and program optimization.
- Langfuse for tracing, prompt versioning, datasets, experiments, and evaluations.
- AWS CloudWatch for operational logging, metrics, latency, failures, and infrastructure monitoring.

Preserve these technologies wherever possible. Do not replace the existing architecture merely
to accommodate a new framework.

Use typed Python, Pydantic schemas, structured logging, pytest, configuration management, and
asynchronous I/O where appropriate.

2. FIRST INSPECT THE REPOSITORY
Before modifying any code:
1. Inspect the repository tree, README, pyproject.toml or requirements files, configuration,
   entry points, LangGraph definitions, DSPy modules, Bedrock clients, tool integrations, tests,
   and existing Langfuse instrumentation.
2. Identify the current model IDs, AWS Regions, inference APIs, model invocation patterns, and
   whether the system uses a direct Bedrock Runtime API or an OpenAI-compatible endpoint.
3. Determine how prompts, graph state, tools, retries, timeouts, and business outcomes are
   currently represented.
4. Identify which LangGraph nodes contain model decisions and which contain deterministic
   business logic.
5. Identify existing traces, evaluation datasets, test fixtures, and operational metrics.
6. Produce an architecture assessment and an implementation plan before making significant changes.

Do not assume that all models support fine-tuning, log probabilities, token-level training
data, or gradient updates.

3. DEFINE THE BUSINESS OBJECTIVE
Design the system around measurable business outcomes rather than generic model quality.

Create a configurable task-specific reward interface that can evaluate:
- End-to-end task completion.
- Correctness against reference answers or business-system state.
- Compliance with business policies and constraints.
- Tool selection and tool-call success.
- Data grounding and factual consistency.
- Latency, including end-to-end latency and time to first token where applicable.
- Cost per successful task.
- Retry counts and failure rates.
- Human intervention requirements.
- Safety violations and unauthorized actions.

Use task-specific evaluators wherever possible. Use an LLM judge only for dimensions that cannot
be reliably evaluated through deterministic checks.

Do not use a generic reward that treats all tasks alike.

Make reward weights configurable per business use case. Normalize component scores and document
the reward scale.

Implement hard safety constraints separately from the aggregate reward. A high task-completion
score must never compensate for a critical policy violation.

Do not reward verbosity, extra tool calls, or longer reasoning unless they demonstrably improve
the business outcome.

4. IMPLEMENT A DATASET MANAGEMENT AND EVALUATION FRAMEWORK
Support the following dataset partitions:
- Training/development: prompt optimization and training-data construction.
- Validation: candidate selection and hyperparameter tuning.
- Sealed holdout: final generalization measurement.

Ensure the same examples cannot silently leak across partitions.

Define a typed evaluation example containing, as appropriate:
- Stable example ID.
- Business task and input.
- Expected output or reference criteria.
- Expected business-state transition.
- Permitted tools and actions.
- Policy constraints.
- Relevant metadata.
- Dataset version and partition.

Support importing examples from JSONL and, where appropriate, Langfuse datasets.

Implement a reproducible experiment runner that can execute a selected agent version against a dataset.

For every run, record:
- Dataset and example IDs.
- Agent code commit.
- Model provider, model ID, and Region.
- Prompt version and DSPy configuration.
- LangGraph configuration and tool versions.
- Sampling parameters.
- Evaluator and reward-function versions.
- Start/end time and execution status.
- Final output and business outcome.
- Latency, token usage where available, and cost where measurable.
- Trace IDs and failure details.

Use deterministic settings where supported. Where the provider is nondeterministic, measure
variability through repeated runs on a predefined subset.

5. INSTRUMENT COMPLETE AGENT TRAJECTORIES
Implement or extend Langfuse tracing so each agent run can be reconstructed.

Capture:
- Initial business request.
- Relevant context and retrieved evidence.
- Model invocations and prompt versions.
- LangGraph node transitions.
- Model-selected actions and tool names.
- Validated tool arguments, with secrets and sensitive fields redacted.
- Tool results, errors, retries, and timeouts.
- State transitions and intermediate outputs.
- Final response.
- Verified business outcome.
- Reward components and aggregate reward.
- Human feedback or review outcomes, when available.

Represent each run as a trajectory containing ordered events.

For RL-oriented experiments, define an explicit transition schema:

(state_t, action_t, reward_t, state_t+1, terminal, metadata)

Document the meaning of state and action for the actual agent architecture. Do not equate every
LangGraph node with an RL action.

A trajectory must distinguish model-generated actions from deterministic tool execution and
external observations.

For multi-step tasks, support delayed outcome rewards and preserve the association between the
final business result and the decisions that contributed to it.

Avoid collecting hidden chain-of-thought as a training target. Capture structured decisions,
tool actions, observations, outcomes, and concise diagnostic explanations instead.

Ensure sensitive enterprise data, credentials, customer information, and proprietary documents
are protected through redaction, access controls, retention policies, and least-privilege IAM.

6. INTEGRATE LANGFUSE AND CLOUDWATCH
Use Langfuse for:
- End-to-end trace inspection.
- Versioned prompts.
- Dataset creation and management.
- Offline experiments.
- Custom evaluation scores.
- Comparison of agent versions.
- Feedback collection and failure analysis.

Use CloudWatch for:
- Operational logs and metrics.
- Request latency and error rates.
- Throttling, retries, and timeouts.
- Bedrock invocation failures.
- Cost and throughput monitoring where supported.
- Production health and alarms.

Use correlation IDs to link business tasks, LangGraph executions, Langfuse traces, and
CloudWatch log events.

Prefer supported SDKs and OpenTelemetry integrations. Inspect the installed versions before
choosing integration APIs.

Do not invent Langfuse SDK methods or rely on deprecated APIs. Verify the installed version and
its current official documentation.

7. IMPLEMENT DSPY OPTIMIZATION FIRST
Build an offline optimization pipeline that uses existing agent evaluations and failure feedback.

Evaluate suitable DSPy optimizers, especially:
- GEPA for reflective optimization using textual feedback.
- MIPROv2 for prompt instruction and demonstration optimization.
- Other optimizers only where they fit the existing program and data size.

The pipeline must:
1. Establish a baseline on the development and validation datasets.
2. Collect failure examples and diagnostic feedback from traces.
3. Define a task-specific DSPy metric.
4. Optimize only approved DSPy modules and prompt parameters.
5. Respect model invocation budgets, concurrency limits, and AWS quotas.
6. Evaluate candidate versions against validation data.
7. Compare aggregate scores and per-example regressions.
8. Save the selected program, prompts, configuration, and experiment results.
9. Preserve the baseline so every change can be rolled back.

Where possible, return both a numerical score and concise diagnostic feedback from the DSPy metric.

Do not automatically rewrite application code, production prompts, tool permissions, or business
policies based on model-generated suggestions.

Prompt optimization and RL must remain separate experiment types.

8. INVESTIGATE REAL REINFORCEMENT LEARNING FOR THE AGENT
Research the feasibility of reinforcement learning for this exact repository.

Start with these references:
- Amazon Bedrock reinforcement fine-tuning:
  https://docs.aws.amazon.com/bedrock/latest/userguide/reinforcement-fine-tuning.html
- Bedrock reinforcement fine-tuning data preparation:
  https://docs.aws.amazon.com/bedrock/latest/userguide/rft-prepare-data.html
- AWS reinforcement fine-tuning best practices:
  https://aws.amazon.com/blogs/machine-learning/reinforcement-fine-tuning-on-amazon-bedrock-best-practices/
- Microsoft Agent Lightning:
  https://github.com/microsoft/agent-lightning
- Agent Lightning documentation:
  https://microsoft.github.io/agent-lightning/
- DSPy optimizer documentation:
  https://github.com/stanfordnlp/dspy/blob/main/docs/docs/learn/optimization/optimizers.md
- Langfuse evaluation documentation:
  https://langfuse.com/docs/evaluation

Investigate these distinct approaches:
A. DSPy-based prompt/program optimization.
B. Supervised fine-tuning using high-quality, reviewed examples when supported.
C. Reinforcement fine-tuning through Amazon Bedrock for explicitly supported models and Regions.
D. External policy training using a suitable RL framework and a trainable model, if genuine
   agent-level RL is required.

For each approach, explain:
- What parameters are actually updated.
- Which model APIs and model architectures are supported.
- Whether model weights, gradients, token probabilities, or training checkpoints are required.
- How training trajectories are converted into examples.
- How rewards are computed.
- How the resulting model is deployed.
- Whether the existing Bedrock Runtime endpoint can serve the trained model.
- What infrastructure, permissions, GPU requirements, and costs are involved.

Do not claim that ordinary Bedrock Runtime inference supports weight updates or that arbitrary
proprietary models can be trained through the inference API.

Verify the current Bedrock reinforcement fine-tuning support matrix for the configured Region.
If the existing model is unsupported, preserve it for inference and implement a compatible
training path only after documenting the additional requirements.

Investigate Agent Lightning's current capabilities, installation instructions, supported
integrations, training backends, and licensing. Check whether a compatible skill exists for
Claude Code and explain how to install it using the project's official instructions.

Do not introduce an external RL training stack into the production service until a working,
isolated proof of concept has demonstrated compatibility.

9. DESIGN AN RL-COMPATIBLE TRAINING PIPELINE
If an appropriate trainable model and backend are available, implement a modular training interface.

The architecture should separate:
- Task and environment generation.
- Agent rollout execution.
- Trajectory collection.
- Reward computation.
- Dataset export.
- Training orchestration.
- Model artifact management.
- Evaluation.
- Model deployment.

For the first proof of concept, prefer a bounded, reproducible task with objectively verifiable
outcomes, such as structured extraction, SQL generation against a test database, or controlled
tool orchestration.

Implement reward functions that combine business success with correctness and efficiency while
enforcing safety constraints.

Avoid reward hacking. Include negative tests for malformed outputs, unauthorized tool use,
fabricated success, repeated tool calls, and shortcut behaviors that maximize the score without
completing the task correctly.

Do not perform unrestricted online RL against production systems. Use a sandbox or simulator for
training and validate candidate policies offline.

Keep the inference model and training model explicitly versioned and independently configurable.

10. PREVENT EVALUATION LEAKAGE AND OVERFITTING
Enforce these rules:
- Never optimize prompts or reward weights directly against the sealed holdout.
- Never use holdout examples for training or optimizer feedback.
- Select candidates using validation results.
- Run the holdout only at defined evaluation milestones.
- Version all datasets, prompts, reward functions, and code.
- Detect duplicate or near-duplicate examples across partitions where practical.
- Record every candidate, including failed experiments.
- Report confidence intervals or appropriate uncertainty estimates for measured improvements.
- Compare against a fixed baseline and a simple deterministic baseline where applicable.

If the candidate does not improve validation performance or causes safety regressions, reject it.

11. ADD AUTOMATED REGRESSION TESTING AND RELEASE GATES
Implement a command-line workflow along these lines, adapting it to the repository:

python -m agent_optimization.evaluate --dataset development
python -m agent_optimization.optimize_prompts
python -m agent_optimization.evaluate --dataset validation
python -m agent_optimization.compare --baseline baseline --candidate candidate
python -m agent_optimization.evaluate --dataset holdout --sealed

These are proposed interfaces, not mandatory names. Implement the actual commands consistently
with the repository.

Add tests for:
- Reward calculations.
- Dataset partition isolation.
- Trajectory serialization.
- Trace and metric correlation.
- Async concurrency and cancellation.
- Retries and timeouts.
- Tool authorization and safety constraints.
- Experiment reproducibility.
- Candidate promotion and rollback.
- Failure handling when Langfuse, CloudWatch, or Bedrock is unavailable.

Do not invoke paid models during ordinary unit tests. Use mocked clients and deterministic
fixtures. Mark integration tests separately.

12. DEFINE THE BUSINESS EVALUATION REPORT
Generate a machine-readable JSON report and a human-readable Markdown report for each experiment.

Include:
- Baseline versus candidate.
- Task completion rate.
- Business correctness.
- Policy and safety violation rate.
- Tool-call success rate.
- End-to-end latency, including p50 and p95.
- Time to first token where applicable.
- Cost per successful task.
- Retry and failure rates.
- Human intervention rate.
- Reward distribution and component scores.
- Per-category performance.
- Confidence intervals where appropriate.
- Regressions and representative failure cases.
- Dataset, model, prompt, code, and evaluator versions.
- A release recommendation and its justification.

Do not optimize a single composite score at the expense of critical business constraints.

13. IMPLEMENTATION STANDARDS
- Use the existing package manager and coding conventions.
- Keep training and evaluation code separate from production execution paths.
- Use async-safe concurrency controls and bounded parallelism.
- Implement timeouts, exponential backoff with jitter where appropriate, and retry budgets.
- Respect Bedrock quotas and throttling.
- Avoid blocking the asyncio event loop.
- Use environment variables or approved secret management for credentials.
- Never hardcode AWS credentials or API keys.
- Do not send customer data to external services without explicit authorization.
- Make all new infrastructure optional and configuration-driven.
- Document required IAM permissions and AWS costs.
- Do not silently upgrade major dependencies.
- Pin or constrain compatible dependency versions.
- Do not fabricate successful experiments, scores, or deployment results.

14. REQUIRED DELIVERABLES
After inspecting the repository, implement the smallest complete, end-to-end vertical slice.

Deliver:
1. An architecture assessment and implementation plan.
2. A documented evaluation and reward schema.
3. A dataset loader and partition validation.
4. Langfuse and CloudWatch instrumentation improvements.
5. A reproducible baseline evaluation runner.
6. DSPy optimization integration with validation-based candidate selection.
7. A trajectory export format suitable for future training.
8. A compatibility assessment for Bedrock reinforcement fine-tuning and external agent RL.
9. A sandboxed RL proof of concept if the required training backend is available and justified.
10. Unit tests, integration-test instructions, configuration examples, and documentation.
11. A sample experiment report with clearly identified real or synthetic data.
12. A roadmap for progressing from prompt optimization to model training.

15. EXECUTION INSTRUCTIONS
Start by inspecting the repository and presenting a concise assessment.

Then implement the evaluation, tracing, and DSPy optimization path first.

Research the RL options in parallel at the design level, but do not block the baseline
evaluation work on an unverified training framework.

Do not make irreversible changes, alter production resources, launch expensive training jobs, or
deploy a new model without explicit approval.

At completion, explain exactly what was implemented, what tests passed, what remains incomplete,
which RL approaches are compatible with the current models, and the next commands I should execute.

The primary success criterion is a statistically credible improvement in the real business KPI
without degrading safety, reliability, latency, or cost beyond the configured limits.

Additional Architecture Requirement: Modular Agent Optimization

Refactor the existing agent into independently observable and optimizable modules without
unnecessarily rewriting the production system.

1. CREATE EXPLICIT OPTIMIZATION BOUNDARIES
Identify the actual components in the repository and represent them as independently versioned
modules where practical:
- Routing and task classification.
- Document or image classification, if applicable.
- Decision and tool selection.
- Planning and multi-step orchestration.
- Retrieval and context construction.
- Final response generation.
- Validation, policy enforcement, and execution.

Use LangGraph nodes as workflow boundaries and DSPy modules for components that benefit from
prompt/program optimization. Do not force every node to become a DSPy module.

2. EVALUATE EVERY COMPONENT INDEPENDENTLY
Define component-specific test datasets and metrics, in addition to end-to-end business outcome tests.

For example:
- Routing: correct route selection and unnecessary handoff rate.
- Classification: per-class precision, recall, and F1.
- Decisions: business-rule compliance and action correctness.
- Retrieval: evidence recall, relevance, and grounding.
- Planning: task completion, unnecessary steps, and tool-call efficiency.
- Generation: factual correctness, completeness, and format compliance.
- Execution: successful state transitions, unauthorized actions, and rollback behavior.

Use deterministic verifiers wherever possible. Keep the overall business KPI as the primary
release criterion.

3. ADD TRACE-TO-TEST CONVERSION
Use Langfuse traces and CloudWatch operational events to identify failures and construct
regression tests.

For each failure, retain the sanitized input, expected behavior, observed behavior, relevant
tool results, component version, and evaluation result.

Require human review for ambiguous labels. Prevent traces from the sealed holdout from entering
optimization datasets.

4. IMPLEMENT CONTROLLED COMPONENT OPTIMIZATION
For each optimizable component:
1. Establish a baseline.
2. Run its development dataset.
3. Collect failure diagnostics.
4. Generate candidate prompts or configurations using DSPy.
5. Evaluate candidates on validation data.
6. Run end-to-end regression tests to detect downstream effects.
7. Promote only if the overall business KPI improves and critical constraints remain satisfied.

Where component changes interact, support joint evaluation of combinations rather than assuming
independent improvements always compose.

5. IMPLEMENT QUALITY-COST PARETO ROUTING
Evaluate eligible Bedrock models for each task category.

Measure quality, latency, and cost per successful task. Maintain a configurable routing policy
that selects among models satisfying the minimum quality and safety thresholds.

Do not select a model based on price alone or assume a smaller model is better without evaluation.

6. SEPARATE HARNESS OPTIMIZATION FROM MODEL TRAINING
Treat prompt optimization, tool configuration, routing, retrieval, and graph changes as harness
optimization.

Treat SFT, DPO, PPO, GRPO, and other parameter-learning methods as separate training experiments
with explicit model and backend compatibility checks.

Record which layer changed in every experiment so improvements can be attributed correctly.

7. IMPLEMENT A PROMOTION AND ROLLBACK WORKFLOW
Maintain immutable candidate versions containing the model identifier, prompt versions, graph
configuration, tool versions, and evaluator versions.

Run validation and end-to-end regression tests for every candidate. Require a sealed-holdout
evaluation and appropriate human approval before production promotion.

Support a rollback to the last approved version.

Do not modify the production agent automatically based on a single successful evaluation run.