Remote Agent Evaluation Engineer Jobs

As our first AI Agent Evaluation Engineer, you define what quality means for every agent we run, you build the systems that measure it across

engineering workflows Build evaluation benchmarks and diagnostic tooling to identify where models fail on engineering tasks, and trace those failures

…sits at the intersection of agentic workflows, product engineering, developer infrastructure, and AI-native user experience.

Familiarity with prompt engineering, RAG, or agent evaluation.

…have deep familiarity with prompt engineering and evaluation frameworks. Excellent written and verbal communication.

You'll combine full-stack engineering (MCP servers, agentic systems, web applications, etc.) with hands-on evaluation work (behavior benchmarks, production monitoring

Take agents from prototype to production, including the evaluation criteria, guardrails, and human review steps prospect-facing work requires.

Build evaluation datasets that include realistic legal queries, contract markups, and/or synthetic legal documents.

, or AI Solutions Engineering role Experience with MCP, agent orchestration frameworks, and enterprise tool-calling patterns Knowledge of AWS Bedrock

We’re looking for research engineers who can build reliable data and evaluation systems, question what a metric actually tells us, and work closely with researchers

Experience with RAG systems, agentic workflows, multi-step reasoning systems, or LLM-as-a-judge evaluation methods.

…We are hiring a Research Engineer to turn questions about agent learning into reliable systems and reproducible experiments.

You’ll strengthen the team’s model, agent system, and evaluation work through hands-on studies, clear rubrics, and reliable measures of quality that reflect user

Agentic tooling you built that other engineers use: skills, plugins, hooks, subagents, MCP servers, or custom agents.

Familiarity with LLMs, AI agents, model evaluation, and approaches to model customization, including fine-tuning.