Llm Evaluation Jobs in United States

Responsibilities Build and own end-to-end evaluation suites for LLM capabilities: task specs, grading logic, and the dashboards

Collaborate closely with modeling & evaluation teams on developing new evals, maintaining evaluation signal quality, and building internal tooling

Design and build benchmarks and evaluation methods that measure LLM capabilities in both text and multimodal modalities.

evaluation: LLM-as-judge, benchmark design, agent harnesses, calibration, and understanding how to integrate them into professional settings Sound

Hands-on experience in LLM evaluation, beyond calling a model API.

You’ll design and ship the LLM-as-a-judge evaluations that score voice and chat conversations at scale, then sharpen them with LoRA fine-tunes on open-weight models

…infrastructure engineering, production ML systems, LLM/LVM inference, evaluation harnesses, and inference

Experience with synthetic data, LLM evaluations, automated raters, or preference data is a plus.

…experience designing, building, or operating evaluation systems for LLM/GenAI products, including model-based evaluators or autograders

Evaluation: Build evaluation frameworks with customers, including task-specific benchmarks, LLM-as-judge pipelines, and held-out test

Our team owns data and evaluation science for Glean's AI products.

…preference studies, ablations, and held-out evaluations to measure behavior.