Llm Evaluation Jobs in San Francisco, CA

…large language or multimodal model evaluation, including model-based grading, human evaluation, or synthetic data.

Design and build benchmarks and evaluation methods that measure LLM capabilities in both text and multimodal modalities.

evaluation: LLM-as-judge, benchmark design, agent harnesses, calibration, and understanding how to integrate them into professional settings Sound

You’ll design and ship the LLM-as-a-judge evaluations that score voice and chat conversations at scale, then sharpen them with LoRA fine-tunes on open-weight models

VLM/LLM training: Track record in training VLMs or LLMs.

…experience designing, building, or operating evaluation systems for LLM/GenAI products, including model-based evaluators or autograders

Evaluation: Build evaluation frameworks with customers, including task-specific benchmarks, LLM-as-judge pipelines, and held-out test

Our team owns data and evaluation science for Glean's AI products.

…preference studies, ablations, and held-out evaluations to measure behavior.

…Work closely with community on LLM engines like vLLM, TensorRT-LLM

Experience with synthetic data, LLM evaluations, automated raters, or preference data is a plus.

Hands-on experience post-training LLMs including adapting open-weight models to specialized tasks.

…fast and automatic training and evaluation of LLM's, as well as evaluation of

…affect how your candidacy is evaluated.

…especially with generative models, e.g., LLMs/VLMs, and/or reinforcement learning