Llm Tools Jobs in San Francisco, CA

…systems like TGI, vLLM, TensorRT- LLM, and Optimum, and comfortable creating custom tooling for testing and optimization.

evaluation: LLM-as-judge, benchmark design, agent harnesses, calibration, and understanding how to integrate them into professional settings Sound

Familiarity with LLMs, RAG, AI agents, or evaluation frameworks.

LLM-as-judge on every event is easy.

and non-tuning work like tool use and context engineering.

Own critical integrations: model adapters, LLM gateway hooks, vector DBs, tools & external APIs, and the platform’s LLMops flows.

…build and lead our Internal Tooling team in San Francisco or New York City.

Experience shaping LLM behavior through prompts, tool and skill design, or memory systems.

Hands-on experience post-training LLMs including adapting open-weight models to specialized tasks.

…of AI infrastructure and developer tooling.

You are fluent in LLM evaluation concepts: evalsets, LLM-as-judge, pairwise/side-by-side judging, judge calibration against human labels,

Familiarity with LLM observability and tracing tools.

We may use AI-enabled tools to screen and assess applicants against the criteria for this position.