Job Description
As a Member of Technical Staff on Evals, you'll own how we measure AI Employee performance on the job.
Areas you may work in:
- Evaluation systems: datasets, offline replay, scorers, and regression alerts
- Feedback loops for models and agents
- Autonomous failure analysis
- Defining what good, degraded, and failed runs look like, and alerting on them
You may be a good fit if you:
- Have built evaluation or measurement systems, such as AI evals, experimentation, ranking, or search quality
- Can turn ambiguous quality questions into concrete metrics and decisions
- Have strong software engineering fundamentals and ship production systems
- Are comfortable where there are few established patterns
- Want to work in person in San Francisco
Even better:
- You've shipped LLM systems to production and learned where they break
- You have strong data instincts and work well with researchers
- You contribute to open source projects
Optimize Your Resume for This Job
Get a match score and see exactly which keywords you're missing
Job Details
- Category
- Software
- Employment Type
- Full Time
- Location
- San Francisco, CA
- Posted
- Compensation
- $130,000 - $180,000 per year
About Jarmin
Jarmin.ai is your 24/7 ML eng employee that you hire. Just talk to Jarmin like any other employee and Jarmin handles your AI/ML work from there. Completely hand over full initiatives or individual tasks. Jarmin’s founding team is from Meta SuperIntelligence Labs (Staff Engineers), Apple (Machine Learning Engineer), AWS (scaling infra), Lockheed-Martin (Research), and JPMorganChase (SWE) with experience building AI/ML and automating complex work of Researchers, Data Scientists, ML Engineers with AI.
More Roles at Jarmin
Similar Software Roles
Found this role interesting?