Confident AI Expands in San Francisco to Scale LLM Evaluation Beyond Open Source
Teams running large language models in production face a persistent challenge: prompts and model architectures that pass manual inspection during development routinely fail when exposed to real users. Confident AI tackles this testing gap by turning local script evaluations into an organizational system of record. Founded in 2023 and backed by Y Combinator's W25 batch, the seven-person startup is now hiring founding engineers in San Francisco to scale its cloud platform.
The company is adding two onsite roles: a Founding Growth Engineer offering $140,000 to $200,000 and a Founding Product Engineer (Frontend) offering $175,000 to $250,000. Both hires report directly to the leadership team as the company expands commercial adoption beyond its open-source user base.
Cloud Infrastructure for Production LLM Testing
Confident AI developed DeepEval, an open-source evaluation framework that has accumulated more than 600,000 monthly downloads, 5,300 GitHub stars, and over 40 million conducted evaluations. According to a listing on Crunchbase, the company provides open-source unit testing solutions for applications utilizing large language models.
While DeepEval provides algorithms and local testing runs inside Python notebooks or CI/CD pipelines, Confident AI operates as the commercial cloud layer above it. In local environments, engineers run tests against custom criteria, but tracking regressions across multi-developer teams requires centralized state. Confident AI allows engineering teams to store benchmarks, generate testing reports, monitor applications deployed in production, and run continuous regression checks across updates.
In a video breakdown by The Inference, reviewers noted that Confident AI supports non-engineers testing AI applications directly over HTTP without writing code. The platform applies more than 50 research-backed metrics spanning retrieval-augmented generation (RAG) to safety, and curates evaluation datasets automatically from production traffic so product managers and QA personnel can test prompts before release.
Engineering Roots and Technical Architecture
The company was established by Jeffrey Ip and Kritin Vongthongsri. Ip, who serves as CEO, previously worked as a software engineer at Google and Microsoft before creating DeepEval. CTO Vongthongsri studied operations research, financial engineering, and computer science at Princeton, authored CHI-published research, developed NLP pipelines for a fintech startup, and researched autonomous vehicles and human-computer interaction.
Alongside Y Combinator, Confident AI has raised Seed funding from Liquid 2 Ventures, January Capital, Flex Capital, and Vermilion Cliffs Ventures.
The technical workflow connecting DeepEval to Confident AI was detailed in a technical walkthrough on DevStoriesEU. Engineers authenticate locally through a CLI command using a workspace API key. Test runs execute in the local terminal while simultaneously streaming results to Confident AI's hosted dashboard. By logging hyperparameters—such as model architectures, temperature settings, retrieval chunk sizes, and prompt template versions—teams can compare changes across test runs. A dashboard timeline reveals whether moving from one model to another improves relevancy while causing factual consistency to degrade.
A technical tutorial by Yash Patil demonstrated the evaluation mechanics for retrieval-augmented generation. The system defines "goldens"—pairings of inputs and expected outputs—and extracts retrieved contexts and actual outputs from LLM pipelines into test case objects. Confident AI's evaluation engine then scores the run across specific parameters like answer relevancy and hallucination rates before pushing aggregated results back to named datasets in the cloud interface.
Founding Engineering Roles in San Francisco
Confident AI is filling two full-time, onsite engineering roles at its San Francisco headquarters to support product delivery and customer onboarding.
The Founding Product Engineer (Frontend) takes ownership of user-facing interfaces end-to-end. The role entails building data-dense workflows for evaluation and testing without relying on dedicated product managers or designers. The hire will make independent design decisions and write both frontend interfaces and backend API routes. Compensation ranges from $175,000 to $250,000.
The Founding Growth Engineer works alongside the founders to design and construct an automated growth engine. Responsibilities include building tooling and experiments to acquire and activate engineering teams evaluating and monitoring AI systems. The position pays $140,000 to $200,000.
| Role | Location | Work Mode | Salary |
|---|---|---|---|
| Founding Product Engineer (Frontend) | San Francisco, CA, US | Onsite | $175,000 – $250,000 |
| Founding Growth Engineer | San Francisco, CA, US | Onsite | $140,000 – $200,000 |
