Human Evaluation Researcher

Why this role exists

"Does this avatar feel human?" is the question our whole company is organized around — and no automated metric can answer it. Lip-sync error and video quality scores say nothing about whether a smile landed as sincere or unsettling, whether a conversation felt warm or hollow, or whether someone would want to talk to our avatar again tomorrow.

Your job is to turn human judgment into a reliable signal our researchers can train and ship against. You'll take the most ambiguous problems in our field (naturalness, emotional resonance, trust, presence) and design studies whose results people actually agree on. When two models differ, your study is the tiebreaker. When a model "feels off" and nobody can say why, your investigation finds the cause.

This is a hands-on IC role and our first hire dedicated to human evaluation. You'll own it end to end (what to measure, how to measure it, who rates it, and what the results mean) working directly with the founders and the modeling team. Your findings will decide which models ship and what we train next.

What you'll do

• Design and run qualitative and quantitative studies of our AI avatars: side-by-side comparisons, controlled rating experiments, in-depth interviews, think-alouds, diary studies, and longitudinal panels.
• Turn ambiguous judgments into instruments people can align on (rubrics, anchored scales, annotation guidelines) then measure and improve inter-rater agreement without flattening real signal.
• Use ethnographic techniques (observation of live conversations, contextual inquiry, field work) to understand how people actually experience face-to-face AI, not just what they report in a survey.
• Build the human-eval pipeline itself: participant panels, rater training and calibration, tooling, and an evaluation cadence tied to model releases.
• Calibrate automated and model-based metrics (including LLM-as-judge) against human judgment, so the team knows when to trust them and when not to.

You may be a good fit if you have

• 5+ years designing and running human-subjects research in industry or academia (UX research, HCI, experimental psychology, behavioral science, or a related field).
• Examples of study designs you can walk us through, especially ones where you got humans to converge on an ambiguous judgment (tone, emotion, quality, trust), including the rubrics, anchors, and protocols that made alignment possible.
• Strong grounding in both qualitative methods (interviews, ethnography, contextual inquiry) and quantitative methods (survey and psychometric design, experimental design, statistics for rating and pairwise-comparison data).
• Fluency with agreement and reliability. For example: You know your Cohen's kappa from your Krippendorff's alpha, and more importantly, how to raise them.
• A bias toward running a scrappy, sound study this week over a perfect one next quarter, and the ability to explain findings crisply to ML researchers.

Strong candidates may also have

• Experience evaluating generative AI: avatars or digital humans, speech or video generation, conversational agents, or emotion expression and recognition.
• A background in perceptual science or psychophysics (how people perceive faces, voices, motion, and emotion). MS/PhD in a related field welcome.
• Familiarity with human evaluation at scale: crowdsourcing platforms, annotation tooling, golden datasets.
• Enough statistics and scripting (Python or R) to analyze your own data.

No candidate checks every box. If the "good fit" list sounds like you but your background is unconventional, we'd like to hear from you anyway.

Logistics

• Location: In-person in Seattle, five days a week — we believe in the compounding value of working shoulder-to-shoulder.
• Visa sponsorship: We sponsor visas (O-1, H-1B, green card, etc.) from day one.
• AI-native tooling: Do your best work with the best tools, including unlimited tokens.

Benefits

• Health: We offer a variety of plans that meet your needs, including an HDHP with ~$2,000 in annual HSA contributions by the company (roughly 2x what most big tech companies put in).
• Time off: 15 days of PTO, 10 public holidays, and we close the office for a full week at year-end.
• Food: Lunch, drinks, and snacks on us every workday. We observe boba Tuesdays and Thursdays.
• Commuter benefits: Utilize pre-tax money (up to $340/month) for parking and transportation.
• 401(k): 4% match (100% of 1st 1% + 50% of next 5% contributions).