# Research Engineer, Web Crawling **Company:** [Thinking Machines Lab](https://zerogtalent.com/ai-companies/thinking-machines-lab) **Location:** San Francisco **Work mode:** On-site **Seniority:** Senior **Employment type:** Full Time **Salary:** USD 350K–475K/yr **Category:** Aerospace Engineering **Department:** Other **Listed:** 2026-09-24 [View and apply on Zero G Talent](https://zerogtalent.com/ai-jobs/thinking-machines-lab/web-crawling-research-engineer-a5559c53-7085-45ff-8a0f-5e47b5e822bc) ## What you'll do - Design/scale web crawler and ingestion infrastructure for pretraining data - Build large-scale extraction, deduplication, data quality pipelines - Create specialized crawlers for high-value/hard-to-reach sources - Collaborate with pretraining team on data impact on model performance - Improve reliability/efficiency of crawling at petabyte scale ## What you need - 8+ yrs building/scaling web crawlers at internet scale - Owned crawler/data-acquisition infrastructure at scale - Strong Python/Go/Rust with distributed systems experience - Knowledge of legal/practical web data collection (robots.txt, licensing) ## Job description ### **About Thinking Machines** The mission of Thinking Machines is to build AI that extends human will and judgment. We are training frontier models with Inkling, developing Tinker to let people make models their own, and crafting interfaces that broaden human-AI communication. We believe the future worth building is human, and we're hiring people who want to build it. ### **About the Role** We're hiring a Software Engineer to build and own our web-crawling systems, from distributed collection at internet scale through filtering, deduplication, and deciding what data we keep. The ideal candidate has built and scaled a web crawler or large-scale data-acquisition systems. In this role, you'll write and own production systems: the crawler itself, the infrastructure that runs it at scale, and the pipelines that turn raw crawls into usable pretraining data. You'll work closely with our pretraining and data teams to understand what's actually moving model quality, but this is fundamentally an engineering role, not a research one. ### **What You'll Do** - Design and scale the web crawler and ingestion infrastructure that sources Inkling's pretraining data - Build pipelines for large-scale extraction, deduplication, and data quality filtering - Build specialized crawlers for high-value or hard-to-reach data sources - Work with the pretraining team to understand how changes in crawled data affect model performance - Improve the reliability and efficiency of crawling and ingestion infrastructure at petabyte scale - Help set technical direction for this area as it grows, and bring other engineers up to speed on what you've learned ### **Skills & Qualifications** ### **Minimum Qualifications** - 8+ years designing, building, and scaling web crawlers, scrapers, or large-scale distributed data-acquisition systems - A track record of owning crawler or data-acquisition infrastructure at internet scale - Strong software engineering skills in a language such as Python, Go, or Rust, with real experience in distributed systems - Working knowledge of the practical and legal considerations of large-scale web data collection (robots.txt, rate limiting, licensing) ### **Preferred Qualifications** - Experience applying machine learning to crawl selection, extraction, or data quality classification at internet scale - Experience setting technical direction for a crawling, data acquisition, or search infrastructure team, whether or not that was your formal title - Experience designing systems for petabyte-scale storage and processing - Track record of open-source contributions to crawling, scraping, or data infrastructure tools - Background at a search engine (crawling, indexing) or a frontier AI lab's data acquisition team ### **Logistics** - Location: This role is based in San Francisco, CA or New York, NY. - Compensation: Depending on background, skills and experience, the expected annual salary range for this position is $350,000-$475,000 USD. - Visa sponsorship: We sponsor visas. While we can't guarantee success for every candidate or role, if you're the right fit, we're committed to working through the visa process together. - Benefits: Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed. _As set forth in Thinking Machines' Equal Employment Opportunity policy, we do not discriminate on the basis of any protected group status under any applicable law._ _Thinking Machines Lab will consider for employment qualified applicants with criminal histories in a manner consistent with the requirements of the California Fair Chance Act, the San Francisco Fair Chance Ordinance, and any other applicable state or local fair chance ordinance or law._ ## Other open roles at Thinking Machines Lab - [Business Recruiter](https://zerogtalent.com/ai-jobs/thinking-machines-lab/business-recruiter-b47907dc-a08e-4a58-9b46-5d86417cfa57) — San Francisco - [Product Management Recruiter](https://zerogtalent.com/ai-jobs/thinking-machines-lab/product-management-recruiter-3d90697c-362e-4b6e-8600-e1d39115c6cd) — San Francisco - [Technical Recruiter: Security](https://zerogtalent.com/ai-jobs/thinking-machines-lab/technical-recruiter-security-fa72912b-78b7-40ae-8a54-c2a524675f2c) — San Francisco - [Head of Revenue Operations](https://zerogtalent.com/ai-jobs/thinking-machines-lab/head-of-revenue-operations-c199591a-01b0-445d-9ad2-70a46b2e34a5) — San Francisco - [Brand Designer](https://zerogtalent.com/ai-jobs/thinking-machines-lab/brand-designer-9ab3e86f-b636-4a75-867c-4ffd04d7b87d) — New York --- *Source: [Zero G Talent](https://zerogtalent.com), the job board for space, defense, AI, robotics and frontier-tech companies.*