Engineering Manager, Inference Cloud

Location: Sunnyvale

About the Role

We're looking for a deeply technical, hands-on engineering leader to scale our Inference Cloud Platform. This team owns the cloud layer that powers our Inference Service, with direct responsibility for availability, reliability, latency, and global scalability.

You'll lead a high-performing team building the systems that keep inference fast and reliable at massive scale: multi-region traffic management, intelligent routing, graceful degradation under load, and best-in-class observability. The work sits at the intersection of distributed systems, cloud-native infrastructure, and the unique demands of serving AI workloads in production, including bursty, unpredictable traffic patterns unique to model serving.

If you're passionate about building resilient, globally distributed systems and solving hard infrastructure problems for AI, we'd love to talk.

Responsibilities

Technical Strategy & Architecture

• Platform Vision & Roadmap. Own the technical direction for the Inference Cloud Platform, prioritizing cloud-native scalability, reliability, and multi-region architecture.

• Core Infrastructure. Lead the design of foundational cloud-layer systems including service discovery, service mesh, request routing, load balancing, caching, and batching to optimize latency, throughput, and cost efficiency.

• Resilience & High Availability. Build and operate fault-tolerant, active-active multi-region systems with strong SLAs/SLOs, rapid failover, and graceful degradation strategies such as circuit breaking, backpressure, and load shedding.

• Traffic Control & Quality of Service. Design traffic prioritization, rate limiting, quota management, and admission control systems, including differentiated service tiers with independent SLOs, to ensure fairness and protect stability under extreme load.

Operational Excellence

• Observability. Establish monitoring, logging, and alerting systems; define and own key metrics for latency, availability, and system health.

• Reliability Practices. Drive operational maturity through incident response leadership, blameless postmortems, chaos engineering, and continuous improvement of platform reliability.

People & Partnerships

• Team Building. Recruit, mentor, and scale a high-caliber engineering team; foster a culture of ownership, technical excellence, and rapid execution.

• Cross-Functional Collaboration. Partner with ML, Product, Infrastructure, and Platform teams to align the roadmap with customer needs and deliver scalable cloud services.

Skills & Qualifications

• 8+ years in high-scale software engineering, with 4+ years leading distributed systems or cloud infrastructure teams. Strong hands-on coding and system design skills.

• Deep expertise in building and operating large-scale distributed systems in cloud environments (AWS; GCP or Azure experience), including compute orchestration, networking, and container platforms (Kubernetes/EKS).

• Proven track record delivering highly available, multi-region production services with well-defined SLIs/SLOs/SLAs and measurable reliability outcomes.

• Experience driving latency optimization (including TTFT and tail latency), throughput improvements, and system efficiency gains in high-QPS environments.

• Strong proficiency in systems-level or backend languages such as Go, C++ or Python.

• Hands-on experience building observability stacks including metrics, logging, distributed tracing, and alerting (e.g., Prometheus, Grafana, or equivalents).

• Demonstrated ability to recruit, develop, and lead high-performing engineering teams and to collaborate effectively across engineering, product, and operations.

• Experience with ML inference infrastructure, model serving frameworks, or GPU-accelerated workloads is a plus.