
Sr. Inference Engineer (LLM Serving)
What you need
- 5+ yrs software engineering, 2+ yrs ML/LLM production
- Production experience with vLLM, SGLang or TensorRT-LLM
- Docker, Kubernetes and GPU instances in public cloud
- Inference bottlenecks: KV cache, attention, memory bandwidth, batching
- Clear communication in English with engineers and product
What you'll do
- Deploy, operate and upgrade open-weight LLMs on GPUs in production
- Optimize inference: batching, caching, quantization, parallelism, latency and cost
- Evaluate open-weight models and recommend production candidates
- Build inference gateway with routing, semantic cache and answer validation
- Adapt models with LoRA adapters and fine-tuning for analytics use cases
Our platform turns governed business data into answers, reports and agents that companies put in front of their own customers. To make that AI trustworthy, private and affordable, we run it on our own inference: open-weight models on our own GPUs, so customer data never has to leave our boundary.
The role
You will own the serving layer of GoodData Native Inference. That means the models, the engine and the GPUs behind every AI answer our customers get. You will join the team that builds GoodData AI and become the person the whole company turns to for anything about running models in production.
What you will do
Deploy, operate and upgrade open-weight LLMs and smaller task models (guardrails, PII detection, classifiers, judges) on our GPUs, in production and across regions.
Make inference fast and cheap: tune batching, KV and prefix caching, quantization and parallelism, and track time to first token, tokens per second per GPU, p95 latency and cost per answer.
Choose the models: put new open-weight models through our evaluation harness on real analytics workloads, and recommend what goes to production.
Grow our inference gateway: routing by intent and complexity across models, a semantic cache, and validation that escalates weak answers.
Adapt models to our use cases with LoRA adapters and fine-tuning, such as generating queries for our semantic layer, root-cause analysis and narrative reports.
What you bring
5+ years of software engineering, including 2+ years running ML or LLM models in production.
Hands-on production experience with at least one modern LLM serving engine: vLLM, SGLang or TensorRT-LLM.
A working understanding of inference bottlenecks (KV cache, attention, memory bandwidth, batching, parallelism), and the habit of reasoning about latency, throughput and cost in numbers.
Experience with Docker, Kubernetes and GPU instances in a public cloud.
Discipline in benchmarks and evaluation: you do not ship a model you have not measured.
Clear communication in English, with engineers and with product people.
Nice to have
Speculative decoding, multi-LoRA serving, or PEFT fine-tuning.
Contributions to vLLM, SGLang, PyTorch or similar open-source projects.
CUDA or Triton kernels, and the NVIDIA GPU stack (drivers, NCCL).
LLM routing, guardrail models, embeddings and rerankers.
Go or Rust; Terraform and Helm.
Observability for LLM systems (OpenTelemetry, Langfuse, Prometheus, Grafana).
Experience with analytics, semantic layers or text-to-SQL.
Our stack
Python, Kubernetes on AWS, NVIDIA GPUs, SIE (an open-source inference engine on SGLang), vLLM, PyTorch, gRPC, Langfuse, and our own evaluation harness.
About GoodData.AI
At GoodData.AI, we’re building the future of agentic AI for data intelligence.
Our full-stack data intelligence platform empowers companies to turn raw data into real business impact — from agentic AI apps to embedded analytics that scale securely and seamlessly. With GoodData.AI, organizations don’t just analyze data, they monetize it.
But what really sets GoodData.AI apart is our mission: to put intelligence wherever decisions are made. That’s why we’ve built a platform that’s AI-native, composable, and built for the way modern enterprises work.
#LI-Hybrid
Optimize your resume for this job
Get a match score and the keywords you're missing
About GoodData
GoodData is the full-stack data intelligence platform that helps businesses turn data into actionable, enterprise-grade insights. Build, embed, and monetize agentic AI and analytics apps — fully governed, auditable, and ready to scale. Our mission: Turn data into intelligence that’s secure and embedded wherever decisions happen. That’s why we built a platform that is composable, AI-native, flexible, and designed for the way modern enterprises deliver value. GoodData is built for CIOs, product leaders, and data teams who need to innovate quickly while maintaining governance and compliance. Our platform gives teams the tools to deliver trusted AI at enterprise speed. The GoodData platform unifies semantic modeling, governance, analytics, and agentic AI into a single secure foundation. This enables enterprises to: 🚀 Design & deploy fast: Create custom agentic apps and workflows for any use case — delivered in days, not years. 🚀 Scale & expand effortlessly: Seamlessly size up to meet client demand without sacrificing cost, compute, or performance. 🚀 Customize flexibly: Modify everything with developer-first tools, SDKs, and an API-driven architecture. 🚀 Built secure by design: Run trusted agents with enterprise-grade security, deployed in the cloud or on-premises. 🚀 Leverage our expertise: Partner with our forward-deployed engineers to co-design, test, and deliver custom solutions. We serve all industries worldwide — from fintech and retail to healthcare, SaaS, and manufacturing. More than 140,000 organizations and 3.2 million users rely on GoodData, including Visa, Travelodge, Twilio, and Mews. One AI platform — limitless ways to build, embed, and monetize intelligent agents. 👉 Start building smarter intelligence today at www.gooddata.com.
Similar roles


