AI Economics Flip: Inference Costs Now Outpace Model Training

From Open Source to Commercialization

The core maintainers of vLLM have formed Inferact, a startup dedicated to commercializing the open-source inference engine. The vLLM project began in 2023 at UC Berkeley's Sky Computing Lab, run by Ion Stoica. The engine introduced PagedAttention, a memory-management technique that lets GPUs serve multiple requests with far less wasted VRAM than the standard key-value cache. That single innovation made vLLM the go-to runtime for open-weight models across Hugging Face, major cloud providers, and internal platforms at companies running inference at scale. By the time Inferact launched, vLLM had become the de facto standard for open-source LLM serving — adopted because it delivered higher throughput per GPU than the alternatives.

Inferact's founding team includes the project's primary architects. They kept the project under the same open-source license and pledged continued investment in the public engine. The company's strategy spans TPUs, AMD GPUs, and low-level kernel work — a bet on hardware-agnostic performance rather than a single-vendor stack.

The move reflects a broader market shift. Inference now consumes the majority of AI compute spend, and the cost-per-token metric has replaced training FLOPs as the key performance indicator for deployed systems. Investors are betting that the team best positioned to optimize that metric across every major accelerator is the one that wrote the engine everyone already uses.

Why Deployment Economics Have Inverted

The economics of AI have flipped. Training a foundation model is a one-time capital expense; serving it is a recurring operational burden that now dwarfs the upfront cost.

The hardware mix is fragmenting. AMD's Helios rack (72 MI255X GPUs with up to 30 terabytes of HBM4 memory) and Cerebras' wafer-scale engine (900,000 cores, 44 gigabytes on-chip SRAM) now pair in heterogeneous serving: AMD handles prefill, Cerebras handles decode. AWS pairs Trainium for prefill with Cerebras CS-3 for decode. Nvidia announced Vera Rubin GPUs with Grove LPX accelerators for the same split. The GPU monopoly is ending; procurement decisions are moving from "which GPU" to "which serving architecture."

The architectural shift driving this is phase disaggregation. Prefill (processing the prompt) is compute-intensive and parallel. Decode (generating tokens sequentially) is memory-bandwidth bound and latency-sensitive. Microsoft Research characterized this split in 2023; Moonshot AI's Mooncake system put it into production at scale, processing over 100 billion tokens per day across thousands of nodes by 2025. Nvidia's Dinamo framework vendor-supported the KV cache movement. By 2026, the boundary has migrated from research to commercial hardware partnerships.

The economics work because the two phases grow differently. Agents and multimodal systems ingest more code, documents, images, and video — expanding prefill work. Longer reasoning chains and tool loops expand decode work. Separate pools scale independently. AMD and Cerebras model a fivefold tokens-per-second-per-watt gain for their combined system versus Cerebras alone at comparable interactivity. OpenAI cut logged-out traffic serving costs by more than half through undisclosed optimizations, then used its own model to rewrite production serving code in Triton and Gluon for another 20 percent reduction. Speculative decoding (a smaller draft model proposing tokens the main model verifies in parallel) compounds the gains.

Installed compute (processors, memory, networking, power owned or rented) is no longer the right metric. Effective compute (useful inference work at required latency and quality) is what matters. Better batching, caching, routing, and numerical precision let the same fleet handle more requests. Phase disaggregation goes further: it changes where each request routes, letting providers choose different hardware and scale each phase independently. The useful unit of AI infrastructure is no longer a chip or a rack; it's the complete route one request takes through the data center.

The Talent War: Rise of the AI Infrastructure Engineer

The shift from model training to inference deployment has birthed a distinct engineering category. The role sits at the intersection of distributed systems, kernel-level optimization, and hardware-specific tuning — exactly the skills needed to squeeze tokens per second from GPU clusters running vLLM, TensorRT-LLM, or DeepSpeed.

Titles vary across organizations: AI Platform Engineer, ML Infrastructure Engineer, AI Systems Engineer, and AI Platform Engineering. The substance remains nearly identical: engineers who understand memory management, kernel fusion, quantization pipelines, and multi-GPU parallelism well enough to choose — or build — the right runtime for a given workload.

Inferact's own hiring board makes the demand concrete. As of early 2026, the company lists 15 salaried roles with a compensation band of $156,000 to $400,000 (median $400,000). Recent postings target specialized profiles: Member of Technical Staff for CI/CD Infrastructure in San Francisco, TPU Performance Engineering in Singapore, AMD GPU Performance Engineering in both San Francisco and Singapore, Performance and Scale in Singapore, and Kernel Engineering in Singapore. Each role commands $200,000–$400,000 annually. The specificity (TPU, AMD GPU, kernel engineering) signals that generic backend experience no longer suffices. Companies need engineers who can optimize PagedAttention for H100s one week and port the same stack to AMD MI300X or Google TPU v5p the next.

The salary premium reflects scarcity. Traditional ML engineers focus on model architecture, data pipelines, and training loops. Platform engineers build CI/CD, observability, and developer tooling. AI infrastructure engineers occupy a narrower slice: they profile kernel latency, debug NCCL communication hangs, implement speculative decoding without quality regression, and tune batch scheduling for variable-length generation. These skills were niche until inference became the dominant compute budget.

Companies adding kernel engineers, CUDA specialists, and runtime maintainers are betting on deployment throughput as their competitive moat. The talent pool remains thin. Universities graduate engineers fluent in PyTorch but rarely in PTX assembly or Triton kernel writing. Open-source contributors to vLLM, TensorRT-LLM, and DeepSpeed form a de facto credentialing system. Inferact's founding team emerged directly from the vLLM project at UC Berkeley; their first hires mirror that profile.

The next bottleneck isn't model quality. It's the engineer who can keep 1,024 GPUs fed without a single bubble in the pipeline.

Three Philosophies: PagedAttention vs. Proprietary Stacks

The inference runtime market has consolidated around three distinct philosophies. vLLM, born there and now stewarded by Inferact, bets on runtime flexibility and memory efficiency through PagedAttention. NVIDIA's TensorRT-LLM pursues peak hardware utilization through ahead-of-time compilation and kernel fusion. Microsoft's DeepSpeed-FastGen targets variable-length generation workloads with dynamic splitting techniques. Each approach reflects a different assumption about where the bottleneck lives — and each has carved out a production niche the others struggle to displace.

PagedAttention, vLLM's founding innovation, treats the key-value cache like virtual memory. Instead of allocating contiguous GPU memory for each request's full context window, it pages KV-cache blocks (typically 16 tokens each) into a shared pool. The result: near-optimal memory usage with under 4 percent waste. Continuous batching lets the scheduler interleave new requests with ongoing generations at token-level granularity, keeping GPUs saturated even as prompt lengths and output lengths vary wildly. This architecture gives vLLM its edge in high-concurrency, variable-batch scenarios — the dominant pattern for chat, coding assistants, and agentic workloads where request arrival is unpredictable and context lengths span orders of magnitude.

TensorRT-LLM takes the opposite tack. It compiles a model into a hardware-specific engine, fusing operators into monolithic CUDA graphs that eliminate kernel launch overhead and maximize Tensor Core occupancy. The payoff: on NVIDIA H100s, TensorRT-LLM delivers 30–50 percent higher throughput than vLLM on the same hardware. FP8 quantization on Hopper architecture cuts memory in half with near-lossless quality. But the compilation step introduces friction: rebuilds for each model variant, each GPU architecture, each batch-size profile. TensorRT-LLM also ties deployment to the NVIDIA ecosystem: Triton Inference Server for serving, NeMo for model preparation, CUDA for kernels. Teams committed to H100 clusters and willing to invest in per-model optimization reap the rewards. Teams targeting AMD Instinct, Intel Gaudi, Google TPUs, or Apple Silicon hit a wall.

DeepSpeed-FastGen occupies a third lane. Its dynamic splitting technique partitions generation workloads across GPUs at the token level rather than the layer or tensor level, aiming to smooth throughput for long-form generation where batch sizes shrink as sequences diverge. DeepSpeed Inference adds model offloading to CPU and NVMe for models that exceed aggregate GPU memory, a capability vLLM and TensorRT-LLM address through tensor and pipeline parallelism instead. In practice, production teams mix engines: TensorRT-LLM for latency-critical endpoints on NVIDIA iron, vLLM for development, staging, and multi-hardware deployments, DeepSpeed for offloading-heavy large-model serving.

The fundamental difference lies in compilation versus runtime optimization. TensorRT-LLM builds hardware-specific engines that maximize performance for specific GPU configurations, while vLLM focuses on those aspects.

The hybrid reality is already standard. Production inference teams commonly combine vLLM for staging and multi-architecture flexibility with TensorRT-LLM for a handful of latency-sensitive production endpoints. Inferact's commercial bet — that vLLM's hardware-agnostic runtime and open-source momentum make it the default control plane, with TensorRT-LLM as a specialized accelerator — mirrors how Kubernetes won orchestration while cloud vendors still sell proprietary accelerators underneath.

Hardware-Software Co-Design and the Kernel Bottleneck

When an LLM generates a token, the GPU executes a sequence of highly parallel operations: matrix multiplications, attention, normalization, activation functions. These operations are implemented using GPU kernels: functions that run across thousands of threads simultaneously. "Think of kernels as the inner loops of LLM inference," the Modular handbook explains. "They handle the core numerical work behind each token generation step and are where most of the compute time is actually spent." System-level optimizations (continuous batching, prefix caching, KV cache routing) operate above this layer. They cannot compensate for inefficient kernels. If the bottleneck sits in the kernel itself, no runtime change will close the gap.

The dominant operation in transformer inference is dense matrix multiplication (GEMM). For a batch size B and sequence length S, the attention QKV multiplication costs O(B·S·H²). Scaling to longer contexts quickly saturates the GPU's tensor cores. The attention softmax and value multiplication add extra passes over the same data, compounding latency. A 70-billion-parameter model translates to roughly 280 gigabytes of FP16 weights — far beyond the 80 gigabytes of a flagship A100. Even after 4-bit quantization, the footprint often exceeds a single device's capacity. Production services typically target sub-100-millisecond latency for a single user request while batch processing demands higher throughput. Achieving both requires kernel fusion to reduce intermediate memory traffic, fine-grained parallelism to keep GPUs busy at batch size one, and efficient inter-GPU communication to avoid network bottlenecks.

Hardware specificity makes this harder. NVIDIA's CUDA ecosystem is extensively documented; AMD's server-class MI300 documentation is thin. Profiling tools for AMD were so limited that researchers in the AMD Developer Challenge 2025 had only end-to-end timing results for each M×K×N configuration, with no fine-grained feedback. A team using an LLM-driven "GPU Kernel Scientist" framework started with a naïve HIP kernel approximately six times slower than PyTorch, then iterated through matrix-core utilization, shared-memory pipelining, and dynamic LDS re-purposing for scale caching. Their best result reached 450 microseconds versus a human first-place time of 105 microseconds on the same hardware; the human team had access to actual MI300 hardware and profiling tools. The LLM system had to probe compiler intrinsics through compilation errors, effectively reverse-engineering the hardware behavior.

The performance ceiling is real: inference speed cannot exceed what the underlying kernels allow. System-level optimizations can help, but they can't fully compensate for inefficient kernels.

Kernel fusion illustrates the payoff. Fusing bias addition and GELU avoids an extra kernel launch and a full write-back to global memory, cutting latency by 15–20 percent on a 70-billion-parameter model inference path. Separate kernels for mean, variance, and scaling would each read the token vector from global memory, tripling bandwidth consumption. Fusion reduces reads to a single pass and keeps data in registers throughout. OpenAI's multi-stage scaling strategy reflects this: early GPT-3 relied on Megatron-LM model parallelism; ChatGPT-4 introduced custom CUDA kernels for attention and feed-forward layers, achieving 1.8× speed-up per token by combining bias addition, activation, and quantization in a single pass; inference-only deployment moved to FP8 quantization on Hopper and TensorRT-accelerated kernels, reducing memory by 75 percent while keeping latency under 50 milliseconds for 32-token prompts. Meta reported a 3.5× increase in queries-per-second compared to a naïve FP16 baseline using FlashAttention, 8-way tensor parallelism, 2-way pipeline parallelism, and KV-cache offloading to CPU pinned memory.

Portability collides with peak performance. Custom CUDA kernels only run on NVIDIA GPUs. Moving to AMD or other accelerators often means rewriting from scratch. Compiler-based solutions like TVM or XLA generate optimized kernels from higher-level descriptions and can target multiple backends, but they lag behind hand-written kernels when new model architectures appear. Frameworks like vLLM, SGLang, and TensorRT-LLM ultimately dispatch kernels for actual computation; they are orchestration layers built on kernel collections. When a new attention mechanism or tensor shape arrives, someone must write and integrate efficient kernels before the framework supports it efficiently. That gap between model release and production-ready performance is where the talent war lives.

Risks and Realities: The Open-Source Commercialization Dilemma

Inferact's formation introduced a new set of stakeholders into a project that had thrived on volunteer labor. vLLM grew to over 2,000 contributors across academic institutions and companies without a commercial entity directing its roadmap.

Open-source maintainers were already stretched before this infusion. Asghar, a maintainer interviewed by IBM, described the baseline: "There's a level of rigor that open-source maintainers have to maintain that is exhausting." That was before generative tooling flooded project inboxes with what Asghar called "AI slop": pull requests that look correct but break the codebase, and security reports hallucinated by large language models. Seth Larson documented an uptick in "extremely low-quality, spammy, and LLM-hallucinated security reports" that waste maintainer cycles on false positives. "The overhead in the open-source community of AI trash being upstreamed is crippling to some projects, to the point where actual things can't get done," Asghar said. Inferact inherits this burden while its investors expect velocity.

The tension is structural: venture capital operates on a seven-to-ten-year horizon with a mandate for outsized returns. Open-source communities operate on trust, transparency, and the long arc of adoption.

Historical parallels offer limited comfort. Linux and Android outpaced proprietary competitors because their open, distributed innovation models mobilized global talent at scale, as the Berkeley CMR analysis notes. But neither Linux nor Android launched with a venture-backed commercial entity holding equity in the core project. The Linux Foundation's 2024 report on open-source AI underscored that "open AI is a different story because AI systems don't behave like traditional software"; they require distinct definitions, protocols, and development processes. The governance model that worked for an operating system kernel may not survive the pressure to monetize an inference engine.

The "open core" pattern looms. Companies that commercialize open-source infrastructure often reserve the highest-value features (enterprise authentication, advanced optimization, managed hosting) for a paid tier. Inferact has not announced such a split. Its job postings signal where the engineering effort flows: the same specialized roles outlined above, all banded at $200,000–$400,000. That talent comes from the same contributor pool that sustains vLLM.

Community trust erodes when the roadmap serves a product strategy. The Berkeley CMR research found that open-source communities iterate weekly while closed-model providers release quarterly. That velocity depends on contributors believing their work benefits the commons, not a single company's margin. If Inferact's enterprise features diverge from the open-source core, the project faces a fork risk, or a slow drain of external contributions toward competing runtimes like TensorRT-LLM or DeepSpeed.

For Inferact, the immediate risk is simpler: can a venture-backed entity steward a communal asset without capturing its value? The answer will show in the commit logs.

Working in AI? Zero G Talent tracks the openings: see every open Inferact role, browse AI jobs, the companies hiring, and the people building the field.