46‑Minute Bank API Outage Fixed in 72 Seconds

Workers IO Builds Deterministic Simulation for Mission-Critical Software

Gartner forecasts worldwide generative AI spending at $644 billion in 2025 — a 76% jump from the prior year. Yet the software these systems produce ships with a reliability problem that no amount of capital has solved. When an agent can rewrite an entire service, the cost of a production failure compresses from weeks to minutes, and the failures that matter are the ones no test suite can catch.

Workers IO emerged from Y Combinator's Fall 2026 batch as a two-person team in San Francisco. Chaitanya Choudhary, who founded and sold a company a decade ago and later worked at Grit and OnDeck, launched the venture on a simple premise: conventional test suites cannot verify the next generation of mission-critical software. The company's platform maps every scenario a system can face in production and verifies each release against all of them in deterministic simulation environments, returning every failure as an exact, replayable run.

These are not edge cases — they are the default condition of any distributed system under load. A bank API that goes unreachable mid-transaction, two database writes that collide in the same millisecond, a timeout that arrives after the retry window has closed. These are the failures that slip past unit tests, integration tests, and even chaos engineering tools like Jepsen.

The platform walks teams through three steps. Step 01 asks engineers to define what must never break, turning system guarantees into explicit promises: no lost writes, no double fills, no split-brain. Step 02 simulates production scenarios across timing, network, dependency, retry, crash, and service behavior, exploring millions of interleavings. Step 03 isolates failures with exact context so agents and engineers can reproduce, fix, and verify the fix before release. Every failed run reproduces perfectly: the injected faults, the full execution trace, and each dependency's return values. Reproduce it, shrink it to the root cause, and verify the fix under the same conditions.

The concrete cost is visible in scenarios Workers IO has already simulated. A card payment attempt while the bank API is unreachable eats 46 minutes of production time — and resolves in 1 minute 12 seconds. In one path, no money is lost even while the bank is unreachable; in another, no customer gets charged twice because retries never produce a duplicate charge. A third failure mode shows every charge gets a receipt, but the receipt was never written — a customer pays with nothing to show for it. Across those 46 minutes, two of three promises held.

Workers IO is hiring for one engineering role — a specialized problem. Building a deterministic virtual machine that replays exact production failures across millions of scenarios demands depth in kernel development, hypervisor logic, and virtual clock architecture, work at the intersection of distributed systems and systems verification.

The agent era demands a new kind of reliability infrastructure. When a change is proposed and verified against the scenarios the system will face in production, the feedback loop compresses from days to minutes, but only if the verification substrate is fast enough to be useful. Workers IO bets deterministic simulation is that substrate, and that engineers building for space, defense, and robotics will adopt it first.

How Deterministic Virtual Machines Replay Exact Failures

Traditional test suites assume the world behaves the same way twice. In production, it does not. A retry fires during a partial outage. Two concurrent writes interleave at the database layer. A dependency times out at exactly the wrong millisecond. These are the norm for any distributed system under load. They are the failures that no conventional test suite can catch.

The difference is architectural. Jepsen, created by Kyle Kingsbury, stands outside the system and throws rocks at it. It can tell you that a bug exists, but it cannot tell you why, and it certainly cannot guarantee reproducibility. When a fuzzer finds a crash in a deterministic simulator, it does not send a 5 gigabyte core dump. It sends a single 64-bit integer: the seed. A developer plugs that seed into their local environment, hits run, and the database fails in the exact same way, at the exact same instruction, with the exact same state transition.

This capability rests on a well-established pattern in systems engineering: record-replay debugging. The same idea that game engines use for replays and distributed systems use for fault diagnosis applies directly to agent systems. FoundationDB, the backbone of Apple's iCloud, was built this way from day one. Its simulator is so powerful it can simulate an entire cluster, including the hardware, in a single process. The team will not merge code unless it survives millions of simulation hours. More recently, TigerBeetle, a high-performance financial ledger, has pushed this further with a technique called Viewstamped Replication Revisited and a deterministic fuzzer called Vulture. They simulate bit-rot on disks, corrupted memory, and Byzantine network behavior while maintaining 100 percent reproducibility.

Deterministic replay requires virtualizing the entire world around the state machine. You must turn the environment into a pure function. In a cloud-native environment, input is not just the data sent by the user — it includes the exact nanosecond a packet arrives, the order in which epoll returns file descriptors, the specific latency of an NVMe write, and the memory address malloc returns, which affects hash map iteration order. Three sources of non-determinism break replay: the network, where packets are delayed, reordered, dropped, or duplicated; concurrency, where modern CPUs handle thread scheduling through the OS kernel; and the local environment, including hidden inputs from system calls like getrandom, reads from /dev/urandom, and the CPU's RDTSC instruction, all of which inject entropy that changes every run.

For AI agents, the problem is even sharper. Traditional software is deterministic, so you can reproduce bugs by recreating inputs. Agent systems are not. Every run is a unique combination of model sampling, live API responses, and time-dependent state. Setting temperature to zero does not solve this. Temperature only controls the token sampling step. It makes greedy decoding select the highest-probability token, but the logits feeding that selection are themselves variable. The primary culprit is batch composition. Hardware heterogeneity makes it worse. A cloud provider's GPU fleet includes multiple architectures, such as H100s, A100s, and sometimes older cards, and different GPU architectures implement matrix operations with slightly different numerical behavior.

Research on five large language models configured for deterministic output found accuracy variations up to 15 percent across runs, with a 70 percent gap between best and worst performance. Even OpenAI and Anthropic acknowledge this. OpenAI's seed parameter improves reproducibility but does not guarantee it. Anthropic's documentation states that even at zero temperature, results will not be fully deterministic.

The tools themselves are nondeterministic because they call the real world. A search returns different results today than yesterday. An inventory lookup returns a number that has since changed. An external API was slow, rate limited, or returned an error that one time and never again. The clock is nondeterministic. If a prompt contains "Today is August 22, 2026" or logic branches on the current time, the run behaves differently depending on when it ran. And the plumbing is nondeterministic: random IDs, request ordering when tool calls run in parallel, retries that fire on one run and not another.

Record what each source actually produced during the run, then feed those exact values back in during debugging. A trace is a photograph of the run; what you want is the ability to press play on it. That distinction is the whole game. One layer tells you what occurred; the other reconstructs it. A durable execution layer checkpoints state so a crashed run can resume forward from where it stopped, moving on with fresh model calls and live tool responses. A replay tape captures every input and output so you can re-run a finished execution backward, feeding it recorded responses instead of calling anything live, to reproduce a specific failure and step through it.

If your replay calls the model again, it is a new run wearing the old run's clothes, and it will not reproduce the bug. Record every run, not just the errored ones. You do not know in advance which run a customer will complain about. If you keep tapes only for errored runs, you miss every failure that returned a two hundred and did the wrong thing anyway — and those are most agent failures.

Deterministic replay turns debugging into a precise, repeatable scientific method. It guarantees bit-for-bit identical reconstruction of past execution states for root-cause analysis. The system must transition through identical internal states on every replay, which requires capturing the complete initial state, including all memory, register values, and random seeds. Any divergence, even a single bit, breaks the replay guarantee.

The replay log must be tamper-evident to serve as an audit artifact. Each state transition gets hashed in a Merkle tree, the root hash is timestamped via a Trusted Timestamp Authority, and the log lives on write-once-read-many media. Replaying the same log N times must produce the exact same final state and outputs every time. That idempotency generates identical model inference fingerprints for audit comparison, lets a compliance officer see the same result as the original system, and supports exactly-once semantics in distributed event sourcing.

You turn a deterministic simulator into a cloud-native fuzzing engine. A CI/CD pipeline generates 100,000 different seeds, each representing a unique universe with different network latencies, disk failures, and clock drifts. A 24-hour cluster stress test with multiple node failures and network splits executes in seconds, because the simulator jumps to the next event instead of waiting for timers to expire.

The biggest challenge in cloud-native databases is the cloud part. Your database likely interacts with S3, an IAM service, or a Kubernetes API. These are external, non-deterministic actors. To keep the simulation deterministic, you must build mock providers that are also driven by the simulator's pseudo-random number generator. In Rust, you might define a TimeProvider trait. In production, this resolves to std::time. In simulation, it resolves to SimTime. The actual business logic, including the Raft log, the LSM-tree, and the query optimizer, remains exactly the same.

Deterministic replay is not an add-on feature — it shapes how you write every line of code. Banish std::thread, std::chrono, and rand() from your core logic. When a customer reports a bizarre, once-in-a-year edge case, you do not ask for logs and hope for the best. You ask for the seed, spin up the replay, and watch the bug happen right before your eyes.

Replay also serves as a regression testing tool. Capture production runs as golden traces, then replay them against new code versions. You are not checking for exact output matches — the model might produce different tokens. Instead, you check for structural equivalence: did the agent call the same tools in the same order, extract the same key information, reach the same conclusion? This catches prompt regressions that are nearly invisible otherwise, such as updating a system prompt to improve one scenario and inadvertently breaking five others. The same pattern works for model upgrades. Before switching from one model version to another, replay a representative sample of production traces and compare behavioral metrics. This is more reliable than benchmark scores because it measures performance on your actual workload, not synthetic tasks.

Deterministic replay has real limitations. It cannot replay what it did not record. If an agent has untraced side effects, such as writing to a database, sending emails, or modifying external state, those will not be captured or replayed. It does not explain why the model said what it said. Replay shows the exact sequence of inputs and outputs but does not provide mechanistic interpretability. Trace storage becomes a data governance problem. Traces contain the full prompts and responses from production, which often include user data. Model deprecation breaks historical replay. If you recorded traces against GPT-4 and the model is later deprecated, you can still replay using the recorded outputs, but you cannot do counterfactual replay against a model that no longer exists.

For teams running agents at scale, this capability transforms incident response from guesswork into engineering. Start by instrumenting your most failure-prone agent workflow, capture a week of production traces, and replay the failures. You will learn more about your agent's behavior in that first week than months of log-reading ever taught you.

Why Space, Defense, and Robotics Engineers Are Watching

Space missions cannot be fully tested before launch. Space environments cannot be replicated on the ground, and launching a spacecraft just to test it is neither practical nor affordable. That constraint means validation must maximize coverage with realistic, efficient test strategies, achieving the highest confidence in software behavior without exhaustive end-to-end testing in real conditions.

Modern defense and robotic systems pack structures, electronics, sensors, software, communications, propulsion, and thermal management into increasingly compact, mission-critical platforms. Multiphysics simulation helps teams evaluate these interactions earlier, reducing uncertainty, limiting costly prototypes, and building confidence before systems reach the lab, test range, or field. Structural durability demands that frames, mounts, enclosures, and payload hardware withstand shock, vibration, fatigue, drop, and impact. Thermal performance matters because ruggedized electronics, sensors, compute modules, batteries, and actuators generate heat in tight spaces. Radio frequency and EMI/EMC risks require evaluating antenna placement, co-site interference, radome effects, and coupling in crowded electromagnetic environments. Blast, impact, and high-strain events add another dimension: defense systems and unmanned platforms may face crash, penetration, debris impact, blast loading, and transport shock. Advanced simulation lets teams evaluate these risks virtually before committing to costly or destructive tests.

The market data explains why engineers in these domains are paying attention.

Botondynamics reports 6+ technology verticals, 14+ active missions, and 99.97% platform uptime — a sense of the scale and criticality these systems demand.

RISC-V offers a deterministic foundation relevant to both simulation infrastructure and the labs these domains serve. RISC-V processors from Microchip, built on SiFive's X280 core, power NASA's Jet Propulsion Labs High-Performance Spaceflight Computing processor, designed to deliver at least 100 times the computational capacity of existing spaceflight computers. In December 2025, Qualcomm acquired Ventana Micro Systems, a RISC-V server chip designer, signaling accelerating adoption. RISC-V's open instruction set eliminates vendor lock-in and simplifies certification. Pre-certified cores and virtual models cut certification costs and risks. The architecture supports deterministic runtime execution and built-in security for safety-critical applications, including DO-178C Design Assurance Level A certification through dissimilar redundancy strategies. Pre-certified IP cores from Synopsys, Microchip, SiFive, and CAST further reduce certification overhead, integrating error detection and correction, watchdog timers, and memory protection units.

Defense virtualization connects to the deterministic simulation approach. PikeOS is a certified RTOS and hypervisor built on a separation kernel architecture. It consolidates diverse, high-assurance code, from vehicle diagnostics (ISO 26262) to encrypted communications (Common Criteria), on a single hardware platform. That separation is non-negotiable for defense, where hardware failure or a cyberattack cannot compromise the mission. PikeOS meets stringent standards across sectors, including ISO 26262 for automotive and DO-178C for avionics. Its modular certification lets teams add mission features without re-certifying the entire stack, cutting time-to-deployment. By consolidating functions onto fewer processing units, PikeOS reduces size, weight, and power, which is critical for defense vehicles under severe mass, energy, and cooling constraints. Its guaranteed determinism keeps Edge AI and critical control loops running reliably in real time.

Deterministic simulation addresses a problem common to all three domains: exhaustive testing is impossible, but reproducible failure understanding is non-negotiable. Space missions cannot replicate their operating environment on the ground. Defense platforms face electromagnetic extremes, thermal stress, and physical shocks that are hard to reproduce consistently. Robotics systems must operate reliably in unpredictable real-world conditions while managing countless interleaved operations. These are the failure modes deterministic simulation is built to catch — the retry that lands when the network is partially down, the writes that interleave without warning, the dependency timeout that arrives at the worst possible instant. No test suite surfaces them. The tech sector faces a costly paradox: companies pour money into AI infrastructure while software reliability problems persist. Simulation platforms that can replay failures across millions of scenarios offer a path from chasing bugs after they hit users to preventing them before deployment. These engineers are watching because this approach addresses a gap that has persisted since the earliest days of mission-critical software — the distance between what teams need to prove and what their tests can show.

One Hypervisor Posting Reveals the Depth of the Problem

A Workday posting for a hypervisor engineer, listed without a discernible date and based in Spring, Texas, signals how deeply Workers IO must burrow into system architecture. A platform mapping millions of production scenarios and replayable failures cannot stay at the application layer. Trapping non-deterministic events, such as retries during partial outages, writes that interleave unpredictably, and dependencies timing out at the wrong instant, demands understanding how requests pass through kernel boundaries and how virtualization layers intercept those paths. This hiring need reveals that deterministic simulation is a systems engineering problem, not a testing tool problem.

Traditional test suites operate at higher levels of abstraction. They mock dependencies, they record happy-path sequences, they assert outcomes against expected inputs. But the failure modes that deterministic simulation exists to catch, including interleaved writes during partial outages, retries that succeed only because timing shifted, and timeouts that cascade through nested calls, happen in the spaces between application logic and the kernel's scheduling decisions. A hypervisor engineer would need to work at the level where virtual machines request CPU time, where memory pages are allocated and freed, where interrupt handlers decide which pending operation gets serviced first. These are the zones where non-determinism either originates or can be reliably reproduced.

The kernel hypervisor boundary is where determinism gets technically expensive. To replay an exact failure across millions of scenarios, the simulation must freeze state at any point, restore it precisely, and re-execute along the same path. That requires hooks into the kernel's scheduler, page table management, and the mechanisms governing when one virtual machine yields to another. Workers IO's need for this depth suggests the platform is building replay capability closer to the metal than any conventional CI/CD tooling.

This hiring signal aligns with broader AI infrastructure trends. The same companies investing heavily in AI infrastructure this year are buying compute capacity, not software reliability. Mission-critical software in space, defense, and robotics cannot tolerate a development approach that prioritizes velocity over stability. Workers IO, founded in 2026 and already hiring for kernel-level work, suggests the market for deterministic assurance is shifting from academic curiosity to operational necessity.

This niche has been underserved because building systems that simulate production behavior with fidelity and guarantee replayable execution demands expertise at the intersection of operating systems, virtualization, and testing methodology. Tests that cover code paths are not enough; the tests must cover behavior under timing pressure, resource contention, and concurrent-operation interleaving that only emerges at scale. A hypervisor engineer brings the low-level knowledge to ensure that what gets simulated matches what actually happens in production — not a sanitized approximation.

Hypervisor engineers must understand not just how VMs run, but how the kernel mediates their execution to ensure replayable determinism across millions of scenarios. This is the layer where such issues can be reliably reproduced. As AI agents ship mission-critical software at agent speed, the failures traditional testing cannot catch will multiply. Companies building infrastructure to catch those failures need engineers who understand the stack from the application layer down to the kernel's deepest scheduling decisions. That is the signal Workers IO's posting encodes.

Capital Outpaces Engineering

Gartner forecasts worldwide GenAI spending, yet such software has a reliability problem that spending has not solved. The gap between capital deployment and verified correctness is widening, not narrowing.

AI agents can write and deploy code faster than traditional verification can catch the failures that matter. Workers IO is one of a few companies trying to close that gap. Its platform runs millions of production scenarios through deterministic simulation, replaying every failure exactly as it occurred — the kind no standard test suite catches. A retry landing during a partial outage. Two writes interleaving at exactly the wrong moment. A dependency timing out under load. These are the failure modes that emerge when software runs at agent speed across distributed systems.

The scale of the AI infrastructure buildout makes the stakes explicit. Anthropic alone carries 530 salaried roles at a median band of $405,000 and added 45 in the past week, according to Zero G Talent's board data; Databricks added 42, Zero G Talent's data shows. But hiring more engineers does not automatically close the verification gap. The buildout depends on the same core workforce, including computer specialists, engineers, and technicians, and rising competition for those roles has emerged as a strategic risk.

Capital cannot solve every reliability problem. You can throw compute at a model, but not at a race condition. As AI agents compress development cycles from weeks to hours, the window for catching failures before production shrinks with them. Platforms like Workers IO represent a shift in how the industry thinks about verification — not as a final gate before shipping, but as a continuous simulation layer running millions of scenarios in parallel. Every failure leaves a seed: a single integer that reproduces the exact moment everything broke. The industry's challenge is not finding those seeds but heeding them before millions more are generated.

Gartner forecast on worldwide GenAI spending

Working in AI? Zero G Talent tracks the openings: see every open Databricks role, browse AI jobs, openings at Anthropic and Harvey AI, and the people building the field.