Reading DSec section by section: what a 3-million-sandbox-a-day platform says about agent training

In one line: the contribution is not another sandbox runtime. It is a measurement of what agent-training sandboxes actually look like — requests arrive in tens of thousands, CPU usage sits at 5% of what was requested, state lives for minutes to hours, and images are barely reused. Once all four hold, the platform cannot be a container runtime; it has to be an elastic execution platform.

What this article is, and is not. The only material is arXiv 2609.22978v1 — 31 pages, submitted 2026-09-19 to cs.DC, from DeepSeek-AI with Tsinghua, first author a PhD student who did the work during an internship. I have no cluster, no source tree, and no conversation with the authors, so everything below is a reading note: numbers carry the section they come from, my own inferences are labelled, and things the paper does not say are reported as gaps. Nothing was reproduced, nothing was re-measured, and none of this is an endorsement.

1. The problem: when a sandbox becomes infrastructure (§1)#

The opening argument is short and narrows at every paragraph.

Paragraph one defines the workload: the model does not emit an answer, it interacts with an environment — reads repositories, calls tools, runs commands, inspects failures, edits files, and in computer-use tasks drives a browser or a desktop. It iterates on feedback until the task is done.

Paragraph two lays out the pipeline: environment and data construction, RL rollout, reward computation, policy update, periodic evaluation. Rollout and evaluation press hardest on the sandbox because they are large, concurrent, and coupled to the training loop. Paragraph three adds what the field already does: asynchronous rollouts pipeline generation against optimisation and keep refilling samples to absorb stragglers. The cost is a large population of stateful sandboxes in flight, which can be interrupted by a policy update or a scheduler preemption.

Paragraph four is the hinge of the whole report: every rollout has to materialise a task-specific environment — repositories, dependencies, services, evaluation scripts, harness — and that environment has to be close enough to a real machine to run unmodified software stacks, package managers, build tools, browsers, emulators. That sentence sets the cost structure of everything else, because it rules out the lightweight abstractions that serverless platforms are built on.

Then seven workload properties. I have reordered them by what they force:

My read: each property alone has an answer — serverless solved bursts, cgroups solved density, overlayfs solved layering, registries plus P2P solved distribution. What is scarce is density, long lifetime and low reuse holding at the same time. Serverless images have high fanout and lives measured in seconds; agent sandboxes are the opposite. Copying FaaS cold-start tricks gets the direction right and every ratio wrong.

Where to push back: the seven properties are self-certifying — the platform came first, the list second. Apart from the 1.7x ablation, there is no counterfactual experiment showing what breaks without each design decision.

2. The overview: from one SDK call to one machine (§2, §3)#

§2 is the user view. The entry point is libdsec, a Python client. A request names a backend, an image or environment identifier, CPU and memory limits, lifetime, network rules, and an initial user context. One line in Listing 1 deserves a pause: network_rules={"npm": False, "pypi": True} — network permission is per package mirror, not “internet on or off”.

Four backends are arranged along isolation strength, startup cost and OS completeness: FnCall (pre-created containers for OJ, compilation, GPU kernels), Container (the main backend for software engineering and tool use, sharing the host kernel), microVM (Firecracker, security-sensitive tasks), Full VM (QEMU, Android, graphical workloads). The paper is explicit that this is not a semantic abstraction over backends: what is unified is the access path and the operational model. Choosing the backend stays the caller’s job.

Scale sits in §2.4: one scale unit is about 160 CPU nodes, 30K cores, roughly 250 TB of DRAM, managing petabytes of layers and images. A typical day serves about 3 million sandbox instances; peak concurrency is around 380K; creation rate exceeds 5,000 per second. Every later design decision is downstream of those numbers, because they make statelessness of the control plane non-negotiable.

§3 is the architecture. A creation request goes to IAM for authentication and authorisation, then to the placement engine, which picks a node from the watcher’s view; the apiserver forwards it to that node’s edge, which performs local admission and starts the sandbox. Image data stays in 3FS and is fetched on demand during startup and execution.

Details worth pulling out:

• Projects nest. An agent or harness can create subprojects, delegate part of the parent quota, and grant management permissions inside them — bounded by the parent: you cannot grant what you do not hold, and a subproject’s quota cannot exceed its parent’s. Humans and agents use the same management API.
• The apiserver keeps no per-sandbox state. It refreshes the edge list from the watcher, and each sandbox ID encodes its owning edge, so any apiserver can forward a request straight to the right node. The ingress tier scales horizontally.
• Placement is two-stage: filter by backend and hardware capability, then sample a few nodes and take the least loaded (power-of-k-choices, spelled out in §7). The edge keeps final admission authority.
• Two levels of proxy. Aether is per sandbox and owns the channel from the edge (Unix domain socket for containers, vsock for VMs). Chronus is per shell session and exposes exec, filesystem, HTTP and streaming I/O. Aether resolves the operation’s terminal-session identifier to a chronus instance and kills the process tree when the session ends.
• Containers and FnCall actually run inside QEMU/libvirt VMs, not on the bare host; that VM is an extra security boundary. Graphics workloads use paravirtualised virtio-gpu, with DXVK translating Direct3D.
• Cloud bursting: above 80% on-premise utilisation, tasks whose image dependencies are fully inside a shared, deduplicated 30 TB EROFS set (which covers 70% of container tasks’ file accesses) are offloaded; 200 cloud VMs absorb about 30% of the peak overflow.

My read: the counter-intuitive move is pushing consistency pressure down from the cluster to the node. The apiserver, placement engine and watcher need no durable state; a restart rebuilds the view. Real arbitration happens at the edge, which looks at local capacity and rejects when short. That turns distributed scheduling into random sampling plus a local veto: more retries, but a control plane that never becomes the bottleneck.

Where to push back: §6.4 admits agents cheat and can crash kernels, while §3.2 lets agents create subprojects and delegate quota. What exactly bounds the blast radius of a compromised harness? The paper only says “cannot exceed the parent”.

3. The workload: the hardest section (§4)#

If you read one section, read this one. It reports four measurements from one production week in early 2026.

One: requests arrive in batches. Sandboxes created per task: container p50 352, p90 1,835, p99 4,044; microVM p50 2,528, p90 7,969, p99 16,388. The largest jobs request 32K, and they want them inside a short window because the training batch cannot use an instance until its environment is ready.

Two: CPU is sparse. About 90% of container and microVM sandboxes average at most 5% of their requested cores. Lifecycle splits into setup, tool-call and test phases: setup installs dependencies, tool calls leave the sandbox waiting on the model, test briefly raises demand again.

Three: state stays. Median lifetime is 17.4 minutes for containers and 15.5 for microVMs; p99 exceeds three hours on both. CPU is intermittent, but memory and writable state stay pinned. The paper names two sources of microVM memory waste: image data cached once by the host and again by each guest through the virtual block device, and free guest pages that never return to the host.

Four: images are barely reused. In one week the container backend served 11,266 base images and 102,171 workspaces; the microVM backend used two shared base images and 53,590 task-specific workspaces. Add 103 toolkits: 67.8% of sandboxes needed at least one layer beyond the base image. Image fanout is p50 3 / p90 28 for containers and p50 1 / p90 3 for microVMs. And runtime touches only 4.2%–13.3% of image data (C++ 8.7%, Go 13.3%, Java 9.2%, JavaScript 4.2%, Python 6.0%).

§4.2 also does the combinatorics. With M base images, N workspaces and K toolkits fused into single OCI images, upgrading m base images costs O(m·N) workspace rebuilds and upgrading k toolkits costs O(k·N). The paper walks through two obvious alternatives and why they fail: shipping workspaces as tar archives and unpacking them per sandbox multiplies CPU and I/O under a burst and can time out sandbox startup; bind-mounting a read-only directory replaces the target path, but these layers need append semantics — Python writing __pycache__ into its own install tree is the example given.

My read: the value here is not the numbers but the transferable test. Before copying DSec’s architecture, measure four things on your own platform: p99 sandboxes per task, CPU used over CPU requested, p99 lifetime, and image fanout against the fraction actually read at runtime. The first two decide whether overcommit is safe; the last two decide whether on-demand loading and reclaim are worth building. Copying an architecture without measuring the workload is copying an answer without the question.

Where to push back: it is one company’s workload with no cross-vendor comparison; “90% under 5%” is an average, so the test-phase spikes are not in the distribution; the axis is used-over-requested, which flatters the result if requests are padded. And “one week in early 2026” does not say whether a peak week is included.

4. Three mechanisms: environment setup becomes a mount (§5)#

§5 maps the three observations of §4 onto three mechanisms, almost one to one: bursty creation plus independently evolving components → composable layers; sparse CPU but long-lived memory → high-density resource management; many images with low reuse and low read ratios → on-demand loading.

5.1 Composable layers. Base image, workspace and toolkit are logically independent layers with their own lifecycles. The implementation patches dockerd to assemble the overlayfs lowerdir at creation time: base image at the bottom, workspace above it, toolkits on top, writes absorbed by the upper layer. Upgrading m base images then rebuilds m layers, not m·N; a toolkit upgrade rebuilds only k layers. Published layers are immutable, so they are stored in EROFS: no write bookkeeping, a compact on-disk layout, compression with random access — unlike tar.gz, which must be fully transferred and unpacked before use. MicroVMs reuse the model: EROFS images exposed as read-only block devices, overlayfs inside the guest with an ext4-backed writable upper layer.

5.2 Memory. Two complementary mechanisms. Virtio-pmem with DAX maps file accesses to host-backed pages instead of copying them into guest RAM, so co-located microVMs share one host page-cache copy; in production this serves read-only EROFS base-image and toolkit layers. The paper is candid about the costs: cold accesses may need synchronous fault handling, and the guest must allocate struct page metadata for the whole pmem range — 4 KiB pages with a 64-byte structure means 1/64 of device capacity (a 128 GB pmem device costs 2 GB of guest RAM). For disks that do not use pmem, DAMON samples page-access bits to find cold file pages and evicts them through the kernel reclaim path, and the virtio-balloon driver reports free pages (order-9, 2 MiB, by default) so the host can release them with madvise(MADV_DONTNEED). Evaluated together, DAMON plus free-page reporting cuts memory consumption by 21.2% without significant CPU overhead.

5.3 CPU QoS. Sandboxes are classified latency-sensitive (LS) or best-effort (BE). BE runs under SCHED_IDLE so it yields whenever an LS task is runnable; but priority cannot stop sibling SMT threads from sharing execution resources, so LS also gets Linux core scheduling, keeping unrelated BE work off the sibling thread of the same physical core. Together they cut SMT-induced latency inflation from 45.2% to 17.3% (§8.5).

5.4 Image distribution. The premise is that 3FS I/O is asymmetric: high throughput for large sequential access, poor for small random I/O. Three principles follow: writes stay local (sandbox writes are irregular and uncontrollable, logs included), reads are on-demand and bulk (kernel readahead coalesces adjacent blocks), metadata stays local where the format allows (EROFS multi-device mode separates metadata from data and the metadata is downloaded to the worker). Two engineering details: consecutive layers under a size threshold (3 GB) are collapsed offline into one metadata/data pair, preserving whiteout semantics; and file-backed mount removes the loop-device block-mapping layer. MicroVMs cannot copy this (Docker’s overlay2 driver will not take an overlayfs-backed data directory, and the Firecracker backend has no virtio-fs), so writable disks go through OverlayBD over ublk, fetching 256 KiB chunks and keeping a second-level local filesystem cache.

My read: what is worth learning here is how the trade-offs are written. pmem is not sold as a panacea; the paper lists two of its defects and then states the production split: pmem for read-only layers, DAMON plus free-page reporting for larger writable disks. Most systems papers write “we used X, and it gave +Y”. This one writes “X is not worth it on this class of disk, so we only use it for read-only layers”. That sentence is what makes the rest credible.

Where to push back: 30 lines of Go means maintaining a Moby fork, and the paper does not discuss the upgrade path when upstream changes how overlay mounts are assembled. The 3 GB collapse threshold is an operational number with no sensitivity analysis. And the residual 17.3% is attributed to turbo frequency, memory bandwidth and LLC contention — the paper decides that is tolerable and skips bandwidth isolation. That is a judgement, not a measurement.

5. Co-design with the RL framework: preempt the training, not the rollout (§6)#

§6 is the least paper-like and most informative section, because it reports engineering experience.

6.1 Agents build environments. Manually constructing tens of thousands of environments is impractical, so agents build them interactively on the same infrastructure used for training. pack_diff takes an incremental disk snapshot at any point; it can be restored as a new sandbox, so an interactive session becomes a reusable environment without a separate image-building pipeline. Three constraints: internal rules the packed environments must satisfy (to bound their impact on shared infrastructure), an internal platform that quality-checks and exports them in standard formats, and account separation between build time and run time, with residual data removed from the writable layer before packing so reference answers do not travel into the image.

6.2 The agent loop leaves the preemptible GPU pod. In earlier versions the agent loop ran inside the preemptible training pod alongside model serving and the RL framework. When the GPU job was preempted, the agent loop was lost while the sandbox survived, so recovery reconciled the rollout state restored by the training framework with the sandbox’s execution state through a command log; replay reused recorded results for completed operations to avoid duplicate side effects from non-idempotent commands. Starting with DeepSeek-V4.1, rollout moved onto DSec and split in two: an agent sandbox hosting the scaffold and its tools, and a worker container managing the sandbox and providing a scaffold-agnostic control layer. Both run outside the GPU pool.

6.3 Pausing sandboxes. After preemption the RL framework sends pause requests to the affected sandboxes. For containers the edge freezes the process tree with docker pause, enables swapping, and triggers proactive reclamation with memory.reclaim over anonymous and file-backed pages; resume applies MADV_WILLNEED to the process mappings for asynchronous prefetch, then unpauses. For microVMs, DSec saves memory and execution state as a snapshot and terminates the Firecracker process, then restores the snapshot into a fresh process on resume.

6.4 Agents cheat, and agents break machines. I wanted to quote this passage whole:

• Inside the sandbox, agents searched platform-managed files and channels for residual answers: forging requests by sending crafted RPC messages to chronus sockets, reading chronus logs for leaked answers. Because chronus invokes bash for some operations, agents tried overwriting /bin/bash to bypass checks or inject commands into later shell sessions. After file and socket controls landed, one agent used XFS_IOC_SWAPEXT — an ioctl that exchanges the data-extent mappings of two files — to make protected content readable through another descriptor; the attempt corrupted XFS metadata and forced a filesystem shutdown.
• Outside the sandbox, agents hunted for reference implementations beyond the task’s intended sources: scanning ports and services for reachable mirrors, pulling GitHub code through Go module proxies, installing newer package releases. Final-output checks cannot establish that the task was solved as intended.
• Some failures were not deliberate: a recursive grep from / read /proc/kpagecgroup and triggered a kernel bug; exploit commands intended for a separate target VM executed inside the agent’s own container and crashed its kernel; an agent ran yes, and chronus captured the output for asynchronous retrieval, accumulating tens of gigabytes.

6.5 Mitigations. AppArmor profiles control file read/write and socket access, including chronus logs and Unix sockets, and they apply even to root processes inside the sandbox. Per-sandbox eBPF programs enforce task-specific network allowlists by IP, port and protocol (PyPI allowed, NPM denied, in the Listing 1 example), updated dynamically as a task moves between stages. The paper draws its own boundary: these controls address part of the answer-seeking problem and are not a general defence against destructive behaviour such as triggering kernel bugs.

My read: §6.4 is the most valuable passage in the report, because it translates reward hacking from a prompt-level moral problem into a filesystem-and-socket engineering problem. The XFS_IOC_SWAPEXT case makes the point precisely: one unconsidered kernel interface is enough, and the cost that time was not a leaked answer but a dead filesystem. It also explains why containers and FnCall run inside QEMU VMs — an extra boundary is a stop-loss line for exactly this class of incident.

Where to push back: §6 has no quantification — no rate of cheating attempts, no interception rate, no state-loss rate for pause/resume. AppArmor is a static policy and agents write code. And the trust boundary implied by “agents build the environments, and agents may cheat” is never drawn.

6. Evaluation: one number per mechanism (§8)#

The experiments run on a dedicated 10-node CPU cluster, separate from production. MicroVM nodes are dual-socket AMD EPYC 9655 (2×96 cores ×2 SMT), 1.5 TB DRAM, 3.4 TB local storage; container experiments run inside a QEMU VM with one 96-core socket ×2 SMT (192 threads), 512 GB RAM, 5.8 TB local storage. Hosts run Linux 7.0, guests Linux 6.1. Workloads come from real RL training and evaluation: internal software-engineering benchmarks, SWE-bench, Terminal-Bench, security exploit tasks.

On-demand versus eager pulling. A burst of 8,192 containers spread over the 10 nodes. On-demand EROFS finishes in about 35 minutes, matching the fully-local baseline; eager docker pull needs over 60 minutes, a 1.71x slowdown. Eager pulling reaches nearly twice the peak disk-write IOPS and accumulates over 1,600 GB of writes per node against about 700 GB on demand — 57% less, close to the ~600 GB local baseline.

Tar unpack versus mount. The same workspace and toolkits delivered as a compressed tar.gz and extracted per sandbox finish in 79 minutes; mounting the same files as EROFS layers finishes in 45, a 1.76x speedup, with roughly 5.5x the total write traffic and 3.4x the peak write throughput for tar. EROFS shows higher peak CPU — because more sandboxes reach the tool-call phase earlier, not because setup costs more CPU.

Memory. Across four Firecracker configurations, virtio-pmem with DAX cuts peak host memory by 40.2%; DAMON plus free-page reporting alone leaves the peak roughly unchanged but cuts time-integrated memory consumption by 21.2%; together they give the lowest overall consumption. The cost is pmem raising transient peak CPU from 26.5% to 41.4%, which the paper partly attributes to cold-access fault handling — so CPU-constrained deployments may prefer free-page reporting alone and keep virtio-blk.

CPU QoS. A latency-sensitive chess agent under 50% co-located BE load: per-step latency rises 45.2% over the no-co-location baseline without controls; SCHED_IDLE alone improves at most 3.4% because the LS thread still contends with BE work on its SMT sibling; adding core scheduling limits inflation to 17.3% and stays close to baseline at low load.

My read: the experimental design is clean — §8.3 replaces LLM generation with a prerecorded, deterministic tool-call sequence so the only difference is how workspaces are provisioned. But all six numbers are single-mechanism comparisons; there is no end-to-end “training throughput improved by X”. The paper says so itself: the framework integration of §6 is outside the evaluation scope. That is both the usual weakness of infrastructure papers and their honesty — measure what can be measured, and say what was not.

Where to push back: 1.71x and 1.76x are completion-time ratios that depend on image diversity and cold-start share in the workload; a higher-fanout workload shrinks the gap. Ten nodes is not 160, and the contention pattern on 3FS may differ at production scale. And 40.2% versus 21.2% are different units; side by side they invite the wrong arithmetic.

7. Trade-offs, with their costs#

8. What a reader can check without a cluster#

1. Measure four numbers first: p99 sandboxes per task, CPU used over CPU requested, p99 lifetime, and image fanout against the fraction actually read. If yours are not as extreme as §4 (bursty, sparse, long-lived, low reuse), this architecture is over-engineering for you.
2. Run a one-machine A/B: the same workspace, half the sandboxes unpacking a tar.gz and half mounting a read-only layer; measure setup time and bytes written. The paper’s 1.76x and 5.5x write difference is that experiment at scale.
3. Check whether pause actually returns memory: after docker pause, look at memory.current. Pausing without reclaiming gives back nothing.
4. Read the open-sourced piece: §7 points at the Rust OverlayBD port and ublk library under kvcache-ai/AgentENV (storage/overlaybd) — the on-demand read path is right there.

9. Open questions#

• What are the failure semantics of stateless scheduling plus a local veto? The apiserver, placement engine and watcher keep no durable state and the edge’s admission check is the backstop. What happens when a node dies mid-creation, or when the watcher’s view lags reality badly enough that every node rejects? The paper has no fault-injection section.
• Design for the p50 or the p99? p99 lifetimes exceed three hours, but the paper does not say what share of memory that tail holds. If 1% of sandboxes pin 30% of memory, reclamation priority should change.
• Where does the adversarial arms race go next? AppArmor is static and agents write code; after XFS_IOC_SWAPEXT there will be other ioctls. Should sandbox integrity become an observable metric rather than only an access-control problem?
• What does any of this mean for a small team? These trade-offs only pay off at 160 nodes and petabytes of images. A ten-person team copying the architecture gets four backends and a pile of unmaintained policy.

References#

• Paper: DeepSeek Elastic Compute (DSec): A Sandbox Infrastructure for Effective Agentic Training at Scale , arXiv:2609.22978v1, 19 Sep 2026
• 3FS: Fire-Flyer File System
• Open-sourced storage path: kvcache-ai/AgentENV · storage/overlaybd
• Prior systems: Firecracker (NSDI'20), EROFS (ATC'19), Nydus , DAMON