Not a claim. A documented fact.
Documented, falsifiable, and peer-reviewable — with the methodology and dataset published.
DOI 10.5281/zenodo.19503175 · ZenodoMost vendors ask you to trust a number on a slide. We published ours. The full methodology, hardware configuration, NUMA-anomaly data, WAL-determinism analysis, and per-concurrency throughput curves live in a paper with a permanent DOI — and the benchmark harness plus a real public dataset are free to download, so you can reproduce the result on your own hardware. The paper's finding is blunt: across every environment tested, the engine is never the limiting factor. The hardware is.
The engine introduced no ceiling of its own.
On a Pure Storage FlashArray X90R4, the bottleneck migrated entirely off storage and onto CPU. The software never became the limit.
8,151,597 rows/sec, sustained.
3,032,564,040 rows. One machine. Zero failures.
The full 401 GB federal corpus, in 60.5 minutes.
Not a synthetic stress test. The Free Law Project's complete CourtListener archive — 32 structurally heterogeneous datasets, including the 347 GB opinions table — reconstituted into an empty PostgreSQL 18 database in a single concurrent pass, on a consumer desktop.
60.5 minutes wall time. Lossless.
A homogeneous concurrent test proves the scheduler is stable. A heterogeneous corpus proves the engine: citation tables are wide and sparse; opinion clusters are dense with long text; opinions itself is billions of small rows across 347 GB; dockets carry 51 wide columns of highly variable size. The engine adapted batch sizing and arena allocation per job, simultaneously — zero failures across 2.6 billion rows.
No staging. No orchestration. No intermediary.
DataForge streams from a source database straight into a destination database — cross-engine — through the same Parse → Filter → Accumulate → Write pipeline. Source cursor to destination write, one pass.
| Transfer | Rows | Elapsed | Rows/sec | Parity |
|---|---|---|---|---|
| SQL Server → PostgreSQL | 75,814,101 | 3m 03s | 413,325 | Lossless · scanned = written |
| PostgreSQL → SQL Server | 75,814,101 | 5m 14s | 241,009 | Lossless · scanned = written |
The 1.7× difference between directions is protocol-driven, not engine-driven: PostgreSQL's COPY FROM STDIN is one streaming call per batch, while SQL Server's TDS bulk path carries per-row encoding overhead. The DataForge execution path is identical in both directions. The asymmetry is a property of the destination; the engine is a constant.
The same binary, measured honestly in the cloud.
Cloud numbers carry real overhead — cold start, network transport, writes over the wire. These are end-to-end, into managed PostgreSQL, 75,814,101 rows each, zero malformed, zero dropped.
| Cloud | Infrastructure | Rows/sec | Wall time | Status |
|---|---|---|---|---|
| Amazon Web Services | EC2 c6i.4xlarge → RDS PostgreSQL r6g.2xlarge (8 vCPU / 64 GB) | 1,181,807 | 1m 04s | Measured |
| Google Cloud | GCS → Cloud Run Jobs → Cloud SQL Enterprise Plus (8 vCPU / 64 GB) | 895,259 | 1m 25s | Live |
| Microsoft Azure | Azure Database for PostgreSQL · capacity-constrained region | ~440,000 | constrained run | Premier-region run forthcoming |
What the published study establishes
- Bottleneck migration. As storage got faster, the limiting factor moved off disk and onto CPU — the engine kept scaling until the hardware, not the software, said stop.
- A NUMA anomaly in SQL ingestion under high concurrency — documented empirically, with the pressure curves to show it.
- Deterministic output across heterogeneous environments — the same run produces the same reconciled result, machine to machine.
The validation model
Every result on this page is end-to-end — source file (or source cursor) through destination write, with full row-count and checksum validation. Nothing is a partial-pipeline microbenchmark. The runs are derived from a real public dataset, executed under observable and repeatable conditions, and reproducible with a comparable hardware class. Bulk-load runs create tables fresh and load without secondary indexes or foreign-key constraints — standard practice; index builds run after load and are not counted in the movement time.
The honest bound: performance is limited primarily by external system constraints — destination protocol, storage write ceiling, cloud CPU-allocation policy — rather than by DataForge itself. That is the whole point.