Where does DataForge run?

DataForge runs as a single binary inside your own environment, next to your data. In its marketplace form it is a zero-trust, ephemeral engine: it stands up, runs one job from your source to your destination inside your walls, sends back only a signed, content-free run meter, and self-destructs. Your data never leaves your premises.

How it works

One pass. Every row accounted for.

Q: How does DataForge move data without a staging layer?

DataForge streams directly from the source cursor to the destination write in one pass — Parse, Filter, Accumulate, Write. There is no intermediate file, no extract step, no message queue, and no coordination service. The same pipeline that ingests flat files applies unchanged to live database sources across engines.

Q: What is Δ0 (delta-zero)?

Δ0 means every accepted unit reconciles to exactly one terminal state — inserted, malformed, dropped, or skipped — and scanned equals written. The net row-count delta is zero. There is no silent discard: nothing disappears in the middle of the pipeline unaccounted for.

DataForge moves data in a single streaming pass and reconciles the entire population to zero unaccounted. No staging tables. No orchestration. No reverse path.

Ask most tools how they move data and the honest answer is in stages, and we hope nothing falls out along the way. DataForge answers differently: it streams straight from the source cursor to the destination write in one pass — Parse → Filter → Accumulate → Write — and every single row ends in exactly one of four terminal states. Scanned equals written. The net delta is zero. Nothing disappears in the middle.

The four-stage core

Parse → Filter → Accumulate → Write

The same pipeline handles a flat file, an object store, an HTTP feed, or a live database. It does not change shape per dataset.

Parse

Read the source stream and infer structure on the fly — schema, column mapping, row boundaries — without a pre-declared model per source.

Filter

Evaluate every row against policy. Malformed and policy-excluded rows are classified here — never silently swallowed downstream.

Accumulate

Batch into the destination's most efficient write shape, sizing batches and arena allocation to the workload and the pressure on the system.

Write

Stream into the destination in a single protocol call per batch — PostgreSQL COPY FROM STDIN, SQL Server bulk — then verify what landed.

Δ0 accounting

Four terminal states. Zero ambiguity.

Every accepted unit resolves to one of these — and the four counts reconcile against the source. That reconciliation is Δ0.

INSERTED

Landed and verified in the destination.

MALFORMED

Failed structural validation — counted, not hidden.

DROPPED

Excluded by an explicit policy — counted, with the reason.

SKIPPED

Intentionally not processed — counted and reported.

In the full 401 GB CourtListener corpus run — 32 heterogeneous datasets, 2,608,874,016 rows — the result was 0 malformed, 0 dropped, 0 row-count delta. Lossless is not a marketing word here. It is a number that has to close.

The governing framework

Three clocks, not one

"How fast is it?" is the wrong question. What a business actually waits on is three distinct intervals.

Clock 1

Data Movement Time

How long the bytes take to travel from source to verified destination. This is the interval most tools quote — and the one DataForge collapses.

Clock 2

Restoration Time

How long until the data is whole and trustworthy at the destination — reconciled, verified, and safe to build on. Movement without this is just copying.

Clock 3

Time to Function

How long until the organization can actually do something with the data. This is the clock the customer feels. Turning hours into minutes means moving all three.

Zero-trust by construction

It runs where your data lives.

DataForge does not ask you to ship your data to us. It ships itself to your data.

One binary, inside your walls

The engine deploys as a single binary in your own environment — on the marketplace, an ephemeral per-job container that stands up, runs one job from your source to your destination, and self-destructs.

Only telemetry leaves

What comes back to us is a signed, content-free run meter — counts and bytes for billing and assurance. Never your rows, never your schema, never your data.

Any source, any target

Files, cloud object stores, HTTP, and live databases. PostgreSQL and SQL Server are interchangeable as source or destination — cross-engine transfer is the same one pass.

Verified on arrival

An independent destination verification and the closed per-item ledger run as part of the job — not a separate audit you have to remember to do.

Questions

Common questions

How does DataForge move data without a staging layer?

It streams directly from the source cursor to the destination write in one pass. There is no intermediate file, no extract step, no message queue, and no coordination service. The same pipeline that ingests flat files applies unchanged to live database sources across engines.

What exactly is Δ0?

Every accepted unit reconciles to exactly one terminal state — inserted, malformed, dropped, or skipped — and scanned equals written. The net row-count delta is zero. Nothing disappears in the middle of the pipeline unaccounted for.

Does my data ever leave my environment?

No. In its marketplace form DataForge is a zero-trust, ephemeral engine that runs entirely inside your premises. Only a signed, content-free run meter is sent back — never your data.

What sources and targets are supported?

Any source to any target: flat files, cloud object stores, HTTP endpoints, and live databases including PostgreSQL and SQL Server. Databases are treated as interchangeable sources and destinations.

See the evidence Run it on your own data