kPivot

The starting problem

A data migration rarely fails on the transfer. It fails on everything around it.

• The script that loaded the data the first time cannot say what it wrote, nor what it left behind.
• Run again after a fix, it duplicates what had already made it across — or the target has to be emptied and the whole thing redone.
• Interrupted on the four hundred thousandth row, it starts again at the first.
• Refused rows end up in a log nobody reads: the gap surfaces months later, while looking for a customer record.
• The project itself — waves, freeze windows, cutover, sign-off — lives in a spreadsheet alongside, with no link to what actually ran.

What kPivot does today

Everything below works on record migrations — databases, files, APIs, search engines: whatever the Kore platform connectors can read and write. The screens shown in a demonstration run on an entirely fictional dataset: an invented company, invented systems, synthetic runs and rejects.

A plan, and what it declares

A migration plan names a source, a target, a field mapping and a replay rule. It also declares the batch size, how many rows the transform handles in parallel, the share of rejects beyond which a run must stop, and whether it is a dry run. The plan is inert: only a run writes. It can be cloned — the second wave is usually the first with another filter — and it leaves the working catalogue by being archived rather than deleted, because its runs are the record of what happened to production data.

Four ways to read the source

Full: the whole source, on every run. Partial: a declared subset — one wave, a list of identifiers, the corrected rejects. Incremental: whatever changed since the marker left by the last successful run, for the nightly syncs of the period when both systems are live. Continuous: a stream, where the source can produce one. All four compose on a real project, and replay in whatever order you need.

Replay without duplicating

This is the question that settles everything else. Three answers, declared by the plan: the natural key of the target, which updates instead of inserting; the pivot log, which remembers which target object each source object became; or none at all, reserved for an empty target and a single run — and which has to be chosen explicitly, never by default.

A dry run that writes nothing

A dry run reads and transforms for real, then writes nothing. It produces the same counters, the same rejects and the same diagnostics as a live run: a dress rehearsal, not a preview. A dry run can also be forced when launching a live plan; the reverse cannot happen: going back to real writes means editing the plan.

A resume point at every batch

Every committed batch writes its resume point: how far the source has been read, the date reached and the batch number. An interrupted run — network failure, restart, error threshold crossed — resumes where it stopped instead of starting over. That point only advances once the batch is safely written: an interruption makes the run re-read rows, never skip them.

An error threshold that really stops

Beyond the share of rejects declared by the plan, the run stops on a clean batch rather than producing wrong data at scale. A plan can demand a stop on the first reject, or never stop at all — that is a choice, and it is written into the plan.

Rejects you can act on

Every refused row keeps the phase where it died — read, transform, validate, write —, a stable reason code to group by, a readable message and a diagnostic. The dashboard breaks down by reason the rejects of the last run that ended badly: four hundred rejects sharing one cause are one fix to make, not four hundred decisions to take. The original row is kept exactly as it was read, so the replay has something to start from; it never leaves through the API and no screen displays it — carrying personal data that is never shown anywhere would be a risk taken for nothing.

The project in the same place

Projects, waves, tasks and defects are Kore objects with their own lifecycles: a project runs from scoping to cutover and then verification, a wave from planning to the final check on discrepancies — or to rollback. Lists, kanban, timeline and "my work" build on them, with assignments, a timer and history. A migration plan attaches to its project: what ran and what remains to be decided no longer live in two different tools.

Field mappings, and the return migration

The transform is not reinvented: field mappings are the platform ones, shared with the other applications, with their validation, their lookup tables and — what matters on the day of a rollback — their inversion. A return plan is the outbound plan with its mapping inverted.

The pivot log, object by object

A run’s counters say four hundred thousand rows made it across. The pivot log says that this ticket became that one, through which mapping and in which direction. It is the trace that answers a precise question months later, and it is also what the pivot-log replay rule reads.

Trends, not just the present moment

A long migration replays the same plan dozens of times. The volume processed, the rejects and the duration of each plan are measured on every run and kept: a curve shows whether rejects are falling week by week, and whether a run is taking longer and longer. Ready-made dashboards save starting from a blank page.

One API, and access for assistants

Everything the screens do goes through the same GraphQL API, under the same permissions and the same separation between organisations. Three tools exposed to assistants give the state of plans, runs and rejects, read-only: an agent can report where the migration stands without being able to launch anything.

What the exploration showed

kPivot came out of an internal observation: every data migration rewrote its own script, and none of them left a usable trace. The question was whether the Kore platform made it possible to turn that into a tool.

Frequently asked questions