Data Migration: How to Move Business Data to a New System Without Losing or Corrupting It

When a business replaces an old system, most of the discussion is about features, screens and price. The data that has to move across usually gets a single line in the project plan. Yet a new system is only as trustworthy as the records loaded into it on the first day.

Data migration is the work of moving customers, products, orders, balances and history from the system you are leaving into the one you are adopting. It applies whether you are buying an off-the-shelf product or commissioning custom software development, and it is where many otherwise sound projects slip.

This guide explains why the work is harder than it looks, the steps that keep data from being lost or corrupted, and the decisions you will be asked to make along the way.

Why data migration is routinely underestimated

From a distance, migration looks like a copy job: export from one place, import to another. In practice the two systems rarely agree on what a customer, an order or a status is. The old system also holds years of workarounds, such as codes typed into notes fields and records kept "just in case", that nobody documented.

The people who know what the data really means are operational staff with full-time jobs, and the problems stay invisible until the first test load fails. Microsoft's implementation guidance for Dynamics 365 calls migrating data "a complex and time-consuming process" that needs coordination, testing and validation, and advises giving it its own time and people in the project plan. That advice holds for any business system.

Take inventory, then decide what moves and what is archived

Start by listing everywhere business data lives: the main database, spreadsheets kept alongside it, document attachments, shared mailboxes and any other system that exchanges data with it. For each source, record who owns it, roughly how much there is and who relies on it.

Then sort the data into three groups: move, archive, and delete. Master data (customers, products, suppliers) and open transactions (unpaid invoices, orders in progress, stock on hand) normally move. Closed history is often better kept in a searchable archive than loaded into the new system, where it adds cost and risk without helping daily work.

Retention rules affect this decision. In the UK, the Information Commissioner's Office states that you must not keep personal data for longer than you need it. In the US, the Federal Trade Commission's Start with Security guide advises holding on to information only as long as you have a legitimate business need. Tax, employment and industry rules may equally require you to keep some records, so check before deleting anything. This is general guidance, not legal advice.

Map old fields to new ones

A field mapping is a document that lists every field being moved, where it lands in the new system, and what has to happen to it on the way. Some fields map directly. Others must be split (one "name" field into first and last name), merged, translated (old status codes into new ones) or given a default value.

Every rule in that document is a business decision, not a technical one. If the old system had seven customer types and the new one has four, someone in the business must decide where the other three go. Assign an owner to each data area and have them approve the mapping before any code is written.

The data quality problems that surface

Old data is never as clean as people believe. Four problems appear in almost every migration:

• Duplicates: the same customer entered several times with slightly different spellings, each with its own order history.
• Free-text fields: notes boxes holding phone numbers, credit terms or delivery instructions that the new system expects in structured fields.
• Missing values: fields that were optional in the old system but are required in the new one.
• Inconsistent formats: dates, phone numbers, postal codes, units and currencies recorded in several ways. A date written 03/04/2025 means March 4 to a US office and 3 April to a UK one.

Decide early whether each problem is fixed in the old system before the move, corrected automatically by a rule during the move, or accepted and flagged for later. Deciding which of two duplicate records survives is a judgment for the people who use the data.

Trial runs, reconciliation and sign-off

Never run a migration for the first time on go-live day. Run the complete process into a test copy of the new system, with the full data set rather than a sample, and expect to repeat it several times. Each run exposes records the new system rejects and shows how long the load takes. Fix problems in the migration scripts or the source data, not by hand in the target, so the next run is repeatable. Microsoft's guidance recommends verifying the migration in both system integration and user acceptance test environments.

Reconciliation is how you prove that what arrived matches what left. It usually combines record counts for each type of data, control totals such as the sum of open invoice balances or stock quantities, a report of every rejected record with its reason, and spot checks of sampled records by staff who know those accounts.

Agree the acceptance criteria before the first trial, and have a named business owner sign off each data area. A sign-off from the developers alone only confirms that the scripts ran.

Big-bang or phased cutover

In a big-bang cutover, everyone stops using the old system and starts on the new one at the same moment, often over a weekend. There is one source of truth and no temporary bridging between systems, but all the risk lands at once, so it depends on rehearsal and a tested way back.

A phased cutover moves one department, location or module at a time. Each step is smaller and teaches lessons for the next, but two systems run side by side for a while. That means temporary integrations or double entry, and firm rules about which system is the master for each record. Big-bang tends to suit tightly interlinked data and a business that can tolerate a short stoppage. Phased suits units that operate fairly independently.

Either way, the cutover follows the same sequence: freeze changes in the old system, take the final extract, load, reconcile, then make a go or no-go decision. Afterward, keep the old system available in read-only mode for an agreed period. Staff can look things up and you keep a reference if a discrepancy appears, while nobody can enter new data in the wrong place.

Keeping data secure during the move

A migration multiplies copies of your data: export files, staging databases, spreadsheets sent for cleaning, backups on laptops. Each copy needs protection. The ICO notes that UK GDPR requires personal data to be processed securely using appropriate technical and organisational measures, and the FTC guide cited above advises against using personal information where it is not necessary.

In practice: encrypt files in transit and at rest, limit access to named individuals, keep extracts out of email, use masked or fictitious data for early testing where possible, and delete temporary copies once sign-off is complete. If a vendor handles the data, set these terms out in writing. Our web application security checklist covers the wider controls.

Common mistakes to avoid

• Leaving migration until the end of the project, when there is no time left to clean anything.
• Treating it as an IT task with no business owner for the data.
• Moving everything because deciding what to leave behind feels harder.
• Testing with a small sample, then meeting the difficult records for the first time at go-live.
• Having no rollback plan, or switching the old system off before reconciliation is signed.

What to do next

Before you talk to a vendor, name one person as owner of the migration on the business side. Export a sample of your most important data and read it; an hour with a real customer list tells you more about the effort ahead than any estimate.

Then ask whoever is building or supplying the new system four questions: how many trial runs are planned, what reconciliation reports you will receive, who is responsible for cleaning, and what happens to copies of your data afterward.

Conclusion

A safe data migration is unglamorous, repeatable work: know what you have, move only what you need, write down every mapping rule, rehearse the full load, prove the totals match and keep the old system read-only until you are sure. Most failures trace back to starting too late or leaving the business out of the decisions.

If you are planning to replace a system and want the migration scoped alongside the build, you can request a quote and describe the data you need to bring with you.