In this guide 6 sections
Duplicates usually have four causes. There is no shared match key. Retries resend a message already processed. A two way sync echoes its own changes back. Staff type the same customer into both systems. Each cause has a specific fix.
No match key between the two systems
A sync needs to know that customer 4471 in the CRM is the same company as account ACME-09 in the accounts package. Without a shared key, it sees a new name and creates a new record.
Names are a weak key. Spelling, abbreviations and trading names all differ between the CRM and the accounts package. Tax numbers, registered email addresses or an internal customer code hold up better.
Store the other system’s ID on each record. Once both sides carry each other’s ID, the sync can update instead of insert.
- Add an external ID field to both systems
- Back fill it for existing customers before go live
- Reject records that arrive without a key
- Review near matches by hand, not by script
Retries without an idempotency check
Networks time out. The sender cannot tell whether the order arrived, so it sends again. If the receiver has no memory of what it already accepted, you now have two sales orders.
An idempotency key fixes this. Each message carries a unique reference, such as the order number plus a version, and the receiver stores references it has processed and ignores a repeat.
Check the integration log for pairs of identical records created seconds apart. That is a retry.
Two way syncs that echo each other
System A updates a contact and pushes the change to B. B saves it, sees a changed record and pushes it back to A, which sees another change, and the loop starts. Some loops settle. Others create a copy on every pass.
Break the echo by tagging each change with its origin. A record last written by the integration user is not sent onward. Compare a hash of the fields as well, so a save that changed nothing is dropped.
Decide which system owns each field. Ownership turns one two way sync into several clear one way flows.
- Flag changes made by the integration account
- Skip sends when no field value differs
- Give each field one owner
- Add a short delay to merge rapid edits
Customers entered by hand on both sides
Even a good integration loses when staff do not trust it. A salesperson creates the customer in the CRM. The accounts clerk, seeing nothing yet, creates it again. When the sync arrives, there are two.
Decide where a customer is born. Say it plainly: new customers are created in the CRM only, and accounts receives them, and remove the create button from the other system where you can.
Teach one matching rule for exceptions. Search by tax number first. Then try the email. Only then create anything.
Clean up the duplicates you already have
Export both systems and sort by the match key. Group records that share a tax number or an email. Choose a surviving record for each group, keeping the one with the most invoice history.
Merge in the system of record first, then let the sync carry the merged result across. Merging on both sides separately only creates new differences.
Run the duplicate report weekly for a month. Any new pair points to a cause still live.
Duplicate diagnosis sheet
Four rows, one per cause, fill this sheet. Columns give the symptom to look for in the data, the query or export that finds it, the likely fix and the owner. A lower block lists the merge steps for a duplicate group: survivor, records retired, history moved and date merged.
Open a blank worksheet to printQuestions owners ask
Why do duplicates appear only after go live?
Before the link, each system was used alone. Once data crosses between them, any weak match key, retry or echo shows up as extra records.
What is an idempotency key?
A unique reference sent with each message so the receiver can recognise a repeat. If it has seen the reference, it skips the message instead of creating another record.
Should we delete the duplicate records?
Merge them rather than delete. Merging keeps invoices, notes and order history attached to the surviving record. Deleting can orphan linked documents.
Can the integration prevent duplicates entirely?
It can prevent most. Hand entry and unusual spellings still slip through, so a weekly duplicate report remains useful.