Data Deduplication FAQ: Answers to the Most Common Questions
This page answers the questions about data deduplication that come up most often — in sales conversations, support tickets, and from data teams encountering the problem for the first time.
What is data deduplication?
Data deduplication is the process of identifying and resolving duplicate records in a database or data set. A duplicate is any record that refers to the same real-world entity as at least one other record — the same customer, the same company, the same product — regardless of whether the data was entered identically.
Deduplication usually involves two steps: first finding duplicates (comparing records and identifying groups that likely represent the same entity), then processing them (merging, deleting, or otherwise resolving the duplicates so only one canonical record remains).
What is the difference between data deduplication and data cleansing?
Data cleansing (or data scrubbing) is a broad term covering the correction of errors, inconsistencies, and formatting issues in data — standardising phone number formats, correcting misspellings, filling in missing values, and so on.
Deduplication is a subset of data cleansing specifically focused on removing duplicate records. A full data quality programme typically includes both, but they are distinct operations — you can deduplicate data that is otherwise clean, and you can cleanse data that has no duplicates.
How do duplicates get into databases?
The most common causes:
- Manual data entry by multiple users — different people enter the same customer or lead without knowing the other already exists
- System integrations — when syncing data between CRMs, ERPs, or marketing tools, the matching logic may fail to recognise existing records
- Data migrations — imports from legacy systems that were never deduplicated
- API-driven record creation — external systems creating records via API without first checking for an existing match
- Acquired data — merging data from an acquisition, a purchased list, or a partner import
What percentage of CRM records are typically duplicates?
Studies and audits suggest that mature CRM environments (5+ years of active use) typically have duplication rates of 10–25%. Fresh environments with active data quality controls have rates closer to 2–5%. Without any deduplication controls in place, the rate grows by roughly 1–3 percentage points per year.
What is fuzzy matching and why does deduplication need it?
Fuzzy matching compares how similar two strings are rather than whether they are identical. It is necessary because real-world data has noise: names are abbreviated, addresses are formatted differently, companies rename themselves, and typos occur.
“Acme Corporation Ltd” and “ACME Corp Limited” are almost certainly the same company, but an exact-match query won’t find them. A fuzzy matching algorithm computes a similarity score (typically 0–100%) and flags pairs above a configured threshold as candidate duplicates.
DeDuplica exposes purpose-built comparators configurable per field: Levenshtein (general text), Weighted Levenshtein (addresses), Jaro-Winkler (short text and names), QGram (order-independent text), Person Name (personal names), Metaphone (pronunciation-based), and Exact. See the fuzzy matching guide for a detailed explanation of each.
What is the risk of deduplication? Can I accidentally delete good data?
The risk is real if the process is automated without safeguards. The correct approach is to find duplicates first and review the identified groups before processing any of them. Automatic merging without review is appropriate only for very high-confidence matches (near-identical records).
DeDuplica enforces this separation: a Find Duplicates job produces a list of groups for review; a Process Duplicates job acts on those groups. You choose what gets processed and when. The testing workflow lets you validate your matching rules against a sample before running on production data.
How do I deduplicate a SQL Server database?
The most direct approach:
- Connect DeDuplica to your SQL Server instance using the MSSQL connection.
- Create a job with a Source Definition pointing to the table you want to deduplicate.
- Configure matching rules (the fields to compare and the comparison strategy for each).
- Run a job that will find duplicates, create Duplicate in admin panel for review.
- You can confirm the duplicate pair once confirmed it is representing same object. DeDuplica can notify multiple sub systems about the change so that you keep you records in sync.
For a detailed walkthrough, see the How to Start guide and the technical post on removing duplicates from SQL databases.
How do I deduplicate data in Dynamics 365?
Dynamics 365 uses the Dataverse API. DeDuplica connects via OAuth (an Azure app registration) and processes records in-place — no data export required. The main steps are registering an Azure AD application, configuring the connection in DeDuplica, and running a Find Duplicates job against the relevant Dynamics entity (Contact, Account, Lead, etc.).
See the complete Dynamics 365 deduplication guide and the Dynamics connection documentation.
Can deduplication run automatically on a schedule?
Yes. DeDuplica supports scheduled jobs that run daily, weekly, or monthly. Once configured, the job runs without manual intervention. Results are stored for review; if you’ve enabled automatic processing for high-confidence matches, those can be resolved automatically too.
Scheduling is available from the Standard plan upward. See the scheduling documentation.
Does the data have to leave our network?
Not with DeDuplica’s local agent option. The agent runs on a server in your own infrastructure. It connects to your database directly and processes data locally — only job execution metadata (counts, timestamps, status) is sent to DeDuplica’s cloud. Your actual records never leave your network.
This is particularly important for GDPR-sensitive data, healthcare records, and financial services environments. See the full explanation in the on-premises deduplication guide.
How long does deduplication take on a large table?
Performance depends on table size, matching complexity, and the processing environment. Rough benchmarks with DeDuplica:
| Table size | Simple matching (1–2 fields, exact) | Complex matching (4+ fields, fuzzy) |
|---|---|---|
| 50,000 rows | 1–3 minutes | 5–15 minutes |
| 500,000 rows | 5–20 minutes | 30–120 minutes |
| 5,000,000 rows | 30–90 minutes | 2–8 hours |
There are techniques to speed up the process, like splitting data in smaller groups by distinctive parameters - like country.
What is DeDuplica, and who makes it?
DeDuplica is a B2B SaaS deduplication platform developed for enterprise data teams. It supports SQL Server, PostgreSQL, MySQL, MariaDB, Oracle, and Microsoft Dynamics 365. It is available at deduplica.net, with a free tier, transparent subscription pricing, and no credit card required to start. It is run by UK registered OWWARE LTD company.
Have a question not answered here? Contact us or read the full documentation.