Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

dedupe

github.com

  • Free plan
  • Windows
  • macOS
  • Linux
  • API
  • Self-hosted
7.6 Club score Visit github.com

What dedupe is

dedupe is a free, MIT-licensed Python library for fuzzy matching, deduplication, and entity resolution on structured data. It learns matching rules from human-labeled record pairs, including datasets held in very large databases. The project describes removing duplicate spreadsheet rows, linking customer and order records without unique identifiers, and recognizing people across campaign-contribution records. Python developers can integrate it into applications or import scripts, run it on laptops, local machines, or servers, and build self-hosted pipelines. Documented capabilities include extensible data types and comparators, machine learning from labeled data, and an API. Installation uses `pip install dedupe`; the separate csvdedupe command-line tool handles CSV deduplication. The library runs on Linux, macOS, and Windows, with API and self-hosted use listed. Dedupe.io, the related web service, shut down on January 31, 2023, and its team now focuses on consulting through DataMade.

Who it suits

It suits Python developers building matching or deduplication into applications, import scripts, or self-hosted pipelines. Teams working with structured records and human-labeled examples can use its learning approach.

Good to know, and what to watch

  • Free and open source
  • Learns from human-labeled examples
  • Supports fuzzy matching and entity resolution
  • Runs on local machines or servers
  • Integrates into applications and import scripts
  • Former Dedupe.io web service shut down
  • Training requires labeled record-pair examples
  • Python library requires developer integration

The verdict

dedupe is a developer-focused option for machine-learning record matching and deduplication. Its current offering is the open-source library, not the discontinued Dedupe.io hosted workflow.

What the maker publishes

API
The former Dedupe.io web API supported matching and training against projects through REST calls, including incremental checks, matches and additions after deduplication. source
Compliance
The former service’s FAQ states it was certified compliant with FERPA as of July 2019. source
Compliance
The former service's FAQ states that Dedupe.io was certified FERPA compliant as of July 2019. source
Data formats
The former service’s FAQ says uploads accepted Excel spreadsheets and CSV files, with one record per row and column names in the first row. source
Data privacy
The former service's FAQ says it would not rent or sell submitted data and would not allow third-party access without signed consent. source
Features
The documentation lists machine learning from labeled data, laptop execution, library integration, extensible data types and comparators, and open-source licensing as features. source
Features
Documented features include learned weights and blocking rules, laptop execution, library integration, and support for custom data types, comparators, and blocking rules. source
Headquarters
Chicago, Illinois, United States source
How it learns
It uses human-labeled training data to learn rules for finding similar records, including in large databases. source
Input formats
The former Dedupe.io service accepted spreadsheet files (.xls, .xslx) and comma-separated text (.csv). source
Installation
The documented installation command is `pip install dedupe`. source
Installation
The documentation gives `pip install dedupe` as the installation command. source
Integrations
The project describes dedupe as a library that can be integrated into applications or import scripts; it also documents CSV deduplication through the separate csvdedupe command-line tool. source
Intended users
The maintainers describe the library as suitable for integration into applications or import scripts, while the former Dedupe.io service was designed as an online tool for non-developers. source
License
The GitHub repository identifies the project as MIT licensed. source
Open source library
dedupe is a Python library that uses machine learning for fuzzy matching, deduplication and entity resolution on structured data. source
Privacy
The former service’s FAQ says its privacy policy prohibited renting or selling user data and third-party access without signed consent, and limited company access to debugging and service improvement. source
Product status
Dedupe.io was shut down on January 31, 2023; its team says it now focuses on consulting through DataMade. source
Product status
Dedupe.io, the hosted service, shut down on January 31, 2023. source
Self-hosting
The developer page says Python developers can run the library on local machines or servers and build self-hosted entity-resolution pipelines. source
Support
The project directs users to GitHub Issues for bugs and lists a mailing list for the open-source library; the former service also offered project setup consulting. source
Support
The library documentation directs bug reports to GitHub Issues and lists a mailing list for users. source
Training
The library uses human-labeled examples to learn rules for finding similar records, including in very large databases. source
Training limit
The former service required at least 10 positive and 10 negative record-pair examples during training. source

Checked against the maker's pages on 10 Oct 2026.