dedupe
github.com
- Free plan
- Windows
- macOS
- Linux
- API
- Self-hosted
What dedupe is
dedupe is a free, MIT-licensed Python library for fuzzy matching, deduplication, and entity resolution on structured data. It learns matching rules from human-labeled record pairs, including datasets held in very large databases. The project describes removing duplicate spreadsheet rows, linking customer and order records without unique identifiers, and recognizing people across campaign-contribution records. Python developers can integrate it into applications or import scripts, run it on laptops, local machines, or servers, and build self-hosted pipelines. Documented capabilities include extensible data types and comparators, machine learning from labeled data, and an API. Installation uses `pip install dedupe`; the separate csvdedupe command-line tool handles CSV deduplication. The library runs on Linux, macOS, and Windows, with API and self-hosted use listed. Dedupe.io, the related web service, shut down on January 31, 2023, and its team now focuses on consulting through DataMade.
Who it suits
It suits Python developers building matching or deduplication into applications, import scripts, or self-hosted pipelines. Teams working with structured records and human-labeled examples can use its learning approach.
Good to know, and what to watch
- Free and open source
- Learns from human-labeled examples
- Supports fuzzy matching and entity resolution
- Runs on local machines or servers
- Integrates into applications and import scripts
- Former Dedupe.io web service shut down
- Training requires labeled record-pair examples
- Python library requires developer integration
The verdict
dedupe is a developer-focused option for machine-learning record matching and deduplication. Its current offering is the open-source library, not the discontinued Dedupe.io hosted workflow.
What the maker publishes
- API
- The former Dedupe.io web API supported matching and training against projects through REST calls, including incremental checks, matches and additions after deduplication. source
- Compliance
- The former service’s FAQ states it was certified compliant with FERPA as of July 2019. source
- Compliance
- The former service's FAQ states that Dedupe.io was certified FERPA compliant as of July 2019. source
- Data formats
- The former service’s FAQ says uploads accepted Excel spreadsheets and CSV files, with one record per row and column names in the first row. source
- Data privacy
- The former service's FAQ says it would not rent or sell submitted data and would not allow third-party access without signed consent. source
- Features
- The documentation lists machine learning from labeled data, laptop execution, library integration, extensible data types and comparators, and open-source licensing as features. source
- Features
- Documented features include learned weights and blocking rules, laptop execution, library integration, and support for custom data types, comparators, and blocking rules. source
- Headquarters
- Chicago, Illinois, United States source
- How it learns
- It uses human-labeled training data to learn rules for finding similar records, including in large databases. source
- Input formats
- The former Dedupe.io service accepted spreadsheet files (.xls, .xslx) and comma-separated text (.csv). source
- Installation
- The documented installation command is `pip install dedupe`. source
- Installation
- The documentation gives `pip install dedupe` as the installation command. source
- Integrations
- The project describes dedupe as a library that can be integrated into applications or import scripts; it also documents CSV deduplication through the separate csvdedupe command-line tool. source
- Intended users
- The maintainers describe the library as suitable for integration into applications or import scripts, while the former Dedupe.io service was designed as an online tool for non-developers. source
- License
- The GitHub repository identifies the project as MIT licensed. source
- Open source library
- dedupe is a Python library that uses machine learning for fuzzy matching, deduplication and entity resolution on structured data. source
- Privacy
- The former service’s FAQ says its privacy policy prohibited renting or selling user data and third-party access without signed consent, and limited company access to debugging and service improvement. source
- Product status
- Dedupe.io was shut down on January 31, 2023; its team says it now focuses on consulting through DataMade. source
- Product status
- Dedupe.io, the hosted service, shut down on January 31, 2023. source
- Self-hosting
- The developer page says Python developers can run the library on local machines or servers and build self-hosted entity-resolution pipelines. source
- Support
- The project directs users to GitHub Issues for bugs and lists a mailing list for the open-source library; the former service also offered project setup consulting. source
- Support
- The library documentation directs bug reports to GitHub Issues and lists a mailing list for users. source
- Training
- The library uses human-labeled examples to learn rules for finding similar records, including in very large databases. source
- Training limit
- The former service required at least 10 positive and 10 negative record-pair examples during training. source
Checked against the maker's pages on 10 Oct 2026.