Amazon Deequ
github.com
- Free plan
- Windows
- macOS
- Linux
- API
- Self-hosted
What Amazon Deequ is
Amazon Deequ is a free, open-source Apache Spark library for defining data unit tests, so teams can check data quality before datasets reach consuming systems or machine-learning algorithms. Checks cover row counts, completeness, uniqueness, allowed and nonnegative values, patterns, and approximate quantiles. Its examples also include data profiling, saved and queryable metrics, anomaly detection over time, automatic constraint suggestions, and incremental metric computation. The declarative Data Quality Definition Language supports rules for counts, completeness, uniqueness, statistics, schema matching, freshness, and custom SQL. Row-level evaluation marks records that pass or fail supported rules, but skips dataset-level rules such as RowCount and Mean. Deequ is intended for very large datasets, including billions of rows, commonly held in distributed filesystems or data warehouses. It requires Apache Spark, and releases target specific Spark versions; Deequ 2.1.0 and later also require Java 11. The README provides Maven and sbt dependency examples and points Python users to PyDeequ. The library uses the Apache 2.0 license.
Who it suits
Deequ suits data teams using Apache Spark who want to define data checks and compute quality metrics on large datasets. Python users can access it through PyDeequ.
Good to know, and what to watch
- Checks completeness, uniqueness, patterns, and allowed values.
- Supports profiling, anomaly detection, and incremental metrics.
- DQDL includes schema, freshness, and custom SQL rules.
- Apache 2.0 licensed and open source.
- Requires Apache Spark.
- The library release must match the Spark version.
- Deequ 2.1.0 and later require Java 11.
- Dataset-level rules are skipped in row-level evaluation.
Plans
Every plan and priceApache 2.0 open-source library
Open source
Getting started
- Open the Deequ GitHub website
- Choose a Deequ release compatible with your Apache Spark version
- Add the dependency using Maven or sbt
- Use Scala, Java, DQDL, or SQL to define checks
- Use PyDeequ when working through its Python API
- Run the checks in your self-hosted environment
Limits to check
The library requires Apache Spark, and its release must match the Spark version in use. Row-level evaluation skips dataset-level rules such as RowCount and Mean; Deequ 2.1.0 and later also require Java 11.
The Club's review
Overview
Amazon Deequ is a free, open-source library for defining data-quality tests in Apache Spark. It best suits engineering teams already running Spark who want to validate large datasets before they reach downstream systems or machine-learning workloads. Its range of checks and metrics is a strength; the Spark and release-version requirements make it a poor fit for teams seeking a standalone service.
Key features
Deequ can test row counts, completeness, uniqueness, allowed and nonnegative values, patterns, and approximate quantiles. That gives teams a useful set of checks for common data-quality problems, while its distributed Spark foundation is designed for very large datasets, including those with billions of rows. The scale is most relevant when data already lives in a distributed filesystem or warehouse; it does not remove the need to operate a compatible Spark environment.
Its examples extend beyond pass-or-fail validation to profiling, storing and querying metrics, detecting anomalies over time, suggesting constraints, and computing metrics incrementally. These capabilities can support recurring quality monitoring, but they call for integration into a team's own Spark workflows rather than a ready-made hosted service.
DQDL lets teams express rules declaratively, including checks for counts, completeness, uniqueness, statistics, schema matching, freshness, and custom SQL. Row-level evaluation can identify records that pass or fail supported rules, but dataset-level checks such as RowCount and Mean are skipped in that mode. Teams should account for that distinction when they need both record-level diagnosis and aggregate validation.
Deequ releases target specific Spark versions. Releases 2.1.0 and later require Java 11, and Deequ 2.x supports Spark 3.1 through 3.5. Maven and sbt dependency examples are provided, but selecting a release that matches the Spark installation is essential. Python users can use PyDeequ, a Python API for Deequ. The library is licensed under Apache 2.0; AWS Glue Data Quality is a managed service built on the open-source framework and uses DQDL.
Pricing
Apache 2.0 open-source library: free. The plan includes the library with no subscription charge, making it a practical option for teams able to run and maintain Spark themselves. It requires Apache Spark, and the chosen Deequ release must match the Spark version; the free price does not eliminate those operational requirements.
Platforms
Deequ supports API use, Linux, macOS, Windows, and self-hosted deployment. Test execution is self-hosted, with Scala, Java, DQDL, and SQL supported. Git is the supported version-control system. Schema migration tests, data-quality checks, pre-commit scanning, and custom detection rules are supported; CI/CD scanning, pull-request scanning, and push protection are not.
Who it's for
Choose Deequ if your team already uses Apache Spark and needs customizable quality checks and metrics for large datasets. Its Scala and Java support, declarative DQDL rules, and PyDeequ interface give data teams several ways to incorporate checks into existing workflows. Look elsewhere if you need a managed testing service, or if Spark compatibility and self-hosting are unacceptable overhead.
Pros and cons
- Broad validation coverage: Checks span completeness, uniqueness, value constraints, patterns, quantiles, and schema-related rules, covering more than basic row counts.
- Monitoring beyond single runs: Profiling, metric persistence, anomaly detection, and incremental computation can support ongoing quality work.
- Works at large scale: Its Spark foundation is designed for datasets that can reach billions of rows.
- Environment and version coupling: Teams need Spark, must match Deequ to its version, and may need Java 11 for newer releases.
- Row-level limits: Aggregate rules such as RowCount and Mean are skipped during row-level evaluation, so the two evaluation modes do not provide interchangeable results.
- No built-in CI/CD or pull-request scanning: Teams looking for those scanning capabilities will need another approach.
Alternatives
Atlas is worth considering for database migration work, particularly for teams that want an open-core tool with a free plan and trial. Bytebase Enterprise is an option for teams that prefer a database-focused product with a free Community plan for up to 20 users and 10 database instances, plus a paid Pro plan billed per user. DataKitchen TestGen may suit a small team looking for a free, single-user setup with one connection and one project, while allowing unlimited tables and data volume.
Great Expectations is an alternative for teams that want a free Developer plan capped at five data assets under test per month and three users, with unlimited rows per asset. Soda may fit teams seeking pipeline testing, metrics observability, and alerting or ticketing integrations in a free plan. HammerDB is a free, open-source database benchmarking tool for Windows and Linux, rather than a data-quality testing library.
pgroll is a free option for PostgreSQL schema changes, requiring PostgreSQL 14.0 or later. Redgate SQL Provision is a paid alternative for provisioning or masking production data, with capacity tiers based on total production data volume. For broader tool categories, browse Database Testing Tools and Secrets Scanning Software.
Verdict
Deequ is a strong choice for Spark teams that want a free, extensible way to test and monitor data quality at scale. Its range of checks and metrics is the main reason to choose it; its dependency on a compatible, self-hosted Spark environment is the main reason to look elsewhere.
The verdict
Deequ offers a broad set of data-quality checks and metrics for Spark workflows, including declarative rules and row-level results. Confirm Spark and Java compatibility before selecting a release.
Questions people ask
Is Amazon Deequ free?
Yes. It is an open-source library priced at no cost under Apache 2.0.
Which platforms does Deequ support?
The listed platforms are API, Linux, macOS, self-hosted, and Windows.
What languages can define Deequ tests?
The listed test languages are Scala, Java, DQDL, and SQL. PyDeequ provides a Python API.
Does Deequ support schema migration tests?
Yes, schema migration tests are supported.
Where are Deequ tests executed?
Test execution is self-hosted and built on Apache Spark.
What the maker publishes
- AWS relationship
- AWS Glue Data Quality documentation says that managed service is built on the open-source Deequ framework and uses DQDL. source
- Checks
- Checks can validate row counts, completeness, uniqueness, allowed values, nonnegative values, patterns, and approximate quantiles. source
- Checks
- Checks can validate row counts, completeness, uniqueness, allowed values, nonnegative values, URL patterns, and approximate quantiles. source
- Compatibility
- Deequ releases target specific Apache Spark versions; versions 2.1.0 and later require Java 11, and the README lists Spark 3.1 through 3.5 compatibility for Deequ 2.x. source
- Compatibility
- Deequ 2.1.0 and later require Java 11, and the project lists Spark 3.1 through 3.5 compatibility for Deequ 2.x. source
- Contribution and feedback
- The README welcomes feedback and contributions. source
- Data scale
- The project says Deequ is designed for very large datasets, including billions of rows, typically stored in a distributed filesystem or data warehouse. source
- DQDL
- Deequ supports the declarative Data Quality Definition Language, including rules for counts, completeness, uniqueness, statistics, schema matching, freshness, and custom SQL. source
- DQDL
- Deequ supports the declarative Data Quality Definition Language, including composite rules using AND and OR. source
- Installation
- The README provides Maven and sbt dependency examples and directs users to select a release matching their Spark version. source
- Integrations
- Deequ is built on Apache Spark, is distributed through Maven artifacts, and has a Python interface called PyDeequ. source
- Intended use
- The README describes using data checks to catch errors before datasets reach consuming systems or machine-learning algorithms. source
- License
- The library is licensed under Apache 2.0. source
- License
- The library is licensed under the Apache 2.0 License. source
- Metrics and profiling
- The project examples include metrics persistence and querying, data profiling, anomaly detection over time, automatic constraint suggestions, and incremental metric computation. source
- Profiling and monitoring
- Examples cover data profiling, persisting and querying computed metrics, anomaly detection over time, automatic constraint suggestions, and incremental metrics computation. source
- Purpose
- Deequ is an Apache Spark library for defining data unit tests that measure quality in large datasets. source
- Purpose
- Deequ is an Apache Spark library for defining unit tests that measure data quality in large datasets. source
- Python interface
- The project points Python users to PyDeequ, described on its repository as a Python API for Deequ. source
- Row-level results
- Row-level evaluation identifies rows that pass or fail supported rules, while dataset-level rules such as RowCount and Mean are marked as skipped. source
- Row-level results
- DQDL row-level evaluation identifies rows passing or failing supported rules, while dataset-level rules such as RowCount and Mean are skipped. source
- Scale
- The project says it is designed for very large datasets, including billions of rows, typically stored in distributed filesystems or data warehouses. source
- Security reporting
- The repository security policy asks users not to report security concerns in public GitHub issues and directs them to AWS's Vulnerability Disclosure Program or email. source
- Security reporting
- The contribution guide asks users to report potential security issues through AWS/Amazon Security's vulnerability reporting page rather than a public GitHub issue. source
Checked against the maker's pages on 5 Oct 2026.