Hadoop Distributed File System
hadoop.apache.org
- Free plan
- Windows
- macOS
- Linux
- API
- Self-hosted
What Hadoop Distributed File System is
Hadoop Distributed File System (HDFS) is the primary distributed storage system used by Hadoop applications. Its clusters use a NameNode to manage filesystem metadata and DataNodes to store data. HDFS is designed for fault tolerance on commodity hardware and for high-throughput streaming access and batch processing, not low-latency workloads. Its design is tuned for files ranging from gigabytes to terabytes, clusters with hundreds of nodes, and tens of millions of files in one instance. Written in Java, Hadoop is supported on major platforms and HDFS is deployed on self-hosted clusters. Administrators can use shell-like commands, NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer. Permissions share much with POSIX, with optional POSIX ACLs; Hadoop supports operating-system identity and Kerberos authentication. When configured, encryption protects data at rest and in transit. WebHDFS provides REST access, and listed storage integrations include Amazon S3, Azure Blob Storage, Azure Data Lake Storage, and Tencent COS. Apache Hadoop is free under Apache License 2.0.
Who it suits
HDFS suits organizations building self-hosted Hadoop storage for large files and high-throughput batch or streaming work. It is less suited to workloads where low latency is the priority.
Good to know, and what to watch
- Designed for fault tolerance on commodity hardware
- Supports high-throughput streaming and batch processing
- Provides NameNode and DataNode administration tools
- Supports Kerberos authentication and configured encryption
- Free under Apache License 2.0
- Designed for throughput rather than low-latency work
- Requires self-hosted cluster deployment
- Project support is through volunteer-maintained mailing lists
Plans
Every plan and priceApache Hadoop (open source)
Open source
Getting started
- Open the Apache Hadoop HDFS documentation website
- Download a source or binary Hadoop tarball
- Verify the release with its GPG signature or SHA-512 checksum
- Deploy an Apache Hadoop self-hosted cluster
- Configure NameNode and DataNode services
- Choose simple operating-system identity or Kerberos authentication
Limits to check
The free Apache Hadoop plan is a self-hosted cluster deployment. HDFS is optimized for streaming and batch throughput, not low-latency workloads.
The verdict
HDFS is oriented toward large-scale distributed storage and throughput-focused processing, with administration, access, and security features for self-hosted clusters. Its workload profile makes latency-sensitive use a poor fit.
Questions people ask
Is HDFS free to use?
Yes. The Apache Hadoop open-source plan costs nothing, billed no charge, under the Apache License 2.0.
Which platforms does HDFS support?
HDFS is available for self-hosted use on Linux, macOS and Windows, with Hadoop supported on major platforms.
How does an HDFS cluster store data?
A NameNode manages filesystem metadata, while DataNodes store the data.
Does HDFS provide APIs for file access?
Yes. Access options include Java, C and REST APIs, HTTP browser access, an NFS gateway and the FS shell. WebHDFS supports the complete FileSystem and FileContext interfaces.
Can HDFS encrypt stored and transferred data?
Configured HDFS encryption provides end-to-end protection for data at rest and in transit through encryption zones and the Hadoop Key Management Server.
What the maker publishes
- Administration
- HDFS provides shell-like commands, built-in NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer. source
- Architecture
- An HDFS cluster primarily consists of a NameNode managing filesystem metadata and DataNodes storing data. source
- Authentication
- Hadoop supports simple operating-system identity mode and Kerberos authentication mode. source
- Downloads and integrity
- Apache distributes source and binary tarballs and recommends verifying releases with GPG signatures or SHA-512 checksums. source
- Encryption
- Configured HDFS encryption provides transparent end-to-end encryption for data at rest and in transit using encryption zones and the Hadoop Key Management Server. source
- Fault tolerance
- HDFS is highly fault-tolerant and designed to run on low-cost commodity hardware. source
- Language and platforms
- Hadoop is written in Java and supported on all major platforms. source
- License
- Apache Hadoop is licensed under the Apache License 2.0. source
- Permissions
- HDFS implements a permissions model sharing much of POSIX and optionally supports POSIX access control lists. source
- Purpose
- HDFS is the primary distributed storage used by Hadoop applications. source
- REST integration
- The WebHDFS HTTP REST API supports the complete FileSystem and FileContext interfaces for HDFS. source
- Scale
- HDFS is tuned for gigabyte-to-terabyte files, hundreds of nodes per cluster, and tens of millions of files in one instance. source
- Storage integrations
- The documentation lists Amazon S3, Azure Blob Storage, Azure Data Lake Storage, Tencent COS, and other Hadoop-compatible file systems. source
- Support
- Apache Hadoop support is provided through project mailing lists maintained by non-paid volunteers. source
- Workload fit
- HDFS is designed for streaming access and batch processing with high throughput rather than low latency. source
Checked against the maker's pages on 30 Sept 2026.