Skip to the content
Know your tech before you buy it. Makers: get listed from $49/mo

Hadoop Distributed File System

hadoop.apache.org

  • Free plan
  • Windows
  • macOS
  • Linux
  • API
  • Self-hosted
7.6 Club score Visit hadoop.apache.org

What Hadoop Distributed File System is

Hadoop Distributed File System (HDFS) is the primary distributed storage system used by Hadoop applications. Its clusters use a NameNode to manage filesystem metadata and DataNodes to store data. HDFS is designed for fault tolerance on commodity hardware and for high-throughput streaming access and batch processing, not low-latency workloads. Its design is tuned for files ranging from gigabytes to terabytes, clusters with hundreds of nodes, and tens of millions of files in one instance. Written in Java, Hadoop is supported on major platforms and HDFS is deployed on self-hosted clusters. Administrators can use shell-like commands, NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer. Permissions share much with POSIX, with optional POSIX ACLs; Hadoop supports operating-system identity and Kerberos authentication. When configured, encryption protects data at rest and in transit. WebHDFS provides REST access, and listed storage integrations include Amazon S3, Azure Blob Storage, Azure Data Lake Storage, and Tencent COS. Apache Hadoop is free under Apache License 2.0.

Who it suits

HDFS suits organizations building self-hosted Hadoop storage for large files and high-throughput batch or streaming work. It is less suited to workloads where low latency is the priority.

Good to know, and what to watch

  • Designed for fault tolerance on commodity hardware
  • Supports high-throughput streaming and batch processing
  • Provides NameNode and DataNode administration tools
  • Supports Kerberos authentication and configured encryption
  • Free under Apache License 2.0
  • Designed for throughput rather than low-latency work
  • Requires self-hosted cluster deployment
  • Project support is through volunteer-maintained mailing lists

Apache Hadoop (open source)

Free

Open source

Getting started

  1. Open the Apache Hadoop HDFS documentation website
  2. Download a source or binary Hadoop tarball
  3. Verify the release with its GPG signature or SHA-512 checksum
  4. Deploy an Apache Hadoop self-hosted cluster
  5. Configure NameNode and DataNode services
  6. Choose simple operating-system identity or Kerberos authentication

Limits to check

The free Apache Hadoop plan is a self-hosted cluster deployment. HDFS is optimized for streaming and batch throughput, not low-latency workloads.

The verdict

HDFS is oriented toward large-scale distributed storage and throughput-focused processing, with administration, access, and security features for self-hosted clusters. Its workload profile makes latency-sensitive use a poor fit.

Questions people ask

Is HDFS free to use?

Yes. The Apache Hadoop open-source plan costs nothing, billed no charge, under the Apache License 2.0.

Which platforms does HDFS support?

HDFS is available for self-hosted use on Linux, macOS and Windows, with Hadoop supported on major platforms.

How does an HDFS cluster store data?

A NameNode manages filesystem metadata, while DataNodes store the data.

Does HDFS provide APIs for file access?

Yes. Access options include Java, C and REST APIs, HTTP browser access, an NFS gateway and the FS shell. WebHDFS supports the complete FileSystem and FileContext interfaces.

Can HDFS encrypt stored and transferred data?

Configured HDFS encryption provides end-to-end protection for data at rest and in transit through encryption zones and the Hadoop Key Management Server.

What the maker publishes

Administration
HDFS provides shell-like commands, built-in NameNode and DataNode web servers, fsck diagnostics, safemode, and a cluster balancer. source
Architecture
An HDFS cluster primarily consists of a NameNode managing filesystem metadata and DataNodes storing data. source
Authentication
Hadoop supports simple operating-system identity mode and Kerberos authentication mode. source
Downloads and integrity
Apache distributes source and binary tarballs and recommends verifying releases with GPG signatures or SHA-512 checksums. source
Encryption
Configured HDFS encryption provides transparent end-to-end encryption for data at rest and in transit using encryption zones and the Hadoop Key Management Server. source
Fault tolerance
HDFS is highly fault-tolerant and designed to run on low-cost commodity hardware. source
Language and platforms
Hadoop is written in Java and supported on all major platforms. source
License
Apache Hadoop is licensed under the Apache License 2.0. source
Permissions
HDFS implements a permissions model sharing much of POSIX and optionally supports POSIX access control lists. source
Purpose
HDFS is the primary distributed storage used by Hadoop applications. source
REST integration
The WebHDFS HTTP REST API supports the complete FileSystem and FileContext interfaces for HDFS. source
Scale
HDFS is tuned for gigabyte-to-terabyte files, hundreds of nodes per cluster, and tens of millions of files in one instance. source
Storage integrations
The documentation lists Amazon S3, Azure Blob Storage, Azure Data Lake Storage, Tencent COS, and other Hadoop-compatible file systems. source
Support
Apache Hadoop support is provided through project mailing lists maintained by non-paid volunteers. source
Workload fit
HDFS is designed for streaming access and batch processing with high throughput rather than low latency. source

Checked against the maker's pages on 30 Sept 2026.