Distributed File Systems

Imagine you've written a book that is so incredibly large that no single bookshelf in the world can hold it. To store it, you have to rip out the pages, bundle them into chapters, and store different chapters in different libraries across the country.

If someone wants to read your book, they can't just go to one library. They need a master index that tells them exactly which library holds Chapter 1, which holds Chapter 2, and so on.

This is the exact problem a Distributed File System (DFS) solves for data.

The Core Concept

When data grows beyond the capacity of a single hard drive (or even a single high-end server), you have no choice but to distribute it across multiple machines. A Distributed File System allows multiple servers (often hundreds or thousands) to act like one giant, unified file system.

To the user or application, it just looks like a regular folder on their computer. But under the hood, the files are chopped into blocks and scattered across a fleet of commodity servers.

Anatomy of a DFS

Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.

Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.

algobase.dev

A Distributed File System (HDFS, Google File System) separates metadata from data storage. The Name Node is the master directory — it holds no file data, only the mapping from filenames to chunk locations. Files are split into fixed-size chunks (128 MB in HDFS) and distributed across Data Nodes (ordinary commodity servers). When a client wants to read a file, it first asks the Name Node: "Where are the chunks for log_data.csv?" The Name Node responds with a list of Data Node addresses. The client then reads each chunk directly from the Data Nodes in parallel — the Name Node is not in the data path at all. This separation means the Name Node handles lightweight metadata requests while Data Nodes handle the heavy byte transfer, enabling throughput that scales linearly with the number of Data Nodes.

1 / 10.5×1×2×3×Playground

DFS read path — Name Node metadata lookup, then direct chunk reads

Most distributed file systems (like Hadoop HDFS or Google File System) use a similar architecture:

1. The Master Node: This is the librarian. It doesn't hold the actual file data. Instead, it holds the metadata—the directory tree and the mapping of which chunks of data live on which servers.
2. The Chunk Servers: These are the grunts. They store the actual pieces of the files.

When a client wants to read a file, it first asks the Master Node, "Where are the chunks for log_data.csv?" The Master Node replies with a list of IP addresses. The client then talks directly to those Chunk Servers to get the data.

Real-World Examples

• Google File System (GFS): Google created GFS to store the massive amounts of web crawling data needed for their search engine. They relied on cheap, easily replaceable hardware, relying on software to handle the inevitable hardware failures.
• Hadoop Distributed File System (HDFS): Inspired by GFS, HDFS became the open-source standard for big data. Companies like Yahoo and Facebook used HDFS extensively to store massive data lakes for analytics.

Dealing with Failures

Press enter or space to select a node. You can then use the arrow keys to move the node around. Press delete to remove it and escape to cancel.

Press enter or space to select an edge. You can then press delete to remove it or escape to cancel.

algobase.dev

Durability in a DFS comes from replication, not RAID. HDFS defaults to 3x replication: every chunk is written to three Data Nodes placed on different racks. When a node fails (and with hundreds of commodity servers, some fail every week), the Name Node detects the missing replica via heartbeat timeouts and instructs surviving nodes to re-replicate the chunk to restore the replication factor. This lets clusters survive simultaneous failure of two nodes without data loss. The cost is storage overhead: 1 TB of actual data requires 3 TB of disk. HDFS 3.0 introduced erasure coding for cold data to cut this cost. DFS is optimized for high sequential throughput on large files — MapReduce jobs reading petabytes of log data. It is the wrong choice for low-latency random reads or small files.

1 / 10.5×1×2×3×Playground

Fault tolerance — 3x chunk replication across different racks

If your data is spread across 1,000 cheap servers, some of those servers will crash. By default, systems like HDFS replicate every chunk of data three times across different servers.

[!NOTE] This replication ensures high availability and fault tolerance, but it means a 1TB file actually consumes 3TB of physical disk space.

Trade-offs and Workloads

Distributed file systems are heavily optimized for throughput (reading massive amounts of data quickly) rather than latency (reading small amounts of data instantly). They are also optimized for append-only workloads.

Summary

• A DFS allows hundreds of servers to act as one giant storage drive.
• Files are broken into chunks and distributed across "Chunk Servers."
• A "Master Node" keeps track of where all the chunks live.
• Perfect for Big Data and analytics; terrible for low-latency database storage.