Introduction
The retailer's files have names and directories, but machines must also store their contents somewhere. How can one large file be spread across storage nodes? What does HDFS copy when it keeps a replica, and does a nearly empty block reserve a whole block's worth of disk space?
You already know how to organize and preserve files. In this lab, you will look beneath their paths. An HDFS block is a portion of a file that HDFS tracks and stores. The NameNode keeps metadata about files and block locations; DataNodes hold the block data. A replica is another stored copy of a block, rather than a new retail record or another user-visible file.
You will upload two prepared files with a deliberately small, valid block size, predict their block counts, and inspect the real locations. This single-DataNode environment uses replication factor 1. It provides real observations without pretending to demonstrate multi-machine recovery. All data and services are prepared independently. Use the Terminal; no programming is needed.
Connect File Size to Blocks
In this step, you will upload a file with a known block size and inspect its metadata.
A filesystem path names a logical file. HDFS divides its contents into blocks so different parts can be stored on different DataNodes in a larger cluster. A block's configured size is a maximum data size for that block, not a promise that every block is full.
Start in the project directory:
cd /home/labex/project
The local wc -c command counts bytes. Use it on the first prepared file:
wc -c /home/labex/project/data/products-a.txt
The result begins with 1572864: exactly 1.5 MiB. A MiB is 1,048,576 bytes. This file repeats a simple product label so the storage experiment has no complicated business rules.
Create a personal HDFS directory:
hdfs dfs -mkdir -p /user/labex/block-study
For this demonstration, -Ddfs.blocksize=1048576 overrides the client's block-size setting for the following upload. -D supplies a configuration property for this command; it does not edit the shared configuration files. One MiB satisfies this image's minimum block-size limit and keeps the demonstration small:
hdfs dfs -Ddfs.blocksize=1048576 -put /home/labex/project/data/products-a.txt /user/labex/block-study/products-a.txt
Predict the layout: 1.5 MiB with a 1 MiB block size requires one full block and one half-full block, for two blocks.
The HDFS -stat operation prints selected metadata. In its quoted format, %b means file length in bytes, %o means block size, and %r means replication factor:
hdfs dfs -stat 'bytes=%b blocksize=%o replication=%r' /user/labex/block-study/products-a.txt
Expected values:
bytes=1572864 blocksize=1048576 replication=1
The file length is still 1.5 MiB. Replication factor 1 means one stored copy per block, not one block for the whole file. Save the actual metadata:
hdfs dfs -stat 'bytes=%b blocksize=%o replication=%r' /user/labex/block-study/products-a.txt > /home/labex/project/output/file-a-metadata.txt
If the file already exists from a completed upload, inspect it before repeating the upload; do not delete a file merely because -put reports an existing destination.
Inspect Locations and Predict a Change
In this step, you will find the real blocks for the first file and test your prediction with a larger file.
The hdfs fsck command can inspect a file's storage state. In the following read-only inspection, -files includes file details, -blocks includes block information, and -locations includes the DataNodes holding each block. None of these options repairs or deletes files:
hdfs fsck /user/labex/block-study/products-a.txt -files -blocks -locations
Find the file-length line, the block entries, and the summary's Total blocks (validated) field. There should be two blocks. One contains 1,048,576 bytes and the other 524,288 bytes. Block identifiers, addresses, and timestamps vary. The final filesystem health summary should be HEALTHY.
Save the inspection:
hdfs fsck /user/labex/block-study/products-a.txt -files -blocks -locations > /home/labex/project/output/file-a-blocks.txt
A healthy result means this check found the required blocks and replicas. With one replica per block on one machine, it does not mean the file can survive losing that machine.
Now inspect the second local file's size:
wc -c /home/labex/project/data/products-b.txt
Its length is 2621440 bytes, or 2.5 MiB. Before uploading, predict the number and sizes of its blocks at the same 1 MiB block size. Only the file length changes:
hdfs dfs -Ddfs.blocksize=1048576 -put /home/labex/project/data/products-b.txt /user/labex/block-study/products-b.txt
Inspect the result:
hdfs fsck /user/labex/block-study/products-b.txt -files -blocks -locations
You should find three blocks: two full blocks and one 524,288-byte block. All locations refer to the same DataNode in this lab. Save the observation:
hdfs fsck /user/labex/block-study/products-b.txt -files -blocks -locations > /home/labex/project/output/file-b-blocks.txt
In a larger cluster, block locations can differ across machines. The NameNode provides location metadata to clients; it is not a central pipe through which every byte of file content must pass.
Interpret Space, Replicas, and the Data Path
In this step, you will relate the block observations to file size and the actual storage-node report.
A half-full final block does not reserve a full MiB of data space just because the block-size setting is 1 MiB. Use -du without -h to keep exact byte counts visible:
hdfs dfs -du /user/labex/block-study
The first two columns for each file should be:
1572864 1572864 .../products-a.txt
2621440 2621440 .../products-b.txt
The first column is logical file length; the second accounts for replication. Here they agree because replication is 1. These values describe HDFS file data, not every filesystem metadata byte or the VM's total disk use. Save the rows:
hdfs dfs -du /user/labex/block-study > /home/labex/project/output/block-report.txt
Check the actual storage-node report with the read-only administrative command introduced in the environment lab:
sudo -u hdfs /usr/local/bin/hdfs dfsadmin -report
Look for Live datanodes (1). Setting replication to 3 would not create three machines; this lab keeps replication at 1 instead of leaving blocks waiting for unavailable replicas.
Connect the observations to the data path:
Write: client asks NameNode to create a file, then sends block data to DataNode(s).
Read: client asks NameNode for locations, then reads block data from DataNode(s).
With several replicas in a larger cluster, a client can use an available copy, and a processing framework can try to run work near the data. This idea is called data locality. A named copy created with -cp is a second logical file; replicas are storage copies of blocks inside a logical file. They solve different problems.
Your prediction exercise changed the file size, not the replication factor. File A still has two blocks, file B has three, and both have one replica per block. Next, you will learn why a file can be present and healthy yet unreadable to a particular user: access permissions.
Summary
You predicted and observed how 1.5 MiB and 2.5 MiB files split into two and three 1 MiB blocks. You inspected actual block locations and distinguished logical file length, replica storage, and extra named files.
You also connected those results to NameNode metadata and DataNode transfers. The single-node environment illustrates these responsibilities while leaving multi-machine capacity and fault tolerance for later exploration.
