Introduction
The retailer's batch succeeded yesterday, but today's command reports an error. Repeating it without reading the message may hide the real cause or encourage a destructive shortcut. A missing input, a wrong service address, and a failed processing task require different responses.
You will follow a small diagnostic sequence: identify the filesystem and connection, check input existence and readability, check the output path, then inspect an accepted application's status and logs. A client-side error may occur before an application is submitted. A submitted application has a YARN identity whose diagnostics provide further evidence.
This lab deliberately provides several failures, including a precompiled training application that rejects a marked input record. These are controlled learning fixtures, not random infrastructure failures. You will preserve them as evidence and finish a clean WordCount run using a different input and output path. No Java coding, service reinstallation, formatting, or broad deletion is required. All fixtures are independent; use the Terminal.
Diagnose Connection and Input Errors
In this step, you will distinguish a wrong service address from a missing file by reading real client errors.
Start in the local workspace:
cd /home/labex/project
First recall the configured filesystem:
hdfs getconf -confKey fs.defaultFS
It should be hdfs://localhost:9000. Now deliberately ask the client to connect to port 1, where no NameNode is listening. The -D properties below limit connection retries and set a 2,000-millisecond connection timeout for this invocation, keeping the intentional error bounded. They do not edit shared configuration.
The redirection > file 2>&1 saves ordinary output in a local file and sends error output to that same destination. The next command is expected to fail.
hdfs dfs -Dipc.client.connect.max.retries=0 -Dipc.client.connect.timeout=2000 -ls hdfs://localhost:1/user/labex > /home/labex/project/output/connection-error.log 2>&1
Read the captured error:
cat /home/labex/project/output/connection-error.log
Look for connection failure or refusal and the wrong address. This happens before a filesystem listing can succeed. Do not format HDFS or reinstall services; the command used the wrong endpoint. Return to the configured endpoint by using the default path:
hdfs dfs -ls /user/labex/job-diagnostics/data
You should see clean.txt and rejected.txt. The connection works. A different problem is asking that healthy filesystem for a nonexistent file. The next command is expected to fail.
hdfs dfs -ls /user/labex/job-diagnostics/data/missing.txt > /home/labex/project/output/missing-input.log 2>&1
Inspect the message:
cat /home/labex/project/output/missing-input.log
It should report that the named path does not exist. Check the intended input with the familiar read operation:
hdfs dfs -cat /user/labex/job-diagnostics/data/clean.txt
The two lines are notebook pen and notebook folder. A corrected input path solves a missing-file problem; changing server configuration would not. Keep the error logs as evidence.
Recognize an Existing Output Path
In this step, you will observe why a job refuses an output directory that already exists and choose preservation over a broad deletion.
Setup has prepared an empty existing result directory to represent an already reserved output location. Inspect that directory entry:
hdfs dfs -ls -d /user/labex/job-diagnostics/results/existing
The example MapReduce output format expects to create a new output directory. An existing directory can trigger a client-side rejection before a processing application is accepted. The next command is expected to fail.
hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar wordcount /user/labex/job-diagnostics/data/clean.txt /user/labex/job-diagnostics/results/existing > /home/labex/project/output/existing-output.log 2>&1
Read the captured message:
cat /home/labex/project/output/existing-output.log
Look for an output-directory-already-exists message or FileAlreadyExistsException. This is a path conflict, not evidence that the product-counting algorithm is wrong.
List YARN's accepted applications:
yarn application -list -appStates ALL
These client mistakes should not produce a completed processing application. A client may contact cluster services or obtain an identifier during submission preparation; an accepted application and its status are separate evidence.
You will preserve the reserved directory and use a new output path in the successful run. Removing all of results or deleting raw input would be a much broader change than the problem requires.
Inspect a Real Failed Application
In this step, you will submit a controlled training application and use its real YARN identity and logs to explain the failure.
The prepared controlled-failure.jar accepts an input path and a new output path. Its mapper deliberately rejects the marker BAD_RECORD. You do not need to inspect or edit Java code; the purpose is to distinguish an accepted application's task failure from the earlier client errors.
Read the marked input first:
hdfs dfs -cat /user/labex/job-diagnostics/data/rejected.txt
It contains a normal product line followed by BAD_RECORD. The training job uses one map attempt so the same intentional failure is not repeatedly retried. The next command is expected to fail.
hadoop jar /home/labex/project/tools/controlled-failure.jar /user/labex/job-diagnostics/data/rejected.txt /user/labex/job-diagnostics/results/failed-example > /home/labex/project/output/failed-job.log 2>&1
Wait for the prompt, then read the end of its log:
tail -n 25 /home/labex/project/output/failed-job.log
Unlike the previous cases, this run should have a submitted YARN application and task-failure evidence. Capture its actual ID using the same application-list pattern as the previous lab:
yarn application -list -appStates ALL | grep -o 'application_[0-9]*_[0-9]*' > /home/labex/project/output/failed-application-id.txt
Query and save its report:
yarn application -status "$(cat /home/labex/project/output/failed-application-id.txt)" > /home/labex/project/output/failed-application-status.txt
Read the report:
cat /home/labex/project/output/failed-application-status.txt
Find the application identity, final status, and diagnostics. The final status should be FAILED; the report is evidence from YARN, not a handwritten failure label. Read the real aggregated logs too:
yarn logs -applicationId "$(cat /home/labex/project/output/failed-application-id.txt)" > /home/labex/project/output/failed-application.log
Use grep to find the known training diagnostic inside them:
grep 'Training input rejected' /home/labex/project/output/failed-application.log
The message points to the marked record. You have moved from a generic nonzero command result to a specific explanation supported by the application and task logs. Preserve this evidence instead of treating every failure as a reason to restart the cluster.
Repair the Batch Selection and Complete a Run
In this step, you will use the clean input and a new output directory, then retain a short evidence-based diagnosis.
The intended reporting input is clean.txt; the rejected file was a training fixture. Keep both originals. Submit the familiar WordCount example against only the clean file, using a new result directory:
hadoop jar /opt/hadoop/share/hadoop/mapreduce/hadoop-mapreduce-examples-3.5.0.jar wordcount /user/labex/job-diagnostics/data/clean.txt /user/labex/job-diagnostics/results/successful-run > /home/labex/project/output/successful-job.log 2>&1
This command should return successfully. The input has two notebook appearances, one pen, and one folder. Read the result:
hdfs dfs -cat '/user/labex/job-diagnostics/results/successful-run/part-*' | sort
Expected counts:
folder 1
notebook 2
pen 1
The successful driver's log includes its application ID. sort -u sorts and removes repeated identical ID lines, leaving the one ID from this invocation:
grep -o 'application_[0-9]*_[0-9]*' /home/labex/project/output/successful-job.log | sort -u > /home/labex/project/output/successful-application-id.txt
Confirm the actual successful application status:
yarn application -status "$(cat /home/labex/project/output/successful-application-id.txt)"
The final status should be SUCCEEDED. Assemble the earlier evidence into one local diagnosis record:
cat /home/labex/project/output/connection-error.log /home/labex/project/output/missing-input.log /home/labex/project/output/existing-output.log /home/labex/project/output/failed-application-status.txt > /home/labex/project/output/diagnosis.md
Append the repair choice. printf prints the text and \n creates line breaks; >> appends rather than replaces the captured evidence:
printf '\nRepair: select /user/labex/job-diagnostics/data/clean.txt and use the new /user/labex/job-diagnostics/results/successful-run output. Preserve original inputs and the existing result directory.\n' >> /home/labex/project/output/diagnosis.md
This record connects error, observation, and action. For an unavailable service or platform fault outside your access, record the endpoint and message and report the issue rather than reformatting storage. For the problems taught here, a specific path or input choice was enough.
Summary
You distinguished connection errors, missing inputs, and existing output paths from a task failure in an accepted YARN application. You used actual application state and task logs to explain a controlled rejection, then completed a correct batch with a clean input and new output path.
The final course challenge combines these habits with input organization and permissions. It asks you to complete a new batch independently using methods you have already practiced.
