When learning about a new technology, it is helpful to view existing examples to get an understanding of how things work. One example included in the Hadoop 2.0 package from Apache is the "Distributed Shell" example. This example is the equivalent of the "Hello World" example that "Word Count" served as for MapReduce exploration. One issue with the code in its current state is that although the example works well, it does not lend itself to an explanation of the general flow of a YARN-based application. This blog post is meant to break down that application flow and provide a clear example of the steps that are necessary to build a successful YARN application.
In another blog post, "Hadoop Fundamentals - YARN Concepts", the details of the concepts added to the Hadoop paradigm due to the addition of YARN are described in great detail. To add context for this blog post, a short glossary of the definitions is provided below:
- Application - In YARN terminology, the cluster supports a collection of "Applications", instead of a set of "Jobs". This is because YARN supports many different paradigms, not just MapReduce, so there was a need for a more general term.
- Container - A Container represents the specific set of resources that a particular application has the rights to use. The resources that make up a container are any subset of memory, CPU, network bandwidth, etc. on a specific node of the cluster.
- Resource Manager - In Hadoop 2.0, the Resource Manager is analogous to the JobTracker, but with the roles separated into two sub-processes - the Scheduler and the Applications Manager.
- Application Master - Each application running in the cluster must have its own Application Master, including traditional MapReduce jobs. The Application Master must negotiate any necessary resource containers from the Resource Manager's Scheduler. It is also responsible for communicating with the Node Managers of the cluster in order to track the status for all containers associated with the application.
- Client - The Client is responsible for launching the Application Master that contains the logic for the Application through the Applications Manager.
When launching a YARN application, the user is interacting with the "Application Submission Client". During this interaction, the user must pass any appropriate arguments to the Client so that it is able to specify the details of the Application Master when appropriate. Once given those specifications, the Client must submit an "Application" to the Applications Manager inside the Resource Manager. This notifies the Applications Manager of the existence of the Application so that it can be monitored.
Once the Application is accepted by the Resource Manager, it provides sufficient information to launch the first container - the Application Master (AM). This process involves three major steps. First, the Client must transfer any local files or jars that need to be available to the AM or any containers that the AM will create. Second, it must configure any environment settings that the AM container requires. Finally, it must submit the actual command that needs to be executed, along with any necessary command line arguments. The Resource Manager will launch the specified Application Master on an allocated container, and once started, the Application Master will register itself with the Resource Manager.
Once started, the Application Master can request and receive additional containers as necessary to complete the task for which it was created. When there is a need for a new container, the Application Master communicates with the Node Manager to launch the container for its task. Similar to the Application Master launch, sufficient information needs to be provided - including any necessary files or environment variables.
At any point, the client can monitor the application's status by querying the Resource Manager or by directly querying the Application Master. Once the application is completed, the Application Master has to signal the Resource Manager of its completion. This will prompt the end any monitoring that is taking place, and the status should appropriately reflect the success or failure of the job.