Hadoop
Distributed
File
System (HDFS) Architectural
Documentation
This section describes the communication among the HDFS elements. The application process has two elements for which we will discuss the communication—the application code and the client library. We will also discuss the communication between the various processes. Each of the three elements of HDFS is described in a different section, see NameNode Decomposition, Client decomposition, and DataNode Block Management.
HDFS provides a Java API for
applications
to
use.
The API that the application uses is described in http://hadoop.apache.org/core/docs/current/api/.
Fundamentally,
the
application
uses the standard java.io interface. A C language
wrapper for this Java API is also available.
The client and the application code are bound into the same address space.
The connection between NameNode and the client is governed by the Client Protocol documented in …\hdfs\protocol\ClientProtocol.java. A client establishes a connection to a configurable TCP port on the NameNode machine. This is an RPC connection. The communication between the Client and the NameNode uses the ClientProtocol.
The major functions of the Client Protocol are:
A client communicates with a DataNode directly to
transfer
(send/receive) data using the DataTransferProtocol,
defined
in
DataTransferProtocol.java.
For performance purposes this protocol is a streaming protocol,
not RPC.
The client buffers data until a full block (the default is 64 Mbytes)
has been
created and then the block is streamed to the DataNode.
The DataTransferProtocol
defines
operations to read a block (opReadBlock()), write a block (opWriteBlock()),
replace
a
block
(opReplaceBlock()), copy a block (opCopyBlock()), and to get a block’s Checksum (opBlockChecksum()).
All communication between Namenode and Datanode is
initiated by the Datanode, and responded to by the Namenode. The
Namenode never initiates communication to the Datanode, although
Namenode responses may include commands to the Datanode that cause it
to send further communications.
DataNode sends information to NameNode through four major interfaces defined in the DataNodeProtocol. These four are
Registration is triggered when a new DataNode is initiated, an old one is re-initiated, or when a new NameNode is initiated.
In addition, there are several bookkeeping interfaces such as verify request, verify version, and get names or locations of various files or images.