Configuring HDFS (Hadoop Distributed File System)

In this chapter

This document will show you how to configure a cluster using QueryIO. Following are the steps to configure HDFS cluster:

Go to Admin > Manage Hosts.
Here you can add the machines on which you want to install nodes.
To add a machine, click on Add button.
In the Add Host wizard, specify the credentials for the system which you want to add.
If you want to use private key based authentication mechanism, you can do so by changing the selection in the Authentication Method dropdown box.
Click Next once you have entered all the required credentials.
In Installation Path textbox, enter the path of the directory where you want to install QueryIO services.
Change the value in the System JAVA_HOME textbox if it currently points to invalid java home directory.
In the QueryIO Agent Port textbox, specify the port for QueryIO agent service. QueryIO utilizes agent to perform service operations on different machines.
In the Rack Name textbox, specify the name of the rack to which your system belongs. Typically large Hadoop clusters are arranged in racks and network traffic between different nodes within the same rack is much more desirable than network traffic across the racks. A default installation assumes all the nodes belong to the same rack.
Click Save to add this machine to your cluster.

The NameNode is the centerpiece of an HDFS file system. It keeps the directory tree of all files in the file system in form of metadata, and tracks where across the cluster, the file data is kept. After you have added required machines to the cluster, you can add new NameNodes.
To add NameNode, Go to Hadoop > HDFS > NameNode.
Click on Add button.
In the HostName/IP dropdown, select the IP/hostname of the system on which you want to install the node.
Database: Select database to be linked with NameNode.

Metadata Database : To store HDFS metadata, Extended Metadata & user defined tags.
AdHoc Analysis Database : To store processed data as a result of big data analysis.

In the Unique Identifier textbox, enter a unique identifier for the node.
Let NameNode Type be Active.
If you want to utilize high availability feature of HDFS, you should also add standby nodes to your system. To add stand by node, change the namenode type selection to "StandBy".
Click Next.
Specify the ports for different services that QueryIO will start. Make sure that the ports configured for all of the services are unique. The ports should not conflict with that of any other services running on your system.
Click Next.
In Directory Path textbox, enter the path for the directory where you want your NameNode to store its metadata.
Select associated disk from the Disk dropdown box.
Make sure Start Namenode Service after installation checkbox is enabled. It will automatically start all the services for this namenode.
Click Save.

A DataNode stores data in the [HadoopFileSystem]. After you have added required NameNodes to the cluster, you can add new DataNodes.
To add DataNode, Go to Hadoop > HDFS > DataNode.
Click on Add button.
In the HostName/IP dropdown, select the IP/hostname of the system on which you want to install the node.
In the Unique Identifier textbox, enter a unique identifier for the node.
Click Next.
Specify the ports for different services that QueryIO will start.
Click Next.
In Volume Path textbox, enter the path for the directory where you want your DataNode to store the uploaded data.
Select associated disk from the Disk dropdown box.
If you want to configure multiple volumes for the datanode, you can do so using +/- buttons.
Make sure Start Datanode Service after installation checkbox is enabled. It will automatically start all the services for this datanode.
Click Save.

Your cluster is now configured. You can now upload data to the cluster and manage all resources through QueryIO UI very easily.