Hadoop High Availability

In this chapter

This chapter explains High Availability feature supported by QueryIO. Various aspects of high availability explained are:

• Description
• HA feature with QJM (Quorum Journal Manager)
• HA feature with NFS Mount

Description

High Availability(HA) feature of QueryIO will make sure that QueryIO agent service on host is always available. To make it possible, in every two minutes(configurable), host checks itself whether the QueryIO agent process is up or not and if it is down, then host will start QueryIO agent process on itself. This is made possible through crontab, which is a time-based job scheduler in Unix-like computer operating systems.

HDFS has always had a well-known single point of failure which impacts HDFS' availability: The system relies on a single NameNode to coordinate access to the file system data and if NameNode is down, then the whole cluster is unavailable. Hadoop introduced its own High Availability feature to ease this problem. The HA-Enabled cluster comprises of two redundant NameNodes, one is in Active state and the other is in StandBy state. These two nodes access a shared directory(example: NFS Mounted disk) to synchronize their FileSystem Image through Audit Logs. Now the single point of failure is reduced to availability of this shared directory. This feature provides a hot-standby between these two nodes and in case of failure, manual failover can be performed through QueryIO UI to switch states between these nodes.

NameNode federation does not support HA feature. There must be single active NameNode in the cluster for HA feature to work.

HA feature with QJM (Quorum Journal Manager)

For HA feature to use QJM, Add Standby NameNode with QJM configured.
For High availability, Standby NameNode needs to be synchronized with active NameNode, so both NameNodes communicate with each other through Journal Node. When any namespace modification is performed by the Active node, it durably logs a record of the modification to a majority of these JNs. Standby NameNode constantly monitors edit logs at journal nodes and updates its namespace accordingly.
In the event of failover, standby NameNode will ensure that its namespace is completely updated according to edit logs before it is changes to Active state. Thus namespace is fully synchronized before a failover.

Click on FailOver to initiate HA feature.

HA feature with NFS Mount

How to use HA Feature

For HA feature to use QJM, Add Standby NameNode with NFS mount configured.

Use following guidelines to configure HA feature:

• Cluster should be HA-Enabled.
  • By default, cluster is non-HA cluster.
  • Add standby NameNode to change cluster to enable HA.
  • After standby NameNode is successfully configured, Type of NameNodes will change to Active and Standby from Non-HA.
  • Make sure that NFS shared directory mount point for both Active and Standby NameNodes should be same.
• FailOver feature can only be used when cluster is HA-Enabled.
• Click on FailOver to switch active NameNode to Standby mode and standby NameNode to active mode.

NFS Mount Steps

Both the NFS server and NFS client must have parts of the NFS package installed and running. NFS is installed by default, and also by default NFS is activated when the system boots. If you want to install NFS package manually, use following command:

• yum install nfs-utils

Configuring NFS on The Server

• The /etc/exports File: This is the main NFS configuration file, and it has got two columns:
  • The first column contains the directories which is made available to the network.
  • The second column is further divided into two parts. The first part contains the networks or DNS domains that can access the directory, and the second part contains NFS options in brackets.
• Add directory to be mounted in "/etc/exports" file. Suppose "/data" directory is to be mounted, then add this directory :

• # vi /etc/exports
  /data *(rw,sync)
  	

• Use the chkconfig command to configure the required nfs and RPC rpcbind daemons to start at boot.

• # chkconfig --level 35 nfs on
  # chkconfig --level 35 nfslock on
  # chkconfig --level 35 rpcbind on			
  			

• Three services of the NFS are required to be active on server: rpcbind, nfs, nfslock
  Services can be started using start option. You can also stop and restart the processes with the stop and restart options. For example:
  • service rpcbind start
  • service nfs start
  • service nfslock start
• Use rpcinfo command to test whether NFS is running correctly. You should get a listing of running RPC programs that must include mountd, portmapper, nfs, and nlockmgr.

• # rpcinfo -p
  program vers proto   port  service
      100000    4   tcp    111  portmapper
      100000    3   tcp    111  portmapper
      100024    1   udp  59471  status
      100024    1   tcp  14485  status
      100011    1   udp    875  rquotad
      100005    1   udp  20780  mountd
      100005    1   tcp  57275  mountd
      100003    2   tcp   2049  nfs
      100003    3   tcp   2049  nfs
      100227    2   tcp   2049  nfs_acl
      100227    3   tcp   2049  nfs_acl
      100021    1   udp  54505  nlockmgr
      100021    3   udp  54505  nlockmgr
  			

Configuring NFS on The Client

• chkconfig command is used to configure the required nfs and RPC rpcbind daemons to start at boot.

• # chkconfig --level 35 netfs on
  # chkconfig --level 35 nfslock on
  # chkconfig --level 35 rpcbind on			
  			

• Three services of the nfs are required to be active on client: rpcbind, netfs, nfslock
  Services can be started using start command. You can also stop and restart the processes with the stop and restart options. For example:
  • # service rpcbind start
  • # service netfs start
  • # service nfslock start
• Use rpcinfo command to test whether NFS is running correctly. The listing of running RPC programs one gets must include status, portmapper, and nlockmgr.

• # rpcinfo -p
  program vers proto   port  service
      100000    4   tcp    111  portmapper
      100000    3   tcp    111  portmapper
      100024    1   udp  59471  status
      100024    1   tcp  14485  status
      100011    1   udp    875  rquotad
      100005    1   udp  20780  mountd
      100005    1   tcp  57275  mountd
      100003    2   tcp   2049  nfs
      100003    3   tcp   2049  nfs
      100227    2   tcp   2049  nfs_acl
      100227    3   tcp   2049  nfs_acl
      100021    1   udp  54505  nlockmgr
      100021    3   udp  54505  nlockmgr
  			

Mounting the NFS Directory

• In case you want to mount "/data" directory as NFS-type filesystem on the /mnt/nfs mount point. NFS server is 192.168.0.1 .
• To mount a NFS directory, use the following command:
  

  • # mkdir /mnt/nfs			
    # mount -t nfs <NFSServer>:<directory> <mount point directory>

  • <NFSServer> : IP address of the nfs server. In our case , it is 192.168.0.1.
  • <directory>: Directory to be mounted as NFS-type filesystem. In our case, it is "/data"
  • <mount point directory> : Path to mount point directory. (/mnt/nfs).
• For example:

  mount -t nfs 192.168.0.1:/data /mnt/nfs

• You will notice that all the file in the "/data" directory at NFs server appears in the "/mnt/nfs" directory.

Copyright © 2018 QueryIO Corporation. All Rights Reserved.

QueryIO, "Big Data Intelligence" and the QueryIO Logo are trademarks of QueryIO Corporation. Apache, Hadoop and HDFS are trademarks of The Apache Software Foundation.