HDFS

Fresh HA HDFS Startup Process (Automatic Failover):

1. Initialize required state in ZooKeeper. You can do so by running the following command from one of the NameNode hosts :
   $HADOOP_HOME/bin/hdfs zkfc -formatZK -force

   This will create a znode in ZooKeeper inside of which the automatic failover system stores its data.

2. Start up the JournalNode daemons using the following command on each of the JournalNode servers :
   $HADOOP_HOME/sbin/hadoop-daemon.sh start journalnode

3. Format one of the NameNodes (nn1) as we are setting up a fresh cluster.
   $HADOOP_HOME/bin/hdfs namenode -format

4. Initialize JournalNodes, run the following command on Namenode nn1:
   $HADOOP_HOME/bin/hdfs namenode -initializeSharedEdits -force
   Running this command will initialize all JournalNodes with the edits data recorded after the most recent checkpoint from NameNode nn1 to the edits directory of all the JournalNodes (对 JouranlNode 集群的共享存储目录进行格式化，并且将原有的 NameNode 本地磁盘上最近一次 checkpoint 操作生成 FSImage 文件之后的 EditLog 拷贝到 JournalNode 集群上的共享目录之中)

   (如果都是HA HDFS就不需要执行此步骤)

5. Start nn1 NameNode(启动原有的 NameNode 节点):
   $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode

6. Run the following command on the new unformatted NameNode(nn2) in order to bootstrap it before starting it:
   $HADOOP_HOME/bin/hdfs namenode -bootstrapStandby -force

   This command synchronizes the namespace metadata stored on disk on both the NameNode by copying the latest fsimage file from nn1(active) to nn2(standby). It formats the storage on the Standby NameNode first and afterwards copies the lastest namespace snapshot from the active NameNode.

7. Start nn2 NameNode
   $HADOOP_HOME/sbin/hadoop-daemon.sh start namenode

8. Start the ZKFC service on both NameNode hosts
   $HADOOP_HOME/sbin/hadoop-daemon.sh start zkfc

9. Start all the DataNodes by issuing the following command:
   $HADOOP_HOME/sbin/hadoop-daemon.sh start datanode
   $HBASE_HOME/bin/graceful_stop.sh -d --restart --reload hostname

Administer HA HDFS Cluster

• Check the status of NameNode
  $HADOOP_HOME/bin/hdfs haadmin -getServiceState nn1

• Transition one of NameNodes to the active state
  $HADOOP_HOME/bin/hdfs haadmin -failover -forcefence -forceactive nn2 nn1

  $HADOOP_HOME/bin/hdfs haadmin -failover nn1 nn2
  Before running above commands, nn1 is in active state; after running this command, nn2 is in active state

  Transitions the Standby NameNode to the active status

  $HADOOP_HOME/bin/hdfs haadmin -transitionToActive <serviceId>**

• Verify the HA NameNode configuration
  $HADOOP_HOME/bin/hdfs getconf -namenodes

• Copy to multiple targets
  pdcp -w ^/pang/logs/datanodes /pang/hadoop265/etc/hadoop/* /pang/hadoop265/etc/hadoop/
  scp ./* [email protected]:/pang/hadoop265/etc/hadoop

• $HADOOP_HOME/sbin/start-balancer.sh -threshold 2
  $HADOOP_HOME/sbin/start-balancer.sh -threshold 5 -include apm-datanode02,apm-datanode10

• dd

• References

• HDFS Architecture

• HDFS File Block And Input Split

• Setup a multiple node cluster in Hadoop 2.0

• 详解HDFS Short Circuit Local Reads

• Sizing NameNode Heap Memory

• NameNode内存全景

• Tune Hadoop Cluster to get Maximum Performance

• How to Set Up Hadoop Cluster with HDFS High Availability

• Hadoop HDFS NameNode High Availability

• How to find the blocks/chunks of a file in HDFS?

• How to Setup ZooKeeper Cluster?

• ZooKeeper Cluster (Multi-Server) Setup

• HDFS Disk Scanner

• How to control size of log files for various HDP components?

• How to Change Hadoop Daemon log4j.properties

NameNode HA References

1. Hadoop高可用集群（HA+JournalNode+zookeeper）
2. 基于QJM/Qurom Journal Manager/Paxos的HDFS HA原理及代码分析
3. HDFS之Qurom Journal Manager（QJM）实现机制分析
4. Hadoop NameNode 高可用 (High Availability) 实现解析---Great
5. HDFS HA机制解析
6. NameNode 的ZKFC机制
7. ZKFC设计文档（ZK Failover Controller Design）
8. 利用QJM实现HDFS的HA策略部署与验证工作记录分享

9. 给Hadoop集群中添加Snappy解压缩库

10. Compression in HBase
11. HDFS Balancer (HDFS均衡操作分析)

HDFS Federation(联盟)

1. HDFS Federation设计动机与基本原理
2. HDFS Federation机制

Java Client HA Connection

1. Sample HDFS HA Client