Monitoring Hadoop With Graphite

Monitoring Hadoop, HBase, ZooKeeper with Graphite and Grafana Using Hadoop Metrics2.(2017/04/19)

Most technologies has built-in instrumentation for collecting monitoring metrics. In Hadoop/HBase/Hive, it's Hadoop Metrics2. For Spark/Cassandra and others, it's likely to be Codahale/DropWizard Metrics.

Hadoop Version: 2.8.0
HBase Version: 1.2.5
ZooKeeper Version: 3.4.9

HBase will send metrics data to Carbon service (Carbon is one of three components within the Graphite project) via socket directly.

1 Setting Up the Hadoop Metrics

• Stop all Hadoop Services

• Modify $HADOOP_HOME/etc/hadoop/hadoop-metrics2.properties
  datanode.sink.graphite.server_host="Graphite Carbon service ip"
  datanode.sink.graphite.server_port="Graphite Carbon service port"
  datanode.sink.graphite.metrics_prefix=apm.pinpoint

  Above configuration use TCP transport protocol

• Restart all the Hadoop services.

2 Metric Keys Convention

hosts.$service.$role.$instanceId : hosts.apm.apm_collector
TCP Half Connection Issue

Restart HA NameNode

1. Failover to standby namenode: hdfs haadmin -failover nn1 nn2
2. Stop namenode nn1
3. Start namenode nn1

References

1. hadoop运维之NameNode重启 (With SecondaryNameNode)
2. Hadoop NameNode重启优化
3. Too Many Socket Connections In CLOSE_WAIT
4. HBase Metrics Filetering ---- 指标过滤
5. Hadoop metrics filtering

6. What is Hadoop Metrics2?

7. Java Regex - Tutorial