Managing Remote JMX Metrics in ODH Components

Most of ODH components expose metrics over JMX. To access these metrics outside of this process, use the remoteJMX metrics utility helper function to manage remote JMX metrics on demand through a custom bootstrap script.

Enabling remoteJMX requires updating JVM arguments for java process of each ODH component. Each Ambari service has service-env shell script template which deals with JVM arguments. This shell script template can be modified to enable and manage remoteJMX. To simplify management of remoteJMX there's a config property in Ambari for each component to enable/disable remoteJMX. For each component, go to the Configs section for each service in the Ambari UI, and then look for properties with following syntax:

  • odh.<service-name>.<component-name>.jmxremote

    This property is exposed as a toggle in the Ambari UI and is used for enabling and disabling remoteJMX for respective service and component.

  • odh.<service-name>.<component-name>.jmxremote.port

    This property denotes the remoteJMX port that the component JVM process exposes metrics.

The config toggle in Ambari UI enables remoteJMX for the component with default values as described below. To add JVM properties for remoteJMX, edit the <service>-env.sh config template in the Ambari UI.

  • -Dcom.sun.management.jmxremote=true
  • -Dcom.sun.management.jmxremote.ssl=false
  • -Dcom.sun.management.jmxremote.port=<port_value_for_component>
  • -Dcom.sun.management.jmxremote.authenticate=false

The following is the complete list of all Ambari config properties to manage remote JMX:

Service Component/Properties Remote JMX Port JVM args Config Property
HDFS NAMENODE
  • odh.hdfs.namenode.remotejmx
  • odh.hdfs.namenode.remotejmx.port
18011 HDFS_NAMENODE_OPTS in hadoop-env
DATANODE
  • odh.hdfs.datanode.remotejmx
  • odh.hdfs.datanode.remotejmx.port
18012 HDFS_DATANODE_OPTS in hadoop-env
JOURNALNODE
  • odh.hdfs.journalnode.remotejmx
  • odh.hdfs.journalnode.remotejmx.port
18013 HADOOP_JOURNALNODE_OPTS in hadoop-env
ZKFC
  • odh.hdfs.zkfc.remotejmx
  • odh.hdfs.zkfc.remotejmx.port
18014 HDFS_ZKFC_OPTS in hadoop-env
YARN RESOURCEMANAGER
  • odh.yarn.rm.remotejmx
  • odh.yarn.rm.remotejmx.port
18021 YARN_RESOURCEMANAGER_OPTS in yarn-env
NODEMANAGER
  • odh.yarn.nm.remotejmx
  • odh.yarn.nm.remotejmx.port
18022 YARN_NODEMANAGER_OPTS in yarn-env
APP_TIMELINE_SERVER
  • odh.yarn.timelineservice.remotejmx
  • odh.yarn.timelineservice.remotejmx.port
18023 YARN_TIMELINESERVER_OPTS in yarn-env
MAPREDUCE2 HISTORYSERVER
  • odh.mapred.historyserver.remotejmx
  • odh.mapred.historyserver.remotejmx.port
18031 MAPRED_HISTORYSERVER_OPTS in mapred-env
HIVE HIVE_METASTORE
  • odh.hive.metastore.remotejmx
  • odh.hive. metastore.remotejmx.port
18041 HADOOP_OPTS in hive-env (conditioned block)
HIVE_SERVER
  • odh.hive.hiveserver2.remotejmx
  • odh.hive.hiveserver2.remotejmx.port
18042 HADOOP_OPTS in hive-env (conditioned block)
HBASE HBASE_MASTER
  • odh.hbase.master.remotejmx
  • odh.hbase.master.remotejmx.port
18051 HBASE_MASTER_OPTS in hbase-env
HBASE_REGIONSERVER
  • odh.hbase.regionserver.remotejmx
  • odh.hbase.regionserver.remotejmx.port
18052 HBASE_REGIONSERVER_OPTS in hbase-env
ZOOKEEPER ZOOKEEPER_SERVER
  • odh.zookeeper.server.remotejmx
  • odh.zookeeper.server.remotejmx.port
18061 SERVER_JVMFLAGS in zookeeper-env
SPARK3 SPARK3_JOBHISTORYSERVER
  • odh.spark3.historyserver.remotejmx
  • odh.spark3.historyserver.remotejmx.port
18071 SPARK_HISTORY_OPTS in spark3-env
SPARK3_THRIFTSERVER
  • odh.spark3.thriftserver.remotejmx
  • odh.spark3.thriftserver.remotejmx.port
18072 SPARK_THRIFT_OPTS in spark3-env
KAFKA KAFKA_BROKER
  • odh.kafka.broker.remotejmx
  • odh.kafka.broker.remotejmx.port
18081 KAFKA_OPTS in kafka-env