Services and State with Ambari REST API

  • 时间:
  • 浏览:0
  • 来源:幸运飞艇_幸运飞艇官方

15

HCat ClientHCATHIVE1+

curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X PUT 

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/services/KNOX

curl -u admin:admin -H "X-Requested-By:ambari" -i -X DELETE 

8

UPGRADINGIn the process of upgrading the deployed bits.

Falcon ServerFALCON_SERVERFALCON1

5

Kafka BrokerKAFKA_BROKERKAFKA1+

"Body":{"HostRoles":{"state":"INSTALLED"}}}' 

11

Ranger Admin ServerRANGER_ADMINRANGER1-3

10

"Body":{"HostRoles": {"state""STARTED"}}}' 

25

/<NEW_HOST_FQDN>/host_components/OOZIE_SERVER

WIPING_OUTIn the process of wiping out the install.

Secondary NameNodeSECONDARY_NAMENODEHDFS1

9

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT 

HBase MasterHBASE_MASTERHBASE1+

/<NEW_HOST_FQDN>/host_components/OOZIE_SERVER

STARTEDState when start completed successfully.

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X DELETE

20

"Body":{"HostRoles": {"state""INSTALLED"}}}' 

Ganglia Metrics CollectorGANGLIA_MONITORGANGLIAALL

Application Timeline ServerAPP_TIMELINE_SERVERYARN1

STARTED <-INSTALLED, STARTING, STARTED

WebHcat ServerWEBHCAT_SERVERHIVE1

// Delete old component

// Install component by achieving stated INSTALLED

5

INSTALLED <-INIT, UNINSTALLED, INSTALLED, INSTALLING, STARTED, INSTALL_FAILED, UPGRADING, STOPPING, UNKNOWN, DISABLED

The Ambari management tool for Hadoop offers among other handy tools a comprehensive REST API for cluster administration. Logically a cluster is divided into hosts, services and service components. While the UI might not always has support for all needed scenarios sure the REST API can be used to achieve it. For example moving a master component of a service from one host to another.

INSTALLEDState when install completed successfully.

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X GET

Allowed State transitions of a component:

Nagios ServerNAGIOS_SERVERNAGIOS1

1

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/hosts

Hive ClientHIVE_CLIENTHIVE1+

UNINSTALLINGIn the process of uninstalling.

Mahout ClientMAHOUTMAHOUT0+

ServiceCardinality

INIT <-UNINSTALLED, INIT, WIPING_OUT

HiveServer2HIVE_SERVERHIVE1-2

4

ResourceMangerRESOURCEMANAGERYARN1-2

Ranger User SyncRANGER_USERSYNCRANGER1

2

7

-d '{"RequestInfo":{"context":"Install YARN_CLIENT"},

10

-d '{"ServiceInfo": {"state": "INSTALLED"}}'

14

17

Ambari Metrics ServiceMETRICS_COLLECTORAMS1

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/hosts

NameNodeNAMENODEHDFS1-2

6

15

DISABLEDDisabled master’s backup state.

The possible states a service component can be in is listed below. To change or get the state of a service component the following API calls can be issued.

2

HBase RegionServerHBASE_REGIONSERVERHBASE1+

// Setting the state of the component to INSTALLED

Spark Job History ServerSPARK_JOBHISTORYSERVERSPARK1

3

7

// Get service info

// Starting the new component

Nimbus ServerNIMBUSSTORM1

-d '{"ServiceComponentInfo": {"state": "INSTALLED"}}'

Service

Oozie ServerOOZIE_SERVEROOZIE1

NameAmbari Component Name

Ganglia ServerGANGLIA_SERVERGANGLIA1

// Add component to host

curl -u admin:admin -H "X-Requested-By:ambari" -i -X POST -d 

2

26

4

/<NEW_HOST_FQDN>/host_components/OOZIE_SERVER

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/YARN_CLIENT

Ranger Key Management ServerRANGER_KMS_SERVERRANGER_KMS1+

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/hosts

13

Among the need to change the state of a service component a very common issue is the need to actually move one service – most likely a master service – to different host. Users faced with this problems often need to issue the existing API due to a lack of support by the existing UI – at least with older releases of Ambari Web UI. This is an example of moving the OOZIE_SERVER to a new host.

1

Desired StateStart State

16

UNINSTALLEDState when uninstall completed successfully.

Sqoop ClientSQOOPSQOOP1+

DISABLED <-INSTALLED, INSTALL_FAILED, UNKNOWN

HBase ClientHBASE_CLIENTHBASE1+

9

NameAmbari Component Name

5

-d '{"RequestInfo": {"context": "Start Oozie","query":"HostRoles/component_name.in('OOZIE_SERVER')"}, 

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/KNOX_GATEWAY

-d '{"RequestInfo": {"context": "Stop Oozie","query":"HostRoles/component_name.in('OOZIE_SERVER')"}, 

STARTINGIn the process of starting.

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/services/KNOX/

12

Removing a complete service from the cluster would require all the removal of all it’s components together with the service itself. For example for removing the Knox service we would need to remove the KNOX_GATEWAY component from the cluster, but before let’s get their state:

16

ZookeeperZOOKEEPER_SERVERZOOKEEPER1+ (odd #)

Storm UISTORM_UI_SERVERSTORM1

curl -k -u admin:admin -H "X-Requested-By:ambari" -i -X GET

Cardinality

1

Falcon ClientFALCON_CLIENTFALCON1+

13

STOPPINGIn the process of stopping.

11

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT 

INSTALL_FAILEDInstall failed.

YARN ClientYARN_CLIENTYARN1+

Phoneix Query ServerPHOENIX_QUERY_SERVERHBASE0+

Storm REST ServerSTORM_REST_APISTORM1

Oozie ClientOOZIE_CLIENTOOZIE1+

Zookeeper Failover ServiceZKFCHDFS0+

MapReduce ClientMAPREDUCE2_CLIENTMAPREDUCE21+

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/hosts

23

-d '{"RequestInfo": {"context": "Install Oozie","query":"HostRoles/component_name.in('OOZIE_SERVER')"}, 

6

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/hosts/<HOST_FQDN>/host_components/YARN_CLIENT 

Spark ClientSPARK_CLIENTSPARK1+

22

/<OLD_HOST_FQDN>/host_components/OOZIE_SERVER

Knox GatewayKNOX_GATEWAYKNOX1+

// Stop component KNOX_GATEWAY

Zookeeper ClientZOOKEEPER_CLIENTZOOKEEPER1+

// Stop the old component by putting it back in state INSTALLED

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/services/KNOX

Node ManagerNODEMANAGERYARN1+

/<OLD_HOST_FQDN>/host_components/OOZIE_SERVER

StateDescription

INSTALLINGIn the process of installing.

24

8

8

// Delete service

6

7

INITInitial/Clean state.

UNINSTALLED <-INSTALLED, UNINSTALLED, UNINSTALLING

3

21

Ambari Metrics CollectorMETRICS_MONITORAMSALL

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X PUT 

17

14

A Hadoop cluster contains of multiple services that individually have either master, slave or client components. Below you will find a list of all currently supported components as part of a HDP stack divided in master, slave, or client groups. The cardinality notates the number of components that can exist at the same time in one cluster.

18

Possible states of a component:

curl -u admin:admin -H "X-Requested-By:ambari" -i -X PUT 

Storm SupervisorSUPERVISORSTORM1+

NameAmbari Component Name

// Delete component

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/hosts

HistoryServerHISTORYSERVERMAPREDUCE2 1

ServiceCardinality

"Body":{"HostRoles": {"state""INSTALLED"}}}' 

Journale Nodes for NameNode HAJOURNALNODEHDFS0+ (odd #)

DataNodeDATANODEHDFS1+

Zeppelin ServerZEPPELIN_MASTERSPARK / HIVE1

// Getting the state of the component

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X PUT 

UNKNOWNState could not be determined.

curl -u admin:admin -H "X-Requested-by:ambari" -i -k -X DELETE

4

In this post we are going to look a little closer at the way the Ambari API can be used to manage Hadoop services. At the end of this post you will find a list of all the currently supported Hadoop services with all the needed master, slave and client components that can be manged and administrated within your HDP stack. Also this posts contains the possible states and state transitions a component might have which could become useful when facing problems like Host config is in invalid state.

HDFS ClientHDFS_CLIENTHDFS1+

19

Secondary NameNodeNFS_GATEWAYHDFS0+

12

Hive MetastoreHIVE_METASTOREHIVE1-2

// Stop service KNOX

3

http://<HOST>:150150/api/v1/clusters/<CLUSTER_NAME>/services/KNOX/components/KNOX_GATEWAY

DRPC ServerDRPC_SERVERSTORM1