Acquiring Kafka Delegation Token for Oozie Workflow
Acquire Kafka delegation tokens for use with an Oozie workflow.
-
Add the credentials of type
kafka
under the credentials block in yourworkflow.xml
and include the following properties:oozie.kafka.bootstrap.servers : {Your kafka bootstrap servers} oozie.kafka.sasl.kerberos.service.name : kafka oozie.kafka.security.protocol : SASL_PLAINTEXT oozie.kafka.sasl.mechanism : GSSAPI
The
oozie.kafka.bootstrap.servers:
field is a comma-separated list of "host and port" pairs that are the addresses of the Kafka brokers. Specify one or more.To pass other properties, add the
oozie.kafka.
prefix to the property key as shown above. You can pass these either from the Oozie site orworkflow.xml
.Note
The properties defined inworkflow.xml
will take higher precedence than the Oozie site.Optionally, you can enter the properties from the Ambari. For more information, see Entering Properties Using Ambari.
-
Include the credentials inside your workflow action:
<action name='mr-node' cred='my-kafka-creds'>
Example:
<workflow-app xmlns="uri:oozie:workflow:1.0" name="example-wf"> <credentials> <credential name="my-kafka-creds" type="kafka"> <property> <name>oozie.kafka.bootstrap.servers</name> <value>bootstrapServer1:6667,bootstrapServer2:6667,bootstrapServer3:6667,</value> </property> <property> <name>oozie.kafka.sasl.kerberos.service.name</name> <value>kafka</value> </property> <property> <name>oozie.kafka.security.protocol</name> <value>SASL_PLAINTEXT</value> </property> <property> <name>oozie.kafka.sasl.mechanism</name> <value>GSSAPI</value> </property> </credential> </credentials> <start to="mr-node"/> <action name="mr-node" cred='my-kafka-creds'> ... </action> </workflow-app>
-
Use the text identifier
KAFKA_DELEGATION_TOKEN
to fetch the Kafka delegation token from the job configuration.Note
-
If you are using the
AclAuthorizer
for Kafka, ensure the correct ACLs are set for users. Add the ACLs on the Kafka broker as follows:bin/kafka-acls.sh --authorizer-properties zookeeper.connect=localhost:2181 --add --allow-principal User:tokenRequester --allow-host * --operation CreateTokens --user-principal User:tokenOwner
-
If you are using the
RangerKafkaAuthorizer
for Kafka, ensure the appropriate policies are set. Specifically, add the token requester (in this case Oozie user) under theall - User policy
and grant theCreate Tokens
andDescribe Tokens
permission. This enables the token requester the permission to acquire a delegation token on behalf of the user who submitted the job.The above ACL or Ranger authorization comes into effect only if impersonation is used. For example,
token requester != token owner
.
- The
Oozie server's Keytab and Principal
defined on the Oozie site (oozie.authentication.kerberos.keytab
andoozie.authentication.kerberos.principal
) are used for authenticating with the Kafka broker.
-
Entering Properties Using Ambari (Optional)
Enter Kafka configuration properties in Ambari to acquire the Kafka delegation token for an Oozie workflow.
- Access Apache Ambari.
- From the side toolbar, under Services click Oozie.
- Click Configs.
- On the Advanced tab, navigate to Custom oozie-site.
-
Click Add Property and add the following properties:
oozie.kafka.bootstrap.servers
: Enter your Kafka bootstrap servers in a comma-separated list of "host and port" pairs that are the addresses of the Kafka brokers. Specify one or more.oozie.kafka.sasl.kerberos.service.name
: Enterkafka
.oozie.kafka.security.protocol
: EnterGSSAPI
.oozie.kafka.security.protocol
: EnterSASL_PLAINTEXT