Oracle Cloud Infrastructure Documentation

Creating and Managing Streams

Before publishing messages to a stream or consuming messages from a stream, you must first create a stream.

When creating a stream, consider your partitioning and security strategies.

You can use the Oracle Cloud Infrastructure (OCI) Console, the Command Line Interface (CLI), the Streaming API, the OCI SDKs, and Resource Manager to create your Streaming resources.

The following information describes how to create and manage streams by using the Console, CLI, and API, as applicable. Steps for using OCI SDKs and Resource Manager are supplied below.

For requirements to create and manage streams, see Getting Started with Streaming.

Stream Pools

Stream pools are logical groupings for streams. Every stream needs to be a member of a stream pool. If you don't create a stream pool, the Streaming service uses a default pool to contain your streams.

You can use stream pools to:

  • Organize streams into groups matching your organizational structure or a specific solution
  • Restrict access to a specified virtual cloud network (VCN) inside your tenancy so that streams in the pool are not accessible through the internet
  • Specify whether the data in the pool's streams should be encrypted using your own Vault encryption key or an Oracle-managed key

When you create a stream, you need to specify whether it should become a member of an existing stream pool, or a member of a new, automatically created stream pool. There is no limit to the number of stream pools you can create. See Creating a Stream Pool for more information.

Note

Stream names must be unique within a stream pool.

Stream Pools and Apache Kafka

Stream pools serve as the root of a virtual Apache Kafka cluster when you use Kafka with Streaming. All streams within the pool share the same Kafka configuration, encryption, and access control settings. Every action on that virtual cluster is scoped to that stream pool.

You can configure the stream pool to automatically create streams, or Kafka topics, and call KafkaAdminClient::createTopic to create a stream or topic in that stream pool.

Note

When specifying the SASL Connection string to use with Kafka Java client, the following user name is required:

username="<Namespace>/<identity_domain_name>/<username>/<stream_pool_id>"

For more information, see Using Streaming with Apache Kafka.

Using OCI SDKs to Create a Stream

To create a stream, use the createStream method of StreamAdminClient.

See the Developer Guide to Streaming for detailed SDK examples.

Using OCI SDKs to Create a Stream Pool

To create a stream pool, use the createStreamPool method of StreamAdminClient.

See the Developer Guide to Streaming for detailed SDK examples.

Using Resource Manager to Create a Stream

Resource Manager is an Oracle Cloud Infrastructure (OCI) service that lets you automate the process of provisioning your OCI resources. Using Terraform, Resource Manager helps you install, configure, and manage resources through the "infrastructure-as-code" model.

A Terraform configuration codifies your infrastructure in declarative configuration files. The configuration defines the resources you intend to provision, variables, and specific instructions for provisioning the resources

You can use Resource Manager or the Terraform CLI with the OCI Terraform provider to provision Streaming resources like streams and stream pools.

Stream Resource

You can use the oci_streaming_stream resource in Terraform configurations to create a stream in your compartment.

For example:

resource "oci_streaming_stream" "stream" {
  compartment_id     = var.compartment_ocid
  name               = "<stream_name>"
  partitions         = "<number_of_partitions>"
  retention_in_hours = "<retention_in_hours>"
}

For more information about writing configurations for use with Resource Manager, see Terraform Configurations for Resource Manager and Terraform Configuration.

Using Resource Manager to Create a Stream Pool

Resource Manager is an Oracle Cloud Infrastructure (OCI) service that lets you automate the process of provisioning your OCI resources. Using Terraform, Resource Manager helps you install, configure, and manage resources through the "infrastructure-as-code" model.

A Terraform configuration codifies your infrastructure in declarative configuration files. The configuration defines the resources you intend to provision, variables, and specific instructions for provisioning the resources

You can use Resource Manager or the Terraform CLI with the OCI Terraform provider to provision Streaming resources like streams and stream pools.

Stream Pool Resource

You can use the oci_streaming_stream_pool resource to create a stream pool with optional private endpoint and Kafka compatibility settings. Private endpoint settings require a VCN, a subnet, and a network security group. This example Terraform configuration creates those resources as well.

For example:

resource "oci_streaming_stream_pool" "test_stream_pool" {
  #Required
  compartment_id = var.compartment_ocid
  name           = "<stream_pool_name>"

  #Optional
  private_endpoint_settings {
    nsg_ids             = [oci_core_network_security_group.test_nsg.id]
    private_endpoint_ip = "10.0.0.5"
    subnet_id           = oci_core_subnet.test_subnet.id
  }

  kafka_settings {
    #Optional
    auto_create_topics_enable = true
    log_retention_hours       = 24
    num_partitions            = 1
  }
}

resource "oci_core_vcn" "test_vcn" {
  cidr_block     = "10.0.0.0/16"
  compartment_id = var.compartment_ocid
  display_name   = "testvcn"
  dns_label      = "dnslabel"
}

resource "oci_core_subnet" "test_subnet" {
  cidr_block     = "10.0.0.0/24"
  compartment_id = var.compartment_ocid
  vcn_id         = oci_core_vcn.test_vcn.id
}

resource "oci_core_network_security_group" "test_nsg" {
  compartment_id = var.compartment_ocid
  vcn_id         = oci_core_vcn.test_vcn.id
}

For more information about writing configurations for use with Resource Manager, see Terraform Configurations for Resource Manager and Terraform Configuration.