Harvest from Autonomous Databases with Private Access
Harvesting is a process that extracts technical metadata from your data source into your data catalog. This tutorial provides the steps to harvest from a data source that is only accessible privately.
In this tutorial, you:
- Create the policies needed to harvest an autonomous database with private URL.
- Obtain the autonomous database access details.
- Create a private endpoint in Data Catalog.
- Attach the private endpoint to your data catalog.
- Create a data asset.
- Add a connection for the data asset.
- Harvest the data asset.
For more information, see configuring a private network.
Before You Begin
To successfully perform this tutorial, you must have the following:
- An Oracle Cloud Infrastructure account. See signing up for Oracle Cloud Infrastructure.
- Access to use the Data Catalog resources. See prerequisites and policy examples.
- A created data catalog instance. See creating a data catalog instance.
If you already have the autonomous database you want to harvest from, you can use the details for that database to complete this tutorial. If you don't have an existing autonomous database with private access and want to try this tutorial, you can follow the following instructions to set up the resources needed to perform this tutorial.
Setting up the Resources Needed for this Tutorial
You create policies in Oracle Cloud Infrastructure to allow access to the various resources.
Before you can create a private network in your tenancy, you must have the required networking permissions.
In this setup, you create a policy to allow you to perform all networking operations in any compartment in your tenancy.
Perform the following steps:
A Virtual Cloud Network (VCN) is a virtual, private network that you set up in a single Oracle Cloud Infrastructure region. A VCN has a single, contiguous IPv4 CIDR block of your choice.
The allowable VCN size range is /16 to /30. Decide on the CIDR block before you create a VCN. You can't change the CIDR value later. For your reference, here's a CIDR Calculator.
To create a VCN, complete the following steps:
- Open the navigation menu, click Networking, and then click Virtual cloud networks.
- Click Create VCN.
- Enter a name to identify the VPN and select the compartment you have permission to work in.
- Enter the CIDR block for the VCN. For this tutorial, you can enter 10.0.0.0/16.
- Select DNS RESOLUTION and enter a DNS label.
- Click Create VCN.
The VCN is created and the Virtual Cloud Networks Details page for the VCN displays.
By default, a route table, DHCP option, and security list are automatically created for the VCN. You use these default components when you create a private subnet.
Subnets are divisions you create in a VCN. Each subnet consists of a contiguous range of IP addresses that don't overlap with other subnets in the VCN. You create a private subnet when you don't want the resources created in the subnet to have public IP addresses.
Complete the following steps to create a private subnet:
- Click Create Subnet from the Virtual Cloud Networks Details page of the VCN you created in the previous step.
- Enter a name for the private subnet.
- Retain the default regional selection for subnet type.
- Enter the CIDR block for the private subnet. For this tutorial, you can enter 10.0.0.0/24.
- Select the default route table.
- Select PRIVATE SUBNET for subnet access.
- Select Use DNS Hostnames in this Subnet for DNS resolution.
- Enter a DNS label.
- Select the default DHCP options and the default security lists.
- Click Create Subnet.
The private subnet is created and displayed on the Subnets page in the compartment you chose.
When you create an autonomous database in a VCN, you need to specify the Network Security Group (NSG) for the database. An NSG consists of security rules that apply to only a group of VNICs.
Without security rules, no traffic is allowed in and out of VNICs in the VCN.
Complete the following steps to create an NSG with an ingress rule:
- Click Network Security Groups from the Virtual Cloud Networks Details page of the VCN you created before.
- Click Create Network Security Group.
- Enter a name for the NSG.
- Ensure you have permissions to work in the selected compartment and click Next.
- Select Ingress for DIRECTION.
- Select CIDR for SOURCE TYPE and enter the CIDR for the private subnet in SOURCE CIDR. For this tutorial, you can enter 10.0.0.0/24.
- Select TCP for IP PROTOCOL.
- Enter 1522 in DESTINATION PORT RANGE.
- Click + Another Rule.
- Select Egress for DIRECTION.
- Select CIDR for DESTINATION TYPE and enter the CIDR for the private subnet in DESTINATION CIDR. For this tutorial, you can enter 10.0.0.0/24.
- Retain All Protocols for IP PROTOCOL.
- Click Create.
In this tutorial, the autonomous database is created in the same subnet that's used in Data Catalog to configure the private network for harvesting. For this scenario, you have created the ingress and egress rules specifying the CIDR of the private subnet.
Your autonomous database might be in a different private subnet than the subnet used in Data Catalog to configure the private network for harvesting. In that case, you must create the ingress and egress rules specifying the CIDR of the VCN.
Complete the following steps to create an Autonomous Database with private access.
For example, to create an Autonomous Database with a Data Warehouse workload type:
- Open the navigation menu and click Oracle Database. Under Autonomous Database, click Autonomous Data Warehouse.
- Click Create Autonomous Database.
- Ensure you have permission to work in the selected compartment and enter a display name and database name for the autonomous database.
- Retain Data Warehouse as the workload type.
- Retain Serverless as the deployment type.
- Configure the database with database version 19c, 1 OCPU count, 1-TB storage, and enable Auto scaling.
- Enter a password for the ADMIN user. You need this password when you connect to this database later in this tutorial.
- In the Choose network access section, select Private endpoint access only.
- Select the VCN, subnet, and NSG you created in the previous steps.
- Enter a hostname prefix. This text appears in the database's private URL.
- Retain the BYOL option for license type.
- Click Create Autonomous Database.
When you create a VCN, a security list is created by default for the VCN. You can add more security rules to this default security list or create a security list to permit traffic in and out of your VCN. In this tutorial, you add security rules to the default security list.
Complete the following steps to create a security list with the required security rules:
- Open the navigation menu, click Networking, and then click Virtual cloud networks.
- Click the VCN you created before to view the VCN details.
- Click Security Lists from the Virtual Cloud Networks Details page of the VCN you created before.
- Click the Default Security List for <your vcn>.
- Click Egress Rules.
- Click Add Egress Rules.
- Enter the CIDR of your private subnet. For this tutorial, enter 10.0.0.0/24.
- Select All Protocols for IP PROTOCOL.
- Click Add Egress Rules.
- Click Ingress Rules.
- Click Add Ingress Rules.
- Enter the CIDR of your private subnet. For this tutorial, enter 10.0.0.0/24.
- Select TCP for IP PROTOCOL.
- Enter 1521–1522 for DESTINATION PORT RANGE.
- Click Add Ingress Rules.
In this tutorial, the autonomous database is created in the same subnet that is used in Data Catalog to configure the private network for harvesting. For this scenario, you have created the ingress and egress rules specifying the CIDR of the private subnet.
Your autonomous database may be in a different private subnet than the subnet used in Data Catalog to configure the private network for harvesting. In that case, you must create the ingress and egress rules specifying the CIDR of the VCN.
1. Create Access Policies
To configure Data Catalog to access the private network of a data source, you need access to networking and data catalog resources.
If you already have access to perform all Data Catalog and Networking operations in your required compartments, you may skip this step.
To create the policy needed to configure a private network in data catalog, perform the following steps:
2. Obtain Data Source Details
You need the private network and database connection information for the autonomous database you want to harvest.
Obtain the following details for the autonomous database:
Information Needed | Instructions to Obtain Information |
---|---|
For configuring the private network, you need the VCN and subnet name and the private URL of the database. |
Note: If you have more database in this network (same VCN and subnet) that you want to harvest, make note of the Private URL of those databases too. |
For creating the data asset, you need the database name. | From the autonomous database details page, note the database name from the General Information section. |
For adding a connection, you need the database wallet and sign in credentials. |
You also need the credentials (username and password) for the database that you specified when you created the autonomous database. If you didn't create the autonomous database, obtain the credentials from your admin. While harvesting, you can view only the database entities you have access to. |
3. Create a Private Endpoint
You create a Data Catalog private endpoint to configure the network access details for the autonomous database data sources you want to harvest.
To create a private endpoint in Data Catalog, perform the following steps:
ACTIVE
status.If the private endpoint status changes to
FAILED
, ensure you have the created the access policies and set up your private network
correctly.
4. Attach a Private Endpoint
You attach a private endpoint to a data catalog to allow data assets to be created for data sources available in the private network.
To attach a private endpoint to a data catalog, perform the following steps:
- Click Data Catalogs.
- Click the Actions menu for the data catalog where you want to attach the private endpoint and select Attach Private Endpoint.
- Select the private endpoint you created in the previous step and click Attach.
Updating
, and the private
endpoint is being attached. After the private endpoint is attached successfully, the
status of the data catalog changes to Active
.5. Create an Autonomous Database Data Asset
You are now ready to register your private IP autonomous database with Data Catalog as a data asset . In this tutorial, you create an Autonomous Data Warehouse data asset.
To create an autonomous database data asset, perform the following steps:
6. Add a Connection
After you register a data source as a data asset in your data catalog, you create a connection to your data asset to harvest it. You can create multiple connections to your data source. At least one connection is needed to harvest a data asset.
For autonomous database data source types, you can use secrets in Oracle Cloud Infrastructure Vault to store the password that you need to connect to the source using a connection. By using OCI Vault, you provide the OCID of the secret when specifying the connection details, so you don't have to enter the actual password when you create the data asset. You can also use secrets for the Oracle wallet and passwords instead of uploading the wallet when you create your data asset.
A vault is a container for keys and secrets. Secrets store credentials such as required passwords for connecting to data sources. You use an encryption key in a vault to encrypt and import secret contents to the vault. Secret contents are based64-encoded. Data Catalog uses the same key to retrieve and decrypt secrets while connecting a data asset to the data source. For more information about vault, key, and secret, see Overview of Vault. For information about copying the secret OCID, see View Secret Details.
- Provide a wallet password when you download the wallet.
- Remove the
.p12
file from the downloaded wallet zip. - Use any base64 encoder to encode the modified wallet zip to base64.
- Copy the base64-encoded data to a secret in a vault.
- Create a secret for the database password.
To add a connection for the autonomous database data asset, follow these steps:
7. Harvest the Data Asset
You're now ready to harvest your autonomous database data asset. Your autonomous database must have the data from which you want to harvest the technical metadata. If you used the setup instructions in this tutorial, you can harvest metadata from the default data that's available in your autonomous database.
To harvest your autonomous database data asset, perform the following steps:
What's Next
Now, you can explore the data asset, create a glossary, and link terms and tags to data objects.