Bring Your Own Container (BYOC) for Pipelines
Apart from defining pipeline steps based on jobs and scripts, you can use custom container images to define the step runtime.
You can choose the container image, command, or entry point for container execution. You can provide the custom code in a script or a compressed archive, which lets you update the code without rebuilding the image.
BYOC Step Configuration
Model Artifact
Uploading a step artifact when using BYOC is optional. However, it lets you change the code that runs inside the container without rebuilding the image.
The step artifacts are mounted in the container to the folder
/home/datascience/
. If an artifact is an archive, its content is
extracted to the folder /home/datascience/decompressed_artifact
.
- Common docker image
- It's convenient to build a generic container image with the required environment (for example, Python 3.8 and basic libraries as shown in Quick start) and add Python scripts later as step artifacts.
- Custom artifacts - folder override
- When you use a custom step artifact, the service mounts a volume with the artifact to
the
/home/datascience
folder, overriding the folder in your container image. Archive artifacts (zip/tar/...
) are decompressed and the content is presented in the folder/home/datascience/decompressed_artifact
.
Create a Container Pipeline - Quick Start
Follow these steps to create a container pipeline step.
Building the Container
Testing the Container
Before pushing the image to a container registry, you can test it locally.
Pushing the Container to OCIR
Follow the steps in the Container Registry documentation to generate an auth token to log in to OCIR.
Creating the Pipeline
See the Pipeline Policies section to ensure that you have policies that let the pipeline run resource pull container images from the container registry.
- Create a pipeline with an appropriate name, for example,
BYOC Demo name
. - Click Add pipeline steps.
- Give the step a name, for example,
Step 1
. - To use Bring Your Own Container, select From container.
- In Container configuration, click Configure.
- In Configure your container environment:
- Select the repository quickstart or byoc from the list. If the repository is in a different compartment, click Change compartment.
- Select the image,
iad.ocir.io/idtlxnfdweil/quickstart/byoc:1.0.0
, from the list. - Click Select.
- Click Save.
- Optional: Define logging.
- Click Create.
Enabling the Pipeline Logs
- From the list of pipelines, click the pipeline you want to enable.
- From the pipeline details page, click Logs.
- Click Enable logs.
Supported Configurations
Important information about configurations that are supported.
- ML Pipelines only support container images residing in the OCI Registry.
- The size of the container image is limited to 40 GB in uncompressed form.
- The user who creates the ML Pipeline resource must have access to the container image in the OCI Registry. If not, create a user access IAM policy before creating the ML Pipeline resource.
Container images on Apple Silicon M1 Mac
For more information, see Docker Image on Apple an M1 MacBook.
Image digests
Images in a container registry are identified by repository, name, and a tag. Also, Docker gives each version of an image a unique alphanumeric digest. When pushing an updated container image, we recommend giving the updated image a new tag to identify it, rather than reusing an existing tag, as best practice. However, even if you push an updated image and give it the same name and tag as an earlier version, the newly pushed version has a different digest to the earlier version.
When you create a pipeline resource, you specify the name and tag of a particular version of an image. To avoid inconsistencies later on, pipelines also record the unique digest of that particular version of the image. You can also provide the digest of the image when creating the pipeline resource.
By default, if you push an updated version of an image to the Docker registry with the same name and tag as the original version of the image, pipelines continue to use the original digest to pull the original version of the image. This might be the behavior that you require. However, if you want pipelines to pull the later version of the image, you can explicitly change the image name with the tag and digest that pipelines use to identify which version of the image to pull.