Deployment using OCI Data Science Job
=======================================

ML Monitoring Application can be quickly set up using OCI Data Science Job and ML Monitoring Application managed container image.

- Configure required policy and dynamic rules
- Provision OCI Data Science Job
- Trigger Data science job run passing the required runtime configuration

Add Dynamic Groups Rules
--------------------------
User needs to create Dynamic Groups and add following rules for setting up OCI resources

.. code-block::

    ALL {resource.type='datasciencejobrun', resource.compartment.id = '<COMPARTMENT_ID>'}


Here COMPARTMENT_ID is the OCID of Compartment in which user need to setup OCI Data Science Job resource and push logs for ML Monitoring Application run .

.. seealso::
    `Dynamic Groups <https://docs.oracle.com/en-us/iaas/Content/Identity/getstarted/identity-domains.htm#identity_documentation>`_

Add Policy Rules
----------------------
Policy rules required to be added for authorizing and giving access permission to OCI resources within user’s compartment via dynamic-group . Here user is giving required permission using following policy to all the resources defined in the dynamic group created above.

.. code-block::

    Allow dynamic-group <ml-monitoring-application-dynamic-group> to manage log-groups in compartment <COMPARTMENT NAME>
    Allow dynamic-group <ml-monitoring-application-dynamic-group> to use log-content in compartment <COMPARTMENT NAME>
    Allow dynamic-group <ml-monitoring-application-dynamic-group> to manage objects in compartment <COMPARTMENT NAME>
    Allow dynamic-group <ml-monitoring-application-dynamic-group> to read buckets in compartment <COMPARTMENT NAME>

OCI Autonomous Data Warehouse (ADW) reader and post processor need to connect to OCI ADW database using database credentials and wallet files present in OCI vault, Following permission needs to be added by the user to access the OCI resource

.. code-block::

    Allow dynamic-group <ml-monitoring-application-dynamic-group> to read secret-bundles in compartment <COMPARTMENT NAME>


Configure OCI Data Science Job
----------------------
Steps to setup OCI Data Science Job :

- Create OCI Data Science Job using the `link <https://docs.oracle.com/iaas/data-science/using/jobs-create.htm#jobs_creating>`_.

    .. note::

        - Do not use BYOC while creating the job for ML Monitoring Application.
        - Do not use Fast Job. Only Custom Configuration is supported.
        - Do not use ARM shapes. Only Intel / AMD shapes are supported.


- Set the below environment variable while configuring the job:
    .. list-table::
       :widths: 25 15 25
       :header-rows: 1

       * - Key
         - Description
         - Value Example
       * - CONTAINER_CUSTOM_IMAGE
         - Managed Container moniker for ML Monitoring Application. `Available versions <../release_notes.html>`_
         - .. code-block:: bash

                  "dsmc://odsc-ml-monitoring-application:<version>"


Example

    .. image:: ../resources/job_created.png
       :width: 800
       :alt: Partition Data

    .. image:: ../resources/job_default_config.png
       :width: 800
       :alt: Partition Data


Run OCI Data Science Job
-------------------------------

After the OCI Data Science Job is created using ML Monitoring Application managed container, we can use OCI Data Science Job Run to repeatedly run the job.

.. seealso::
    `How to configure ML Monitoring Application config file <../config_setup.html>`_

Steps to configure OCI Data Science Job Run :

- Create OCI Data Science Job Run using the `link <https://docs.oracle.com/iaas/data-science/using/job-runs.htm>`_.
- Override the default job configuration that was defined when the job was created or add new runtime job configuration
- Refer the `input contract <../user_guide/input_configuration.html>`_ to add runtime configuration

Example

    .. image:: ../resources/job_run_created.png
       :width: 800
       :alt: Partition Data

    .. image:: ../resources/job_run_runtime_config.png
       :width: 800
       :alt: Partition Data


Other ways of creating Job and Job Runs
----------------------------------------
**CLI**

Use the `Data Science CLI <https://docs.oracle.com/en-us/iaas/tools/oci-cli/3.40.2/oci_cli_docs/cmdref/data-science/job/create.html>`_ to create a job as in this example:

1. Create a job with:
    .. code-block:: text

	oci data-science job create \
	--display-name <job_name>\
	--compartment-id <compartment_ocid>\
	--project-id <project_ocid> \
	--configuration-details file://<jobs_configuration_json_file> \
	--infrastructure-configuration-details file://<jobs_infrastructure_configuration_json_file> \
	--log-configuration-details file://<optional_jobs_infrastructure_configuration_json_file>

2. Use this jobs configuration JSON file:
    .. code-block:: json

	{
	  "jobType": "DEFAULT",
	  "environmentVariables": {
		"CONTAINER_CUSTOM_IMAGE": "dsmc://odsc-ml-monitoring-application:<version>"
	  }
	}

3. Use this jobs infrastructure configuration JSON file:
    .. code-block:: json

	{
	  "jobInfrastructureType":"ME_STANDALONE",
	  "shapeName": "<shape_name>",
	  "blockStorageSizeInGBs": "block_storage_size_in_GBs"
	}

4. (Optional) Use this jobs logging configuration JSON file:
    .. code-block:: json

	{
	  "enableLogging": true,
	  "enableAutoLogCreation": true,
	  "logGroupId": "<log_group_ocid>"
	}

5. Create a job run with:
    .. code-block:: text

	oci data-science job-run create \
	--display-name <job_name>\
	--compartment-id <compartment_ocid>\
	--project-id <project_ocid> \
	--job-id <job_ocid> \
	--configuration-override-details file://<jobs_override_configuration_json_file>

6. Use this job run override configuration JSON file:
    .. code-block:: json

	{
	  "jobType": "DEFAULT",
	  "environmentVariables": {
		"CONFIG_LOCATION": "<HTTP_LOCATION_OF_OCI_STORAGE_APPLICATION_CONFIG_FILE_PATH>"
		"RUNTIME_PARAMETER": "{"ACTION_TYPE":"RUN_CONFIG_VALIDATION"}"
	  }
	}

**API**

The ADS SDK is also a publicly available Python library that you can install with this command:

    .. code-block:: text

	pip install oracle-ads


It provides the wrapper that makes the creation and running jobs from notebooks or on your client machine easy.

Use the `ADS SDK <https://accelerated-data-science.readthedocs.io/en/latest/user_guide/jobs/data_science_job.html>`_ to create job and run jobs

.. collapse:: job_and_job_run_example.py

	.. code-block:: code

		import ads
		import os
		from ads.jobs import Job, DataScienceJob, PythonRuntime

		# Setting OCI_ODSC_SERVICE_ENDPOINT
		os.environ['OCI_ODSC_SERVICE_ENDPOINT'] = 'https://datascience-int.us-ashburn-1.oci.oc-test.com'

		def create_job_and_job_run():
			rp_version = os.environ.get("OCI_RESOURCE_PRINCIPAL_VERSION", "UNDEFINED")

			# Setting authentication

			# Setting local authentication
			if not rp_version or rp_version == "UNDEFINED":
				ads.set_auth("security_token", oci_config_location="~/.oci/config", profile="DEFAULT")
			else:
				# Setting Resource Principal authentication
				ads.set_auth("resource_principal")

			job = (
				Job(name="Job_creation_using_Ads")
					.with_infrastructure(
					DataScienceJob()
						.with_compartment_id(COMPARTMENT_OCID)
						.with_project_id(PROJECT_OCID)
						.with_job_type('DEFAULT')
						.with_shape_name("SHAPE_NAME")
						.with_block_storage_size(BLOCK_STORAGE_SIZE)
						.with_log_group_id(LOG_GROUP_OCID)
						.with_job_infrastructure_type('ME_STANDALONE')
				)
					.with_runtime(
					PythonRuntime()
						.with_environment_variable(CONTAINER_CUSTOM_IMAGE="dsmc://odsc-ml-monitoring-application:<version>")
				)
			)

			# Create Job
			job.create()

			# Create Job Run
			job_run = job.run(
				name="ML_Monitoring_Validation_Run_ads_1",
				env_var={
					"CONFIG_LOCATION": "<HTTP_LOCATION_OF_OCI_STORAGE_APPLICATION_CONFIG_FILE_PATH>",
					"RUNTIME_PARAMETER": "{\"ACTION_TYPE\":\"RUN_CONFIG_VALIDATION\"}"
				}
			)


**Terraform**

Use the `Data Science Create Job Terraform <https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/datascience_job>`_ to create a job.

Use the `Data Science Create Job Run Terraform <https://registry.terraform.io/providers/oracle/oci/latest/docs/resources/datascience_job_run>`_ to create a job run.