Cloning Cluster Configurations
Cloning cluster configuration based bootstrap helper functions exist to import and export cluster configurations.
The export and import support for Big Data Service ODH service configurations across source and target clusters provides a convenient way to move Big Data Service ODH environment between clusters for various reasons, saving time and effort.
Exporting and importing Big Data Service ODH service configurations is helpful in several scenarios;
- Disaster recovery
- To recover from a failed primary Big Data Service ODH cluster, use the import helper function to quickly create a new cluster and import configuration of your services.
- Testing and development
- To create a testing or development environment, use the export helper function to create a backup of the Big Data Service ODH cluster configuration.
To export Big Data Service ODH service configuration, use the OCI Console. The exported configuration includes information about your cluster configurations.
To import the Big Data Service ODH service configuration to a target cluster, use the Console. The import updates configurations of all services that are present in the zip file created by export operation.
The export/import process doesn't include any data or metadata associated with your jobs. You must separately migrate data to the target cluster if needed.
To run a bootstrap script, see Running the Bootstrap Script.
For more information on the export and import helper functions, see:
Cloning Cluster Configurations Bootstrap Helper Functions
For more information on cloning cluster configurations bootstrap script, see Cloning Cluster Configurations.
For Cloning Cluster import and export helper function examples, see Export/Import Cluster Configuration Examples
To run a bootstrap script, see Running the Bootstrap Script.
Category | Helper Function | Functionality |
---|---|---|
Config Clusters | exportClusterConfs(output_path, service_list=[] spcl_service_config_dict={}) |
This helper function creates a zipped file that contains the configurations of all services in JSON format. Note:
When the job is successful. It gathers all the configurations for the services in the Big Data Service ODH environment and saves them in a JSON file and creates a zip of all such files in |
importClusterConfs(zip_file_path, override_group=False) |
This helper function extracts the JSON file from the zip file and uses the JSON file to recreate the configurations for the services in the target cluster. Note:
|
Export/Import Cluster Configuration Examples
For more information on the exportClusterConfs
and importClusterConfs
helper functions, see Cloning Cluster Configurations Bootstrap Helper Functions.
sys.path.append(os.path.abspath("/home/opc/cloud/flask-microservice/"))
def execute(updateConfigHelper):
# existing utility functions, mention output location as "/tmp"
cluster_name = updateConfigHelper.exportClusterConfs("/tmp")
sys.path.append(os.path.abspath("/home/opc/cloud/flask-microservice/"))
def execute(updateConfigHelper):
# existing utility functions
print("working hurrah")
#Run export operation for mentioned list of services.
service = ["HDFS","SPARK3","HIVE","YARN","MAPREDUCE2"]
cluster_name = updateConfigHelper.exportClusterConfs("/tmp",service)
sys.path.append(os.path.abspath("/home/opc/cloud/flask-microservice/"))
def execute(updateConfigHelper):
# existing utility functions
service_list = ["HDFS","SPARK3","HIVE","YARN","MAPREDUCE2"]
spcl_service_config_dict={"HIVE":["javax.jdo.option.ConnectionURL"], "HDFS":["fs.defaultFS","ranger.plugin.hdfs.policy.rest.url"]}
cluster_name = updateConfigHelper.exportClusterConfs("/tmp",service_list,spcl_service_config_dict)
sys.path.append(os.path.abspath("/home/opc/cloud/flask-microservice/"))
def execute(updateConfigHelper):
# existing utility functions
cluster_name = updateConfigHelper.importClusterConfs('/tmp/daily-cluster-ha6_configs_1683618209.zip')
# Exported zip can be present in object store and you can provide object store path in zip_file_path location
cluster_name = updateConfigHelper.importClusterConfs('https://objectstorage.us-ashburn-1.oraclecloud.com/.../cluster_configs_1683606387.zip')