Oracle Cloud Infrastructure Documentation

Copying the Data to the Import Disk

Learn the steps for copying data to the user-supplied transfer disk for migration to Oracle Cloud Infrastructure.

Copy phase indicator for disk transfer

This topic describes the tasks associated with running the data transfer from the Data Host to the import disk. The Data Administrator role typically performs these tasks. See Roles and Responsibilities.

Information Prerequisites

Before performing any disk copying tasks, you must obtain the following information:

  • Disk block device path. The Infrastructure Engineer typically provides this information.

  • IAM login information, Data Transfer Utility configuration files, transfer job ID, and package label. The Project Sponsor typically provides this information.

Creating the Transfer Disk

The transfer disk is the logical representation of the physical import disk that has been configured for use for receiving data as part of the disk-based data transfer. See Transfer Disks for complete details on all tasks related to transfer disks.

Note

You can only use the Data Transfer Utility to create a transfer disk.

When you create a transfer disk for use with the disk on which you are copying your files, the Data Transfer Utility:

  • Sets up the disk for encryption using the passphrase.

  • Creates a file system on the disk.

  • Mounts the file system at /mnt/orcdts_label.

For example:

/mnt/orcdts_DJZNWK3ET

When you register a transfer disk, Oracle Cloud Infrastructure generates a strong encryption passphrase that is used to encrypt the contents on the disk. The encryption passphrase is displayed to standard output to the data transfer administrator user and cannot be retrieved again. Create a local, secure copy of the encryption passphrase, so you can reference the passphrase again.

Creating a transfer disk requires the job ID returned from when you created the transfer job and the path to the attached disk (for example, /dev/sdb).

Using the Data Transfer Utility

Use the dts disk create command and required parameters to create a transfer disk.

dts disk create --job-id job_id --block-device block_device

Run dts disk create --help to view the complete list of flags and variable options.

Copying Files to the Disk

You can only copy regular files to the disk. You cannot copy special files, such as symbolic links, device special, sockets, and pipes, directly to the disk. See the following section for instructions on how to prepare special files.

Important

  • Individual files being copied to the disk cannot exceed 10,000,000,000,000 bytes (10 TB).

  • Do not fill up the disk to 100% capacity. There must be space available to generate metadata and for the manifest file to perform the upload to Object Storage. At least 1 GB of free disk space is needed for this area.

  • File name characters must be UTF-8 and cannot contain a new line or a return character. Before copying data to the appliance, check the filesystem or source with the following command:

    find . -print0 | perl -n0e 'chomp; print $_, "\n" if /[[:^ascii:][:cntrl:]]/'

  • The maximum character file name length is 1024 characters.

Attach the disk to the Data Host and copy files to the mount point created by disk through the Data Transfer Utility.

Note

Only Linux machines can be used as Data Hosts.

Note

Copy all Files Before Disconnecting the Disk.

Do not disconnect the disk until you copy all files from the Data Host and generate the manifest file. If you accidentally disconnect the disk before copying all files, you must unlock the disk using the encryption passphrase. The encryption passphrase was generated and displayed when you created the transfer disk. If the generated encryption passphrase is not available, you must delete the transfer disk from the transfer job and re-create the transfer disk. All data previously copied to that disk is lost.

Copying Special Files

To transfer special files, create a tar archive of these files and copy the tar archive to the Data Transfer Appliance. We recommend copying many small files using a tar archive. Copying a single compressed archive file should also take less time than running copy commands such as cp -r or rsync.

Here are some examples of creating a tar archive and getting it onto the Data Transfer Appliance:

  • Running a simple tar command:

    tar -cvzf /mnt/nfs-dts-1/filesystem.tgz filesystem/

  • Running a command to create a file with md5sum hashes for each file in addition to the tar archive:

    tar cvzf /mnt/nfs-dts-1/filesystem.tgz filesystem/ |xargs -I '{}' sh -c "test -f '{}' && md5sum '{}'"|tee tarzip_md5

    The tar archive file filesystem.tgz has a base64 md5sum once it is uploaded to OCI Object Storage. Store the tarzip_md5 file where you can retrieve it. After the compressed tar archive file is downloaded from Object Storage and unpacked, you can compare the individual files against the hashes in the file.

Manifest File and Locking Information

Throughput for manifest file and locking is the average throughput for the disk used for disk import process. The manifest process initially opens threads based on 80% of available cores, but declines as the file count falls below the thread count.

Note

  • The sealing process uses a file tree walk and processes whichever files it comes across with no particular ordering.

  • When the file count falls below the thread count based on 80% of available cores, the thread count/ file count falls and throughput appears to drop.

  • Only one file is processed per a thread because of hash/checksum. computation.

  • Smaller files are processed faster than larger files.

  • Larger files can take hours to process.

Generating the Manifest File

Note

You can only use the Data Transfer Utility to generate a manifest file. The amount of time to generate the manifest file depends on the size of the upload files, disk speed, and available processing power.

After copying your data to a transfer disk, generate a manifest file using the Data Transfer Utility. The manifest contains an index of all of the copied files and generated data integrity hashes. The Data Transfer Utility copies the config_upload_user configuration file and referenced IAM credentials to the encrypted transfer disk. This configuration file describes the temporary IAM data transfer upload user. Oracle uses the credentials and entries defined in the config_upload_user file when processing the transfer disk and uploading files to Oracle Cloud Infrastructure Object Storage.

Note

Data Transfer Service Does Not Support Passphrases on Private Keys

While we recommend encrypting a private key with a passphrase when generating API signing keys, Data Transfer does not support passphrases on the key file required for the config_upload_user. If you use a passphrase, Oracle personnel cannot upload your data.

Oracle cannot upload data from a transfer disk without the correct credentials defined in this configuration file. See Installing the Data Transfer Utility for more information about the required configuration files.

Using the Data Transfer Utility

Use the dts disk manifest command and required parameters to create a manifest file. 

dts disk manifest --job-id job_id --disk-label disk_label 
--object-name-prefix object_name_prefix

Run dts disk manifest --help to view the complete list of flags and variable options.

Note

If you add, remove, or modify any files on the disk after generating the manifest file, you must regenerate the file. If the manifest file does not match the contents of the target bucket, Oracle cannot upload the data.

Locking the Transfer Disk

Note

You can only use the Data Transfer Utility to lock a transfer disk.

Locking a transfer disk safely unmounts the disk and removes the encryption passphrase from the Data Host.

Using the Data Transfer Utility

Use the dts disk lock command and required parameters to lock a transfer disk. 

dts disk lock --job-id job_id --disk-label disk_label 
--block-device block_device

Run dts disk lock --help to view the complete list of flags and variable options.

Unlocking the Transfer Disk

Note

You can only use the Data Transfer Utility to unlock a transfer disk.

When unlocking the transfer disk, you are prompted for the encryption passphrase that was generated when you created the transfer disk.

Using the Data Transfer Utility

Use the dts disk unlock command and required parameters to unlock a transfer disk. 

dts disk unlock --job-id job_id --disk-label disk_label 
--block-device block_device --encryption-passphrase encryption_passphrase

Run dts disk unlock --help to view the complete list of flags and variable options.

Attaching the Transfer Disk to the Transfer Package

Attach a transfer disk to a transfer package after you have performed the following tasks:

  1. Copied your data onto the disk.

  2. Generated the required manifest file.

  3. Run and reviewed the dry-run report.

  4. Locked the transfer disk in preparation for shipment.

A disk can be attached to one package, detached, and then attached to another package. In some cases, you have attached a transfer disk to a transfer package, but have changed your mind about shipping that disk with the transfer package. You can also detach a transfer disk from one transfer package and attach that disk to a different transfer package.

Using the Console

  1. Open the navigation menu and click Migration & Disaster Recovery. Under Data Transfer, click Imports. The Transfer Jobs page appears.

  2. Choose a Compartment you have permission to work in under List scope. All transfer jobs in that compartment are listed in tabular form.

  3. Click the transfer job associated with the transfer package for which you want to attach a disk.The transfer job's Details page appears.

  4. Click Transfer Disks under Resources. The Transfer Disks page appears. All transfer disks are listed in tabular form.

  5. Click Attach Transfer Disk. The Attach Transfer Disk dialog box appears.

  6. Select a transfer package to which the transfer disk is being attached from the Transfer Package list.

  7. Select the Transfer Disks that you want to attach to the transfer package.

  8. Click Attach.

Note

You can also attach a transfer disk to a transfer package through the transfer package's Details page.

Using the Data Transfer Utility

Use the dts disk attach command and required parameters to attach a disk to a transfer package.

dts disk attach --job-id job_id --package-label package_label 
--disk-label disk_label

Run dts disk attach --help to view the complete list of flags and variable options.

Detaching the Transfer Disk from the Transfer Package

Using the Console

  1. Open the navigation menu and click Migration & Disaster Recovery. Under Data Transfer, click Imports. The Transfer Jobs page appears.

  2. Choose a Compartment you have permission to work in under List scope. All transfer jobs in that compartment are listed in tabular form.

  3. Click the transfer job associated with the transfer package for which you want to detach a disk.The transfer job's Details page appears.

  4. Click Transfer Packages under Resources. The Transfer Packages page appears. All transfer packages are listed in tabular form.

  5. Click the transfer package from which you want to detach a transfer disk. The transfer package's Details page appears.

  6. Click Attached Transfer Disks under Resources. The Attached Transfer Disks page appears. All attached transfer disk are listed in tabular form.

  7. Click the attached transfer disk you want to detach. The attached transfer disk's Details page appears.

  8. Click Detach Transfer Disk. .

  9. Confirm the detachment.

Using the Data Transfer Utility

Use the dts disk detach command and required parameters to attach a disk to a transfer package.to detach a disk from a transfer package.

dts disk detach --job-id job_id --package-label package_label 
--disk-label disk_label

Run dts disk detach --help to view the complete list of flags and variable options.

Setting Tracking Details on the Transfer Package

After delivering the transfer package to the shipping vendor, update the transfer package with the tracking information.

Important

Oracle cannot process a transfer package until you update the tracking information.

Using the Console

  1. Open the navigation menu and click Migration & Disaster Recovery. Under Data Transfer, click Imports. The Transfer Jobs page appears.

  2. Choose a Compartment you have permission to work in under List scope. All transfer jobs in that compartment are listed in tabular form.

  3. Click the transfer job for which you want to see the associated transfer packages.The transfer job's Details page appears.

  4. Click Transfer Packages under Resources. The Transfer Packages page appears. All transfer packages are listed in tabular form.

  5. Click the transfer package that you want to update. The transfer packages Details page appears.

  6. Click Edit.

  7. Enter values into the Tracking ID and the Return Tracking ID boxes.

  8. Click Edit Transfer Package.

Using the Data Transfer Utility

Use the dts package ship command and required parameters to update the transfer package tracking information. 

dts package ship --job-id job_id --package-label package_label 
--package-vendor vendor_name --tracking-number tracking_number 
--return-tracking-number return_tracking_number

Run dts package ship --help to view the complete list of flags and variable options.

Notifying the Infrastructure Engineer

After completing the tasks listed in this topic, notify the Infrastructure Engineer of the following:

  • Disconnect the physical disk from the Data Host.

  • Package the disk for shipment.

What's Next

You are now ready to ship your disk with the copied data to Oracle. See Shipping Import Disks.