Troubleshooting Data Transfer

You may need to change the version of the Oracle Cloud Infrastructure command line interface (CLI) to address issues with a particular feature. Installation of a CLI version other than the one currently installed requires the following steps in order:

pip uninstall oci-cli

virtualenv_path/bin/pip uninstall oci-cli

You can generate performance information for troubleshooting issues with the appliance through the terminal emulator on the host machine. Select Collect Appliance Diagnostic Information from the serial console configuration menu. The diagnostic tool generates system, network, storage, and performance data while the transfer job is running. It then forwards the data to the appliance serial console. Here you can scroll through the terminal to view it.

You can also use the log capture feature of the serial port emulator to capture the output. Serial port emulators often support the ability to copy the session to a file. Refer to the documentation of your serial port emulation package for instructions. Copying to a log file is useful if you need assistance from Oracle or if your emulation session does not allow you to scroll back and see all the output.

For each operation, the display shows exactly what command was executed and all the options.

Here is an example of the diagnostic output:

--------------------------------------------------------------------------------
- systemctl -l --type service --state=active                                   -
--------------------------------------------------------------------------------
UNIT LOAD ACTIVE SUB DESCRIPTION
auditd.service loaded active running Security Auditing Service
blk-availability.service loaded active exited Availability of block devices
chronyd.service loaded active running NTP client/server
console-diags@39-3147-1001.service loaded active running Diagnostic Collection Server for the XA (PID 3147/UID 1001)
crond.service loaded active running Command Scheduler
data-transfer-appliance.service loaded active running Data Transfer Appliance
data-transfer-console.service loaded active running Data Transfer Serial Console

Any problem with the diagnostic data collection results in the console output being written to the log file of the service. Failure of the commands is indicative of a serious problem, perhaps requiring the return of the appliance. Here is an example of the log:

Mar 6 17:55:33 localhost console-diags: {"Module": "main", "Type": "Info", "Message": "Received message {\"cmd\": \"collect\"}"}
Mar 6 17:55:33 localhost console-diags: {"Module": "main", "Type": "Info", "Message": "Setting up output file. First to remove all /tmp/xa-diags-results"}
Mar 6 17:55:33 localhost console-diags: {"Module": "main", "Type": "Info", "Message": "Removing /tmp/xa-diags-results.2019-03-06T17:54:56.000471"}

Initializing the Appliance can fail because of using the incorrect IP address. The IP address for initialize-auth can differ from the IP address obtained when running ping or ssl connect. If you experience an initialization failure, ensure that you are using the correct IP address for your Appliance and try initializing again.

If you try to configure networking using the appliance serial console but fail with a "connection refused" or "connection timed out" message, follow these troubleshooting steps.

Run the following command at the command prompt on the host:

ping appliance_ip

If a failure occurs, run the following command to verify appliance IP and the path to appliance.

ping -I local_interface appliance_ip

To determine expected interface, run ip route or an equivalent command. Verify that routing table is sane. Try running traceroute if you're not sure to see the network path to the appliance IP.

Run the following command:

curl -k https://appliance_ip

You should receive the response "Not found." This failure can indicate the IP address may be wrong. For example, nothing is listening on port 443.If you receive a failure message, run the following command:

openssl s_client -showcerts -connect appliance_ip:443

You should see a certificate issued for "Oracle Cloud Infrastructure" / "Data Transfer Appliance."

This command is similar to curl but does not use HTTPS and so proxies do not affect it. If this command works, and curl fails, then verify there are no proxy environmental variables.

The dataset sealing process can fail sometimes because there are special files in the dataset:

dts nfs-dataset seal-status --name nfs-ds-1

Seal Status :
 success           : false
 failureReason     :
Number of special files : 5
 startTime         : 2019/03/26 11:52:37 PDT
 endTime           : 2019/03/26 11:52:39 PDT
 numFilesToProcess : 0
 numFilesProcessed : 0
 bytesToProcess    : 0.00 KB
 bytesProcessed    : 0.00 KB
 bytesToProcess    : 0.00 KB

At the command prompt on the host, reactivate the NFS dataset.

oci dts nfs-dataset activate --name dataset_name

Then run find to get the full list of all special files and the specific type of each one.

find mountpoint \! -type f \! -type d | xargs file

For example:

$ find /mnt/nfs-ds-1 \! -type f \! -type d | xargs file
/mnt/nfs-ds-1/myfile1: symbolic link to `/home/user1/myfile1'
/mnt/nfs-ds-1/myfile2: symbolic link to `/home/user1/myfile2'

Next, review the list and remove all special files from the NFS mount point.

find mountpoint \! -type f \! -type d | xargs rm

Deactivate the NFS dataset.

oci dts nfs-dataset deactivate --name dataset_name

Finally, reseal the dataset.

oci dts nfs-dataset seal --name dataset_name [--wait]

Monitor the seal progress. Wait for it to complete successfully and continue with the subsequent steps.

Data sealing fails if the names of the files being transferred contain characters that are not UTF-8, contain a newline, or include a return. The error returned is similar to the following:

failureReason": "\nNumber of file paths with invalid chars: 1

If this error occurs, activate the data set, mount it, and run the following find command on the filesystem:

find . -print0 | perl -n0e 'chomp; print $_, "\n" if /[[:^ascii:][:cntrl:]]/'

Rename or delete those files that appear in the returned list.

The Data Transfer Utility requires exclusive access to the disk. If you have any drivers that already claim exclusive access to the disk, then the Data Transfer Utility fails. For example, if you employ a devicemapper multipath driver over all your disk devices, you must first remove the disk used for the data transfer from the list of devices managed by the multipath driver.

Be sure that access to the disk is not done through any devicemapper or volume manager. During the data transfer, the expectation is that the file system is created on a "raw" device. Any layering or mapping through intermediate drivers or abstraction layers makes it impossible for the disk to be uploaded at the transfer site. The source of these failures can include drivers like multipath, md, striping, logical volume managers, and potentially others as well.

You can confirm that the Data Transfer Utility has exclusive access by attempting to manually format the disk being used for your data transfer. The Data Transfer Utility uses the cryptsetup utility to create an encrypted device. You can run cryptsetup from the command line (root privileges required):

cryptsetup luksFormat -c aes-xts-plain64 -s 512 -h sha512 --iter-time 2000 --use-random /dev/sdXX

sdXX is the name of the disk being used for the data transfer.

When prompted, respond that you do want to encrypt the device. You are required to provide a passphrase. Any passphrase is acceptable as the cryptsetup utility can run on a disk repeatedly without any problems.

If the command succeeds, then you know that the Data Transfer Utility can gain exclusive ownership of the disk to do the necessary for the data transfer.

Check if your environment has proxies to the internet. If so, update them to the latest version and set "https_proxy." If you are using the appliance, set "no_proxy" environmental variables. See Installing the Data Transfer Utility for more information on proxies.

If you attempt to run Data Transfer commands and receive the error message "invalid configuration file," verify that the following files are present on your host and are correctly set up:

• ~/.oci/config and

• ~/.oci/config_upload_user

Both files must have "[DEFAULT]" as the first line. Use of the "~" character in a path is not valid in the file's contents.

Creating a transfer disk using the Data Transfer can fail because of a serial number error:

dts disk create --job-id ocid1.datatransferjob.oc1..exampleuniqueID --block-device /dev/sdb
ERROR : Unable to determine serial number for device /dev/sdb

This error may result from a garbled serial number resulting from the hdparm -I command. For example:

/bin/sh -c "hdparm -I /dev/sdb"
/dev/sdb:
SG_IO: bad/missing sense data, sb[]: 70 00 05 00 00 00 00 0a 00 00 00 00 24 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
ATA device, with non-removable media
Standards:
Likely used: 1
Configuration:
Logical max current
cylinders 0 0
heads 0 0
sectors/track 0 0
–
Logical/Physical Sector size: 512 bytes
device size with M = 1024*1024: 0 MBytes
device size with M = 1000*1000: 0 MBytes
cache/buffer size = unknown
Capabilities:
IORDY not likely
Cannot perform double-word IO
R/W multiple sector transfer: not supported
DMA: not supported
PIO: pio0

If you see this type of error, use the following workaround:

1. Run the following command at the prompt:

   lsblk --nodeps -no serial /dev/device
   serial_number

2. Create an hdparm script in your Home directory using the following command:

   vi $HOME/hdparm
   #!/usr/bin/bash
   while getopts ":Iht" opt;do
   case ${opt} in
   h) # process option h
   ;;
   t)
   ;;
   I)
   echo "Serial Number: serial_number"
   ;;
   esac
   done

   Use the same serial number in your script that was returned when you ran the lsblk command in the previous step.

3. Make the script you just created executable.

4. Change your path using the following command:

   export PATH=/home_dir_path:$PATH

5. Retry creating the transfer disk.