Object Storage for Roving Edge Infrastructure

The Object Storage service provided by Oracle Cloud Infrastructure and Amazon S3 use similar concepts and terminology. In both cases, data is stored as objects in buckets. The differences are in the implementation of features and tools for working with objects.

The following highlights the differences between the two storage technologies:

• Compartments

  Amazon S3 does not use compartments. By default, buckets created using the Amazon S3 Compatibility API or the Swift API are created in the root compartment of the Oracle Cloud Infrastructure tenancy. Instead, you can designate a different compartment for the Amazon S3 Compatibility API or Swift API to create buckets in.

• Global bucket namespace

  Object Storage does not use a global bucket namespace. Instead, the namespace serves as the top-level container for all buckets and objects. At account creation time, each tenant is assigned one unique system-generated and immutable namespace name. The namespace spans all compartments within a region. You control bucket names, but those bucket names must be unique within a namespace. While the namespace is region-specific, the namespace name itself is the same in all regions. You can have a bucket named MyBucket in US West (Phoenix) and a bucket named MyBucket in Germany Central (Frankfurt).

• Encryption

  The Object Storage service encrypts all data at rest by default. Encryption can't be turned on or off using the API.

• Object Level Access Control Lists (ACLs)

  Oracle Cloud Infrastructure does not use ACLs for objects. Instead, an administrator needs to set up groups , compartments , and policies  that control which users can access which services, which resources, and the type of access. For example, the policies control who can create users and groups, create buckets, download objects, and manage Object Storage-related policies and rules.

For more information, see Overview of Object Storage.

To enable application access from Amazon S3 to Object Storage, you need to set up access to Oracle Cloud Infrastructure and modify your application.

1. Sign up for Oracle Cloud Infrastructure and obtain a unique namespace.

2. Any user of the Amazon S3 Compatibility API with Object Storage needs permission to work with the service. If you are not sure if you have permission, contact your administrator. For basic information about policies, see How Policies Work. For policies that enable use of Object Storage, see Common Policies and the Policy Reference.

3. Use an existing or create a customer secret key. A customer secret key consists of an access key/secret key pair. The creation and use of these keys is described in the following section on setting up the S3 configuration file.

Object Storage on Roving Edge Infrastructure supports the Amazon S3cmd tool in addition to supporting the Oracle Cloud Infrastructure command line interface. Support of the Amazon S3cmd tool requires you to set up the S3 configuration file but also supports S3cmd.

You can download s3cmd from the following site:

https://s3tools.org/download

On Macintosh and Linux, the s3cmd configuration file is named .s3cfg and it is located in the your home directory, for example:

/home/username/($HOME)

On Windows, the s3cmd configuration file is named s3cmd.ini and it is located in %USERPROFILE% -> Application Data, which is typically the following:

c:\users\username\AppData\Roaming\s3cmd.ini

For more details, reference: https://s3tools.org/kb/item14.htm

The following example shows contents the S3cmd configuration file:

[default]
access_key = 9aa40ec9-bec3-4fab-9e5a-3584a5979d48
access_token =
add_encoding_exts =
add_headers =
bucket_location = US
cache_file =
ca_certs_file = <Full path to the bundle.pem file>
check_ssl_hostname = False
default_mime_type = binary/octet-stream
delay_updates = False
delete_after = False
delete_after_fetch = False
delete_removed = False
dry_run = False
enable_multipart = True
encoding = UTF-8
encrypt = False
expiry_date =
expiry_days =
expiry_prefix =
follow_symlinks = False
force = False
get_continue = False
gpg_command = /usr/bin/gpg
gpg_decrypt = %(gpg_command)s -d --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_encrypt = %(gpg_command)s -c --verbose --no-use-agent --batch --yes --passphrase-fd %(passphrase_fd)s -o %(output_file)s %(input_file)s
gpg_passphrase =
guess_mime_type = True
host_base = <rover_node_ip>:8019
host_bucket = <rover_node_ip>:8019
human_readable_sizes = False
ignore_failed_copy = False
invalidate_default_index_on_cf = False
invalidate_default_index_root_on_cf = True
invalidate_on_cf = False
list_md5 = False
log_target_prefix =
max_delete = -1
mime_type =
multipart_chunk_size_mb = 15
preserve_attrs = True
progress_meter = True
proxy_host =
proxy_port = 0
put_continue = False
recursive = False
recv_chunk = 4096
reduced_redundancy = False
restore_days = 1
secret_key = <Your secret key, see below>
send_chunk = 4096
server_side_encryption = False
skip_existing = False
socket_timeout = 300
urlencoding_mode = normal
use_https = True
use_mime_magic = True
verbosity = WARNING
website_endpoint = https://<rover_node_ip>:8015
website_error =

To set up the S3cmd configuration file for use with Roving Edge Infrastructure:

1. Update the access_key field:

   1. Log into the Roving Edge Infrastructure Device Console of the appropriate device.

   2. Open the navigation menu and select Identity Management > Users. The Users page appears. All users are listed in tabular form.

   3. Click the user whose access key you want to use in the S3cmd configuration file. The user's Details page appears.

   4. Click Customer Secret Keys under Resources. The Customer Secret Keys page appears. All customer secret keys are listed in tabular form.

   5. Copy the access key and past it into the access_key field value in the s3cmd configuration file.

2. Update the ca_certs_file field: Enter the full path of your bundle.pem file, for example:

   /Users/user_name/.oci/bundle.pem

   If you do not have the ca certs file, download it using the following command:

   echo -n | openssl s_client -showcerts -connect rover_node_IP:8015 | 
   sed -ne '/-BEGIN CERTIFICATE-/,/-END CERTIFICATE-/p' > bundle.pem

3. Set the check_ssl_hostname field to false.

4. Update the host_base field: Enter the rover_node_IP:8019.

5. Update the host_bucket field: Enter the rover_node_IP:8019.

6. Update the secret_key field:

   1. Log into the Roving Edge Infrastructure Device Console of the appropriate device.

   2. Open the navigation menu and select Identity Management > Users. The Users page appears. All users are listed in tabular form.

   3. Click the user whose secret key you want to use in the S3cmd configuration file. The user's Details page appears.

   4. Click Customer Secret Keys under Resources. The Customer Secret Keys page appears. All customer secret keys are listed in tabular form.

   5. Click Generate Secret Key. The generated Secret Key is displayed in the Generate Secret Key dialog box. At the same time, Oracle generates the Access Key that is paired with the Secret Key. The newly generated Customer Secret key is added to the list of Customer Secret Keys.

   6. Copy the key and past it into the secret_key field value in the s3cmd configuration file.

7. Save and close the s3cmd configuration file.

Amazon S3 Compatibility API support is provided at the bucket level and object level.

Bucket APIs

The following bucket APIs are supported:

• DeleteBucket
• HeadBucket
• GetService (list all my buckets)
• ListObjects
• PutBucket

Object APIs

The following object APIs are supported:

• BulkDelete
• DeleteObject
• GetObject
• HeadObject
• PutObject

Multipart Upload APIs

The following multipart upload APIs are supported:

• AbortMultipartUpload
• CompleteMultipartUpload
• InitiateMultipartUpload
• ListParts
• ListUploads
• UploadPart

Using optional API headers, you can provide your own 256-bit AES encryption key that is used to encrypt and decrypt objects uploaded to and downloaded from Object Storage.

If you want to use your own keys for server-side encryption, specify the following three request headers with the encryption key information:

Object Storage has distinct APIs for copying objects and copying parts. Amazon S3 uses the presence of the following headers in PutObject and UploadPart to determine copy operations. To copy a source object that is encrypted with an SSE-C key, you must specify these three headers so that Object Storage can decrypt the object.

Using optional API headers, you can provide your own encryption key in Vault that is used to encrypt objects uploaded to Object Storage.

If you want to use your own keys in Vault for server-side encryption, specify the following request header with the OCID of the key in Vault:

You can configure various client applications to talk to Object Storage's Amazon S3-compatible endpoints. This topic provides some configuration examples for supported Amazon S3 Clients. Review the prerequisites in Setting up Access to Oracle Cloud Infrastructure.

The AWS SDK for Java repository, file download, and documentation links are available on GitHub: https://github.com/aws/aws-sdk-java.

Here is an example of configuring AWS SDK for Java to use Object Storage

                // Put the Access Key and Secret Key here
                
AWSCredentialsProvider credentials = new AWSStaticCredentialsProvider(new BasicAWSCredentials(
 "gQ4+YC530sBa8qZI6WcbUbtH8oar0exampleuniqueID",
 "7fa22331ebe62bf4605dc9a42aaeexampleuniqueID"))));

// Your namespace
String namespace = "rover-namespace";

// The region to connect to
String region = "us-ashburn-1";

// Create an S3 client pointing at the region
String endpoint = String.format("%s.compat.objectstorage.%s.oraclecloud.com",namespace,region);
AwsClientBuilder.EndpointConfiguration endpointConfiguration = new AwsClientBuilder.EndpointConfiguration(endpoint, region);
AmazonS3 client = AmazonS3Client.builder()
 .standard()
 .withCredentials(credentials)
 .withEndpointConfiguration(endpointConfiguration)
 .disableChunkedEncoding()
 .enablePathStyleAccess()
 .build();

s3 = new AWS.S3({
  region: 'us-ashburn-1',
  endpoint: 'https://' + rover-namespace + '.compat.objectstorage.us-ashburn-1.oraclecloud.com',
  accessKeyId: 'gQ4+YC530sBa8qZI6WcbUbtH8oar0exampleuniqueID',
  secretAccessKey: '7fa22331ebe62bf4605dc9a42aaeexampleuniqueID',
  s3ForcePathStyle: true,
  signatureVersion: 'v4',
});

import boto3
  
s3 = boto3.resource(
    's3',
    aws_access_key_id="gQ4+YC530sBa8qZI6WcbUbtH8oar0exampleuniqueID",
    aws_secret_access_key="7fa22331ebe62bf4605dc9a42aaeexampleuniqueID",
    region_name="us-phoenix-1", # Region name here that matches the endpoint
    endpoint_url="https://rover-namespace.compat.objectstorage.us-phoenix-1.oraclecloud.com" # Include your rover-namespace in the URL
)
  
# Print out the bucket names
for bucket in s3.buckets.all():
    print bucket.name

umount /path/to/local_directory_name