Skip to main content

Object storage

Overview​

Object storage is large-capacity storage that exposes an S3-compatible API. It is used for dataset storage, checkpoint backups, and sharing data between VMs.

S3 endpoint

Right after creating an object storage user, check the exact S3 endpoint URL for your environment on the Access info screen that appears. The endpoint can differ by environment.

Creating a bucket and a user​

  1. Go to Object Storage > Buckets and click Create Bucket.
  2. Under Object Storage > Users, click Create User.
    • Key generation: Auto-generate (recommended) or Enter manually
    • For manual entry, the access key must be at least 20 alphanumeric characters and the secret key at least 40
  3. Save the following immediately from the Access info screen shown after the user is created.
    • Access Key
    • Secret Key
    • S3 endpoint: https://s3.elice.cloud
  4. On the user detail page, grant the user permissions on each bucket (read, write, etc.).
Storing the secret key

The secret key is masked on the user detail page, but you can unmask it any time with the toggle. Once leaked, it gives full access to whoever has it, so keep it somewhere safe (secret manager, env file, etc.) and never share it. If you suspect a leak, click Regenerate keys to rotate them.

aws-cli integration​

Install​

# Linux
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
unzip awscliv2.zip && sudo ./aws/install

Configure credentials​

aws configure
# AWS Access Key ID: YOUR_ACCESS_KEY
# AWS Secret Access Key: YOUR_SECRET_KEY
# Default region name: (Enter)
# Default output format: (Enter)

Add the endpoint to ~/.aws/config:

[default]
services = s3

[services s3]
s3 =
    endpoint_url = https://s3.elice.cloud

Common commands​

# Upload a file
aws s3 cp ./dataset.tar.gz s3://YOUR_BUCKET/dataset.tar.gz

# Upload a directory
aws s3 sync ./checkpoints s3://YOUR_BUCKET/checkpoints

# List
aws s3 ls s3://YOUR_BUCKET/

# Download
aws s3 cp s3://YOUR_BUCKET/dataset.tar.gz ./

rclone integration​

Install​

sudo -v ; curl https://rclone.org/install.sh | sudo bash

Configure​

rclone config
# n) New remote
# name> elicecloud
# Storage> 4  (Amazon S3)
# provider> 4  (AWS)
# env_auth> (Enter)
# access_key_id> YOUR_ACCESS_KEY
# secret_access_key> YOUR_SECRET_KEY
# region> (Enter)
# endpoint> https://s3.elice.cloud
# Press Enter for everything else

Common commands​

# Upload
rclone copy ./dataset elicecloud:YOUR_BUCKET/dataset --s3-no-check-bucket --progress

# Download
rclone copy elicecloud:YOUR_BUCKET/dataset ./dataset --s3-no-check-bucket --progress

# Sync (files missing in source are deleted from destination)
rclone sync ./checkpoints elicecloud:YOUR_BUCKET/checkpoints --s3-no-check-bucket --dry-run
rclone sync ./checkpoints elicecloud:YOUR_BUCKET/checkpoints --s3-no-check-bucket

# Usage
rclone size elicecloud:YOUR_BUCKET
--s3-no-check-bucket is required

ECI object storage does not support bucket creation through the API. When using rclone, always pass --s3-no-check-bucket.

Mounting a bucket as a directory (rclone mount)​

You can mount object storage as a local directory.

mkdir -p /mnt/object-storage

# Mount in the background
rclone mount elicecloud:YOUR_BUCKET /mnt/object-storage --daemon

# Unmount
umount /mnt/object-storage
Things to keep in mind when mounting
  • Network latency makes it slower than a local disk.
  • The capacity shown by df -h is a synthetic value, not actual usage. Check actual usage with rclone size.
  • Single files larger than 48 GiB need --s3-chunk-size=128M.

Next steps​