Ctrlk
Sign up

Use s3cmd with Object Storage

The s3cmd tool is a widely used tool for interacting with S3-compatible buckets.

S3cmd is a command-line utility for any S3-compatible object storage service. It is one of the most common tools for working with S3-compatible storage. It can be used to create new buckets, upload and download files, delete files, and modify permissions (ACL).

This is an introduction to how s3cmd can be installed on your computer, as well as an overview of what can be achieved with the program.

Install s3cmd

The following commands will install s3cmd on various common operating systems. Additional methods of installing s3cmd can be found within the s3cmd GitHub repository under the Installation of s3cmd package file.

To install s3cmd on a Mac, you can use Homebrew:

Command
brew install s3cmd

On macOS, s3cmd may fail to install if you do not have Xcode Command Line Tools installed. If that is the case, run the following command:

xcode-select --install

Configure s3cmd

After installing s3cmd, it must be configured to work with the object storage instance in your Glesys account.

  1. Run the following command to start the configuration process:

Command
s3cmd --configure

  1. This command will prompt you with a series of questions. Answer them based on the recommendations below:

  • Access Key: Enter the access key you wish to use. See Manage access keys.

  • Secret Key: Enter the secret key that corresponds to the access key. This was displayed once when generating the access key.

  • Default Region: Enter the data center where you have chosen to place your instance. For Falkenberg, this is dc-fbg1, and for Stockholm, it is dc-sto1.

  • S3 Endpoint: Enter the endpoint hostname provided when the instance was created. For Falkenberg, this is objects.dc-fbg1.glesys.net; for Stockholm, it is objects.dc-sto1.glesys.net.

  • DNS-style bucket+hostname:port template for accessing a bucket: Here, specify how to access your buckets using the bucket+hostname naming convention. For Falkenberg, enter (%bucket).objects.dc-fbg1.glesys.net. For Stockholm, enter (%bucket).objects.dc-sto1.glesys.net.

  • Encryption password: Input a password for encrypting files before uploading them (optional).

  • Path to GPG program: Enter the path to your GPG encryption program (optional).

  • Use HTTPS protocol: Yes

  • HTTP Proxy server name: (Leave blank)

  • HTTP Proxy server port: (Leave blank)

You may leave the encryption password and the path to the GPG program blank if you do not intend to use encryption.

However, if you want to encrypt the files with GPG before uploading, you should enter a password for the files here. This enables you to encrypt files during upload using the -e flag. They will be automatically decrypted during download. This is ideal for backups, as no one can read the files without the password—not even Glesys. For encryption to work, you need to have GPG installed. The path suggested by s3cmd for gpg is usually correct.

Remember not to encrypt files intended for public use, such as video files on a website. These should be accessible to everyone.

  1. When the prompt appears to test access with the supplied credentials, enter y to test the login.

  2. When the prompt appears to save your settings, enter y. A configuration file named .s3cfg is created within your home directory.

Interact with buckets

When uploading or downloading files, listing files, or making other requests to a bucket, you use the format s3://bucket-name. In this example, the automatically created bucket is named patient-credit.

Start by displaying information about the automatically created bucket. You received the name when you created the instance.

This will output something like this:

List buckets

List all buckets within the object storage instance.

Command: s3cmd ls

Create a bucket

Create a bucket with a specified bucket name.

Command: s3cmd mb s3://[bucket-name], replacing [bucket-name] with the desired name for the bucket.

Example: Create a bucket with the name example-bucket:

Delete a bucket

Delete a bucket with the specified name.

Command: s3cmd rb s3://[bucket-name], replacing [bucket-name] with the name of the bucket you want to delete.

Example: Delete the bucket with the name example-bucket:

To delete a bucket with files, include the --recursive (or -r) option and the --force (or -f) option. Use caution when running this command:

Check disk usage by bucket

Find the amount of content within a bucket (in bytes) and the number of objects.

Command: s3cmd du s3://[bucket-name], replacing [bucket-name] with the specific bucket you want to check for usage.

Example: Show the amount of content within a bucket (in bytes) and the number of objects:

Interact with objects

Upload an object

Command: s3cmd put [file] s3://[bucket-name]/[path], replacing [file] with the file's name and path you wish to upload, [bucket-name] with the bucket's name, and [path] with the optional directory within the bucket.

Example: Upload the file file.txt to the bucket called example-bucket:

Additional command options:

  • -P: Makes the object publicly accessible. This allows anyone with the URL to access the object. Once uploaded successfully, s3cmd will output the public URL.

  • -e: Encrypts the object (if you have configured the correct s3cmd options to enable encryption).

Uploading or renaming objects with non-standard special characters and unusual ASCII or Unicode characters may cause issues. This includes the following characters: : " ' < > & + =

Download an object or directory

Command: s3cmd get s3://[bucket-name]/[path], replacing [bucket-name] with the bucket's name and [path] with the complete path and optional filename of the file or directory you want to download.

Example: Download the file file.txt from the bucket called example-bucket:

Delete an object or directory

Command: s3cmd rm s3://[bucket-name]/[path], replacing [bucket-name] with the bucket's name and [path] with the complete path and optional filename of the file or directory you want to delete.

Example: Delete the file.txt file on the bucket called example-bucket:

To delete all files in a bucket, include the --recursive (or -r) option along with the --force (or -f) option. Use caution when executing this command:

Make files publicly available

Making files publicly available means that anyone on the internet can see and download the files. This can, for example, be used to host video files for a website.

Verify that files are private by default

By default, all files that are uploaded are private, unless the bucket's policy or ACL has been changed. To verify this, first upload a small text file named, for example, test.txt:

Now, let's list all available information about the file:

The output will look like this. Notice there is only one ACL line, which is the owner (os-bab6e) that has full control. There's no anonymous access.

You can also try accessing the file with cURL, which should not work:

You should now see an output similar to this:

Here, you were denied access to the file, which is entirely correct.

Make a specific file public

For the file to become public, you must explicitly set it as public (unless other ACLs or policies have been modified for the bucket).

Try accessing the file using cURL again:

You should now see the content of the text file, for example:

If you run the s3cmd info s3://example-bucket/test.txt command again, you should see that there are now two ACL lines, one of which lists anon for anonymous with READ access:

Make all files in a directory public

It's also possible to make all files in a directory public using the --recursive option. For example, if you want to make all files in the directory my-files public:

You should now see all files in that directory being made public.

Permissions and access controls

It's possible to set up policies for buckets. These policies are applied using s3cmd. For more information on how to write policy files and what is supported, refer to Ceph's S3 documentation on policy.

Apply a bucket policy

Command: s3cmd setpolicy [policy-file] s3://[bucket-name], replacing [bucket-name] with the bucket's name and [policy-file] with the filename and path of the bucket policy file.

Example: Implement the bucket policies from a file called policy.json for the bucket named example-bucket:

To ensure it has been applied correctly, you can use the info command:

You can expect output similar to this:

Use multiple configuration files with s3cmd

If you have multiple Object Storage instances, it could be time-consuming to reconfigure s3cmd each time you want to switch instances. Fortunately, with s3cmd, it's possible to specify a configuration file for each command, enabling you to work with different instances quickly.

Before running s3cmd --configure, save your current .s3cfg file with a new name, for example, .s3cfg-os-bab6e, after an instance's name.

Now, configure s3cmd for the new instance with s3cmd --configure.

After configuring the new instance, you can easily run commands against the previous instance by using the -c option (or --config), for example:

To run a command against the new instance, leave out the -c .s3cfg-os-bab6e part.

PreviousClients and toolingNextBackups using Restic

Last updated

Was this helpful?