> For the complete documentation index, see [llms.txt](https://docs.glesys.com/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.glesys.com/products/storage/object-storage/how-tos/clients-and-tooling/use-s3cmd-with-object-storage.md).
# Use s3cmd with Object Storage
***
[S3cmd](https://s3tools.org/s3cmd) is a command-line utility for any S3-compatible object storage service. It is one of the most common tools for working with S3-compatible storage. It can be used to create new buckets, upload and download files, delete files, and modify permissions (ACL).
This is an introduction to how `s3cmd` can be installed on your computer, as well as an overview of what can be achieved with the program.
## Install s3cmd
The following commands will install `s3cmd` on various common operating systems. Additional methods of installing s3cmd can be found within the s3cmd GitHub repository under the [Installation of s3cmd package](https://github.com/s3tools/s3cmd/blob/master/INSTALL.md) file.
{% tabs %}
{% tab title="macOS" %}
To install s3cmd on a Mac, you can use [Homebrew](https://brew.sh/):
{% code title="Command" %}
```
brew install s3cmd
```
{% endcode %}
{% hint style="info" %}
On macOS, s3cmd may fail to install if you do not have Xcode Command Line Tools installed. If that is the case, run the following command:
`xcode-select --install`
{% endhint %}
{% endtab %}
{% tab title="Debian and Ubuntu" %}
The tool can be easily installed using the package manager:
{% code title="Command" %}
```
sudo apt install s3cmd
```
{% endcode %}
{% endtab %}
{% tab title="Fedora and AlmaLinux" %}
In Fedora and AlmaLinux, the tool is also available through the package manager. However, we first need to enable the EPEL repository:
{% code title="Command" %}
```
sudo dnf install epel-release
```
{% endcode %}
After that, we install `s3cmd` using the package manager:
{% code title="Command" %}
```
sudo dnf install s3cmd
```
{% endcode %}
{% endtab %}
{% tab title="Microsoft Windows" %}
On Microsoft Windows 10 or 11, the most straightforward approach is first to install Python 3 via the Microsoft Store and then use `pip` to install `s3cmd`.
1. Open the Microsoft Store and search for *Python*. Select the latest version of Python 3, then click the **Get** button to install Python.
2. Once the installation is complete, open the Command Prompt by searching for **cmd** in the Start menu. Click on **Command Prompt** when it appears.
3. Create a directory to install `s3cmd`, for example:
{% code title="Command" %}
```
mkdir s3cmd
```
{% endcode %}
4. Create a virtual Python environment:
{% code title="Multiple commands" %}
```
cd s3cmd
python3 -m venv env
```
{% endcode %}
5. Now, activate the virtual Python environment and install `s3cmd` within it:
{% code title="Multiple commands. Python venv prompt shown as '(env)'" %}
```
env\Scripts\activate.bat
(env) pip install s3cmd
```
{% endcode %}
6. Let's test the command:
{% code title="Command. Python venv prompt shown as '(env)'" %}
```
(env) python env\Scripts\s3cmd
```
{% endcode %}
{% code title="Output" %}
```
ERROR: C:\Users\glesys\AppData\Roaming\s3cmd.ini: None
ERROR: Configuration file not available.
ERROR: Consider using --configure parameter to create one.
```
{% endcode %}
The command works!
7. To simplify working with the command, you can create an alias for it:
{% code title="Command. Python venv prompt shown as '(env)'" %}
```
(env) DOSKEY s3cmd=python C:\Users\glesys\s3cmd\env\Scripts\s3cmd
```
{% endcode %}
Now, it is sufficient to type `s3cmd` to execute the command. However, please note that the alias needs to be added again each time the Command Prompt is restarted.
{% endtab %}
{% endtabs %}
## Configure s3cmd
After installing `s3cmd`, it must be configured to work with the object storage instance in your Glesys account.
1. Run the following command to start the configuration process:
{% code title="Command" %}
```
s3cmd --configure
```
{% endcode %}
2. This command will prompt you with a series of questions. Answer them based on the recommendations below:
* **Access Key:** Enter the access key you wish to use. See [Manage access keys](/products/storage/object-storage/how-tos/manage-access-keys.md).
* **Secret Key:** Enter the secret key that corresponds to the access key. This was displayed once when generating the access key.
* **Default Region:** Enter the data center where you have chosen to place your instance. For Falkenberg, this is `dc-fbg1`, and for Stockholm, it is `dc-sto1`.
* **S3 Endpoint:** Enter the endpoint hostname provided when the instance was created. For Falkenberg, this is `objects.dc-fbg1.glesys.net`; for Stockholm, it is `objects.dc-sto1.glesys.net`.
* **DNS-style bucket+hostname:port template for accessing a bucket:** Here, specify how to access your buckets using the bucket+hostname naming convention. For Falkenberg, enter `(%bucket).objects.dc-fbg1.glesys.net`. For Stockholm, enter `(%bucket).objects.dc-sto1.glesys.net`.
* **Encryption password:** Input a password for encrypting files before uploading them (optional).
* **Path to GPG program:** Enter the path to your GPG encryption program (optional).
* **Use HTTPS protocol:** `Yes`
* **HTTP Proxy server name:** (Leave blank)
* **HTTP Proxy server port:** (Leave blank)
{% hint style="info" %}
You may leave the **encryption password** and the **path to the GPG program** blank if you do not intend to use encryption.
However, if you want to encrypt the files with GPG before uploading, you should enter a password for the files here. This enables you to encrypt files during upload using the `-e` flag. They will be automatically decrypted during download. This is ideal for backups, as no one can read the files without the password—not even Glesys. For encryption to work, you need to have GPG installed. The path suggested by `s3cmd` for `gpg` is usually correct.
Remember not to encrypt files intended for public use, such as video files on a website. These should be accessible to everyone.
{% endhint %}
3. When the prompt appears to test access with the supplied credentials, enter `y` to test the login.
4. When the prompt appears to save your settings, enter `y`. A configuration file named `.s3cfg` is created within your home directory.
Test access with supplied credentials? [Y/n] y
Please wait, attempting to list all buckets...
Success. Your access key and secret key worked fine :-)
Now verifying that encryption works...
Not configured. Never mind.
Save settings? [y/N] y
Configuration saved to '/home/glesys/.s3cfg'
## Interact with buckets
When uploading or downloading files, listing files, or making other requests to a bucket, you use the format *s3://bucket-name*. In this example, the automatically created bucket is named *patient-credit*.
Start by displaying information about the automatically created bucket. You received the name when you created the instance.
{% code title="Command" %}
```
s3cmd info s3://patient-credit
```
{% endcode %}
This will output something like this:
{% code title="Output" %}
```
s3://patient-credit/ (bucket):
Location: dc-fbg1
Payer: BucketOwner
Versioning:none
Expiration Rule: none
Policy: none
CORS: none
ACL: os-bab6e: FULL_CONTROL
```
{% endcode %}
#### List buckets
List all buckets within the object storage instance.
**Command:** `s3cmd ls`
#### Create a bucket
Create a bucket with a specified bucket name.
**Command:** `s3cmd mb s3://[bucket-name]`, replacing *\[bucket-name]* with the desired name for the bucket.
**Example:** Create a bucket with the name *example-bucket*:
{% code title="Command" %}
```
s3cmd mb s3://example-bucket
```
{% endcode %}
#### Delete a bucket
Delete a bucket with the specified name.
**Command:** `s3cmd rb s3://[bucket-name]`, replacing *\[bucket-name]* with the name of the bucket you want to delete.
**Example:** Delete the bucket with the name *example-bucket*:
{% code title="Command" %}
```
s3cmd rb s3://example-bucket
```
{% endcode %}
To delete a bucket with files, include the `--recursive` (or `-r`) option and the `--force` (or `-f`) option. Use caution when running this command:
{% code title="Command" %}
```
s3cmd rb -r -f s3://example-bucket/
```
{% endcode %}
#### Check disk usage by bucket
Find the amount of content within a bucket (in bytes) and the number of objects.
**Command:** `s3cmd du s3://[bucket-name]`, replacing *\[bucket-name]* with the specific bucket you want to check for usage.
**Example:** Show the amount of content within a bucket (in bytes) and the number of objects:
{% code title="Command" %}
```
s3cmd du s3://example-bucket
```
{% endcode %}
## Interact with objects
#### Upload an object
**Command:** `s3cmd put [file] s3://[bucket-name]/[path]`, replacing *\[file]* with the file's name and path you wish to upload, *\[bucket-name]* with the bucket's name, and *\[path]* with the optional directory within the bucket.
**Example:** Upload the file *file.txt* to the bucket called *example-bucket*:
{% code title="Command" %}
```
s3cmd put file.txt s3://example-bucket/
```
{% endcode %}
**Additional command options:**
* `-P`: Makes the object publicly accessible. This allows anyone with the URL to access the object. Once uploaded successfully, `s3cmd` will output the public URL.
* `-e`: Encrypts the object (if you have configured the correct `s3cmd` options to enable encryption).
{% hint style="info" %}
Uploading or renaming objects with non-standard special characters and unusual ASCII or Unicode characters may cause issues. This includes the following characters:\
`: " ' < > & + =`
{% endhint %}
#### Download an object or directory
**Command:** `s3cmd get s3://[bucket-name]/[path]`, replacing *\[bucket-name]* with the bucket's name and *\[path]* with the complete path and optional filename of the file or directory you want to download.
**Example:** Download the file *file.txt* from the bucket called *example-bucket*:
{% code title="Command" %}
```
s3cmd get s3://example-bucket/file.txt
```
{% endcode %}
#### Delete an object or directory
**Command:** `s3cmd rm s3://[bucket-name]/[path]`, replacing *\[bucket-name]* with the bucket's name and *\[path]* with the complete path and optional filename of the file or directory you want to delete.
**Example:** Delete the *file.txt* file on the bucket called *example-bucket*:
{% code title="Command" %}
```
s3cmd rm s3://example-bucket/file.txt
```
{% endcode %}
To delete all files in a bucket, include the `--recursive` (or `-r`) option along with the `--force` (or `-f`) option. Use caution when executing this command:
{% code title="Command" %}
```
s3cmd rm -r -f s3://example-bucket/
```
{% endcode %}
## Make files publicly available
Making files publicly available means that anyone on the internet can see and download the files. This can, for example, be used to host video files for a website.
### Verify that files are private by default
By default, all files that are uploaded are private, unless the bucket's policy or ACL has been changed. To verify this, first upload a small text file named, for example, *test.txt:*
{% code title="Command" %}
```terminal
s3cmd put test.txt s3://example-bucket
```
{% endcode %}
Now, let's list all available information about the file:
{% code title="Command" %}
```terminal
s3cmd info s3://example-bucket/test.txt
```
{% endcode %}
The output will look like this. Notice there is only one ACL line, which is the owner (os-bab6e) that has full control. There's no anonymous access.
{% code title="Output" %}
```
s3://example-bucket/test.txt (object):
File size: 6
Last mod: Sat, 25 Nov 2023 15:12:29 GMT
MIME type: text/plain
Storage: STANDARD
MD5 sum: d41d8cd98f00b204e9800998ecf8427e
SSE: none
Policy: none
CORS: none
ACL: os-bab6e: FULL_CONTROL
x-amz-meta-s3cmd-attrs: atime:1700856660/ctime:1700856657/gid:1000/gname:glesys/md5:ab726f7349c380669bc81c87bc9d3696/mode:33216/mtime:1700852790/uid:1000/uname:glesys
```
{% endcode %}
You can also try accessing the file with cURL, which should not work:
{% code title="Command" %}
```terminal
curl https://objects.dc-fbg1.glesys.net/example-bucket/test.txt
```
{% endcode %}
You should now see an output similar to this:
{% code title="Output" %}
```
AccessDenied
example-bucket
tx00000876ea47fc42b21d0-006562101a-e740b7-default
e740b7-default-default
```
{% endcode %}
Here, you were denied access to the file, which is entirely correct.
### Make a specific file public
For the file to become public, you must explicitly set it as public (unless other ACLs or policies have been modified for the bucket).
{% code title="Command" %}
```
s3cmd setacl s3://example-bucket/test.txt --acl-public
```
{% endcode %}
Try accessing the file using cURL again:
{% code title="Command" %}
```terminal
curl https://objects.dc-fbg1.glesys.net/example-bucket/test.txt
```
{% endcode %}
You should now see the content of the text file, for example:
{% code title="Output" %}
```
Hello
```
{% endcode %}
If you run the `s3cmd info s3://example-bucket/test.txt` command again, you should see that there are now two ACL lines, one of which lists `anon` for anonymous with `READ` access:
{% code title="Output" %}
```
s3://example-bucket/test.txt (object):
File size: 6
Last mod: Sat, 25 Nov 2023 15:12:29 GMT
MIME type: text/plain
Storage: STANDARD
MD5 sum: d41d8cd98f00b204e9800998ecf8427e
SSE: none
Policy: none
CORS: none
ACL: *anon*: READ
ACL: os-bab6e: FULL_CONTROL
x-amz-meta-s3cmd-attrs: atime:1700856660/ctime:1700856657/gid:1000/gname:glesys/md5:ab726f7349c380669bc81c87bc9d3696/mode:33216/mtime:1700852790/uid:1000/uname:glesys
```
{% endcode %}
### Make all files in a directory public
It's also possible to make all files in a directory public using the `--recursive` option. For example, if you want to make all files in the directory *my-files* public:
{% code title="Command" %}
```
s3cmd setacl s3://example-bucket/my-files/ --acl-public --recursive
```
{% endcode %}
You should now see all files in that directory being made public.
{% code title="Output" %}
```
s3://broken-fire/my-files/g.txt: ACL set to Public [1 of 2]
s3://broken-fire/my-files/h.txt: ACL set to Public [2 of 2]
```
{% endcode %}
## Permissions and access controls
It's possible to set up policies for buckets. These policies are applied using `s3cmd`. For more information on how to write policy files and what is supported, refer to [Ceph's S3 documentation on policy](https://docs.ceph.com/en/quincy/radosgw/bucketpolicy/).
#### Apply a bucket policy
**Command:** `s3cmd setpolicy [policy-file] s3://[bucket-name]`, replacing *\[bucket-name]* with the bucket's name and *\[policy-file]* with the filename and path of the bucket policy file.
**Example:** Implement the bucket policies from a file called *policy.json* for the bucket named *example-bucket*:
{% code title="Command" %}
```
s3cmd setpolicy policy.json s3://example-bucket
```
{% endcode %}
To ensure it has been applied correctly, you can use the info command:
{% code title="Command" %}
```
s3cmd info s3://example-bucket
```
{% endcode %}
You can expect output similar to this:
{% code title="Output" %}
```
s3://example-bucket/ (bucket):
Location: dc-fbg1
Payer: BucketOwner
Ownership: none
Versioning:none
Expiration rule: none
Block Public Access: none
Policy: {
"Version": "2012-10-17",
"Statement": [{
"Effect": "Allow",
"Principal": "*",
"Action": [
"s3:GetObject"
],
"Resource": [
"arn:aws:s3:::example-bucket/assets/*"
]
}]
}
CORS: none
ACL: os-bab6e: FULL_CONTROL
```
{% endcode %}
## Use multiple configuration files with s3cmd
If you have multiple Object Storage instances, it could be time-consuming to reconfigure `s3cmd` each time you want to switch instances. Fortunately, with `s3cmd`, it's possible to specify a configuration file for each command, enabling you to work with different instances quickly.
Before running `s3cmd --configure`, save your current `.s3cfg` file with a new name, for example, `.s3cfg-os-bab6e`, after an instance's name.
{% code title="Multiple commands" %}
```
cd
cp .s3cfg .s3cfg-os-bab6e
```
{% endcode %}
Now, configure `s3cmd` for the new instance with `s3cmd --configure`.
After configuring the new instance, you can easily run commands against the previous instance by using the `-c` option (or `--config`), for example:
{% code title="Command" %}
```
s3cmd -c .s3cfg-os-bab6e ls s3://example-bucket
```
{% endcode %}
To run a command against the new instance, leave out the `-c .s3cfg-os-bab6e` part.
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.glesys.com/products/storage/object-storage/how-tos/clients-and-tooling/use-s3cmd-with-object-storage.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.