How Can I Control Space Being Used On My Object Storage?

With S3-compatible Object Storage in general, you may sometimes find that more space is reported as being used than you actually have. In practice, this happens for the following reasons

• Aborted multipart uploads
• Versioning is enabled

The next sections cover these topics in more detail. The following assumes that aws cli is properly installed and configured.

What are aborted multipart uploads?

Typically, uploads of larger files are broken into smaller chunks, say 10 MB, and uploaded one at a time. These file pieces are not displayed by default, but they do take up disk space. Multi-part uploads are managed by buckets.

How can I find aborted multipart uploads?

You can run this API request to find aborted multipart uploads:

aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api list-multipart-uploads --bucket mybucket

This will return a JSON output containing files that are currently being uploaded or that have been canceled. If you get nothing back, then there are no multipart uploads in progress.

How do I delete aborted multipart uploads?

To cancel a single upload, copy the key (filename) and upload id and run the following API:

aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api abort-multipart-upload --bucket mybucket --key filename.ext --upload-id 2~Pw5t8pu1uBnxu_qMyJRnjnZKYvav41c

You can run the script below to automate the process of deleting any multipart upload that is in progress or that has been aborted:

BUCKETNAME=mybucket
aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api list-multipart-uploads --bucket $BUCKETNAME \
| jq -r '.Uploads[] | "--key \"\(.Key)\" --upload-id \(.UploadId)"' \
| while read -r param; do
eval "aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api abort-multipart-upload --bucket $BUCKETNAME $param";
done

What is enabled versioning?

Enabled versioning holds also storage space as there are many versions of a file. Those files are usually not visible.

How do I delete an old version of a file?

You can run the following API request to delete old versions of a file that may be taking up space:

aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api list-object-versions --bucket mybucket
aws --profile eu2 --region default --endpoint-url https://eu2.contabostorage.com s3api delete-object --bucket mybucket --key file-to-delete.ext --version-id 3424sadfds2344