Figure.1
| PolarSPARC |
Hands-on with MinIO
| Bhaskar S | 10/29/2022 |
Overview
The distributed object storage Simple Storage Service (or S3 for short) was made popular by Amazon in the late 2000s in their cloud offering. Ever since, it has gained traction even in the private cloud environments of the Enterprises.
The idea of the distributed object store S3 is to provide highly available, durable, scable, and performant object storage, where objects (documents, images, videos, etc) are stored into containers called Buckets.
Ever wondered if one can have such a storage setup in one's home labs ???
The answer is YES and this is where the open source distributed object store MinIO comes to the rescue !!!
In other words, MinIO is a highly performant, open source distributed object storage solution, which is compatible to the S3 offering from Amazon, with support for all the core features of S3.
Installation and Setup
The installation is on a Ubuntu 22.04 LTS based Linux desktop.
Ensure Docker is installed and setup. Else, refer to the article Introduction to Docker for help.
Assuming that we are logged in as polarsparc and the current working directory is the home directory /home/polarsparc, we will setup a directory structure by executing the following commands in a terminal window:
$ mkdir -p minio/data
$ mkdir -p minio/client
Now, change the current working directory to the directory /home/polarsparc/minio. In the following paragraphs we will refer to this location as $MINIO_HOME.
For our exploration, we will be downloading and using the following two docker images:
MinIO Server - bitnami/minio:2022.10.24
MinIO Client - bitnami/minio-client:2022.10.20
To pull and download the docker image for MinIO server, execute the following command:
$ docker pull bitnami/minio:2022.10.24
The following should be the typical output:
2022.10.24: Pulling from bitnami/minio 1d8866550bdd: Already exists f44865fd5b5b: Pull complete Digest: sha256:945f9a5c942b25a85f8edc6e222ffd752870bc014304aa4ae45a163160fbe7e8 Status: Downloaded newer image for bitnami/minio:2022.10.24 docker.io/bitnami/minio:2022.10.24
Once the download is complete, execute the following command to check everything was ok with the image for MinIO:
$ docker run --rm --name minio bitnami/minio:2022.10.24 /opt/bitnami/minio/bin/minio --help
The following would be the typical output:
23:31:14.68 23:31:14.69 Welcome to the Bitnami minio container 23:31:14.69 Subscribe to project updates by watching https://github.com/bitnami/containers 23:31:14.69 Submit issues and feature requests at https://github.com/bitnami/containers/issues 23:31:14.69 NAME: minio - High Performance Object Storage DESCRIPTION: Build high performance data infrastructure for machine learning, analytics and application data workloads with MinIO USAGE: minio [FLAGS] COMMAND [ARGS...] COMMANDS: server start object storage server gateway start object storage gateway FLAGS: --certs-dir value, -S value path to certs directory (default: "/.minio/certs") --quiet disable startup and info messages --anonymous hide sensitive information from logging --json output logs in JSON format --help, -h show help --version, -v print the version VERSION: DEVELOPMENT.2022-10-24T18-35-07Z
To pull and download the docker image for MinIO client, execute the following command:
$ docker pull bitnami/minio-client:2022.10.20
The following should be the typical output:
2022.10.20: Pulling from bitnami/minio-client 1d8866550bdd: Already exists b85eadab25a9: Pull complete Digest: sha256:c074eb6bf78944d8e9c5c6f4d37136ae0b41b7f747ae106207ce903514ff33a1 Status: Downloaded newer image for bitnami/minio-client:2022.10.20 docker.io/bitnami/minio-client:2022.10.20
Once the download is complete, execute the following command to check everything was ok with the image for MinIO:
$ docker run --rm --name mc bitnami/minio-client:2022.10.20 --help
The following would be the typical output:
12:54:18.37 12:54:18.38 Welcome to the Bitnami minio-client container 12:54:18.38 Subscribe to project updates by watching https://github.com/bitnami/containers 12:54:18.38 Submit issues and feature requests at https://github.com/bitnami/containers/issues 12:54:18.38 12:54:18.38 INFO ==> ** Starting MinIO Client setup ** 12:54:18.39 INFO ==> ** MinIO Client setup finished! ** NAME: mc - MinIO Client for object storage and filesystems. USAGE: mc [FLAGS] COMMAND [COMMAND FLAGS | -h] [ARGUMENTS...] COMMANDS: alias manage server credentials in configuration file ls list buckets and objects mb make a bucket rb remove a bucket cp copy objects mv move objects rm remove object(s) mirror synchronize object(s) to a remote site cat display object contents head display first 'n' lines of an object pipe stream STDIN to an object find search for objects sql run sql queries on objects stat show object metadata tree list buckets and objects in a tree format du summarize disk usage recursively retention set retention for object(s) legalhold manage legal hold for object(s) support support related commands license license related commands share generate URL for temporary access to an object version manage bucket versioning ilm manage bucket lifecycle encrypt manage bucket encryption config event manage object notifications watch listen for object notification events undo undo PUT/DELETE operations anonymous manage anonymous access to buckets and objects tag manage tags for bucket and object(s) diff list differences in object name, size, and date between two buckets replicate configure server side bucket replication admin manage MinIO servers update update mc to latest release ready checks if the cluster is ready or not ping perform liveness check od measure single stream upload and download batch manage batch jobs GLOBAL FLAGS: --autocompletion install auto-completion for your shell --config-dir value, -C value path to configuration folder (default: "/.mc") --quiet, -q disable progress bar display --no-color disable color theme --json enable JSON lines formatted output --debug enable debug output --insecure disable SSL certificate verification --help, -h show help --version, -v print the version TIP: Use 'mc --autocompletion' to enable shell autocompletion COPYRIGHT: Copyright (c) 2015-2022 MinIO, Inc. LICENSE: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html>
The following is the MinIO client configuration file (in JSON format) located under $MINIO_HOME/client:
{
"version": "10",
"aliases": {
"local": {
"url": "http://172.17.0.2:9000",
"accessKey": "minio_adm",
"secretKey": "s3cr3t_4ce",
"api": "S3v4",
"path": "auto"
}
}
}
Notice the definition for the MinIO target alias local in the above client configuration file. This is the target name that will be used in the MinIO client commands later in this article.
Hands-on with MinIO
For the demonstration, we will use MinIO in a Single Node mode (versus deploying in a multi node cluster mode).
To run the MinIO object storage server, execute the following command in the terminal window:
$ docker run --rm --name minio -e "MINIO_ROOT_USER=minio_adm" -e "MINIO_ROOT_PASSWORD=s3cr3t_4ce" -u $(id -u $USER):$(id -g $USER) -p 9000:9000 -p 9001:9001 -v $MINIO_HOME/data:/mnt/data bitnami/minio:2022.10.24 /opt/bitnami/minio/bin/minio server /mnt/data --console-address ":9001"
The following would be the typical output:
23:32:20.83 23:32:20.83 Welcome to the Bitnami minio container 23:32:20.83 Subscribe to project updates by watching https://github.com/bitnami/containers 23:32:20.83 Submit issues and feature requests at https://github.com/bitnami/containers/issues 23:32:20.83 Formatting 1st pool, 1 set(s), 1 drives per set. WARNING: Host local has more than 0 drives of set. A host failure will result in data becoming unavailable. MinIO Object Storage Server Copyright: 2015-2022 MinIO, Inc. License: GNU AGPLv3 <https://www.gnu.org/licenses/agpl-3.0.html> Version: DEVELOPMENT.2022-10-24T18-35-07Z (go1.18.7 linux/amd64) Status: 1 Online, 0 Offline. API: http://172.17.0.2:9000 http://127.0.0.1:9000 Console: http://172.17.0.2:9001 http://127.0.0.1:9001 Documentation: https://min.io/docs/minio/linux/index.html
Launch a browser and open the console url http://localhost:9001. This will oprn the MinIO web console login as shown in the illustration below:
Enter the user name minio_adm, the password s3cr3t_4ce, and then click on the Login button as shown in the illustration below:
On successful login, one is presented with a screen for creating Buckets as shown in the illustration below:
Click on the Create Bucket + button as shown in the illustration below:
One is presented with the Create Bucket screen as shown in the illustration below:
Click on the View Bucket Naming Rules drop-down as shown in the illustration below:
One is presented with the Create Bucket screen as shown in the illustration below:
Enter the S3 bucket name b-test-1, toggle the Versioning switch, and then click on the Create Bucket button as shown in the illustration below:
One is presented with the b-test-1 Bucket screen as shown in the illustration below:
Click on the Configure Bucket icon as shown in the illustration below:
One is presented with the Bucket Summary screen (with options to configure the bucket) as shown in the illustration below:
Click on the Browse Bucket icon as shown in the illustration below:
We will land back on the screen as shown in the illustration of Figure.9 above.
Click on the Upload button to upload any object as shown in the illustration below:
One is presented with the option to either Upload File or Upload Folder as shown in the illustration below:
Click on the Upload File option to upload any file object (document or image) as shown in the illustration below:
One is presented with the desktop Open Files dialog to choose a file to upload. In this demo, we choose a sample JPEG image to upload.
When the upload completes, one is presented with a screen as shown in the illustration below:
Click on the Exit icon of the Downloads/Uploads dialog box as shown in the illustration below:
One is presented with Bucket Browser screen as shown in the illustration below:
Now, we will switch gears to use the command-line MinIO client (using the docker minio-client).
Before we get started, open a new terminal window (referred to as the client terminal window) for executing the various MinIO client commands.
To list all the buckets and objects on the MinIO server (referred by alias local), execute the following command in the client terminal window:
$ docker run --rm --name mc -u $(id -u $USER):$(id -g $USER) -v $MINIO_HOME/client:/.mc bitnami/minio-client:2022.10.20 ls --versions local
The following would be the typical output:
15:29:10.44 15:29:10.44 Welcome to the Bitnami minio-client container 15:29:10.44 Subscribe to project updates by watching https://github.com/bitnami/containers 15:29:10.44 Submit issues and feature requests at https://github.com/bitnami/containers/issues 15:29:10.44 15:29:10.44 INFO ==> ** Starting MinIO Client setup ** 15:29:10.45 INFO ==> ** MinIO Client setup finished! ** [2022-10-29 00:10:52 UTC] 0B b-test-1/ [2022-10-29 00:32:59 UTC] 30KiB STANDARD ac05b181-d5be-43d2-bf3a-6f3c1440126d v1 PUT b-test-1/jpeg-sample.jpg
To create a new bucket named b-test-2 with versioning enabled, execute the following command in the client terminal window:
$ docker run --rm --name mc -u $(id -u $USER):$(id -g $USER) -v $MINIO_HOME/client:/.mc bitnami/minio-client:2022.10.20 mb --with-versioning local/b-test-2
The following would be the typical output:
15:51:00.98 15:51:00.98 Welcome to the Bitnami minio-client container 15:51:00.98 Subscribe to project updates by watching https://github.com/bitnami/containers 15:51:00.98 Submit issues and feature requests at https://github.com/bitnami/containers/issues 15:51:00.98 15:51:00.98 INFO ==> ** Starting MinIO Client setup ** 15:51:00.99 INFO ==> ** MinIO Client setup finished! ** Bucket created successfully `local/b-test-2`.
Checking on the Buckets screen in the web console, we see a screen as shown in the illustration below:
To copy the image file object jpeg-sample.jpg in the bucket b-test-1 to the bucket b-test-2, execute the following command in the client terminal window:
$ docker run --rm --name mc -u $(id -u $USER):$(id -g $USER) -v $MINIO_HOME/client:/.mc bitnami/minio-client:2022.10.20 cp local/b-test-1/jpeg-sample.jpg local/b-test-2/jpeg-sample.jpg
The following would be the typical output:
16:02:33.16 16:02:33.17 Welcome to the Bitnami minio-client container 16:02:33.17 Subscribe to project updates by watching https://github.com/bitnami/containers 16:02:33.17 Submit issues and feature requests at https://github.com/bitnami/containers/issues 16:02:33.17 16:02:33.17 INFO ==> ** Starting MinIO Client setup ** 16:02:33.17 INFO ==> ** MinIO Client setup finished! ** `local/b-test-1/jpeg-sample.jpg` -> `local/b-test-2/jpeg-sample.jpg` Total: 0 B, Transferred: 29.50 KiB, Speed: 2.25 MiB/s
Checking on the Bucket browser in the web console, we see a screen as shown in the illustration below:
To delete the image file object jpeg-sample.jpg from the bucket b-test-1, execute the following command in the client terminal window:
$ docker run --rm --name mc -u $(id -u $USER):$(id -g $USER) -v $MINIO_HOME/client:/.mc bitnami/minio-client:2022.10.20 rm local/b-test-1/jpeg-sample.jpg
The following would be the typical output:
16:40:48.29 16:40:48.29 Welcome to the Bitnami minio-client container 16:40:48.29 Subscribe to project updates by watching https://github.com/bitnami/containers 16:40:48.29 Submit issues and feature requests at https://github.com/bitnami/containers/issues 16:40:48.30 16:40:48.30 INFO ==> ** Starting MinIO Client setup ** 16:40:48.30 INFO ==> ** MinIO Client setup finished! ** Created delete marker `local/b-test-1/jpeg-sample.jpg` (versionId=a59b853d-e3e0-48b6-b71e-05f75de3815b).
Once again, to list all the buckets and objects, execute the following command in the client terminal window:
$ docker run --rm --name mc -u $(id -u $USER):$(id -g $USER) -v $MINIO_HOME/client:/.mc bitnami/minio-client:2022.10.20 ls --versions local
The following would be the typical output:
16:45:41.23 16:45:41.23 Welcome to the Bitnami minio-client container 16:45:41.23 Subscribe to project updates by watching https://github.com/bitnami/containers 16:45:41.24 Submit issues and feature requests at https://github.com/bitnami/containers/issues 16:45:41.24 16:45:41.24 INFO ==> ** Starting MinIO Client setup ** 16:45:41.24 INFO ==> ** MinIO Client setup finished! ** [2022-10-29 00:10:52 UTC] 0B b-test-1/ [2022-10-29 16:40:48 UTC] 0B STANDARD a59b853d-e3e0-48b6-b71e-05f75de3815b v2 DEL b-test-1/jpeg-sample.jpg [2022-10-29 00:32:59 UTC] 30KiB STANDARD ac05b181-d5be-43d2-bf3a-6f3c1440126d v1 PUT b-test-1/jpeg-sample.jpg [2022-10-29 15:51:01 UTC] 0B b-test-2/ [2022-10-29 16:02:33 UTC] 30KiB STANDARD 1284fa60-ff17-43c0-9db3-fc1136ff07f4 v1 PUT b-test-2/jpeg-sample.jpg
With this last MinIO client command, we conclude this hands-on demonstration of MinIO. Note that we have barely scratched the surface of the MinIO object storage capabilities.
References