Distributed MinIO Quickstart Guide Slack Docker Pulls

MinIO in distributed mode lets you pool multiple drives (even on different machines) into a single object storage server. As drives are distributed across several nodes, distributed MinIO can withstand multiple node failures and yet ensure full data protection.

Why distributed MinIO?

MinIO in distributed mode can help you setup a highly-available storage system with a single object storage deployment. With distributed MinIO, you can optimally use storage devices, irrespective of their location in a network.

Data protection

Distributed MinIO provides protection against multiple node/drive failures and bit rot using erasure code. As the minimum disks required for distributed MinIO is 2 (same as minimum disks required for erasure coding), erasure code automatically kicks in as you launch distributed MinIO.

If one or more disks are offline at the start of a PutObject or NewMultipartUpload operation the object will have additional data protection bits added automatically to provide additional safety for these objects.

High availability

A stand-alone MinIO server would go down if the server hosting the disks goes offline. In contrast, a distributed MinIO setup with m servers and n disks will have your data safe as long as m/2 servers or m*n/2 or more disks are online.

For example, an 16-server distributed setup with 200 disks per node would continue serving files, up to 4 servers can be offline in default configuration i.e around 800 disks down MinIO would continue to read and write objects.

Refer to sizing guide for more understanding on default values chosen depending on your erasure stripe size here. Parity settings can be changed using storage classes.

Consistency Guarantees

MinIO follows strict read-after-write and list-after-write consistency model for all i/o operations both in distributed and standalone modes. This consistency model is only guaranteed if you use disk filesystems such as xfs, zfs or btrfs etc.. for distributed setup.

In our tests we also found ext4 does not honor POSIX O_DIRECT/Fdatasync semantics, ext4 trades performance for consistency guarantees. Please avoid ext4 in your setup.

If MinIO distributed setup is using NFS volumes underneath it is not guaranteed MinIO will provide these consistency guarantees since NFS is not strictly consistent (If you must use NFS we recommend that you atleast use NFSv4 instead of NFSv3 for relatively better outcomes).

Get started

If you're aware of stand-alone MinIO set up, the process remains largely the same. MinIO server automatically switches to stand-alone or distributed mode, depending on the command line parameters.

1. Prerequisites

Install MinIO - MinIO Quickstart Guide.

2. Run distributed MinIO

To start a distributed MinIO instance, you just need to pass drive locations as parameters to the minio server command. Then, you’ll need to run the same command on all the participating nodes.


Example 1: Start distributed MinIO instance on n nodes with m drives each mounted at /export1 to /exportm (pictured below), by running this command on all the n nodes:

Distributed MinIO, n nodes with m drives each

GNU/Linux and macOS

minio server http://host{1...n}/export{1...m}

NOTE: In above example n and m represent positive integers, do not copy paste and expect it work make the changes according to local deployment and setup.
NOTE: {1...n} shown have 3 dots! Using only 2 dots {1..n} will be interpreted by your shell and won't be passed to MinIO server, affecting the erasure coding order, which would impact performance and high availability. Always use ellipses syntax {1...n} (3 dots!) for optimal erasure-code distribution

Expanding existing distributed setup

MinIO supports expanding distributed erasure coded clusters by specifying new set of clusters on the command-line as shown below:

minio server http://host{1...n}/export{1...m} http://host{o...z}/export{1...m}

For example:

minio server http://host{1...4}/export{1...16} http://host{5...12}/export{1...16}

Now the server has expanded total storage by (newly_added_servers*m) more disks, taking the total count to (existing_servers*m)+(newly_added_servers*m) disks. New object upload requests automatically start using the least used cluster. This expansion strategy works endlessly, so you can perpetually expand your clusters as needed. When you restart, it is immediate and non-disruptive to the applications. Each group of servers in the command-line is called a pool. There are 2 server pools in this example. New objects are placed in server pools in proportion to the amount of free space in each pool. Within each pool, the location of the erasure-set of drives is determined based on a deterministic hashing algorithm.

NOTE: Each pool you add must have the same erasure coding parity configuration as the original pool, so the same data redundancy SLA is maintained.

3. Test your setup

To test this setup, access the MinIO server via browser or mc.

Explore Further