Table of Contents
This procedure documents deploying a MinIO Tenant using the MinIO Operator Console.
The Operator Console provides a rich user interface for deploying and managing MinIO Tenants on Kubernetes infrastructure. Installing the MinIO Kubernetes Operator automatically installs and configures the Operator Console.
This documentation assumes familiarity with all referenced Kubernetes concepts, utilities, and procedures. While this documentation may provide guidance for configuring or deploying Kubernetes-related resources on a best-effort basis, it is not a replacement for the official Kubernetes Documentation.
The Operator Console requires the MinIO Kubernetes Operator. This procedure assumes the latest stable Operator version 4.4.16.
See Deploy MinIO Operator on Kubernetes for complete documentation on deploying the MinIO Operator.
Run the following commands to install the MinIO Operator and Plugin using the
Kubernetes krew plugin manager:
Hybrid Cloud Object Storage
Baremetal
Erasure Code Calculator
Reference Hardware
kubectl krew update
kubectl krew install minio
See the krew installation documentation for specific
instructions.
You can also download the kubectl-minio plugin directly and install it to
your system PATH. The following code downloads the latest stable version
4.4.16 of the MinIO Kubernetes Plugin and installs it to the
system $PATH:
wget https://github.com/minio/operator/releases/download/v4.4.16/kubectl-minio_4.4.16_linux_amd64 -O kubectl-minio
chmod +x kubectl-minio
mv kubectl-minio /usr/local/bin/
Run the following command to verify installation of the plugin:
kubectl minio version
The output should display the Operator version as 4.4.16.
Starting with v4.0.0, the MinIO Operator requires Kubernetes 1.19.0 and later.
The Kubernetes infrastructure and the kubectl CLI tool must have the same
version of 1.19.0+.
This procedure assumes the host machine has kubectl installed and
configured with access to the target Kubernetes cluster. The host machine
must have access to a web browser application.
MinIO strongly recommends using locally attached drives on each node intended to support the MinIO Tenant. MinIO’s strict read-after-write and list-after-write consistency model requires local disk filesystems (xfs, ext4, etc.). MinIO also shows best performance with locally-attached drives.
MinIO automatically generates Persistent Volume Claims (PVC) as part of
deploying a MinIO Tenant. The Operator generates one PVC for each volume in the
tenant plus two PVC to support collecting Tenant Metrics and logs. For
example, deploying a Tenant with 16 volumes requires 18 (16 + 2) PV.
This procedure uses the MinIO DirectCSI driver to automatically provision Persistent Volumes from locally attached drives to support the generated PVC. See the DirectCSI Documentation for installation and configuration instructions.
For clusters which cannot deploy MinIO Direct CSI, Local Persistent Volumes.
The following tabs provide example YAML objects for a local persistent volume and a supporting StorageClass:
The following YAML describes a Local Persistent Volume:
apiVersion: v1
kind: PersistentVolume
metadata:
name: <PV-NAME>
spec:
capacity:
storage: 1Ti
volumeMode: Filesystem
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storage-class: <STORAGE-CLASS>
local:
path: <PATH-TO-DISK>
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- <NODE-NAME>
Replace values in brackets <VALUE> with the appropriate
value for the local drive.
The following YAML describes a StorageClass that meets the requirements for a MinIO Tenant:
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: minio-local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
The storage class must have volumeBindingMode: WaitForFirstConsumer.
Ensure all Persistent Volumes provisioned to support the MinIO Tenant
use this storage class.
Use the kubectl minio proxy command to temporarily forward
traffic between the local host machine and the MinIO Operator Console:
kubectl minio proxy
The command returns output similar to the following:
Starting port forward of the Console UI.
To connect open a browser and go to http://localhost:9090
Current JWT to login: TOKEN
Open your browser to the specified URL and enter the JWT Token into the login page. You should see the Tenants page:
Click the + Create Tenant to start creating a MinIO Tenant.
The Setup pane contains all core configuration settings for the MinIO Tenant.
Settings marked with an asterisk * are required:
Field |
Description |
|---|---|
Name |
The name of the MinIO Tenant |
Namespace |
The Kubernetes Namespace in which to deploy the tenant. You can create the namespace by selecting the plus + icon if it does not exist. The Operator supports at most one MinIO Tenant per namespace. |
Storage Class |
Specify the Kubernetes Storage Class the Operator uses when generating Persistent Volume Claims for the Tenant. This procedure assumes using the DirectCSI
storage class |
Number of Servers |
The total number of MinIO server pods to deploy in the Tenant. The Operator by default uses pod anti-affinity, such that the Kubernetes cluster must have at least one worker node per MinIO server pod. Use the Pod Placement pane to modify the pod scheduling settings for the Tenant. |
Number of Drives per Server |
The number of storage volumes (Persistent Volume Claims) the Operator requests per Server. The Operator displays the Total Volumes under the Resource Allocation section. The Operator generates an equal number of PVC plus two for supporting Tenant services (Metrics and Log Search). The specified Storage Class must correspond to a set of Persistent Volumes sufficient in number to match each generated PVC. |
Total Size |
The total raw storage size for the Tenant. Specify both the total storage size and the Unit of that storage. All storage units are in SI values, e.g. Gi = GiB = 10243 bytes. The Operator displays the Drive Capacity under the Resource Allocation section. The Operator sets this value as the requested storage capacity in each generated PVC. The specified Storage Class must correspond to a set of Persistent Volumes sufficient in capacity to match each generated PVC. |
Memory per Node [Gi] |
Specify the total amount of memory (RAM) to allocate per MinIO server pod. See Memory Allocation for guidance on setting this value. The Kubernetes cluster must have worker nodes with sufficient free RAM to match the pod request. |
Erasure Code Parity |
The Erasure Code Parity to set for the deployment. The Operator displays the selected parity and its effect on the deployment under the Erasure Code Configuration section. Erasure Code parity defines the overall resiliency and availability of data on the cluster. Higher parity values increase tolerance to drive or node failure at the cost of total storage. See Erasure Coding for more complete documentation. |
Select Create to create the Tenant using the current configuration. While all subsequent sections are optional, MinIO recommends reviewing them prior to deploying the Tenant.
The Configure section contains optional configuration settings for the MinIO Tenant and its supporting services.
Field |
Description |
|---|---|
Expose Services |
The MinIO Operator by default directs the MinIO Tenant services to request an externally accessible IP address from the Kubernetes cluster Load Balancer if one is available. Most public cloud Kubernetes infrastructures include a global Load Balancer which meets this requirements. Other Kubernetes distributions may include a load balancer that can respond to these requests. You can direct the Tenant to not make this request by toggling the option to Off for the MinIO Service and Console Service. |
Override Tenant Defaults |
The MinIO Operator sets the Kubernetes Security Context for pods to
a default of You can modify the Security Context to direct MinIO to run using a different User, Group, or FsGroup ID. You can also direct MinIO to not run as the Root user. |
Override Log Search Defaults |
The MinIO Operator deploys a Log Search service (SQL Database and Log Search API) to support Audit Log search in the MinIO Tenant Console. You can modify the Security Context to run the associated pod commands using a different User, Group, or FsGroup ID. You can also direct the pod to not run commands as the Root user. You can also modify the storage class and requested capacity associated to the PVC generated to support the Log Search service. |
Override Prometheus Search Defaults |
The MinIO Operator deploys a Prometheus service to support detailed metrics in the MinIO Tenant Console. You can modify the Security Context to run the associated pod commands using a different User, Group, or FsGroup ID. You can also direct the pod to not run commands as the Root user. You can also modify the storage class and requested capacity associated to the PVC generated to support the Prometheus service. |
The Images section contains container image settings used by the MinIO Tenant.
Field |
Description |
|---|---|
MinIO’s Image |
The container image to use for the MinIO Server. See the MinIO Quay or the MinIO DockerHub repositories for a list of valid tags. |
Log Search API’s Image |
The container image to use for MinIO Log Search API. |
KES Image |
The container image to use for MinIO KES. |
Log Search Postgres Image
Log Search Postgres Init Image
|
The container images to use for starting the PostgreSQL service supporting the Log Search API |
Prometheus Image
Prometheus Sidecar Image
Prometheus Init Image
|
The container images to use for starting the Prometheus service supporting the Log Search API. |
The Pod Placement section contains pod scheduler settings for the MinIO Tenant.
Field |
Description |
|---|---|
None |
Disables pod scheduling constraints for the tenant. This allows Kubernetes to schedule multiple Tenant pods onto the same node. This may decrease resiliency, as a single Kubernetes worker can host multiple MinIO pods. If that worker is down or lost, objects may also unavailable or lost. Consider using this setting only in early development or sandbox environments with a limited number of worker nodes. |
Default (Pod Anti-Affinity) |
Directs the Operator to set anti-affinity settings such that no Kubernetes worker can host more than one MinIO server pod for this Tenant. |
Node Selector |
Directs the operator to set a Node Selector such that pods only deploy onto Kubernetes workers whose labels match the selector. |
The Identity Provider section contains the Identity Provider settings for the MinIO Tenant. This includes configuring an external IDP such as OpenID or Active Directory / LDAP.
Field |
Description |
|---|---|
Built-In |
Configure additional internal MinIO users for the Operator to create as part of deploying the Tenant. |
OpenID |
Configure an OpenID Connect-compatible servce as an external Identity Provider (e.g. Keycloak, Okta, Google, Facebook, Dex) to manage MinIO users. |
Active Directory |
Configure an Active Directory or OpenLDAP service as the external Identity Provider to manage MinIO users. |
The Security section contains TLS certificate settings for the MinIO Tenant.
Field |
Description |
|---|---|
Enable TLS |
Enable or disable TLS for the MinIO Tenant. |
Enable AutoCert |
Directs the Operator to generate Certificate Signing Requests for submission to the Kubernetes TLS API. The MinIO Tenant uses the generated certificates for enabling and establishing TLS connections. |
Custom Certificates |
Specify one or more custom TLS certificates for use by the MinIO Tenant. MinIO supports Server Name Indication (SNI) such that the Tenant can select the appropriate TLS certificate based on the request hostname and the certificate Subject Alternative Name. MinIO also supports specifying Certificate Authority certificates for validating client certificates minted by that CA. |
The Encryption section contains the Server-Side Encryption settings for the MinIO Tenant.
Enabling SSE also deploys a MinIO KES service in the Tenant to faciliate SSE operations.
Field |
Description |
|---|---|
Vault |
Configure Hashicorp Vault as the external KMS for storing root encryption keys. See Server-Side Object Encryption with Hashicorp Vault Root KMS for guidance on the displayed fields. |
AWS |
Configure AWS Secrets Manager as the external KMS for storing root encryption keys. See Server-Side Object Encryption with AWS Root KMS for guidance on the displayed fields. |
GCP |
Configure Google Cloud Platform Secret Manager as the external KMS for storing root encryption keys. See Server-Side Object Encryption with GCP Secret Manager Root KMS for guidance on the displayed fields. |
Azure |
Configure Azure Key Vault as the external KMS for storing root encryption keys. See Server-Side Object Encryption with Azure Key Vault Root KMS for guidance on the displayed fields. |
Select Create at any time to begin the deployment process. The MinIO Operator displays the root user credentials once as part of deploying the Tenant. Copy these credentials to a secure location.
You can monitor the Tenant creation process from the Tenants view. The State column updates throughout the deployment process.
Tenant deployment can take several minutes to complete. Once the State reads as Initialized, click the Tenant to view its details.
Each tab provides additional details or configuration options for the MinIO Tenant.
METRICS - Displays metrics collected from the MinIO Tenant.
SECURITY - Provides TLS-related configuration options.
POOLS - Supports expanding the tenant by adding more Server Pools.
LICENSE - Enter your SUBNET license.
The MinIO Operator creates services for the MinIO Tenant. Use the
kubectl get svc -n NAMESPACE command to review the deployed services:
kubectl get svc -n minio-tenant-1
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
minio LoadBalancer 10.97.114.60 <pending> 443:30979/TCP 2d3h
minio-tenant-1-console LoadBalancer 10.106.103.247 <pending> 9443:32095/TCP 2d3h
minio-tenant-1-hl ClusterIP None <none> 9000/TCP 2d3h
minio-tenant-1-log-hl-svc ClusterIP None <none> 5432/TCP 2d3h
minio-tenant-1-log-search-api ClusterIP 10.103.5.235 <none> 8080/TCP 2d3h
minio-tenant-1-prometheus-hl-svc ClusterIP None <none> 9090/TCP 7h39m
The minio service corresponds to the MinIO Tenant service. Applications
should use this service for performing operations against the MinIO Tenant.
The *-console service corresponds to the MinIO Console. Administrators should use this service for accessing the MinIO
Console and performing administrative operations on the MinIO Tenant.
The remaining services support Tenant operations and are not intended for consumption by users or administrators.
By default each service is visible only within the Kubernetes cluster.
Applications deployed inside the cluster can access the services using the
CLUSTER-IP.
Applications external to the Kubernetes cluster can access the services using
the EXTERNAL-IP. This value is only populated for Kubernetes clusters
configured for Ingress or a similar network access service. Kubernetes provides
multiple options for configuring external access to services. See the Kubernetes
documentation on Publishing Services (ServiceTypes) and
Ingress for more complete
information on configuring external access to services.
You can temporarily expose each service using the
kubectl port-forward utility. Run the following examples to forward
traffic from the local host running kubectl to the services running inside
the Kubernetes cluster.
kubectl port-forward service/minio 443:443
kubectl port-forward service/minio-tenant-1-console 9443:9443
