Deploy AIStor in an Airgapped Environment

This procedure deploys AIStor Object Store on Kubernetes clusters in an airgapped environment. An airgapped environment is a network environment that has no direct connection to external networks or the internet. Installing AIStor in an airgapped environment requires copying all required container images and Helm charts to private registries accessible within the airgapped network.

Airgap deployment consists of two phases:

External Preparation (requires internet access): Download Helm charts and copy container images to your private registries from a system with network access to both the internet and your private registries.
Airgapped Deployment (within airgapped environment): Install AIStor using the mirrored resources from within your airgapped Kubernetes environment.

This procedure assumes you or your organization have existing procedures and tooling for facilitating such copying. Defer to your organization’s procedures on importing data from the internet into the airgapped environment. You cannot complete the first phase without tooling to facilitate the download-and-copy steps.

This procedure requires installation of Kubernetes operators and associated resources including CustomResourceDefinitions, StatefulSets, and secrets into new or existing namespaces.

When to use this procedure

Use this deployment method when:

Your Kubernetes cluster has no internet access to reach helm.min.io for charts or quay.io for images.
Your organization requires all container images and charts to be pulled from a private registry instead of public registries, such as for security, compliance, or governance reasons.

If your Kubernetes cluster only requires using a private registry, but otherwise has internet access, use the Deploy AIStor with a Private Container Registry procedure instead.

Prerequisites

Before beginning this procedure, ensure you have the following:

Kubernetes cluster running an in-support release
Access to a user with broad permissions to create Kubernetes resources (CustomResourceDefinitions, StatefulSets, and secrets) into multiple new or existing namespaces
Helm version 3.17 or later
Private container registry accessible from the Kubernetes cluster and means to copy public images to the private registry
Private Helm chart registry for storing Helm charts and means to copy public charts to a private registry
Skopeo installed for copying container images
Valid AIStor license from SUBNET
kubectl configured to access the target Kubernetes cluster

Deploy AIStor using Helm in an airgapped environment

The following procedure copies required container images and Helm charts to private registries and deploys AIStor using those resources.

Copy container images to private registry

Copy mandatory images and any optional images you need from quay.io to your private container registry. Perform these steps from a system that has network access to both quay.io and your private registry.

In each step, replace PRIVATE_REGISTRY with the hostname and path to your private container registry. If you use a different software or method for copying images to your private repository, defer to the documentation or procedure for that tooling.

The following commands use --override-os linux --override-arch amd64 flags to ensure images are copied for Linux AMD64 architecture. If your Kubernetes cluster uses a different architecture (such as ARM64), replace amd64 with the appropriate architecture for your cluster.

If your private registry does not have TLS configured, add the --dest-tls-verify=false flag to each skopeo command. For production environments, configure your registry with proper TLS certificates instead of disabling verification.

Copy the operator image:

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/aistor/operator:RELEASE.2025-08-19T17-53-00Z \
  docker://PRIVATE_REGISTRY/aistor/operator:RELEASE.2025-08-19T17-53-00Z

Copy the minio image:

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/aistor/minio:RELEASE.2025-08-21T02-50-21Z \
  docker://PRIVATE_REGISTRY/aistor/minio:RELEASE.2025-08-21T02-50-21Z

Copy the minio-sidecar image:

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/aistor/minio-sidecar:RELEASE.2025-06-30T21-59-04Z \
  docker://PRIVATE_REGISTRY/aistor/minio-sidecar:RELEASE.2025-06-30T21-59-04Z

Copy optional images based on the features you plan to use.

Copy the kes and kes-sidecar images if you plan to use server-side encryption with Key Encryption Service:

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/aistor/kes:RELEASE.2025-07-15T11-40-19Z \
  docker://PRIVATE_REGISTRY/aistor/kes:RELEASE.2025-07-15T11-40-19Z

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/aistor/kes-sidecar:RELEASE.2025-07-01T00-09-56Z \
  docker://PRIVATE_REGISTRY/aistor/kes-sidecar:RELEASE.2025-07-01T00-09-56Z

Copy the mc image if you plan to automate jobs using the AIStor Client:

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/aistor/mc:RELEASE.2025-08-21T03-14-05Z \
  docker://PRIVATE_REGISTRY/aistor/mc:RELEASE.2025-08-21T03-14-05Z

Copy the aihub image if you plan to store AI models and datasets:

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/aistor/aihub:RELEASE.2024-12-13T23-00-47Z \
  docker://PRIVATE_REGISTRY/aistor/aihub:RELEASE.2024-12-13T23-00-47Z

Copy the prompt image if you plan to query your objects with generative AI:

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/aistor/prompt:RELEASE.2025-01-17T21-55-43Z \
  docker://PRIVATE_REGISTRY/aistor/prompt:RELEASE.2025-01-17T21-55-43Z

Copy the warp image if you plan to run benchmark tests on your cluster:

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/warp:v1.3.0 \
  docker://PRIVATE_REGISTRY/minio/warp:v1.3.0

Verify that the images are accessible in your private registry.

You can use skopeo inspect to verify each image was copied successfully:
```
skopeo inspect docker://PRIVATE_REGISTRY/aistor/operator:RELEASE.2025-08-19T17-53-00Z
```
If your registry has a REST API endpoint, you can also verify using curl:
```
curl -k https://PRIVATE_REGISTRY/v2/_catalog
curl -k https://PRIVATE_REGISTRY/v2/aistor/operator/tags/list
```
Replace https:// with http:// if your registry does not use TLS.

Verify Before Proceeding

Before moving continuing the procedure, verify that all necessary components are in place:

All required container images (operator, minio, minio-sidecar) are in your private registry
Any optional container images for features you plan to use are in your private registry
Helm charts are downloaded locally or pushed to your Helm chart registry
You have verified image accessibility using skopeo inspect or registry API
If using physical media transfer, all tar archives are prepared and verified

This verification ensures you have all required resources before crossing into the airgapped environment where external resources cannot be retrieved.

Create image pull secret

From within the airgapped environment, create the image pull secret for your private container registry.

If your private container registry requires authentication, create a Kubernetes secret for pulling images.

Create the image pull secret in the aistor namespace:

kubectl create secret docker-registry private-repo-pull-secret \
  --docker-server=PRIVATE_REGISTRY \
  --docker-username=USERNAME \
  --docker-password=PASSWORD \
  --docker-email=EMAIL \
  -n aistor

Replace the following values:

PRIVATE_REGISTRY: The hostname of your private container registry
USERNAME: Username for authenticating to the private container registry
PASSWORD: Password for authenticating to the private container registry
EMAIL: Email address associated with the container registry account

If using an HTTP registry without TLS in a test environment, you may need to configure your Kubernetes nodes’ container runtime to allow insecure registry access. Consult your container runtime documentation (containerd, Docker, etc.) for instructions on configuring insecure registries. For production deployments, always use TLS-enabled registries.

Mirror AIStor Helm charts to private registry

Perform these steps from a system with network access to both the internet and your private Helm chart registry. This step is part of the external preparation phase.

Mirror the AIStor Helm charts from the public MinIO Helm repository to your private Helm chart registry. Perform these steps from a system that has network access to both the internet and your private registry.

Add the MinIO Helm repository:

helm repo add minio https://helm.min.io/
helm repo update

Pull the AIStor Helm charts locally:
```
helm pull minio/aistor-objectstore-operator
helm pull minio/aistor-objectstore
```
This downloads the chart files as .tgz archives to your current directory.
Push the Helm charts to your private Helm chart registry.

Common private Helm registries include Harbor, Artifactory, ChartMuseum, and OCI-compatible registries. Many container registries also support storing Helm charts as OCI artifacts.

For non-OCI registries, consult your registry documentation for specific upload instructions.

For OCI-compatible registries, use the following commands:
```
helm push aistor-objectstore-operator-*.tgz oci://PRIVATE_REGISTRY/helm-charts
helm push aistor-objectstore-*.tgz oci://PRIVATE_REGISTRY/helm-charts
```
Replace PRIVATE_REGISTRY with your registry hostname (for example, registry.example.com).

If your registry requires authentication:
```
helm registry login PRIVATE_REGISTRY --username USERNAME --password PASSWORD
helm push aistor-objectstore-operator-*.tgz oci://PRIVATE_REGISTRY/helm-charts
helm push aistor-objectstore-*.tgz oci://PRIVATE_REGISTRY/helm-charts
```

Add private Helm repository

From within the airgapped environment, configure access to your private Helm charts.

OCI Registry

If you pushed charts to an OCI-compatible registry, you do not need to add a repository. You can install directly from the OCI registry using the full OCI path (see installation steps below).

Authenticate to the OCI registry if required:

helm registry login PRIVATE_REGISTRY --username USERNAME --password PASSWORD

HTTP Repository

Add your private Helm repository:

helm repo add private-minio https://PRIVATE_HELM_REGISTRY_URL
helm repo update

Replace PRIVATE_HELM_REGISTRY_URL with the URL to your private Helm chart registry.

If your registry requires authentication, add credentials:

helm repo add private-minio https://PRIVATE_HELM_REGISTRY_URL \
  --username USERNAME \
  --password PASSWORD
helm repo update

Configure Helm values for private container registry

Create a Helm values file that configures AIStor to use your private container registry for pulling images.

Retrieve your license file.

Log in to SUBNET and select the License button in the Deployments view. Save the content of the file in a secure location for use in the next step.
Create a YAML manifest to customize the private-minio/aistor-objectstore-operator chart.

Use your preferred text editor to create a YAML manifest for the chart named aistor-objectstore-operator-values.yaml. The following YAML example configures the private container registry settings:
```
repositories:
  aistor:
    hostname: PRIVATE_CONTAINER_REGISTRY
    pathPrefix: aistor/
    imagePullSecrets:
    - name: private-repo-pull-secret
  minio:
    hostname: PRIVATE_CONTAINER_REGISTRY
    pathPrefix: minio/
    imagePullSecrets:
    - name: private-repo-pull-secret
```
Replace PRIVATE_CONTAINER_REGISTRY with the hostname of your private container registry (for example, registry.example.com or registry.example.com:5000).

The repositories section configures the Helm chart to pull all AIStor images from your private container registry instead of quay.io. The pathPrefix value specifies the path within your container registry where you copied the images.

The Helm chart uses the operator image directly in the deployment. The chart stores all other images in the object-store-operator configuration map in the aistor namespace. Do not update values in the configuration map directly because helm upgrade overwrites them. Always update the aistor-objectstore-operator-values.yaml file you use to deploy or redeploy the AIStor Operator to ensure changes persist.
If you need to override specific image versions, add an images section to the values file:
```
images:
  minio:
    repository: aistor
    image: minio:RELEASE.2025-08-29T21-27-49Z
```
This example overrides the default minio image version with a different release. Refer to the AIStor Operator values reference for all available image customizations.

Do not update values in the object-store-operator configuration map directly, as helm upgrade overwrites them. Instead, always use an custom YAML file such as aistor-objectstore-operator-values.yaml to install and upgrade AIStor. This ensures that changes persist.

Install the AIStor Operator

Install the AIStor Operator using the customized values file and your SUBNET license.

The license parameter requires the decoded JWT token value from your SUBNET license file (the value begins with eyJ...).

OCI Registry

Install the chart from your OCI registry:

helm install aistor oci://PRIVATE_REGISTRY/helm-charts/aistor-objectstore-operator \
  -n aistor --create-namespace \
  --set license="eyJhbGciOiJFUzM4NCIsInR..." \
  -f aistor-objectstore-operator-values.yaml

Replace PRIVATE_REGISTRY with your registry hostname and replace the license value with your decoded JWT token from SUBNET.

HTTP Repository

Install the chart from your HTTP-based Helm repository:

helm install aistor private-minio/aistor-objectstore-operator \
  -n aistor --create-namespace \
  --set license="eyJhbGciOiJFUzM4NCIsInR..." \
  -f aistor-objectstore-operator-values.yaml

Replace the license value with your decoded JWT token from SUBNET.

Local Chart File

If your airgapped Kubernetes environment cannot access the Helm chart registry, install directly from the local chart file:

helm install aistor ./aistor-objectstore-operator-*.tgz \
  -n aistor --create-namespace \
  --set license="eyJhbGciOiJFUzM4NCIsInR..." \
  -f aistor-objectstore-operator-values.yaml

Replace the license value with your decoded JWT token from SUBNET.

This method is useful for environments where the Helm chart registry is not accessible from within the airgapped network.

If successful, the command outputs a summary of installed resources.

Validate the installation by running the following command:

kubectl get all -n aistor

The output should show running pods similar to the following:

NAME                                         READY   STATUS    RESTARTS   AGE
pod/adminjob-operator-cfc97d9f-hjbp5         1/1     Running   0          4m16s
pod/object-store-operator-78c9f84b85-kmwlv   1/1     Running   0          4m16s

NAME                            TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)    AGE
service/object-store-operator   ClusterIP   10.43.210.230   <none>        4221/TCP   4m16s

NAME                                    READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/adminjob-operator       1/1     1            1           4m16s
deployment.apps/object-store-operator   1/1     1            1           4m16s

NAME                                               DESIRED   CURRENT   READY   AGE
replicaset.apps/adminjob-operator-cfc97d9f         1         1         1       4m16s
replicaset.apps/object-store-operator-78c9f84b85   1         1         1       4m16s

Deploy an AIStor Object Store

After the Operator is running, deploy an AIStor Object Store instance.

Create a YAML manifest to customize the object store chart.

Use your preferred text editor to create a file named aistor-objectstore-values.yaml. The following example has a minimal set of fields for deploying an 8x8 object store using the AIStor Volume Manager for storage provisioning:
```
objectStore:
  name: primary-object-store
  pools:
  - name: pool-0
    servers: 8
    volumesPerServer: 8
    size: 2Ti
  services:
    minio:
      serviceType: NodePort
      nodePort: 31000
```
Modify the values to reflect your deployment requirements. Remove any default or unmodified values such that the file reflects only your changes.

Install the object store chart:

OCI Registry

Install the chart from your OCI registry:

helm install primary-object-store oci://PRIVATE_REGISTRY/helm-charts/aistor-objectstore \
  -n primary-object-store --create-namespace \
  -f aistor-objectstore-values.yaml

Replace PRIVATE_REGISTRY with your registry hostname.

HTTP Repository

Install the chart from your HTTP-based Helm repository:

helm install primary-object-store private-minio/aistor-objectstore \
  -n primary-object-store --create-namespace \
  -f aistor-objectstore-values.yaml

Local Chart File

Install directly from the local chart file:

helm install primary-object-store ./aistor-objectstore-*.tgz \
  -n primary-object-store --create-namespace \
  -f aistor-objectstore-values.yaml

The command deploys an AIStor Server with the name and namespace of primary-object-store.

Connect to the deployment

After the object store is deployed, connect to it using either the Console web interface or the AIStor Client command-line tool.

Console

Configure ingress to route traffic to the -console service in the object store namespace. You can temporarily access the Console using kubectl port-forward:

kubectl port-forward -n primary-object-store svc/primary-object-store-console 9443:9443

You can use the browser interface for managing the object store installation.

CLI

Download the mc tool to your local host machine.

See AIStor Client Quickstart for installation instructions.
Create a new alias for the AIStor server.

The S3 API services are configured via NodePort so you can access them through the IP or hostname of any worker node in the Kubernetes cluster. Use kubectl get nodes -o wide to determine the appropriate IP or hostname to use for access.
```
mc --insecure alias set myaistor https://NODE_IP_OR_HOSTNAME:31000 minio minio123
```
Replace NODE_IP_OR_HOSTNAME with the externally accessible hostname or IP of any worker node in the deployment.

Create a bucket and copy a file to that bucket:

mc --insecure mb myaistor/data
mc --insecure cp ~/somefile.txt myaistor/data

Troubleshooting

Architecture mismatch errors when copying images

If you encounter errors like no image found in image index for architecture when using Skopeo, ensure you are using the --override-os and --override-arch flags with the correct values for your Kubernetes cluster.

The images must match the architecture of your Kubernetes nodes, not the system where you run the skopeo commands. For example, if copying from a Mac to a Linux AMD64 cluster:

skopeo copy --override-os linux --override-arch amd64 \
  docker://quay.io/minio/aistor/operator:RELEASE.2025-08-19T17-53-00Z \
  docker://PRIVATE_REGISTRY/aistor/operator:RELEASE.2025-08-19T17-53-00Z

TLS/SSL certificate errors

If you see errors related to certificate verification when copying images, you can use one of the following options depending on your environment:

For development/testing: Add --dest-tls-verify=false to disable TLS verification:

skopeo copy --override-os linux --override-arch amd64 \
  --dest-tls-verify=false \
  docker://quay.io/minio/aistor/operator:RELEASE.2025-08-19T17-53-00Z \
  docker://PRIVATE_REGISTRY/aistor/operator:RELEASE.2025-08-19T17-53-00Z

For production environments: Configure your private registry with valid TLS certificates instead of disabling verification.

Connection refused or EOF errors

If Skopeo cannot connect to your private registry:

Verify the registry is running and accessible:
```
curl -k https://PRIVATE_REGISTRY/v2/
```
This should return an empty JSON object {} if the registry is accessible.
Check that the hostname and port are correct in your PRIVATE_REGISTRY value.
Verify network connectivity between the system running Skopeo and the private registry.
If using a firewall, ensure the registry port is open.

Images fail to pull in Kubernetes

If images were copied successfully but fail to pull in your Kubernetes cluster:

Verify the image pull secret was created correctly:

kubectl get secret private-repo-pull-secret -n aistor

Verify the Helm values file contains the correct registry hostname and image pull secret name.
Check pod events for specific error messages:
```
kubectl describe pod POD_NAME -n NAMESPACE
```
Verify network connectivity from the Kubernetes nodes to the private registry.