Kubernetes

Mount a MinIO AIStor Memory Bucket (cortex) into a pod with the AIStor Memory CSI node driver — the workload pod gains no privilege, because kubelet mounts on the node.

Status — v1 driver, published image, no end-to-end CI

The driver image is published for each release and the driver is in v1. There is no automated end-to-end test that mounts through a live cluster yet, so validate the driver on a staging cluster before you depend on it in production. The driver has no Controller service: it mounts a cortex that already exists and never creates one.

A pod is disposable. Anything an agent writes to the container filesystem disappears when the pod does. The AIStor Memory CSI node driver gives that pod a durable workspace instead: it mounts a Memory Bucket — a cortex — as an ordinary directory, so everything written there persists to AIStor on your own storage with your own keys. The next pod mounts the same cortex and starts with what the last one wrote.

Why the CSI driver rather than mounting inside the pod

You can run aimem inside a pod, as the OpenShift page shows. That pod needs CAP_SYS_ADMIN, privilege escalation for the setuid mount helper, and access to /dev/fuse. A restrictive PodSecurity policy or an OpenShift SCC blocks all three, and granting them to a pod that runs model-authored code widens what that code can reach.

The CSI driver moves the mount off the pod. kubelet asks the driver to mount on the node before the container starts, and the container then sees a plain directory. The workload pod needs no capabilities at all:

securityContext:
  allowPrivilegeEscalation: false
  runAsNonRoot: true
  runAsUser: 1000
  capabilities:
    drop: ["ALL"]
  seccompProfile:
    type: RuntimeDefault

The privilege moves to one DaemonSet you install and audit once, instead of every workload pod.

What your nodes need

The mount runs on the node under systemd, so each node must provide:

systemd, with PID 1 visible to the driver pod. The DaemonSet sets hostPID: true and uses nsenter to reach the host's systemd. Minimal hosts that do not run systemd, such as Talos, are out of scope in v1.
/dev/fuse present on the node.
A kubelet directory mounted with Bidirectional propagation, so a mount made on the node reaches the workload pod. The DaemonSet below sets this.

You also need a cortex that already exists, and credentials that can mount it. aimem cortex credentials <name> mints scoped credentials for one cortex.

The endpoint must resolve from the node

This is the mistake that costs the most time. Because the mount executes on the node, endpointUrl has to resolve and connect from the node, not from the pod.

A Kubernetes Service name does not work. The node sits outside the cluster network and does not use cluster DNS, so http://aistor:9000 fails with a transport error while a pod on the same cluster reaches it without trouble. Use one of these instead:

A Service ClusterIP, which kube-proxy programs on the node.
Any address routable from the host. An external AIStor endpoint is the normal production case.

Install the driver

Pin the image to the release you are deploying. Do not use :latest: this container performs the mount every workload on the node depends on, so a floating tag lets a node restart change the filesystem implementation under running pods.

Save this as aimem-csi.yaml, replace RELEASE_TAG with your release, and apply it:

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: csi.aimem.min.io
spec:
  # Node-only driver: no Controller service, nothing to attach, so kubelet
  # must not wait on an external-attacher.
  attachRequired: false
  podInfoOnMount: true
  volumeLifecycleModes:
    - Persistent
    - Ephemeral
  fsGroupPolicy: None
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: aimem-csi-node-sa
  namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aimem-csi-node
  namespace: kube-system
  labels:
    app: aimem-csi-node
spec:
  selector:
    matchLabels:
      app: aimem-csi-node
  template:
    metadata:
      labels:
        app: aimem-csi-node
    spec:
      serviceAccountName: aimem-csi-node-sa
      # Lets the driver nsenter into the host namespaces and launch aimem
      # under the host's systemd.
      hostPID: true
      priorityClassName: system-node-critical
      tolerations:
        - operator: Exists
      initContainers:
        # Stage the aimem binary onto the host: the mount runs on the host,
        # not in this pod.
        - name: install-aimem
          image: quay.io/minio/aistor/aimem-csi-driver:RELEASE_TAG
          command:
            - sh
            - -c
            - install -D -m 0755 /usr/local/bin/aimem /host/opt/aimem/bin/aimem
          volumeMounts:
            - name: aimem-host-bin
              mountPath: /host/opt/aimem/bin
      containers:
        - name: aimem-csi-driver
          image: quay.io/minio/aistor/aimem-csi-driver:RELEASE_TAG
          args:
            - --endpoint=$(CSI_ENDPOINT)
            - --node-id=$(CSI_NODE_ID)
          env:
            - name: CSI_ENDPOINT
              value: unix:///csi/csi.sock
            - name: CSI_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: RUST_LOG
              value: info
          securityContext:
            privileged: true
          livenessProbe:
            httpGet:
              path: /healthz
              port: healthz
            initialDelaySeconds: 10
            periodSeconds: 60
            timeoutSeconds: 3
          ports:
            - name: healthz
              containerPort: 9808
              protocol: TCP
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: kubelet-dir
              mountPath: /var/lib/kubelet
              # Bidirectional so the host-side mount propagates into workload
              # pods.
              mountPropagation: Bidirectional
            - name: dev-fuse
              mountPath: /dev/fuse
            - name: aimem-host-bin
              mountPath: /opt/aimem/bin
        - name: node-driver-registrar
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0
          args:
            - --csi-address=/csi/csi.sock
            - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.aimem.min.io/csi.sock
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
        - name: liveness-probe
          image: registry.k8s.io/sig-storage/livenessprobe:v2.16.0
          args:
            - --csi-address=/csi/csi.sock
            - --health-port=9808
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
      volumes:
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi.aimem.min.io/
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory
        - name: kubelet-dir
          hostPath:
            path: /var/lib/kubelet
            type: Directory
        - name: dev-fuse
          hostPath:
            path: /dev/fuse
            type: CharDevice
        - name: aimem-host-bin
          hostPath:
            path: /opt/aimem/bin
            type: DirectoryOrCreate

kubectl apply -f aimem-csi.yaml
kubectl -n kube-system rollout status daemonset/aimem-csi-node

A node-only driver that provisions nothing needs no cluster-scoped permissions, which is why the ServiceAccount above carries no Role. It exists to give the DaemonSet a stable identity.

Choose a volume style

Both styles mount a cortex that already exists. They differ in who authors the mount and how long it lives.

Style	Who authors it	Lives as long as	Use it when
Inline ephemeral	Whoever creates the pod	The pod	One mount per unit of work — agent sandboxes, CI jobs
PersistentVolume	A cluster admin	The PV	A long-lived, shared workspace, or when only admins may name buckets

Inline ephemeral leaves nothing behind for a controller to garbage-collect. A PersistentVolume puts the mount's settings under admin control, which matters because localDir is accepted only there.

Mount with an inline ephemeral volume

The pod names the cortex directly. bucketName is required here: kubelet generates the volume handle from the pod, so the fallback that a PersistentVolume uses would name a bucket that does not exist, and the driver rejects the request rather than failing later.

apiVersion: v1
kind: Secret
metadata:
  name: aimem-inline-credentials
type: Opaque
stringData:
  # Scope these to one cortex where you can:
  # `aimem cortex credentials <name>`.
  accessKeyID: REPLACE_ME
  secretAccessKey: REPLACE_ME
  # sessionToken: REPLACE_ME   # required for STS credentials
---
apiVersion: v1
kind: Pod
metadata:
  name: aimem-inline-workspace
spec:
  restartPolicy: Never
  containers:
    - name: workload
      image: ubuntu:24.04
      command: ["bash", "-lc"]
      args:
        - |
          set -euo pipefail
          ls -la /workspace
          # This write is durable: it lands in the cortex, not in the pod.
          printf 'hello from %s\n' "$(hostname)" > /workspace/probe.txt
          sync
      volumeMounts:
        - name: workspace
          mountPath: /workspace
      securityContext:
        allowPrivilegeEscalation: false
        runAsNonRoot: true
        runAsUser: 1000
        capabilities:
          drop: ["ALL"]
        seccompProfile:
          type: RuntimeDefault
  volumes:
    - name: workspace
      csi:
        driver: csi.aimem.min.io
        volumeAttributes:
          bucketName: my-project
          # Mount a sub-prefix instead of the whole cortex. Leading and
          # trailing slashes are optional.
          # prefix: workspaces/alice/app-7/
          endpointUrl: https://aistor.example.com:9000
        nodePublishSecretRef:
          name: aimem-inline-credentials

Mount with a PersistentVolume

An admin defines the volume, and the pod claims it. volumeHandle names the cortex when bucketName is absent.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: aimem-my-project
spec:
  capacity:
    storage: 1Ti # Ignored by the driver; Kubernetes requires a value.
  accessModes: ["ReadWriteMany"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "" # Static: bind by claim, not by class.
  csi:
    driver: csi.aimem.min.io
    volumeHandle: my-project
    volumeAttributes:
      endpointUrl: https://aistor.example.com:9000
      # Staging and read cache on the node. PersistentVolume only.
      localDir: /var/lib/aimem/staging/my-project
    nodePublishSecretRef:
      name: aimem-credentials
      namespace: kube-system

Volume attributes

Set these under volumeAttributes.

Attribute	Effect
`bucketName`	The cortex to mount. Defaults to `volumeHandle`; required on an inline volume.
`prefix`	Mount a sub-prefix, as `bucket/prefix/`. Surrounding slashes are normalised.
`endpointUrl`	The AIStor endpoint. Must resolve from the node.
`region`	The region to sign requests for.
`agent`	The agent identity AIStor stamps on writes.
`metadataTtl`	How long metadata stays cached.
`tlsCaFile`	A CA bundle path on the node, not in the pod.
`localDir`	Staging and read cache on the node. PersistentVolume only.
`readOnly`	Mount read-only. Also honoured from the CSI request flag.

Credentials come from the Secret named by nodePublishSecretRef:

Secret key	Passed to aimem as
`accessKeyID`	`AIMEM_ACCESS_KEY`
`secretAccessKey`	`AIMEM_SECRET_KEY`
`sessionToken`	`AIMEM_SESSION_TOKEN`

Attributes the driver rejects

uid, gid, allowOther, allowRoot, and virtualHostStyle are rejected with InvalidArgument naming the attribute. The aimem CLI has no matching flag, so there is nothing to translate them to. The driver refuses the mount rather than ignoring the request, because a volume that asks for allowOther and comes up without it looks like it worked while behaving differently from what it declares.

localDir is rejected on an inline ephemeral volume, and only there. It names a host path, and an inline volume's attributes are authored by whoever creates the pod — honouring it would let a pod stage into another volume's directory, or anywhere else on the node. A PersistentVolume's attributes are an administrator's, so it is accepted there. It must still be absolute and free of ...

How a mount comes up and goes away

On NodePublishVolume the driver launches aimem on the host under a transient systemd unit, then polls /proc/self/mountinfo until the mount appears. Because host PID 1 owns the mount, restarting or upgrading the driver DaemonSet does not tear down live mounts.

On NodeUnpublishVolume it stops the unit, unmounts, and removes the target directory.

Limitations in v1

No dynamic provisioning. There is no Controller service, so the driver never creates a cortex. Both volume styles name one that already exists.
Credentials are visible to host root. They reach aimem through systemd-run --setenv, so a root user on the node can read them with systemctl show. Scope credentials to a single cortex to limit what that exposes. A credentials-file approach is planned.
systemd nodes only. See What your nodes need.

On this page