Kubernetes

A pod is temporary: anything an agent writes to the container filesystem is lost when the pod goes. The AIStor Memory CSI node driver mounts a Memory Bucket into the pod as an ordinary directory. Everything written there is stored in AIStor, and the next pod that mounts the same Memory Bucket starts with it.

The driver does the privileged work on the node, so the workload pod needs no capabilities, no privilege escalation, and no access to /dev/fuse. It needs only an fsGroup, so its non-root user can write to the mount. The only privileged component is the driver DaemonSet, which you install once.

Before you start

• Nodes with /dev/fuse. Every node that runs workloads must have it.
• A Memory Bucket. The driver mounts an existing Memory Bucket; it never creates one. Create it with aimem create <name>. See Prerequisites.
• Credentials for that bucket. aimem credentials <name> creates credentials that reach only that bucket.
• An AIStor endpoint the pod network can reach. Each mount runs in a pod in the aimem-csi-mounters namespace, so endpointUrl must resolve and connect from the pod network. A Service DNS name such as http://aistor.aistor.svc.cluster.local:9000 works.

Install the driver

Save the manifest below as aimem-csi.yaml and replace RELEASE_TAG, in both places, with the release you are deploying. Do not use :latest: this image serves every mount on the node, so a floating tag lets a node restart change the filesystem under running pods.

apiVersion: storage.k8s.io/v1
kind: CSIDriver
metadata:
  name: csi.aimem.min.io
  labels:
    # Lets OpenShift admit an inline volume into a restricted namespace.
    security.openshift.io/csi-ephemeral-volume-profile: restricted
spec:
  # Node-only driver: no Controller service, nothing to attach, so kubelet
  # must not wait on an external-attacher.
  attachRequired: false
  podInfoOnMount: true
  # kubelet calls the driver again while a volume is mounted, and each call
  # brings the mount's credentials up to date.
  requiresRepublish: true
  volumeLifecycleModes:
    - Persistent
    - Ephemeral
  # The driver applies the pod's fsGroup itself; kubelet never chowns the bucket.
  fsGroupPolicy: None
---
apiVersion: v1
kind: ServiceAccount
metadata:
  name: aimem-csi-node-sa
  namespace: kube-system
---
# OpenShift admits the privileged DaemonSet only under the privileged SCC.
# On other clusters this grants nothing.
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: aimem-csi-node-scc
rules:
  - apiGroups: ["security.openshift.io"]
    resources: ["securitycontextconstraints"]
    resourceNames: ["privileged"]
    verbs: ["use"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: aimem-csi-node-scc
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: aimem-csi-node-scc
subjects:
  - kind: ServiceAccount
    name: aimem-csi-node-sa
    namespace: kube-system
---
# One pod per mounted volume runs here, unprivileged. The namespace enforces
# the restricted Pod Security profile.
apiVersion: v1
kind: Namespace
metadata:
  name: aimem-csi-mounters
  labels:
    pod-security.kubernetes.io/enforce: restricted
    pod-security.kubernetes.io/audit: restricted
    pod-security.kubernetes.io/warn: restricted
---
# The driver creates, reads, lists and deletes those pods, and nothing else.
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: aimem-csi-mounter
  namespace: aimem-csi-mounters
rules:
  - apiGroups: [""]
    resources: ["pods"]
    verbs: ["create", "get", "list", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: aimem-csi-mounter
  namespace: aimem-csi-mounters
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: aimem-csi-mounter
subjects:
  - kind: ServiceAccount
    name: aimem-csi-node-sa
    namespace: kube-system
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: aimem-csi-node
  namespace: kube-system
  labels:
    app: aimem-csi-node
spec:
  selector:
    matchLabels:
      app: aimem-csi-node
  template:
    metadata:
      labels:
        app: aimem-csi-node
    spec:
      serviceAccountName: aimem-csi-node-sa
      priorityClassName: system-node-critical
      tolerations:
        - operator: Exists
      containers:
        - name: aimem-csi-driver
          image: quay.io/minio/aistor/aimem-csi-driver:RELEASE_TAG
          args:
            - --endpoint=$(CSI_ENDPOINT)
            - --node-id=$(CSI_NODE_ID)
            # kubelet pins a node-critical pod's OOM score to -997 and never
            # evicts it for node pressure: a mounter killed takes its
            # workload's mount with it.
            - --mounter-priority-class=system-node-critical
          env:
            - name: CSI_ENDPOINT
              value: unix:///csi/csi.sock
            - name: CSI_NODE_ID
              valueFrom:
                fieldRef:
                  fieldPath: spec.nodeName
            - name: AIMEM_CSI_MOUNTER_IMAGE
              value: quay.io/minio/aistor/aimem-csi-driver:RELEASE_TAG
            - name: RUST_LOG
              value: info
          # Mounting FUSE takes privilege. The image's own user is non-root,
          # for the mounter pods, so the driver sets root here.
          securityContext:
            privileged: true
            runAsUser: 0
          livenessProbe:
            httpGet:
              path: /healthz
              port: healthz
            initialDelaySeconds: 10
            periodSeconds: 60
            timeoutSeconds: 3
          ports:
            - name: healthz
              containerPort: 9808
              protocol: TCP
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: kubelet-dir
              mountPath: /var/lib/kubelet
              # Bidirectional so the mount reaches workload pods, and the
              # driver reaches each mounter pod's emptyDir.
              mountPropagation: Bidirectional
            - name: host-sys-fs
              # Where the driver learns whether the node runs SELinux.
              mountPath: /host/sys/fs
              readOnly: true
        - name: node-driver-registrar
          image: registry.k8s.io/sig-storage/csi-node-driver-registrar:v2.13.0
          args:
            - --csi-address=/csi/csi.sock
            - --kubelet-registration-path=/var/lib/kubelet/plugins/csi.aimem.min.io/csi.sock
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
            - name: registration-dir
              mountPath: /registration
        - name: liveness-probe
          image: registry.k8s.io/sig-storage/livenessprobe:v2.16.0
          args:
            - --csi-address=/csi/csi.sock
            - --health-port=9808
          volumeMounts:
            - name: plugin-dir
              mountPath: /csi
      volumes:
        - name: plugin-dir
          hostPath:
            path: /var/lib/kubelet/plugins/csi.aimem.min.io/
            type: DirectoryOrCreate
        - name: registration-dir
          hostPath:
            path: /var/lib/kubelet/plugins_registry/
            type: Directory
        - name: kubelet-dir
          hostPath:
            path: /var/lib/kubelet
            type: Directory
        - name: host-sys-fs
          hostPath:
            path: /sys/fs
            type: Directory

Apply it and wait for the driver to start on every node:

kubectl apply -f aimem-csi.yaml
kubectl -n kube-system rollout status daemonset/aimem-csi-node

The driver needs no cluster-wide permissions. Its one Role lets it manage pods in aimem-csi-mounters, where it starts one pod for each mounted volume.

Mount a Memory Bucket in a pod

Use an inline volume: the pod names the Memory Bucket, and the mount lasts as long as the pod. This suits one mount per unit of work, such as an agent sandbox or a CI job, and leaves nothing to clean up.

Save the following as workspace-pod.yaml. It creates a Secret with the credentials, and a pod that mounts the bucket at /workspace. Replace my-project, the endpoint, and the keys:

apiVersion: v1
kind: Secret
metadata:
  name: aimem-inline-credentials
type: Opaque
stringData:
  # Use credentials scoped to one Memory Bucket where you can:
  # `aimem credentials <name>`.
  accessKeyID: REPLACE_ME
  secretAccessKey: REPLACE_ME
  # sessionToken: REPLACE_ME   # required for STS credentials
---
apiVersion: v1
kind: Pod
metadata:
  name: aimem-inline-workspace
spec:
  restartPolicy: Never
  securityContext:
    fsGroup: 1000
  containers:
    - name: workload
      image: ubuntu:24.04
      command: ["bash", "-lc"]
      args:
        - |
          set -euo pipefail
          ls -la /workspace
          # This write is durable: it lands in the Memory Bucket, not in the pod.
          printf 'hello from %s\n' "$(hostname)" > /workspace/probe.txt
          sync
      volumeMounts:
        - name: workspace
          mountPath: /workspace
      securityContext:
        allowPrivilegeEscalation: false
        runAsNonRoot: true
        runAsUser: 1000
        capabilities:
          drop: ["ALL"]
        seccompProfile:
          type: RuntimeDefault
  volumes:
    - name: workspace
      csi:
        driver: csi.aimem.min.io
        volumeAttributes:
          bucketName: my-project
          # Mount a sub-prefix instead of the whole Memory Bucket. Leading and
          # trailing slashes are optional.
          # prefix: workspaces/alice/app-7/
          endpointUrl: https://aistor.example.com:9000
        nodePublishSecretRef:
          name: aimem-inline-credentials

bucketName is required on an inline volume. Apply the file, then check the pod's output:

kubectl apply -f workspace-pod.yaml
kubectl logs aimem-inline-workspace

The pod writes probe.txt to the Memory Bucket. The next pod that mounts my-project sees it.

For a long-running pod, such as an agent, also set mountPropagation: HostToContainer on the volume mount. See When the pod serving a mount dies.

Volume attributes

Set these under volumeAttributes.

Attributes the driver rejects

The driver refuses a volume that sets any of these, with InvalidArgument naming the attribute. It refuses instead of ignoring them, so a volume never mounts with settings other than the ones it declares.

• allowOther, allowRoot, and virtualHostStyle: the driver does not support them. Every mount already allows other users; see Non-root pods and OpenShift.
• localDir: use cacheSize instead.
• tlsCaFile: put the PEM text in caCert instead.

The cache

aimem keeps open files and cached reads in an emptyDir of the pod that serves the mount, on the node's disk. cacheSize sets that emptyDir's size limit. aimem keeps its own use to 90% of it, because kubelet also counts filesystem blocks and directories, and evicts the pod, and its mount, once the directory passes the limit. A write that does not fit fails with ENOSPC.

A volume can ask for at most the driver's --max-cache-size, which defaults to 100Gi.

Credentials

The driver gives each mount its credentials in a file in memory, readable only by the pod that serves the mount. The credentials never appear in a pod spec, environment, or command line. The file is removed when the volume is unmounted.

kubelet calls the driver again from time to time while a volume is mounted, and each call updates the credentials. aimem switches to new credentials within a few seconds, so a mount can outlive the session it started with. Use sessions of an hour or more: a session that expires before kubelet's next call leaves the mount unable to reach the bucket until that call arrives.

From a Secret

Name the Secret with nodePublishSecretRef. It holds these keys:

To rotate credentials, update the Secret. A running mount picks up the new keys on kubelet's next call, without a remount.

Other ways to mount and authenticate

Mount with a PersistentVolume

Use a PersistentVolume when a cluster admin, not the pod author, should choose the bucket and its settings, or for a long-lived, shared workspace. The admin creates the volume and a claim; the pod uses the claim. volumeHandle names the Memory Bucket when bucketName is not set. First create the Secret aimem-credentials in kube-system, with the keys in From a Secret.

apiVersion: v1
kind: PersistentVolume
metadata:
  name: aimem-my-project
spec:
  capacity:
    storage: 1Ti # Ignored by the driver; Kubernetes requires a value.
  accessModes: ["ReadWriteMany"]
  persistentVolumeReclaimPolicy: Retain
  storageClassName: "" # Static: bind by claim, not by class.
  csi:
    driver: csi.aimem.min.io
    volumeHandle: my-project
    volumeAttributes:
      endpointUrl: https://aistor.example.com:9000
      # The mount's staging and read cache. Defaults to 10Gi.
      cacheSize: 20Gi
    nodePublishSecretRef:
      name: aimem-credentials
      namespace: kube-system
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: aimem-my-project
spec:
  accessModes: ["ReadWriteMany"]
  storageClassName: ""
  resources:
    requests:
      storage: 1Ti
  volumeName: aimem-my-project

In the pod, use the claim with persistentVolumeClaim.claimName: aimem-my-project.

Credentials from the pod's service account

Instead of a Secret, the driver can exchange the pod's service-account token for a session at an endpoint you run. A volume with no nodePublishSecretRef uses the exchange.

1. Run an endpoint that accepts this request and returns a session:

   POST <url>
   Authorization: Bearer <service-account token>
   {"bucket": "...", "prefix": "...", "generation": "..."}
   
   200 {"access_key_id": "...", "secret_access_key": "...",
        "session_token": "...", "expiration": "<RFC 3339>"}

2. Add these arguments to the aimem-csi-driver container (or set AIMEM_CSI_TOKEN_EXCHANGE_URL and AIMEM_CSI_TOKEN_AUDIENCE):

   args:
     - --token-exchange-url=https://sessions.example.com/exchange
     - --token-audience=aimem

3. Ask kubelet for a token of that audience in the CSIDriver spec:

   tokenRequests:
     - audience: aimem
       expirationSeconds: 3600

4. Leave nodePublishSecretRef off the volume, or point it at a Secret that holds only a generation key (below).

The driver exchanges the token again when half the session's lifetime has passed. A volume with neither a Secret nor an exchange fails to mount, with an error naming the missing nodePublishSecretRef.

Moving a mount to a new session at once. Put a generation key in the volume's Secret. kubelet passes it on every call, and the driver sends it with the exchange. When the value changes, the driver exchanges again at once. Updating the pod too, for example an annotation, makes kubelet call within about a second. If the Secret also holds accessKeyID and secretAccessKey, those keys are used and generation is ignored. The Secret must exist before the pod starts and for as long as it runs.

A private CA. By default the driver checks an https exchange endpoint against the public web roots. For a private CA, use one of these (not both):

• --token-exchange-ca-file (AIMEM_CSI_TOKEN_EXCHANGE_CA_FILE): a PEM file mounted into the driver container.
• --token-exchange-ca-configmap (AIMEM_CSI_TOKEN_EXCHANGE_CA_CONFIGMAP): a ConfigMap of that name in the namespace of the pod being mounted. The driver reads the PEM from its ca.crt key, or from the key named by --token-exchange-ca-key (AIMEM_CSI_TOKEN_EXCHANGE_CA_KEY). A changed CA takes effect within five minutes. A missing or invalid ConfigMap fails mounts in that namespace only; the driver never falls back to the public roots.

To read the ConfigMap, the driver needs this ClusterRole. Set resourceNames to the name you pass to --token-exchange-ca-configmap:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: aimem-csi-node-exchange-ca
rules:
  - apiGroups: [""]
    resources: ["configmaps"]
    resourceNames: ["sessions-ca"]
    verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: aimem-csi-node-exchange-ca
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: aimem-csi-node-exchange-ca
subjects:
  - kind: ServiceAccount
    name: aimem-csi-node-sa
    namespace: kube-system

One endpoint per namespace. On a cluster shared by several tenants, put {namespace} in the URL. The driver replaces it with the namespace of the pod being mounted:

args:
  - --token-exchange-url=https://sessions.{namespace}.svc.cluster.local:8443/exchange
  - --token-audience=aimem
  - --token-exchange-ca-configmap=sessions-ca

This needs podInfoOnMount: true in the CSIDriver, as in the manifest above. Without it, a {namespace} URL or a ConfigMap CA refuses every mount. A volume attribute cannot change the namespace.

Restrict network access to AIStor

Every mount connects from a pod labeled app: aimem-csi-mounter in the aimem-csi-mounters namespace. A NetworkPolicy in front of AIStor can admit these mounts and nothing else on the pod network:

apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: aimem-mounts
  namespace: aistor
spec:
  podSelector:
    matchLabels:
      app: aistor
  policyTypes: ["Ingress"]
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              kubernetes.io/metadata.name: aimem-csi-mounters
          podSelector:
            matchLabels:
              app: aimem-csi-mounter
      ports:
        - port: 9000

Add rules for AIStor's other clients. Your network plugin must enforce NetworkPolicy.

Non-root pods and OpenShift

aimem runs as a non-root user in the pod that serves the mount. Two settings let a non-root workload pod use the mount:

• Every mount allows other users. Other users on the node still cannot reach it, because kubelet keeps each pod's volume directory private to root.
• The pod's fsGroup owns the mount. kubelet passes the pod's fsGroup to the driver instead of changing ownership of every object. Every file and directory reads as owned by that group, with the group's permissions copied from the owner's. Any user in the pod can then read and write what another user created. A mode set with chmod still applies, so chmod 600 on an SSH key or a .pgpass file keeps it private to its owner.

A non-root pod that sets no fsGroup can read the mount but not write to it.

Files the workload did not create

A file created through the mount records its creator's user ID. A file that records none, such as an object written through S3, reads as owned by the user of the pod that serves the mount. The workload then cannot chmod it: chmod fails with EPERM. Set the uid and gid volume attributes to the workload's user and group to show such files as theirs:

volumeAttributes:
  bucketName: my-project
  uid: "1000"
  gid: "1000"

These attributes change only the owner shown. The pod that serves the mount keeps its own unprivileged user. 0 and non-numeric values are refused.

chmod needs only uid. When the pod sets an fsGroup, every entry reads as owned by that group whatever gid says, so gid matters only in a pod without one.

OpenShift

• The default restricted-v2 SCC always sets an fsGroup and allows CSI volumes, so a workload pod needs no custom SCC.
• The ClusterRoleBinding in the manifest lets the driver DaemonSet run under the privileged SCC. Workload pods and the pods that serve mounts do not use it; they run under restricted-v2.
• The security.openshift.io/csi-ephemeral-volume-profile: restricted label on the CSIDriver allows inline volumes in namespaces that enforce the restricted profile.
• On a node with SELinux enabled, the driver labels every mount container_file_t, so a confined container can use it.

How a mount comes up and goes away

When a pod with an AIStor Memory volume starts, the driver starts a pod for that volume in aimem-csi-mounters, on the same node. That pod runs aimem as a non-root user with every capability dropped, no host path, and no host network. Its annotation csi.aimem.min.io/target-path names the volume it serves. The driver mounts the volume on the node and hands it to that pod, which serves it.

The kernel holds the mount, so restarting or upgrading the driver DaemonSet does not interrupt live mounts.

When the workload pod stops, the driver unmounts the volume, lets aimem flush and exit, and deletes the serving pod.

When the pod serving a mount dies

If aimem exits, or its pod is killed, while the workload runs, the mount fails with ENOTCONN. On kubelet's next call, the driver starts a new pod and mounts the volume again.

A container sees the new mount without a restart only when it mounts the volume with mountPropagation: HostToContainer. Kubernetes allows this for an unprivileged pod. Set it for a long-running pod, such as an agent:

volumeMounts:
  - name: workspace
    mountPath: /workspace
    mountPropagation: HostToContainer

Any other container fails until it restarts. Files that were open when the pod died stay broken either way.

To see the pods serving mounts, and why one failed:

kubectl -n aimem-csi-mounters get pods -o wide
kubectl -n aimem-csi-mounters logs <pod>
kubectl -n kube-system logs -l app=aimem-csi-node -c aimem-csi-driver

Draining a node

kubectl drain refuses a node while pods serving mounts run on it, because they have no controller and use emptyDir. Drain with:

kubectl drain <node> --ignore-daemonsets --delete-emptydir-data --force

A serving pod that is asked to stop keeps serving until its workload is gone and the volume is unmounted, for up to its 60-second grace period. Keep the workload's terminationGracePeriodSeconds under 60, or stop its writes before you drain.

Uninstalling

Pods serving mounts have no owner, so deleting the DaemonSet alone leaves them running. Delete the workloads first, so the driver stops their serving pods. Then run kubectl delete -f aimem-csi.yaml. This also deletes the aimem-csi-mounters namespace: any serving pod still in it stops, and its workload's mount fails.

Upgrading from a driver that mounted with systemd

Earlier driver versions ran each mount on the node under systemd. After you upgrade, those mounts keep working, but their credentials are no longer renewed. Restart each such workload to move its volume to a serving pod.

Limitations

• No dynamic provisioning. The driver never creates a Memory Bucket. Every volume names one that already exists.
• A dead mount needs a container restart, unless the workload mounts the volume with mountPropagation: HostToContainer. See When the pod serving a mount dies.
• A non-root pod needs an fsGroup to write. See Non-root pods and OpenShift.