Operations

AIStor Memory is the memory your agents keep — one mount of a MinIO AIStor Memory Bucket (a cortex) that an agent both works in and remembers through. This page is the production playbook for the operators who run that mount: how to bring a cortex up inside each sandbox runtime, at boot, and as a daemon; how to observe it; how to seed the Vault that leases secrets into an agent; and how to recover when something goes wrong.

Memory takes three forms on one cortex, and each has an operational surface here. The Workspace is the mount you stand up per runtime (see the runtime table). Long-term memory — the memory files an agent looks for, warmed onto local disk on mount — is best-effort at startup and observable through the metrics below. The Vault is the cortex's own secret store, which you seed and then provision into an agent at launch (see Managing the Vault).

How the mount is configured

Every example on this page uses a --config YAML file so the mount is reviewable and diffable alongside the rest of your infrastructure config. The equivalent long-form CLI flags exist on the binary but are intentionally not documented here.

aimem mounts only a Memory Bucket (cortex), never a plain S3 bucket. Before a mount can come up, four things must be true: the target is a cortex (create one with aimem cortex create), an explicit endpoint_url is set, real signing credentials are present (AIMEM_ACCESS_KEY / AIMEM_SECRET_KEY — anonymous or default-chain mounts are refused), and the AIStor server is on a paid/Enterprise tier that serves the Memory API. See Prerequisites below.

The local staging directory is carried by the local: key inside --config — the canonical form, shown in every YAML example on this page. The --local <DIR> CLI flag and the AIMEM_LOCAL environment variable exist as reference alternatives, and exactly one of the three is required. See Local staging directory.

Prerequisites

aimem mounts a cortex (Memory Bucket), not a plain S3 bucket, and the mount gate verifies every precondition before the FUSE session comes up. If any of these is missing the mount fails with an explicit message:

• A cortex to mount. Create one launcher-side with the cortex command family (all verbs run against the Memory API with admin credentials):

  # Create a new cortex (encrypted + compressed at rest).
  aimem cortex create my-cortex --endpoint-url http://aistor:9000
  
  # Or convert an existing regular bucket in place. Objects written
  # before conversion are NOT retro-compressed/encrypted/stamped;
  # only post-conversion writes honor the cortex contract.
  aimem cortex convert my-bucket
  
  # Inspect and manage.
  aimem cortex list
  aimem cortex get my-cortex
  aimem cortex delete my-cortex --force   # --force deletes a non-empty cortex
  
  # Mint short-lived, cortex-scoped credentials (STS AssumeRole with an
  # inline session policy) for handing to an agent.
  aimem cortex credentials my-cortex --duration 3600

  aimem cortex create optionally pins a KMS key with --kms-key-id (omitted → the server default key). There is no update verb — encryption/compression/KMS properties are fixed at create/convert time.

• An explicit endpoint URL. Pass endpoint_url in the config (or --endpoint-url / AIMEM_ENDPOINT_URL to the cortex commands). The gate cannot verify a cortex without one.

• Real signing credentials. AIMEM_ACCESS_KEY / AIMEM_SECRET_KEY (plus optional AIMEM_SESSION_TOKEN). Anonymous or default-chain mounts are refused — the gate needs a real credential to probe the Memory API. AIMEM_TLS_CA_FILE supplies a custom CA; SSL_CERT_FILE / SSL_CERT_DIR are also honored by the TLS loader.

• A paid/Enterprise-tier AIStor server. The entire Memory API family (cortex, secrets, search) is an Enterprise capability. On a free-tier server the gate reports that the credentials lack memory:GetCortex or the server is not on a paid tier.

Running AIStor Memory inside a sandbox provider

The per-runtime Integrations pages ship full templates (Dockerfile fragments, devfiles, devcontainer Features, Python helpers). If you'd rather assemble it yourself, the table below is the minimum each runtime needs — drop a mount.yaml at the right path, mount before the agent runs, unmount on session teardown.

Minimum configuration

# /etc/aimem/mount.yaml
version: "1.0"
local: /var/cache/aimem
endpoint_url:
  - http://aistor{1...4}:9000
region: us-east-1
agent: auto
metadata_ttl: indefinite

# Credentials are supplied via the environment only — never on the command line.
export AIMEM_ACCESS_KEY=... AIMEM_SECRET_KEY=...

aimem my-cortex /workspace --config /etc/aimem/mount.yaml

The version: "1.0" key is required — a config file without it (or with any other value) is rejected at startup. It must be a quoted string; bare version: 1.0 parses as a float and fails. The local: key carries the mandatory staging directory; see Local staging directory below for the CLI and env alternatives. The positional my-cortex must name an existing cortex, not a plain bucket — see Prerequisites.

Add otlp_endpoint, log_directory, etc. as needed — the full key list is in Configuration.

fstab

When you want the mount to come up at boot on a regular Linux host, AIStor Memory can be invoked from /etc/fstab via mount.fuse3. Linux's mount infrastructure passes the bucket as the source, the mountpoint as the target, and forwards everything after -o as a single comma-separated options string:

my-cortex /workspace fuse.aimem _netdev,config=/etc/aimem/mount.yaml 0 0

The config= option is the YAML form — point it at the same file you'd use for an interactive mount, and everything else (including the local: staging directory) lives there. If you prefer to keep the staging directory out of the YAML, the fstab option local=<dir> sets it inline; the legacy spellings staging_dir=<dir> and staging-dir=<dir> are rewritten to the same flag. Quoting rules: backslash-escape literal commas, backslashes, and double quotes inside any value.

Daemonization

By default, aimem forks into the background after the FUSE session is up. The parent process waits up to 30 seconds for the child to signal "mount succeeded" through a pipe; on success it prints the mount path and exits, on failure it reports that the mount failed (the child writes the underlying error to its own log).

To keep aimem in the foreground (useful inside containers, systemd, or under strace), set foreground: true in the YAML.

Logging

Logs are emitted via the tracing crate. Defaults: warn for everything, info for the aimem target.

# mount.yaml — logging block
debug: false # bumps default filter to `debug` when true
no_log: false # disables logging entirely
log_directory: /var/log/aimem
log_metrics: false # emit metric records inline with logs

The AIMEM_LOG env var overrides the YAML-derived default filter when set, using standard tracing EnvFilter syntax (e.g. AIMEM_LOG=aimem=debug,warn).

OpenTelemetry metrics

# mount.yaml — telemetry block
otlp_endpoint: http://otel-collector:4318
otlp_export_interval: 60

Exports metrics over OTLP/HTTP at the given endpoint. The metric set includes:

• fuse.request_latency and fuse.request_errors keyed by FUSE op.
• fuse.io_size keyed by read / write.
• fuse.idle_threads / fuse.total_threads.
• fs.current_handles{type=read|staged_read|write}, fs.inodes, fs.inode_kinds{kind=…}.
• fuse.cache_hit.
• prefetch.reset_state.
• s3.request_count / s3.request_errors / s3.request_canceled / s3.request_total_latency / s3.request_first_byte_latency — the S3 client layer, keyed by request type and HTTP status.
• process.memory_usage.

The metric names above are the full set emitted by the mount.

Local staging directory

The local staging directory is required. It can be supplied three ways, and at least one must be present (if more than one is set, they resolve by precedence — CLI/env win over the config file):

• the local: key in the --config YAML (the canonical form used throughout this page);
• the --local <DIR> CLI flag;
• the AIMEM_LOCAL environment variable.

If none of them is set the mount is rejected with a local staging directory is required; set --local <DIR>, the AIMEM_LOCAL env var, or the local: key in --config. When more than one is set, the CLI flag and env var take precedence over the config key. There is no auto-detection.

Point it at fast local storage (e.g. NVMe) with room for the open-write working set — each file opened for writing gets a staging file here. The chosen path is logged at mount.

Managing the Vault: secrets in a cortex

The Vault is the third form of memory — the authority an approved agent receives. A cortex is itself a secret store: it holds secret values encrypted server-side and exposes them through the Memory API. Use the aimem secret subcommands to seed and rotate the secrets that aimem secret provision --cortex later leases into an agent. All four commands take the cortex name and connect with the workspace credentials (the connection flags, or the AIMEM_* env vars):

# Store or rotate a secret (value read from stdin — preferred over --value,
# which is visible in the process list and shell history).
printf '%s' "$OPENAI_KEY" | aimem secret put my-cortex openai

# Read a secret's plaintext value back.
aimem secret get my-cortex openai

# List the secret names held in the cortex.
aimem secret list my-cortex

# Delete a secret.
aimem secret delete my-cortex openai

aimem does not store the secret bytes itself — it passes them through to the cortex, which envelope-encrypts them under KMS at rest (a plain S3 GET of the backing object returns ciphertext; only the Memory API GET returns server-decrypted plaintext). Each secret is capped at 1 MiB and versioned — every put bumps the version counter. These commands require the same connection prerequisites as the cortex family (explicit endpoint, real credentials, paid tier). To hand a stored secret to an agent at launch, use aimem secret provision with a matching --grant.

Unmounting

umount /workspace

If aimem daemonized, umount triggers a clean shutdown. The foreground binary also responds to SIGINT and SIGTERM by unmounting and exiting.

If a stale FUSE mount blocks unmount (e.g., after a crash), force it with fusermount3 -uz /workspace.

This is safe — the underlying bucket state is unaffected.

Troubleshooting

• Transport endpoint is not connected after a crash — stale mount; fusermount3 -uz <path> to clear it.

• AIStor Memory warmup never completes — warmup (preloading the agent's memory files into the local cache at mount time) is best-effort; failures are logged and the mount continues. Check that the agent-memory prefixes (e.g. CLAUDE.md) actually exist in the bucket; missing prefixes are silently skipped.

• Out of space in staging — staging holds a copy of every file currently open for write. Either increase the volume backing local, or close intermediate files between batches.

• Need to migrate the staging dir mid-run — unmount and remount with the new local. There's no live-migration path.

• aimem secret provision errors reaching the source — provisioning resolves granted secrets from exactly one of two mutually exclusive sources, and at least one --grant VAR=name is required:

  • --cortex <name> — the AIStor Memory-API secrets endpoint. It signs requests with the workspace credentials (AIMEM_ENDPOINT_URL, AIMEM_ACCESS_KEY, AIMEM_SECRET_KEY, plus optional AIMEM_REGION / AIMEM_SESSION_TOKEN).
  • --store-endpoint <url> (AIMEM_SECRET_STORE_ENDPOINT) — a generic HTTP secret store (HashiCorp Vault/OpenBao, a cloud secret manager). The bearer token is optional and defaults to empty; prefer the AIMEM_SECRET_STORE_TOKEN env var over the --store-token flag — CLI arguments can show up in process listings and shell history. AIMEM_SECRET_STORE_ENDPOINT is required only on this path (not when --cortex is used).

  Production endpoints must use HTTPS/TLS; plain HTTP is only for controlled local/test use. The fetch credential stays with the launcher and is never injected into the agent. The endpoint must be reachable from the launcher host, and the credential must be authorized for the granted secrets.

• unknown field 'X' at startup — unrecognised key in the YAML. YAML keys mirror the long-flag names with dashes converted to underscores; check the Configuration key reference.