Operations

AIStor Memory is the memory your agents keep — one mount of a MinIO AIStor Memory Bucket that an agent both works in and remembers through. This page is the production playbook for the operators who run that mount: how to bring a Memory Bucket up inside each sandbox runtime, at boot, and as a daemon; how to observe it; how to seed the Vault that leases secrets into an agent; and how to recover when something goes wrong.

Memory takes three forms on one Memory Bucket, and each has an operational surface here. The Workspace is the mount you stand up per runtime (see the runtime table). Long-term memory — the memory files an agent looks for, warmed onto local disk on mount — is best-effort at startup and observable through the metrics below. The Vault is the Memory Bucket's own secret store, which you seed and then provision into an agent at launch (see Managing the Vault).

How the mount is configured

Every example on this page uses a --config YAML file so the mount is reviewable and diffable alongside the rest of your infrastructure config. The equivalent long-form CLI flags exist on the binary but are intentionally not documented here.

aimem mounts only a Memory Bucket, never a plain S3 bucket. Before a mount can come up, four things must be true: the target is a Memory Bucket (create one with aimem bucket create), an explicit endpoint_url is set, real signing credentials are present (AIMEM_ACCESS_KEY / AIMEM_SECRET_KEY), and the AIStor server is on a paid/Enterprise tier that serves the Memory API. Anonymous and default-chain mounts are refused. See Prerequisites below.

The local staging directory is carried by the local: key inside --config — the canonical form, shown in every YAML example on this page. The --local <DIR> CLI flag and the AIMEM_LOCAL environment variable exist as reference alternatives, and at least one of the three is required (CLI/env win when more than one is set). See Local staging directory.

Prerequisites

aimem mounts a Memory Bucket, not a plain S3 bucket, and the mount gate verifies every precondition before the FUSE session comes up. If any of these is missing the mount fails with an explicit message:

• A Memory Bucket to mount. Create one launcher-side with the Memory Bucket command family (all verbs run against the Memory API with admin credentials):

  # Create a new Memory Bucket (encrypted + compressed at rest).
  aimem bucket create my-memory-bucket --endpoint-url http://aistor:9000
  
  # Or convert an existing regular bucket in place. Objects written
  # before conversion are NOT retro-compressed/encrypted/stamped;
  # only post-conversion writes honor the Memory Bucket contract.
  aimem bucket convert my-bucket
  
  # Inspect and manage.
  aimem bucket list
  aimem bucket get my-memory-bucket
  aimem bucket delete my-memory-bucket --force   # --force deletes a non-empty Memory Bucket
  
  # Mint short-lived, Memory Bucket-scoped credentials (STS AssumeRole with an
  # inline session policy) for handing to an agent.
  aimem bucket credentials my-memory-bucket --duration 3600

  aimem bucket create optionally pins a KMS key with --kms-key-id (omitted → the server default key). There is no update verb — encryption/compression/KMS properties are fixed at create/convert time.

• An explicit endpoint URL. Pass endpoint_url in the config (or --endpoint-url / AIMEM_ENDPOINT_URL to the Memory Bucket commands). The gate cannot verify a Memory Bucket without one.

• Real signing credentials. AIMEM_ACCESS_KEY / AIMEM_SECRET_KEY (plus optional AIMEM_SESSION_TOKEN). Anonymous or default-chain mounts are refused — the gate needs a real credential to probe the Memory API. AIMEM_TLS_CA_FILE supplies a custom CA; SSL_CERT_FILE / SSL_CERT_DIR are also honored by the TLS loader.

• A paid/Enterprise-tier AIStor server. The entire Memory API family (Memory Bucket, secrets, search) is an Enterprise capability. On a free-tier server the gate reports that the credentials lack memory:GetCortex or the server is not on a paid tier.

Running AIStor Memory inside a sandbox provider

The per-runtime Integrations pages ship full templates (Dockerfile fragments, devfiles, devcontainer Features, Python helpers). If you'd rather assemble it yourself, the table below is the minimum each runtime needs: drop a mount.yaml at the right path, mount before the agent runs, and unmount on session teardown.

Minimum configuration

# /etc/aimem/mount.yaml
version: "1.0"
local: /var/cache/aimem
endpoint_url:
  - http://aistor{1...4}:9000
region: us-east-1
agent: auto
metadata_ttl: indefinite

# Credentials are supplied via the environment only — never on the command line.
export AIMEM_ACCESS_KEY=... AIMEM_SECRET_KEY=...

aimem my-memory-bucket /workspace --config /etc/aimem/mount.yaml

The version: "1.0" key is required — a config file without it (or with any other value) is rejected at startup. It must be a quoted string; bare version: 1.0 parses as a float and fails. The local: key carries the mandatory staging directory; see Local staging directory below for the CLI and env alternatives. The positional my-memory-bucket must name an existing Memory Bucket, not a plain bucket — see Prerequisites.

Add otlp_endpoint, log_directory, etc. as needed — the full key list is in Configuration.

fstab

To bring the mount up at boot on a regular Linux host, invoke AIStor Memory from /etc/fstab via mount.fuse3. Linux's mount infrastructure passes the bucket as the source, the mountpoint as the target, and forwards everything after -o as a single comma-separated options string:

my-memory-bucket /workspace fuse.aimem _netdev,config=/etc/aimem/mount.yaml 0 0

The config= option is the YAML form — point it at the same file you'd use for an interactive mount, and everything else (including the local: staging directory) lives there. If you prefer to keep the staging directory out of the YAML, the fstab option local=<dir> sets it inline. The legacy spellings staging_dir=<dir> and staging-dir=<dir> are rewritten to the same flag. Quoting rules: backslash-escape literal commas, backslashes, and double quotes inside any value.

Daemonization

By default, aimem forks into the background after the FUSE session is up. The parent process waits up to 30 seconds for the child to signal "mount succeeded" through a pipe. On success, the parent prints the mount path and exits. On failure, it reports that the mount failed; the child writes the underlying error to its own log.

To keep aimem in the foreground (useful inside containers, systemd, or under strace), set foreground: true in the YAML.

Logging

Logs are emitted via the tracing crate. Defaults: warn for everything, info for the aimem target.

# mount.yaml — logging block
debug: false # bumps default filter to `debug` when true
no_log: false # disables logging entirely
log_directory: /var/log/aimem
log_metrics: false # emit metric records inline with logs

The AIMEM_LOG env var overrides the YAML-derived default filter when set, using standard tracing EnvFilter syntax (e.g. AIMEM_LOG=aimem=debug,warn).

OpenTelemetry metrics

# mount.yaml — telemetry block
otlp_endpoint: http://otel-collector:4318
otlp_export_interval: 60

The mount exports metrics over OTLP/HTTP at the given endpoint. The metric set includes:

• fuse.request_latency and fuse.request_errors keyed by FUSE op.
• fuse.io_size keyed by read / write.
• fuse.idle_threads / fuse.total_threads.
• fs.current_handles{type=read|staged_read|write}, fs.inodes, fs.inode_kinds{kind=…}.
• fuse.cache_hit.
• prefetch.reset_state.
• s3.request_count / s3.request_errors / s3.request_canceled / s3.request_total_latency / s3.request_first_byte_latency — the S3 client layer, keyed by request type and HTTP status.
• process.memory_usage.

The metric names above are the full set emitted by the mount.

Agent telemetry

The metrics above describe the mount. The agent's own telemetry — what it did, in traces, logs and metrics — is captured separately by the aimem bio serve collector, which stores it in the same Memory Bucket.

Run one collector per agent, next to the agent, and give the harness the endpoint it publishes:

export AIMEM_ENDPOINT_URL=https://aistor.example.com:9000

# The credential the Memory Bucket issued when the agent was created. It may
# not read the agent record, so name the session-id attribute the record's
# capture_descriptors declare.
export AIMEM_ACCESS_KEY=... AIMEM_SECRET_KEY=...
aimem bio serve "${BUCKET}" --agent checkout-bot \
  --session-key session.id \
  --endpoint-file /run/agent/otlp-endpoint &

# The collector publishes its endpoint once it is listening, so wait for the
# file rather than racing it, and start the agent only after it appears.
for _ in $(seq 1 30); do [ -s /run/agent/otlp-endpoint ] && break; sleep 1; done
[ -s /run/agent/otlp-endpoint ] || { echo "collector never published an endpoint"; exit 1; }
export OTEL_EXPORTER_OTLP_ENDPOINT="$(cat /run/agent/otlp-endpoint)"

The collector binds loopback, so it has to share a network namespace with the agent — the same pod, or the same container. Give it the connection settings from Connection flags; they are read from the environment, and the credential belongs to the collector, not to the agent.

There is no separate provisioning step. The Memory Bucket leaves an agent's telemetry tables to the collector and grants the agent's credential just enough to create them in its own namespace. Before it publishes its endpoint, serve creates that namespace, stamped with the agent id, and the agent's tables, and checks any that already exist. It exits instead if the namespace belongs to another agent or a table does not match the schema it writes. It never drops a table another collector created: when a create collides with a table that is still becoming visible, it waits for it. Only a registration that stays unloadable for the whole wait, which AIStor can leave behind when a Memory Bucket is deleted and recreated under the same name, is dropped and created again. That drop removes the registration, never the data files. Deleting the agent reclaims the namespace and its tables within a minute or two: the server removes them on its next background reclaim pass.

Four operational properties are worth planning around:

• It fails loudly, not quietly. A wrong Memory Bucket or an under-privileged credential makes every write fail, and the collector reports each failure rather than reporting success. Once it is clear the writes will not start succeeding it stops accepting and exits non-zero, so the problem cannot be mistaken for an idle agent. Run it under a supervisor that restarts it and reports that exit.
• An acknowledged payload survives a crash. The collector answers 200 only once the payload is written to its spool and the write is on disk. A payload leaves the spool only when it is committed or quarantined, so if the collector is killed or exits first, the next start replays it. The spool lives beside the endpoint file (<endpoint-file>.spool) unless --spool-dir or AIMEM_OTLP_SPOOL_DIR names another place. It is as durable as the filesystem it is on: a tmpfs such as /run keeps it across a crash but not a reboot. Its files hold prompts and tool output, so it is readable only by the collector's user.
• It holds a slow backend's backlog on disk instead of refusing. When the Memory Bucket is slow and memory fills up, new payloads wait in the spool, so a slow backend no longer produces 503s. The collector still answers 503 with Retry-After when the spool is full (1 GiB or 100,000 payloads by default, AIMEM_OTLP_SPOOL_MAX_BYTES and AIMEM_OTLP_SPOOL_MAX_COUNT), when a spool write fails, or when it cannot read the request body. It has not taken the payload in any of those cases, so an exporter must retry every retryable 503 to deliver it.
• It is exactly-once per table and raw payload bytes. A replay after a crash, or a retry after a lost commit response, checks each table before writing, so a payload that had already landed in some tables is not written to them again. Quarantine replay deliberately writes rows again when an operator asks it to.

Scoping a collector to one agent

The collector stores every payload under the agent named by --agent. A payload-supplied aimem.agent.id remains in the attribute map like any other sender field; it does not route the write.

Run the collector with the credential the Memory Bucket issues when the agent is created. It is already scoped to that agent's telemetry namespace, and the response that issues it names the namespace.

To build a credential by hand instead, it needs two grants, and the split matters. Reading the warehouse config is a warehouse-level operation with no namespace to test, so it cannot carry the namespace condition. Everything the collector does in the agent's namespace can: it creates the namespace and its tables on first write, loads and commits to them, and drops a table registration that has stayed unloadable (see above). The direct S3 data-file statement below grants object access to the whole Memory Bucket. Use the AIStor Tables data-file variant after the policy if the data-file permissions must be agent-isolated too.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketWideTelemetryDataFiles",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": ["arn:aws:s3:::MY-MEMORY-BUCKET/*"]
    },
    {
      "Sid": "ReadTheCatalog",
      "Effect": "Allow",
      "Action": ["s3tables:GetConfig"],
      "Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET"]
    },
    {
      "Sid": "ThisAgentsNamespaceOnly",
      "Effect": "Allow",
      "Action": [
        "s3tables:CreateNamespace",
        "s3tables:GetNamespace",
        "s3tables:CreateTable",
        "s3tables:GetTable",
        "s3tables:UpdateTable",
        "s3tables:DeleteTable"
      ],
      "Resource": [
        "arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET",
        "arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET/table/*"
      ],
      "Condition": { "StringEquals": { "s3tables:namespace": "MY-AGENT" } }
    }
  ]
}

Replace MY-MEMORY-BUCKET and MY-AGENT, attach it to the identity the collector runs as, and run one collector per agent with that agent's credential. Do not grant s3tables:UpdateNamespaceProperties: it would let a collector rewrite the aimem.agent.id stamp that marks which agent owns a namespace. For agent-isolated data-file access, replace the BucketWideTelemetryDataFiles statement with s3tables:GetTableData and s3tables:PutTableData on the same table resource and namespace condition as ThisAgentsNamespaceOnly.

Reading telemetry outside AIStor Memory

AIStor Memory writes Iceberg v3 tables with nanosecond OTLP timestamps (timestamptz_ns). Supporting Iceberg v2 is not a goal, and the schemas are not weakened for engines that cannot map that type.

PyIceberg works with the AIStor REST catalog and SigV4. Install it with the SigV4 extra, pip install "pyiceberg[pyarrow,rest-sigv4]". The client.* credentials sign the catalog requests and read the data files:

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "aimem",
    type="rest",
    uri="https://aistor.example.com:9000/_iceberg",
    warehouse="MY-MEMORY-BUCKET",
    **{
        "rest.sigv4-enabled": "true",
        "rest.signing-name": "s3tables",
        "s3.endpoint": "https://aistor.example.com:9000",
        "client.region": "us-east-1",
        "client.access-key-id": "...",
        "client.secret-access-key": "...",
    },
)

table = catalog.load_table(("MY-AGENT", "otel_spans"))
arrow_table = table.scan(row_filter="session_id = 'my-session'").to_arrow()

A filter on session_id or any other string column works. A filter on a timestamp column fails with TypeError: Cannot convert ... into timestamptz_ns, because PyIceberg cannot bind a value to that type yet. Filter on the session, then on time in Arrow.

A read-only external credential needs only the read half of the collector policy. The direct S3 statement is Memory Bucket-wide; use s3tables:GetTableData with the namespace condition when external-reader data-file permissions also need to be agent-isolated.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketWideTelemetryDataFilesRead",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::MY-MEMORY-BUCKET/*"]
    },
    {
      "Sid": "ReadTheCatalog",
      "Effect": "Allow",
      "Action": ["s3tables:GetConfig"],
      "Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET"]
    },
    {
      "Sid": "ThisAgentsTablesReadOnly",
      "Effect": "Allow",
      "Action": ["s3tables:GetTable"],
      "Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET/table/*"],
      "Condition": { "StringEquals": { "s3tables:namespace": "MY-AGENT" } }
    }
  ]
}

DuckDB and Spark can list the tables but cannot read their rows, because their Iceberg readers do not map timestamptz_ns yet (DuckDB 1.5.5 and Spark 3.5.6 with Iceberg 1.9.2 were checked).

For those engines, export the rows to Parquet with aimem bio export. The export writes each timestamp in microseconds, the unit DuckDB and Spark map, and keeps the exact value beside it as an integer <column>_unix_nano, so no precision is lost:

aimem bio export "${BUCKET}" --agent checkout-bot --table otel_spans \
  --output spans.parquet
duckdb -c "SELECT span_name, start_time FROM 'spans.parquet' ORDER BY start_time"

Spark reads the same file as timestamps with every microsecond:

spark.read.parquet("spans.parquet").where("session_id = 'my-session'")

The live test suite reads the export with DuckDB 1.5.5 and Spark 3.5.6, and the tables with PyIceberg 0.12.0 configured as above. The export reads the table with the same credential as bio query.

Local staging directory

The local staging directory is required. You can supply it three ways, and at least one must be present:

• the local: key in the --config YAML (the canonical form used throughout this page);
• the --local <DIR> CLI flag;
• the AIMEM_LOCAL environment variable.

If none of them is set the mount is rejected with a local staging directory is required; set --local <DIR>, the AIMEM_LOCAL env var, or the local: key in --config. When more than one is set, the CLI flag and env var take precedence over the config key. There is no auto-detection.

Point it at fast local storage (e.g. NVMe) with room for the open-write working set — each file opened for writing gets a staging file here. The chosen path is logged at mount.

The .aimem-lostfound/ prefix

POSIX keeps a file readable through every descriptor that still holds it open, even after its name is gone. So when something deletes a file, or renames another file over it, while a reader still has it open, AIStor Memory moves that object to a key under .aimem-lostfound/ rather than destroying it. The reader keeps reading. This is the object-store equivalent of lost+found on a local filesystem.

The move is a server-side rename, so no data is copied and the cost is the same whatever the file weighs. On a backend with no server-side rename, AIStor Memory copies the bytes into the local staging directory instead, and nothing appears under the prefix.

The mount hides the prefix. It is absent from a directory listing of the mount, and you cannot create a file with that name. Other S3 clients do see it: it is an ordinary prefix inside the path the mount points at, so it appears in mc ls, counts toward stored capacity, and is matched by bucket lifecycle rules.

When objects appear, and when they go

An object lands there only when a file is deleted or renamed over while a descriptor still holds it open. It is removed as soon as the last descriptor closes, which is normally within milliseconds — a listing rarely catches one.

An object outlives that only when a mount exits without closing its descriptors, after a crash or a kill -9. Nothing is left to reclaim it, so the next mount sweeps the prefix in the background and deletes everything older than 24 hours. The age check is deliberate: a recent object may belong to a different mount that is still serving it to a reader.

Two consequences to plan for:

• A mount that crashes and restarts repeatedly can accumulate up to a day of these objects before a sweep collects them.
• The sweep runs at mount time. A mount that stays up for months does not sweep again.

Managing the Vault: secrets in a Memory Bucket

The Vault is the third form of memory — the authority an approved agent receives. A Memory Bucket is itself a secret store: it holds secret values encrypted server-side and exposes them through the Memory API. Use the aimem secret subcommands to seed and rotate the secrets that aimem secret provision --bucket later leases into an agent. All four commands take the Memory Bucket name and connect with the workspace credentials (the connection flags, or the AIMEM_* env vars):

# Store or rotate a secret (value read from stdin — preferred over --value,
# which is visible in the process list and shell history).
printf '%s' "$OPENAI_KEY" | aimem secret put my-memory-bucket openai

# Read a secret's plaintext value back.
aimem secret get my-memory-bucket openai

# List the secret names held in the Memory Bucket.
aimem secret list my-memory-bucket

# Delete a secret.
aimem secret delete my-memory-bucket openai

aimem does not store the secret bytes itself — it passes them through to the Memory Bucket, which envelope-encrypts them under KMS at rest (a plain S3 GET of the backing object returns ciphertext; only the Memory API GET returns server-decrypted plaintext). Each secret is capped at 1 MiB and versioned — every put bumps the version counter. These commands require the same connection prerequisites as the Memory Bucket family (explicit endpoint, real credentials, paid tier). To hand a stored secret to an agent at launch, use aimem secret provision with a matching --grant.

Unmounting

umount /workspace

If aimem daemonized, umount triggers a clean shutdown. The foreground binary also responds to SIGINT and SIGTERM by unmounting and exiting.

If a stale FUSE mount blocks unmount (e.g., after a crash), force it with fusermount3 -uz /workspace.

This is safe — the underlying bucket state is unaffected.

Troubleshooting

• Transport endpoint is not connected after a crash — stale mount; fusermount3 -uz <path> to clear it.

• AIStor Memory warmup never completes — warmup (preloading the agent's memory files into the local cache at mount time) is best-effort; failures are logged and the mount continues. Check that the agent-memory prefixes (e.g. CLAUDE.md) actually exist in the bucket; missing prefixes are silently skipped.

• Out of space in staging — staging holds a copy of every file currently open for write. Either increase the volume backing local, or close intermediate files between batches.

• Need to migrate the staging dir mid-run — unmount and remount with the new local. There's no live-migration path.

• aimem secret provision errors reaching the source — provisioning resolves granted secrets from exactly one of two mutually exclusive sources, and at least one --grant VAR=name is required:

  • --bucket <name> — the AIStor Memory-API secrets endpoint. It signs requests with the workspace credentials (AIMEM_ENDPOINT_URL, AIMEM_ACCESS_KEY, AIMEM_SECRET_KEY, plus optional AIMEM_REGION / AIMEM_SESSION_TOKEN).
  • --store-endpoint <url> (AIMEM_SECRET_STORE_ENDPOINT) — a generic HTTP secret store (HashiCorp Vault/OpenBao, a cloud secret manager). The bearer token is optional and defaults to empty; prefer the AIMEM_SECRET_STORE_TOKEN env var over the --store-token flag — CLI arguments can show up in process listings and shell history. AIMEM_SECRET_STORE_ENDPOINT is required only on this path (not when --bucket is used).

  Production endpoints must use HTTPS/TLS; plain HTTP is only for controlled local/test use. The fetch credential stays with the launcher and is never injected into the agent. The endpoint must be reachable from the launcher host, and the credential must be authorized for the granted secrets.

• unknown field 'X' at startup — unrecognised key in the YAML. YAML keys mirror the long-flag names with dashes converted to underscores; check the Configuration key reference.