AIStor Memory
Operate

Operations

Run AIStor Memory in production. Mount a Memory Bucket in each sandbox runtime or at boot, collect logs, metrics, and agent telemetry, store secrets in the Vault, and fix common problems.

This page is for operators who run AIStor Memory mounts. It covers what a mount needs, how to start one in each sandbox runtime or at boot, how to watch it, how to store secrets in the Vault, and how to fix common problems.

How the mount is configured

The examples on this page configure the mount with a --config YAML file. A file is easy to review and to keep with the rest of your infrastructure configuration. Credentials come from the environment, never from the file.

Before a mount starts, AIStor Memory checks that the target is a Memory Bucket it can reach and sign requests for. See Prerequisites.

Prerequisites

AIStor Memory mounts only a Memory Bucket, never a plain S3 bucket. Before it starts the mount, it asks the AIStor Memory API to confirm the bucket. If any condition below is not met, the mount stops with an error that names the problem. There is no option to skip the check.

  1. The bucket is a Memory Bucket. Create one with aimem create, or convert an existing bucket with aimem convert. If the target is a plain bucket, the mount fails with '<bucket>' is not a Memory Bucket.
  2. An endpoint is set. Set endpoint_url in the config file, or pass --endpoint-url. AIStor Memory needs the endpoint to check the bucket.
  3. Signing credentials are set. Set AIMEM_ACCESS_KEY and AIMEM_SECRET_KEY (and AIMEM_SESSION_TOKEN for temporary credentials), or set AIMEM_CREDENTIALS_FILE to a credentials file. Anonymous mounts and mounts that rely on a default credential chain are refused.
  4. The credentials can read the Memory Bucket, and AIStor is on a paid tier. The Memory API is an AIStor Enterprise feature. The credentials need the memory:GetCortex permission on the bucket. If either condition fails, the mount reports access denied by the Memory API (the credentials lack memory:GetCortex, or the server is not on a paid tier).
  5. A local staging directory is set. See Local staging directory.

When endpoint_url lists several endpoints, AIStor Memory tries each in turn. It retries network errors and HTTP 408, 429, and 5xx responses up to five times per endpoint. The check uses the same TLS settings as the mount (tls_ca_file, tls_server_name, insecure_skip_tls_verify). For a custom CA, set tls_ca_file or AIMEM_TLS_CA_FILE; AIStor Memory also honors SSL_CERT_FILE and SSL_CERT_DIR.

To create a Memory Bucket and give an agent credentials that reach only that bucket:

aimem create my-memory-bucket --endpoint-url https://aistor.example.com:9000
aimem credentials my-memory-bucket --duration 3600

aimem create compresses the bucket, and encrypts it when AIStor has a KMS. It uses the server's default KMS key unless you pass --kms-key-id. You cannot change these properties later. For every Memory Bucket command, see the CLI reference.

Running AIStor Memory inside a sandbox provider

The Integrations pages give a full setup for each runtime. To set one up yourself, put a mount.yaml where the runtime can read it, mount before the agent starts, and unmount when the session ends.

RuntimeWhere to put mount.yamlMount stepNotes
Vercel Sandbox/vercel/sandbox/aimem/mount.yamlaimem "$BUCKET" /workspace --config <yaml>Set local: in the YAML to a path on /vercel/sandbox.
DaytonaRuntime env vars or /etc/aimem/mount.yamlsandbox.process starts aimem-mountSee Daytona for the sandbox image recipe
GitHub Codespaces.devcontainer/aimem/mount.yamldevcontainer Feature postCreateCommandSee Codespaces
E2BBaked into the templatesb.commands.run starts aimem-mountSee E2B
OpenShellHost-side, before sandbox launchHost runs aimem; policy allows the pathSee OpenShell
Plain Linux VM/etc/aimem/mount.yamlsystemd unit or /etc/fstabSee fstab below

Minimum configuration

# /etc/aimem/mount.yaml
version: "1.0"
local: /var/cache/aimem
endpoint_url:
  - http://aistor{1...4}:9000
region: us-east-1
agent: auto
metadata_ttl: indefinite
# Set credentials in the environment, never on the command line.
export AIMEM_ACCESS_KEY=... AIMEM_SECRET_KEY=...

aimem my-memory-bucket /workspace --config /etc/aimem/mount.yaml
  • version: "1.0" is required, as a quoted string. Without quotes, YAML reads 1.0 as a number and the file is rejected.
  • local: sets the staging directory, which every mount needs.
  • my-memory-bucket must be an existing Memory Bucket.

For logging, metrics, and every other key, see Configuration.

fstab

To mount at boot on a Linux host, add a line to /etc/fstab. Linux runs mount.fuse3, which passes the bucket, the mountpoint, and the options after -o to AIStor Memory:

my-memory-bucket /workspace fuse.aimem _netdev,config=/etc/aimem/mount.yaml 0 0
  • config= points at the same YAML file you use for an interactive mount.
  • To set the staging directory in fstab instead of the YAML, add local=<dir>. The older spellings staging_dir=<dir> and staging-dir=<dir> also work.
  • Inside a value, put a backslash before a literal comma, backslash, or double quote.

Daemonization

By default, aimem starts the mount and then moves to the background. The starting process waits until the mount is up or has failed, however long start-up takes on a busy host. On success, it prints the mount path and exits. On failure, it reports that the mount failed; the background process writes the cause to its log.

To keep aimem in the foreground, for example in a container or under systemd, set foreground: true in the YAML.

Logging

By default, AIStor Memory logs warnings from every component and info messages from aimem itself. Logs go to syslog, or to files when you set log_directory. In the foreground, logs also go to stdout.

# mount.yaml: logging keys
debug: false # true logs everything at debug level
no_log: false # true turns logging off
log_directory: /var/log/aimem
log_metrics: false # true writes metric records to the log

To set the log filter directly, set AIMEM_LOG. It takes the tracing EnvFilter syntax, for example AIMEM_LOG=aimem=debug,warn, and overrides the YAML settings.

OpenTelemetry metrics

To export mount metrics over OTLP/HTTP, set an endpoint:

# mount.yaml: metrics keys
otlp_endpoint: http://otel-collector:4318
otlp_export_interval: 60

The metrics include:

  • fuse.request_latency and fuse.request_errors, by FUSE operation.
  • fuse.io_size, by read and write.
  • fuse.idle_threads and fuse.total_threads.
  • fs.current_handles{type=read|staged_read|write}, fs.inodes, and fs.inode_kinds{kind=…}.
  • fuse.cache_hit.
  • prefetch.reset_state.
  • s3.request_count, s3.request_errors, s3.request_canceled, s3.request_total_latency, and s3.request_first_byte_latency, by request type and HTTP status.
  • process.memory_usage.

Agent telemetry

The metrics above describe the mount. To record what the agent itself does, its traces, logs, and metrics, run the aimem bio serve collector. It stores the telemetry in the same Memory Bucket.

Run one collector for each agent, next to the agent, and point the agent's OpenTelemetry exporter at the endpoint the collector publishes:

export AIMEM_ENDPOINT_URL=https://aistor.example.com:9000

# Use the credential the Memory Bucket issued when the agent was created.
# It cannot read the agent record, so name the session-id attribute that the
# record's capture_descriptors declare.
export AIMEM_ACCESS_KEY=... AIMEM_SECRET_KEY=...
aimem bio serve "${BUCKET}" --agent checkout-bot \
  --session-key session.id \
  --endpoint-file /run/agent/otlp-endpoint &

# Wait for the collector to publish its endpoint, then start the agent.
for _ in $(seq 1 30); do [ -s /run/agent/otlp-endpoint ] && break; sleep 1; done
[ -s /run/agent/otlp-endpoint ] || { echo "collector never published an endpoint"; exit 1; }
export OTEL_EXPORTER_OTLP_ENDPOINT="$(cat /run/agent/otlp-endpoint)"

The collector listens only on loopback, so it must share a network namespace with the agent: the same pod or the same container. It reads the connection settings from the environment. The credential belongs to the collector; do not give it to the agent.

You do not need to create tables first. Before it publishes its endpoint, the collector creates the agent's namespace, or checks the one that exists, and exits if it belongs to another agent. It creates each table the first time it has rows for it, and checks a table that exists the first time it writes to it. A table that does not match its schema is not written. When you delete the agent, AIStor removes its namespace and tables within a minute or two.

Plan for these behaviors:

  • It fails loudly. If the Memory Bucket name is wrong or the credential lacks permissions, every write fails and the collector reports each failure. When it is clear the writes will not succeed, it stops accepting data and exits with a non-zero status. Run it under a supervisor that restarts it and reports the exit.
  • An acknowledged payload survives a crash. The collector answers 200 only after it writes the payload to disk in its spool. If the collector stops before it commits the payload, the next start replays it. The spool is at <endpoint-file>.spool, or where --spool-dir or AIMEM_OTLP_SPOOL_DIR says. It is only as durable as its filesystem: a tmpfs such as /run keeps it across a crash but not a reboot. The spool holds prompts and tool output, so only the collector's user can read it. A payload received more than 7 days ago is dropped instead of delivered, at the next start or before any export attempt; set AIMEM_OTLP_SPOOL_MAX_AGE_SECS to a shorter time, down to one hour.
  • A slow backend fills the spool, not the agent. When the Memory Bucket is slow, new payloads wait in the spool. The collector answers 503 with Retry-After only when the spool is full (1 GiB or 100,000 payloads by default; set AIMEM_OTLP_SPOOL_MAX_BYTES and AIMEM_OTLP_SPOOL_MAX_COUNT to change), when a spool write fails, or when it cannot read the request. In each case it has not kept the payload, so the exporter must retry every retryable 503.
  • Each payload is written once per table, within 24 hours. A replay after a crash, or a retry after a lost response, checks each table first, so rows are not duplicated. The check reads the table's snapshots from the last 24 hours, so a payload re-sent later than that is written again. Replaying the quarantine table with aimem bio quarantine replay writes rows again on purpose, for payloads quarantined in the last 7 days.
  • Old snapshots are expired. Each flush adds a snapshot to each table it writes to. About once an hour the collector removes those older than 24 hours, keeping the current one, and sets that retention as the table's history.expire.* properties. Expiry changes metadata only: it loses no row, and it does not reclaim storage.
  • Tables are maintained. When it first creates or loads one of the agent's tables, the collector gives it the maintenance Fabric gives its own tables: unreferenced-file removal (a file no snapshot references is deleted after 1 day unreferenced and 1 day noncurrent), snapshot management (24 hours, 1 kept) and compaction. Each table keeps at most 100 old metadata files. A table that already has its own setting of a type keeps it. AIStor 11.0.0-edge accepts only unreferenced-file removal on a Memory Bucket's tables: the collector warns once for each refused type, counts it, expires snapshots itself, and keeps writing.
  • Old rows are removed with --retention-days. About once an hour the collector removes each day of a table that is wholly older than the retention, by event_time (received_time for quarantine). A row stays for up to one day past the retention. A row with no event_time, or one in the future, ages by when the collector received it. Without the flag, no row is removed by age. --retention-days-file reads the number from a file on each pass, so a change applies without a restart.
  • A session is erased with aimem bio erase. It removes every row of the given sessions, and with --user of the given users (when the collector runs with --user-key), from the six telemetry tables, and every quarantined payload that carries one of them. It records them as erased so the collector drops any of their rows that arrive later, from a redelivery or a replay, and does not quarantine them unless a payload cannot be decoded. The record is kept for 8 days, longer than a payload may wait in a spool. It then expires every snapshot from before the erase, so table history no longer reaches the erased rows; unreferenced-file removal deletes their files 1 + 1 days later by default. Rows without a session are not erased, unless they belong to an erased user. Quarantined payloads that cannot be decoded are not erased.

Scoping a collector to one agent

The collector stores every payload under the agent named by --agent. An aimem.agent.id attribute inside a payload is stored like any other attribute; it does not choose the agent.

Run the collector with the credential the Memory Bucket issues when you create the agent. That credential can reach only the agent's telemetry namespace, and the response that issues it names the namespace.

To write the policy yourself, use the one below. It has two parts:

  • Reading the catalog configuration (s3tables:GetConfig) applies to the whole bucket, so it cannot be limited to a namespace.
  • Everything else the collector does is limited to the agent's namespace: it creates the namespace and tables, loads and commits to them, and drops a table registration that stays unloadable.

The first statement gives object access to the whole Memory Bucket. To limit data-file access to the agent's tables too, see the note after the policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketWideTelemetryDataFiles",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": ["arn:aws:s3:::MY-MEMORY-BUCKET/*"]
    },
    {
      "Sid": "ReadTheCatalog",
      "Effect": "Allow",
      "Action": ["s3tables:GetConfig"],
      "Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET"]
    },
    {
      "Sid": "ThisAgentsNamespaceOnly",
      "Effect": "Allow",
      "Action": [
        "s3tables:CreateNamespace",
        "s3tables:GetNamespace",
        "s3tables:CreateTable",
        "s3tables:GetTable",
        "s3tables:UpdateTable",
        "s3tables:DeleteTable"
      ],
      "Resource": [
        "arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET",
        "arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET/table/*"
      ],
      "Condition": { "StringEquals": { "s3tables:namespace": "MY-AGENT" } }
    }
  ]
}

Replace MY-MEMORY-BUCKET and MY-AGENT, and attach the policy to the identity the collector runs as.

  • To limit data-file access to the agent's tables, replace the BucketWideTelemetryDataFiles statement with s3tables:GetTableData and s3tables:PutTableData on the same resources and namespace condition as ThisAgentsNamespaceOnly.
  • Do not grant s3tables:UpdateNamespaceProperties. It would let a collector change the aimem.agent.id value that marks which agent owns a namespace.

The quarantine table uses the same namespace and credential. Its body column can hold prompts, tool arguments, and tool output, so protect it at least as closely as the six telemetry tables. In a healthy deployment quarantined_payloads is zero; a row in otel_quarantine is an incident to investigate.

Reading telemetry outside AIStor Memory

AIStor Memory writes Iceberg v2 tables. Each OTLP timestamp is a timestamptz column in microseconds, with the exact nanoseconds OTLP sent in a long column named <column>_unix_nano beside it, for example start_time and start_time_unix_nano.

PyIceberg reads the tables through the AIStor REST catalog with SigV4. Install it with pip install "pyiceberg[pyarrow,rest-sigv4]". The client.* credentials sign catalog requests and read the data files:

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "aimem",
    type="rest",
    uri="https://aistor.example.com:9000/_iceberg",
    warehouse="MY-MEMORY-BUCKET",
    **{
        "rest.sigv4-enabled": "true",
        "rest.signing-name": "s3tables",
        "s3.endpoint": "https://aistor.example.com:9000",
        "client.region": "us-east-1",
        "client.access-key-id": "...",
        "client.secret-access-key": "...",
    },
)

table = catalog.load_table(("MY-AGENT", "otel_spans"))
arrow_table = table.scan(row_filter="session_id = 'my-session'").to_arrow()

You can filter on session_id, on a timestamp column, or on any other column.

A read-only credential needs only the read part of the collector policy. As before, the first statement covers the whole Memory Bucket; use s3tables:GetTableData with the namespace condition to limit data-file reads to the agent's tables.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketWideTelemetryDataFilesRead",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::MY-MEMORY-BUCKET/*"]
    },
    {
      "Sid": "ReadTheCatalog",
      "Effect": "Allow",
      "Action": ["s3tables:GetConfig"],
      "Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET"]
    },
    {
      "Sid": "ThisAgentsTablesReadOnly",
      "Effect": "Allow",
      "Action": ["s3tables:GetTable"],
      "Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET/table/*"],
      "Condition": { "StringEquals": { "s3tables:namespace": "MY-AGENT" } }
    }
  ]
}

DuckDB and Spark read a Parquet export of a table's rows, written by aimem bio export. The export keeps the table's columns, <column>_unix_nano included, so no precision is lost. It uses the same credential as aimem bio query.

aimem bio export "${BUCKET}" --agent checkout-bot --table otel_spans \
  --output spans.parquet
duckdb -c "SELECT span_name, start_time FROM 'spans.parquet' ORDER BY start_time"
spark.read.parquet("spans.parquet").where("session_id = 'my-session'")

Tested with PyIceberg 0.12.0, DuckDB 1.5.5, and Spark 3.5.6 with Iceberg 1.9.2.

Local staging directory

Every mount needs a local staging directory. Set it in one of three ways:

  • the local: key in the --config file (used on this page);
  • the --local <DIR> flag;
  • the AIMEM_LOCAL environment variable.

If you set more than one, the flag or environment variable wins over the config key. If you set none, the mount stops with a local staging directory is required; set --local <DIR>, the AIMEM_LOCAL env var, or the local: key in --config.

Put it on fast local storage, such as NVMe, with room for every file that is open for writing at once. Each such file gets a staging file here. The mount logs the path it uses. To limit how much space it uses, see Local store size.

The .aimem-lostfound/ prefix

On a POSIX filesystem, a reader can keep reading a file after the file is deleted or replaced. AIStor Memory keeps this behavior. When a file is deleted, or another file is renamed over it, while a reader still has it open, AIStor Memory moves the object to a key under .aimem-lostfound/ instead of deleting it. This works like lost+found on a local filesystem.

The move is a server-side rename, so no data is copied. On a backend without server-side rename, AIStor Memory copies the data to the local staging directory instead, and nothing appears under the prefix.

The mount hides the prefix. It does not appear in a listing of the mount, and you cannot create a file with that name. Other S3 clients do see it: it appears in mc ls, counts toward stored capacity, and matches bucket lifecycle rules.

When objects appear, and when they go

An object appears under the prefix only while a reader holds a deleted or replaced file open. AIStor Memory removes it when the last reader closes the file, which is usually within milliseconds.

An object stays longer only when a mount exits without closing its files, for example after a crash or a kill -9. The next mount then deletes, in the background, every object under the prefix that is older than 24 hours. It skips newer objects because another running mount may still be serving them.

Two results to plan for:

  • A mount that crashes and restarts often can leave up to a day of these objects before a sweep removes them.
  • The sweep runs only when a mount starts. A mount that stays up for months does not sweep again.

Do not delete objects under this prefix by hand while mounts are running: a reader may still be reading one. As a backstop, add a bucket lifecycle rule that expires the prefix after longer than your longest-open file.

Managing the Vault: secrets in a Memory Bucket

The Vault is the secret store inside a Memory Bucket. Use the aimem secret commands to store and rotate secrets. Then use aimem secret provision --bucket to give them to an agent when it starts. Each command takes the Memory Bucket name and connects with the connection settings:

# Store or rotate a secret. Pipe the value on stdin: a value passed with
# --value shows in the process list and shell history.
printf '%s' "$OPENAI_KEY" | aimem secret put my-memory-bucket openai

# Read a secret's value.
aimem secret get my-memory-bucket openai

# List the secret names in the Memory Bucket.
aimem secret list my-memory-bucket

# Delete a secret.
aimem secret delete my-memory-bucket openai

AIStor Memory does not keep the secret itself. It passes the value to the Memory Bucket, which encrypts it with KMS. A plain S3 GET of the stored object returns ciphertext; only the Memory API returns the plaintext. Each secret can be up to 1 MiB. Each put creates a new version. These commands need the same prerequisites as a mount: an endpoint, signing credentials, and a paid-tier server.

To give a stored secret to an agent at launch, see aimem secret provision.

Unmounting

umount /workspace

umount shuts down a background aimem cleanly. A foreground aimem also unmounts and exits on SIGINT or SIGTERM.

If a stale mount blocks umount, for example after a crash, run fusermount3 -uz /workspace. This does not change the data in the bucket.

Troubleshooting

  • Transport endpoint is not connected. Run df on the mount point itself, such as df /workspace, not on a path inside it. The mount point needs no lookup, so df reaches aimem even when the nodes do not answer.
    • If df answers, the mount is running but no AIStor node answered the request: every node refused the connection or timed out. Check the nodes and the network between them and the mount. Do not unmount: the mount works again as soon as a node answers.
    • If df fails with the same error, check that no aimem process is left for the mount, for example with pgrep -a aimem. If none is, the mount is stale, for example after a crash: run fusermount3 -uz <path>.
  • The mount refuses to start with a Memory Bucket or credential error. Check each condition in Prerequisites.
  • Memory files are not warmed. On mount, AIStor Memory preloads the agent's memory files, such as CLAUDE.md, into the local cache. This step is best-effort: it logs failures and the mount continues. Files that do not exist in the bucket are skipped without a message, so check that they exist.
  • The staging directory runs out of space. It holds a copy of every file open for writing. Give local a larger volume, or close files between batches.
  • You need to move the staging directory. Unmount, then mount again with the new local. You cannot move it while mounted.
  • aimem secret provision cannot reach the secret source. It reads from exactly one source: --bucket (the Memory Bucket, with the connection settings) or --store-endpoint (a generic HTTP secret store), and needs at least one --grant VAR=name. The source must be reachable from the launcher host, the credential must be allowed to read the granted secrets, and a production endpoint must use HTTPS. See aimem secret provision.
  • unknown field 'X' at startup. The YAML has a key AIStor Memory does not know. YAML keys are the long flag names with dashes changed to underscores; see the Configuration key reference.