Operations
Run AIStor Memory in production. Mount a Memory Bucket in each sandbox runtime or at boot, collect logs, metrics, and agent telemetry, store secrets in the Vault, and fix common problems.
This page is for operators who run AIStor Memory mounts. It covers what a mount needs, how to start one in each sandbox runtime or at boot, how to watch it, how to store secrets in the Vault, and how to fix common problems.
How the mount is configured
The examples on this page configure the mount with a --config
YAML file. A file is easy to review and to keep with the rest of your
infrastructure configuration. Credentials come from the environment, never from
the file.
Before a mount starts, AIStor Memory checks that the target is a Memory Bucket it can reach and sign requests for. See Prerequisites.
Prerequisites
AIStor Memory mounts only a Memory Bucket, never a plain S3 bucket. Before it starts the mount, it asks the AIStor Memory API to confirm the bucket. If any condition below is not met, the mount stops with an error that names the problem. There is no option to skip the check.
- The bucket is a Memory Bucket. Create one with
aimem create, or convert an existing bucket withaimem convert. If the target is a plain bucket, the mount fails with'<bucket>' is not a Memory Bucket. - An endpoint is set. Set
endpoint_urlin the config file, or pass--endpoint-url. AIStor Memory needs the endpoint to check the bucket. - Signing credentials are set. Set
AIMEM_ACCESS_KEYandAIMEM_SECRET_KEY(andAIMEM_SESSION_TOKENfor temporary credentials), or setAIMEM_CREDENTIALS_FILEto a credentials file. Anonymous mounts and mounts that rely on a default credential chain are refused. - The credentials can read the Memory Bucket, and AIStor is on a paid tier.
The Memory API is an AIStor Enterprise feature. The credentials need the
memory:GetCortexpermission on the bucket. If either condition fails, the mount reportsaccess denied by the Memory API (the credentials lack memory:GetCortex, or the server is not on a paid tier). - A local staging directory is set. See Local staging directory.
When endpoint_url lists several endpoints, AIStor Memory tries each in turn.
It retries network errors and HTTP 408, 429, and 5xx responses up to five times
per endpoint. The check uses the same TLS settings as the mount
(tls_ca_file, tls_server_name, insecure_skip_tls_verify). For a custom CA,
set tls_ca_file or AIMEM_TLS_CA_FILE; AIStor Memory also honors
SSL_CERT_FILE and SSL_CERT_DIR.
To create a Memory Bucket and give an agent credentials that reach only that bucket:
aimem create my-memory-bucket --endpoint-url https://aistor.example.com:9000
aimem credentials my-memory-bucket --duration 3600aimem create compresses the bucket, and encrypts it when AIStor has a
KMS. It uses the server's default KMS key unless you pass --kms-key-id. You cannot change these
properties later. For every Memory Bucket command, see the
CLI reference.
Running AIStor Memory inside a sandbox provider
The Integrations pages give a full setup for each runtime. To
set one up yourself, put a mount.yaml where the runtime can read it, mount
before the agent starts, and unmount when the session ends.
| Runtime | Where to put mount.yaml | Mount step | Notes |
|---|---|---|---|
| Vercel Sandbox | /vercel/sandbox/aimem/mount.yaml | aimem "$BUCKET" /workspace --config <yaml> | Set local: in the YAML to a path on /vercel/sandbox. |
| Daytona | Runtime env vars or /etc/aimem/mount.yaml | sandbox.process starts aimem-mount | See Daytona for the sandbox image recipe |
| GitHub Codespaces | .devcontainer/aimem/mount.yaml | devcontainer Feature postCreateCommand | See Codespaces |
| E2B | Baked into the template | sb.commands.run starts aimem-mount | See E2B |
| OpenShell | Host-side, before sandbox launch | Host runs aimem; policy allows the path | See OpenShell |
| Plain Linux VM | /etc/aimem/mount.yaml | systemd unit or /etc/fstab | See fstab below |
Minimum configuration
# /etc/aimem/mount.yaml
version: "1.0"
local: /var/cache/aimem
endpoint_url:
- http://aistor{1...4}:9000
region: us-east-1
agent: auto
metadata_ttl: indefinite# Set credentials in the environment, never on the command line.
export AIMEM_ACCESS_KEY=... AIMEM_SECRET_KEY=...
aimem my-memory-bucket /workspace --config /etc/aimem/mount.yamlversion: "1.0"is required, as a quoted string. Without quotes, YAML reads1.0as a number and the file is rejected.local:sets the staging directory, which every mount needs.my-memory-bucketmust be an existing Memory Bucket.
For logging, metrics, and every other key, see Configuration.
fstab
To mount at boot on a Linux host, add a line to /etc/fstab. Linux runs
mount.fuse3, which passes the bucket, the mountpoint, and the options after
-o to AIStor Memory:
my-memory-bucket /workspace fuse.aimem _netdev,config=/etc/aimem/mount.yaml 0 0config=points at the same YAML file you use for an interactive mount.- To set the staging directory in fstab instead of the YAML, add
local=<dir>. The older spellingsstaging_dir=<dir>andstaging-dir=<dir>also work. - Inside a value, put a backslash before a literal comma, backslash, or double quote.
Daemonization
By default, aimem starts the mount and then moves to the background. The
starting process waits until the mount is up or has failed, however long start-up
takes on a busy host. On success, it prints the mount path and exits. On failure, it reports that the mount failed;
the background process writes the cause to its log.
To keep aimem in the foreground, for example in a container or under systemd,
set foreground: true in the YAML.
Logging
By default, AIStor Memory logs warnings from every component and info messages
from aimem itself. Logs go to syslog, or to files when you set
log_directory. In the foreground, logs also go to stdout.
# mount.yaml: logging keys
debug: false # true logs everything at debug level
no_log: false # true turns logging off
log_directory: /var/log/aimem
log_metrics: false # true writes metric records to the logTo set the log filter directly, set AIMEM_LOG. It takes the tracing
EnvFilter syntax, for example AIMEM_LOG=aimem=debug,warn, and overrides the
YAML settings.
OpenTelemetry metrics
To export mount metrics over OTLP/HTTP, set an endpoint:
# mount.yaml: metrics keys
otlp_endpoint: http://otel-collector:4318
otlp_export_interval: 60The metrics include:
fuse.request_latencyandfuse.request_errors, by FUSE operation.fuse.io_size, byreadandwrite.fuse.idle_threadsandfuse.total_threads.fs.current_handles{type=read|staged_read|write},fs.inodes, andfs.inode_kinds{kind=…}.fuse.cache_hit.prefetch.reset_state.s3.request_count,s3.request_errors,s3.request_canceled,s3.request_total_latency, ands3.request_first_byte_latency, by request type and HTTP status.process.memory_usage.
Agent telemetry
The metrics above describe the mount. To record what the agent itself does, its
traces, logs, and metrics, run the
aimem bio serve
collector. It stores the telemetry in the same Memory Bucket.
Run one collector for each agent, next to the agent, and point the agent's OpenTelemetry exporter at the endpoint the collector publishes:
export AIMEM_ENDPOINT_URL=https://aistor.example.com:9000
# Use the credential the Memory Bucket issued when the agent was created.
# It cannot read the agent record, so name the session-id attribute that the
# record's capture_descriptors declare.
export AIMEM_ACCESS_KEY=... AIMEM_SECRET_KEY=...
aimem bio serve "${BUCKET}" --agent checkout-bot \
--session-key session.id \
--endpoint-file /run/agent/otlp-endpoint &
# Wait for the collector to publish its endpoint, then start the agent.
for _ in $(seq 1 30); do [ -s /run/agent/otlp-endpoint ] && break; sleep 1; done
[ -s /run/agent/otlp-endpoint ] || { echo "collector never published an endpoint"; exit 1; }
export OTEL_EXPORTER_OTLP_ENDPOINT="$(cat /run/agent/otlp-endpoint)"The collector listens only on loopback, so it must share a network namespace with the agent: the same pod or the same container. It reads the connection settings from the environment. The credential belongs to the collector; do not give it to the agent.
You do not need to create tables first. Before it publishes its endpoint, the collector creates the agent's namespace, or checks the one that exists, and exits if it belongs to another agent. It creates each table the first time it has rows for it, and checks a table that exists the first time it writes to it. A table that does not match its schema is not written. When you delete the agent, AIStor removes its namespace and tables within a minute or two.
Plan for these behaviors:
- It fails loudly. If the Memory Bucket name is wrong or the credential lacks permissions, every write fails and the collector reports each failure. When it is clear the writes will not succeed, it stops accepting data and exits with a non-zero status. Run it under a supervisor that restarts it and reports the exit.
- An acknowledged payload survives a crash. The collector answers
200only after it writes the payload to disk in its spool. If the collector stops before it commits the payload, the next start replays it. The spool is at<endpoint-file>.spool, or where--spool-dirorAIMEM_OTLP_SPOOL_DIRsays. It is only as durable as its filesystem: a tmpfs such as/runkeeps it across a crash but not a reboot. The spool holds prompts and tool output, so only the collector's user can read it. A payload received more than 7 days ago is dropped instead of delivered, at the next start or before any export attempt; setAIMEM_OTLP_SPOOL_MAX_AGE_SECSto a shorter time, down to one hour. - A slow backend fills the spool, not the agent. When the Memory Bucket is
slow, new payloads wait in the spool. The collector answers
503withRetry-Afteronly when the spool is full (1 GiB or 100,000 payloads by default; setAIMEM_OTLP_SPOOL_MAX_BYTESandAIMEM_OTLP_SPOOL_MAX_COUNTto change), when a spool write fails, or when it cannot read the request. In each case it has not kept the payload, so the exporter must retry every retryable503. - Each payload is written once per table, within 24 hours. A replay after
a crash, or a retry after a lost response, checks each table first, so rows
are not duplicated. The check reads the table's snapshots from the last 24
hours, so a payload re-sent later than that is written again. Replaying the
quarantine table with
aimem bio quarantine replaywrites rows again on purpose, for payloads quarantined in the last 7 days. - Old snapshots are expired. Each flush adds a snapshot to each table it
writes to. About once an hour the collector removes those older than 24
hours, keeping the current one, and sets that retention as the table's
history.expire.*properties. Expiry changes metadata only: it loses no row, and it does not reclaim storage. - Tables are maintained. When it first creates or loads one of the agent's tables, the collector gives it the maintenance Fabric gives its own tables: unreferenced-file removal (a file no snapshot references is deleted after 1 day unreferenced and 1 day noncurrent), snapshot management (24 hours, 1 kept) and compaction. Each table keeps at most 100 old metadata files. A table that already has its own setting of a type keeps it. AIStor 11.0.0-edge accepts only unreferenced-file removal on a Memory Bucket's tables: the collector warns once for each refused type, counts it, expires snapshots itself, and keeps writing.
- Old rows are removed with
--retention-days. About once an hour the collector removes each day of a table that is wholly older than the retention, byevent_time(received_timefor quarantine). A row stays for up to one day past the retention. A row with noevent_time, or one in the future, ages by when the collector received it. Without the flag, no row is removed by age.--retention-days-filereads the number from a file on each pass, so a change applies without a restart. - A session is erased with
aimem bio erase. It removes every row of the given sessions, and with--userof the given users (when the collector runs with--user-key), from the six telemetry tables, and every quarantined payload that carries one of them. It records them as erased so the collector drops any of their rows that arrive later, from a redelivery or a replay, and does not quarantine them unless a payload cannot be decoded. The record is kept for 8 days, longer than a payload may wait in a spool. It then expires every snapshot from before the erase, so table history no longer reaches the erased rows; unreferenced-file removal deletes their files 1 + 1 days later by default. Rows without a session are not erased, unless they belong to an erased user. Quarantined payloads that cannot be decoded are not erased.
Scoping a collector to one agent
The collector stores every payload under the agent named by --agent. An
aimem.agent.id attribute inside a payload is stored like any other attribute;
it does not choose the agent.
Run the collector with the credential the Memory Bucket issues when you create the agent. That credential can reach only the agent's telemetry namespace, and the response that issues it names the namespace.
To write the policy yourself, use the one below. It has two parts:
- Reading the catalog configuration (
s3tables:GetConfig) applies to the whole bucket, so it cannot be limited to a namespace. - Everything else the collector does is limited to the agent's namespace: it creates the namespace and tables, loads and commits to them, and drops a table registration that stays unloadable.
The first statement gives object access to the whole Memory Bucket. To limit data-file access to the agent's tables too, see the note after the policy.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BucketWideTelemetryDataFiles",
"Effect": "Allow",
"Action": ["s3:GetObject", "s3:PutObject"],
"Resource": ["arn:aws:s3:::MY-MEMORY-BUCKET/*"]
},
{
"Sid": "ReadTheCatalog",
"Effect": "Allow",
"Action": ["s3tables:GetConfig"],
"Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET"]
},
{
"Sid": "ThisAgentsNamespaceOnly",
"Effect": "Allow",
"Action": [
"s3tables:CreateNamespace",
"s3tables:GetNamespace",
"s3tables:CreateTable",
"s3tables:GetTable",
"s3tables:UpdateTable",
"s3tables:DeleteTable"
],
"Resource": [
"arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET",
"arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET/table/*"
],
"Condition": { "StringEquals": { "s3tables:namespace": "MY-AGENT" } }
}
]
}Replace MY-MEMORY-BUCKET and MY-AGENT, and attach the policy to the identity
the collector runs as.
- To limit data-file access to the agent's tables, replace the
BucketWideTelemetryDataFilesstatement withs3tables:GetTableDataands3tables:PutTableDataon the same resources and namespace condition asThisAgentsNamespaceOnly. - Do not grant
s3tables:UpdateNamespaceProperties. It would let a collector change theaimem.agent.idvalue that marks which agent owns a namespace.
The quarantine table uses the same namespace and credential. Its body column
can hold prompts, tool arguments, and tool output, so protect it at least as
closely as the six telemetry tables. In a healthy deployment
quarantined_payloads is zero; a row in otel_quarantine is an incident to
investigate.
Reading telemetry outside AIStor Memory
AIStor Memory writes Iceberg v2 tables. Each OTLP timestamp is a timestamptz
column in microseconds, with the exact nanoseconds OTLP sent in a long column
named <column>_unix_nano beside it, for example start_time and
start_time_unix_nano.
PyIceberg reads the tables through the AIStor REST catalog with SigV4.
Install it with pip install "pyiceberg[pyarrow,rest-sigv4]". The client.*
credentials sign catalog requests and read the data files:
from pyiceberg.catalog import load_catalog
catalog = load_catalog(
"aimem",
type="rest",
uri="https://aistor.example.com:9000/_iceberg",
warehouse="MY-MEMORY-BUCKET",
**{
"rest.sigv4-enabled": "true",
"rest.signing-name": "s3tables",
"s3.endpoint": "https://aistor.example.com:9000",
"client.region": "us-east-1",
"client.access-key-id": "...",
"client.secret-access-key": "...",
},
)
table = catalog.load_table(("MY-AGENT", "otel_spans"))
arrow_table = table.scan(row_filter="session_id = 'my-session'").to_arrow()You can filter on session_id, on a timestamp column, or on any other
column.
A read-only credential needs only the read part of the collector policy. As
before, the first statement covers the whole Memory Bucket; use
s3tables:GetTableData with the namespace condition to limit data-file reads to
the agent's tables.
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "BucketWideTelemetryDataFilesRead",
"Effect": "Allow",
"Action": ["s3:GetObject"],
"Resource": ["arn:aws:s3:::MY-MEMORY-BUCKET/*"]
},
{
"Sid": "ReadTheCatalog",
"Effect": "Allow",
"Action": ["s3tables:GetConfig"],
"Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET"]
},
{
"Sid": "ThisAgentsTablesReadOnly",
"Effect": "Allow",
"Action": ["s3tables:GetTable"],
"Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET/table/*"],
"Condition": { "StringEquals": { "s3tables:namespace": "MY-AGENT" } }
}
]
}DuckDB and Spark read a Parquet export of a table's rows, written by
aimem bio export.
The export keeps the table's columns, <column>_unix_nano included, so no
precision is lost. It uses the same credential as aimem bio query.
aimem bio export "${BUCKET}" --agent checkout-bot --table otel_spans \
--output spans.parquet
duckdb -c "SELECT span_name, start_time FROM 'spans.parquet' ORDER BY start_time"spark.read.parquet("spans.parquet").where("session_id = 'my-session'")Tested with PyIceberg 0.12.0, DuckDB 1.5.5, and Spark 3.5.6 with Iceberg 1.9.2.
Local staging directory
Every mount needs a local staging directory. Set it in one of three ways:
- the
local:key in the--configfile (used on this page); - the
--local <DIR>flag; - the
AIMEM_LOCALenvironment variable.
If you set more than one, the flag or environment variable wins over the config
key. If you set none, the mount stops with a local staging directory is required; set --local <DIR>, the AIMEM_LOCAL env var, or the local: key in --config.
Put it on fast local storage, such as NVMe, with room for every file that is open for writing at once. Each such file gets a staging file here. The mount logs the path it uses. To limit how much space it uses, see Local store size.
The .aimem-lostfound/ prefix
On a POSIX filesystem, a reader can keep reading a file after the file is
deleted or replaced. AIStor Memory keeps this behavior. When a file is deleted,
or another file is renamed over it, while a reader still has it open, AIStor
Memory moves the object to a key under .aimem-lostfound/ instead of deleting
it. This works like lost+found on a local filesystem.
The move is a server-side rename, so no data is copied. On a backend without server-side rename, AIStor Memory copies the data to the local staging directory instead, and nothing appears under the prefix.
The mount hides the prefix. It does not appear in a listing of the mount,
and you cannot create a file with that name. Other S3 clients do see it: it
appears in mc ls, counts toward stored capacity, and matches bucket lifecycle
rules.
When objects appear, and when they go
An object appears under the prefix only while a reader holds a deleted or replaced file open. AIStor Memory removes it when the last reader closes the file, which is usually within milliseconds.
An object stays longer only when a mount exits without closing its files, for
example after a crash or a kill -9. The next mount then deletes, in the
background, every object under the prefix that is older than 24 hours. It skips
newer objects because another running mount may still be serving them.
Two results to plan for:
- A mount that crashes and restarts often can leave up to a day of these objects before a sweep removes them.
- The sweep runs only when a mount starts. A mount that stays up for months does not sweep again.
Do not delete objects under this prefix by hand while mounts are running: a reader may still be reading one. As a backstop, add a bucket lifecycle rule that expires the prefix after longer than your longest-open file.
Managing the Vault: secrets in a Memory Bucket
The Vault is the secret store inside a Memory Bucket. Use the
aimem secret commands to store and rotate secrets. Then use
aimem secret provision --bucket to give them to an agent when it starts. Each
command takes the Memory Bucket name and connects with the
connection settings:
# Store or rotate a secret. Pipe the value on stdin: a value passed with
# --value shows in the process list and shell history.
printf '%s' "$OPENAI_KEY" | aimem secret put my-memory-bucket openai
# Read a secret's value.
aimem secret get my-memory-bucket openai
# List the secret names in the Memory Bucket.
aimem secret list my-memory-bucket
# Delete a secret.
aimem secret delete my-memory-bucket openaiAIStor Memory does not keep the secret itself. It passes the value to the Memory
Bucket, which encrypts it with KMS. A plain S3 GET of the stored object returns
ciphertext; only the Memory API returns the plaintext. Each secret can be up to
1 MiB. Each put creates a new version. These commands need the same
prerequisites as a mount: an endpoint, signing credentials, and
a paid-tier server.
To give a stored secret to an agent at launch, see
aimem secret provision.
Unmounting
umount /workspaceumount shuts down a background aimem cleanly. A foreground aimem also
unmounts and exits on SIGINT or SIGTERM.
If a stale mount blocks umount, for example after a crash, run
fusermount3 -uz /workspace. This does not change the data in the bucket.
Troubleshooting
Transport endpoint is not connected. Rundfon the mount point itself, such asdf /workspace, not on a path inside it. The mount point needs no lookup, sodfreachesaimemeven when the nodes do not answer.- If
dfanswers, the mount is running but no AIStor node answered the request: every node refused the connection or timed out. Check the nodes and the network between them and the mount. Do not unmount: the mount works again as soon as a node answers. - If
dffails with the same error, check that noaimemprocess is left for the mount, for example withpgrep -a aimem. If none is, the mount is stale, for example after a crash: runfusermount3 -uz <path>.
- If
- The mount refuses to start with a Memory Bucket or credential error. Check each condition in Prerequisites.
- Memory files are not warmed. On mount, AIStor Memory preloads the agent's
memory files, such as
CLAUDE.md, into the local cache. This step is best-effort: it logs failures and the mount continues. Files that do not exist in the bucket are skipped without a message, so check that they exist. - The staging directory runs out of space. It holds a copy of every file
open for writing. Give
locala larger volume, or close files between batches. - You need to move the staging directory. Unmount, then mount again with
the new
local. You cannot move it while mounted. aimem secret provisioncannot reach the secret source. It reads from exactly one source:--bucket(the Memory Bucket, with the connection settings) or--store-endpoint(a generic HTTP secret store), and needs at least one--grant VAR=name. The source must be reachable from the launcher host, the credential must be allowed to read the granted secrets, and a production endpoint must use HTTPS. Seeaimem secret provision.unknown field 'X'at startup. The YAML has a key AIStor Memory does not know. YAML keys are the long flag names with dashes changed to underscores; see the Configuration key reference.
Devin (Cognition)
How a Devin session could mount an AIStor Memory Bucket, so its files and memory survive sandbox teardown and any host can resume it.
Configuration
Reference for the AIStor Memory mount configuration file. Covers the local staging store, endpoints and failover, health checks, request timeout, metadata caching, and FUSE and transfer settings.