Operations

This page is for operators who run AIStor Memory mounts. It covers what a mount needs, how to start one in each sandbox runtime or at boot, how to watch it, how to store secrets in the Vault, and how to fix common problems.

How the mount is configured

The examples on this page configure the mount with a --config YAML file. A file is easy to review and to keep with the rest of your infrastructure configuration. Credentials come from the environment, never from the file.

Before a mount starts, AIStor Memory checks that the target is a Memory Bucket it can reach and sign requests for. See Prerequisites.

Prerequisites

AIStor Memory mounts only a Memory Bucket, never a plain S3 bucket. Before it starts the mount, it asks the AIStor Memory API to confirm the bucket. If any condition below is not met, the mount stops with an error that names the problem. There is no option to skip the check.

1. The bucket is a Memory Bucket. Create one with aimem create, or convert an existing bucket with aimem convert. If the target is a plain bucket, the mount fails with '<bucket>' is not a Memory Bucket.
2. An endpoint is set. Set endpoint_url in the config file, or pass --endpoint-url. AIStor Memory needs the endpoint to check the bucket.
3. Signing credentials are set. Set AIMEM_ACCESS_KEY and AIMEM_SECRET_KEY (and AIMEM_SESSION_TOKEN for temporary credentials), or set AIMEM_CREDENTIALS_FILE to a credentials file. Anonymous mounts and mounts that rely on a default credential chain are refused.
4. The credentials can read the Memory Bucket, and AIStor is on a paid tier. The Memory API is an AIStor Enterprise feature. The credentials need the memory:GetCortex permission on the bucket. If either condition fails, the mount reports access denied by the Memory API (the credentials lack memory:GetCortex, or the server is not on a paid tier).
5. A local staging directory is set. See Local staging directory.

When endpoint_url lists several endpoints, AIStor Memory tries each in turn. It retries network errors and HTTP 408, 429, and 5xx responses up to five times per endpoint. The check uses the same TLS settings as the mount (tls_ca_file, tls_server_name, insecure_skip_tls_verify). For a custom CA, set tls_ca_file or AIMEM_TLS_CA_FILE; AIStor Memory also honors SSL_CERT_FILE and SSL_CERT_DIR.

To create a Memory Bucket and give an agent credentials that reach only that bucket:

aimem create my-memory-bucket --endpoint-url https://aistor.example.com:9000
aimem credentials my-memory-bucket --duration 3600

aimem create compresses the bucket, and encrypts it when AIStor has a KMS. It uses the server's default KMS key unless you pass --kms-key-id. You cannot change these properties later. For every Memory Bucket command, see the CLI reference.

Running AIStor Memory inside a sandbox provider

The Integrations pages give a full setup for each runtime. To set one up yourself, put a mount.yaml where the runtime can read it, mount before the agent starts, and unmount when the session ends.

Minimum configuration

# /etc/aimem/mount.yaml
version: "1.0"
local: /var/cache/aimem
endpoint_url:
  - http://aistor{1...4}:9000
region: us-east-1
agent: auto
metadata_ttl: indefinite

# Set credentials in the environment, never on the command line.
export AIMEM_ACCESS_KEY=... AIMEM_SECRET_KEY=...

aimem my-memory-bucket /workspace --config /etc/aimem/mount.yaml

• version: "1.0" is required, as a quoted string. Without quotes, YAML reads 1.0 as a number and the file is rejected.
• local: sets the staging directory, which every mount needs.
• my-memory-bucket must be an existing Memory Bucket.

For logging, metrics, and every other key, see Configuration.

fstab

To mount at boot on a Linux host, add a line to /etc/fstab. Linux runs mount.fuse3, which passes the bucket, the mountpoint, and the options after -o to AIStor Memory:

my-memory-bucket /workspace fuse.aimem _netdev,config=/etc/aimem/mount.yaml 0 0

• config= points at the same YAML file you use for an interactive mount.
• To set the staging directory in fstab instead of the YAML, add local=<dir>. The older spellings staging_dir=<dir> and staging-dir=<dir> also work.
• Inside a value, put a backslash before a literal comma, backslash, or double quote.

Daemonization

By default, aimem starts the mount and then moves to the background. The starting process waits up to 30 seconds for the mount to come up. On success, it prints the mount path and exits. On failure, it reports that the mount failed; the background process writes the cause to its log.

To keep aimem in the foreground, for example in a container or under systemd, set foreground: true in the YAML.

Logging

By default, AIStor Memory logs warnings from every component and info messages from aimem itself. Logs go to syslog, or to files when you set log_directory. In the foreground, logs also go to stdout.

# mount.yaml: logging keys
debug: false # true logs everything at debug level
no_log: false # true turns logging off
log_directory: /var/log/aimem
log_metrics: false # true writes metric records to the log

To set the log filter directly, set AIMEM_LOG. It takes the tracing EnvFilter syntax, for example AIMEM_LOG=aimem=debug,warn, and overrides the YAML settings.

OpenTelemetry metrics

To export mount metrics over OTLP/HTTP, set an endpoint:

# mount.yaml: metrics keys
otlp_endpoint: http://otel-collector:4318
otlp_export_interval: 60

The metrics include:

• fuse.request_latency and fuse.request_errors, by FUSE operation.
• fuse.io_size, by read and write.
• fuse.idle_threads and fuse.total_threads.
• fs.current_handles{type=read|staged_read|write}, fs.inodes, and fs.inode_kinds{kind=…}.
• fuse.cache_hit.
• prefetch.reset_state.
• s3.request_count, s3.request_errors, s3.request_canceled, s3.request_total_latency, and s3.request_first_byte_latency, by request type and HTTP status.
• process.memory_usage.

Agent telemetry

The metrics above describe the mount. To record what the agent itself does, its traces, logs, and metrics, run the aimem bio serve collector. It stores the telemetry in the same Memory Bucket.

Run one collector for each agent, next to the agent, and point the agent's OpenTelemetry exporter at the endpoint the collector publishes:

export AIMEM_ENDPOINT_URL=https://aistor.example.com:9000

# Use the credential the Memory Bucket issued when the agent was created.
# It cannot read the agent record, so name the session-id attribute that the
# record's capture_descriptors declare.
export AIMEM_ACCESS_KEY=... AIMEM_SECRET_KEY=...
aimem bio serve "${BUCKET}" --agent checkout-bot \
  --session-key session.id \
  --endpoint-file /run/agent/otlp-endpoint &

# Wait for the collector to publish its endpoint, then start the agent.
for _ in $(seq 1 30); do [ -s /run/agent/otlp-endpoint ] && break; sleep 1; done
[ -s /run/agent/otlp-endpoint ] || { echo "collector never published an endpoint"; exit 1; }
export OTEL_EXPORTER_OTLP_ENDPOINT="$(cat /run/agent/otlp-endpoint)"

The collector listens only on loopback, so it must share a network namespace with the agent: the same pod or the same container. It reads the connection settings from the environment. The credential belongs to the collector; do not give it to the agent.

You do not need to create tables first. Before it publishes its endpoint, the collector creates the agent's namespace and tables, or checks the ones that exist. It exits if the namespace belongs to another agent or a table does not match its schema. When you delete the agent, AIStor removes its namespace and tables within a minute or two.

Plan for these behaviors:

• It fails loudly. If the Memory Bucket name is wrong or the credential lacks permissions, every write fails and the collector reports each failure. When it is clear the writes will not succeed, it stops accepting data and exits with a non-zero status. Run it under a supervisor that restarts it and reports the exit.
• An acknowledged payload survives a crash. The collector answers 200 only after it writes the payload to disk in its spool. If the collector stops before it commits the payload, the next start replays it. The spool is at <endpoint-file>.spool, or where --spool-dir or AIMEM_OTLP_SPOOL_DIR says. It is only as durable as its filesystem: a tmpfs such as /run keeps it across a crash but not a reboot. The spool holds prompts and tool output, so only the collector's user can read it.
• A slow backend fills the spool, not the agent. When the Memory Bucket is slow, new payloads wait in the spool. The collector answers 503 with Retry-After only when the spool is full (1 GiB or 100,000 payloads by default; set AIMEM_OTLP_SPOOL_MAX_BYTES and AIMEM_OTLP_SPOOL_MAX_COUNT to change), when a spool write fails, or when it cannot read the request. In each case it has not kept the payload, so the exporter must retry every retryable 503.
• Each payload is written once per table. A replay after a crash, or a retry after a lost response, checks each table first, so rows are not duplicated. Replaying the quarantine table with aimem bio quarantine replay writes rows again on purpose.

Scoping a collector to one agent

The collector stores every payload under the agent named by --agent. An aimem.agent.id attribute inside a payload is stored like any other attribute; it does not choose the agent.

Run the collector with the credential the Memory Bucket issues when you create the agent. That credential can reach only the agent's telemetry namespace, and the response that issues it names the namespace.

To write the policy yourself, use the one below. It has two parts:

• Reading the catalog configuration (s3tables:GetConfig) applies to the whole bucket, so it cannot be limited to a namespace.
• Everything else the collector does is limited to the agent's namespace: it creates the namespace and tables, loads and commits to them, and drops a table registration that stays unloadable.

The first statement gives object access to the whole Memory Bucket. To limit data-file access to the agent's tables too, see the note after the policy.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketWideTelemetryDataFiles",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": ["arn:aws:s3:::MY-MEMORY-BUCKET/*"]
    },
    {
      "Sid": "ReadTheCatalog",
      "Effect": "Allow",
      "Action": ["s3tables:GetConfig"],
      "Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET"]
    },
    {
      "Sid": "ThisAgentsNamespaceOnly",
      "Effect": "Allow",
      "Action": [
        "s3tables:CreateNamespace",
        "s3tables:GetNamespace",
        "s3tables:CreateTable",
        "s3tables:GetTable",
        "s3tables:UpdateTable",
        "s3tables:DeleteTable"
      ],
      "Resource": [
        "arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET",
        "arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET/table/*"
      ],
      "Condition": { "StringEquals": { "s3tables:namespace": "MY-AGENT" } }
    }
  ]
}

Replace MY-MEMORY-BUCKET and MY-AGENT, and attach the policy to the identity the collector runs as.

• To limit data-file access to the agent's tables, replace the BucketWideTelemetryDataFiles statement with s3tables:GetTableData and s3tables:PutTableData on the same resources and namespace condition as ThisAgentsNamespaceOnly.
• Do not grant s3tables:UpdateNamespaceProperties. It would let a collector change the aimem.agent.id value that marks which agent owns a namespace.

Reading telemetry outside AIStor Memory

AIStor Memory writes Iceberg v3 tables with nanosecond OTLP timestamps (timestamptz_ns). Iceberg v2 is not supported.

PyIceberg reads the tables through the AIStor REST catalog with SigV4. Install it with pip install "pyiceberg[pyarrow,rest-sigv4]". The client.* credentials sign catalog requests and read the data files:

from pyiceberg.catalog import load_catalog

catalog = load_catalog(
    "aimem",
    type="rest",
    uri="https://aistor.example.com:9000/_iceberg",
    warehouse="MY-MEMORY-BUCKET",
    **{
        "rest.sigv4-enabled": "true",
        "rest.signing-name": "s3tables",
        "s3.endpoint": "https://aistor.example.com:9000",
        "client.region": "us-east-1",
        "client.access-key-id": "...",
        "client.secret-access-key": "...",
    },
)

table = catalog.load_table(("MY-AGENT", "otel_spans"))
arrow_table = table.scan(row_filter="session_id = 'my-session'").to_arrow()

You can filter on session_id or any other string column. A filter on a timestamp column fails with TypeError: Cannot convert ... into timestamptz_ns, because PyIceberg cannot yet bind a value to that type. Filter on the session, then filter on time in Arrow.

A read-only credential needs only the read part of the collector policy. As before, the first statement covers the whole Memory Bucket; use s3tables:GetTableData with the namespace condition to limit data-file reads to the agent's tables.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "BucketWideTelemetryDataFilesRead",
      "Effect": "Allow",
      "Action": ["s3:GetObject"],
      "Resource": ["arn:aws:s3:::MY-MEMORY-BUCKET/*"]
    },
    {
      "Sid": "ReadTheCatalog",
      "Effect": "Allow",
      "Action": ["s3tables:GetConfig"],
      "Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET"]
    },
    {
      "Sid": "ThisAgentsTablesReadOnly",
      "Effect": "Allow",
      "Action": ["s3tables:GetTable"],
      "Resource": ["arn:aws:s3tables:::bucket/MY-MEMORY-BUCKET/table/*"],
      "Condition": { "StringEquals": { "s3tables:namespace": "MY-AGENT" } }
    }
  ]
}

DuckDB and Spark can list the tables but cannot read their rows, because their Iceberg readers do not yet support timestamptz_ns. Export the rows to Parquet with aimem bio export instead. The export writes each timestamp in microseconds, which these engines read, and keeps the exact value in an integer column named <column>_unix_nano, so no precision is lost. It uses the same credential as aimem bio query.

aimem bio export "${BUCKET}" --agent checkout-bot --table otel_spans \
  --output spans.parquet
duckdb -c "SELECT span_name, start_time FROM 'spans.parquet' ORDER BY start_time"

spark.read.parquet("spans.parquet").where("session_id = 'my-session'")

Tested with PyIceberg 0.12.0, DuckDB 1.5.5, and Spark 3.5.6 with Iceberg 1.9.2.

Local staging directory

Every mount needs a local staging directory. Set it in one of three ways:

• the local: key in the --config file (used on this page);
• the --local <DIR> flag;
• the AIMEM_LOCAL environment variable.

If you set more than one, the flag or environment variable wins over the config key. If you set none, the mount stops with a local staging directory is required; set --local <DIR>, the AIMEM_LOCAL env var, or the local: key in --config.

Put it on fast local storage, such as NVMe, with room for every file that is open for writing at once. Each such file gets a staging file here. The mount logs the path it uses. To limit how much space it uses, see Local store size.

The .aimem-lostfound/ prefix

On a POSIX filesystem, a reader can keep reading a file after the file is deleted or replaced. AIStor Memory keeps this behavior. When a file is deleted, or another file is renamed over it, while a reader still has it open, AIStor Memory moves the object to a key under .aimem-lostfound/ instead of deleting it. This works like lost+found on a local filesystem.

The move is a server-side rename, so no data is copied. On a backend without server-side rename, AIStor Memory copies the data to the local staging directory instead, and nothing appears under the prefix.

The mount hides the prefix. It does not appear in a listing of the mount, and you cannot create a file with that name. Other S3 clients do see it: it appears in mc ls, counts toward stored capacity, and matches bucket lifecycle rules.

When objects appear, and when they go

An object appears under the prefix only while a reader holds a deleted or replaced file open. AIStor Memory removes it when the last reader closes the file, which is usually within milliseconds.

An object stays longer only when a mount exits without closing its files, for example after a crash or a kill -9. The next mount then deletes, in the background, every object under the prefix that is older than 24 hours. It skips newer objects because another running mount may still be serving them.

Two results to plan for:

• A mount that crashes and restarts often can leave up to a day of these objects before a sweep removes them.
• The sweep runs only when a mount starts. A mount that stays up for months does not sweep again.

Managing the Vault: secrets in a Memory Bucket

The Vault is the secret store inside a Memory Bucket. Use the aimem secret commands to store and rotate secrets. Then use aimem secret provision --bucket to give them to an agent when it starts. Each command takes the Memory Bucket name and connects with the connection settings:

# Store or rotate a secret. Pipe the value on stdin: a value passed with
# --value shows in the process list and shell history.
printf '%s' "$OPENAI_KEY" | aimem secret put my-memory-bucket openai

# Read a secret's value.
aimem secret get my-memory-bucket openai

# List the secret names in the Memory Bucket.
aimem secret list my-memory-bucket

# Delete a secret.
aimem secret delete my-memory-bucket openai

AIStor Memory does not keep the secret itself. It passes the value to the Memory Bucket, which encrypts it with KMS. A plain S3 GET of the stored object returns ciphertext; only the Memory API returns the plaintext. Each secret can be up to 1 MiB. Each put creates a new version. These commands need the same prerequisites as a mount: an endpoint, signing credentials, and a paid-tier server.

To give a stored secret to an agent at launch, see aimem secret provision.

Unmounting

umount /workspace

umount shuts down a background aimem cleanly. A foreground aimem also unmounts and exits on SIGINT or SIGTERM.

If a stale mount blocks umount, for example after a crash, run fusermount3 -uz /workspace. This does not change the data in the bucket.

Troubleshooting

• Transport endpoint is not connected after a crash. The mount is stale. Run fusermount3 -uz <path>.
• The mount refuses to start with a Memory Bucket or credential error. Check each condition in Prerequisites.
• Memory files are not warmed. On mount, AIStor Memory preloads the agent's memory files, such as CLAUDE.md, into the local cache. This step is best-effort: it logs failures and the mount continues. Files that do not exist in the bucket are skipped without a message, so check that they exist.
• The staging directory runs out of space. It holds a copy of every file open for writing. Give local a larger volume, or close files between batches.
• You need to move the staging directory. Unmount, then mount again with the new local. You cannot move it while mounted.
• aimem secret provision cannot reach the secret source. It reads from exactly one source: --bucket (the Memory Bucket, with the connection settings) or --store-endpoint (a generic HTTP secret store), and needs at least one --grant VAR=name. The source must be reachable from the launcher host, the credential must be allowed to read the granted secrets, and a production endpoint must use HTTPS. See aimem secret provision.
• unknown field 'X' at startup. The YAML has a key AIStor Memory does not know. YAML keys are the long flag names with dashes changed to underscores; see the Configuration key reference.