Transfer Objects over RDMA

Move object data between your application’s memory and MinIO AIStor over RDMA, bypassing the kernel network stack.

RDMA suits pipelines that read large objects straight into GPU memory, where the copy between the network stack and the device is the bottleneck.

The deployment must run the RDMA build

S3 over RDMA needs the RDMA build of AIStor Server. Against a standard deployment, an RDMA request is declined and your client falls back to HTTP.

See RDMA acceleration.

How it works

Your client allocates a buffer, registers it, and sends its descriptor in the x-amz-rdma-token header. The server transfers the object through that buffer directly and answers with x-amz-rdma-reply: 200, or declines with 501.

A ranged request, or one naming a part number, replies 206 instead of 200. Both mean the transfer succeeded — treat only 501 as a decline.

Three operations have an RDMA path: GetObject, PutObject, and UploadPart. GetObject writes into your buffer; PutObject and UploadPart read from it. Everything else uses HTTP whatever the header says.

For the wire-level detail, see S3 over RDMA.

Requirements

Requirement Detail
Server An AIStor deployment running the RDMA build
Fabric An RDMA-capable NIC on the client, reaching the server’s RDMA fabric
GPU Only for transfers into device memory. Host-memory transfers need no GPU.
Library libminiocpp.so, built with RDMA enabled. How each SDK finds it differs — see the sections below.

Every SDK uses NVIDIA cuObjClient underneath to move the payload.

Build minio-cpp with -DMINIO_CPP_ENABLE_RDMA=ON to get that library. The option is off by default and is supported on Linux only.

Buffers. Go, C++, and Rust need a page-aligned, contiguous buffer, and each provides a way to allocate one — do not hand them an arbitrary slice or array. Python is the exception: it accepts any buffer-protocol object, such as a bytearray, and takes the address directly.

Go

RDMA is behind a build tag, and the build links libminiocpp.so through cgo, so the library must be present when you compile and when you run:

go build -tags=rdma ./...

Set EnableRDMA on the client, then pass a buffer through the request options:

package main

import (
	"context"
	"log"
	"unsafe"

	minio "github.com/minio/minio-go/v7"
	"github.com/minio/minio-go/v7/pkg/credentials"
)

func main() {
	const size = 1 << 20 // 1 MiB

	client, err := minio.New("aistor.example.net:9000", &minio.Options{
		Creds:      credentials.NewStaticV4("ACCESS_KEY", "SECRET_KEY", ""),
		Secure:     false,
		EnableRDMA: true,
	})
	if err != nil {
		log.Fatal(err)
	}

	if !minio.IsRDMAAvailable() {
		log.Fatal("no cuObjServer connection; the transfer would fall back to HTTP")
	}

	ctx := context.Background()

	src := minio.AlignedBuffer(size)
	if src == nil {
		log.Fatal("AlignedBuffer failed")
	}
	defer minio.FreeAlignedBuffer(src)

	// Fill the registered buffer in place. Do not build the payload in a
	// separate slice and pass that instead: the transfer reads from src.
	payload := unsafe.Slice((*byte)(src), size)
	for i := range payload {
		payload[i] = byte(i)
	}

	info, err := client.PutObject(ctx, "my-bucket", "my-object", nil, 0, minio.PutObjectOptions{
		RDMABuffer:     src,
		RDMABufferSize: size,
	})
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("uploaded etag=%s size=%d", info.ETag, info.Size)

	dst := minio.AlignedBuffer(size)
	if dst == nil {
		log.Fatal("AlignedBuffer failed")
	}
	defer minio.FreeAlignedBuffer(dst)

	obj, err := client.GetObject(ctx, "my-bucket", "my-object", minio.GetObjectOptions{
		RDMABuffer:     dst,
		RDMABufferSize: size,
	})
	if err != nil {
		log.Fatal(err)
	}
	stat, err := obj.Stat()
	if err != nil {
		log.Fatal(err)
	}
	log.Printf("downloaded size=%d", stat.Size)
}

PutObject takes nil for the reader and 0 for the size when RDMABuffer is set: the data comes from the buffer, not the reader.

Function Purpose
minio.IsRDMAAvailable() Reports whether the client is connected to a cuObjServer
minio.AlignedBuffer(n) Allocates a page-aligned buffer. Returns nil on failure.
minio.FreeAlignedBuffer(p) Releases a buffer from AlignedBuffer

Python

Set enable_rdma on the client. A buffer-protocol object such as a bytearray selects the RDMA path:

from minio import Minio

client = Minio(
    endpoint="aistor.example.net:9000",
    access_key="ACCESS_KEY",
    secret_key="SECRET_KEY",
    secure=False,
    enable_rdma=True,
)

size = 1 << 20
payload = bytearray(b"x" * size)

resp = client.put_object(
    bucket_name="my-bucket",
    object_name="my-object",
    data=payload,
    length=len(payload),
)
print(f"etag={resp.etag}")

dst = bytearray(size)
n = client.get_object(
    bucket_name="my-bucket",
    object_name="my-object",
    into=dst,
    length=size,
)
print(f"bytes_transferred={n}")

The RDMA download uses into= to name a pre-allocated destination buffer, rather than returning a response stream.

Python loads libminiocpp.so at runtime through ctypes. Set MINIOCPP_LIB to an explicit path if the library is not on the default search path. That variable is specific to this SDK; the others locate the library at link time.

C++

Set buf and size on the request arguments. Their presence selects the RDMA path:

#include <miniocpp/client.h>
#include <stdlib.h>   // posix_memalign
#include <string.h>   // memset
#include <unistd.h>   // getpagesize

int main() {
  const size_t bufsize = 1 << 20;

  minio::s3::BaseUrl base_url("aistor.example.net:9000", false, "us-east-1");
  minio::creds::StaticProvider provider("ACCESS_KEY", "SECRET_KEY");
  minio::s3::Client client(base_url, &provider);

  char* bufptr = nullptr;
  if (posix_memalign(reinterpret_cast<void**>(&bufptr), getpagesize(), bufsize) != 0) {
    return 1;
  }
  memset(bufptr, 'A', bufsize);

  minio::s3::PutObjectArgs pargs;
  pargs.buf = bufptr;
  pargs.size = bufsize;
  pargs.bucket = "my-bucket";
  pargs.object = "my-object";

  minio::s3::PutObjectResponse presp = client.PutObject(pargs);
  if (!presp) return 1;

  minio::s3::GetObjectArgs gargs;
  gargs.buf = bufptr;
  gargs.size = bufsize;
  gargs.bucket = "my-bucket";
  gargs.object = "my-object";

  minio::s3::GetObjectResponse gresp = client.GetObject(gargs);
  if (!gresp) return 1;

  return 0;
}

For a GPU transfer, allocate with cudaMalloc and pass the device pointer as buf.

Rust

Wrap a raw pointer in an RdmaBuffer and use the RDMA-specific methods:

use minio::s3::MinioClient;
use minio::s3::creds::StaticProvider;
use minio::s3::http::BaseUrl;
use minio::s3::rdma::RdmaBuffer;

#[tokio::main]
async fn main() {
    let bufsize = 1 << 20;
    let base_url: BaseUrl = "aistor.example.net:9000".parse().unwrap();

    let client = MinioClient::new(
        base_url,
        Some(StaticProvider::new("ACCESS_KEY", "SECRET_KEY", None)),
        None,
        None,
    )
    .unwrap();

    // Page-aligned host allocation. Use cudaMalloc for a device pointer.
    let mut ptr: *mut libc::c_void = std::ptr::null_mut();
    let page = unsafe { libc::sysconf(libc::_SC_PAGESIZE) } as usize;
    assert_eq!(unsafe { libc::posix_memalign(&mut ptr, page, bufsize) }, 0);
    unsafe { std::ptr::write_bytes(ptr as *mut u8, b'A', bufsize) };

    let buffer = unsafe { RdmaBuffer::from_raw(ptr, bufsize) };

    let resp = client
        .rdma_put_object("my-bucket", "my-object", buffer)
        .await
        .unwrap();
    println!("etag={} bytes={}", resp.etag, resp.bytes_transferred);

    let resp = client
        .rdma_get_object("my-bucket", "my-object", buffer)
        .await
        .unwrap();
    println!("bytes={}", resp.bytes_transferred);
}

RdmaBuffer::from_raw is unsafe: the pointer must stay valid and unaliased for the duration of the transfer.

Handle the fallback

The server does not serve the object over HTTP when it declines an RDMA request. It returns an S3 error, and retrying is the client’s job. Each SDK above does this for you.

If you implement the protocol directly, retry the same request without the x-amz-rdma-token header whenever any of these occur:

Signal Meaning
x-amz-rdma-reply: 501 The server declined the request.
RDMATransferError, HTTP 503 The transfer started and failed. Retryable.
An S3 error with no x-amz-rdma-reply header A malformed token, or transfer bounds that do not fit the object.

Do not key your fallback solely on the 501 header: two of the three signals above do not carry it.

Confirm a transfer used RDMA

Read the response headers, or watch the server-side counters:

curl -s http://aistor.example.net:9000/minio/metrics/v3/api/rdma

minio_api_rdma_read_ops_total increments for a GET and minio_api_rdma_write_ops_total for a PUT or part upload. A counter that stays flat while your client reports success means the transfer used HTTP.

See Validate the RDMA deployment.