warp snowball

Test TAR archive upload and extraction performance for bulk data import scenarios.

warp snowball [FLAGS]

warp snowball --host myaistor.example.com:9000 --access-key ACCESS_KEY --secret-key SECRET_KEY

Default: 512KiB

Size of each object inside the TAR archive. The object size affects TAR creation time and extraction performance. Use this flag to test snowball behavior with different object sizes.

Valid formats include numbers or values with KiB, MiB, or GiB suffixes (base 2 binary).

Example:

warp snowball --host myaistor.example.com:9000 --obj.size 1MiB --access-key YOUR_ACCESS_KEY --secret-key YOUR_SECRET_KEY

Default: 50

Number of objects to include in each TAR archive. The command creates TAR files containing this many objects per upload. Use this flag to test different archive sizes and object counts.

Example:

warp snowball --host myaistor.example.com:9000 --objs.per 100 --access-key YOUR_ACCESS_KEY --secret-key YOUR_SECRET_KEY

Default: false

Enable compression for each TAR archive. The default behavior uploads uncompressed TAR files. Use this flag to test compressed snowball uploads on MinIO servers.

This feature is specific to MinIO servers.

Example:

warp snowball --host myaistor.example.com:9000 --compress --access-key YOUR_ACCESS_KEY --secret-key YOUR_SECRET_KEY

This command supports all common flags including connection, performance, object, upload, benchmark control, analysis, and monitoring flags.

The snowball benchmark tests bulk data import performance using TAR archives.

During the TAR creation phase, the command generates TAR archives containing multiple objects. Each TAR file includes the configured number of objects with the specified size. The command creates TAR archives dynamically during the benchmark execution. When compression is enabled, the command compresses TAR archives before upload.

During the upload phase, the command uploads TAR archives to the target bucket. The storage system automatically extracts objects from uploaded TAR files. Extraction occurs server-side without downloading the archive. The benchmark continues uploading and extracting TAR archives throughout the specified duration.

The command measures archive upload throughput, extraction rates, and error rates. Snowball performance tests bulk import efficiency for large object collections. Compressed archives test extraction performance with decompression overhead.

The snowball benchmark provides metrics for bulk import operations.

The output includes archive upload throughput measured in MB/s showing TAR transfer rates. Objects extracted per second displays the rate of object creation from archives. Latency percentiles show response times at p50, p90, p99, and p99.9 levels for archive operations.

The benchmark reports extraction throughput showing how quickly objects become available. Error rates display counts and percentages of failed upload or extraction operations. Total object counts show how many objects were successfully imported during the test.

Compression metrics reveal overhead and space savings for compressed archives. Archive processing time shows end-to-end duration from upload to extraction completion.

Test TAR archive upload and extraction with default settings:

warp snowball --host myaistor.example.com:9000 --access-key ACCESS_KEY --secret-key SECRET_KEY

The command uploads TAR archives containing 50 objects of 512KiB each.

Test snowball with larger objects per archive:

warp snowball --host myaistor.example.com:9000 --obj.size 5MiB --objs.per 100 --access-key YOUR_ACCESS_KEY --secret-key YOUR_SECRET_KEY

This tests bulk import of larger objects in each archive.

Test archives with many small objects:

warp snowball --host myaistor.example.com:9000 --obj.size 100KiB --objs.per 200 --access-key YOUR_ACCESS_KEY --secret-key YOUR_SECRET_KEY

Many small objects test extraction efficiency for numerous files.

Test compressed TAR upload and extraction:

warp snowball --host myaistor.example.com:9000 --compress --obj.size 1MiB --access-key YOUR_ACCESS_KEY --secret-key YOUR_SECRET_KEY

Compression testing reveals performance impact of decompression during extraction.

Test archives with fewer but larger objects:

warp snowball --host myaistor.example.com:9000 --obj.size 10MiB --objs.per 10 --access-key YOUR_ACCESS_KEY --secret-key YOUR_SECRET_KEY

Fewer objects test overhead of archive processing versus extraction.

Test concurrent TAR archive uploads:

warp snowball --host myaistor.example.com:9000 --concurrent 20 --objs.per 50 --access-key YOUR_ACCESS_KEY --secret-key YOUR_SECRET_KEY

This tests how the system handles many simultaneous bulk import operations.