mc pipe

The mc pipe command streams content from STDIN to a target object.

echo "My Meeting Notes" | mc pipe s3/engineering/meeting-notes.txt

mc [GLOBALFLAGS] pipe                              \
                 TARGET                            \
                 [--attr "string"]                 \
                 [--checksum "string"]             \
                 [--enc-kms "string"]              \
                 [--enc-s3 "string"]               \
                 [--enc-c "string"]                \
                 [--storage-class, --sc "string"]  \
                 [--tags "string"]

mc pipe supports concurrent uploads for better throughput of large streams.

Required

The full path to the alias or prefix where the command should run.

Optional

Add custom metadata for the object.

Specify key-value pairs as KEY=VALUE\;, separating each pair with a back slash and semicolon (\;). For example, --attr key1=value1\;key2=value2\;key3=value3.

Optional

Add a checksum to an uploaded object.

Valid values are:

• MD5
• CRC32
• CRC32C
• SHA1
• SHA256

The flag requires server trailing headers and works with AWS or AIStor targets.

Encrypt or decrypt objects using server-side SSE-KMS encryption with client-managed keys.

The parameter accepts a key-value pair formatted as KEY=VALUE

• KEY- The full path to the object as alias/bucket/path/object.ext. You can specify only the top-level path to use a single encryption key for all operations in that path.
• VALUE - Specify an existing data key on the external KMS. See the mc admin kms key create reference for creating data keys.

For example:

--enc-kms "myminio/mybucket/prefix/object.obj=mybucketencryptionkey"

You can specify multiple encryption keys by repeating the parameter.

Specify the path to a prefix to apply encryption to all matching objects at that path:

--enc-kms "myminio/mybucket/prefix/=mybucketencryptionkey"

Optional

Encrypt or decrypt objects using server-side SSE-S3 encryption with KMS-managed keys. Specify the full path to the object as alias/bucket/prefix/object.

For example:

--enc-s3 "myminio/mybucket/prefix/object.obj"

You can specify the parameter multiple times to denote different object(s) to encrypt:

--enc-s3 "myminio/mybucket/foo/fooobject.obj" --enc-s3 "myminio/mybucket/bar/barobject.obj"

Specify the path to a prefix to apply encryption to all matching objects at that path:

--enc-s3 "myminio/mybucket/foo"

Optional

Encrypt or decrypt objects using server-side SSE-C encryption with client-managed keys.

The parameter accepts a key-value pair formatted as KEY=VALUE

• KEY: The full path to the object as alias/bucket/path/object.ext. You can specify only the top-level path to use a single encryption key for all operations in that path.
• VALUE: Specify either a 32-byte RawBase64-encoded key or a 64-byte hex-encoded key for use with SSE-C encryption. Raw Base64 encoding rejects =-padded keys. Omit the padding or use a Base64 encoder that supports RAW formatting.

For example:

# RawBase64-Encoded string "mybucket32byteencryptionkeyssec"
--enc-c "myminio/mybucket/prefix/object.obj=bXlidWNrZXQzMmJ5dGVlbmNyeXB0aW9ua2V5c3NlYwo"

You can specify multiple encryption keys by repeating the parameter.

Specify the path to a prefix to apply encryption to all matching objects at that path:

--enc-c "myminio/mybucket/prefix/=bXlidWNrZXQzMmJ5dGVlbmNyeXB0aW9ua2V5c3NlYwo"

Optional

Set the storage class for the new object at the TARGET.

See Amazons documentation for more information on S3 storage classes.

Optional

Applies one or more tags to the TARGET.

Specify an ampersand-separated list of key-value pairs as KEY1=VALUE1&KEY2=VALUE2, where each pair represents one tag to assign to the objects.

This command supports any of the global flags.

The following command writes the contents of STDIN to the /tmp folder on the local filesystem.

mc pipe /tmp/hello-world.go

The following command first streams the contents of an iso image for Debian and then uses the stream to create the object at an S3 path.

cat debian-live-11.5.0-amd64-mate.iso | mc pipe s3/opensource-isos/debian-11-5.iso

The following command first streams a MySQL database and uses the stream to create a backup on S3 with mc pipe:

mysqldump -u root -p ******* accountsdb | mc pipe s3/sql-backups/backups/accountsdb-sep-28-2022.sql

The following command takes the STDIN stream and creates an object on the Reduced Redundancy storage class on S3.

 mc pipe --storage-class REDUCED_REDUNDANCY s3/personalbuck/meeting-notes.txt

The following command uploads an MP3 file to a AIStor deployment with an ALIAS of myminio and a music bucket. The object writes with some metadata for Cache-Control and Artist.

cat music.mp3 | mc pipe --attr "Cache-Control=max-age=90000,min-fresh=9000;Artist=Unknown" myminio/music/guitar.mp3

The following command creates an object on a AIStor deployment with an ALIAS of myminio in bucket mybucket with two tags. AIStor supports adding up to 10 custom tags to an object.

tar cvf - . | mc pipe --tags "category=prod&type=backup" myminio/mybucket/backup.tar