Recent changes (master)
The following changes since commit 9dc528b1638b625b5e167983a74de4e85c5859ea:
lib/rand: get rid of unused MAX_SEED_BUCKETS (2022-08-10 09:51:49 -0600)
are available in the Git repository at:
git://git.kernel.dk/fio.git master
for you to fetch changes up to 7a7bcae0610d872951bc22dc310105c7ec1157af:
Merge branch 's3_crypto' of github.com:hualongfeng/fio (2022-08-11 15:39:02 -0400)
----------------------------------------------------------------
Feng, Hualong (3):
engines/http: Add storage class option for s3 engines/http: Add s3 crypto options for s3
doc: Add usage and example about s3 storage class and crypto
[email protected] (1):
ioengines: merge filecreate, filestat, filedelete engines to fileoperations.c
Vincent Fu (1):
Merge branch 's3_crypto' of github.com:hualongfeng/fio
HOWTO.rst | 14 ++ Makefile | 2 +- engines/filecreate.c | 118 -------------- engines/filedelete.c | 115 -------------- engines/fileoperations.c | 318 +++++++++++++++++++++++++++++++++++++ engines/filestat.c | 190 ---------------------- engines/http.c | 178 ++++++++++++++++++--- examples/http-s3-crypto.fio | 38 +++++ examples/http-s3-storage-class.fio | 37 +++++ fio.1 | 9 ++ 10 files changed, 577 insertions(+), 442 deletions(-) delete mode 100644 engines/filecreate.c delete mode 100644 engines/filedelete.c create mode 100644 engines/fileoperations.c delete mode 100644 engines/filestat.c create mode 100644 examples/http-s3-crypto.fio create mode 100644 examples/http-s3-storage-class.fio
---
Diff of recent changes:
HOWTO.rst +14 -0
@@ -2692,6 +2692,20 @@ with the caveat that when used on the command line, they must come after the The S3 key/access id. +.. option:: http_s3_sse_customer_key=str : [http] + + The encryption customer key in SSE server side. + +.. option:: http_s3_sse_customer_algorithm=str : [http] + + The encryption customer algorithm in SSE server side. + Default is **AES256** + +.. option:: http_s3_storage_class=str : [http] + + Which storage class to access. User-customizable settings. + Default is **STANDARD** + .. option:: http_swift_auth_token=str : [http] The Swift auth token. See the example configuration file on how
Makefile +1 -1
@@ -56,7 +56,7 @@ SOURCE := $(sort $(patsubst $(SRCDIR)/%,%,$(wildcard $(SRCDIR)/crc/*.c)) \ pshared.c options.c \ smalloc.c filehash.c profile.c debug.c engines/cpu.c \ engines/mmap.c engines/sync.c engines/null.c engines/net.c \ - engines/ftruncate.c engines/filecreate.c engines/filestat.c engines/filedelete.c \ + engines/ftruncate.c engines/fileoperations.c \ engines/exec.c \ server.c client.c iolog.c backend.c libfio.c flow.c cconv.c \ gettime-thread.c helpers.c json.c idletime.c td_error.c \
engines/filecreate.c deleted +0 -118
118 lines not on this page · raw message
engines/filedelete.c deleted +0 -115
115 lines not on this page · raw message
engines/fileoperations.c new file +318 -0
318 lines not on this page · raw message
engines/filestat.c deleted +0 -190
190 lines not on this page · raw message
engines/http.c +160 -18
235 lines not on this page · raw message
examples/http-s3-crypto.fio new file +38 -0
@@ -0,0 +1,38 @@ +# Example test for the HTTP engine's S3 support against Amazon AWS. +# Obviously, you have to adjust the S3 credentials; for this example, +# they're passed in via the environment. +# And you can set the SSE Customer Key and Algorithm to test Server +# Side Encryption. +# + +[global] +ioengine=http +name=test +direct=1 +filename=/larsmb-fio-test/object +http_verbose=0 +https=on +http_mode=s3 +http_s3_key=${S3_KEY} +http_s3_keyid=${S3_ID} +http_host=s3.eu-central-1.amazonaws.com +http_s3_region=eu-central-1 +http_s3_sse_customer_key=${SSE_KEY} +http_s3_sse_customer_algorithm=AES256 +group_reporting + +# With verify, this both writes and reads the object +[create] +rw=write +bs=4k +size=64k +io_size=4k +verify=sha256 + +[trim] +stonewall +rw=trim +bs=4k +size=64k +io_size=4k +
examples/http-s3-storage-class.fio new file +37 -0
@@ -0,0 +1,37 @@ +# Example test for the HTTP engine's S3 support against Amazon AWS. +# Obviously, you have to adjust the S3 credentials; for this example, +# they're passed in via the environment. +# And here add storage class parameter, you can set normal test for +# STANDARD and compression test for another storage class. +# + +[global] +ioengine=http +name=test +direct=1 +filename=/larsmb-fio-test/object +http_verbose=0 +https=on +http_mode=s3 +http_s3_key=${S3_KEY} +http_s3_keyid=${S3_ID} +http_host=s3.eu-central-1.amazonaws.com +http_s3_region=eu-central-1 +http_s3_storage_class=${STORAGE_CLASS} +group_reporting + +# With verify, this both writes and reads the object +[create] +rw=write +bs=4k +size=64k +io_size=4k +verify=sha256 + +[trim] +stonewall +rw=trim +bs=4k +size=64k +io_size=4k +
fio.1 +9 -0
@@ -2308,6 +2308,15 @@ The S3 secret key. .BI (http)http_s3_keyid \fR=\fPstr The S3 key/access id. .TP +.BI (http)http_s3_sse_customer_key \fR=\fPstr +The encryption customer key in SSE server side. +.TP +.BI (http)http_s3_sse_customer_algorithm \fR=\fPstr +The encryption customer algorithm in SSE server side. Default is \fBAES256\fR +.TP +.BI (http)http_s3_storage_class \fR=\fPstr +Which storage class to access. User-customizable settings. Default is \fBSTANDARD\fR +.TP .BI (http)http_swift_auth_token \fR=\fPstr The Swift auth token. See the example configuration file on how to retrieve this.