Skip to contentSkip to conversation

Recent changes (master)

PATCH Jens Axboe 1 participant 1 message Aug 12, 2022 Kbuild

lore
Share & tools
lore

Jens Axboe

author
#0 lore raw

The following changes since commit 9dc528b1638b625b5e167983a74de4e85c5859ea:

lib/rand: get rid of unused MAX_SEED_BUCKETS (2022-08-10 09:51:49 -0600)

are available in the Git repository at:

git://git.kernel.dk/fio.git master

for you to fetch changes up to 7a7bcae0610d872951bc22dc310105c7ec1157af:

Merge branch 's3_crypto' of github.com:hualongfeng/fio (2022-08-11 15:39:02 -0400)

----------------------------------------------------------------
Feng, Hualong (3):
engines/http: Add storage class option for s3 engines/http: Add s3 crypto options for s3
doc: Add usage and example about s3 storage class and crypto

[email protected] (1):
ioengines: merge filecreate, filestat, filedelete engines to fileoperations.c

Vincent Fu (1):
Merge branch 's3_crypto' of github.com:hualongfeng/fio

 HOWTO.rst                          |  14 ++
 Makefile                           |   2 +-
 engines/filecreate.c               | 118 --------------
 engines/filedelete.c               | 115 --------------
 engines/fileoperations.c           | 318 +++++++++++++++++++++++++++++++++++++
 engines/filestat.c                 | 190 ----------------------
 engines/http.c                     | 178 ++++++++++++++++++---
 examples/http-s3-crypto.fio        |  38 +++++
 examples/http-s3-storage-class.fio |  37 +++++
 fio.1                              |   9 ++
 10 files changed, 577 insertions(+), 442 deletions(-)
 delete mode 100644 engines/filecreate.c
 delete mode 100644 engines/filedelete.c
 create mode 100644 engines/fileoperations.c
 delete mode 100644 engines/filestat.c
 create mode 100644 examples/http-s3-crypto.fio
 create mode 100644 examples/http-s3-storage-class.fio

---

Diff of recent changes:

HOWTO.rst +14 -0
index 104cce2d..05fc117f 100644
@@ -2692,6 +2692,20 @@ with the caveat that when used on the command line, they must come after the
 
 	The S3 key/access id.
 
+.. option:: http_s3_sse_customer_key=str : [http]
+
+        The encryption customer key in SSE server side.
+
+.. option:: http_s3_sse_customer_algorithm=str : [http]
+
+        The encryption customer algorithm in SSE server side.
+        Default is **AES256**
+
+.. option:: http_s3_storage_class=str : [http]
+
+        Which storage class to access. User-customizable settings.
+        Default is **STANDARD**
+
 .. option:: http_swift_auth_token=str : [http]
 
 	The Swift auth token. See the example configuration file on how
Makefile +1 -1
index 188a74d7..634d2c93 100644
@@ -56,7 +56,7 @@ SOURCE :=	$(sort $(patsubst $(SRCDIR)/%,%,$(wildcard $(SRCDIR)/crc/*.c)) \
 		pshared.c options.c \
 		smalloc.c filehash.c profile.c debug.c engines/cpu.c \
 		engines/mmap.c engines/sync.c engines/null.c engines/net.c \
-		engines/ftruncate.c engines/filecreate.c engines/filestat.c engines/filedelete.c \
+		engines/ftruncate.c engines/fileoperations.c \
 		engines/exec.c \
 		server.c client.c iolog.c backend.c libfio.c flow.c cconv.c \
 		gettime-thread.c helpers.c json.c idletime.c td_error.c \
engines/filecreate.c deleted +0 -118

118 lines not on this page · raw message

engines/filedelete.c deleted +0 -115

115 lines not on this page · raw message

engines/fileoperations.c new file +318 -0

318 lines not on this page · raw message

engines/filestat.c deleted +0 -190

190 lines not on this page · raw message

engines/http.c +160 -18

235 lines not on this page · raw message

examples/http-s3-crypto.fio new file +38 -0
new file mode 100644
index 00000000..2403746e
@@ -0,0 +1,38 @@
+# Example test for the HTTP engine's S3 support against Amazon AWS.
+# Obviously, you have to adjust the S3 credentials; for this example,
+# they're passed in via the environment.
+# And you can set the SSE Customer Key and Algorithm to test Server
+# Side Encryption.
+#
+
+[global]
+ioengine=http
+name=test
+direct=1
+filename=/larsmb-fio-test/object
+http_verbose=0
+https=on
+http_mode=s3
+http_s3_key=${S3_KEY}
+http_s3_keyid=${S3_ID}
+http_host=s3.eu-central-1.amazonaws.com
+http_s3_region=eu-central-1
+http_s3_sse_customer_key=${SSE_KEY}
+http_s3_sse_customer_algorithm=AES256
+group_reporting
+
+# With verify, this both writes and reads the object
+[create]
+rw=write
+bs=4k
+size=64k
+io_size=4k
+verify=sha256
+
+[trim]
+stonewall
+rw=trim
+bs=4k
+size=64k
+io_size=4k
+
examples/http-s3-storage-class.fio new file +37 -0
new file mode 100644
index 00000000..9ee23837
@@ -0,0 +1,37 @@
+# Example test for the HTTP engine's S3 support against Amazon AWS.
+# Obviously, you have to adjust the S3 credentials; for this example,
+# they're passed in via the environment.
+# And here add storage class parameter, you can set normal test for
+# STANDARD and compression test for another storage class.
+#
+
+[global]
+ioengine=http
+name=test
+direct=1
+filename=/larsmb-fio-test/object
+http_verbose=0
+https=on
+http_mode=s3
+http_s3_key=${S3_KEY}
+http_s3_keyid=${S3_ID}
+http_host=s3.eu-central-1.amazonaws.com
+http_s3_region=eu-central-1
+http_s3_storage_class=${STORAGE_CLASS}
+group_reporting
+
+# With verify, this both writes and reads the object
+[create]
+rw=write
+bs=4k
+size=64k
+io_size=4k
+verify=sha256
+
+[trim]
+stonewall
+rw=trim
+bs=4k
+size=64k
+io_size=4k
+
fio.1 +9 -0
index ce9bf3ef..6630525f 100644
@@ -2308,6 +2308,15 @@ The S3 secret key.
 .BI (http)http_s3_keyid \fR=\fPstr
 The S3 key/access id.
 .TP
+.BI (http)http_s3_sse_customer_key \fR=\fPstr
+The encryption customer key in SSE server side.
+.TP
+.BI (http)http_s3_sse_customer_algorithm \fR=\fPstr
+The encryption customer algorithm in SSE server side. Default is \fBAES256\fR
+.TP
+.BI (http)http_s3_storage_class \fR=\fPstr
+Which storage class to access. User-customizable settings. Default is \fBSTANDARD\fR
+.TP
 .BI (http)http_swift_auth_token \fR=\fPstr
 The Swift auth token. See the example configuration file on how to
 retrieve this.