Recent changes (master)
The following changes since commit 8ac0eec77e9c25a054ff1fbff243ffdffcd6e858:
nvme/streams: avoid allocating too large a buffer (2024-08-05 18:15:24 +0000)
are available in the Git repository at:
git://git.kernel.dk/fio.git master
for you to fetch changes up to 55b10ccca6a2ba623360a38c3fdf7332461646bb:
Merge branch 'master' of https://github.com/scaleoutsean/fio (2024-08-06 09:17:03 -0600)
----------------------------------------------------------------
Jens Axboe (1):
Merge branch 'master' of https://github.com/scaleoutsean/fio
Sean Lee (1):
Improve http_host, filename in docs and example/http-s3.fio
HOWTO.rst | 14 ++++++++++++-- examples/http-s3.fio | 16 +++++++++++++--- fio.1 | 11 +++++++++-- 3 files changed, 34 insertions(+), 7 deletions(-)
---
Diff of recent changes:
HOWTO.rst +12 -2
@@ -805,6 +805,11 @@ Target file/device Note: Windows and FreeBSD (refer to geom(4)) prevent write access to areas of the disk containing in-use data (e.g. filesystems). + For HTTP and S3 access, specify a valid URL path or S3 key, respectively. + A filename for path-style S3 includes a bucket name (:file:`/bucket/k/e.y`) + while a virtual-hosted-style S3 filename :file:`/k/e.y` does not because + its bucket name is specified in :option:`http_host`. + The filename "`-`" is a reserved name, meaning *stdin* or *stdout*. Which of the two depends on the read/write direction set.
@@ -2934,8 +2939,13 @@ with the caveat that when used on the command line, they must come after the .. option:: http_host=str : [http] - Hostname to connect to. For S3, this could be the bucket hostname. - Default is **localhost** + Hostname to connect to. HTTP port 80 is used automatically when the value of + the https parameter is *off*, and HTTPS port 443 if it is *on*. A + virtual-hosted-style S3 hostname starts with a bucket name, while a + path-style S3 hostname does not. See + https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html for + detailed examples. + Default is **localhost** (path-style S3 hostname) .. option:: http_user=str : [http]
examples/http-s3.fio +13 -3
@@ -1,19 +1,29 @@ # Example test for the HTTP engine's S3 support against Amazon AWS. # Obviously, you have to adjust the S3 credentials; for this example, # they're passed in via the environment. -# +# For non-AWS S3 implementations, refer to your S3 vendor's region +# settings. Note that the region value appears twice, in http_host and +# http_s3_region. +# This example uses virtual-hosted-style requests: +# https://bucket-name.s3.region-code.amazonaws.com/k/e.y +# For path-style, prefix the key with the bucket name in the filename +# so that filename=/bucket-name/k/e.y: +# https://s3.region-code.amazonaws.com/bucket-name/k/e.y + +# Reference for Virtual-hosted-style vs. Path-style URLs: +# https://docs.aws.amazon.com/AmazonS3/latest/userguide/VirtualHosting.html [global] ioengine=http name=test direct=1 -filename=/larsmb-fio-test/object +filename=/k/e.y http_verbose=0 https=on http_mode=s3 http_s3_key=${S3_KEY} http_s3_keyid=${S3_ID} -http_host=s3.eu-central-1.amazonaws.com +http_host=bucket-name.s3.eu-central-1.amazonaws.com http_s3_region=eu-central-1 group_reporting
fio.1 +9 -2
@@ -585,6 +585,11 @@ the first device, `\\\\.\\PhysicalDrive1' for the second etc. Note: Windows and FreeBSD prevent write access to areas of the disk containing in-use data (e.g. filesystems). .P +For HTTP and S3 access, specify a valid URL path or S3 key, respectively. +A filename for path-style S3 includes a bucket name (`/bucket/k/e.y') +while a virtual-hosted-style S3 filename (`/k/e.y') does not because its +bucket name is specified in \fBhttp_host\fR. +.P The filename `\-' is a reserved name, meaning *stdin* or *stdout*. Which of the two depends on the read/write direction set. .RE
@@ -2554,8 +2559,10 @@ Touching all objects affects ceph caches and likely impacts test results. Enabled by default. .TP .BI (http)http_host \fR=\fPstr -Hostname to connect to. For S3, this could be the bucket name. Default -is \fBlocalhost\fR +Hostname to connect to. HTTP port 80 is used automatically when the value +of the \fBhttps\fP parameter is \fRoff\fP, and HTTPS port 443 if it is \Ron\fP. +A virtual-hosted-style S3 hostname starts with a bucket name, while a +path-style S3 hostname does not. Default is \fBlocalhost\fR. .TP .BI (http)http_user \fR=\fPstr Username for HTTP authentication.