Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by smtp.lore.kernel.org (Postfix) with ESMTPS id DBDCBC433EF for ; Wed, 2 Feb 2022 09:05:51 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=lists.infradead.org; s=bombadil.20210309; h=Sender:List-Subscribe:List-Help :List-Post:List-Archive:List-Unsubscribe:List-Id:Content-Transfer-Encoding: Content-Type:MIME-Version:Message-ID:Date:Subject:CC:To:From:Reply-To: Content-ID:Content-Description:Resent-Date:Resent-From:Resent-Sender: Resent-To:Resent-Cc:Resent-Message-ID:In-Reply-To:References:List-Owner; bh=yQBGg+LfdQfY8yxFce5VZtIWY7w2JdH536b0FkvmTfk=; b=Z9hIXGW5sd68TkuIWZ+S8SmTTM sbQ/dAV+d7/mp7wYBsw7GhjVrFocxEAxtmVWeNpev+jnkIkumFkfjJ0LglK1tXlDtTzqMfb6C5Dd8 2hoLEedF/t+eCPMRu7o3McAuzth0E6vTRwt42EETfc3ViNPanv1q2kBswpVZKNAkmp/g1xwJhcu9n 2iJgaOATAEGx1ndanbro74Iuh8DFYbe8RlTKgESMo9vvBF388f2quIdpAIInhPO74tjYjgSUQzrV6 Bc2iZDdoRGs93A5ABriCz6hqqTR61+V7H3AYJYWLkLWweLfsFO6vFt7IL7yj5klB/EoTxSiCub9zc SkunlaRw==; Received: from localhost ([::1] helo=bombadil.infradead.org) by bombadil.infradead.org with esmtp (Exim 4.94.2 #2 (Red Hat Linux)) id 1nFBaG-00EgzD-VM; Wed, 02 Feb 2022 09:05:49 +0000 Received: from mail-dm6nam10on2061a.outbound.protection.outlook.com ([2a01:111:f400:7e88::61a] helo=NAM10-DM6-obe.outbound.protection.outlook.com) by bombadil.infradead.org with esmtps (Exim 4.94.2 #2 (Red Hat Linux)) id 1nFBZq-00Egma-79 for linux-nvme@lists.infradead.org; Wed, 02 Feb 2022 09:05:29 +0000 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=KCEp+U6A/mFbb/ouaI3HfaQxRvIxG72GrUlkclFzMwwh1bmIW5X8tXid0CJyLZzemZbmhIP1QFQJvvSrClFFdzYzHoKJ2IL3L1qkxkk5v+edb7eyOg91yOi1TCPkZXQ0vhD6XNnGSyKuu7OuaFH3eC30uR+OsolNCIThae5pVaR9qlBEYT7nb0Eps7zx9lIxaHvFZNZqz885tSWqIYfz+m5U42H6le1pI7rdMcyDEHSTzF0yFDOAPLJ295+MH4iMi1mGgpzLpov43PnpnL/Sj0IJRxn/fvGFV++YSp3xkS8Zrj3ShXuU0q7JGcRiHbA48VCV+XudRbNPNN+k9uMFtA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-AntiSpam-MessageData-ChunkCount:X-MS-Exchange-AntiSpam-MessageData-0:X-MS-Exchange-AntiSpam-MessageData-1; bh=yQBGg+LfdQfY8yxFce5VZtIWY7w2JdH536b0FkvmTfk=; b=Vg5dBW4b/YHDEMAbz4LWkA4tAkySLoojy3ZmIbGCCQHyEhVMFgUfjO4QrDKuP7wkj2j6E4nOO9twP4l2ogkQ+Banem3bctzcRbXHB6mxcKmTNQSqwyftEhLOxWB2DrGJU5oYk9UHCSLuCRUL3s1eLEELZoAfm52LMWPXnhIadxLI1d3KRWM+MOXEFint4bATyCwk2yzTp7JSVP6xiOhvNNi6d+8yFouIdGAYztqVWpKk8hiFNVdFjoDjr2+Q0qMojCtQ31nWKOX6/QkcBFTi7SlOvfSqN2mPJwHqBJfJBQgGgKiPkj9HYfS/pvpKG9FSRvrRYWym+K23WuwKT5aK2Q== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass (sender ip is 12.22.5.235) smtp.rcpttodomain=kernel.org smtp.mailfrom=nvidia.com; dmarc=pass (p=reject sp=reject pct=100) action=none header.from=nvidia.com; dkim=none (message not signed); arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=Nvidia.com; s=selector2; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=yQBGg+LfdQfY8yxFce5VZtIWY7w2JdH536b0FkvmTfk=; b=iZ18WGlcNNKyqzy+GVqlbnUw30YS6Dz3m+WsdIsMT3HJ2pIntm28M2abb1D4r92CZBOzECzfj+WLPNPwvkyG/6J94RNf6fWQhx1hfUi/YSQt/LryKQ+FZGGmsCx2XWX2Hc93LL/zoVrSFv+plINHW78F0mdcht2PjbsVsiGE9i/ocdR9e1CC+G1dK7twJm5gJv72F6lUmrjrG5PsEp5wy5wC2awnOVTCrWKYWsnYurf1mweYLesxPDGpxa1NbsLOTiSnfdGYEIRWEw4T9WODFjR89jRU4yjfwWCmKJ0jHpBjc4kmT6RHaoCS5eDMu83icgGoZfvCyiwsYTMkPWHGwQ== Received: from DM6PR04CA0030.namprd04.prod.outlook.com (2603:10b6:5:334::35) by DM5PR12MB1163.namprd12.prod.outlook.com (2603:10b6:3:7a::18) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4930.20; Wed, 2 Feb 2022 09:05:17 +0000 Received: from DM6NAM11FT008.eop-nam11.prod.protection.outlook.com (2603:10b6:5:334:cafe::95) by DM6PR04CA0030.outlook.office365.com (2603:10b6:5:334::35) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.4951.11 via Frontend Transport; Wed, 2 Feb 2022 09:05:16 +0000 X-MS-Exchange-Authentication-Results: spf=pass (sender IP is 12.22.5.235) smtp.mailfrom=nvidia.com; dkim=none (message not signed) header.d=none;dmarc=pass action=none header.from=nvidia.com; Received-SPF: Pass (protection.outlook.com: domain of nvidia.com designates 12.22.5.235 as permitted sender) receiver=protection.outlook.com; client-ip=12.22.5.235; helo=mail.nvidia.com; Received: from mail.nvidia.com (12.22.5.235) by DM6NAM11FT008.mail.protection.outlook.com (10.13.172.85) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384) id 15.20.4951.12 via Frontend Transport; Wed, 2 Feb 2022 09:05:16 +0000 Received: from rnnvmail201.nvidia.com (10.129.68.8) by DRHQMAIL107.nvidia.com (10.27.9.16) with Microsoft SMTP Server (TLS) id 15.0.1497.18; Wed, 2 Feb 2022 09:04:55 +0000 Received: from dev.nvidia.com (10.126.230.35) by rnnvmail201.nvidia.com (10.129.68.8) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.2.986.9; Wed, 2 Feb 2022 01:04:54 -0800 From: Chaitanya Kulkarni To: CC: , , , Chaitanya Kulkarni Subject: [PATCH V3 0/2] nvmet: add buffered_io support for bdev-ns Date: Wed, 2 Feb 2022 01:04:43 -0800 Message-ID: <20220202090445.43615-1-chaitanyak@nvidia.com> X-Mailer: git-send-email 2.29.0 MIME-Version: 1.0 Content-Type: text/plain; charset="y" Content-Transfer-Encoding: 8bit X-Originating-IP: [10.126.230.35] X-ClientProxiedBy: rnnvmail203.nvidia.com (10.129.68.9) To rnnvmail201.nvidia.com (10.129.68.8) X-EOPAttributedMessage: 0 X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 099b6a1b-afd4-4e83-1244-08d9e62b213d X-MS-TrafficTypeDiagnostic: DM5PR12MB1163:EE_ X-Microsoft-Antispam-PRVS: X-MS-Oob-TLC-OOBClassifiers: OLM:2399; X-MS-Exchange-SenderADCheck: 1 X-MS-Exchange-AntiSpam-Relay: 0 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: 1RR96CYv5kED0aBR440ArUaiQwrgUXLocw852Hey3WnJe3QHftU2n6o8vP8oS1jigX3+/WCwog+5Sin/DHJkStajIdqAsJnVBzDYayYyoJYNL6FUjtydOs5wB/1lFFoV4P2ARI91DNW0R+eIVQoJmHjs7318gXiNBFbslHjk7+AljsGnUvuE49wuHWoXdeuIpIAbyG8NmdBfmi3Z+bFY4WW+TN7vr9YnXFCYuyzNSf7UVxQY2LGs8HeMuIPrsMSI20wyOxZZvyO/+An2T97TQzZN6rmCOooloULeUumkQUjBSJqlkFhh03u5+H0une5Gki0NGd1WU5LLDImrIQ87sJyyEfBvS3k1lYa1JIK3v6pJzoscSsEKctMiMckzRgA1xDdFMsK0tvVO6hPLgymvSgyGVH+jFW+ntpUzYdmiwSCMAIAXJVMyQRBtF3XPcAaHiwdoOhFqHrTWeJp3lc5btMmYbZXas/fispLSYAhbAvoC4LUzshZxzxFBU2QDCZrK3O7rrydMqk0HLFWyeQGEQ6X30c2KIDNEaT+cAkan6hMw7mFQsOfDA/XzQeuuWtxCEg6njUGdQtCwDFpHPwe3WLr38xe0zT+1QAq11P5pwrJ3z86Bw1FIvxkofo2a5w1++cJ6xcoBZU9rwnan5+T2RMTY/42z/qlhJw3T9iHT+hzdBr/RB+ToHUXP7PPpjB3ajd2K2FaVVcWUS0MoF5CstA== X-Forefront-Antispam-Report: CIP:12.22.5.235; CTRY:US; LANG:en; SCL:1; SRV:; IPV:CAL; SFV:NSPM; H:mail.nvidia.com; PTR:InfoNoRecords; CAT:NONE; SFS:(13230001)(4636009)(36840700001)(46966006)(40470700004)(36756003)(2906002)(81166007)(16526019)(107886003)(2616005)(7696005)(6666004)(186003)(426003)(26005)(54906003)(336012)(8936002)(70206006)(70586007)(82310400004)(8676002)(6916009)(1076003)(4326008)(30864003)(316002)(83380400001)(508600001)(86362001)(47076005)(19627235002)(356005)(5660300002)(36860700001)(40460700003)(36900700001)(559001)(579004); DIR:OUT; SFP:1101; X-OriginatorOrg: Nvidia.com X-MS-Exchange-CrossTenant-OriginalArrivalTime: 02 Feb 2022 09:05:16.4018 (UTC) X-MS-Exchange-CrossTenant-Network-Message-Id: 099b6a1b-afd4-4e83-1244-08d9e62b213d X-MS-Exchange-CrossTenant-Id: 43083d15-7273-40c1-b7db-39efd9ccc17a X-MS-Exchange-CrossTenant-OriginalAttributedTenantConnectingIp: TenantId=43083d15-7273-40c1-b7db-39efd9ccc17a; Ip=[12.22.5.235]; Helo=[mail.nvidia.com] X-MS-Exchange-CrossTenant-AuthSource: DM6NAM11FT008.eop-nam11.prod.protection.outlook.com X-MS-Exchange-CrossTenant-AuthAs: Anonymous X-MS-Exchange-CrossTenant-FromEntityHeader: HybridOnPrem X-MS-Exchange-Transport-CrossTenantHeadersStamped: DM5PR12MB1163 X-BeenThere: linux-nvme@lists.infradead.org X-Mailman-Version: 2.1.34 Precedence: list List-Id: List-Unsubscribe: , List-Archive: List-Post: List-Help: List-Subscribe: , Sender: "Linux-nvme" Errors-To: linux-nvme-bounces+linux-nvme=archiver.kernel.org@lists.infradead.org From: Chaitanya Kulkarni Hi, In current implementation like NVMeOF target file backed namespace we don't support buffered_io for block device backed namesapce. This allows user to configure block device backend in buffered_io mode so user can get the benefits of caching. Please note that using this combination host needs to periodically sync between cache and storage or set the system cache parameters accordingly. NVMeOF target is not responsible for data consistency between cache and storage. fio data verification, rand read/write performance numbers :- 1. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=0 :- fio Verify job :- ---------------------- read: IOPS=24.3k, BW=94.9MiB/s (99.5MB/s)(506MiB/5335msec) write: IOPS=27.9k, BW=109MiB/s (114MB/s)(800MiB/7353msec); fio Random Write :- ---------------------- write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60002msec); write: IOPS=68.6k, BW=268MiB/s (281MB/s)(15.7GiB/60003msec); write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60003msec); write: IOPS=68.3k, BW=267MiB/s (280MB/s)(15.6GiB/60003msec); write: IOPS=69.0k, BW=269MiB/s (283MB/s)(15.8GiB/60002msec); write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec); write: IOPS=69.8k, BW=272MiB/s (286MB/s)(16.0GiB/60003msec); write: IOPS=69.8k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec); write: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60003msec); write: IOPS=71.0k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec); fio Random Read :- ---------------------- read: IOPS=72.8k, BW=284MiB/s (298MB/s)(16.7GiB/60037msec) read: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60037msec) read: IOPS=67.5k, BW=264MiB/s (277MB/s)(15.5GiB/60033msec) read: IOPS=74.5k, BW=291MiB/s (305MB/s)(17.1GiB/60035msec) read: IOPS=72.4k, BW=283MiB/s (296MB/s)(16.6GiB/60036msec) read: IOPS=71.8k, BW=281MiB/s (294MB/s)(16.5GiB/60036msec) read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60035msec) read: IOPS=72.3k, BW=283MiB/s (296MB/s)(16.6GiB/60039msec) read: IOPS=72.8k, BW=285MiB/s (298MB/s)(16.7GiB/60037msec) read: IOPS=74.6k, BW=291MiB/s (306MB/s)(17.1GiB/60036msec) 2. NVMeOF Target ns device_path=/dev/nvme0n1 & buffered_io=1 :- fio Verify job :- ---------------------- read: IOPS=195k, BW=763MiB/s (800MB/s)(506MiB/663msec) write: IOPS=123k, BW=481MiB/s (505MB/s)(800MiB/1662msec); fio Random Write :- ---------------------- write: IOPS=4248k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec); write: IOPS=4157k, BW=15.9GiB/s (17.0GB/s)(951GiB/60002msec); write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec); write: IOPS=4130k, BW=15.8GiB/s (16.9GB/s)(945GiB/60002msec); write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60002msec); write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec); write: IOPS=4166k, BW=15.9GiB/s (17.1GB/s)(954GiB/60002msec); write: IOPS=4106k, BW=15.7GiB/s (16.8GB/s)(940GiB/60002msec); write: IOPS=4181k, BW=15.9GiB/s (17.1GB/s)(957GiB/60001msec); write: IOPS=4153k, BW=15.8GiB/s (17.0GB/s)(951GiB/60002msec); fio Random Read :- ---------------------- read: IOPS=4942k, BW=18.9GiB/s (20.2GB/s)(1131GiB/60002msec) read: IOPS=4940k, BW=18.8GiB/s (20.2GB/s)(1131GiB/60002msec) read: IOPS=5041k, BW=19.2GiB/s (20.6GB/s)(1154GiB/60002msec) read: IOPS=4989k, BW=19.0GiB/s (20.4GB/s)(1142GiB/60001msec) read: IOPS=4982k, BW=19.0GiB/s (20.4GB/s)(1140GiB/60001msec) read: IOPS=5031k, BW=19.2GiB/s (20.6GB/s)(1152GiB/60001msec) read: IOPS=4948k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec) read: IOPS=4720k, BW=18.0GiB/s (19.3GB/s)(1080GiB/60002msec) read: IOPS=4830k, BW=18.4GiB/s (19.8GB/s)(1106GiB/60002msec) read: IOPS=4915k, BW=18.7GiB/s (20.1GB/s)(1125GiB/60002msec) -ck changes from V2: 1. Add a comment in the nvmet_bdev_ns_enable() when returning early for buffered_io case. Feel free to edit the comment at the time of applying patch. changes from v1: 1. Remove first two patches and don't check for stat mode for block device. 2. use i_size_read() to get the size of the file. Chaitanya Kulkarni (2): nvmet: use i_size_read() to set size for file-ns nvmet: allow bdev in buffered_io mode drivers/nvme/target/io-cmd-bdev.c | 8 ++++++++ drivers/nvme/target/io-cmd-file.c | 17 ++++------------- drivers/nvme/target/nvmet.h | 2 +- 3 files changed, 13 insertions(+), 14 deletions(-) root@dev nvme (nvme-5.17) # ./bdev-buffered-io bdev-buffered-io/ bdev-buffered-io-test.sh root@dev nvme (nvme-5.17) # ./bdev-buffered-io-test.sh + NQN=testnqn + nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) real 0m0.350s user 0m0.003s sys 0m0.002s + for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/* + echo 0 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + rmdir /sys/kernel/config/nvmet/ports/1 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn + rmdir 'config/nullb/nullb*' rmdir: failed to remove 'config/nullb/nullb*': No such file or directory + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files + umount /mnt/nvme0n1 + clear_dmesg ./compile_nvme.sh: line 3: clear_dmesg: command not found umount: /mnt/nvme0n1: no mount point specified. + ./delete.sh 1 + NQN=testnqn + nvme disconnect -n testnqn Failed to disconnect by NQN: testnqn real 0m0.001s user 0m0.000s sys 0m0.001s + for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/* + echo 0 ./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory + rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*' rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + rmdir /sys/kernel/config/nvmet/ports/1 rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory + rmdir /sys/kernel/config/nvmet/subsystems/testnqn rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory + rmdir 'config/nullb/nullb*' rmdir: failed to remove 'config/nullb/nullb*': No such file or directory + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files + modprobe -r nvme-fabrics + modprobe -r nvme_loop + modprobe -r nvmet + modprobe -r nvme + sleep 1 + modprobe -r nvme-core + lsmod + grep nvme + sleep 1 + git diff + sleep 1 ++ nproc + make -j 48 M=drivers/nvme/ modules + HOST=drivers/nvme/host + TARGET=drivers/nvme/target ++ uname -r + HOST_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/ ++ uname -r + TARGET_DEST=/lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target/ + cp drivers/nvme/host/nvme-core.ko drivers/nvme/host/nvme-fabrics.ko drivers/nvme/host/nvme-fc.ko drivers/nvme/host/nvme.ko drivers/nvme/host/nvme-rdma.ko drivers/nvme/host/nvme-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host// + cp drivers/nvme/target/nvme-fcloop.ko drivers/nvme/target/nvme-loop.ko drivers/nvme/target/nvmet-fc.ko drivers/nvme/target/nvmet.ko drivers/nvme/target/nvmet-rdma.ko drivers/nvme/target/nvmet-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target// + ls -lrth /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/ /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target// /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/host/: total 6.7M -rw-r--r--. 1 root root 2.8M Feb 1 23:42 nvme-core.ko -rw-r--r--. 1 root root 448K Feb 1 23:42 nvme-fabrics.ko -rw-r--r--. 1 root root 1015K Feb 1 23:42 nvme-fc.ko -rw-r--r--. 1 root root 745K Feb 1 23:42 nvme.ko -rw-r--r--. 1 root root 958K Feb 1 23:42 nvme-rdma.ko -rw-r--r--. 1 root root 902K Feb 1 23:42 nvme-tcp.ko /lib/modules/5.17.0-rc1nvme+/kernel/drivers/nvme/target//: total 6.7M -rw-r--r--. 1 root root 498K Feb 1 23:42 nvme-fcloop.ko -rw-r--r--. 1 root root 439K Feb 1 23:42 nvme-loop.ko -rw-r--r--. 1 root root 768K Feb 1 23:42 nvmet-fc.ko -rw-r--r--. 1 root root 3.3M Feb 1 23:42 nvmet.ko -rw-r--r--. 1 root root 929K Feb 1 23:42 nvmet-rdma.ko -rw-r--r--. 1 root root 775K Feb 1 23:42 nvmet-tcp.ko + modprobe nvme + NQN=testnqn + nvme disconnect -n testnqn NQN:testnqn disconnected 0 controller(s) real 0m0.001s user 0m0.001s sys 0m0.001s + for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/* + echo 0 ./delete.sh: line 9: /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*/enable: No such file or directory + rmdir '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*' rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn/namespaces/*': No such file or directory + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + rmdir /sys/kernel/config/nvmet/ports/1 rmdir: failed to remove '/sys/kernel/config/nvmet/ports/1': No such file or directory + rmdir /sys/kernel/config/nvmet/subsystems/testnqn rmdir: failed to remove '/sys/kernel/config/nvmet/subsystems/testnqn': No such file or directory + rmdir 'config/nullb/nullb*' rmdir: failed to remove 'config/nullb/nullb*': No such file or directory + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files +++ NN=1 +++ NQN=testnqn +++ let NR_DEVICES=NN+1 +++ modprobe -r null_blk +++ modprobe -r nvme +++ modprobe null_blk nr_devices=0 +++ modprobe nvme +++ modprobe nvme-fabrics +++ modprobe nvmet +++ modprobe nvme-loop +++ dmesg -c +++ sleep 2 +++ tree /sys/kernel/config /sys/kernel/config ├── nullb │   └── features └── nvmet ├── hosts ├── ports └── subsystems 5 directories, 1 file +++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn +++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 +++ echo -n /dev/nvme0n1 +++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nvme0n1 +++ echo 1 +++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io 1 +++ echo 1 +++ mkdir /sys/kernel/config/nvmet/ports/1/ +++ echo -n loop +++ echo -n 1 +++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ +++ sleep 1 +++ echo transport=loop,nqn=testnqn +++ sleep 1 +++ mount +++ column -t +++ grep nvme +++ dmesg -c [ 2348.622209] nvme nvme0: 48/0/0 default/read/poll queues [ 2350.605991] nvmet: adding nsid 1 to subsystem testnqn [ 2351.616737] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:bb4f2c46-9ac7-4283-902a-420883b62f3c. [ 2351.616857] nvme nvme1: creating 48 I/O queues. [ 2351.621168] nvme nvme1: new ctrl: "testnqn" write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4 fio-3.27 Starting 1 process Jobs: 1 (f=1) write-and-verify: (groupid=0, jobs=1): err= 0: pid=20757: Tue Feb 1 23:42:57 2022 read: IOPS=195k, BW=763MiB/s (800MB/s)(506MiB/663msec) slat (usec): min=3, max=422, avg= 3.96, stdev= 1.90 clat (nsec): min=1072, max=483948, avg=15610.29, stdev=5064.35 lat (usec): min=4, max=494, avg=19.64, stdev= 6.20 clat percentiles (nsec): | 1.00th=[13248], 5.00th=[13376], 10.00th=[13504], 20.00th=[13632], | 30.00th=[13760], 40.00th=[13888], 50.00th=[14016], 60.00th=[14144], | 70.00th=[14272], 80.00th=[14528], 90.00th=[23424], 95.00th=[26752], | 99.00th=[28544], 99.50th=[32384], 99.90th=[42752], 99.95th=[47360], | 99.99th=[82432] write: IOPS=123k, BW=481MiB/s (505MB/s)(800MiB/1662msec); slat (usec): min=4, max=221, avg= 7.36, stdev= 2.87 clat (nsec): min=841, max=222422, avg=24669.34, stdev=8735.58 lat (usec): min=6, max=263, avg=32.11, stdev=11.14 clat percentiles (usec): | 1.00th=[ 18], 5.00th=[ 19], 10.00th=[ 19], 20.00th=[ 19], | 30.00th=[ 20], 40.00th=[ 20], 50.00th=[ 20], 60.00th=[ 21], | 70.00th=[ 28], 80.00th=[ 36], 90.00th=[ 38], 95.00th=[ 40], | 99.00th=[ 51], 99.50th=[ 58], 99.90th=[ 70], 99.95th=[ 81], | 99.99th=[ 155] bw ( KiB/s): min=150160, max=525424, per=83.10%, avg=409600.00, stdev=174551.65, samples=4 iops : min=37540, max=131356, avg=102400.00, stdev=43637.91, samples=4 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=64.99%, 50=34.34%, 100=0.65% lat (usec) : 250=0.02%, 500=0.01% cpu : usr=27.97%, sys=32.66%, ctx=333157, majf=0, minf=3053 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=4 Run status group 0 (all jobs): READ: bw=763MiB/s (800MB/s), 763MiB/s-763MiB/s (800MB/s-800MB/s), io=506MiB (531MB), run=663-663msec WRITE: bw=481MiB/s (505MB/s), 481MiB/s-481MiB/s (505MB/s-505MB/s), io=800MiB (839MB), run=1662-1662msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=16.2GiB/s][w=4236k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=20761: Tue Feb 1 23:43:57 2022 write: IOPS=4248k, BW=16.2GiB/s (17.4GB/s)(972GiB/60002msec); slat (nsec): min=1112, max=24135k, avg=6779.81, stdev=5320.08 clat (nsec): min=721, max=24222k, avg=53033.78, stdev=21465.11 lat (usec): min=6, max=24230, avg=59.89, stdev=23.52 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 43], 40.00th=[ 44], 50.00th=[ 46], 60.00th=[ 57], | 70.00th=[ 58], 80.00th=[ 60], 90.00th=[ 63], 95.00th=[ 88], | 99.00th=[ 126], 99.50th=[ 133], 99.90th=[ 186], 99.95th=[ 223], | 99.99th=[ 277] bw ( MiB/s): min=11452, max=20170, per=100.00%, avg=16608.41, stdev=84.92, samples=3808 iops : min=2931936, max=5163618, avg=4251753.76, stdev=21739.07, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=54.82%, 100=42.95% lat (usec) : 250=2.21%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=11.32%, sys=36.88%, ctx=254865246, majf=0, minf=439 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,254859948,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=16.2GiB/s (17.4GB/s), 16.2GiB/s-16.2GiB/s (17.4GB/s-17.4GB/s), io=972GiB (1044GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=15.3GiB/s][w=4015k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=20878: Tue Feb 1 23:44:57 2022 write: IOPS=4157k, BW=15.9GiB/s (17.0GB/s)(951GiB/60002msec); slat (nsec): min=1773, max=23957k, avg=6930.59, stdev=5690.00 clat (nsec): min=672, max=24058k, avg=54190.75, stdev=22713.63 lat (usec): min=6, max=24086, avg=61.20, stdev=24.89 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 57], | 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 94], | 99.00th=[ 129], 99.50th=[ 137], 99.90th=[ 192], 99.95th=[ 225], | 99.99th=[ 269] bw ( MiB/s): min=11359, max=20085, per=100.00%, avg=16257.07, stdev=84.39, samples=3808 iops : min=2907927, max=5141948, avg=4161809.03, stdev=21605.08, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=54.02%, 100=42.99% lat (usec) : 250=2.97%, 500=0.02%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=11.06%, sys=36.43%, ctx=249428122, majf=0, minf=464 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,249423456,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=15.9GiB/s (17.0GB/s), 15.9GiB/s-15.9GiB/s (17.0GB/s-17.0GB/s), io=951GiB (1022GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4283k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=20954: Tue Feb 1 23:45:57 2022 write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec); slat (nsec): min=1783, max=23922k, avg=6853.24, stdev=4821.07 clat (nsec): min=711, max=24147k, avg=53603.75, stdev=19747.55 lat (usec): min=5, max=24155, avg=60.53, stdev=21.64 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57], | 70.00th=[ 59], 80.00th=[ 60], 90.00th=[ 65], 95.00th=[ 85], | 99.00th=[ 118], 99.50th=[ 133], 99.90th=[ 167], 99.95th=[ 190], | 99.99th=[ 251] bw ( MiB/s): min=11876, max=20042, per=100.00%, avg=16429.89, stdev=81.89, samples=3808 iops : min=3040376, max=5130956, avg=4206051.45, stdev=20963.19, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.72%, 100=45.12% lat (usec) : 250=2.14%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=11.02%, sys=37.05%, ctx=252169523, majf=0, minf=473 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,252164251,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=14.1GiB/s][w=3695k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=21089: Tue Feb 1 23:46:58 2022 write: IOPS=4130k, BW=15.8GiB/s (16.9GB/s)(945GiB/60002msec); slat (nsec): min=1222, max=12690k, avg=6987.00, stdev=5499.44 clat (nsec): min=711, max=16691k, avg=54537.79, stdev=23234.80 lat (usec): min=5, max=16701, avg=61.61, stdev=25.56 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57], | 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 69], 95.00th=[ 93], | 99.00th=[ 131], 99.50th=[ 141], 99.90th=[ 184], 99.95th=[ 204], | 99.99th=[ 260] bw ( MiB/s): min=11551, max=20105, per=100.00%, avg=16156.07, stdev=74.80, samples=3808 iops : min=2957123, max=5146990, avg=4135953.66, stdev=19148.37, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.25%, 100=44.71% lat (usec) : 250=3.03%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=10.94%, sys=36.18%, ctx=247812660, majf=0, minf=472 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,247805833,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=15.8GiB/s (16.9GB/s), 15.8GiB/s-15.8GiB/s (16.9GB/s-16.9GB/s), io=945GiB (1015GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=16.3GiB/s][w=4268k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=22043: Tue Feb 1 23:47:58 2022 write: IOPS=4170k, BW=15.9GiB/s (17.1GB/s)(955GiB/60002msec); slat (nsec): min=1763, max=17410k, avg=6908.89, stdev=5766.44 clat (nsec): min=712, max=17505k, avg=54011.25, stdev=23339.67 lat (usec): min=5, max=17513, avg=61.00, stdev=25.50 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57], | 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 68], 95.00th=[ 87], | 99.00th=[ 121], 99.50th=[ 135], 99.90th=[ 178], 99.95th=[ 204], | 99.99th=[ 310] bw ( MiB/s): min=11707, max=19992, per=100.00%, avg=16303.82, stdev=73.70, samples=3808 iops : min=2997012, max=5118104, avg=4173776.76, stdev=18867.69, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.47%, 100=45.42% lat (usec) : 250=2.10%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=11.04%, sys=36.68%, ctx=250242780, majf=0, minf=461 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,250236670,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=955GiB (1025GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=16.0GiB/s][w=4186k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=22119: Tue Feb 1 23:48:58 2022 write: IOPS=4203k, BW=16.0GiB/s (17.2GB/s)(962GiB/60002msec); slat (nsec): min=1162, max=13647k, avg=6841.84, stdev=4990.36 clat (nsec): min=832, max=17776k, avg=53604.58, stdev=21052.59 lat (usec): min=6, max=17783, avg=60.53, stdev=22.89 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 50], 60.00th=[ 58], | 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 65], 95.00th=[ 79], | 99.00th=[ 103], 99.50th=[ 121], 99.90th=[ 153], 99.95th=[ 176], | 99.99th=[ 306] bw ( MiB/s): min=11133, max=20082, per=100.00%, avg=16434.73, stdev=79.43, samples=3808 iops : min=2850202, max=5141134, avg=4207290.84, stdev=20334.84, samples=3808 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=50.62% lat (usec) : 100=48.12%, 250=1.25%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=11.77%, sys=36.68%, ctx=252168793, majf=0, minf=462 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,252163391,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=16.0GiB/s (17.2GB/s), 16.0GiB/s-16.0GiB/s (17.2GB/s-17.2GB/s), io=962GiB (1033GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=16.5GiB/s][w=4326k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=22221: Tue Feb 1 23:49:58 2022 write: IOPS=4166k, BW=15.9GiB/s (17.1GB/s)(954GiB/60002msec); slat (nsec): min=1804, max=16555k, avg=6911.03, stdev=4216.56 clat (nsec): min=731, max=16639k, avg=54075.05, stdev=17110.09 lat (usec): min=5, max=16647, avg=61.07, stdev=18.98 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 49], 60.00th=[ 58], | 70.00th=[ 60], 80.00th=[ 62], 90.00th=[ 67], 95.00th=[ 81], | 99.00th=[ 123], 99.50th=[ 135], 99.90th=[ 165], 99.95th=[ 184], | 99.99th=[ 210] bw ( MiB/s): min=11319, max=20001, per=100.00%, avg=16285.86, stdev=86.40, samples=3808 iops : min=2897836, max=5120368, avg=4169180.79, stdev=22118.90, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=51.04% lat (usec) : 100=47.21%, 250=1.75%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=11.43%, sys=36.52%, ctx=249967922, majf=0, minf=464 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,249962350,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=954GiB (1024GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4224k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=22298: Tue Feb 1 23:50:59 2022 write: IOPS=4106k, BW=15.7GiB/s (16.8GB/s)(940GiB/60002msec); slat (nsec): min=1372, max=18974k, avg=7033.73, stdev=4335.83 clat (nsec): min=721, max=19073k, avg=54854.61, stdev=19806.39 lat (usec): min=6, max=19082, avg=61.97, stdev=22.12 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 47], 60.00th=[ 57], | 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 74], 95.00th=[ 95], | 99.00th=[ 133], 99.50th=[ 147], 99.90th=[ 184], 99.95th=[ 196], | 99.99th=[ 219] bw ( MiB/s): min=11060, max=20108, per=100.00%, avg=16051.21, stdev=92.86, samples=3808 iops : min=2831464, max=5147868, avg=4109110.77, stdev=23772.60, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=53.51% lat (usec) : 100=42.99%, 250=3.49%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=10.98%, sys=35.65%, ctx=246371950, majf=0, minf=458 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,246367496,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=15.7GiB/s (16.8GB/s), 15.7GiB/s-15.7GiB/s (16.8GB/s-16.8GB/s), io=940GiB (1009GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=14.9GiB/s][w=3895k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=22372: Tue Feb 1 23:51:59 2022 write: IOPS=4181k, BW=15.9GiB/s (17.1GB/s)(957GiB/60001msec); slat (nsec): min=1623, max=16114k, avg=6882.50, stdev=4431.39 clat (nsec): min=712, max=16232k, avg=53879.81, stdev=16614.37 lat (usec): min=6, max=16241, avg=60.84, stdev=18.29 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 51], 60.00th=[ 58], | 70.00th=[ 60], 80.00th=[ 61], 90.00th=[ 67], 95.00th=[ 80], | 99.00th=[ 104], 99.50th=[ 129], 99.90th=[ 161], 99.95th=[ 182], | 99.99th=[ 219] bw ( MiB/s): min=12006, max=19952, per=100.00%, avg=16360.11, stdev=79.02, samples=3808 iops : min=3073692, max=5107912, avg=4188189.50, stdev=20228.57, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=49.59%, 100=49.27% lat (usec) : 250=1.14%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=11.16%, sys=37.05%, ctx=250877181, majf=0, minf=498 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,250866850,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=15.9GiB/s (17.1GB/s), 15.9GiB/s-15.9GiB/s (17.1GB/s-17.1GB/s), io=957GiB (1028GB), run=60001-60001msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=16.1GiB/s][w=4230k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=23453: Tue Feb 1 23:52:59 2022 write: IOPS=4153k, BW=15.8GiB/s (17.0GB/s)(951GiB/60002msec); slat (nsec): min=1824, max=12568k, avg=6939.60, stdev=3999.18 clat (nsec): min=701, max=16670k, avg=54238.72, stdev=17869.32 lat (usec): min=6, max=16677, avg=61.26, stdev=19.85 clat percentiles (usec): | 1.00th=[ 41], 5.00th=[ 42], 10.00th=[ 42], 20.00th=[ 43], | 30.00th=[ 44], 40.00th=[ 45], 50.00th=[ 48], 60.00th=[ 57], | 70.00th=[ 59], 80.00th=[ 61], 90.00th=[ 69], 95.00th=[ 84], | 99.00th=[ 129], 99.50th=[ 141], 99.90th=[ 176], 99.95th=[ 194], | 99.99th=[ 217] bw ( MiB/s): min=11198, max=20030, per=100.00%, avg=16238.98, stdev=82.50, samples=3808 iops : min=2866772, max=5127914, avg=4157179.99, stdev=21119.50, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=52.21%, 100=45.55% lat (usec) : 250=2.23%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01% cpu : usr=10.99%, sys=36.59%, ctx=249190314, majf=0, minf=490 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,249183943,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=15.8GiB/s (17.0GB/s), 15.8GiB/s-15.8GiB/s (17.0GB/s-17.0GB/s), io=951GiB (1021GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=18.9GiB/s][r=4960k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=23530: Tue Feb 1 23:53:59 2022 read: IOPS=4942k, BW=18.9GiB/s (20.2GB/s)(1131GiB/60002msec) slat (nsec): min=1142, max=49607k, avg=8828.54, stdev=15569.98 clat (nsec): min=441, max=243040k, avg=612190.41, stdev=339908.18 lat (usec): min=6, max=243043, avg=621.12, stdev=343.94 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 523], | 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 537], | 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1139], 95.00th=[ 1188], | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1926], 99.95th=[ 4817], | 99.99th=[11469] bw ( MiB/s): min= 1691, max=27419, per=100.00%, avg=19320.99, stdev=89.34, samples=5712 iops : min=433098, max=7019516, avg=4946173.24, stdev=22869.85, samples=5712 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.01%, 500=15.40%, 750=69.25%, 1000=4.03% lat (msec) : 2=11.24%, 4=0.03%, 10=0.04%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01%, 250=0.01% cpu : usr=9.29%, sys=35.84%, ctx=296529230, majf=0, minf=3836 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=296546544,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=18.9GiB/s (20.2GB/s), 18.9GiB/s-18.9GiB/s (20.2GB/s-20.2GB/s), io=1131GiB (1215GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=18.4GiB/s][r=4834k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=23882: Tue Feb 1 23:55:00 2022 read: IOPS=4940k, BW=18.8GiB/s (20.2GB/s)(1131GiB/60002msec) slat (usec): min=2, max=16996, avg= 8.84, stdev= 6.11 clat (nsec): min=911, max=21566k, avg=612493.08, stdev=234517.40 lat (usec): min=4, max=21574, avg=621.43, stdev=238.08 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 523], | 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 537], | 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1123], 95.00th=[ 1172], | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1549], 99.95th=[ 2024], | 99.99th=[ 3261] bw ( MiB/s): min=12590, max=26589, per=100.00%, avg=19314.78, stdev=65.21, samples=5712 iops : min=3223108, max=6806844, avg=4944582.49, stdev=16693.22, samples=5712 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.01%, 500=14.27%, 750=69.32%, 1000=5.38% lat (msec) : 2=10.98%, 4=0.05%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=9.29%, sys=36.21%, ctx=296412664, majf=0, minf=3763 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=296403321,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=18.8GiB/s (20.2GB/s), 18.8GiB/s-18.8GiB/s (20.2GB/s-20.2GB/s), io=1131GiB (1214GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=18.6GiB/s][r=4885k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=24112: Tue Feb 1 23:56:00 2022 read: IOPS=5041k, BW=19.2GiB/s (20.6GB/s)(1154GiB/60002msec) slat (usec): min=2, max=22808, avg= 8.65, stdev= 6.86 clat (nsec): min=912, max=26105k, avg=600221.79, stdev=221404.88 lat (usec): min=5, max=26112, avg=608.96, stdev=224.71 clat percentiles (usec): | 1.00th=[ 404], 5.00th=[ 412], 10.00th=[ 424], 20.00th=[ 523], | 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537], | 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1106], 95.00th=[ 1188], | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1287], 99.95th=[ 1319], | 99.99th=[ 2057] bw ( MiB/s): min=16300, max=26946, per=100.00%, avg=19713.39, stdev=67.57, samples=5712 iops : min=4173052, max=6898318, avg=5046626.47, stdev=17299.17, samples=5712 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.01%, 500=12.91%, 750=73.53%, 1000=3.26% lat (msec) : 2=10.28%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=9.37%, sys=36.74%, ctx=302479289, majf=0, minf=3758 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=302468143,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=19.2GiB/s (20.6GB/s), 19.2GiB/s-19.2GiB/s (20.6GB/s-20.6GB/s), io=1154GiB (1239GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=17.9GiB/s][r=4683k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=24212: Tue Feb 1 23:57:00 2022 read: IOPS=4989k, BW=19.0GiB/s (20.4GB/s)(1142GiB/60001msec) slat (nsec): min=1953, max=52925k, avg=8751.89, stdev=11303.93 clat (nsec): min=912, max=56316k, avg=606515.90, stdev=246434.14 lat (usec): min=5, max=56326, avg=615.36, stdev=249.97 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 519], | 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537], | 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1123], 95.00th=[ 1188], | 99.00th=[ 1237], 99.50th=[ 1270], 99.90th=[ 1745], 99.95th=[ 1827], | 99.99th=[ 4752] bw ( MiB/s): min=13300, max=27673, per=100.00%, avg=19511.23, stdev=82.41, samples=5712 iops : min=3405011, max=7084445, avg=4994872.61, stdev=21096.46, samples=5712 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.01%, 500=14.24%, 750=70.95%, 1000=3.71% lat (msec) : 2=11.08%, 4=0.02%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=9.31%, sys=36.30%, ctx=299345988, majf=0, minf=3730 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=299326708,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=19.0GiB/s (20.4GB/s), 19.0GiB/s-19.0GiB/s (20.4GB/s-20.4GB/s), io=1142GiB (1226GB), run=60001-60001msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=17.8GiB/s][r=4674k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=25485: Tue Feb 1 23:58:00 2022 read: IOPS=4982k, BW=19.0GiB/s (20.4GB/s)(1140GiB/60001msec) slat (nsec): min=1473, max=41255k, avg=8761.08, stdev=7916.28 clat (nsec): min=871, max=45519k, avg=607322.90, stdev=229101.48 lat (usec): min=5, max=45526, avg=616.18, stdev=232.51 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 420], 20.00th=[ 519], | 30.00th=[ 529], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545], | 70.00th=[ 545], 80.00th=[ 553], 90.00th=[ 1106], 95.00th=[ 1172], | 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1287], 99.95th=[ 1303], | 99.99th=[ 1745] bw ( MiB/s): min=15677, max=27351, per=100.00%, avg=19485.07, stdev=73.52, samples=5712 iops : min=4013328, max=7002048, avg=4988177.24, stdev=18821.75, samples=5712 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=14.54%, 750=69.53%, 1000=5.29% lat (msec) : 2=10.62%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=9.33%, sys=36.13%, ctx=298932803, majf=0, minf=3800 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=298924191,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=19.0GiB/s (20.4GB/s), 19.0GiB/s-19.0GiB/s (20.4GB/s-20.4GB/s), io=1140GiB (1224GB), run=60001-60001msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=18.2GiB/s][r=4763k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=25572: Tue Feb 1 23:59:01 2022 read: IOPS=5031k, BW=19.2GiB/s (20.6GB/s)(1152GiB/60001msec) slat (nsec): min=1985, max=21488k, avg=8673.54, stdev=5859.90 clat (nsec): min=902, max=22056k, avg=601397.58, stdev=214885.23 lat (usec): min=5, max=22064, avg=610.17, stdev=218.14 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 424], 20.00th=[ 523], | 30.00th=[ 529], 40.00th=[ 537], 50.00th=[ 537], 60.00th=[ 545], | 70.00th=[ 553], 80.00th=[ 553], 90.00th=[ 971], 95.00th=[ 1172], | 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1270], 99.95th=[ 1287], | 99.99th=[ 1631] bw ( MiB/s): min=14781, max=27237, per=100.00%, avg=19676.38, stdev=75.29, samples=5712 iops : min=3784135, max=6972871, avg=5037153.12, stdev=19275.07, samples=5712 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.01%, 500=13.17%, 750=72.94%, 1000=4.22% lat (msec) : 2=9.67%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=9.32%, sys=36.93%, ctx=301882480, majf=0, minf=3751 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=301872061,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=19.2GiB/s (20.6GB/s), 19.2GiB/s-19.2GiB/s (20.6GB/s-20.6GB/s), io=1152GiB (1236GB), run=60001-60001msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=17.6GiB/s][r=4607k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=25664: Wed Feb 2 00:00:01 2022 read: IOPS=4948k, BW=18.9GiB/s (20.3GB/s)(1133GiB/60002msec) slat (usec): min=2, max=21123, avg= 8.83, stdev= 5.96 clat (nsec): min=942, max=22372k, avg=611470.57, stdev=236265.95 lat (usec): min=6, max=22391, avg=620.40, stdev=239.84 clat percentiles (usec): | 1.00th=[ 396], 5.00th=[ 408], 10.00th=[ 416], 20.00th=[ 515], | 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545], | 70.00th=[ 545], 80.00th=[ 562], 90.00th=[ 1139], 95.00th=[ 1188], | 99.00th=[ 1221], 99.50th=[ 1237], 99.90th=[ 1287], 99.95th=[ 1303], | 99.99th=[ 1795] bw ( MiB/s): min=15222, max=27543, per=100.00%, avg=19354.61, stdev=79.33, samples=5712 iops : min=3896862, max=7051174, avg=4954779.41, stdev=20309.68, samples=5712 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=16.87%, 750=66.30%, 1000=4.54% lat (msec) : 2=12.28%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=9.19%, sys=35.87%, ctx=296904832, majf=0, minf=3743 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=296894931,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=18.9GiB/s (20.3GB/s), 18.9GiB/s-18.9GiB/s (20.3GB/s-20.3GB/s), io=1133GiB (1216GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=17.4GiB/s][r=4554k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=25785: Wed Feb 2 00:01:01 2022 read: IOPS=4720k, BW=18.0GiB/s (19.3GB/s)(1080GiB/60002msec) slat (usec): min=2, max=28533, avg= 9.30, stdev= 7.03 clat (nsec): min=921, max=29089k, avg=640985.06, stdev=258045.84 lat (usec): min=5, max=29096, avg=650.39, stdev=261.95 clat percentiles (usec): | 1.00th=[ 392], 5.00th=[ 404], 10.00th=[ 412], 20.00th=[ 510], | 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545], | 70.00th=[ 553], 80.00th=[ 906], 90.00th=[ 1156], 95.00th=[ 1188], | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1303], 99.95th=[ 1401], | 99.99th=[ 1876] bw ( MiB/s): min=13222, max=27706, per=100.00%, avg=18458.01, stdev=98.55, samples=5712 iops : min=3384868, max=7092862, avg=4725251.34, stdev=25229.04, samples=5712 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.01%, 500=18.90%, 750=56.44%, 1000=10.71% lat (msec) : 2=13.94%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% cpu : usr=8.95%, sys=33.65%, ctx=283232675, majf=0, minf=3746 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=283217843,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=18.0GiB/s (19.3GB/s), 18.0GiB/s-18.0GiB/s (19.3GB/s-19.3GB/s), io=1080GiB (1160GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=18.2GiB/s][r=4774k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=25897: Wed Feb 2 00:02:01 2022 read: IOPS=4830k, BW=18.4GiB/s (19.8GB/s)(1106GiB/60002msec) slat (usec): min=2, max=37297, avg= 9.06, stdev=11.53 clat (nsec): min=892, max=37911k, avg=626423.87, stdev=281337.79 lat (usec): min=5, max=37919, avg=635.59, stdev=285.46 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 412], 10.00th=[ 429], 20.00th=[ 515], | 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 537], | 70.00th=[ 545], 80.00th=[ 562], 90.00th=[ 1156], 95.00th=[ 1205], | 99.00th=[ 1549], 99.50th=[ 1680], 99.90th=[ 2835], 99.95th=[ 3392], | 99.99th=[ 5407] bw ( MiB/s): min= 6580, max=27466, per=100.00%, avg=18885.17, stdev=100.72, samples=5712 iops : min=1684599, max=7031464, avg=4834602.87, stdev=25783.81, samples=5712 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 10=0.01%, 20=0.01%, 50=0.01%, 100=0.01% lat (usec) : 250=0.01%, 500=12.90%, 750=70.01%, 1000=4.60% lat (msec) : 2=12.35%, 4=0.13%, 10=0.02%, 20=0.01%, 50=0.01% cpu : usr=9.17%, sys=37.36%, ctx=289835159, majf=0, minf=3741 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=289804479,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=18.4GiB/s (19.8GB/s), 18.4GiB/s-18.4GiB/s (19.8GB/s-19.8GB/s), io=1106GiB (1187GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=18.3GiB/s][r=4805k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=26489: Wed Feb 2 00:03:02 2022 read: IOPS=4915k, BW=18.7GiB/s (20.1GB/s)(1125GiB/60002msec) slat (nsec): min=1714, max=56074k, avg=8893.16, stdev=12899.01 clat (nsec): min=922, max=57329k, avg=615610.19, stdev=258832.65 lat (usec): min=5, max=57337, avg=624.60, stdev=262.51 clat percentiles (usec): | 1.00th=[ 400], 5.00th=[ 408], 10.00th=[ 420], 20.00th=[ 515], | 30.00th=[ 523], 40.00th=[ 529], 50.00th=[ 537], 60.00th=[ 545], | 70.00th=[ 545], 80.00th=[ 562], 90.00th=[ 1139], 95.00th=[ 1188], | 99.00th=[ 1237], 99.50th=[ 1254], 99.90th=[ 1418], 99.95th=[ 1811], | 99.99th=[ 4948] bw ( MiB/s): min=14198, max=26839, per=100.00%, avg=19217.89, stdev=63.10, samples=5712 iops : min=3634881, max=6870838, avg=4919778.50, stdev=16154.49, samples=5712 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=15.84%, 750=67.55%, 1000=3.63% lat (msec) : 2=12.94%, 4=0.03%, 10=0.01%, 20=0.01%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=9.32%, sys=35.56%, ctx=294925914, majf=0, minf=3702 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=294900613,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=18.7GiB/s (20.1GB/s), 18.7GiB/s-18.7GiB/s (20.1GB/s-20.1GB/s), io=1125GiB (1208GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% + NQN=testnqn + nvme disconnect -n testnqn NQN:testnqn disconnected 1 controller(s) real 0m0.349s user 0m0.000s sys 0m0.003s + for i in /sys/kernel/config/nvmet/subsystems/${NQN}/namespaces/* + echo 0 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 + rm -fr /sys/kernel/config/nvmet/ports/1/subsystems/testnqn + rmdir /sys/kernel/config/nvmet/ports/1 + rmdir /sys/kernel/config/nvmet/subsystems/testnqn + rmdir 'config/nullb/nullb*' rmdir: failed to remove 'config/nullb/nullb*': No such file or directory + umount /mnt/nvme0n1 umount: /mnt/nvme0n1: no mount point specified. + umount /mnt/backend umount: /mnt/backend: not mounted. + modprobe -r nvme_loop + modprobe -r nvme_fabrics + modprobe -r nvmet + modprobe -r nvme + modprobe -r null_blk + tree /sys/kernel/config /sys/kernel/config 0 directories, 0 files +++ NN=1 +++ NQN=testnqn +++ let NR_DEVICES=NN+1 +++ modprobe -r null_blk +++ modprobe -r nvme +++ modprobe null_blk nr_devices=0 +++ modprobe nvme +++ modprobe nvme-fabrics +++ modprobe nvmet +++ modprobe nvme-loop +++ dmesg -c +++ sleep 2 +++ tree /sys/kernel/config /sys/kernel/config ├── nullb │   └── features └── nvmet ├── hosts ├── ports └── subsystems 5 directories, 1 file +++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn +++ mkdir /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1 +++ echo -n /dev/nvme0n1 +++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/device_path /dev/nvme0n1 +++ echo 0 +++ cat /sys/kernel/config/nvmet/subsystems/testnqn/namespaces/1/buffered_io 0 +++ echo 1 +++ mkdir /sys/kernel/config/nvmet/ports/1/ +++ echo -n loop +++ echo -n 1 +++ ln -s /sys/kernel/config/nvmet/subsystems/testnqn /sys/kernel/config/nvmet/ports/1/subsystems/ +++ sleep 1 +++ echo transport=loop,nqn=testnqn +++ sleep 1 +++ mount +++ column -t +++ grep nvme +++ dmesg -c [ 3560.886773] nvme nvme0: 48/0/0 default/read/poll queues [ 3562.872451] nvmet: adding nsid 1 to subsystem testnqn [ 3563.883217] nvmet: creating nvm controller 1 for subsystem testnqn for NQN nqn.2014-08.org.nvmexpress:uuid:bc654437-83be-4628-91bd-a439b8e830d5. [ 3563.883353] nvme nvme1: creating 48 I/O queues. [ 3563.887931] nvme nvme1: new ctrl: "testnqn" write-and-verify: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=4 fio-3.27 Starting 1 process Jobs: 1 (f=1): [V(1)][80.0%][r=99.2MiB/s][r=25.4k IOPS][eta 00m:03s] write-and-verify: (groupid=0, jobs=1): err= 0: pid=27715: Wed Feb 2 00:03:19 2022 read: IOPS=24.3k, BW=94.9MiB/s (99.5MB/s)(506MiB/5335msec) slat (nsec): min=1032, max=1447.4k, avg=8870.69, stdev=27575.11 clat (usec): min=11, max=4669, avg=153.54, stdev=131.98 lat (usec): min=47, max=4672, avg=162.54, stdev=137.66 clat percentiles (usec): | 1.00th=[ 62], 5.00th=[ 86], 10.00th=[ 96], 20.00th=[ 106], | 30.00th=[ 113], 40.00th=[ 116], 50.00th=[ 120], 60.00th=[ 125], | 70.00th=[ 135], 80.00th=[ 198], 90.00th=[ 221], 95.00th=[ 251], | 99.00th=[ 709], 99.50th=[ 947], 99.90th=[ 1663], 99.95th=[ 2114], | 99.99th=[ 2933] write: IOPS=27.9k, BW=109MiB/s (114MB/s)(800MiB/7353msec); slat (usec): min=2, max=3897, avg=13.26, stdev=21.42 clat (usec): min=15, max=5720, avg=129.39, stdev=79.93 lat (usec): min=37, max=6857, avg=142.77, stdev=84.05 clat percentiles (usec): | 1.00th=[ 63], 5.00th=[ 77], 10.00th=[ 84], 20.00th=[ 96], | 30.00th=[ 105], 40.00th=[ 115], 50.00th=[ 123], 60.00th=[ 129], | 70.00th=[ 137], 80.00th=[ 151], 90.00th=[ 169], 95.00th=[ 215], | 99.00th=[ 273], 99.50th=[ 310], 99.90th=[ 1057], 99.95th=[ 1696], | 99.99th=[ 2868] bw ( KiB/s): min=82496, max=143312, per=98.04%, avg=109226.13, stdev=16354.90, samples=15 iops : min=20624, max=35828, avg=27306.67, stdev=4088.95, samples=15 lat (usec) : 20=0.01%, 50=0.28%, 100=19.29%, 250=77.38%, 500=2.09% lat (usec) : 750=0.52%, 1000=0.20% lat (msec) : 2=0.19%, 4=0.05%, 10=0.01% cpu : usr=18.65%, sys=16.14%, ctx=141033, majf=0, minf=3053 IO depths : 1=0.1%, 2=0.1%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=129572,204800,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=4 Run status group 0 (all jobs): READ: bw=94.9MiB/s (99.5MB/s), 94.9MiB/s-94.9MiB/s (99.5MB/s-99.5MB/s), io=506MiB (531MB), run=5335-5335msec WRITE: bw=109MiB/s (114MB/s), 109MiB/s-109MiB/s (114MB/s-114MB/s), io=800MiB (839MB), run=7353-7353msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=253MiB/s][w=64.8k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=27744: Wed Feb 2 00:04:20 2022 write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60002msec); slat (nsec): min=1072, max=42904k, avg=162051.61, stdev=467637.32 clat (nsec): min=752, max=72679k, avg=3588602.81, stdev=2243011.38 lat (usec): min=52, max=72681, avg=3750.87, stdev=2268.70 clat percentiles (usec): | 1.00th=[ 562], 5.00th=[ 1029], 10.00th=[ 1369], 20.00th=[ 1860], | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3589], | 70.00th=[ 4178], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7701], | 99.00th=[11076], 99.50th=[12780], 99.90th=[18220], 99.95th=[21890], | 99.99th=[30278] bw ( KiB/s): min=193029, max=335346, per=100.00%, avg=273269.63, stdev=877.09, samples=3808 iops : min=48248, max=83826, avg=68310.92, stdev=219.29, samples=3808 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.16% lat (usec) : 100=0.04%, 250=0.09%, 500=0.46%, 750=1.38%, 1000=2.46% lat (msec) : 2=18.84%, 4=43.56%, 10=31.38%, 20=1.55%, 50=0.07% lat (msec) : 100=0.01% cpu : usr=0.63%, sys=1.57%, ctx=1935887, majf=0, minf=485 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4093399,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.8GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=251MiB/s][w=64.3k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=27840: Wed Feb 2 00:05:20 2022 write: IOPS=68.6k, BW=268MiB/s (281MB/s)(15.7GiB/60003msec); slat (nsec): min=1052, max=52671k, avg=146009.86, stdev=449344.58 clat (nsec): min=641, max=64251k, avg=3581258.82, stdev=2231979.80 lat (usec): min=55, max=64976, avg=3727.49, stdev=2255.52 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 1037], 10.00th=[ 1369], 20.00th=[ 1844], | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3097], 60.00th=[ 3589], | 70.00th=[ 4178], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7701], | 99.00th=[10945], 99.50th=[12649], 99.90th=[18220], 99.95th=[21627], | 99.99th=[30016] bw ( KiB/s): min=203172, max=338441, per=100.00%, avg=275054.71, stdev=839.49, samples=3808 iops : min=50788, max=84602, avg=68758.55, stdev=209.87, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.15% lat (usec) : 100=0.03%, 250=0.08%, 500=0.45%, 750=1.40%, 1000=2.44% lat (msec) : 2=18.96%, 4=43.62%, 10=31.29%, 20=1.51%, 50=0.07% lat (msec) : 100=0.01% cpu : usr=0.63%, sys=1.56%, ctx=1832905, majf=0, minf=488 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4119099,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=268MiB/s (281MB/s), 268MiB/s-268MiB/s (281MB/s-281MB/s), io=15.7GiB (16.9GB), run=60003-60003msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=270MiB/s][w=69.1k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=27949: Wed Feb 2 00:06:20 2022 write: IOPS=68.2k, BW=266MiB/s (279MB/s)(15.6GiB/60003msec); slat (nsec): min=1082, max=41371k, avg=222396.07, stdev=539362.31 clat (nsec): min=631, max=67998k, avg=3529440.50, stdev=2240119.90 lat (usec): min=54, max=69361, avg=3752.10, stdev=2276.78 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 1012], 10.00th=[ 1352], 20.00th=[ 1827], | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3523], | 70.00th=[ 4113], 80.00th=[ 4883], 90.00th=[ 6194], 95.00th=[ 7570], | 99.00th=[11076], 99.50th=[12780], 99.90th=[20055], 99.95th=[23987], | 99.99th=[31327] bw ( KiB/s): min=195270, max=336343, per=100.00%, avg=273029.60, stdev=880.99, samples=3808 iops : min=48813, max=84075, avg=68253.36, stdev=220.24, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.20% lat (usec) : 100=0.05%, 250=0.13%, 500=0.52%, 750=1.45%, 1000=2.48% lat (msec) : 2=19.07%, 4=44.45%, 10=30.09%, 20=1.45%, 50=0.10% lat (msec) : 100=0.01% cpu : usr=0.64%, sys=1.69%, ctx=2408175, majf=0, minf=508 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4091921,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=266MiB/s (279MB/s), 266MiB/s-266MiB/s (279MB/s-279MB/s), io=15.6GiB (16.8GB), run=60003-60003msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=28031: Wed Feb 2 00:07:20 2022 write: IOPS=68.3k, BW=267MiB/s (280MB/s)(15.6GiB/60003msec); slat (nsec): min=1062, max=50611k, avg=206709.24, stdev=515844.58 clat (nsec): min=491, max=58217k, avg=3538847.51, stdev=2211954.38 lat (usec): min=46, max=58793, avg=3745.81, stdev=2241.07 clat percentiles (usec): | 1.00th=[ 529], 5.00th=[ 1020], 10.00th=[ 1352], 20.00th=[ 1844], | 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3556], | 70.00th=[ 4146], 80.00th=[ 4883], 90.00th=[ 6259], 95.00th=[ 7570], | 99.00th=[10945], 99.50th=[12649], 99.90th=[18744], 99.95th=[22414], | 99.99th=[30540] bw ( KiB/s): min=205666, max=333724, per=100.00%, avg=273520.75, stdev=810.61, samples=3808 iops : min=51408, max=83423, avg=68374.75, stdev=202.66, samples=3808 lat (nsec) : 500=0.01%, 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.21% lat (usec) : 100=0.05%, 250=0.12%, 500=0.50%, 750=1.42%, 1000=2.45% lat (msec) : 2=18.99%, 4=44.21%, 10=30.54%, 20=1.43%, 50=0.08% lat (msec) : 100=0.01% cpu : usr=0.66%, sys=1.67%, ctx=2291683, majf=0, minf=481 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4098881,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=267MiB/s (280MB/s), 267MiB/s-267MiB/s (280MB/s-280MB/s), io=15.6GiB (16.8GB), run=60003-60003msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][100.0%][w=268MiB/s][w=68.6k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=28274: Wed Feb 2 00:08:21 2022 write: IOPS=69.0k, BW=269MiB/s (283MB/s)(15.8GiB/60002msec); slat (nsec): min=1062, max=68056k, avg=176848.85, stdev=488062.07 clat (nsec): min=692, max=77359k, avg=3532992.98, stdev=2215070.84 lat (usec): min=42, max=78568, avg=3710.06, stdev=2245.78 clat percentiles (usec): | 1.00th=[ 545], 5.00th=[ 1012], 10.00th=[ 1352], 20.00th=[ 1827], | 30.00th=[ 2245], 40.00th=[ 2638], 50.00th=[ 3064], 60.00th=[ 3556], | 70.00th=[ 4146], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7570], | 99.00th=[10814], 99.50th=[12518], 99.90th=[18482], 99.95th=[22414], | 99.99th=[30540] bw ( KiB/s): min=201233, max=338661, per=100.00%, avg=276170.71, stdev=819.64, samples=3808 iops : min=50302, max=84658, avg=69037.91, stdev=204.91, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.16% lat (usec) : 100=0.04%, 250=0.11%, 500=0.50%, 750=1.46%, 1000=2.53% lat (msec) : 2=19.26%, 4=43.92%, 10=30.52%, 20=1.41%, 50=0.08% lat (msec) : 100=0.01% cpu : usr=0.64%, sys=1.61%, ctx=2102892, majf=0, minf=494 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4138405,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=269MiB/s (283MB/s), 269MiB/s-269MiB/s (283MB/s-283MB/s), io=15.8GiB (16.9GB), run=60002-60002msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=0): [f(32)][100.0%][w=277MiB/s][w=71.0k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=29446: Wed Feb 2 00:09:21 2022 write: IOPS=69.6k, BW=272MiB/s (285MB/s)(15.9GiB/60003msec); slat (nsec): min=1052, max=50297k, avg=138125.49, stdev=435715.13 clat (nsec): min=521, max=59414k, avg=3537010.84, stdev=2183018.58 lat (usec): min=51, max=61090, avg=3675.34, stdev=2211.08 clat percentiles (usec): | 1.00th=[ 570], 5.00th=[ 1029], 10.00th=[ 1352], 20.00th=[ 1827], | 30.00th=[ 2245], 40.00th=[ 2671], 50.00th=[ 3064], 60.00th=[ 3556], | 70.00th=[ 4146], 80.00th=[ 4948], 90.00th=[ 6259], 95.00th=[ 7570], | 99.00th=[10683], 99.50th=[12256], 99.90th=[17433], 99.95th=[21365], | 99.99th=[30540] bw ( KiB/s): min=206519, max=341031, per=100.00%, avg=278664.74, stdev=817.73, samples=3808 iops : min=51622, max=85253, avg=69660.15, stdev=204.42, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.12% lat (usec) : 100=0.03%, 250=0.08%, 500=0.47%, 750=1.43%, 1000=2.51% lat (msec) : 2=19.26%, 4=43.83%, 10=30.86%, 20=1.34%, 50=0.06% lat (msec) : 100=0.01% cpu : usr=0.64%, sys=1.55%, ctx=1803573, majf=0, minf=455 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4177602,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=272MiB/s (285MB/s), 272MiB/s-272MiB/s (285MB/s-285MB/s), io=15.9GiB (17.1GB), run=60003-60003msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=0): [f(32)][100.0%][w=272MiB/s][w=69.7k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=29556: Wed Feb 2 00:10:21 2022 write: IOPS=69.8k, BW=272MiB/s (286MB/s)(16.0GiB/60003msec); slat (nsec): min=1052, max=35550k, avg=79344.28, stdev=323613.65 clat (nsec): min=771, max=53581k, avg=3588937.03, stdev=2176963.25 lat (usec): min=58, max=53842, avg=3668.46, stdev=2181.80 clat percentiles (usec): | 1.00th=[ 586], 5.00th=[ 1037], 10.00th=[ 1369], 20.00th=[ 1844], | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3621], | 70.00th=[ 4228], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701], | 99.00th=[10683], 99.50th=[12125], 99.90th=[16057], 99.95th=[18744], | 99.99th=[27395] bw ( KiB/s): min=204774, max=339176, per=100.00%, avg=279266.54, stdev=802.82, samples=3808 iops : min=51191, max=84786, avg=69810.42, stdev=200.71, samples=3808 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.13% lat (usec) : 100=0.03%, 250=0.06%, 500=0.44%, 750=1.37%, 1000=2.49% lat (msec) : 2=18.96%, 4=43.01%, 10=32.04%, 20=1.42%, 50=0.04% lat (msec) : 100=0.01% cpu : usr=0.62%, sys=1.45%, ctx=1356660, majf=0, minf=493 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4185468,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=272MiB/s (286MB/s), 272MiB/s-272MiB/s (286MB/s-286MB/s), io=16.0GiB (17.1GB), run=60003-60003msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=32): [w(32)][98.4%][w=280MiB/s][w=71.7k IOPS][eta 00m:01s] Jobs: 32 (f=32): [w(32)][100.0%][w=277MiB/s][w=70.9k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=29643: Wed Feb 2 00:11:21 2022 write: IOPS=69.8k, BW=273MiB/s (286MB/s)(16.0GiB/60003msec); slat (nsec): min=1052, max=34378k, avg=59061.75, stdev=277238.20 clat (nsec): min=742, max=47279k, avg=3608224.03, stdev=2171757.46 lat (usec): min=48, max=47679, avg=3667.45, stdev=2172.85 clat percentiles (usec): | 1.00th=[ 594], 5.00th=[ 1045], 10.00th=[ 1369], 20.00th=[ 1860], | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3654], | 70.00th=[ 4293], 80.00th=[ 5080], 90.00th=[ 6390], 95.00th=[ 7701], | 99.00th=[10683], 99.50th=[12125], 99.90th=[15664], 99.95th=[17433], | 99.99th=[23725] bw ( KiB/s): min=201486, max=341296, per=100.00%, avg=279357.72, stdev=825.73, samples=3808 iops : min=50369, max=85319, avg=69833.71, stdev=206.44, samples=3808 lat (nsec) : 750=0.01%, 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.11% lat (usec) : 100=0.03%, 250=0.05%, 500=0.42%, 750=1.37%, 1000=2.47% lat (msec) : 2=18.87%, 4=42.63%, 10=32.55%, 20=1.47%, 50=0.02% cpu : usr=0.60%, sys=1.42%, ctx=1195286, majf=0, minf=481 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4186642,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=273MiB/s (286MB/s), 273MiB/s-273MiB/s (286MB/s-286MB/s), io=16.0GiB (17.1GB), run=60003-60003msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 32 (f=0): [f(32)][100.0%][w=272MiB/s][w=69.6k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=29748: Wed Feb 2 00:12:22 2022 write: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60003msec); slat (nsec): min=1072, max=25141k, avg=14918.87, stdev=115673.29 clat (nsec): min=752, max=37983k, avg=3564960.33, stdev=2091487.49 lat (usec): min=57, max=37985, avg=3580.03, stdev=2089.00 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 1057], 10.00th=[ 1369], 20.00th=[ 1860], | 30.00th=[ 2278], 40.00th=[ 2671], 50.00th=[ 3130], 60.00th=[ 3621], | 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 6325], 95.00th=[ 7570], | 99.00th=[10421], 99.50th=[11600], 99.90th=[14615], 99.95th=[15926], | 99.99th=[18744] bw ( KiB/s): min=223459, max=340302, per=100.00%, avg=286170.92, stdev=703.56, samples=3808 iops : min=55856, max=85068, avg=71536.67, stdev=175.90, samples=3808 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.04% lat (usec) : 100=0.02%, 250=0.02%, 500=0.37%, 750=1.33%, 1000=2.48% lat (msec) : 2=19.22%, 4=43.12%, 10=32.14%, 20=1.26%, 50=0.01% cpu : usr=0.61%, sys=1.38%, ctx=872668, majf=0, minf=472 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4288951,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=279MiB/s (293MB/s), 279MiB/s-279MiB/s (293MB/s-293MB/s), io=16.4GiB (17.6GB), run=60003-60003msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randwrite, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=8 ... fio-3.27 Starting 32 processes Jobs: 2 (f=0): [_(12),f(2),_(18)][100.0%][w=266MiB/s][w=68.1k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=32): err= 0: pid=29827: Wed Feb 2 00:13:22 2022 write: IOPS=71.0k, BW=277MiB/s (291MB/s)(16.2GiB/60003msec); slat (nsec): min=1062, max=32432k, avg=19937.63, stdev=148066.81 clat (nsec): min=811, max=59246k, avg=3585601.00, stdev=2115468.91 lat (usec): min=52, max=59254, avg=3605.70, stdev=2113.21 clat percentiles (usec): | 1.00th=[ 635], 5.00th=[ 1074], 10.00th=[ 1385], 20.00th=[ 1860], | 30.00th=[ 2278], 40.00th=[ 2704], 50.00th=[ 3130], 60.00th=[ 3621], | 70.00th=[ 4228], 80.00th=[ 5014], 90.00th=[ 6390], 95.00th=[ 7635], | 99.00th=[10552], 99.50th=[11731], 99.90th=[14877], 99.95th=[16188], | 99.99th=[19792] bw ( KiB/s): min=222503, max=339437, per=100.00%, avg=284148.08, stdev=711.74, samples=3808 iops : min=55619, max=84852, avg=71031.03, stdev=177.94, samples=3808 lat (nsec) : 1000=0.01% lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.05% lat (usec) : 100=0.02%, 250=0.02%, 500=0.36%, 750=1.29%, 1000=2.44% lat (msec) : 2=18.96%, 4=43.25%, 10=32.27%, 20=1.33%, 50=0.01% lat (msec) : 100=0.01% cpu : usr=0.61%, sys=1.42%, ctx=918578, majf=0, minf=478 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=100.0%, 16=0.0%, 32=0.0%, >=64=0.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.1%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% issued rwts: total=0,4258348,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=8 Run status group 0 (all jobs): WRITE: bw=277MiB/s (291MB/s), 277MiB/s-277MiB/s (291MB/s-291MB/s), io=16.2GiB (17.4GB), run=60003-60003msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=295MiB/s][r=75.5k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=30964: Wed Feb 2 00:14:22 2022 read: IOPS=72.8k, BW=284MiB/s (298MB/s)(16.7GiB/60037msec) slat (nsec): min=971, max=173108k, avg=39325.53, stdev=826441.69 clat (nsec): min=1894, max=316249k, avg=42166512.42, stdev=14952004.11 lat (usec): min=417, max=316251, avg=42205.96, stdev=15032.61 clat percentiles (msec): | 1.00th=[ 9], 5.00th=[ 24], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 45], | 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 59], 95.00th=[ 68], | 99.00th=[ 91], 99.50th=[ 99], 99.90th=[ 121], 99.95th=[ 140], | 99.99th=[ 201] bw ( KiB/s): min=142106, max=403311, per=100.00%, avg=291270.21, stdev=902.19, samples=5664 iops : min=35512, max=100811, avg=72801.07, stdev=225.55, samples=5664 lat (usec) : 2=0.01%, 4=0.01%, 10=0.01%, 20=0.01%, 50=0.01% lat (usec) : 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01%, 1000=0.01% lat (msec) : 2=0.05%, 4=0.22%, 10=1.02%, 20=3.07%, 50=76.23% lat (msec) : 100=18.97%, 250=0.42%, 500=0.01% cpu : usr=0.24%, sys=0.55%, ctx=261137, majf=0, minf=3757 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4368373,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=284MiB/s (298MB/s), 284MiB/s-284MiB/s (298MB/s-298MB/s), io=16.7GiB (17.9GB), run=60037-60037msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=300MiB/s][r=76.9k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=31076: Wed Feb 2 00:15:23 2022 read: IOPS=71.5k, BW=279MiB/s (293MB/s)(16.4GiB/60037msec) slat (nsec): min=961, max=88836k, avg=17391.79, stdev=551047.29 clat (usec): min=22, max=221605, avg=42951.57, stdev=12630.75 lat (usec): min=383, max=222644, avg=42969.08, stdev=12673.43 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 32], | 30.00th=[ 39], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45], | 70.00th=[ 47], 80.00th=[ 51], 90.00th=[ 58], 95.00th=[ 63], | 99.00th=[ 84], 99.50th=[ 93], 99.90th=[ 112], 99.95th=[ 123], | 99.99th=[ 150] bw ( KiB/s): min=170354, max=369108, per=100.00%, avg=286058.08, stdev=686.26, samples=5664 iops : min=42573, max=92262, avg=71497.39, stdev=171.56, samples=5664 lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.06%, 10=0.36%, 20=1.78%, 50=77.37% lat (msec) : 100=20.13%, 250=0.27% cpu : usr=0.22%, sys=0.53%, ctx=218767, majf=0, minf=3743 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4290648,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=279MiB/s (293MB/s), 279MiB/s-279MiB/s (293MB/s-293MB/s), io=16.4GiB (17.6GB), run=60037-60037msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=299MiB/s][r=76.4k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=31174: Wed Feb 2 00:16:23 2022 read: IOPS=67.5k, BW=264MiB/s (277MB/s)(15.5GiB/60033msec) slat (nsec): min=941, max=113553k, avg=20605.09, stdev=608598.15 clat (usec): min=20, max=244312, avg=45449.46, stdev=13495.74 lat (usec): min=78, max=244315, avg=45470.18, stdev=13549.72 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 28], 10.00th=[ 30], 20.00th=[ 34], | 30.00th=[ 42], 40.00th=[ 44], 50.00th=[ 45], 60.00th=[ 47], | 70.00th=[ 50], 80.00th=[ 55], 90.00th=[ 61], 95.00th=[ 66], | 99.00th=[ 92], 99.50th=[ 101], 99.90th=[ 120], 99.95th=[ 129], | 99.99th=[ 163] bw ( KiB/s): min=147119, max=347537, per=100.00%, avg=270244.42, stdev=628.01, samples=5664 iops : min=36762, max=86866, avg=67544.04, stdev=157.00, samples=5664 lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.08%, 10=0.40%, 20=1.57%, 50=70.55% lat (msec) : 100=26.85%, 250=0.52% cpu : usr=0.21%, sys=0.51%, ctx=217032, majf=0, minf=3833 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4054701,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=264MiB/s (277MB/s), 264MiB/s-264MiB/s (277MB/s-277MB/s), io=15.5GiB (16.6GB), run=60033-60033msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=296MiB/s][r=75.8k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=31271: Wed Feb 2 00:17:23 2022 read: IOPS=74.5k, BW=291MiB/s (305MB/s)(17.1GiB/60035msec) slat (nsec): min=952, max=94814k, avg=15742.78, stdev=457046.68 clat (usec): min=38, max=202717, avg=41217.82, stdev=11676.36 lat (usec): min=65, max=202721, avg=41233.69, stdev=11708.54 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 44], | 70.00th=[ 46], 80.00th=[ 48], 90.00th=[ 56], 95.00th=[ 61], | 99.00th=[ 79], 99.50th=[ 88], 99.90th=[ 105], 99.95th=[ 112], | 99.99th=[ 140] bw ( KiB/s): min=182011, max=362829, per=100.00%, avg=298134.45, stdev=603.05, samples=5664 iops : min=45487, max=90694, avg=74515.97, stdev=150.76, samples=5664 lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.06%, 10=0.30%, 20=1.80%, 50=82.63% lat (msec) : 100=15.05%, 250=0.14% cpu : usr=0.23%, sys=0.55%, ctx=232008, majf=0, minf=3740 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4471696,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=291MiB/s (305MB/s), 291MiB/s-291MiB/s (305MB/s-305MB/s), io=17.1GiB (18.3GB), run=60035-60035msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=294MiB/s][r=75.2k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=31366: Wed Feb 2 00:18:23 2022 read: IOPS=72.4k, BW=283MiB/s (296MB/s)(16.6GiB/60036msec) slat (nsec): min=952, max=131203k, avg=21974.73, stdev=596139.49 clat (usec): min=3, max=270950, avg=42409.20, stdev=13195.67 lat (usec): min=80, max=270961, avg=42431.29, stdev=13247.16 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 27], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 45], | 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 64], | 99.00th=[ 86], 99.50th=[ 93], 99.90th=[ 109], 99.95th=[ 120], | 99.99th=[ 161] bw ( KiB/s): min=163161, max=388414, per=100.00%, avg=289753.58, stdev=752.52, samples=5664 iops : min=40771, max=97088, avg=72421.87, stdev=188.12, samples=5664 lat (usec) : 4=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.03%, 4=0.14%, 10=0.67%, 20=2.25%, 50=77.40% lat (msec) : 100=19.26%, 250=0.25%, 500=0.01% cpu : usr=0.23%, sys=0.53%, ctx=232559, majf=0, minf=3758 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4345429,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=283MiB/s (296MB/s), 283MiB/s-283MiB/s (296MB/s-296MB/s), io=16.6GiB (17.8GB), run=60036-60036msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=288MiB/s][r=73.8k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=32520: Wed Feb 2 00:19:24 2022 read: IOPS=71.8k, BW=281MiB/s (294MB/s)(16.5GiB/60036msec) slat (nsec): min=961, max=155112k, avg=23329.58, stdev=600906.79 clat (usec): min=18, max=269118, avg=42721.27, stdev=13343.53 lat (usec): min=109, max=269121, avg=42744.72, stdev=13395.21 clat percentiles (msec): | 1.00th=[ 13], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45], | 70.00th=[ 47], 80.00th=[ 51], 90.00th=[ 58], 95.00th=[ 64], | 99.00th=[ 88], 99.50th=[ 96], 99.90th=[ 113], 99.95th=[ 125], | 99.99th=[ 176] bw ( KiB/s): min=155002, max=377376, per=100.00%, avg=287593.81, stdev=736.00, samples=5664 iops : min=38739, max=94329, avg=71881.32, stdev=184.00, samples=5664 lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.02%, 4=0.10%, 10=0.54%, 20=2.09%, 50=77.36% lat (msec) : 100=19.56%, 250=0.33%, 500=0.01% cpu : usr=0.22%, sys=0.54%, ctx=235545, majf=0, minf=3694 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4313415,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=281MiB/s (294MB/s), 281MiB/s-281MiB/s (294MB/s-294MB/s), io=16.5GiB (17.7GB), run=60036-60036msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=291MiB/s][r=74.6k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=32630: Wed Feb 2 00:20:24 2022 read: IOPS=73.7k, BW=288MiB/s (302MB/s)(16.9GiB/60035msec) slat (nsec): min=962, max=177671k, avg=38019.32, stdev=644096.49 clat (usec): min=29, max=287028, avg=41615.50, stdev=13238.93 lat (usec): min=94, max=319545, avg=41653.64, stdev=13291.04 clat percentiles (msec): | 1.00th=[ 12], 5.00th=[ 26], 10.00th=[ 28], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 45], | 70.00th=[ 46], 80.00th=[ 49], 90.00th=[ 57], 95.00th=[ 63], | 99.00th=[ 88], 99.50th=[ 95], 99.90th=[ 112], 99.95th=[ 123], | 99.99th=[ 176] bw ( KiB/s): min=156946, max=373773, per=100.00%, avg=295110.41, stdev=718.25, samples=5664 iops : min=39217, max=93429, avg=73760.01, stdev=179.56, samples=5664 lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.11%, 10=0.60%, 20=2.37%, 50=80.04% lat (msec) : 100=16.54%, 250=0.31%, 500=0.01% cpu : usr=0.23%, sys=0.56%, ctx=327850, majf=0, minf=3771 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4426283,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=288MiB/s (302MB/s), 288MiB/s-288MiB/s (302MB/s-302MB/s), io=16.9GiB (18.1GB), run=60035-60035msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=292MiB/s][r=74.6k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=32752: Wed Feb 2 00:21:24 2022 read: IOPS=72.3k, BW=283MiB/s (296MB/s)(16.6GiB/60039msec) slat (nsec): min=972, max=125387k, avg=20090.72, stdev=496977.06 clat (nsec): min=1563, max=210556k, avg=42435419.14, stdev=12289751.96 lat (usec): min=123, max=210559, avg=42455.63, stdev=12326.24 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 44], 60.00th=[ 45], | 70.00th=[ 47], 80.00th=[ 50], 90.00th=[ 58], 95.00th=[ 63], | 99.00th=[ 82], 99.50th=[ 91], 99.90th=[ 108], 99.95th=[ 116], | 99.99th=[ 146] bw ( KiB/s): min=169789, max=367924, per=100.00%, avg=289626.85, stdev=669.34, samples=5664 iops : min=42433, max=91970, avg=72388.99, stdev=167.32, samples=5664 lat (usec) : 2=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.02%, 4=0.07%, 10=0.34%, 20=1.74%, 50=79.17% lat (msec) : 100=18.44%, 250=0.22% cpu : usr=0.23%, sys=0.52%, ctx=248980, majf=0, minf=3751 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4343055,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=283MiB/s (296MB/s), 283MiB/s-283MiB/s (296MB/s-296MB/s), io=16.6GiB (17.8GB), run=60039-60039msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=294MiB/s][r=75.3k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=32851: Wed Feb 2 00:22:25 2022 read: IOPS=72.8k, BW=285MiB/s (298MB/s)(16.7GiB/60037msec) slat (nsec): min=961, max=88402k, avg=16580.98, stdev=499828.64 clat (usec): min=16, max=190979, avg=42147.65, stdev=12291.43 lat (usec): min=76, max=200180, avg=42164.34, stdev=12331.85 clat percentiles (msec): | 1.00th=[ 14], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 36], 40.00th=[ 42], 50.00th=[ 43], 60.00th=[ 45], | 70.00th=[ 46], 80.00th=[ 50], 90.00th=[ 57], 95.00th=[ 62], | 99.00th=[ 83], 99.50th=[ 92], 99.90th=[ 108], 99.95th=[ 116], | 99.99th=[ 142] bw ( KiB/s): min=170567, max=369399, per=100.00%, avg=291617.48, stdev=663.74, samples=5664 iops : min=42628, max=92333, avg=72887.66, stdev=165.93, samples=5664 lat (usec) : 20=0.01%, 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01% lat (usec) : 750=0.01%, 1000=0.01% lat (msec) : 2=0.01%, 4=0.07%, 10=0.39%, 20=1.84%, 50=79.83% lat (msec) : 100=17.63%, 250=0.22% cpu : usr=0.23%, sys=0.53%, ctx=222791, majf=0, minf=3753 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4372949,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=285MiB/s (298MB/s), 285MiB/s-285MiB/s (298MB/s-298MB/s), io=16.7GiB (17.9GB), run=60037-60037msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% RANDREAD: (g=0): rw=randread, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=libaio, iodepth=64 ... fio-3.27 Starting 48 processes Jobs: 48 (f=48): [r(48)][100.0%][r=296MiB/s][r=75.7k IOPS][eta 00m:00s] RANDREAD: (groupid=0, jobs=48): err= 0: pid=32948: Wed Feb 2 00:23:25 2022 read: IOPS=74.6k, BW=291MiB/s (306MB/s)(17.1GiB/60036msec) slat (nsec): min=982, max=143449k, avg=14518.04, stdev=453275.74 clat (usec): min=39, max=211504, avg=41151.34, stdev=11534.04 lat (usec): min=227, max=230931, avg=41165.97, stdev=11567.45 clat percentiles (msec): | 1.00th=[ 15], 5.00th=[ 27], 10.00th=[ 29], 20.00th=[ 31], | 30.00th=[ 34], 40.00th=[ 41], 50.00th=[ 43], 60.00th=[ 44], | 70.00th=[ 46], 80.00th=[ 48], 90.00th=[ 55], 95.00th=[ 60], | 99.00th=[ 79], 99.50th=[ 87], 99.90th=[ 103], 99.95th=[ 112], | 99.99th=[ 148] bw ( KiB/s): min=182147, max=368337, per=100.00%, avg=298660.44, stdev=576.72, samples=5664 iops : min=45520, max=92066, avg=74647.02, stdev=144.17, samples=5664 lat (usec) : 50=0.01%, 100=0.01%, 250=0.01%, 500=0.01%, 750=0.01% lat (usec) : 1000=0.01% lat (msec) : 2=0.01%, 4=0.07%, 10=0.33%, 20=1.75%, 50=83.09% lat (msec) : 100=14.62%, 250=0.13% cpu : usr=0.23%, sys=0.54%, ctx=233134, majf=0, minf=3732 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=99.9% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.1%, >=64=0.0% issued rwts: total=4479122,0,0,0 short=0,0,0,0 dropped=0,0,0,0 latency : target=0, window=0, percentile=100.00%, depth=64 Run status group 0 (all jobs): READ: bw=291MiB/s (306MB/s), 291MiB/s-291MiB/s (306MB/s-306MB/s), io=17.1GiB (18.3GB), run=60036-60036msec Disk stats (read/write): nvme1n1: ios=0/0, merge=0/0, ticks=0/0, in_queue=0, util=0.00% -- 2.29.0