Help: FIO testing problem on arm64

Hi All:
I met a problem while testing the ramdisk peformance using fio. The platform is arm64 , fio version is 2.1.10.

The ramdisk is enabled on my platform , so there are dev/ram0 ...to dev/ram15 totall 16 ramdisk devices. I tried to test these ramdisks performance uisng the script file list as below.

#!/bin/bash

#disk_list=lsscsi | awk '{print $NF}' | cut -d/ -f3
disk_list="ram1 ram2 ram3 ram4 ram5 ram6 ram7 ram8 ram9 ram10 ram11 ram12 ram13 ram14 ram15" numjobs=1
runtime=180

run_18disk_test()
{
logdir=/root/18disk_performance_test
rm -rf $logdir
mkdir $logdir
for rw in read write randread randwrite

	do
		for bs in 4K 1M
		do
			for iodepth in 1 2 4 8 16 32
			do
				for disk in `echo $disk_list`
				do
					iodepth_2=`printf "%02d" $iodepth`
					fio --time_based --filename=/dev/$disk --rw=$rw --bs=$bs --ioengine=libaio --direct=1 --numjobs=$numjobs --iodepth=$iodepth --name=test --group_reporting --runtime=$runtime &> $logdir/${disk}_${bs}_${rw}_numjobs${numjobs}_iodepth${iodepth_2}.log &
				done
				sleep $runtime
			done
		done

done
}

But the test result is not ok.
The error log is:

test: (g=0): rw=write, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=1 fio-2.1.14
Starting 1 process
fio: cache invalidation of /dev/ram2 failed: Device or resource busy
fio: cache invalidation of /dev/ram2 failed: Device or resource busy
fio: cache invalidation of /dev/ram2 failed: Device or resource busy
fio: cache invalidation of /dev/ram2 failed: Device or resource busy
fio: cache invalidation of /dev/ram2 failed: Device or resource busy
fio: cache invalidation of /dev/ram2 failed: Device or resource busy
fio: cache invalidation of /dev/ram2 failed: Device or resource busy
fio: cache invalidation of /dev/ram2 failed: Device or resource busy

Have you tried sleeping for slightly longer then the runtime passed to fio?

As a result of implementation details and the simple fact that sleep isn't exact, nor is fio's runtime, a script not allowing a sufficient buffer between backgrounded fio operations but intending to run operations sequentially may step on itself.

Since the intention of the script is to run serially, you dont need both '&' and 'sleep', you can remove '&' and let the fio command block and remove sleep

Hi Philip:
Thanks for you replying.
Can you explain the case of "may step on itself" more detail ?