Recent changes
Subsystems (2)
The following changes since commit 15b87723551b424d0db4c53577b567e670c7d4d8:
Fix bad random offset generation for file map (2011-10-12 09:46:01 +0200)
are available in the git repository at:
git://git.kernel.dk/fio.git master
Bruce Cran (2):
Fix compile on FreeBSD
Fix Windows issue with socklen_t
Jens Axboe (183):
Start of client/server
Hide things not appropriate behind is_backend
Use poll() for connect loop
Close listen socket when done
Properly log errors in server
Add start of client, start of real protocol
Start of functional client
Server logging cleanup/functionality
Use length guarded sprintf functions
Pass more arguments to fio_init_net_cmd()
server: debug fixes
crc16: use void * as the argument
Reinstate double logging if f_err != stderr
server: switch to 16-bit crc
server: serial is a 64-bit field
client: fix error in logging output
server: quit client when job run is complete
server: ensure payload larger than max is broken into pieces
client: initial support for multiple connections
client: continue support for multiple connections
server: exit gracefully on ctrl-c
client: re-setup poll fd array in case a client disappears
Add FD_NET debugging value
server/client: add FD_NET debug clues
Fixup some bad file naming
server: malloc/free fix
Style fixup
server: make struct group_run_stats network transfer friendly
server: initial support for daemonizing
server: start conversion of data structures to network friendly types
client: don't setup shared mem area for just the frontend
server: endianness bug and exit command
server: unify shm setup
Move stat_io_bytes/time to thread_data
Move getrusage() out of thread_stat
Allocate thread_stat name arrays statically
server: transmit status as structures, not text
client: fixup quit
Add type checking to 16/32/64 endianness converters
server: package defrag fix
Add TODO list
Abstract out calculation of ETA from display of ETA
server: add ETA as a specific command
server: send network copy of run_str[]
server: attempt to handle client ctrl-c
server: fix typos in conversion
poll: break on EINTR without complaining
server: remove some debug statements
server: silence mem debug warning
client: handle connection failure
server: error handling and probe command
client: probe command support
server: idle loop support
parser: use logging infrastructure
server: fix for non zero appended strings
Style fixup
server: initial support for command line passing
server: send return quit when asked to leave
client: improve poll() loop
server: improve exit handling
client: improve handling of non-newline text strings
server: add a few more debug logging statements
Fix bad indexing of options
server: more debug dumping
Move endianness check to OS parts
log: needs stdarg include for va_list
HPUX endianness
Wider endianness support
More endianness for platforms
Need signal.h for sigaction()
server: fix sk typo and add endian type to probe
server: log where connection is coming from
server: cast sockaddr_in for accept()
Update TODO
Generic endianness typo
server: log locally if connection isn't up yet
Finalize (?) byte swap/endian stuff
server: portability fixups
log: don't use vsyslog
Add fio_socklen_t
Assume AIX is big endian
AIX fixes
Remove endianness TODO, should be done now
Add IEEE 754 pack and unpack helpers
Change network transmitted doubles to fio_fp64_t IEEE 754 type Merge branch 'master' into client-server
Print usage if no arguments are given
t/stest: add log.o smaller helper
Re-enable -O2 optimization
Add IEEE 754 test case
Remove float conversion from TODO
Fio 1.99
Fix warnings about unused fwrite() return value
client: prefix server text messages with <hostname>
Endian sanity check
Move endian support out of server.h
Add support for write_iops_log
client: check and error out on exceeding number of command line args to pass
parser: always use the fio logging instead of stderr/stdout
client: ensure that cmd line arguments are always run
server: send quit if we don't add a job
client: disconnect on read failure
client: improve handling of multiple clients
server: require poll before fio_net_recv_cmd()
server: quit on !block and backend exit
Fio 1.99.1
Only print usage() on error
Correctly handle multiple clients for various command line arguments Add jhash (Jenkins hash) and use that for file names
client: add hashes for fd/name lookups
net: support for unix domain sockets
Add support for client/server connection over unix domain sockets Remember to close sockets on error
Update TODO
server: fix += -> + typo
Unify client/server argument
server: fix bad interpretation of local socket binding
Remember to clear client cookie
Split version into separate include fio
client: remove leftover debug printf()
server: increase default max pdu length to 1024
Fio 1.99.2
Poll server idle loop any time the main status thread sleeps
client: fix mem leak
Pass arch/os in probe
server: ensure to set proper port
client: don't clear client->addr after it's been set
server: properly configure port without argument
client: pretty up probe output
Makefile: move -rdynamic to linking flag
Fix warning when clang is used as the compiler
Makefile: use -O3 by default
Fio 1.99.3
client: used hostname passed back in probe as log prefix
Add protocol support for an arbitrary number of command line arguments Update TODO
client: fix jobs_eta conversion typo
client: sum running ETA of jobs
client/server: request ETA instead of having the server send it automatically
init: typo, remove -> remote.
client: track pending ETA requests
client/server: few select speedups
Abstract out and export summation of thread_stats
client: properly assign client eta in flight
server: write pid file for backgrounded server to specified file
server: improve pidfile and log handling
client: duplicate arguments to "empty" clients
server: fread() - check <= 0 return value
Fio 1.99.4
client/server: track and handle command timeouts
server: assume PID is dead on ESRCH
server: error handling fixes
client: display summed total of all clients when all stats have been received
client: dec sum_stat_clients if one a client is disconnected client/server: fix ptr <-> uint64_t casting warnings on 32-bit builds
server: include 32/64-bit in probe
client: cleanup bit printing
Fix off-by-one in jobs_eta allocation
Merge branch 'master' into client-server
Correct Windows fio version
Fix clat percentile display
Pretty up clat percentile display so it's actually readable
Be a bit more defensive in clat percentile calc and display
server: fix bug in converting/storing clat percentiles
Fio 1.99.5
Remove extra \n before printing run status
Enable completion latency percentiles by default
Disable clat percentiles if gtod_reduce=1 is set
Adapt clat percentiles for min/max values
client/server: add support for passing disk_util structures Break double loop on end-of-clat percentiles
Silence uninitialized mem warning on disk_util send
Update TODO
Add IOPS to terse output
Don't output version for terse output
Add completion latency percentiles to terse output format
Add disk utilization to terse format output
Move IEEE754 support code to lib/
Check string length of ts->description, not value
Fio 1.99.6
Update man page
Man page typo
Only print ts->description if set for non-terse output
Martin Steigerwald (1):
Escape minus signs in manpage to fix lintian warning:
HOWTO | 29 ++- Makefile | 32 +- README | 69 +++- SERVER-TODO | 2 + arch/arch-alpha.h | 2 +- arch/arch-arm.h | 2 +- arch/arch-generic.h | 2 +- arch/arch-hppa.h | 2 +- arch/arch-ia64.h | 2 +- arch/arch-mips.h | 2 +- arch/arch-ppc.h | 2 +- arch/arch-s390.h | 2 +- arch/arch-sh.h | 2 +- arch/arch-sparc.h | 2 +- arch/arch-sparc64.h | 2 +- arch/arch-x86.h | 2 +- arch/arch-x86_64.h | 2 +- arch/arch.h | 4 +- client.c | 995 ++++++++++++++++++++++++++++++++++++++++++ crc/crc16.c | 5 +- crc/crc16.h | 2 +- debug.c | 4 +- debug.h | 1 + diskutil.c | 194 +++++---- diskutil.h | 40 ++- engines/net.c | 209 +++++++-- eta.c | 163 ++++--- examples/netio | 9 +- filehash.c | 4 +- fio.1 | 128 +++++- fio.c | 261 +++++++++--- fio.h | 216 ++-------- fio_generate_plots | 18 + fio_version.h | 8 + hash.h | 77 ++++ init.c | 429 +++++++++++++------ io_u.c | 5 +- iolog.c | 541 +++++++++++++++++++++++ iolog.h | 14 +- lib/ieee754.c | 84 ++++ lib/ieee754.h | 20 + log.c | 574 +++---------------------- log.h | 20 +- options.c | 49 ++- os/os-aix.h | 13 + os/os-freebsd.h | 14 + os/os-hpux.h | 14 + os/os-linux.h | 19 +- os/os-mac.h | 16 + os/os-netbsd.h | 13 + os/os-solaris.h | 13 + os/os-windows.h | 11 + os/os.h | 82 ++++ os/windows/install.wxs | 4 +- os/windows/version.h | 10 +- parse.c | 107 +++--- server.c | 1134 ++++++++++++++++++++++++++++++++++++++++++++++++ server.h | 144 ++++++ stat.c | 494 ++++++++++++++------- stat.h | 201 +++++++++ t/ieee754.c | 21 + t/log.c | 15 + 62 files changed, 5162 insertions(+), 1395 deletions(-) create mode 100644 SERVER-TODO create mode 100644 client.c create mode 100644 fio_version.h create mode 100644 iolog.c create mode 100644 lib/ieee754.c create mode 100644 lib/ieee754.h create mode 100644 server.c create mode 100644 server.h create mode 100644 stat.h create mode 100644 t/ieee754.c create mode 100644 t/log.c
---
Diff of recent changes:
HOWTO +26 -3
@@ -267,7 +267,7 @@ filename=str Fio normally makes up a filename based on the job name, files between threads in a job or several jobs, specify a filename for each of them to override the default. If the ioengine used is 'net', the filename is the host, port, - and protocol to use in the format of =host/port/protocol. + and protocol to use in the format of =host,port,protocol. See ioengine=net for more. If the ioengine is file based, you can specify a number of files by separating the names with a ':' colon. So if you wanted a job to open /dev/sda and /dev/sdb
@@ -864,6 +864,9 @@ exitall When one job finishes, terminate the rest. The default is bwavgtime=int Average the calculated bandwidth over the given time. Value is specified in milliseconds. +iopsavgtime=int Average the calculated IOPS over the given time. Value + is specified in milliseconds. + create_serialize=bool If true, serialize the file creating for the jobs. This may be handy to avoid interleaving of data files, which may greatly depend on the filesystem
@@ -1104,6 +1107,9 @@ write_lat_log=str Same as write_bw_log, except that this option stores io and foo_lat.log. This helps fio_generate_plot fine the logs automatically. +write_bw_log=str If given, write an IOPS log of the jobs in this job + file. See write_bw_log. + lockmem=int Pin down the specified amount of memory with mlock(2). Can potentially be used instead of removing memory or booting with less memory to simulate a smaller amount of memory.
@@ -1356,25 +1362,42 @@ Split up, the format is as follows: version, jobname, groupid, error READ status: - Total IO (KB), bandwidth (KB/sec), runtime (msec) + Total IO (KB), bandwidth (KB/sec), IOPS, runtime (msec) Submission latency: min, max, mean, deviation Completion latency: min, max, mean, deviation + Completion latency percentiles: 20 fields (see below) Total latency: min, max, mean, deviation Bw: min, max, aggregate percentage of total, mean, deviation WRITE status: - Total IO (KB), bandwidth (KB/sec), runtime (msec) + Total IO (KB), bandwidth (KB/sec), IOPS, runtime (msec) Submission latency: min, max, mean, deviation Completion latency: min, max, mean, deviation + Completion latency percentiles: 20 fields (see below) Total latency: min, max, mean, deviation Bw: min, max, aggregate percentage of total, mean, deviation CPU usage: user, system, context switches, major faults, minor faults IO depths: <=1, 2, 4, 8, 16, 32, >=64 IO latencies microseconds: <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000 IO latencies milliseconds: <=2, 4, 10, 20, 50, 100, 250, 500, 750, 1000, 2000, >=2000 + Disk utilization: Disk name, Read ios, write ios, + Read merges, write merges, + Read ticks, write ticks, + Read in-queue time, write in-queue time, + Disk utilization percentage Additional Info (dependant on continue_on_error, default off): total # errors, first error code Additional Info (dependant on description being set): Text description +Completion latency percentiles can be a grouping of up to 20 sets, so +for the terse output fio writes all of them. Each field will look like this: + + 1.00%=6112 + +which is the Xth percentile, and the usec latency associated with it. + +For disk utilization, all disks used by fio are shown. So for each disk +there will be a disk utilization section. + 8.0 Trace file format ---------------------
Makefile +21 -11
@@ -2,7 +2,7 @@ CC = gcc DEBUGFLAGS = -D_FORTIFY_SOURCE=2 -DFIO_INC_DEBUG CPPFLAGS= -D_GNU_SOURCE -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 \ $(DEBUGFLAGS) -OPTFLAGS= -O2 -fno-omit-frame-pointer -g $(EXTFLAGS) +OPTFLAGS= -O3 -fno-omit-frame-pointer -g $(EXTFLAGS) CFLAGS = -std=gnu99 -Wwrite-strings -Wall $(OPTFLAGS) LIBS = -lm $(EXTLIBS) PROGS = fio
@@ -12,9 +12,9 @@ UNAME := $(shell uname) SOURCE = gettime.c fio.c ioengines.c init.c stat.c log.c time.c filesetup.c \ eta.c verify.c memory.c io_u.c parse.c mutex.c options.c \ rbtree.c smalloc.c filehash.c profile.c debug.c lib/rand.c \ - lib/num2str.c $(wildcard crc/*.c) engines/cpu.c \ + lib/num2str.c lib/ieee754.c $(wildcard crc/*.c) engines/cpu.c \ engines/mmap.c engines/sync.c engines/null.c engines/net.c \ - memalign.c + memalign.c server.c client.c iolog.c ifeq ($(UNAME), Linux) SOURCE += diskutil.c fifo.c blktrace.c helpers.c cgroup.c trim.c \
@@ -22,7 +22,7 @@ ifeq ($(UNAME), Linux) engines/splice.c engines/syslet-rw.c engines/guasi.c \ engines/binject.c engines/rdma.c profiles/tiobench.c LIBS += -lpthread -ldl -lrt -laio - CFLAGS += -rdynamic + LDFLAGS += -rdynamic endif ifeq ($(UNAME), SunOS) SOURCE += fifo.c lib/strsep.c helpers.c engines/posixaio.c \
@@ -33,12 +33,12 @@ endif ifeq ($(UNAME), FreeBSD) SOURCE += helpers.c engines/posixaio.c LIBS += -lpthread -lrt - CFLAGS += -rdynamic + LDFLAGS += -rdynamic endif ifeq ($(UNAME), NetBSD) SOURCE += helpers.c engines/posixaio.c LIBS += -lpthread -lrt - CFLAGS += -rdynamic + LDFLAGS += -rdynamic endif ifeq ($(UNAME), AIX) SOURCE += fifo.c helpers.c lib/getopt_long.c engines/posixaio.c
@@ -63,9 +63,16 @@ endif OBJS = $(SOURCE:.c=.o) -T_OBJS = t/stest.o -T_OBJS += mutex.o smalloc.o -T_PROGS = t/stest +T_SMALLOC_OBJS = t/stest.o +T_SMALLOC_OBJS += mutex.o smalloc.o t/log.o +T_SMALLOC_PROGS = t/stest + +T_IEEE_OBJS = t/ieee754.o +T_IEEE_OBJS += ieee754.o +T_IEEE_PROGS = t/ieee754 + +T_OBJS = $(T_SMALLOC_OBJS) +T_OBJS += $(T_IEEE_OBJS) ifneq ($(findstring $(MAKEFLAGS),s),s) ifndef V
@@ -84,8 +91,11 @@ all: .depend $(PROGS) $(SCRIPTS) .c.o: .depend $(QUIET_CC)$(CC) -o $@ -c $(CFLAGS) $(CPPFLAGS) $< -t/stest: $(T_OBJS) - $(QUIET_CC)$(CC) $(LDFLAGS) $(CFLAGS) -o $@ $(T_OBJS) $(LIBS) $(LDFLAGS) +t/stest: $(T_SMALLOC_OBJS) + $(QUIET_CC)$(CC) $(LDFLAGS) $(CFLAGS) -o $@ $(T_SMALLOC_OBJS) $(LIBS) $(LDFLAGS) + +t/ieee754: $(T_IEEE_OBJS) + $(QUIET_CC)$(CC) $(LDFLAGS) $(CFLAGS) -o $@ $(T_IEEE_OBJS) $(LIBS) $(LDFLAGS) fio: $(OBJS) $(QUIET_CC)$(CC) $(LDFLAGS) $(CFLAGS) -o $@ $(OBJS) $(LIBS) $(LDFLAGS)
README +62 -7
91 lines not on this page · raw message
SERVER-TODO new file +2 -0
@@ -0,0 +1,2 @@ +- Collate ETA output from multiple connections into 1 +- If group_reporting is set, collate final output from multiple connections
arch/arch-alpha.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_ALPHA_H #define ARCH_ALPHA_H -#define ARCH (arch_alpha) +#define FIO_ARCH (arch_alpha)#ifndef __NR_ioprio_set#define __NR_ioprio_set 442
arch/arch-arm.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_ARM_H #define ARCH_ARM_H -#define ARCH (arch_arm) +#define FIO_ARCH (arch_arm)#ifndef __NR_ioprio_set#define __NR_ioprio_set 314
arch/arch-generic.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_GENERIC_H #define ARCH_GENERIC_H -#define ARCH (arch_generic) +#define FIO_ARCH (arch_generic)#define nop do { } while (0)#define read_barrier () __asm__ __volatile__("": : :"memory")
arch/arch-hppa.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_HPPA_H #define ARCH_HPPA_H -#define ARCH (arch_hppa) +#define FIO_ARCH (arch_hppa)#ifndef __NR_ioprio_set#define __NR_ioprio_set 267
arch/arch-ia64.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_IA64_H #define ARCH_IA64_H -#define ARCH (arch_ia64) +#define FIO_ARCH (arch_ia64)#ifndef __NR_ioprio_set#define __NR_ioprio_set 1274
arch/arch-mips.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_MIPS64_H #define ARCH_MIPS64_H -#define ARCH (arch_mips) +#define FIO_ARCH (arch_mips)#ifndef __NR_ioprio_set#define __NR_ioprio_set 314
arch/arch-ppc.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_PPC_H #define ARCH_PPH_H -#define ARCH (arch_ppc) +#define FIO_ARCH (arch_ppc)#ifndef __NR_ioprio_set#define __NR_ioprio_set 273
arch/arch-s390.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_S390_H #define ARCH_S390_H -#define ARCH (arch_s390) +#define FIO_ARCH (arch_s390)#ifndef __NR_ioprio_set#define __NR_ioprio_set 282
arch/arch-sh.h +1 -1
@@ -3,7 +3,7 @@#ifndef ARCH_SH_H #define ARCH_SH_H -#define ARCH (arch_sh) +#define FIO_ARCH (arch_sh)#ifndef __NR_ioprio_set#define __NR_ioprio_set 288
arch/arch-sparc.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_SPARC_H #define ARCH_SPARC_H -#define ARCH (arch_sparc) +#define FIO_ARCH (arch_sparc)#ifndef __NR_ioprio_set#define __NR_ioprio_set 196
arch/arch-sparc64.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_SPARC64_H #define ARCH_SPARC64_H -#define ARCH (arch_sparc64) +#define FIO_ARCH (arch_sparc64)#ifndef __NR_ioprio_set#define __NR_ioprio_set 196
arch/arch-x86.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_X86_H #define ARCH_X86_H -#define ARCH (arch_i386) +#define FIO_ARCH (arch_i386)#ifndef __NR_ioprio_set#define __NR_ioprio_set 289
arch/arch-x86_64.h +1 -1
@@ -1,7 +1,7 @@#ifndef ARCH_X86_64_h#define ARCH_X86_64_h -#define ARCH (arch_x86_64) +#define FIO_ARCH (arch_x86_64)#ifndef __NR_ioprio_set#define __NR_ioprio_set 251
arch/arch.h +3 -1
@@ -8,7 +8,7 @@#endif enum { - arch_x86_64, + arch_x86_64 =1 , arch_i386, arch_ppc, arch_ia64,
@@ -21,6 +21,8 @@ enum { arch_hppa, arch_generic, + + arch_nr, }; enum {
client.c new file +995 -0
995 lines not on this page · raw message
crc/crc16.c +3 -2
@@ -43,11 +43,12 @@ unsigned short const crc16_table[256] = { 0x8201, 0x42C0, 0x4380, 0x8341, 0x4100, 0x81C1, 0x8081, 0x4040 }; -unsigned short crc16 (unsigned char const *buffer,unsigned int len) +unsigned short crc16 (const void *buffer,unsigned int len) { +const unsigned char *cp = (const unsigned char *) buffer;unsigned short crc =0 ;while (len--) - crc =crc16_byte (crc, *buffer++); + crc =crc16_byte (crc, *cp++);return crc; }
crc/crc16.h +1 -1
@@ -17,7 +17,7 @@extern unsigned short const crc16_table[256 ]; -extern unsigned short crc16 (const unsigned char *buffer,unsigned int len); +extern unsigned short crc16 (const void *buffer,unsigned int len);static inline unsigned short crc16_byte (unsigned short crc,const unsigned char data)
debug.c +2 -2
@@ -16,8 +16,8 @@ void __dprint(int type, const char *str, ...) && pid != *fio_debug_jobp)return ; -log_info ("%-8s " , debug_levels[type].name); -log_info ("%-5u " , (int ) pid); +log_local ("%-8s " , debug_levels[type].name); +log_local ("%-5u " , (int ) pid);va_start (args, str);log_valist (str, args);
debug.h +1 -0
@@ -18,6 +18,7 @@ enum {FD_MUTEX ,FD_PROFILE ,FD_TIME , +FD_NET , FD_DEBUG_MAX, };
diskutil.c +112 -82
282 lines not on this page · raw message
diskutil.h +31 -9
@@ -1,16 +1,31 @@#ifndef FIO_DISKUTIL_H #define FIO_DISKUTIL_H +#define FIO_DU_NAME_SZ 64 +/* * Disk utils as read in /sys/block/<dev>/stat */ struct disk_util_stat { -unsigned ios[2 ]; -unsigned merges[2 ]; -unsigned long long sectors[2 ]; -unsigned ticks[2 ]; -unsigned io_ticks; -unsigned time_in_queue; +uint8_t name[FIO_DU_NAME_SZ ]; +uint32_t ios[2 ]; +uint32_t merges[2 ]; +uint64_t sectors[2 ]; +uint32_t ticks[2 ]; +uint32_t io_ticks; +uint32_t time_in_queue; +uint64_t msec; +}; + +struct disk_util_agg { +uint32_t ios[2 ]; +uint32_t merges[2 ]; +uint64_t sectors[2 ]; +uint32_t ticks[2 ]; +uint32_t io_ticks; +uint32_t time_in_queue; +uint32_t slavecount; +fio_fp64_t max_util; };/*
@@ -31,6 +46,8 @@ struct disk_util {struct disk_util_stat dus;struct disk_util_stat last_dus; +struct disk_util_agg agg; +/* For software raids, this entry maintains pointers to the * entries for the slave devices. The disk_util entries for * the slaves devices should primarily be maintained through
@@ -40,7 +57,6 @@ struct disk_util {*/ struct flist_head slaves; -unsigned long msec;struct timeval time;struct fio_mutex *lock;
@@ -76,15 +92,21 @@ static inline void disk_util_dec(struct disk_util *du)#define DISK_UTIL_MSEC (250) +extern struct flist_head disk_list; +/* * disk util stuff */ #ifdef FIO_HAVE_DISK_UTIL -extern void show_disk_util (void ); +extern void print_disk_util (struct disk_util_stat *,struct disk_util_agg *,int terse); +extern void show_disk_util (int terse); +extern void free_disk_util (void );extern void init_disk_util (struct thread_data *);extern void update_io_ticks (void );#else -#define show_disk_util () +#define print_disk_util (dus, agg, terse) +#define show_disk_util (terse) +#define free_disk_util ()#define init_disk_util (td)#define update_io_ticks ()#endif
engines/net.c +160 -49
343 lines not on this page · raw message
eta.c +97 -66
260 lines not on this page · raw message
examples/netio +7 -2
@@ -1,8 +1,12 @@ # Example network job, just defines two clients that send/recv data [global] ioengine=net -#the below defaults to a tcp connection, add /udp at the end for udp -filename=localhost/8888 +#this would use UDP over localhost, port 8888 +#filename=localhost,8888,udp +#this would use a local domain socket /tmp/fio.sock +#filename=/tmp/fio.sock,,unix +#TCP, port 8888, localhost +filename=localhost,8888,tcp bs=4k size=10g #set the below option to enable end-to-end data integrity tests
@@ -12,4 +16,5 @@ size=10g rw=read [sender] +startdelay=1 rw=write
filehash.c +2 -2
@@ -3,7 +3,7 @@#include "fio.h" #include "flist.h" -#include "crc/crc16.h" +#include "hash.h" #define HASH_BUCKETS 512#define HASH_MASK (HASH_BUCKETS - 1)
@@ -15,7 +15,7 @@ static struct fio_mutex *hash_lock;static unsigned short hash (const char *name) { -return crc16 ((const unsigned char *) name,strlen (name)) &HASH_MASK ; +return jhash (name,strlen (name),0 ) &HASH_MASK ; }void remove_file_hash (struct fio_file *f)
fio.1 +111 -17
207 lines not on this page · raw message
fio.c +197 -64
408 lines not on this page · raw message
fio.h +39 -177
216 lines not on this page · raw message
hash.h +77 -0
77 lines not on this page · raw message
init.c +301 -128
429 lines not on this page · raw message
iolog.c new file +541 -0
541 lines not on this page · raw message
lib/ieee754.c new file +84 -0
84 lines not on this page · raw message
log.c +64 -510
574 lines not on this page · raw message
options.c +44 -5
49 lines not on this page · raw message
os/os.h +82 -0
82 lines not on this page · raw message
parse.c +55 -52
107 lines not on this page · raw message
server.c new file +1134 -0
1,134 lines not on this page · raw message
server.h new file +144 -0
144 lines not on this page · raw message
stat.c +331 -163
494 lines not on this page · raw message
stat.h new file +201 -0
201 lines not on this page · raw message
The rest of this diff is not shown. View the raw message.
Part of the diff is not shown here. View the raw message.
Fio 2.0 coming (was "Re: Recent changes")
Hi,
As you may have noticed from the change email this morning, I pulled the client-server branch into the master branch. The old master branch now lives in stable-1.x, which is a maintenance branch for 1.60 and on. Only bug fixes should go into that branch from now on, all new developments should go into the 'master' (or 2.x) branch. For now the version is still 1.99.6, I'll release 2.0 once it's seen a few weeks of testing.
Highlights of changes between stable-1.x and master:
- Basic support for client/server. I wrote about this in here before, I wont bore you with more details. See the included documentation for how to play with that, if you are interested.
- Completion latency percentiles is a great feature of 1.x, but I don't think a lot of people have used it. I've prettied up the output and turned it on by default.
- Terse output format has been modified, new additions are:
Fio version
IOPS
Completion latency percentiles
Disk utilization
I've wanted to add IOPS and disk utilization for a while, now seems a good time to introduce a modified format and get all of them done in one sweep.
- Support has been added for write_iops_log. It's similar to write_bw_log, but writes the IOPS instead. For mixed block size workloads, it's not possible to deduct IOPS from the bandwidth.
- Network IO engine has support for local unix domain sockets as well, plus the format for configuring it has been modified to appropriately support that.
- Build is clang clean, and works fine with it (for some quick testing, clang 2.8 used here). There were in fact only 2-3 warnings prior.
Especially the client/server split-up has been an exercise in refactoring lots of code into network friendly structures, and splitting previous "show these results" code into "calculate these results" and "display these results" parts instead.
Please give the current master branch (or the 1.99.x releases) a good testing, it should work fine. I care more about generic testing just moving to the 2.x/master branch than specific client/server functionality testing, since that feature will be stabilized soonish and regressions are the worst enemy of something like fio. As you perhaps saw from the 'Recent changes' email, there's been a few changes since 1.60 in the current branch:
62 files changed, 5162 insertions(+), 1395 deletions(-)
So we do need a few weeks to ensure that everything is fine before I tag 2.0 and release it.
Martin, can I talk you into taking a look at the man page to see whether it's clean (it's not, I guarantee it)?
And folks running on non-linux, please try and build it at least (extra brownie points for actually running it, too).
Thanks!
Signature · 1 line
Jens Axboe
Re: Fio 2.0 coming (was "Re: Recent changes")
Quoted 2 lines · Jens Axboe
And folks running on non-linux, please try and build it at least (extra brownie points for actually running it, too).
It seems the latest version still has problems on non-Linux platforms when running in client/server mode:
Quoted 1 line
./fio -S
fio: server listening on 0.0.0.0:8765
Quoted 1 line
./fio -C localhost examples/tiobench-example
hostname=vfreebsd_amd64, be=0, 64-bit, os=FreeBSD, arch=x64-64, fio=1.99.7
Quoted 1 line
Signature · 1 line
Bruce Cran
Re: Fio 2.0 coming (was "Re: Recent changes")
Quoted 6 lines · Bruce Cran
Quoted 2 lines · Jens Axboe
And folks running on non-linux, please try and build it at least (extra brownie points for actually running it, too).
Quoted 7 lines
It seems the latest version still has problems on non-Linux platforms when running in client/server mode:
Quoted 1 line
./fio -S
fio: server listening on 0.0.0.0:8765
Quoted 1 line
./fio -C localhost examples/tiobench-example
hostname=vfreebsd_amd64, be=0, 64-bit, os=FreeBSD, arch=x64-64, fio=1.99.7
Can you try and add --debug=all before the -S and -C on both ends, perhaps that'll give a clue as to what is going on. I'll see if I can find a FreeBSD machine here for a test spin.
Signature · 1 line
Jens Axboe
Re: Fio 2.0 coming (was "Re: Recent changes")
Quoted 10 lines · Jens Axboe
Quoted 9 lines · Bruce Cran
Quoted 2 lines · Jens Axboe
And folks running on non-linux, please try and build it at least (extra brownie points for actually running it, too).
It seems the latest version still has problems on non-Linux platforms when running in client/server mode:
Quoted 1 line
./fio -S
Quoted 8 lines
Quoted 4 lines
fio: server listening on 0.0.0.0:8765
Quoted 1 line
./fio -C localhost examples/tiobench-example
hostname=vfreebsd_amd64, be=0, 64-bit, os=FreeBSD, arch=x64-64, fio=1.99.7
Can you try and add --debug=all before the -S and -C on both ends, perhaps that'll give a clue as to what is going on. I'll see if I can find a FreeBSD machine here for a test spin.
It's not OS specific, which did sound odd to me. Looks like the parser is broken for some comment types when being passed a buffer instead of a file.
Signature · 1 line
Jens Axboe
Re: Fio 2.0 coming (was "Re: Recent changes")
Quoted 16 lines · Jens Axboe
Quoted 15 lines · Jens Axboe
Quoted 13 lines · Bruce Cran
Quoted 2 lines · Jens Axboe
And folks running on non-linux, please try and build it at least (extra brownie points for actually running it, too).
It seems the latest version still has problems on non-Linux platforms when running in client/server mode:
Quoted 1 line
./fio -S
fio: server listening on 0.0.0.0:8765
Quoted 1 line
./fio -C localhost examples/tiobench-example
hostname=vfreebsd_amd64, be=0, 64-bit, os=FreeBSD, arch=x64-64, fio=1.99.7
Quoted 7 lines
Quoted 3 lines
Can you try and add --debug=all before the -S and -C on both ends, perhaps that'll give a clue as to what is going on. I'll see if I can find a FreeBSD machine here for a test spin.
It's not OS specific, which did sound odd to me. Looks like the parser is broken for some comment types when being passed a buffer instead of a file.
Please try current -git, it should work now.
Signature · 1 line
Jens Axboe
Re: Fio 2.0 coming (was "Re: Recent changes")
Quoted 1 line · Jens Axboe
Please try current -git, it should work now.
Thanks, it's fixed.
I can crash the client by pressing ctrl-c on the server:
Quoted 1 line
./fio -S
fio: server listening on 0.0.0.0:8765
and ^C>
^^^^ there seems to be a random "and" printed?
Quoted 1 line
./fio -C localhost examples/tiobench-example
...
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon>
Assertion failed: (cmdret->opcode == cmd.opcode), function fio_net_recv_cmd, file server.c, line 187.
Program received signal SIGABRT, Aborted. [Switching to Thread 28301140 (LWP 100220/initial thread)] 0x281e7207 in kill () from /lib/libc.so.7
(gdb) bt
#0 0x281e7207 in kill () from /lib/libc.so.7 #1 0x280eb377 in raise () from /lib/libthr.so.3 #2 0x281e5d3a in abort () from /lib/libc.so.7 #3 0x281ce0c6 in __assert () from /lib/libc.so.7 #4 0x08074f8e in fio_net_recv_cmd (sk=3) at server.c:187 #5 0x08077f95 in fio_handle_clients () at client.c:779 #6 0x08051033 in exec_run () at fio.c:1779 #7 0x08051be1 in main (argc=4, argv=0xbfbfec04, envp=Error accessing memory address 0xb: Bad address.
) at fio.c:1904
(gdb)
Signature · 1 line
Bruce
Re: Fio 2.0 coming (was "Re: Recent changes")
Quoted 40 lines · Bruce Cran
Quoted 1 line · Jens Axboe
Please try current -git, it should work now.
Thanks, it's fixed.
I can crash the client by pressing ctrl-c on the server:
Quoted 1 line
./fio -S
fio: server listening on 0.0.0.0:8765
and ^C>
^^^^ there seems to be a random "and" printed?Quoted 1 line
./fio -C localhost examples/tiobench-example
...
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon>
Assertion failed: (cmdret->opcode== cmd.opcode), functionfio_net_recv_cmd, fileserver.c, line 187.Program received signal SIGABRT, Aborted. [Switching to Thread 28301140 (LWP 100220/initial thread)]
0x281e7207in kill () from/lib/libc.so.7
(gdb) bt#0 0x281e7207 in kill () from /lib/libc.so.7 #1 0x280eb377 in raise () from /lib/libthr.so.3 #2 0x281e5d3a in abort () from /lib/libc.so.7 #3 0x281ce0c6 in __assert () from /lib/libc.so.7 #4 0x08074f8e in fio_net_recv_cmd (sk=3) at server.c:187 #5 0x08077f95 in fio_handle_clients () at client.c:779 #6 0x08051033 in exec_run () at fio.c:1779 #7 0x08051be1 in main (argc=4, argv=0xbfbfec04, envp=Error accessing memory address 0xb: Bad address.
Quoted 2 lines
) at fio.c:1904
(gdb)
OK, I'll take a look at that one. You are a good tester :-)
The assert itself is a fragmented packet, where a later fragment does not match the initial opcode. Must be due to a read error not being handled correctly.
Signature · 1 line
Jens Axboe
Re: Fio 2.0 coming (was "Re: Recent changes")
Quoted 44 lines · Jens Axboe
Quoted 40 lines · Bruce Cran
Quoted 1 line · Jens Axboe
Please try current -git, it should work now.
Thanks, it's fixed.
I can crash the client by pressing ctrl-c on the server:
Quoted 1 line
./fio -S
fio: server listening on 0.0.0.0:8765
and ^C>
^^^^ there seems to be a random "and" printed?Quoted 1 line
./fio -C localhost examples/tiobench-example
...
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f3: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon> f4: Laying out IO file(s) (1 file(s) / 512MB)
<muon>
Assertion failed: (cmdret->opcode== cmd.opcode), functionfio_net_recv_cmd, fileserver.c, line 187.Program received signal SIGABRT, Aborted. [Switching to Thread 28301140 (LWP 100220/initial thread)]
0x281e7207in kill () from/lib/libc.so.7
(gdb) bt#0 0x281e7207 in kill () from /lib/libc.so.7 #1 0x280eb377 in raise () from /lib/libthr.so.3 #2 0x281e5d3a in abort () from /lib/libc.so.7 #3 0x281ce0c6 in __assert () from /lib/libc.so.7 #4 0x08074f8e in fio_net_recv_cmd (sk=3) at server.c:187 #5 0x08077f95 in fio_handle_clients () at client.c:779 #6 0x08051033 in exec_run () at fio.c:1779 #7 0x08051be1 in main (argc=4, argv=0xbfbfec04, envp=Error accessing memory address 0xb: Bad address.Quoted 2 lines
) at fio.c:1904
(gdb)
Quoted 5 lines
OK, I'll take a look at that one. You are a good tester :-)
The assert itself is a fragmented packet, where a later fragment does not match the initial opcode. Must be due to a read error not being handled correctly.
Added attempt at failing naturally when that happens, does it reproduce now?
Signature · 1 line
Jens Axboe
Re: Fio 2.0 coming (was "Re: Recent changes")
Quoted 2 lines · Jens Axboe
Added attempt at failing naturally when that happens, does it reproduce now?
It doesn't crash now - it prints a warning and exits normally:
fio: fragment opcode mismatch (6 != 7)
client: host=localhost disconnected
Signature · 1 line
Bruce Cran