Support large pattern buffers
cover Support large pattern buffers
Hello,
When testing storage platforms that have integrated compression it is useful to use specifically compressible real world data. The existing buffer_compress_percentage option doesn't work well for testing the performance of the compression algorithm portion of the stack seeing it results in random data (which can slow down many compression algorithms) followed by trivially compressible data (which can often be compressed at a higher performance than real world data).
Instead of the crude compressibility approximation when using buffer_compress_percentage, it would be useful to specify specific test data for benchmarking purposes. Using real world data or well-known test data like the Calgary Corpus would help to make the benchmarks closer to real world applications.
The existing buffer_pattern option would almost work for this, except it is limited to a maximum length of 512B and will be repeated in the IO buffer and thus can't be used for any arbitrary compression size seeing repeated data is easily compressed.
This patch set proposes allowing much larger buffer patterns, that may be loaded from a file. The first patch provides the infrastructure for passing arbitrary sized buffers in the client/server protocol (which means the server version number must be bumped), the second patch adds a method for pre-calculating the buffer size in parse_and_fill_pattern(), the third patch supports properly reading large files and the final patch enables dynamic allocation of the buffers themselves.
The buffer_pattern will still be truncated to the block size of the IO and repeated for every IO, but this should still be acceptable for testing compression in storage which tends to operate on fixed size chunks anyway.
An arbitrary limit on the pattern buffer of 128MB is still enforced to avoid issues that would crop up with the maximum limit of the client/server commands (FIO_SERVER_MAX_CMD_MB). This should be more than enough seeing it isn't useful to specify a buffer greater than the block size and blocksizes that large are not very practical.
Thanks,
Logan
--
Logan Gunthorpe (4):
cconv: Support pattern buffers of arbitrary size lib/pattern: Support NULL output buffer in parse_and_fill_pattern() lib/pattern: Support short repeated read calls when loading from file
options: Support arbitrarily long pattern buffers
cconv.c | 86 +++++++++++++++++++++++++++++++------------ client.c | 17 ++++++--- gclient.c | 12 ++++-- lib/pattern.c | 96 ++++++++++++++++++++++++++++++++++++++++-------- lib/pattern.h | 21 ++++++++--- options.c | 10 ++--- server.c | 23 +++++++----- server.h | 2 +- stat.h | 1 - thread_options.h | 12 +++--- 10 files changed, 207 insertions(+), 73 deletions(-)
base-commit: c4704c081a54160621227b42238f6e439c28fba3 --
2.30.2
PATCH 1/4 cconv: Support pattern buffers of arbitrary size
From: Logan Gunthorpe <[email protected]>
Change the thread_options_pack structure to support pattern buffers of arbitrary size by using a flexible array at the end of the the structure to store both the verify_pattern and the buffer_pattern in that order.
In this way, only the actual bytes of each pattern will be sent over the wire and patterns of an arbitrary size can be used with the packed structure.
In order to determine the required size of the structure the function thread_options_pack_size() is introduced which returns the total number of bytes required for a given thread_options instance.
The two callsites of convert_thread_options_to_net() are then converted to dynamically allocate a pdu of the appropriate size and the two callsites of convert_thread_options_to_cpu() are modified to take the size of the received data to prevent buffer overruns.
Also add specific testing of this feature in fio_test_cconv().
Seeing this changes the client/server protocol, the FIO_SERVER_VER is bumped.
--- cconv.c | 75 +++++++++++++++++++++++++++++++++--------------- client.c | 17 +++++++---- gclient.c | 12 ++++++-- server.c | 23 +++++++++------ server.h | 2 +- thread_options.h | 8 ++++-- 6 files changed, 93 insertions(+), 44 deletions(-)
cconv.c +52 -23
@@ -54,8 +54,15 @@ static void free_thread_options_to_cpu(struct thread_options *o) } } -void convert_thread_options_to_cpu (struct thread_options *o, -struct thread_options_pack *top) +size_t thread_options_pack_size (struct thread_options *o) +{ +return sizeof (struct thread_options_pack ) + o->verify_pattern_bytes + + o->buffer_pattern_bytes; +} + +int convert_thread_options_to_cpu (struct thread_options *o, +struct thread_options_pack *top, +size_t top_sz) {int i, j;
@@ -171,10 +178,17 @@ void convert_thread_options_to_cpu(struct thread_options *o, o->verify_interval =le32_to_cpu (top->verify_interval); o->verify_offset =le32_to_cpu (top->verify_offset); -memcpy (o->verify_pattern, top->verify_pattern,MAX_PATTERN_SIZE ); -memcpy (o->buffer_pattern, top->buffer_pattern,MAX_PATTERN_SIZE ); - o->verify_pattern_bytes =le32_to_cpu (top->verify_pattern_bytes); + o->buffer_pattern_bytes =le32_to_cpu (top->buffer_pattern_bytes); +if (o->verify_pattern_bytes >=MAX_PATTERN_SIZE || + o->buffer_pattern_bytes >=MAX_PATTERN_SIZE || +thread_options_pack_size (o) > top_sz) +return -EINVAL ; + +memcpy (o->verify_pattern, top->patterns, o->verify_pattern_bytes); +memcpy (o->buffer_pattern, &top->patterns[o->verify_pattern_bytes], + o->buffer_pattern_bytes); + o->verify_fatal =le32_to_cpu (top->verify_fatal); o->verify_dump =le32_to_cpu (top->verify_dump); o->verify_async =le32_to_cpu (top->verify_async);
@@ -268,7 +282,6 @@ void convert_thread_options_to_cpu(struct thread_options *o, o->zero_buffers =le32_to_cpu (top->zero_buffers); o->refill_buffers =le32_to_cpu (top->refill_buffers); o->scramble_buffers =le32_to_cpu (top->scramble_buffers); - o->buffer_pattern_bytes =le32_to_cpu (top->buffer_pattern_bytes); o->time_based =le32_to_cpu (top->time_based); o->disable_lat =le32_to_cpu (top->disable_lat); o->disable_clat =le32_to_cpu (top->disable_clat);
@@ -334,6 +347,8 @@ void convert_thread_options_to_cpu(struct thread_options *o,uint8_t verify_cpumask[FIO_TOP_STR_MAX ];uint8_t log_gz_cpumask[FIO_TOP_STR_MAX ];#endif + +return 0 ; } void convert_thread_options_to_net(struct thread_options_pack *top,
@@ -572,8 +587,9 @@ void convert_thread_options_to_net(struct thread_options_pack *top, top->max_latency[i] =__cpu_to_le64 (o->max_latency[i]); } -memcpy (top->verify_pattern, o->verify_pattern,MAX_PATTERN_SIZE ); -memcpy (top->buffer_pattern, o->buffer_pattern,MAX_PATTERN_SIZE ); +memcpy (top->patterns, o->verify_pattern, o->verify_pattern_bytes); +memcpy (&top->patterns[o->verify_pattern_bytes], o->buffer_pattern, + o->buffer_pattern_bytes); top->size =__cpu_to_le64 (o->size); top->io_size =__cpu_to_le64 (o->io_size);
@@ -620,7 +636,6 @@ void convert_thread_options_to_net(struct thread_options_pack *top,uint8_t verify_cpumask[FIO_TOP_STR_MAX ];uint8_t log_gz_cpumask[FIO_TOP_STR_MAX ];#endif - }/*
@@ -630,18 +645,32 @@ void convert_thread_options_to_net(struct thread_options_pack *top,*/ int fio_test_cconv (struct thread_options *__o) { -struct thread_options o; -struct thread_options_pack top1, top2; - -memset (&top1,0 ,sizeof (top1)); -memset (&top2,0 ,sizeof (top2)); - -convert_thread_options_to_net (&top1, __o); -memset (&o,0 ,sizeof (o)); -convert_thread_options_to_cpu (&o, &top1); -convert_thread_options_to_net (&top2, &o); - -free_thread_options_to_cpu (&o); - -return memcmp (&top1, &top2,sizeof (top1)); +struct thread_options o1 = *__o, o2; +struct thread_options_pack *top1, *top2; +size_t top_sz; +int ret; + + o1.verify_pattern_bytes =61 ; +memset (o1.verify_pattern,'V' , o1.verify_pattern_bytes); + o1.buffer_pattern_bytes =15 ; +memset (o1.buffer_pattern,'B' , o1.buffer_pattern_bytes); + + top_sz =thread_options_pack_size (&o1); + top1 =calloc (1 , top_sz); + top2 =calloc (1 , top_sz); + +convert_thread_options_to_net (top1, &o1); +memset (&o2,0 ,sizeof (o2)); + ret =convert_thread_options_to_cpu (&o2, top1, top_sz); +if (ret) + goto out; + +convert_thread_options_to_net (top2, &o2); + ret =memcmp (top1, top2, top_sz); + +out: +free_thread_options_to_cpu (&o2); +free (top2); +free (top1); +return ret; }
client.c +12 -5
@@ -922,13 +922,20 @@ int fio_clients_send_ini(const char *filename)int fio_client_update_options (struct fio_client *client,struct thread_options *o,uint64_t *tag) { -struct cmd_add_job_pdu pdu; +size_t cmd_sz = offsetof(struct cmd_add_job_pdu , top) + +thread_options_pack_size (o); +struct cmd_add_job_pdu *pdu; +int ret; - pdu.thread_number =cpu_to_le32 (client->thread_number); - pdu.groupid =cpu_to_le32 (client->groupid); -convert_thread_options_to_net (&pdu.top, o); + pdu =malloc (cmd_sz); + pdu->thread_number =cpu_to_le32 (client->thread_number); + pdu->groupid =cpu_to_le32 (client->groupid); +convert_thread_options_to_net (&pdu->top, o); -return fio_net_send_cmd (client->fd,FIO_NET_CMD_UPDATE_JOB , &pdu,sizeof (pdu), tag, &client->cmd_list); + ret =fio_net_send_cmd (client->fd,FIO_NET_CMD_UPDATE_JOB , pdu, + cmd_sz, tag, &client->cmd_list); +free (pdu); +return ret; }static void convert_io_stat (struct io_stat *dst,struct io_stat *src)
gclient.c +9 -3
@@ -553,12 +553,15 @@ static void gfio_quit_op(struct fio_client *client, struct fio_net_cmd *cmd) }static struct thread_options *gfio_client_add_job (struct gfio_client *gc, -struct thread_options_pack *top) +struct thread_options_pack *top,size_t top_sz) {struct gfio_client_options *gco; gco =calloc (1 ,sizeof (*gco)); -convert_thread_options_to_cpu (&gco->o, top); +if (convert_thread_options_to_cpu (&gco->o, top, top_sz)) { +dprint (FD_NET ,"client: failed parsing add_job command\n" ); +return NULL ; + }INIT_FLIST_HEAD (&gco->list);flist_add_tail (&gco->list, &gc->o_list); gc->o_list_nr =1 ;
@@ -577,7 +580,10 @@ static void gfio_add_job_op(struct fio_client *client, struct fio_net_cmd *cmd) p->thread_number =le32_to_cpu (p->thread_number); p->groupid =le32_to_cpu (p->groupid); - o =gfio_client_add_job (gc, &p->top); + o =gfio_client_add_job (gc, &p->top, + cmd->pdu_len - offsetof(struct cmd_add_job_pdu , top)); +if (o ==NULL ) +return ;gdk_threads_enter ();
server.c +14 -9
@@ -1082,6 +1082,7 @@ static int handle_update_job_cmd(struct fio_net_cmd *cmd)struct cmd_add_job_pdu *pdu = (struct cmd_add_job_pdu *) cmd->payload;struct thread_data *td;uint32_t tnumber; +int ret; tnumber =le32_to_cpu (pdu->thread_number);
@@ -1093,8 +1094,9 @@ static int handle_update_job_cmd(struct fio_net_cmd *cmd) } td =tnumber_to_td (tnumber); -convert_thread_options_to_cpu (&td->o, &pdu->top); -send_update_job_reply (cmd->tag,0 ); + ret =convert_thread_options_to_cpu (&td->o, &pdu->top, + cmd->pdu_len - offsetof(struct cmd_add_job_pdu , top)); +send_update_job_reply (cmd->tag, ret);return 0 ; }
@@ -2323,15 +2325,18 @@ int fio_send_iolog(struct thread_data *td, struct io_log *log, const char *name)void fio_server_send_add_job (struct thread_data *td) { -struct cmd_add_job_pdu pdu = { - .thread_number =cpu_to_le32 (td->thread_number), - .groupid =cpu_to_le32 (td->groupid), - }; +struct cmd_add_job_pdu *pdu; +size_t cmd_sz = offsetof(struct cmd_add_job_pdu , top) + +thread_options_pack_size (&td->o); -convert_thread_options_to_net (&pdu.top, &td->o); + pdu =malloc (cmd_sz); + pdu->thread_number =cpu_to_le32 (td->thread_number); + pdu->groupid =cpu_to_le32 (td->groupid); -fio_net_queue_cmd (FIO_NET_CMD_ADD_JOB , &pdu,sizeof (pdu),NULL , -SK_F_COPY ); +convert_thread_options_to_net (&pdu->top, &td->o); + +fio_net_queue_cmd (FIO_NET_CMD_ADD_JOB , pdu, cmd_sz,NULL ,SK_F_COPY ); +free (pdu); }void fio_server_send_start (struct thread_data *td)
server.h +1 -1
@@ -51,7 +51,7 @@ struct fio_net_cmd_reply { };enum { -FIO_SERVER_VER =97 , +FIO_SERVER_VER =98 ,FIO_SERVER_MAX_FRAGMENT_PDU =1024 ,FIO_SERVER_MAX_CMD_MB =2048 ,
thread_options.h +5 -3
@@ -464,7 +464,6 @@ struct thread_options_pack {uint32_t do_verify;uint32_t verify_interval;uint32_t verify_offset; -uint8_t verify_pattern[MAX_PATTERN_SIZE ];uint32_t verify_pattern_bytes;uint32_t verify_fatal;uint32_t verify_dump;
@@ -572,7 +571,6 @@ struct thread_options_pack {uint32_t zero_buffers;uint32_t refill_buffers;uint32_t scramble_buffers; -uint8_t buffer_pattern[MAX_PATTERN_SIZE ];uint32_t buffer_pattern_bytes;uint32_t compress_percentage;uint32_t compress_chunk;
@@ -699,9 +697,13 @@ struct thread_options_pack {uint32_t log_entries;uint32_t log_prio; + +uint8_t patterns[]; } __attribute__((packed)); -extern void convert_thread_options_to_cpu (struct thread_options *o,struct thread_options_pack *top); +extern int convert_thread_options_to_cpu (struct thread_options *o, +struct thread_options_pack *top,size_t top_sz); +extern size_t thread_options_pack_size (struct thread_options *o);extern void convert_thread_options_to_net (struct thread_options_pack *top,struct thread_options *);extern int fio_test_cconv (struct thread_options *);extern void options_default_fill (struct thread_options *o);
Signature · 1 line
2.30.2
Re: [PATCH 1/4] cconv: Support pattern buffers of arbitrary size
Quoted 1 line · Logan Gunthorpe
From: Logan Gunthorpe <[email protected]>
Seems I really messed up sending these patches out. Sorry for all the noise. The From email should be the same as the signed-off by. I'll fix this in v2 after collecting feedback on any code issues.
Logan
Re: [PATCH 1/4] cconv: Support pattern buffers of arbitrary size
In reply to Logan GunthorpeQuoted 27 lines · Logan Gunthorpe
From: Logan Gunthorpe <[email protected]>
Change the
thread_options_packstructure to support pattern buffers of arbitrary size by using a flexible array at the end of the the structure to store both theverify_patternand thebuffer_patternin that order.In this way, only the actual bytes of each pattern will be sent over the wire and patterns of an arbitrary size can be used with the packed structure.
In order to determine the required size of the structure the function
thread_options_pack_size()is introduced which returns the total number of bytes required for a giventhread_optionsinstance.The two callsites of
convert_thread_options_to_net()are then converted to dynamically allocate a pdu of the appropriate size and the two callsites ofconvert_thread_options_to_cpu()are modified to take the size of the received data to prevent buffer overruns.Also add specific testing of this feature in
fio_test_cconv().Seeing this changes the client/server protocol, the
FIO_SERVER_VERis bumped.Signed-off-by Logan Gunthorpe <[email protected]>---
<snip>
Quoted 21 lines
thread_options.h +0
-2@@ -464,7 +464,6 @@ struct thread_options_pack {uint32_t do_verify;uint32_t verify_interval;uint32_t verify_offset; -uint8_t verify_pattern[MAX_PATTERN_SIZE ];uint32_t verify_pattern_bytes;uint32_t verify_fatal;uint32_t verify_dump;@@ -572,7 +571,6 @@ struct thread_options_pack {uint32_t zero_buffers;uint32_t refill_buffers;uint32_t scramble_buffers; -uint8_t buffer_pattern[MAX_PATTERN_SIZE ];uint32_t buffer_pattern_bytes;uint32_t compress_percentage;uint32_t compress_chunk;
thread_options.h struct thread_options_pack { · line 701
Quoted 5 lines
thread_options.h +2
-0@@ -699,9 +697,13 @@ struct thread_options_pack {uint32_t log_entries;uint32_t log_prio; + +uint8_t patterns[];
Since there is typically a 1:1 correspondence between the members of this struct and the unpacked version, I would add a comment here noting that patterns[] includes the contents of verify_pattern and buffer_pattern from the unpacked struct.
PATCH 2/4 lib/pattern: Support NULL output buffer in parse_and_fill_pattern()
In reply to Logan GunthorpeFrom: Logan Gunthorpe <[email protected]>
Support passing a NULL output buffer to parse_and_fill_pattern(). Each formatting function simply needs to avoid accessing the buffer when it is NULL. This allows calculating the required size of the buffer before one might be allocated.
This will be useful in a subsequent patch for dynamically allocating the pattern buffers.
--- lib/pattern.c | 35 +++++++++++++++++++++++++---------- 1 file changed, 25 insertions(+), 10 deletions(-)
lib/pattern.c +25 -10
@@ -51,9 +51,17 @@ static const char *parse_file(const char *beg, char *out,if (fd <0 ) goto err_free_out; - count =read (fd, out, out_len); -if (count ==-1 ) - goto err_free_close_out; +if (out) { + count =read (fd, out, out_len); +if (count ==-1 ) + goto err_free_close_out; + }else { + count =lseek (fd,0 ,SEEK_END ); +if (count ==-1 ) + goto err_free_close_out; +if (count >= out_len) + count = out_len; + } *filled = count;close (fd);
@@ -100,7 +108,8 @@ static const char *parse_string(const char *beg, char *out,if (end - beg > out_len)return NULL ; -memcpy (out, beg, end - beg); +if (out) +memcpy (out, beg, end - beg); *filled = end - beg;/* Catch up quote */
@@ -156,12 +165,14 @@ static const char *parse_number(const char *beg, char *out, i =0 ;if (!lval) { num =0 ; - out[i] =0x00 ; +if (out) + out[i] =0x00 ; i =1 ; }else { val = (unsigned int )lval;for (; val && out_len; out_len--, i++, val >>=8 ) - out[i] = val &0xff ; +if (out) + out[i] = val &0xff ;if (val)return NULL ; }
@@ -183,7 +194,8 @@ static const char *parse_number(const char *beg, char *out,const char *fmt; fmt = (num &1 ?"%1hhx" :"%2hhx" ); -sscanf (beg, fmt, &out[i]); +if (out) +sscanf (beg, fmt, &out[i]);if (num &1 ) { num++; beg--;
@@ -251,7 +263,8 @@ static const char *parse_format(const char *in, char *out, unsigned int parsed,if (f->desc->len > out_len)return NULL ; -memset (out,'\0' , f->desc->len); +if (out) +memset (out,'\0' , f->desc->len); *filled = f->desc->len;return in + len;
@@ -262,7 +275,9 @@ static const char *parse_format(const char *in, char *out, unsigned int parsed,* numbers and pattern formats. * @in - string input * @in_len - size of the input string -* @out - output buffer where parsed result will be put +* @out - output buffer where parsed result will be put, may be NULL +* in which case this function just calculates the required +* length of the buffer * @out_len - lengths of the output buffer * @fmt_desc - array of pattern format descriptors [input] * @fmt - array of pattern formats [output]
@@ -314,7 +329,7 @@ int parse_and_fill_pattern(const char *in, unsigned int in_len,const char *beg, *end, *out_beg = out;unsigned int total =0 , fmt_rem =0 ; -if (!in || !in_len || !out || !out_len) +if (!in || !in_len || !out_len)return -EINVAL ;if (fmt_sz_out) fmt_rem = *fmt_sz_out;
Signature · 1 line
2.30.2
PATCH 3/4 lib/pattern: Support short repeated read calls when loading from file
In reply to Logan GunthorpeFrom: Logan Gunthorpe <[email protected]>
Once a pattern file can be much larger, it will be possible that kernel will return a short read while loading the file and thus may randomly only load part of the file.
Fix this by putting the read in a loop so the entire file will be read.
--- lib/pattern.c | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-)
lib/pattern.c +11 -4
@@ -32,7 +32,7 @@ static const char *parse_file(const char *beg, char *out,const char *end;char *file;int fd; -ssize_t count; +ssize_t rc, count =0 ;if (!out_len) goto err_out;
@@ -52,9 +52,16 @@ static const char *parse_file(const char *beg, char *out, goto err_free_out;if (out) { - count =read (fd, out, out_len); -if (count ==-1 ) - goto err_free_close_out; +while (1 ) { + rc =read (fd, out, out_len - count); +if (rc ==0 ) +break ; +if (rc ==-1 ) + goto err_free_close_out; + + count += rc; + out += rc; + } }else { count =lseek (fd,0 ,SEEK_END );if (count ==-1 )
Signature · 1 line
2.30.2
PATCH 4/4 options: Support arbitrarily long pattern buffers
In reply to Logan GunthorpeFrom: Logan Gunthorpe <[email protected]>
Dynamically allocate the pattern buffer to remove the 512B length restriction. To accomplish this, store a pointer instead of a fixed block of memory for the buffers in the thread_options structure. Then introduce and use the function parse_and_fill_pattern_alloc() which will calculate the approprite size of the buffer and allocate it before filling it.
The buffers will be freed, along with a number of string buffers in free_thread_options_to_cpu(). They will also be reallocated (if necessary) when receiving them over the wire with convert_thread_options_to_cpu().
This allows for specifying real world compressible data (eg. The Calgary Corpus) for the buffer_pattern option.
--- cconv.c | 11 ++++++++++ lib/pattern.c | 52 +++++++++++++++++++++++++++++++++++++++++++----- lib/pattern.h | 21 ++++++++++++++----- options.c | 10 +++++----- stat.h | 1 - thread_options.h | 4 ++-- 6 files changed, 81 insertions(+), 18 deletions(-)
cconv.c +11 -0
@@ -48,6 +48,9 @@ static void free_thread_options_to_cpu(struct thread_options *o)free (o->profile);free (o->cgroup); +free (o->verify_pattern); +free (o->buffer_pattern); +for (i =0 ; i <DDIR_RWDIR_CNT ; i++) {free (o->bssplit[i]);free (o->zone_split[i]);
@@ -185,6 +188,10 @@ int convert_thread_options_to_cpu(struct thread_options *o, thread_options_pack_size(o) > top_sz)return -EINVAL ; + o->verify_pattern =realloc (o->verify_pattern, + o->verify_pattern_bytes); + o->buffer_pattern =realloc (o->buffer_pattern, + o->buffer_pattern_bytes);memcpy (o->verify_pattern, top->patterns, o->verify_pattern_bytes);memcpy (o->buffer_pattern, &top->patterns[o->verify_pattern_bytes], o->buffer_pattern_bytes);
@@ -651,8 +658,10 @@ int fio_test_cconv(struct thread_options *__o)int ret; o1.verify_pattern_bytes =61 ; + o1.verify_pattern =malloc (o1.verify_pattern_bytes);memset (o1.verify_pattern,'V' , o1.verify_pattern_bytes); o1.buffer_pattern_bytes =15 ; + o1.buffer_pattern =malloc (o1.buffer_pattern_bytes);memset (o1.buffer_pattern,'B' , o1.buffer_pattern_bytes); top_sz =thread_options_pack_size (&o1);
@@ -672,5 +681,7 @@ out:free_thread_options_to_cpu (&o2);free (top2);free (top1); +free (o1.buffer_pattern); +free (o1.verify_pattern);return ret; }
lib/pattern.c +47 -5
64 lines not on this page · raw message
lib/pattern.h +16 -5
@@ -1,6 +1,19 @@#ifndef FIO_PARSE_PATTERN_H #define FIO_PARSE_PATTERN_H +/* +* The pattern is dynamically allocated, but that doesn't mean there +* are not limits. The network protocol has a limit of +* FIO_SERVER_MAX_CMD_MB and potentially two patterns must fit in there. +* There's also a need to verify the incoming data from the network and +* this provides a sensible check. +* +* 128MiB is an arbitrary limit that meets these criteria. The patterns +* tend to be truncated at the IO size anyway and IO sizes that large +* aren't terribly practical. +*/ +#define MAX_PATTERN_SIZE (128 << 20) +/** * Pattern format description. The input for 'parse_pattern'. * Describes format with its name and callback, which should
@@ -21,11 +34,9 @@ struct pattern_fmt {const struct pattern_fmt_desc *desc; }; -int parse_and_fill_pattern (const char *in,unsigned int in_len, -char *out,unsigned int out_len, -const struct pattern_fmt_desc *fmt_desc, -struct pattern_fmt *fmt, -unsigned int *fmt_sz_out); +int parse_and_fill_pattern_alloc (const char *in,unsigned int in_len, +char **out,const struct pattern_fmt_desc *fmt_desc, +struct pattern_fmt *fmt,unsigned int *fmt_sz_out); int paste_format_inplace(char *pattern, unsigned int pattern_len, struct pattern_fmt *fmt, unsigned int fmt_sz,
options.c +5 -5
@@ -1488,8 +1488,8 @@ static int str_buffer_pattern_cb(void *data, const char *input)int ret;/* FIXME: for now buffer pattern does not support formats */ - ret =parse_and_fill_pattern (input,strlen (input), td->o.buffer_pattern, -MAX_PATTERN_SIZE ,NULL ,NULL ,NULL ); + ret =parse_and_fill_pattern_alloc (input,strlen (input), + &td->o.buffer_pattern,NULL ,NULL ,NULL );if (ret <0 )return 1 ;
@@ -1537,9 +1537,9 @@ static int str_verify_pattern_cb(void *data, const char *input)int ret; td->o.verify_fmt_sz =FIO_ARRAY_SIZE (td->o.verify_fmt); - ret =parse_and_fill_pattern (input,strlen (input), td->o.verify_pattern, -MAX_PATTERN_SIZE , fmt_desc, - td->o.verify_fmt, &td->o.verify_fmt_sz); + ret =parse_and_fill_pattern_alloc (input,strlen (input), + &td->o.verify_pattern, fmt_desc, td->o.verify_fmt, + &td->o.verify_fmt_sz);if (ret <0 )return 1 ;
stat.h +0 -1
@@ -142,7 +142,6 @@ enum block_info_state { BLOCK_STATE_COUNT, }; -#define MAX_PATTERN_SIZE 512#define FIO_JOBNAME_SIZE 128#define FIO_JOBDESC_SIZE 256#define FIO_VERROR_SIZE 128
thread_options.h +2 -2
@@ -144,7 +144,7 @@ struct thread_options {unsigned int do_verify;unsigned int verify_interval;unsigned int verify_offset; -char verify_pattern[MAX_PATTERN_SIZE ]; +char *verify_pattern;unsigned int verify_pattern_bytes;struct pattern_fmt verify_fmt[8 ];unsigned int verify_fmt_sz;
@@ -256,7 +256,7 @@ struct thread_options {unsigned int zero_buffers;unsigned int refill_buffers;unsigned int scramble_buffers; -char buffer_pattern[MAX_PATTERN_SIZE ]; +char *buffer_pattern;unsigned int buffer_pattern_bytes;unsigned int compress_percentage;unsigned int compress_chunk;
Signature · 1 line
2.30.2
Quoted 66 lines · Logan Gunthorpe
Hello,
When testing storage platforms that have integrated compression it is useful to use specifically compressible real world data. The existing
buffer_compress_percentageoption doesn't work well for testing the performance of the compression algorithm portion of the stack seeing it results in random data (which can slow down many compression algorithms) followed by trivially compressible data (which can often be compressed at a higher performance than real world data).Instead of the crude compressibility approximation when using
buffer_compress_percentage, it would be useful to specify specific test data for benchmarking purposes. Using real world data or well-known test data like the Calgary Corpus would help to make the benchmarks closer to real world applications.The existing
buffer_patternoption would almost work for this, except it is limited to a maximum length of 512B and will be repeated in the IO buffer and thus can't be used for any arbitrary compression size seeing repeated data is easily compressed.This patch set proposes allowing much larger buffer patterns, that may be loaded from a file. The first patch provides the infrastructure for passing arbitrary sized buffers in the client/server protocol (which means the server version number must be bumped), the second patch adds a method for pre-calculating the buffer size in
parse_and_fill_pattern(), the third patch supports properly reading large files and the final patch enables dynamic allocation of the buffers themselves.The
buffer_patternwill still be truncated to the block size of the IO and repeated for every IO, but this should still be acceptable for testing compression in storage which tends to operate on fixed size chunks anyway.An arbitrary limit on the pattern buffer of 128MB is still enforced to avoid issues that would crop up with the maximum limit of the client/server commands (
FIO_SERVER_MAX_CMD_MB). This should be more than enough seeing it isn't useful to specify a buffer greater than the block size and blocksizes that large are not very practical.Thanks,
Logan
--
Logan Gunthorpe (4):
cconv: Support pattern buffers of arbitrary size lib/pattern: Support NULL output buffer inparse_and_fill_pattern()lib/pattern: Support short repeated read calls when loading from file
options: Support arbitrarily long pattern bufferscconv.c | 86 +++++++++++++++++++++++++++++++------------ client.c | 17 ++++++--- gclient.c | 12 ++++-- lib/pattern.c | 96 ++++++++++++++++++++++++++++++++++++++++-------- lib/pattern.h | 21 ++++++++--- options.c | 10 ++--- server.c | 23 +++++++----- server.h | 2 +- stat.h | 1 - thread_options.h | 12 +++--- 10 files changed, 207 insertions(+), 73 deletions(-)
Quoted 3 lines
base-commit:
c4704c081a54160621227b42238f6e439c28fba3--
2.30.2
I've looked through the patches, done some testing and see no problems apart from the small changes already mentioned.
It would be nice to have an addition to our test suite that covers this functionality.
Vincent
Quoted 5 lines · Vincent Fu
I've looked through the patches, done some testing and see no problems apart from the small changes already mentioned.
It would be nice to have an addition to our test suite that covers this functionality.
Thanks! I'll make the change you mentioned and look into writing a test for this. I'll send a v2 in the next week or so.
Logan