Add environment-variable substitution to config options

Hi,

I have a nasty bash script that performs environment variable substitution for running automated benchmarks. This seems like a reasonable feature for FIO... so here it is.

The code is quite perilous, and hasn't been particularly well tested.

Thanks,
-- Aaron

---

Strings of the form ${VARNAME} in config option strings are substituted with the value of the environment variable VARNAME. Only the right hand side of an option assignment undergoes substitution. If VARNAME is empty or undefined, the empty string is substituted.

---
parse.c | 52 +++++++++++++++++++++++++++++++++++++++++++++++++++-
parse.h | 1 +

2 files changed, 52 insertions(+), 1 deletions(-)

index b946690..456e3ba 100644
@@ -8,6 +8,7 @@

#include <string.h>
#include <errno.h>
#include <limits.h>
+#include <stdlib.h>

#include "parse.h"
#include "debug.h"
@@ -529,12 +530,61 @@ int parse_cmd_option(const char *opt, const char *val,

	return 1;
}

+/*
+ * Return a copy of the input string with substrings of the form ${VARNAME}
+ * substituted with the value of the environment variable VARNAME. The
+ * substitution always occurs, even if VARNAME is empty or the corresponding
+ * environment variable undefined.
+ */
+static char *option_dup_subs(const char *opt)
+{
+ char out[OPT_LEN_MAX+1];
+ char in[OPT_LEN_MAX+1];
+ char *outptr = out;
+ char *inptr = in;
+ char *ch1, *ch2, *env;
+ ssize_t nchr = OPT_LEN_MAX;
+ size_t envlen;
+
+ in[OPT_LEN_MAX] = '\0';
+ strncpy(in, opt, OPT_LEN_MAX);
+
+ while (*inptr && nchr > 0) {
+ if (inptr[0] == '$' && inptr[1] == '{') {
+ ch2 = strchr(inptr, '}');
+ if (ch2 && inptr+1 < ch2) {
+ ch1 = inptr+2;
+ inptr = ch2+1;
+ *ch2 = '\0';
+
+ env = getenv(ch1);
+ if (env) {
+ envlen = strlen(env);
+ if (envlen <= nchr) {
+ memcpy(outptr, env, envlen);
+ outptr += envlen;
+ nchr -= envlen;
+ }
+ }
+
+ continue;
+ }
+ }
+
+ *outptr++ = *inptr++;
+ --nchr;
+ }
+
+ *outptr = '\0';
+ return strdup(out);
+}
+
int parse_option(const char *opt, struct fio_option *options, void *data)

{
	struct fio_option *o;
	char *post, *tmp;

- tmp = strdup(opt);
+ tmp = option_dup_subs(opt);

	o = get_option(tmp, options, &post);
	if (!o) {

index 4f3f94d..4546259 100644
@@ -26,6 +26,7 @@ struct value_pair {

const char *help; /* help text for sub option */
};

+#define OPT_LEN_MAX 1024
#define PARSE_MAX_VP 16

/*

I'm fine with including this, but could you also please provide an example (preferably in HOWTO/README)? Thanks :)

Jens Axboe

Add environment variable expansion documentation to HOWTO, and fix a few nearby typo's while we're at it.

---
HOWTO | 31 +++++++++++++++++++++++++++++--

1 files changed, 29 insertions(+), 2 deletions(-)

index 178bc98..129926b 100644
@@ -111,7 +111,7 @@ several global sections if so desired. A job is only affected by a global

section residing above it. If the first character in a line is a ';' or a '#', the entire line is discarded as a comment.

-So lets look at a really simple job file that define to threads, each +So let's look at a really simple job file that defines two processes, each randomly reading from a 128MiB file.

; -- start job file --
@@ -133,7 +133,7 @@ line, this job would look as follows:

$ fio --name=global --rw=randread --size=128m --name=job1 --name=job2

-Lets look at an example that have a number of processes writing randomly +Let's look at an example that has a number of processes writing randomly to files.

; -- start job file --
@@ -158,6 +158,33 @@ specify:

$ fio --name=random-writers --ioengine=libaio --iodepth=4 --rw=randwrite --bs=32k --direct=0 --size=64m --numjobs=4

+fio also supports environment variable expansion in job files. Any +substring of the form "${VARNAME}" as part of an option value (in other +words, on the right of the `='), will be expanded to the value of the +environment variable called VARNAME. If no such environment variable +is defined, or VARNAME is the empty string, the empty string will be +substituted.
+
+As an example, let's look at a sample fio invocation and job file:
+
+$ SIZE=64m NUMJOBS=4 fio jobfile.fio
+
+; -- start job file --
+[random-writers]
+rw=randwrite
+size=${SIZE}
+numjobs=${NUMJOBS}
+; -- end job file --
+
+This will expand to the following equivalent job file at runtime:
+
+; -- start job file --
+[random-writers]
+rw=randwrite
+size=64m
+numjobs=4
+; -- end job file --
+
fio ships with a few example job files, you can also look there for inspiration.

1.6.0.2

Super, thanks! I've applied this and the original patch.

Jens Axboe