Skip to contentSkip to conversation

Random fill data

DISCUSSION Fetters, Alex 1 participant 2 messages Jan 29, 2020 – Jan 30, 2020

lore
Share & tools
lore

Fetters, Alex

author
#0 lore raw

I'm having issue getting a full fill when using the default buffer fill. Specifically, every other 32-bits of fill is 0x00000000. This is similar to the issue seen here: https://www.spinics.net/lists/fio/msg07529.html The command I'm using is:

    fio --filename=test.bin --direct=1 --rw=write --bs=512 --size=512 --name=test --verify=crc32c

which produces:

    xxd test.bin
    00000000: caac 0500 0002 0000 c249 e005 0000 0000  .........I......
    00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    00000020: 0100 0000 c3e9 1fe9 42df bf59 0000 0000  ........B..Y....
    00000030: c320 0c14 0000 0000 1824 fe0e 0000 0000  . .......$......
    00000040: 83c4 0414 0000 0000 9038 c504 0000 0000  .........8......
    00000050: 12a7 7601 0000 0000 e294 4a00 0000 0000  ..v.......J.....
    00000060: 9c12 1b0f 0000 0000 53e2 d30a 0000 0000  ........S.......
    00000070: 4a1c 951c 0000 0000 8963 8f1b 0000 0000  J........c......
    00000080: 71cc 7f01 0000 0000 8ed9 7815 0000 0000  q.........x.....
    00000090: 315b df0e 0000 0000 66cb 8c1c 0000 0000  1[......f.......
    000000a0: 6cd9 6e18 0000 0000 2d5b d40e 0000 0000  l.n.....-[......
    000000b0: 65eb 6f0d 0000 0000 6c5d c41e 0000 0000  e.o.....l]......
    000000c0: ad0b 1f1b 0000 0000 7541 690c 0000 0000  ........uAi.....
    000000d0: 2e88 510a 0000 0000 0571 0616 0000 0000  ..Q......q......
    000000e0: 202e 830e 0000 0000 c465 ec16 0000 0000   ........e......
    000000f0: b80c 811d 0000 0000 9721 210d 0000 0000  .........!!.....
    00000100: 3244 d20e 0000 0000 8608 3212 0000 0000  2D........2.....
    00000110: 1081 5f13 0000 0000 22f0 b915 0000 0000  .._.....".......
    00000120: 04fe 2015 0000 0000 c09f ff03 0000 0000  .. .............
    00000130: f8f3 c71a 0000 0000 7ffe a112 0000 0000  ................
    00000140: cf5f 3d04 0000 0000 f9cb 760b 0000 0000  ._=.......v.....
    00000150: 7fb9 7e11 0000 0000 2ff7 b809 0000 0000  ..~...../.......
    00000160: e53e da04 0000 0000 dca7 c11b 0000 0000  .>..............
    00000170: fbb4 801e 0000 0000 9fb6 4d1d 0000 0000  ..........M.....
    00000180: d3d6 6e0f 0000 0000 da7a 180e 0000 0000  ..n......z......
    00000190: 5bcf dd1f 0000 0000 eb59 cd1a 0000 0000  [........Y......
    000001a0: 3d4b c90e 0000 0000 6789 dc1b 0000 0000  =K......g.......
    000001b0: 2cb1 5f06 0000 0000 2576 da07 0000 0000  ,._.....%v......
    000001c0: c4ae f907 0000 0000 d8b5 8211 0000 0000  ................
    000001d0: bb56 5d0e 0000 0000 d74a 5103 0000 0000  .V]......JQ.....
    000001e0: 5a49 3019 0000 0000 2bc9 f00c 0000 0000  ZI0.....+.......
    000001f0: 25b9 e519 0000 0000 2417 db16 0000 0000  %.......$.......

I included the verify header to show that the byte can be written to.

To me, this looks like a 32 vs 64 bit issue. Specifically, somewhere a 32-bit value is used for the fill and then the buffer is incremented by 64-bits.

fio is 32-bit.

    file /usr/bin/fio
    /usr/bin/fio: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib, for GNU/Linux 3.2.0, BuildID[sha1]=61891ac32f94190d63ea4f182e585b4a093bd8b6, stripped

As is my Kernel (ARMv7 architectures are 32-bit: https://en.wikipedia.org/wiki/ARM_architecture#Cores)

uname -s -r -m
Linux 4.14.5-92 armv7l

Digging into the fio, fill_random_buf() https://github.com/axboe/fio/blob/master/lib/rand.c#L125 is suspicious. Specifically, when the state->use32 is true, __rand() will return a number contained within 32-bits of the uint64_t. __fill_random_buf() then should increment by sizeof(int64_t). Unfortunately, that should only impact the first few values. GOLDEN_RATIO_PRIME is sufficiently large that when multiplied with <seed> in __fill_random_buf() it should quickly grow past 32-bits.

And now I'm lost. Any ideas where the holes are coming from?

I have verified that on a VM running 64-bit Linux and FIO, there are no holes.

--
Alex Fetters

author
#1 lore raw

Some debugging with gdb reveals this issue has been patched in https://github.com/axboe/fio/commit/9781c080385364d24ac18a95544484864a27611a which is included in fio-3.14 or later.

The important detail missing from the commit summary is that the fix is not limited to Windows as I encountered it in a 32-bit Linux kernel.

--
Alex Fetters

-----Original Message-----
From: Fetters, Alex
Sent: Wednesday, January 29, 2020 4:55 PM
To: '[email protected]' <[email protected]>
Subject: Random fill data

I'm having issue getting a full fill when using the default buffer fill. Specifically, every other 32-bits of fill is 0x00000000. This is similar to the issue seen here: https://www.spinics.net/lists/fio/msg07529.html The command I'm using is:

    fio --filename=test.bin --direct=1 --rw=write --bs=512 --size=512 --name=test --verify=crc32c

which produces:

    xxd test.bin
    00000000: caac 0500 0002 0000 c249 e005 0000 0000  .........I......
    00000010: 0000 0000 0000 0000 0000 0000 0000 0000  ................
    00000020: 0100 0000 c3e9 1fe9 42df bf59 0000 0000  ........B..Y....
    00000030: c320 0c14 0000 0000 1824 fe0e 0000 0000  . .......$......
    00000040: 83c4 0414 0000 0000 9038 c504 0000 0000  .........8......
    00000050: 12a7 7601 0000 0000 e294 4a00 0000 0000  ..v.......J.....
    00000060: 9c12 1b0f 0000 0000 53e2 d30a 0000 0000  ........S.......
    00000070: 4a1c 951c 0000 0000 8963 8f1b 0000 0000  J........c......
    00000080: 71cc 7f01 0000 0000 8ed9 7815 0000 0000  q.........x.....
    00000090: 315b df0e 0000 0000 66cb 8c1c 0000 0000  1[......f.......
    000000a0: 6cd9 6e18 0000 0000 2d5b d40e 0000 0000  l.n.....-[......
    000000b0: 65eb 6f0d 0000 0000 6c5d c41e 0000 0000  e.o.....l]......
    000000c0: ad0b 1f1b 0000 0000 7541 690c 0000 0000  ........uAi.....
    000000d0: 2e88 510a 0000 0000 0571 0616 0000 0000  ..Q......q......
    000000e0: 202e 830e 0000 0000 c465 ec16 0000 0000   ........e......
    000000f0: b80c 811d 0000 0000 9721 210d 0000 0000  .........!!.....
    00000100: 3244 d20e 0000 0000 8608 3212 0000 0000  2D........2.....
    00000110: 1081 5f13 0000 0000 22f0 b915 0000 0000  .._.....".......
    00000120: 04fe 2015 0000 0000 c09f ff03 0000 0000  .. .............
    00000130: f8f3 c71a 0000 0000 7ffe a112 0000 0000  ................
    00000140: cf5f 3d04 0000 0000 f9cb 760b 0000 0000  ._=.......v.....
    00000150: 7fb9 7e11 0000 0000 2ff7 b809 0000 0000  ..~...../.......
    00000160: e53e da04 0000 0000 dca7 c11b 0000 0000  .>..............
    00000170: fbb4 801e 0000 0000 9fb6 4d1d 0000 0000  ..........M.....
    00000180: d3d6 6e0f 0000 0000 da7a 180e 0000 0000  ..n......z......
    00000190: 5bcf dd1f 0000 0000 eb59 cd1a 0000 0000  [........Y......
    000001a0: 3d4b c90e 0000 0000 6789 dc1b 0000 0000  =K......g.......
    000001b0: 2cb1 5f06 0000 0000 2576 da07 0000 0000  ,._.....%v......
    000001c0: c4ae f907 0000 0000 d8b5 8211 0000 0000  ................
    000001d0: bb56 5d0e 0000 0000 d74a 5103 0000 0000  .V]......JQ.....
    000001e0: 5a49 3019 0000 0000 2bc9 f00c 0000 0000  ZI0.....+.......
    000001f0: 25b9 e519 0000 0000 2417 db16 0000 0000  %.......$.......

I included the verify header to show that the byte can be written to.

To me, this looks like a 32 vs 64 bit issue. Specifically, somewhere a 32-bit value is used for the fill and then the buffer is incremented by 64-bits.

fio is 32-bit.

    file /usr/bin/fio
    /usr/bin/fio: ELF 32-bit LSB shared object, ARM, EABI5 version 1 (SYSV), dynamically linked, interpreter /lib, for GNU/Linux 3.2.0, BuildID[sha1]=61891ac32f94190d63ea4f182e585b4a093bd8b6, stripped

As is my Kernel (ARMv7 architectures are 32-bit: https://en.wikipedia.org/wiki/ARM_architecture#Cores)

uname -s -r -m
Linux 4.14.5-92 armv7l

Digging into the fio, fill_random_buf() https://github.com/axboe/fio/blob/master/lib/rand.c#L125 is suspicious. Specifically, when the state->use32 is true, __rand() will return a number contained within 32-bits of the uint64_t. __fill_random_buf() then should increment by sizeof(int64_t). Unfortunately, that should only impact the first few values. GOLDEN_RATIO_PRIME is sufficiently large that when multiplied with <seed> in __fill_random_buf() it should quickly grow past 32-bits.

And now I'm lost. Any ideas where the holes are coming from?

I have verified that on a VM running 64-bit Linux and FIO, there are no holes.

--
Alex Fetters