An interesting cascade of failures in ptest
I was recently asked why an strace ptest build/testimage was going slowly. I've looked into it and it is perhaps worth documenting the cascades of failures that were causing problems.
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15651
The first step is strace creating a 43GB sparse file in it's test directory.
There is nothing wrong with that as such, until an strace ptest failure happens. We then tar up the directory as part of the image post processing and archive it for debugging.
Taring a 43GB sparse file with gzip is not efficient at all, it will sit using 100% of a CPU under qemu+kvm for over an hour, I gave up at that point.
The data is transferred over ssh to the host and each chunk of data is printed into the task log file (as binary). The 187MB of data ends up as 1.4GB tar.gz, so that is over 1.4GB of data dumped into the logfile as binary. The code uses string concatenation to do this which is slow in python.
Worse, the logged data is also set over the worker IPC mechanism to the bitbake server. This is done in 64kb (pipesize) chunks, with a memory mmap remap being done between each chunk due to recreation of the bytearray.
In summary, the time taken to do this is around 2 days. We did let the build finish as I was curious to see if it would!
There are multiple patches being sent to try and address this:
* switch qemu images to include GNU tar
* use --sparse option to tar
* increase bitbake's pipesize
* avoid many of the bytearray remap calls
* don't print large/binary data into the logfiles
* use bytearray in the ssh code
So hopefully we get some general performance boosts all around.
Cheers,
Richard