Two issues that prevent process accounting (taskstats) from working correctly
Hello Ingo, Peter, Nicolas, Andrew, Balbir, Shailabh, Jay, Gerlof and Marc,
starting from a Debian bug report of mine, Gerlof Langeveld, developer of system and process monitor atop¹, found two issues with process accounting.
I did some guess work on who might be the maintainer for this, but please feel free to add further Cc´s as you see fit. Or ask for removal for Cc if you are not working on this anymore.
Gerlof found two issues which I also reported to the kernel bug tracker. I copy and paste the summaries that Gerlof prepared:
1) Sometimes process accounting does not work at all.
The acct() system call (to activate process accounting) return value 0, which means that process accounting is activated successfully. However, no process accounting records are written whatsoever. This situation can be reproduced with the program 'acctdemo.c'
that you can find as attachment. When this program gives the message "found a process accounting record!", the situation is okay
and process accounting works fine to the file '/tmp/mypacct'. When the message 'No process accounting record yet....' is repeatedly given, process accounting does not work and will not work at all. It might be that you have to start this program several times before you get this situation (preferably start/finish lots of processes in the mean time). This problem is probably caused by a new mechanism introduced in the kernel code (..../linux/kernel/acct.c) that is called 'slow accounting' and has to be solved in the kernel code.
I experience this problem on Debian8 with a 4.8 kernel and on CentOS7 with a 4.8 kernel.
I reported this as:
Bug 190271 - process accounting sometimes does not work
https://bugzilla.kernel.org/show_bug.cgi?id=190271
2) When using the NETLINK inface, the command TASKSTATS_CMD_GET consequently returns -EINVAL.
The code that is used by the atopacctd daemon is based on the demo code 'getdelays.c' that can be found in the kernel source code tree (..../linux/Documentation/accounting/getdelays.c). Also this 'getdelays' program does not work any more (also -EINVAL on the same call) with the newer kernels. I really spent a lot of time on this issue to get the code running (there are many places in the kernel code where -EINVAL for this call can be given), but I did not succeed. It is really an incompatibility introduced by the kernel code.
It would be nice if the kernel maintainers provide a working version of the getdelays program in the kernel source tree.
I only experience this problem on Debian8 with a 4.8 kernel (virtual machine with 4 cores).
On CentOS7 with a 4.8 kernel it works fine (physical machine with 4 cores).
I will anyhow adapt atopacctd for this issue that it detects and logs the -EINVAL and terminates.
The current version of atopacctd keeps running which is not useful at all.
I reported this as:
Bug 190711 - Process accounting: Using the NETLINK inface, the command TASKSTATS_CMD_GET returns -EINVAL
https://bugzilla.kernel.org/show_bug.cgi?id=190711
Marc Haber, maintainer of atop package, Gerlof Langeveld, developer of atop and I are currently discussing workarounds with atop and/or systemd service fail for the time till upstream kernels with this issues fixed are shipped by distributions. Still it would be nice to remove those work-arounds and have the kernel work correctly again at some time in the future.
Thanks,
Signature · 14 lines
Martin Steigerwald | Trainer teamix GmbH Südwestpark 43 90449 Nürnberg Tel.: +49 911 30999 55 | Fax: +49 911 30999 99 mail: [email protected] | web: http://www.teamix.de | blog: http://blog.teamix.de Amtsgericht Nürnberg, HRB 18320 | Geschäftsführer: Oliver Kügow, Richard Müller teamix Support Hotline: +49 911 30999-112 *** Bitte liken Sie uns auf Facebook: facebook.com/teamix ***
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <stdio.h>
#include <unistd.h>
#include <stdlib.h>
#define ACCTFILE "/tmp/mypacct"
main()
{
int fd;
char buf[1024];
if ( (fd = open(ACCTFILE, O_RDWR|O_CREAT|O_TRUNC, 0777)) == -1)
{
perror("Open " ACCTFILE);
exit(1);
}
if (acct(ACCTFILE) == -1)
{
perror("Switch on accounting");
exit(1);
}
if ( fork() == 0 ) // fork new process
exit(0); // child process: finish
// parent process:
// wait for child to finish
wait((int *)0);
// read the process accounting record of the finished child
while (read(fd, buf, sizeof buf) == 0)
{
printf("No process accounting record yet....\n");
sleep(1);
}
printf("Yeeeeah, found a process accounting record!\n");
}Quoted 5 lines · Martin Steigerwald
Hello Ingo, Peter, Nicolas, Andrew, Balbir, Shailabh, Jay, Gerlof and Marc,
starting from a Debian bug report of mine, Gerlof Langeveld, developer of system and process monitor atop¹, found two issues with process accounting.
While $subject says regression the rest of the email completely fails to mentions if this ever worked, and if so, against what version.
Am Montag, 19. Dezember 2016, 14:19:11 CET schrieb Peter Zijlstra:
Quoted 10 lines · Martin Steigerwald
Quoted 6 lines
Hello Ingo, Peter, Nicolas, Andrew, Balbir, Shailabh, Jay, Gerlof and Marc,
starting from a Debian bug report of mine, Gerlof Langeveld, developer of system and process monitor atop¹, found two issues with process accounting.
While $subject says regression the rest of the email completely fails to mentions if this ever worked, and if so, against what version.
Sorry for not mentioning it here. With kernel 3.16 process accounting worked nicely. We do not have any data on the exact kernel version where it started to fail. I do see these issues since quite a while, but I was never sure whether it was an issue in atop or the kernel and I do not recall on what exact combination of atop + kernel version it failed to work.
As an additional notice: The mail addresses of Balbir and Shailabh which I got from source code annotations of taskstats.c and getdelays.c do not work anymore.
Signature · 1 line
Martin
Quoted 18 lines · Martin Steigerwald
Am Montag, 19. Dezember 2016, 14:19:11 CET schrieb Peter Zijlstra:
Quoted 10 lines · Martin Steigerwald
Quoted 6 lines
Hello Ingo, Peter, Nicolas, Andrew, Balbir, Shailabh, Jay, Gerlof and Marc,
starting from a Debian bug report of mine, Gerlof Langeveld, developer of system and process monitor atop¹, found two issues with process accounting.
While $subject says regression the rest of the email completely fails to mentions if this ever worked, and if so, against what version.
Sorry for not mentioning it here. With kernel 3.16 process accounting worked nicely. We do not have any data on the exact kernel version where it started to fail. I do see these issues since quite a while, but I was never sure whether it was an issue in atop or the kernel and I do not recall on what exact combination of atop + kernel version it failed to work.
Quoted 3 lines
As an additional notice: The mail addresses of Balbir and Shailabh which I got from source code annotations of
taskstats.candgetdelays.cdo not work anymore.
Right, Balbir is back with IBM and working on Linux again, but under a different email address (and on a different continent). I've amended his address and bounced him the initial email.
Quoted 24 lines · Martin Steigerwald
2) When using the NETLINK inface, the command
TASKSTATS_CMD_GETconsequently returns-EINVAL.The code that is used by the atopacctd daemon is based on the demo code '
getdelays.c' that can be found in the kernel source code tree (..../linux/Documentation/accounting/getdelays.c). Also this 'getdelays' program does not work any more (also-EINVALon the same call) with the newer kernels. I really spent a lot of time on this issue to get the code running (there are many places in the kernel code where-EINVALfor this call can be given), but I did not succeed. It is really an incompatibility introduced by the kernel code.
It would be nice if the kernel maintainers provide a working version of the getdelays program in the kernel source tree.I only experience this problem on Debian8 with a 4.8 kernel (virtual machine with 4 cores).
On CentOS7 with a 4.8 kernel it works fine (physical machine with 4 cores).I will anyhow adapt atopacctd for this issue that it detects and logs the
-EINVALand terminates.
The current version of atopacctd keeps running which is not useful at all.
Quoted 6 lines
I reported this as:
Bug 190711 - Process accounting: Using the NETLINK inface, the command
TASKSTATS_CMD_GETreturns-EINVAL
Ben Hutchings reports:
"It looks like the taskstats bug was introduced by 513e3d2d11c9 as that means cpumask_parse() may not initialise as many bits as cpumask_subset() compares"
Am Montag, 19. Dezember 2016, 18:25:49 CET schrieb Peter Zijlstra:
Quoted 30 lines · Martin Steigerwald
Quoted 29 lines
2) When using the NETLINK inface, the command
TASKSTATS_CMD_GETconsequently returns-EINVAL.The code that is used by the atopacctd daemon is based on the demo code '
getdelays.c' that can be found in the kernel source code tree (..../linux/Documentation/accounting/getdelays.c). Also this 'getdelays' program does not work any more (also-EINVALon the same call) with the newer kernels. I really spent a lot of time on this issue to get the code running (there are many places in the kernel code where-EINVALfor this call can be given), but I did not succeed. It is really an incompatibility introduced by the kernel code.
It would be nice if the kernel maintainers provide a working version of the getdelays program in the kernel source tree.I only experience this problem on Debian8 with a 4.8 kernel (virtual machine with 4 cores).
On CentOS7 with a 4.8 kernel it works fine (physical machine with 4 cores).I will anyhow adapt atopacctd for this issue that it detects and logs the
-EINVALand terminates.
The current version of atopacctd keeps running which is not useful at all.I reported this as:
Bug 190711 - Process accounting: Using the NETLINK inface, the command
TASKSTATS_CMD_GETreturns-EINVAL
Quoted 7 lines
Quoted 1 line
Ben Hutchings reports:
"It looks like the taskstats bug was introduced by
513e3d2d11c9as that meanscpumask_parse()may not initialise as many bits ascpumask_subset()compares"
Thank you.
Okay, any conclusion out of this? Any feedback from the maintainers of this code?
Thank you,
Hello,
Quoted 5 lines · Martin Steigerwald
Quoted 5 lines
Ben Hutchings reports:
"It looks like the taskstats bug was introduced by
513e3d2d11c9as that meanscpumask_parse()may not initialise as many bits ascpumask_subset()compares"
I see, so that patch switched parsing and printing to always use nr_cpu_ids but left the comparison functions to keep using nr_cpumask_bits which may be NR_CPUS instead of nr_cpu_ids.
Quoted 2 lines
Okay, any conclusion out of this? Any feedback from the maintainers of this code?
We can switch back the parse functions to nr_cpumask_bits, or just get rid of nr_cpumask_bits and use nr_cpu_ids everywhere. The only reason we use nr_cpumask_bits is because on small configurations the constant NR_CPUS can be more efficient than having to read out nr_cpu_ids variable each time.
Hmm... I'll restore the parse functions to use nr_cpumask_bits instead for now. The confusing part was the output results. Input should be okay even if we flip between nr_cpu_ids and NR_CPUS.
Thanks.
Signature · 1 line
tejun
PATCH 1/1 cpumask: use nr_cpumask_bits for parsing functions
513e3d2d11c9 ("cpumask: always use nr_cpu_ids in formatting and parsing functions") converted both cpumask printing and parsing functions to use nr_cpu_ids instead of nr_cpumask_bits. While this was okay for the printing functions as it just picked one of the two output formats that we were alternating between depending on a kernel config, doing the same for parsing wasn't okay.
nr_cpumask_bits can be either nr_cpu_ids or NR_CPUS. We can always use nr_cpu_ids but that is a variable while NR_CPUS is a constant, so it can be more efficient to use NR_CPUS when we can get away with it. Converting the printing functions to nr_cpu_ids makes sense because it affects how the masks get presented to userspace and doesn't break anything; however, using nr_cpu_ids for parsing functions can incorrectly leave the higher bits uninitialized while reading in these masks from userland. As all testing and comparison functions use nr_cpumask_bits which can be larger than nr_cpu_ids, the parsed cpumasks can erroneously yield false negative results.
This made the taskstats interface incorrectly return -EINVAL even when the inputs were correct.
Fix it by restoring the parse functions to use nr_cpumask_bits instead of nr_cpu_ids.
--- include/linux/cpumask.h | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-)
include/linux/cpumask.h +4 -4
@@ -560,7 +560,7 @@ static inline void cpumask_copy(struct cpumask *dstp,static inline int cpumask_parse_user (const char __user *buf,int len,struct cpumask *dstp) { -return bitmap_parse_user (buf, len,cpumask_bits (dstp), nr_cpu_ids); +return bitmap_parse_user (buf, len,cpumask_bits (dstp), nr_cpumask_bits); }/**
@@ -575,7 +575,7 @@ static inline int cpumask_parselist_user(const char __user *buf, int len, struct cpumask *dstp) {return bitmap_parselist_user (buf, len,cpumask_bits (dstp), - nr_cpu_ids); + nr_cpumask_bits); }/**
@@ -590,7 +590,7 @@ static inline int cpumask_parse(const char *buf, struct cpumask *dstp)char *nl =strchr (buf,'\n' );unsigned int len = nl ? (unsigned int )(nl - buf) :strlen (buf); -return bitmap_parse (buf, len,cpumask_bits (dstp), nr_cpu_ids); +return bitmap_parse (buf, len,cpumask_bits (dstp), nr_cpumask_bits); }/**
@@ -602,7 +602,7 @@ static inline int cpumask_parse(const char *buf, struct cpumask *dstp)*/ static inline int cpulist_parse (const char *buf,struct cpumask *dstp) { -return bitmap_parselist (buf,cpumask_bits (dstp), nr_cpu_ids); +return bitmap_parselist (buf,cpumask_bits (dstp), nr_cpumask_bits); }/**