Skip to contentSkip to conversation

delayacct: alignment changes break iotop

PATCH In review Dan Carpenter 5 participants 12 messages Dec 13, 2010 – Dec 15, 2010 Taskstats statistics interface

lore
Share & tools
lore
#0 lore raw

Iotop uses hardcoded offsets to find the taskstats struct members. This got changed in 2.6.37 so it now iotop doesn't work on amd64. The offending commit is:

commit 85893120699f8bae8caa12a8ee18ab5fceac978e

Author: Jeff Mahoney <[email protected]>
Date:   Wed Oct 27 15:34:43 2010 -0700

delayacct: align to 8 byte boundary on 64-bit systems

Brian Rogers gets the reported-by tag. The bugzilla entry is:
https://bugzilla.kernel.org/show_bug.cgi?id=24272

regards,
dan carpenter

#1 lore raw
Quoted 4 lines · Dan Carpenter

Iotop uses hardcoded offsets to find the taskstats struct members. This got changed in 2.6.37 so it now iotop doesn't work on amd64. The offending commit is:

Quoted 9 lines

commit 85893120699f8bae8caa12a8ee18ab5fceac978e

Author: Jeff Mahoney <[email protected]>
Date:   Wed Oct 27 15:34:43 2010 -0700

delayacct: align to 8 byte boundary on 64-bit systems

Brian Rogers gets the reported-by tag. The bugzilla entry is:
https://bugzilla.kernel.org/show_bug.cgi?id=24272

Thanks for the report, looks like the change did not even bump the version field. Sorry, its my fault, I should have caught it earlier. iotop hard coding member offsets is not bad as long as we don't break ABI (expected from us). Any chance you could dump the offsets before and after the change?

Signature · 2 lines
	Three Cheers,
	Balbir
#2 lore raw
Quoted 6 lines · Balbir Singh

Thanks for the report, looks like the change did not even bump the version field. Sorry, its my fault, I should have caught it earlier. iotop hard coding member offsets is not bad as long as we don't break ABI (expected from us). Any chance you could dump the offsets before and after the change?

Here is what iotop expects from iotop/data.py

    members_offsets = [
        ('blkio_delay_total', 40),
        ('swapin_delay_total', 56),
        ('read_bytes', 248),
        ('write_bytes', 256),
        ('cancelled_write_bytes', 264)
    ]

Bumping the version doesn't help, because iotop doesn't care.

What are the warning messages that prompted the change on IA64 in the first place? Can't we just change the format for ia64 and leave the other arches how they were before?

regards,
dan carpenter

#3 lore raw
In reply to Balbir Singh

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 16 lines · Balbir Singh
Quoted 4 lines · Dan Carpenter

Iotop uses hardcoded offsets to find the taskstats struct members. This got changed in 2.6.37 so it now iotop doesn't work on amd64. The offending commit is:

Quoted 9 lines

commit 85893120699f8bae8caa12a8ee18ab5fceac978e

Author: Jeff Mahoney <[email protected]>
Date:   Wed Oct 27 15:34:43 2010 -0700

delayacct: align to 8 byte boundary on 64-bit systems

Brian Rogers gets the reported-by tag. The bugzilla entry is:
https://bugzilla.kernel.org/show_bug.cgi?id=24272

Quoted 5 lines

Thanks for the report, looks like the change did not even bump the version field. Sorry, its my fault, I should have caught it earlier. iotop hard coding member offsets is not bad as long as we don't break ABI (expected from us). Any chance you could dump the offsets before and after the change?

In your February response and again in September, you did suggest a version bump. I'm not sure why that didn't get integrated but I still don't see how it's necessary for code that actually follows the interface.

iotop doesn't. It's broken. It doesn't even honor that version field and worse yet, it doesn't even honor the packet format which specifically doesn't define hard offsets. Rather it defines a protocol that tags fields and supplies the offsets in the packet.

The getdelays.c code that ships with the kernel even demonstrates this, so there's no excuse for half-assing it like this.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iEYEARECAAYFAk0GObEACgkQLPWxlyuTD7IziQCfYMylgEurblaMRGYXEbWzcYag K9oAmQGeHsXKjcSs++Kx31UFS02ZEEUf
=2v7T
-----END PGP SIGNATURE-----

#4 lore raw
Quoted 28 lines · Jeff Mahoney

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 12 lines · Balbir Singh
Quoted 10 lines · Dan Carpenter

Iotop uses hardcoded offsets to find the taskstats struct members. This got changed in 2.6.37 so it now iotop doesn't work on amd64. The offending commit is:

commit 85893120699f8bae8caa12a8ee18ab5fceac978e

Author: Jeff Mahoney<[email protected]>
Date:   Wed Oct 27 15:34:43 2010 -0700

delayacct: align to 8 byte boundary on 64-bit systems

Quoted 8 lines
Quoted 3 lines

Brian Rogers gets the reported-by tag. The bugzilla entry is:
https://bugzilla.kernel.org/show_bug.cgi?id=24272

Thanks for the report, looks like the change did not even bump the version field. Sorry, its my fault, I should have caught it earlier. iotop hard coding member offsets is not bad as long as we don't break ABI (expected from us). Any chance you could dump the offsets before and after the change?

In your February response and again in September, you did suggest a version bump. I'm not sure why that didn't get integrated but I still don't see how it's necessary for code that actually follows the interface.

Quoted 7 lines

iotop doesn't. It's broken. It doesn't even honor that version field and worse yet, it doesn't even honor the packet format which specifically doesn't define hard offsets. Rather it defines a protocol that tags fields and supplies the offsets in the packet.

The getdelays.c code that ships with the kernel even demonstrates this, so there's no excuse for half-assing it like this.

From a cursory glance, it looks to me like iotop (mostly) does the correct thing. The taskstats struct is received as one big chunk, so the table of fixed offsets (within struct taskstats) is necessary.

There's a bug fix in the git repo that fixes the cause of misalignment:

commit 08211d209ae8fc7e67ea3bebb09979ff61c70f97

Author: Guillaume Chazarain <[email protected]>
Date:   Sat Sep 4 13:57:43 2010 +0200

Instead of assuming the pid field is 4 bytes long, take its length
from the header.
This is needed for http://lkml.org/lkml/2010/2/12/167
[PATCH] delayacct: align to 8 byte boundary on 64-bit systems

With this version of iotop, there is no problem with the latest kernel. I'm CCing the iotop author. It would be nice to have a release with this fix.

Brian

#5 lore raw

Yeah, but the rule is that we don't break userspace... :/

Fixing the bug in iotop was always going to be the easy part. It's great that they've applied a fix to git repo but it still takes a while to get the fix out to users so they don't notice a problem.

I think we're going to need to put an ifdef IA64 then size is 8 else size is 4. I just started this thread because I was hoping someone had a different ifdef to use, like maybe there are other arches which care about the alignment and maybe there is a way to test for it.

regards,
dan carpenter

#6 lore raw

PATCH 1/1 delayacct: fix iotop on x86_64

We changed how the taskstats was exported to user space in:
85893120699 "delayacct: align to 8 byte boundary on 64-bit systems" This was important because it fixes a run time warning on IA64. In theory it shouldn't have broken anything, if you just assume that user space programmers don't smoke crack all day long.

But actually it breaks iotop on x86_64.

kernel/taskstats.c +13 -1
index c8231fb..a0758de 100644
@@ -358,7 +358,19 @@ static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid)
 	 * This causes lots of runtime warnings on systems requiring 8 byte
 	 * alignment */
 	u32 pids[2] = { pid, 0 };
-	int pid_size = ALIGN(sizeof(pid), sizeof(long));
+	int pid_size;
+
+	/*
+	 * IA64 can't be aligned on a 4 byte boundary.  But iotop on x86_64
+	 * depends on the current struct layout.  The next version of iotop
+	 * will fix this so maybe we can move everything to the new code in
+	 * a couple years.
+	 */
+#if defined(CONFIG_IA64)
+	pid_size = ALIGN(sizeof(pid), sizeof(long));
+#else
+	pid_size = sizeof(u32);
+#endif
 
 	aggr = (type == TASKSTATS_TYPE_PID)
 			? TASKSTATS_TYPE_AGGR_PID
#7 lore raw

Re: [patch] delayacct: fix iotop on x86_64

Quoted 23 lines · Dan Carpenter

We changed how the taskstats was exported to user space in:
85893120699 "delayacct: align to 8 byte boundary on 64-bit systems" This was important because it fixes a run time warning on IA64. In theory it shouldn't have broken anything, if you just assume that user space programmers don't smoke crack all day long.

But actually it breaks iotop on x86_64.

kernel/taskstats.c +3 -1
index c8231fb..a0758de 100644
@@ -358,7 +358,19 @@ static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid)
 	 * This causes lots of runtime warnings on systems requiring 8 byte
 	 * alignment */
 	u32 pids[2] = { pid, 0 };
-	int pid_size = ALIGN(sizeof(pid), sizeof(long));
+	int pid_size;
+
+	/*

kernel/taskstats.c static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid) · line 373

Quoted 10 lines
kernel/taskstats.c +10 -0
+	 * IA64 can't be aligned on a 4 byte boundary.  But iotop on x86_64
+	 * depends on the current struct layout.  The next version of iotop
+	 * will fix this so maybe we can move everything to the new code in
+	 * a couple years.
+	 */
+#if defined(CONFIG_IA64)
+	pid_size = ALIGN(sizeof(pid), sizeof(long));
+#else
+	pid_size = sizeof(u32);
+#endif

I would rather abstract this better and I'd be apprehensive about the fix if iotop was at fault to begin with, I would rather fix iotop. IOW, are we fixing what iotop got wrong? Isn't it easier to backport the correct behaviour in iotop. I understand we broke the ABI, but user space can still live.

Quoted 3 lines
kernel/taskstats.c +0 -0
 	aggr = (type == TASKSTATS_TYPE_PID)
 			? TASKSTATS_TYPE_AGGR_PID
Signature · 2 lines
	Three Cheers,
	Balbir
#8 lore raw

Re: [patch] delayacct: fix iotop on x86_64

Quoted 6 lines · Balbir Singh

I would rather abstract this better and I'd be apprehensive about the fix if iotop was at fault to begin with, I would rather fix iotop. IOW, are we fixing what iotop got wrong? Isn't it easier to backport the correct behaviour in iotop. I understand we broke the ABI, but user space can still live.

The iotop people are definitely at fault and we should throw salmon at the developers next time when we see them at a conference. But in the end, it's not really a matter of assigning blame to things. It's just annoying for users if the program stops working and you have to google to figure out why the new kernel it broke iotop.

It's simple enough to paper over the bug for now, then fix it properly in a couple years when everyone has upgraded their user space.

regards,
dan carpenter

#9 lore raw

Re: [patch] delayacct: fix iotop on x86_64

In reply to Balbir Singh
Quoted 29 lines · On Tue, 14 Dec 2010 13:32:39 +0530 Balbir Singh <[email protected]>…
Quoted 27 lines · Dan Carpenter

We changed how the taskstats was exported to user space in:
85893120699 "delayacct: align to 8 byte boundary on 64-bit systems" This was important because it fixes a run time warning on IA64. In theory it shouldn't have broken anything, if you just assume that user space programmers don't smoke crack all day long.

But actually it breaks iotop on x86_64.

kernel/taskstats.c +7 -1
index c8231fb..a0758de 100644
@@ -358,7 +358,19 @@ static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid)
 	 * This causes lots of runtime warnings on systems requiring 8 byte
 	 * alignment */
 	u32 pids[2] = { pid, 0 };
-	int pid_size = ALIGN(sizeof(pid), sizeof(long));
+	int pid_size;
+
+	/*
+	 * IA64 can't be aligned on a 4 byte boundary.  But iotop on x86_64
+	 * depends on the current struct layout.  The next version of iotop
+	 * will fix this so maybe we can move everything to the new code in
+	 * a couple years.
Quoted 8 lines

kernel/taskstats.c static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid) · line 373

Quoted 6 lines
kernel/taskstats.c +6 -0
+	 */
+#if defined(CONFIG_IA64)
+	pid_size = ALIGN(sizeof(pid), sizeof(long));
+#else
+	pid_size = sizeof(u32);
+#endif

I would rather abstract this better

Well. Abstracting something tends to make it permanent. When you have an ugly, special-case temporary hack, there is merit to having it sitting there in the middle of the code staring you in the face. It's very explicit and we won't forget about it.

Quoted 5 lines

and I'd be apprehensive about the
fix if iotop was at fault to begin with, I would rather fix iotop. IOW, are we fixing what iotop got wrong? Isn't it easier to backport the correct behaviour in iotop. I understand we broke the ABI, but user space can still live.

Nah, let's not knowingly break a userspace app.

This is a versioned interface, is it not? How is that supposed to work? Should we have upped the version number when making this change?

#10 lore raw

Re: [patch] delayacct: fix iotop on x86_64

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 52 lines · Andrew Morton
Quoted 29 lines · On Tue, 14 Dec 2010 13:32:39 +0530 Balbir Singh <[email protected]>…
Quoted 27 lines · Dan Carpenter

We changed how the taskstats was exported to user space in:
85893120699 "delayacct: align to 8 byte boundary on 64-bit systems" This was important because it fixes a run time warning on IA64. In theory it shouldn't have broken anything, if you just assume that user space programmers don't smoke crack all day long.

But actually it breaks iotop on x86_64.

kernel/taskstats.c +7 -1
index c8231fb..a0758de 100644
@@ -358,7 +358,19 @@ static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid)
 	 * This causes lots of runtime warnings on systems requiring 8 byte
 	 * alignment */
 	u32 pids[2] = { pid, 0 };
-	int pid_size = ALIGN(sizeof(pid), sizeof(long));
+	int pid_size;
+
+	/*
+	 * IA64 can't be aligned on a 4 byte boundary.  But iotop on x86_64
+	 * depends on the current struct layout.  The next version of iotop
+	 * will fix this so maybe we can move everything to the new code in
+	 * a couple years.
Quoted 8 lines

kernel/taskstats.c static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid) · line 373

Quoted 6 lines
kernel/taskstats.c +6 -0
+	 */
+#if defined(CONFIG_IA64)
+	pid_size = ALIGN(sizeof(pid), sizeof(long));
+#else
+	pid_size = sizeof(u32);
+#endif

I would rather abstract this better

Well. Abstracting something tends to make it permanent. When you have an ugly, special-case temporary hack, there is merit to having it sitting there in the middle of the code staring you in the face. It's very explicit and we won't forget about it.

Quoted 5 lines

and I'd be apprehensive about the
fix if iotop was at fault to begin with, I would rather fix iotop. IOW, are we fixing what iotop got wrong? Isn't it easier to backport the correct behaviour in iotop. I understand we broke the ABI, but user space can still live.

Quoted 6 lines

Nah, let's not knowingly break a userspace app.

This is a versioned interface, is it not? How is that supposed to work? Should we have upped the version number when making this change?

Perhaps. Balbir suggested it, but it didn't make it into the final version. Not that it would have mattered. iotop doesn't look at the version anyway.

- -Jeff

- --
Jeff Mahoney
SUSE Labs
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.16 (GNU/Linux)
Comment: Using GnuPG with SUSE - http://enigmail.mozdev.org/

iEYEARECAAYFAk0H0bUACgkQLPWxlyuTD7LJlwCeKLRuVKXIwZi7XMARDNXmBxkj QC0An0up3AVv/G8T8JZbb+cpDFagKnj0
=ra4a
-----END PGP SIGNATURE-----

#11 lore raw

Re: [patch] delayacct: fix iotop on x86_64

In reply to Andrew Morton
Quoted 41 lines · Andrew Morton
Quoted 29 lines · On Tue, 14 Dec 2010 13:32:39 +0530 Balbir Singh <[email protected]>…
Quoted 27 lines · Dan Carpenter

We changed how the taskstats was exported to user space in:
85893120699 "delayacct: align to 8 byte boundary on 64-bit systems" This was important because it fixes a run time warning on IA64. In theory it shouldn't have broken anything, if you just assume that user space programmers don't smoke crack all day long.

But actually it breaks iotop on x86_64.

kernel/taskstats.c +7 -1
index c8231fb..a0758de 100644
@@ -358,7 +358,19 @@ static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid)
 	 * This causes lots of runtime warnings on systems requiring 8 byte
 	 * alignment */
 	u32 pids[2] = { pid, 0 };
-	int pid_size = ALIGN(sizeof(pid), sizeof(long));
+	int pid_size;
+
+	/*
+	 * IA64 can't be aligned on a 4 byte boundary.  But iotop on x86_64
+	 * depends on the current struct layout.  The next version of iotop
+	 * will fix this so maybe we can move everything to the new code in
+	 * a couple years.
Quoted 8 lines

kernel/taskstats.c static struct taskstats *mk_reply(struct sk_buff *skb, int type, u32 pid) · line 373

Quoted 6 lines
kernel/taskstats.c +6 -0
+	 */
+#if defined(CONFIG_IA64)
+	pid_size = ALIGN(sizeof(pid), sizeof(long));
+#else
+	pid_size = sizeof(u32);
+#endif

I would rather abstract this better

Quoted 5 lines

Well. Abstracting something tends to make it permanent. When you have an ugly, special-case temporary hack, there is merit to having it sitting there in the middle of the code staring you in the face. It's very explicit and we won't forget about it.

OK, agreed and learnt

Quoted 8 lines
Quoted 5 lines

and I'd be apprehensive about the
fix if iotop was at fault to begin with, I would rather fix iotop. IOW, are we fixing what iotop got wrong? Isn't it easier to backport the correct behaviour in iotop. I understand we broke the ABI, but user space can still live.

Nah, let's not knowingly break a userspace app.

Fair enough!

Quoted 4 lines

This is a versioned interface, is it not? How is that supposed to work? Should we have upped the version number when making this change?

Signature · 2 lines
	Three Cheers,
	Balbir