Skip to contentSkip to conversation

New: poor performance while preprocessing source code

Bugme-new Andrew Morton 4 participants 7 messages Jan 28, 2009 – Jan 29, 2009 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore

A message that is not in the archive (sent off-list or not yet received): [email protected]/

#1 lore raw

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 4 lines · On Wed, 28 Jan 2009 11:29:52 -0800 (PST)…

http://bugzilla.kernel.org/show_bug.cgi?id=12564

Summary: poor performance while preprocessing source code
Product: IO/Storage

Thanks for the report.

Quoted 13 lines
           Version: 2.5
     KernelVersion: 2.6.28.2
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Other
        AssignedTo: io_other-ztI5WcYan/[email protected]
        ReportedBy: [email protected]
Quoted 2 lines

Latest working kernel version: 2.6.26-rc3
Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)

(huge performance regression in NFS)

Quoted 23 lines

Distribution: gentoo

Hardware Environment:
Various 32 and 64 bit Intel and AMD machines with various
PATA and SATA disks and various network interfaces.

Problem Description:
Here's my situation. I've recently upgraded the kernels
on ~30 computers at work (from 2.6.21 to 2.6.27). These computers are used to build and test software we develop. We speed up the building process using distcc. However, after the kernel upgrade, the builds are much much slower. The preprocessing stage seems to be at least 10 times slower.
As evidence of this slowdown I am attaching two images created
using distccmon-gnome. Both snapshots were taken shortly after starting builds in a clean sandbox. The only difference is the kernel. "fast.png" was generated while running kernel 2.6.25.20. "slow.png" was generated with 2.6.26. The light purple sections indicate the preprocessing times for each file.
This slowdown is observed on both 32 and 64 bit computers
and using either gcc or the intel compiler. (The intel compiler

Quoted 8 lines

builds do not use distcc, but that are also slower.) Strangely enough, it's still faster to use an NFS mounted sandbox on a machine with an older kernel than the same sandbox on the local machine with a newer kernel. (This suggests to me that it
is neither a disk or network IO problem.)
I've used git bisect to narrow it down to a single commit:
# bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' and/or 'actimeo=0' turn off attribute caching

And thanks for bisecting it.

Quoted 9 lines

This is the first commit after v2.6.26-rc3. I'm not an experienced git user, so I don't know how to narrow it down further. Distccmon-gnome snapshots from the bisection process are similar to the ones noted above.
Naturally, I would like the newer kernels to have similar
performance to the older kernels.
I will be attaching various items. Let me know what other
information might be helpful.

Quoted 6 lines

Steps to reproduce:
distccmon-gnome &

# using a makefile setup to use distcc:
make -j 5 all
# note preprocessing times in distccmon

Something I don't understand from this:

If your normal setup is using distcc then what role does NFS have to play? I mean, distcc kind-of replaces NFS in this workload.

Perhaps you could briefly describe the topology/data-flow/etc so that we can see how NFS could be a bottleneck here?

Thanks.

#2 lore raw
Quoted 64 lines · On Wed, 28 Jan 2009 13:29:42 -0800 Andrew Morton <[email protected]>…

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 4 lines · On Wed, 28 Jan 2009 11:29:52 -0800 (PST)…

http://bugzilla.kernel.org/show_bug.cgi?id=12564

Summary: poor performance while preprocessing source code
Product: IO/Storage

Thanks for the report.

Quoted 13 lines
           Version: 2.5
     KernelVersion: 2.6.28.2
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Other
        AssignedTo: io_other-ztI5WcYan/[email protected]
        ReportedBy: [email protected]
Quoted 2 lines

Latest working kernel version: 2.6.26-rc3
Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)

(huge performance regression in NFS)

Quoted 23 lines

Distribution: gentoo

Hardware Environment:
Various 32 and 64 bit Intel and AMD machines with various
PATA and SATA disks and various network interfaces.

Problem Description:
Here's my situation. I've recently upgraded the kernels
on ~30 computers at work (from 2.6.21 to 2.6.27). These computers are used to build and test software we develop. We speed up the building process using distcc. However, after the kernel upgrade, the builds are much much slower. The preprocessing stage seems to be at least 10 times slower.
As evidence of this slowdown I am attaching two images created
using distccmon-gnome. Both snapshots were taken shortly after starting builds in a clean sandbox. The only difference is the kernel. "fast.png" was generated while running kernel 2.6.25.20. "slow.png" was generated with 2.6.26. The light purple sections indicate the preprocessing times for each file.
This slowdown is observed on both 32 and 64 bit computers
and using either gcc or the intel compiler. (The intel compiler

Quoted 8 lines

builds do not use distcc, but that are also slower.) Strangely enough, it's still faster to use an NFS mounted sandbox on a machine with an older kernel than the same sandbox on the local machine with a newer kernel. (This suggests to me that it
is neither a disk or network IO problem.)
I've used git bisect to narrow it down to a single commit:
# bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' and/or 'actimeo=0' turn off attribute caching

Quoted 2 lines

And thanks for bisecting it.

It's a pretty small patch. The obvious question is: "What mount options are you using on your NFS mounts?"

Quoted 30 lines
Quoted 9 lines

This is the first commit after v2.6.26-rc3. I'm not an experienced git user, so I don't know how to narrow it down further. Distccmon-gnome snapshots from the bisection process are similar to the ones noted above.
Naturally, I would like the newer kernels to have similar
performance to the older kernels.
I will be attaching various items. Let me know what other
information might be helpful.

Quoted 6 lines

Steps to reproduce:
distccmon-gnome &

# using a makefile setup to use distcc:
make -j 5 all
# note preprocessing times in distccmon

Something I don't understand from this:

If your normal setup is using distcc then what role does NFS have to play? I mean, distcc kind-of replaces NFS in this workload.

Perhaps you could briefly describe the topology/data-flow/etc so that we can see how NFS could be a bottleneck here?

Thanks.
--
To unsubscribe from this list: send the line "unsubscribe linux-nfs" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Signature · 1 line
Jeff Layton <[email protected]>
#3 lore raw
Quoted 5 lines · Jeff Layton

It's a pretty small patch. The obvious question is: "What mount options
are you using on your NFS mounts?"

Well, I didn't think that I was using any NFS mounts.
But, it turns out I was in a way. Apparently, amd does its top level mounts with noac. So, where this (using an automounted path):
cd /h/spo/work
make -j 5
would be slow. This (using a non-automounted path):
cd /home/spo/work
make -j 5
showed the faster performance I was expecting.
I will mark my bug as invalid and decide what, if anything,
I want to do about amd.
Thanks for pointing me in the right direction.

Steven

#4 lore raw
Quoted 7 lines · On Wed, 28 Jan 2009 17:02:36 -0800 Steven Patrick…
Quoted 5 lines · Jeff Layton

It's a pretty small patch. The obvious question is: "What mount options
are you using on your NFS mounts?"

Quoted 13 lines

Well, I didn't think that I was using any NFS mounts.
But, it turns out I was in a way. Apparently, amd does its top level mounts with noac. So, where this (using an automounted path):
cd /h/spo/work
make -j 5
would be slow. This (using a non-automounted path):
cd /home/spo/work
make -j 5
showed the faster performance I was expecting.
I will mark my bug as invalid and decide what, if anything,
I want to do about amd.
Thanks for pointing me in the right direction.

This had me all mystified until I clicked on the bugzilla link:

: ------- Comment #9 From Trond Myklebust 2009-01-28 13:58:50 [reply] ------- :
: If you are worried about performance, why are you using noac/actimeo=0? :
: The commit you point to fixed a bug in which the noac/actimeo=0 was : using cached metadata in situations where it should have been
: retrieving the data from the server. We fully expect a significant : performance drop when the client is forced to send more GETATTR
: requests to the server, and as I said, that was precisely the point of : this fix.

Fair enough.

However. We surprised one user (yourself), and we will presumably surprise others. Or, worse, we will slow down people's stuff and they won't even notice.

Should we perhaps warn people about this? A mount-time printk telling them that moac/actime=0 is slow?

#5 lore raw
Quoted 42 lines · Andrew Morton
Quoted 7 lines · On Wed, 28 Jan 2009 17:02:36 -0800 Steven Patrick…
Quoted 5 lines · Jeff Layton

It's a pretty small patch. The obvious question is: "What mount options
are you using on your NFS mounts?"

Quoted 13 lines

Well, I didn't think that I was using any NFS mounts.
But, it turns out I was in a way. Apparently, amd does its top level mounts with noac. So, where this (using an automounted path):
cd /h/spo/work
make -j 5
would be slow. This (using a non-automounted path):
cd /home/spo/work
make -j 5
showed the faster performance I was expecting.
I will mark my bug as invalid and decide what, if anything,
I want to do about amd.
Thanks for pointing me in the right direction.

This had me all mystified until I clicked on the bugzilla link:

: ------- Comment #9 From Trond Myklebust 2009-01-28 13:58:50 [reply] ------- :
: If you are worried about performance, why are you using noac/actimeo=0? :
: The commit you point to fixed a bug in which the noac/actimeo=0 was : using cached metadata in situations where it should have been
: retrieving the data from the server. We fully expect a significant : performance drop when the client is forced to send more GETATTR
: requests to the server, and as I said, that was precisely the point of : this fix.

Fair enough.

Quoted 6 lines

However. We surprised one user (yourself), and we will presumably surprise others. Or, worse, we will slow down people's stuff and they won't even notice.

Should we perhaps warn people about this? A mount-time printk telling them that moac/actime=0 is slow?

No. The current nfs manpages (dated 2 November 2007) have a whole section on data and metadata coherence and about the trade-offs involved. That section is explicitly referenced in the description of the 'noac' mount option. It explains that 'noac' is basically about disabling caching in order to achieve approximate cache coherent behaviour when multiple NFS clients need to access files that are being modified frequently on the server.

Note that 'noac' has _never_ been intended as a default NFS mount option. The kernel default is rather to cache attributes as aggressively as possible for scalability and performance reasons.
All other NFS mount programs therefore require 'noac' to be explicitly set by the user. I see no reason to punish those users with extra printk spam.

#6 lore raw
In reply to Andrew Morton
Quoted 63 lines · Andrew Morton

(switched to email. Please respond via emailed reply-to-all, not via the bugzilla web interface).

Quoted 4 lines · On Wed, 28 Jan 2009 11:29:52 -0800 (PST)…

http://bugzilla.kernel.org/show_bug.cgi?id=12564

Summary: poor performance while preprocessing source code
Product: IO/Storage

Thanks for the report.

Quoted 13 lines
           Version: 2.5
     KernelVersion: 2.6.28.2
          Platform: All
        OS/Version: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: Other
        AssignedTo: io_other-ztI5WcYan/[email protected]
        ReportedBy: [email protected]
Quoted 2 lines

Latest working kernel version: 2.6.26-rc3
Earliest failing kernel version: 2.6.26-rc4 (or 2.6.26-rc3 + patch)

(huge performance regression in NFS)

Quoted 23 lines

Distribution: gentoo

Hardware Environment:
Various 32 and 64 bit Intel and AMD machines with various
PATA and SATA disks and various network interfaces.

Problem Description:
Here's my situation. I've recently upgraded the kernels
on ~30 computers at work (from 2.6.21 to 2.6.27). These computers are used to build and test software we develop. We speed up the building process using distcc. However, after the kernel upgrade, the builds are much much slower. The preprocessing stage seems to be at least 10 times slower.
As evidence of this slowdown I am attaching two images created
using distccmon-gnome. Both snapshots were taken shortly after starting builds in a clean sandbox. The only difference is the kernel. "fast.png" was generated while running kernel 2.6.25.20. "slow.png" was generated with 2.6.26. The light purple sections indicate the preprocessing times for each file.
This slowdown is observed on both 32 and 64 bit computers
and using either gcc or the intel compiler. (The intel compiler

Quoted 8 lines

builds do not use distcc, but that are also slower.) Strangely enough, it's still faster to use an NFS mounted sandbox on a machine with an older kernel than the same sandbox on the local machine with a newer kernel. (This suggests to me that it
is neither a disk or network IO problem.)
I've used git bisect to narrow it down to a single commit:
# bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' and/or 'actimeo=0' turn off attribute caching

Quoted 1 line

And thanks for bisecting it.

I think it might be possible to bisect further, but I
don't know how get git to do it.

Quoted 16 lines
Quoted 9 lines

This is the first commit after v2.6.26-rc3. I'm not an experienced git user, so I don't know how to narrow it down further. Distccmon-gnome snapshots from the bisection process are similar to the ones noted above.
Naturally, I would like the newer kernels to have similar
performance to the older kernels.
I will be attaching various items. Let me know what other
information might be helpful.

Quoted 6 lines

Steps to reproduce:
distccmon-gnome &

# using a makefile setup to use distcc:
make -j 5 all
# note preprocessing times in distccmon
Quoted 4 lines

Something I don't understand from this:

If your normal setup is using distcc then what role does NFS have to play? I mean, distcc kind-of replaces NFS in this workload.

I don't believe that this actually has anything to do
with NFS. The only way I could see NFS coming into this is the fact that I use amd to automount home directories.
The commit mentioned above modifies just over 100 files, very
few of which appear related to NFS. I've attached the diifs. It appears to be a number of fixes rolled up into one patch. This is why I believe it might be possible to continue bisecting with git. But, to do that I suspect I would need to use a different repository and/or tag or version names.
BTW, the diffs were generated with "git diff v2.6.26-rc3"
after having done the bisection. This is why it looks to me like this one commit includes multiple updates. Since all bisection iterations were bad, the diffs against the good version must be the diffs of the one commit. Of course, I reserve the right to have made a mistake or not understand things completely.
There was one other detail that seemed strange. In the
last four bisection iterations, the kernel identified itself as -rc2 instead of -rc3. I assumed an update accidentally changed it back.

Quoted 5 lines

Perhaps you could briefly describe the topology/data-flow/etc so that we can see how NFS could be a bottleneck here?

Thanks.

#7 lore raw
Quoted 26 lines · On Wed, 28 Jan 2009 14:43:37 -0800 Steven Patrick…
Quoted 26 lines
Quoted 25 lines

Problem Description:
Here's my situation. I've recently upgraded the kernels
on ~30 computers at work (from 2.6.21 to 2.6.27). These computers are used to build and test software we develop.
We speed up the building process using distcc. However, after the kernel upgrade, the builds are much much slower. The preprocessing stage seems to be at least 10 times
slower.
As evidence of this slowdown I am attaching two images created
using distccmon-gnome. Both snapshots were taken shortly after starting builds in a clean sandbox. The only difference is the kernel. "fast.png" was generated while running
kernel 2.6.25.20. "slow.png" was generated with 2.6.26.
The light purple sections indicate the preprocessing times for each file.
This slowdown is observed on both 32 and 64 bit computers
and using either gcc or the intel compiler. (The intel compiler builds do not use distcc, but that are also slower.) Strangely enough, it's still faster to use an NFS mounted sandbox on a machine with an older kernel than the same sandbox on the local machine with a newer kernel. (This suggests to me that it
is neither a disk or network IO problem.)
I've used git bisect to narrow it down to a single commit:
# bad: [b0b539739fe9b7d75002412a787cfdf4efddbc33] NFS: Ensure that 'noac' and/or 'actimeo=0' turn off attribute caching

Quoted 4 lines
Quoted 1 line

And thanks for bisecting it.

I think it might be possible to bisect further, but I
don't know how get git to do it.

OK..

Quoted 33 lines
Quoted 16 lines
Quoted 9 lines

This is the first commit after v2.6.26-rc3. I'm not an experienced git user, so I don't know how to narrow it down further. Distccmon-gnome snapshots from the bisection process are similar to the ones noted above.
Naturally, I would like the newer kernels to have similar
performance to the older kernels.
I will be attaching various items. Let me know what other
information might be helpful.

Quoted 6 lines

Steps to reproduce:
distccmon-gnome &

# using a makefile setup to use distcc:
make -j 5 all
# note preprocessing times in distccmon
Quoted 4 lines

Something I don't understand from this:

If your normal setup is using distcc then what role does NFS have to play? I mean, distcc kind-of replaces NFS in this workload.

I don't believe that this actually has anything to do
with NFS. The only way I could see NFS coming into this is the fact that I use amd to automount home directories.
The commit mentioned above modifies just over 100 files, very
few of which appear related to NFS. I've attached the diifs. It appears to be a number of fixes rolled up into one patch. This is why I believe it might be possible to continue bisecting with git. But, to do that I suspect I would need to use a different repository and/or tag or version names.
BTW, the diffs were generated with "git diff v2.6.26-rc3"
after having done the bisection. This is why it looks to me like this one commit includes multiple updates. Since all

Quoted 8 lines

bisection iterations were bad, the diffs against the good version must be the diffs of the one commit. Of course, I reserve the right to have made a mistake or not understand things completely.
There was one other detail that seemed strange. In the
last four bisection iterations, the kernel identified itself as -rc2 instead of -rc3. I assumed an update accidentally changed it back.

Heaven knows.

OK, "This is the first commit after v2.6.26-rc3", so NFS is innocent. Presumably your git bisect landed on a big merge commit.

I'm afaid I'm not the person who can tell you how to fix this - it hasn't happened to me in a while, but I don't know what I did to prevent it happening :(

Here's the diffstat from your diffs.txt.gz:

 arch/m68k/configs/multi_defconfig              | 1269 -------------------------
 b/MAINTAINERS                                  |    4 
 b/Makefile                                     |    2 
 b/arch/arm/common/locomo.c                     |   66 -
 b/arch/arm/kernel/armksyms.c                   |    2 
 b/arch/arm/kernel/arthur.c                     |    2 
 b/arch/arm/mach-omap1/board-palmte.c           |    2 
 b/arch/arm/mach-omap1/board-palmz71.c          |    2 
 b/arch/arm/mach-omap2/board-2430sdp.c          |    1 
 b/arch/arm/mach-omap2/board-apollon.c          |    1 
 b/arch/arm/mach-omap2/board-generic.c          |    1 
 b/arch/arm/mach-omap2/board-h4.c               |    1 
 b/arch/arm/mach-omap2/clock.c                  |    4 
 b/arch/arm/mach-omap2/clock34xx.h              |   21 
 b/arch/arm/mach-omap2/cm-regbits-34xx.h        |    1 
 b/arch/arm/mach-omap2/mailbox.c                |   25 
 b/arch/arm/mach-omap2/prm.h                    |    2 
 b/arch/arm/mach-orion5x/dns323-setup.c         |    2 
 b/arch/arm/mach-orion5x/kurobox_pro-setup.c    |    2 
 b/arch/arm/mach-pxa/colibri.c                  |    3 
 b/arch/arm/mach-pxa/spitz.c                    |    1 
 b/arch/arm/mm/proc-arm925.S                    |    2 
 b/arch/arm/mm/proc-arm926.S                    |    2 
 b/arch/arm/mm/proc-arm940.S                    |    2 
 b/arch/arm/mm/proc-arm946.S                    |    2 
 b/arch/arm/plat-omap/clock.c                   |   10 
 b/arch/arm/plat-omap/dma.c                     |    2 
 b/arch/arm/plat-omap/mailbox.c                 |    1 
 b/arch/blackfin/mach-bf527/boards/ezkit.c      |    2 
 b/arch/m68k/Kconfig                            |    9 
 b/arch/m68k/configs/amiga_defconfig            |  159 +--
 b/arch/m68k/configs/apollo_defconfig           |  140 --
 b/arch/m68k/configs/atari_defconfig            |  143 +-
 b/arch/m68k/configs/bvme6000_defconfig         |  138 --
 b/arch/m68k/configs/hp300_defconfig            |  142 +-
 b/arch/m68k/configs/mac_defconfig              |  144 +-
 b/arch/m68k/configs/mvme147_defconfig          |  138 --
 b/arch/m68k/configs/mvme16x_defconfig          |  138 --
 b/arch/m68k/configs/q40_defconfig              |  159 +--
 b/arch/m68k/configs/sun3_defconfig             |  140 --
 b/arch/m68k/configs/sun3x_defconfig            |  140 --
 b/arch/m68k/kernel/head.S                      |    2 
 b/arch/m68k/kernel/setup.c                     |   15 
 b/arch/powerpc/platforms/pasemi/misc.c         |    7 
 b/arch/sparc64/defconfig                       |   40 
 b/arch/sparc64/mm/init.c                       |    2 
 b/arch/x86/kernel/process.c                    |   36 
 b/arch/x86/mm/pat.c                            |    2 
 b/drivers/block/amiflop.c                      |    6 
 b/drivers/block/z2ram.c                        |    2 
 b/drivers/char/snsc_event.c                    |    2 
 b/drivers/char/vme_scc.c                       |    4 
 b/drivers/i2c/busses/i2c-amd756.c              |    2 
 b/drivers/i2c/busses/i2c-nforce2.c             |   28 
 b/drivers/i2c/chips/max6875.c                  |    3 
 b/drivers/i2c/i2c-core.c                       |   22 
 b/drivers/ide/legacy/macide.c                  |    3 
 b/drivers/input/keyboard/hilkbd.c              |    4 
 b/drivers/input/misc/hp_sdc_rtc.c              |    5 
 b/drivers/input/serio/hp_sdc_mlc.c             |    5 
 b/drivers/input/serio/q40kbd.c                 |    2 
 b/drivers/media/video/cs5345.c                 |    7 
 b/drivers/media/video/cs53l32a.c               |   10 
 b/drivers/media/video/cx18/cx18-i2c.c          |    9 
 b/drivers/media/video/cx25840/cx25840-core.c   |    7 
 b/drivers/media/video/et61x251/et61x251_core.c |    2 
 b/drivers/media/video/ivtv/ivtv-i2c.c          |   13 
 b/drivers/media/video/m52790.c                 |    9 
 b/drivers/media/video/msp3400-driver.c         |   17 
 b/drivers/media/video/saa7115.c                |   40 
 b/drivers/media/video/saa7127.c                |    9 
 b/drivers/media/video/saa717x.c                |    9 
 b/drivers/media/video/sn9c102/sn9c102_core.c   |    2 
 b/drivers/media/video/tuner-core.c             |   17 
 b/drivers/media/video/upd64031a.c              |    6 
 b/drivers/media/video/upd64083.c               |    6 
 b/drivers/media/video/vp27smpx.c               |    9 
 b/drivers/media/video/wm8739.c                 |    7 
 b/drivers/media/video/wm8775.c                 |    7 
 b/drivers/media/video/zc0301/zc0301_core.c     |    2 
 b/drivers/media/video/zoran_device.c           |    2 
 b/drivers/media/video/zoran_driver.c           |    2 
 b/drivers/net/82596.c                          |    7 
 b/drivers/net/apne.c                           |    3 
 b/drivers/net/mac89x0.c                        |    3 
 b/drivers/net/macmace.c                        |    3 
 b/drivers/net/sun3lance.c                      |    3 
 b/drivers/net/wireless/atmel.c                 |    2 
 b/drivers/video/Kconfig                        |    4 
 b/drivers/video/amifb.c                        |    4 
 b/drivers/video/dnfb.c                         |    3 
 b/drivers/video/hpfb.c                         |    2 
 b/drivers/video/pxafb.c                        |    5 
 b/fs/befs/endian.h                             |    2 
 b/fs/nfs/inode.c                               |    7 
 b/include/asm-arm/arch-omap/common.h           |    4 
 b/include/asm-arm/arch-omap/control.h          |    2 
 b/include/asm-arm/arch-omap/mmc.h              |   24 
 b/include/asm-arm/arch-sa1100/irqs.h           |    2 
 b/include/asm-arm/hardware/locomo.h            |   19 
 b/include/asm-m68k/bug.h                       |    4 
 b/include/asm-m68k/io.h                        |   44 
 b/include/asm-m68k/setup.h                     |    2 
 b/include/asm-m68k/uaccess.h                   |    6 
 b/include/linux/i2c.h                          |    7 
 b/include/linux/i2c/pcf857x.h                  |    3 
 106 files changed, 833 insertions(+), 2743 deletions(-)

Now, distcc has a long track record of tripping up bugs and regressions in the networking code - it's very clever that way. But it doesn't look like there were any relevant networking changes in that window.

So I think the next step is to ask you to continue with (or redo) that bisection search.

Is there someone who is more git-competent than I who could help with this please?