New Kernel Bugs

This is the listing of the open bugs that are relatively new, around 2.6.22 and up. They are vaguely classified by specific area. (not a full list, there are more :)

The good part is that reporters of the bugs below are still around and haven't dissipated, or disposed of their hardware, so it is a good time to get the bugs.
Those bugzillas that have been started as regressions on Rafael's list are not mentioned here so far, since they are being tracked as new regressions already.

It would be appreciated if the corresponding maintenance team could take a look, close off any which are fixed and see if they can fix any which aren't.

NOTE: when replying to this email, please add the bug number to the Subject in the form [Bug 1234] so that bugzilla will capture the discussion. Thanks.

ACPI====================================================================

System does not load without acpi=off ide=nodma noapic
http://bugzilla.kernel.org/show_bug.cgi?id=9358
Kernel: 2.6.23.1

ACPI Error attaching device data
http://bugzilla.kernel.org/show_bug.cgi?id=9354
Kernel: 2.6.24-rc2

/proc/acpi/battery displays Incorrect voltages
http://bugzilla.kernel.org/show_bug.cgi?id=9341
Kernel: 2.6.23.1

PATA scan: ACPI Exception AE_AML_PACKAGE_LIMIT... is beyond end of object
http://bugzilla.kernel.org/show_bug.cgi?id=9320
Kernel: 2.6.24-rc2
(Tejun: calling _GTF without calling _STM first. _GTM doesn't have any prerequisite (it can't). Can someone familiar with ACPI tell me why the method is failing? At any rate, libata should work fine regardless of ACPI failures. Maybe it's time to start blacklist to skip ATA-ACPI for some boards to avoid those annoying messages during boot)

ACPI Battery Info in /sys but not /proc/acpi
http://bugzilla.kernel.org/show_bug.cgi?id=9183
Kernel: 2.6.23-rc8-mm2

When using ACPI on a Compaq Presario V6221EU the laptop goes into deadlock after a random amount of time
http://bugzilla.kernel.org/show_bug.cgi?id=9118
Kernel: 2.6.23-rc6

ACPI video driver should validate brightness level before setting it via _BCM
http://bugzilla.kernel.org/show_bug.cgi?id=9277
Kernel: 2.6.23

VIDEO/DVB

dvb driver reboot system
http://bugzilla.kernel.org/show_bug.cgi?id=9357
Kernel: 2.6.21.5

PLATFORM===============================================================

xipImage is built so that uBoot cant run it (ARM)
http://bugzilla.kernel.org/show_bug.cgi?id=9356
Kernel: 2.6.21

Samsung R20 - ACPI: PCI Root Bridge [PCI0] (0000:00)
http://bugzilla.kernel.org/show_bug.cgi?id=9339
Kernel: 2.6.24
(boot is very long
..MP-BIOS bug: 8254 timer not connected to IO-APIC then the boot stop at :
ACPI: PCI Root Bridge [PCI0] (0000:00) (during 3 minutes, and boot continue)

system_64.h: switch_to inline asm should be more robbust wrt optimizations
http://bugzilla.kernel.org/show_bug.cgi?id=9302
Kernel: 2.6.24-rc1

with CONFIG_NO_HZ and/or CONFIG_HPET_TIMER set kernel 2.6.23 doesn't boot (ARM, Timer)
http://bugzilla.kernel.org/show_bug.cgi?id=9229
Kernel: 2.6.23

NETWORKING===========================================================

RTNLGRP_ND_USEROPT does not report ifindex (IPv6)
http://bugzilla.kernel.org/show_bug.cgi?id=9349
Kernel: 2.6.24+

a kernel error happend in the func: __skb_dequeue when using in pfifo_fast_dequeue
http://bugzilla.kernel.org/show_bug.cgi?id=9342
Kernel: 2.6.11.1 - reporter asked to try recent kernel

e100 does not work after boot
http://bugzilla.kernel.org/show_bug.cgi?id=9336
Kernel: 2.6.23.1

2.6.23.1-smp kernel panic (network-related)
http://bugzilla.kernel.org/show_bug.cgi?id=9318
Kernel: 2.6.23.1
Infiniband panic

sundance -> 4port D-Link System Inc DFE-580TX -> Log errors
http://bugzilla.kernel.org/show_bug.cgi?id=9311
Kernel: 2.6.22.9

via-rhine driver stalls with: PHY status 786d, resetting...
http://bugzilla.kernel.org/show_bug.cgi?id=9300
Kernel: 2.6.23+

Weird network problems with 2.6.23-rc2
http://bugzilla.kernel.org/show_bug.cgi?id=9080
http://lkml.org/lkml/2007/8/11/40 - description

rt2500pci: low TCP throughput (wireless)
http://bugzilla.kernel.org/show_bug.cgi?id=9273
Kernel: 2.6.24-rc1
This is a regression

Unable to build wifi network between zd1201 and b43
http://bugzilla.kernel.org/show_bug.cgi?id=9237
Kernel: 2.6.24-rc1

Crash after module unload in b43 (wireless)
http://bugzilla.kernel.org/show_bug.cgi?id=9233
Kernel: 2.6.24-rc1

(net typhoon) "no descs for cmd, had (needed) 0 (1) cmd, 31 (7) resp"
http://bugzilla.kernel.org/show_bug.cgi?id=9225
Kernel: 2.6.23.1

IDE/SATA=========================================================

pata_pdc202xx_old excessive ATA bus errors
http://bugzilla.kernel.org/show_bug.cgi?id=9337
2.6.24-rc2

Drive seagate ST380011AS needs to be blacklisted
http://bugzilla.kernel.org/show_bug.cgi?id=9309
Kernel: 2.6.22.X

DVD-RAM umount and disk free bug
http://bugzilla.kernel.org/show_bug.cgi?id=9265
Kernel: 2.6.15 (asked to try current kernel)

FILE SYSTEMS=======================================================

ext4: delalloc space accounting problem drops data
http://bugzilla.kernel.org/show_bug.cgi?id=9329
Kernel: 2.6.24-rc1

POSIX Access Control Lists cause bogus file system check errors
http://bugzilla.kernel.org/show_bug.cgi?id=9241
Kernel: 2.6.23.1

MEMORY MANAGEMENT================================================

My system hangs when it has no more free memory to allocate via malloc()
http://bugzilla.kernel.org/show_bug.cgi?id=9316
Kernel: 2.6.23
User program, "My system hangs when it has no more free memory to allocate via malloc()"

BUG: unable to handle kernel paging request at virtual address

26121228/kswapd0[231] exited with preempt_count 1
http://bugzilla.kernel.org/show_bug.cgi?id=9305
EIP is at free_block+0x6d/0xe4
Kernel: 2.6.22.6

POWER MANAGEMENT==================================================

IBM X41 looses time after Suspend2Disk
http://bugzilla.kernel.org/show_bug.cgi?id=9314
Kernel: 2.6.23

Suspend to RAM resume hangs on a tickless (NO_HZ) kernel
http://bugzilla.kernel.org/show_bug.cgi?id=9275
Kernel: 2.6.23
This is HP notebook nc6320 T2400 945GM

VIDEO DRIVERS========================================================

No text consoles with FRAMEBUFFER_CONSOLE_DETECT_PRIMARY
http://bugzilla.kernel.org/show_bug.cgi?id=9310
Kernel: 2.6.24-rc1
This is a regression

PARALLEL PORT========================================================

LPC IT8705 POST port making noise on parallel port
http://bugzilla.kernel.org/show_bug.cgi?id=9306
Kernel: 2.6.16+

I/O STORAGE===========================================================

kernel bug from pktcdvd
http://bugzilla.kernel.org/show_bug.cgi?id=9294
Kernel: 2.6.23

After pci-e video card was installed, pci add-on usb card & firewire card fail
http://bugzilla.kernel.org/show_bug.cgi?id=9223
Kernel: 2.6.20 (testing of latest kernel requested)

SCSI==================================================================

qla2xxx: driver initialization does not complete when booting with Port connected
http://bugzilla.kernel.org/show_bug.cgi?id=9267
Kernel: 2.6.23.1

SOUND ALSA============================================================

Unable to load snd-hda-intel module: Unknown symbol in module, or unknown parameter
http://bugzilla.kernel.org/show_bug.cgi?id=9242
Kernel: 2.6.24-rc1

usbaudio microphone: regular sound distortion on several Logitech Webcams
http://bugzilla.kernel.org/show_bug.cgi?id=9230
Kernel: 2.6.22.9

HID====================================================================

Kernel NULL pointer dereference at :usbhid:hiddev_ioctl+0x2f/0xabc
http://bugzilla.kernel.org/show_bug.cgi?id=9216
Kernel: 2.6.23.1
Looks like this is a regression

Thanks.

You're optimistic.

One response from a developer

Zero responses from developers

Zero responses from developers

Tejun doing stuff

Marked as a duplicate of an already-resolved bug.

Someone called Jike Song is trying to help out. Regular developers awol.

Zero responses from developers

Mauro thinks it might be bad hardware

Zero responses from developers

No response from developers

Not really a bug.

No response from developers

No response from developers

No response from developers

No response from developers

No response from developers

No response from developers

No response from developers

No response from developers

No response from developers

No response from developers

No response from developers

No response from developers

No response from developers

Jeff and Tehun did some stuff.

No response from developers

No response from developers

Andreas did some work, seemed to lose interest.

Rafael poked Thomas a week ago, to no effect. Thomas has been travelling.

No response from developers

Rafael poked Thomas a week ago, to no effect. Thomas has been travelling.

No response from developers

No response from developers

No response from developers

I think we might have fixed this.

No response from developers

No response from developers

Takashi has responded

Clemens responded

No response from developers

So I count around seven reports which people are doing something with and twenty seven which have been just ignored.

Three of these reports have been identified as regressions. All three of those remain unresponded to.

It's fixed and merged, I just forgot to close the bugzilla. Did so now.

Jens Axboe

Fixed (extended) in the DaveM's tree (or will be soon - patch was submitted by Pierre Ynard).

Sorry, others are either driver related (and thus require hardware to be tested on and maintainers to be kicked in) or too obscure (like 2.6.11 bug and weird network problem which is undetectible on other systems).

Yes, we suck, but we try to recover :)

	Evgeniy Polyakov

From: Andrew Morton <[email protected]>
Date: Tue, 13 Nov 2007 03:15:53 -0800

That's funny, then how come there was a proper patch fix posted and it's now in my tree ready to go to Linus?

I think you like just saying "No response from developers" over and over again to make some of point about how developers are ignoring lots of bugs. That's fine, but at least be accurate about it :-)

Do you believe that our response to bug reports is adequate?

From: Andrew Morton <[email protected]>
Date: Tue, 13 Nov 2007 03:49:16 -0800

Do you feel that making us feel and look like shit helps?

I guess I'm just masterbating here all night long with the 46 bug fixes I've reviewed fully and queued up into my tree. Along with all the 10 or so -stable submissions I did tonight as well.

When someone like me is bug fixing full time, I take massive offense to the impression you're trying to give especially when it's directed at the networking.

So turn it down a notch Andrew.

I bet if you did things like list explicitly by name every single person who adds a bug fix (however trivial) to an -mm release instead of a new feature, you'll better achieve your goal than what you're doing here.

That doesn't answer my question.

See, first we need to work out whether we have a problem. If we do this, then we can then have a think about what to do about it.

I tried to convince the 2006 KS attendees that we have a problem and I resoundingly failed. People seemed to think that we're doing OK.

But it appears that data such as this contradicts that belief.

This is not a minor matter. If the kernel _is_ slowly deteriorating then this won't become readily apparent until it has been happening for a number of years. By that stage there will be so much work to do to get us back to an acceptable level that it will take a huge effort. And it will take a long time after that for the kerel to get its reputation back.

So it is important that we catch deterioration *early* if it is happening.

From: Andrew Morton <[email protected]>
Date: Tue, 13 Nov 2007 04:12:59 -0800

You tell me what I should spend my time working on, and I promise to do it OK? :-)

For example, if I have a choice between a TCP crash just about anyone can hit and some obscure issue only reported with some device nearly nobody has, which one should I analyze and work on?

That's the problem. All of us prioritize and it means the chaff collects at the bottom. You cannot fix that except by getting more bug fixers so that the chaff pile has a chance to get smaller.

Luckily if the report being ignored isn't chaff, it will show up again (and again and again) and this triggers a reprioritization because not only is the bug no longer chaff, it also now got a lot of information tagged to it so it's a double worthwhile investment to work on the problem.

I think a lot of bugs that "aren't getting looked at" are simply sitting in some early stage of this process.

My suggestion: regressions.

If we're really active in chasing down the regressions then I think we can be confident that the kernel isn't deteriorating. Probably it will be improving as we also fix some always-been-there bugs.

I think that we're fairly good about working the regressions in Adrian/Michal/Rafael's lists but once Linus releases 2.6.x we tend to let the unsolved ones slide, and we don't pay as much attention to the regressions which 2.6.x testers report.

Yes, that's a useful technique. If multiple people are being hurt a lot by a bug then that's a more important one to fix than the single-person minor-irritant bug.

otoh that doesn't work very well with driver/platform bugs. Often these are regressions which only a single person can reproduce within the time window which we have in which we can fix it. If we don't fix it in that window it'll go out to distros and presumably some more people will hit it.

So I don't see much alternative here to the traditional work-with-the-originator way of resolving it.

git bisection should really help us with these regressions but it doesn't appear that people are using as much as one would like. I'm hoping that the very good http://www.kernel.org/doc/local/git-quick.html will help us out here. Thanks to the mystery person who prepared that.

Can't we wait until all regressions[0] are fixed before releasing a new 2.6.x? I'd consider regressions a *literal* show stopper, and with this policy they just have be fixed, nothing would "slide"...

my 2 cents,
Christian.

[0] preferably only reproducible regressions, with responsive reporters.

BOFH excuse #380:

Operators killed when huge stack of backup tapes fell over.

Problem is that everyone would then sit around pumping shiny new features into their trees waiting until someone else fixes the regressions.

There are a number of process things we _could_ do. Like

- have bugfix-only kernel releases

- Just refuse to merge any non-bugfix patches for a subsystem when it is determined that the subsystem has "too many" regressions.

- Create an "if you added a regression, you should fix some other bug too" rule.

- probably other stuff.

But we can't/shouldn't do any of that until it is generally agreed that we have a problem and that the problem is of sufficient magnitude that process changes are needed to address it. We aren't at that stage yet.

Here's an important point: developers have a fixed amount of development time. They spend some of that time fixing bugs and the rest of that time on <otherstuff>. And while one could cook up all sorts of wonderful process changes, they all would be aimed at a single thing: shifting some of the developers' time away from <otherstuff> and onto bugfixing.

At this stage the only tool which is being deployed to attempt to bring about that reprioritisation is suasion. aka "lkml flamewar".

There is another possible solution:
Finding more maintainers.

The problem seems to be that there are many people who want to write drivers for cool shiny new hardware, but not many people willing to learn to know and maintain existing code.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Adrian Bunk does (did?) this with 2.6.16.x, although it always seemed to me like an unrewarded one man show. AFAIK not even the big distros are begging for bugfix-only versions, as they too want to have (sell) new features. Mission critical systems might want to require such versions, but I guess they're using heavily customized trees anyway.

Hm, that's what I had in mind. Has this been tried already?

Naah, I'm not really in favour of blaming someone. The kernel doesn't have SLA contracts (yet), so no need for giving out penalties :)

Keeping track of the (number of) regressions / bugs each release seems to be a good start, IMHO.

True. Implementing "only bugfixes from now on" (i.e. a longer freeze-window) would perhaps speed up the shifting a bit: $developer can still do $otherstuff all day long, but it won't get merged anyway, because we're in "only bugfixes from now on"-mode.

True. But I just noticed that I have to distinguish between "flamewars" and "fierce discussions": if I'd imagine a room with ~50 developers/bystanders brainstorming on a issue like this (at the same time, without the wonderful delay of writing/sending an email), it'd feel much more uncomfortable.

Christian.

BOFH excuse #433:

error: one bad user found in front of screen

And congratulations to him for that. We almost entirely dropped 2.6.16, but there's a regression some time since then that makes large MMAPed files a major pain (specifically the dcc database clean takes about 5 minutes on 2.6.16 and about 12 hours on 2.6.20 or 2.6.23 series kernels)

But we keep putting off writing a small testcase that can repeat the issue so we can bisect it - because it's working fine with 2.6.16 on that machine.

Bron.

Heh. I suspect you don't even need to bisect it.

The big difference with large mmap'ed files is that later kernels will actually track dirty ratios for dirty mmap'ed pages. Earlier kernels never did.

So in older kernels, you can dirty as much memory as you want, and the kernel will never try to write it back (well - "never" here means one of either (a) you ask it to with msync or (b) you run out of memory, when the kernel then totally falls down and the machine is essentially unusuable).

So *if* the symptom seems to be that the later kernels do a lot more IO, then try to change

/proc/sys/vm/dirty_[background_]ratio

which is just a percentage of memory (defaults to 5% for background and 10% for foreground dirtying). Turn them both up a lot (say to 50 and 80 percent respectively) and see if that makes a difference.

If so, you'll be the first one to officially even notice this change, I think.

Linus

# This should help reduce flushing on Cache::FastMmap files vm.dirty_background_ratio = 50 vm.dirty_expire_centisecs = 9000
vm.dirty_ratio = 80
vm.dirty_writeback_centisecs = 3000

So we've already been running those settings for a while. They didn't help.

We also gave this thing its very own dedicated ServeRAID card and associated RAID1 set of high speed SCSI drives (mainly because they were just sitting there already attached to the machine and unused, we don't love DCC that much) and it didn't help. Helped the rest of the machine now that the system drive wasn't being pegged 100% for 12 hours a day, but it didn't speed things up any.

It was making some pretty random little scattered changes all through that file. Hmm.. here's what the developers said about it:

First dbclean creates a new dcc_db file by copying from the old file. As it copies, it decides whether each record is worth keeping. That involves looking up the checksums in the old hash table. This is as almost afast a simple /bin/cp if the old dcc_db and dcc_db.hash files fit in RAM.

The dbclean creates a new dcc_db.hash file. This starts with creating an empty new dcc_db.hash file. Then the new dcc_db and dcc_db.hash files are mapped into memory, and dbclean creates pointers to each checksum in the dcc_db file in the dcc_db.hash file.

While dbclean is running, dccd unmaps everything and tries to stay out of the way.

Yay for us. Thankfully it doesn't affect Cyrus's MMAP usage (read only with direct seek and write calls to change anything, then remap) or we would have suffered pretty badly!

Guess we'd better get on to figuring building a simple test app. The mmap file that DCC uses is about 2Gb if that makes any difference:

-rw-r--r-- 1 dcc dcc 2035138560 Nov 15 00:15 dcc_db
-rw-r--r-- 1 dcc dcc 516612096 Nov 14 06:27 dcc_db.hash

The machine has 6Gb of memory and should be able to fit these files fine:

[root@out1 hm]$ free
total used free shared buffers cached
Mem: 6232364 5758112 474252 0 41756 3002528
-/+ buffers/cache: 2713828 3518536
Swap: 2048248 74944 1973304

And here's what top says about the process:

      15   0 1914m  57m  41m D    5  1.0 346:07.79 dccd

This is on: 2.6.16.55-reiserfix-fai
(one small patch to reiserfs, and built with netboot support for FAI)

So yeah - we'll try to get a clearer idea of what it's doing, but the knob twiddle didn't work for us.

Bron.

Ok, so something else is up. If the mmap file is 2G, and you have 6G of RAM, you shouldn't be hitting the dirty limits with those setups.

Of course, it may still be that some accounting thing is simply off, and the dirty limits trigger *despite* all the proper config settings ;)

Yeah, if you have something that others can see in action, that is sure going to get more people to look at it.

That said - I'm sincerely hoping that you're not running on a 32-bit kernel. Because if so, those percentages are percentages of *normal* memory, not highmem (that got changed at one point after people ran out of lowmem).

So even at 100% dirty limits, it won't let you dirty more than 1GB on the default 32-bit setup.

Linus

Side note: all of these are obviously still just heuristics. If you really *do* run on a 32-bit kernel, and you want to have the pain, I'm sure you can just disable the dirty limits with a one-liner kernel mod. And if it's useful enough, we can certainly expose flags like that.. Not that I expect that much anybody else will ever care, but it's not like it's wrong to expose the silly heuristics the kernel has to users that have very specific loads.

That said, I still do hope you aren't actually using HIGHMEM64G. I was really hoping that the people who had enough moolah to buy >4GB of RAM had long since also upgraded to a 64-bit machine ;)

Linus

I'm afraid we are, which probably explains it.

We have a bunch of 64 bit machines, but this particular machine is one of our somewhat more ancient IBM x235 machines. It's got stacks of fast SCSI drives and a couple of hyperthreading Xeons in it. Very nice machine in its day, and very reliable which is why we have kept them, even though at 6RU it chews through disk space.

Unfortunately none of the 64 bit machines are world facing, and we're running HIGHMEM64G on a bunch of machines both for consistency value and because we only have one machine left with only 2Gb.

I guess we'll be doing the one-liner kernel mod and testing that then. I'd certainly like to build a test case anyway so I'm not spending too much time rebooting that machine, it's also our outbound SMTP gateway.

And I'll keep in mind finding a 64 bit capable machine for the role when I can.

Thanks for the feedback on this - I'll come back with more details once we've done some testing, but this sounds likely, and I don't think DCC is going to change how it works, so we're stuck supporting it.

Bron.

The thing to look at is "get_dirty_limits()" in mm/page-writeback.c, and in this particular case it's the

	unsigned long available_memory = determine_dirtyable_memory();

that's going to bite you. In particular, note the

	x -= highmem_dirtyable_memory(x);

that we do in determine_dirtyable_memory().

So in this case, if you basically remove that line, it will allow all of memory to be dirtied (including highmem), and then the background_ratio will work on the whole 6GB.

HOWEVER! It's worth noting that we also have some other old legacy cruft there that may interfere with your code. In particular, if you look at the top of "get_dirty_limits()", it *also* does a

        unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
                                global_page_state(NR_ANON_PAGES)) * 100) /
                                        available_memory;

        dirty_ratio = vm_dirty_ratio;
        if (dirty_ratio > unmapped_ratio / 2)
                dirty_ratio = unmapped_ratio / 2;

and that whole "unmapped_ratio" comparison is probably bogus these days, since we now take the mapped dirty pages into account. That code harks back to the days before we did that, and dirty ratios only affected non-mapped pages.

And in particular, now that I look at it, I wonder if it can even go negative (because "available_memory" may be *smaller* than the NR_FILE_MAPPED|ANON_PAGES sum!).

We'll fix up a negative value anyway (because of the clamping of dirty_ratio to no less than 5), but the point is that the whole "unmapped_ratio" thing probably doesn't make sense any more, and may well make the dirty_ratio not work for you, because you may have a very small unmapped_ratio that effectively makes all dirty limits always clamp to a very small value.

So regardless, I think you may want to try the appended patch *first*.

If this patch makes a difference, please holler. I think it's the correct thing to do, but I'm not going to actually commit it without somebody saying that it makes a difference (and preferably Peter Zijlstra and Andrew acking it too).

Only *after* testing this change is it probably a good idea to test the real hack of then removing the highmem_dirtyable_memory() thing.

Peter? Andrew?

Linus

---
 mm/page-writeback.c |    8 --------
 1 files changed, 0 insertions(+), 8 deletions(-)

index 81a91e6..d55cfca 100644
@@ -297,20 +297,12 @@ get_dirty_limits(long *pbackground, long *pdirty, long *pbdi_dirty,
 {
 	int background_ratio;		/* Percentages */
 	int dirty_ratio;
-	int unmapped_ratio;
 	long background;
 	long dirty;
 	unsigned long available_memory = determine_dirtyable_memory();
 	struct task_struct *tsk;
 
-	unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
-				global_page_state(NR_ANON_PAGES)) * 100) /
-					available_memory;
-
 	dirty_ratio = vm_dirty_ratio;
-	if (dirty_ratio > unmapped_ratio / 2)
-		dirty_ratio = unmapped_ratio / 2;
-
 	if (dirty_ratio < 5)
 		dirty_ratio = 5;
 

I wondered about that part the other day when I went through the BDI dirty code due to that iozone thing..

The initial commit states:

commit d90e4590519d196004efbb308d0d47596ee4befe

Author: akpm <akpm>
Date:   Sun Oct 13 16:33:20 2002 +0000

[PATCH] reduce the dirty threshold when there's a lot of mapped

Dirty memory thresholds are currently set by /proc/sys/vm/dirty_ratio.

Background writeout levels are controlled by
/proc/sys/vm/dirty_background_ratio.

Problem is that these levels are hard to get right - they are too static. If there is a lot of mapped memory around then the 40% clamping level causes too much dirty data. We do lots of scanning in page reclaim, and the VM generally starts getting into distress. Extra swapping, extra page unmapping.

It would be much better to simply tell the caller of write(2) to slow down - to write out their dirty data sooner, to make those written pages trivially reclaimable. Penalise the offender, not the innocent page allocators.

This patch changes the writer throttling code so that we clamp down much harder on writers if there is a lot of mapped memory in the machine. We only permit memory dirtiers to dirty up to 50% of unmapped memory before forcing them to clean their own pagecache.

BKrev: 3da9a050Mz7H6VkAR9xo6ongavTMrw

But because dirty mapped pages are no longer special, I'd say the reason for its existance is gone. So,

As for the highmem part, that was due to buffer cache, and unfortunately that is still true. Although maybe we can do something smart with the per-bdi stuff.

Something like this ought to do I guess. Although my mapping_is_buffercache() is the ugliest thing. I'm sure that can be done better.

Uncompiled, untested

---
 mm/page-writeback.c |   28 ++++++++++++++++++++--------
 1 file changed, 20 insertions(+), 8 deletions(-)

Index: linux-2.6/mm/page-writeback.c
===================================================================
@@ -280,27 +280,28 @@ static unsigned long highmem_dirtyable_m
 #endif
 }
 
-static unsigned long determine_dirtyable_memory(void)
+static unsigned long determine_dirtyable_memory(int highmem)
 {
 	unsigned long x;
 
 	x = global_page_state(NR_FREE_PAGES)
 		+ global_page_state(NR_INACTIVE)
 		+ global_page_state(NR_ACTIVE);
-	x -= highmem_dirtyable_memory(x);
+	if (!highmem)
+		x -= highmem_dirtyable_memory(x);
 	return x + 1;	/* Ensure that we never return 0 */
 }
 
 static void
 get_dirty_limits(long *pbackground, long *pdirty, long *pbdi_dirty,
-		 struct backing_dev_info *bdi)
+		 struct backing_dev_info *bdi, int highmem)
 {
 	int background_ratio;		/* Percentages */
 	int dirty_ratio;
 	int unmapped_ratio;
 	long background;
 	long dirty;
-	unsigned long available_memory = determine_dirtyable_memory();
+	unsigned long available_memory = determine_dirtyable_memory(highmem);
 	struct task_struct *tsk;
 
 	unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +

@@ -346,6 +347,16 @@ get_dirty_limits(long *pbackground, long
 	}
 }
 
+static inline int mapping_is_buffercache(struct address_space *mapping)
+{
+	struct super_block *sb = mapping->host->i_sb;
+
+	if (sb && sb->s_bdev && sb->s_bdev->bd_inode->i_mapping != mapping)
+		return 0;
+
+	return 1;
+}
+
 /*
  * balance_dirty_pages() must be called by processes which are generating dirty
  * data.  It looks at the number of dirty pages in the machine and will force

@@ -364,6 +375,7 @@ static void balance_dirty_pages(struct a
 	unsigned long write_chunk = sync_writeback_pages();
 
 	struct backing_dev_info *bdi = mapping->backing_dev_info;
+	int highmem = !mapping_is_buffercache(mapping);
 
 	for (;;) {
 		struct writeback_control wbc = {

@@ -375,7 +387,7 @@ static void balance_dirty_pages(struct a
 		};
 
 		get_dirty_limits(&background_thresh, &dirty_thresh,
-				&bdi_thresh, bdi);
+				&bdi_thresh, bdi, highmem);
 		bdi_nr_reclaimable = bdi_stat(bdi, BDI_RECLAIMABLE);
 		bdi_nr_writeback = bdi_stat(bdi, BDI_WRITEBACK);
 		if (bdi_nr_reclaimable + bdi_nr_writeback <= bdi_thresh)

@@ -394,7 +406,7 @@ static void balance_dirty_pages(struct a
 			writeback_inodes(&wbc);
 			pages_written += write_chunk - wbc.nr_to_write;
 			get_dirty_limits(&background_thresh, &dirty_thresh,
-				       &bdi_thresh, bdi);
+				       &bdi_thresh, bdi, highmem);
 		}
 
 		/*

@@ -503,7 +515,7 @@ void throttle_vm_writeout(gfp_t gfp_mask
 	long dirty_thresh;
 
         for ( ; ; ) {
-		get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
+		get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL, 1);
 
                 /*
                  * Boost the allowable dirty threshold a bit for page

@@ -546,7 +558,7 @@ static void background_writeout(unsigned
 		long background_thresh;
 		long dirty_thresh;
 
-		get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL);
+		get_dirty_limits(&background_thresh, &dirty_thresh, NULL, NULL, 1);
 		if (global_page_state(NR_FILE_DIRTY) +
 			global_page_state(NR_UNSTABLE_NFS) < background_thresh
 				&& min_pages <= 0)

No, this absolutely sucks.

Why?

It's totally unacceptable to have per-mapping notions of how much memory we have. We used to do *exactly* that, and it's idiocy.

The reason it's unacceptable idiocy is that it means that two processes that access different files will then have *TOTALLY*DIFFERENT* notions of what the "dirty limit" is. And as a result, one process will happily write lots and lots of dirty stuff and never throttle, and the other process will have to throttle all the time - and clean up after the process that didn't!

See?

The fact is, because we count dirty pages as one resource, we must also have *one* limit.

So this patch is a huge regression. You might not notice it, because if everybody writes to the same kind of mapping, nobody will be hurt (they all have effectively the same global limit anyway), but you *will* notice if you ever have two different values of "highmem".

Unacceptable. We used to do exactly what your patch does, and it got fixed once. We're not introducing that fundamentally broken concept again.

Linus

Agreed, I was just about to send out an email saying that..

Say all buffer cache users were against default_backing_dev_info, and we'd give default_backing_dev_info less, that should work out, right?

( I'm not yet clear on if buffer cache already uses default_backing_dev_info or not, bdget() seems to suggest it does )

Examples of non-broken solutions:
(a) always use lowmem sizes (what we do now)
(b) always use total mem sizes (sane but potentially dangerous: but the VM pressure should work! It has serious bounce-buffer issues, though, which is why I think it's crazy even if it's otherwise consistent)
(c) make all dirty counting be *purely* per-bdi, so that everybody can disagree on what the limits are, but at least they also then use different counters

So it's just the "different writers look at the same dirty counts but then interpret it to mean totally different things" that I think is so fundamentally bogus. I'm not claiming that what we do now is the only way to do things, I just don't think your approach is tenable.

Btw, I actually suspect that while (a) is what we do now, for the specific case that Bron has, we could have a /proc/sys/vm option to just enable (b). So we don't have to have just one consistent model, we can allow odd users (and Bron sounds like one - sorry Bron ;) to just force other, odd, but consistent models.

I'd also like to point out that while the "bounce buffer" issue is not so much a HIGHMEM issue on its own (it's really about the device DMA limits, which are _independent_ of HIGHMEM, of course), the reason HIGHMEM is special is that without HIGHMEM the bounce buffers generally work perfectly fine.

The problem with HIGHMEM is that it causes various metadata (dentries, inodes, page struct tables etc) to eat up memory "prime real estate" under the same kind of conditions that also dirty a lot of memory. So the reason we disallow HIGHMEM from dirty limits is only *partly* the per-device or mapping DMA limits, and to a large degree the fact that non-highmem memory is special in general, and it is usually the non-highmem areas that are constrained - and need to be protected.

Linus

Final note on this (promise):

I'd really be very interested to hear if the patch I *do* think makes sense (ie the removal of the old "unmapped_ratio" logic) actually already solves most of Bron's problems.

It may well be that that unmapped_ratio logic effectively undid the system configuration changes that Bron has done. It doesn't matter if Bron has

>From our sysctl.conf:
# This should help reduce flushing on Cache::FastMmap files vm.dirty_background_ratio = 50 vm.dirty_expire_centisecs = 9000
vm.dirty_ratio = 80
vm.dirty_writeback_centisecs = 3000

if it turns out that the "unmapped_ratio" logic turns the 80% back down to 5%.

It may well be that 80% of the non-highmem memory is plenty good enough!

Sure, older kernels allowed even more of memory to be dirty (since they didn't count dirty mappings at all), but we may have a case where the fact that we discount the HIGHMEM stuff isn't the major problem in itself, and that the dirty_ratio sysctl should be ok - but just gets screwed over by that unmapped_ratio logic.

So Bron, if you can test that patch, I'd love to hear if it matters. It may not make any difference (maybe you don't actually trigger the unmapped_ratio logic at all), but I think it has the potential for being totally broken for you.

People that don't change the dirty_ratio from the default values would generally never care, because the default dirty-ratio is *already* so low that even if the unmapped_ratio logic triggers, it won't much matter!

Linus

I think that (c) is doable. If its worth the effort, who knows, apparently there still are people using 32bit kernels on boxen with mucho memory.

Agreed, the per mapping thing was utter crap.

But this problem is already an issue, Anton recently had a case where a 12GB highmem box locked up due to NTFS running out of lowmem - or something like that.

And I think that with the targeted slab reclaim (or slab defrag as its apparently still called) we can properly fix this side of the problem. I think Rik was looking into doing so.

Yeah. I always considered HIGHMEM to just be unusable. It's ok for extending to 2-4GB (ie HIGHMEM4G, not 64G), and it's probably borderline usable for 4-8G if you are careful.

But quite frankly, I refuse to even care about anything past that. If you have 12G (or heaven forbid, even more) in your machine, and you can't be bothered to just upgrade to a 64-bit CPU, then quite frankly, *I* personally can't be bothered to care.

That's my personal opinion, and I realize that some of the commercial vendors may care about their insane customers' satisfaction, but I'm simply not interested in insane users. If they have that much RAM (and bought it a few years ago when a 64-bit CPU wasn't an option), they can't be poor.

So the _only_ explanation today for 12GB on a 32-bit machine is
(a) insanity
or
(b) being so lazy as to not bother to upgrade
and in either case, my personal reaction is "I'm *not* crazy, and yes, I'm lazy too, and I can't give a rats *ss about those problems".

HIGHMEM was a mistake in the first place. It's one that we can live with, but I refuse to support it more than it needs to be supported. And 12GB is *way* past the end of what is worth supporting.

Linus

How about...

c) they bought it at the beginning of a project and are stuck with it because they aren't getting any more money for hardware

d) they've shipped it to the field and have to support it

We've got some 32-bit 8GB boxes for which both of these would hold true.

Chris

Still not enough of a reason for me to care.

Remember - I'm the guy who refused to merge RH's 4G:4G patches because I thought they were an unsupportable nightmare.

I care a lot about future supportability, and HIGHMEM is there purely as a temporary wart and blip on the screen.

I did acknowledge that others may care more, but the fact is, I suspect that it's going to be cheaper to literally buy and ship a new machine to a customer than to really "suppport" it in any other form.

Side note: HIGHMEM64G works perfectly fine with 12GB of RAM under *limited*loads*. If your customer does certain well-defined and simple things that don't put huge and varied loads on the VFS or VM layer, then 12GB+ is probably fine regardless.

Linus

quite a bit of money on the big RAM (for it's day) + lots of 15k RPM SCSI drives + multi-year support contracts. They're highly IO bound, and barely use 10-20% of their old 2.4Ghz Prestonia Xeon CPUs. It's hard to justify junking those machines < 5 years.

We have a couple of 6G machines and some 8G machines using PAE. On the whole, they actually have been working really well (hmmm, apart from the recent dirty pages issue + reiserfs data=journal leaks + inodes in lowmem limits)

Rob

Junk everything except the 15K drives, you will be glad you did. Too bad about those multi-year support contracts, hopefully you got a deal on them.

Prediction: after these dirty pages issues are gone, there will be more dirty page issues because the notion of dirty page limit is fundamentally broken. Your smartest recourse is to re-motherboard to a place where the dirty page limit borkage does not hurt as much, and in the process you will get a cheap hardware upgrade. Everybody will be happy, the sun will come out, the birds will sing.

Regards,

Daniel

Actually, the 15K drives are the bit we're getting the least use out of now, since we're moving everything to external SATA units that are more easily swapable.

Or just keep running 2.6.16 where it's all been working quite fine thanks very much, or maintain a simple patch that rips all that out since we don't care too much about "fairness" - we only run a couple of things on that machine and they run fine.

Bron ( going to settle down and really test this stuff to make sure
we have an acceptable "fix" for us then do it! )

12GB-16GB worked well historically so its a regression. Above 16GB its all utterly mad.

You forgot reason (c) though

(c) 32bit is a tested approved certified etc environment - essentially conservativsm and paranoia, and its hard to explain to some of these people that the right answer really is less RAM or 64bit, especially as they may already know it but have a 12 month process to prove and certify a system configuration.

Highmem to 4GB was sensible. Highmem to 8GB was pushing it.

Alan

Just around the corner...

$ ftp ftp
Connected to ftp.gwdg.de.
220-====================================================================
220-Gesellschaft fuer wissenschaftliche Datenverarbeitung mbH Goettingen
220-====================================================================
220-This is a Linux PC (Dell PE-2650, 2 CPUs P4/2800, 12 GB RAM)
220-running SuSE-Linux-8.2 with SuSE kernel 2.4.20-64GB-SMP.

There is no reason to upgrade the hardware - if it works, hey good then. And I am pretty sure that a few 2 GB sticks are cheaper than a big opteron (if you only go by that). It sure is now - and probably even back then.

On Thu, Nov 15, 2007 at 01:14:32PM -0800, Linus Torvalds wrote:

Sorry about not replying to this earlier. I actually got a weekend away from the computer pretty much last weekend - took the kids swimming, helped a friend clear dead wood from around her house before the fire season. Shocking I know.

Hey, if Andrew Morton can tell us we find all the interesting bugs, you can call me odd. I've been called worse!

We also run ReiserFS (3 of course, I tried 4 and it et my laptop disk) on all our production IMAP servers. Tried ext3 and the performance was so horrible that our users hated us (and I hated being woken in the night by things timing out and paging me). And I'm spending far too long still writing C thanks to Cyrus having enough bugs to keep me busy for the rest of my natural life if I don't break and go write my own IMAP server at some point. *clears throat*

I'm going to finish off writing a decent test case so I can reliably reproduce the problem first, and then go compile a small set of kernels with the various patches that have been thrown around here and see if they solve the problems for me.

Thankfully I don't have the same problem you do Linus - I don't care if any particular patch isn't consistent - isn't fair in the general sense
- even "doesn't work for anyone else". So long as it's stable and it works on this machine I'm happy to support it through the next couple of years until we either get a world facing 64 bit machine with the spare capacity to run DCC or we drop DCC.

The only reason to upgrade the kernel there at all is keeping up-to-date with security patches, and the relative tradeoffs of backporting (or expecting Adrian Bunk to keep doing it for us) rather than maintaing a small patch to keep the behaviour of one thing we like.

And to all of you in this thread (especially Linus and Peter) - thanks heaps for grabbing on to a throw away line in an unrelated discussion and putting the work in to:

a) explain the problem and the cause to me before I put in heaps of work tracking it down; and

b) putting together some patches for me to test.

I saw a couple of days ago someone posted an "Ask Slashdot" whether 6 weeks was an appropriate time for a software vendor to get a fix out to a customer and implying that the customer was an unrealistic whiner to expect anyone to be able to do better. I'll be able to point to this thread if anyone suggests that you can't get decent support on Linux!

Bron.

A 32 bit machine with HIGHMEM64 enabled running DCC has an MMAPed file of approximately 2Gb size which contains a hash format that is written "randomly" by the dbclean process. On 2.6.16 this process took a few minutes. With lowmem only accounting of dirty ratios, this takes about 12 hours of 100% disk IO, all random writes.

This patch includes some code cleanup from Linus and a toggle in /proc/sys/vm/dirty_highmem which can be set to 1 to add the highmem back to the total available memory count.

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c
===================================================================
@@ -70,6 +70,12 @@ static inline long sync_writeback_pages(
 int dirty_background_ratio = 5;
 
 /*
+ * free highmem will not be subtracted from the total free memory
+ * for calculating free ratios if vm_dirty_highmem is true
+ */
+int vm_dirty_highmem;
+
+/*
  * The generator of dirty data starts writeback at this percentage
  */
 int vm_dirty_ratio = 10;

@@ -153,7 +159,8 @@ static unsigned long determine_dirtyable
 	x = global_page_state(NR_FREE_PAGES)
 		+ global_page_state(NR_INACTIVE)
 		+ global_page_state(NR_ACTIVE);
-	x -= highmem_dirtyable_memory(x);
+	if (!vm_dirty_highmem)
+		x -= highmem_dirtyable_memory(x);
 	return x + 1;	/* Ensure that we never return 0 */
 }
 

@@ -163,20 +170,12 @@ get_dirty_limits(long *pbackground, long
 {
 	int background_ratio;		/* Percentages */
 	int dirty_ratio;
-	int unmapped_ratio;
 	long background;
 	long dirty;
 	unsigned long available_memory = determine_dirtyable_memory();
 	struct task_struct *tsk;
 
-	unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
-				global_page_state(NR_ANON_PAGES)) * 100) /
-					available_memory;
-
 	dirty_ratio = vm_dirty_ratio;
-	if (dirty_ratio > unmapped_ratio / 2)
-		dirty_ratio = unmapped_ratio / 2;
-
 	if (dirty_ratio < 5)
 		dirty_ratio = 5;
 

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
===================================================================
@@ -92,6 +92,7 @@ void throttle_vm_writeout(gfp_t gfp_mask
 
 /* These are exported to sysctl. */
 extern int dirty_background_ratio;
+extern int vm_dirty_highmem;
 extern int vm_dirty_ratio;
 extern int dirty_writeback_interval;
 extern int dirty_expire_interval;

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/sysctl.c
===================================================================
@@ -776,6 +776,7 @@ static ctl_table kern_table[] = {
 /* Constants for minimum and maximum testing in vm_table.
    We use these as one-element integer vectors. */
 static int zero;
+static int one = 1;
 static int two = 2;
 static int one_hundred = 100;
 

@@ -1066,6 +1067,19 @@ static ctl_table vm_table[] = {
 		.extra1		= &zero,
 	},
 #endif
+#ifdef CONFIG_HIGHMEM
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname	= "dirty_highmem",
+		.data		= &vm_dirty_highmem,
+		.maxlen		= sizeof(vm_dirty_highmem),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec_minmax,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+		.extra2		= &one,
+	},
+#endif
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/Documentation/filesystems/proc.txt
===================================================================
@@ -1229,6 +1229,18 @@ dirty_background_ratio
 Contains, as a percentage of total system memory, the number of pages at which
 the pdflush background writeback daemon will start writing out dirty data.
 
+dirty_highmem
+-------------
+
+Contains, as a boolean, a switch to allow highmem to be counted as
+part of the "available" memory against which the dirty ratios will be
+applied.
+
+Setting this to 1 can be useful on 32 bit machines where you want to make
+random changes within an MMAPed file that is larger than your available
+lowmem, however it is potentially dangerous and has serious bounce-buffer
+issues.
+
 dirty_ratio
 -----------------
 

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/Documentation/sysctl/vm.txt
===================================================================
@@ -18,6 +18,7 @@ files can be found in mm/swap.c.
 Currently, these files are in /proc/sys/vm:
 - overcommit_memory
 - page-cluster
+- dirty_highmem
 - dirty_ratio
 - dirty_background_ratio
 - dirty_expire_centisecs

@@ -36,10 +37,10 @@ Currently, these files are in /proc/sys/
 
 ==============================================================
 
-dirty_ratio, dirty_background_ratio, dirty_expire_centisecs,
-dirty_writeback_centisecs, vfs_cache_pressure, laptop_mode,
-block_dump, swap_token_timeout, drop-caches,
-hugepages_treat_as_movable:
+dirty_highmem, dirty_ratio, dirty_background_ratio, 
+dirty_expire_centisecs, dirty_writeback_centisecs, 
+vfs_cache_pressure, laptop_mode, block_dump, 
+swap_token_timeout, drop-caches, hugepages_treat_as_movable:
 
 See Documentation/filesystems/proc.txt
 

Just to verify - can you confirm that this "just fixes it" for you?

I think this is the right approach to take, and seems very safe (ie people who know that their loads are ok can just set the flag), but I do want to verify that there was nothing else going on, and that you now see the same performance as you did in 2.6.16?

The other alternative, of course, would be to simply allow the dirty percentages to be > 100%, but that's just *odd* ;)

Linus

Yes, toggling dirty_highmem "just fixes it" in all our tests. I hadn't tested it on the production machine yet - but I'm just installing it there now since it's been running fine on a less important machine for a few days now.

I did wonder about allowing the dirty percentage to go way up, but that would have cause "this one goes up to 110%" comments in the sysctl limits code and people would have thought I was childish. Can't have that. Much better to have "int one = 1" instead.

Bron.

One would expect that setting dirty_highmem to true would cause highmem to be accounted in dirty-memory calculations. However with this change reality is in fact the inverse of that.

So how about this?

 Documentation/filesystems/proc.txt |    4 ++--
 mm/page-writeback.c                |    8 ++++----
 2 files changed, 6 insertions(+), 6 deletions(-)

diff -puN Documentation/filesystems/proc.txt~mm-add-dirty_highmem-option-fix Documentation/filesystems/proc.txt
@@ -1265,8 +1265,8 @@ Contains, as a boolean, a switch to allo
 part of the "available" memory against which the dirty ratios will be
 applied.
 
-Setting this to 1 can be useful on 32 bit machines where you want to make
-random changes within an MMAPed file that is larger than your available
+Setting this to 0 (false) can be useful on 32 bit machines where you wish to
+make random changes within an MMAPed file that is larger than your available
 lowmem, however it is potentially dangerous and has serious bounce-buffer
 issues.
 

diff -puN mm/page-writeback.c~mm-add-dirty_highmem-option-fix mm/page-writeback.c
@@ -69,10 +69,10 @@ static inline long sync_writeback_pages(
 int dirty_background_ratio = 5;
 
 /*
- * free highmem will not be subtracted from the total free memory
- * for calculating free ratios if vm_dirty_highmem is true
+ * free highmem will be subtracted from the total free memory for calculating
+ * free ratios if vm_dirty_highmem is true
  */
-int vm_dirty_highmem;
+int vm_dirty_highmem = 1;
 
 /*
  * The generator of dirty data starts writeback at this percentage

@@ -293,7 +293,7 @@ static unsigned long determine_dirtyable
 	x = global_page_state(NR_FREE_PAGES)
 		+ global_page_state(NR_INACTIVE)
 		+ global_page_state(NR_ACTIVE);
-	if (!vm_dirty_highmem)
+	if (vm_dirty_highmem)
 		x -= highmem_dirtyable_memory(x);
 	return x + 1;	/* Ensure that we never return 0 */
 }

_

(I dropped the already-merged part of your patch)

(I fixed a build error in kernel/sysctl.c: "one" was defined twice when suitable config options were set).

(It's an unpleasing patch, btw. But it's an unpleasant problem and at least this way people can tell us "hey, I did <this> and it started to work")

Actually, I'm confused now. Maybe I chose a bad name to begin with. Does it mean "I am allowed to dirty high memory" or "my high memory will be dirty if this is on"?

Hmm... I'm even having trouble articulating what's odd about it.

I guess my internal model was: "if this flag is set then you are allowed to make high memory dirty without needing to flush it immediately", which is why I made it that way around.

No - you're wrong. My patch _did_ include high memory in the dirty memory calculations when dirty_highmem was true.

This removes the high memory from the total count.

I think I got it right. If dirty_highmem is set to true, then don't subtract highmem from the total memory count before calculating the percentages. That's what I meant, and that's what the toggle did. Removed the subtraction.

Bron.

  Bron Gondwana
  [email protected]

But we're always allowed to dirty highmem - there'd be no point in having it otherwise. Hence the term dirty_highmem is confusing.

umm, really you want /proc/sys/vm/dont-account-highmem-in-dirty-memory-calculations, only shorter.

Do you agree?

If so, then it's still not a very pleasing interface - setting something to "true" to disable a particular piece of kernel behaviour implies a single negation which we don't really need.

It would be simpler to have /proc/sys/vm/do-account-highmem-in-dirty-memory-calculations, defaulting to "true" - this has no negations.

So... how about /proc/sys/vm/, umm.

<looks at inbox, brain explodes>

OK, I give up. Please see if you can think of something less confusing which involves no negations?

Thanks.

I still read dirty_highmem as:

/proc/sys/vm/do-account-highmem-in-dirty-memory-calculations

... so we're still talking one negative apart!

Well, the particular piece of kernel behaviour is already a negative:

"decrease the amount of memory allowed to get dirty so we never dirty
more than a percentage of available lowmem"

So what this flag is saying is:

"DON'T decrease the amount of memory allowed to get dirty down to just
the lowmem - dirty a percentage of total available including highmem"

As Linus said - the alternative of allowing more than 100% of lowmem to be dirty is just plain too wierd, hence this approach of allowing an option to turn off the new restriction that appeared since 2.6.16.

No, that's not true. The whole point is that between 2.6.16 and 2.6.20 the kernel behaviour changed. It currently doesn't count highmem in dirty memory calculations, which is why the memory pressure appears to be so great when actually there's still 4Gb of unused memory in the box.

/proc/sys/vm/do-account-highmem-in-dirty-memory-calculations would default to "false" to get the current behaviour post-2.6.16 kernels.

Setting a flag to make it true would stop the kernel subtracting highmem from the available count, giving the "old" behaviour of allowing a percentage of all the memory in the system to be dirty before forcing writeback.

I've spent a while thinking about this, and looking at the code.

I think this might be slightly clearer:

/proc/sys/vm/highmem_is_dirtyable - defaults to false

Here's how it would look in the code:

static unsigned long determine_dirtyable_memory(void) {
unsigned long x;

x = global_page_state(NR_FREE_PAGES)
+ global_page_state(NR_INACTIVE)
+ global_page_state(NR_ACTIVE);

if (!vm_highmem_is_dirtyable)
x -= highmem_dirtyable_memory(x);

return x + 1; /* Ensure that we never return 0 */
}

I think that's very clear.

"Unless highmem is dirtyable, subtract the otherwise dirtyable
pages from the total dirtyable memory count if they are in highmem"

Or without negatives:

"If highmem is dirtyable then include highmem pages in the
total dirtyable memory count"

Unfortunately this isn't expressed without negatives in the code because it's easier (I presume without looking at all the different zones on a cruddy piece of old i386) to add up the total and remove the pages in the HIGHMEM zone than to add up all the non-HIGHMEM zones separately.

Does that suit you? Does my explaination make sense?

Would you like me to re-submit the patch based on this? I'm certainly not happy with dirty_highmem as it is now in mm because it looks backwards and unclear to me.

Bron.

Well, it was quick enough to just do - here's the patch. I've also updated the documentation a bit to clarify the intention and the reasons why you might want to use it (based in part on the comments to the original change that made highmem uncountable for dirtyness purposes)

Tested and applied against 2.6.23.9 (our build script makes Debian packages from a clean unpack of kernel major plus patch minor plus svn checkout of out quilt series and apply regardless, so it was just as easy to bump the version number while I was at it).

Builds, boots, passes a quick run of the test program I used last time around.

Bron.

Add vm.highmem_is_dirtyable toggle

A 32 bit machine with HIGHMEM64 enabled running DCC has an MMAPed file of approximately 2Gb size which contains a hash format that is written randomly by the dbclean process. On 2.6.16 this process took a few minutes. With lowmem only accounting of dirty ratios, this takes about 12 hours of 100% disk IO, all random writes.

This patch includes some code cleanup from Linus and a toggle in /proc/sys/vm/highmem_is_dirtyable which can be set to 1 to add the highmem back to the total available memory count.

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/mm/page-writeback.c
===================================================================
@@ -70,6 +70,12 @@
 int dirty_background_ratio = 5;
 
 /*
+ * free highmem will not be subtracted from the total free memory
+ * for calculating free ratios if vm_highmem_is_dirtyable is true
+ */
+int vm_highmem_is_dirtyable;
+
+/*
  * The generator of dirty data starts writeback at this percentage
  */
 int vm_dirty_ratio = 10;

@@ -153,7 +159,10 @@
 	x = global_page_state(NR_FREE_PAGES)
 		+ global_page_state(NR_INACTIVE)
 		+ global_page_state(NR_ACTIVE);
-	x -= highmem_dirtyable_memory(x);
+
+	if (!vm_highmem_is_dirtyable)
+		x -= highmem_dirtyable_memory(x);
+
 	return x + 1;	/* Ensure that we never return 0 */
 }
 

@@ -163,20 +172,12 @@
 {
 	int background_ratio;		/* Percentages */
 	int dirty_ratio;
-	int unmapped_ratio;
 	long background;
 	long dirty;
 	unsigned long available_memory = determine_dirtyable_memory();
 	struct task_struct *tsk;
 
-	unmapped_ratio = 100 - ((global_page_state(NR_FILE_MAPPED) +
-				global_page_state(NR_ANON_PAGES)) * 100) /
-					available_memory;
-
 	dirty_ratio = vm_dirty_ratio;
-	if (dirty_ratio > unmapped_ratio / 2)
-		dirty_ratio = unmapped_ratio / 2;
-
 	if (dirty_ratio < 5)
 		dirty_ratio = 5;
 

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/include/linux/writeback.h
===================================================================
@@ -95,6 +95,7 @@
 extern int vm_dirty_ratio;
 extern int dirty_writeback_interval;
 extern int dirty_expire_interval;
+extern int vm_highmem_is_dirtyable;
 extern int block_dump;
 extern int laptop_mode;
 

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/kernel/sysctl.c
===================================================================
@@ -776,6 +776,7 @@
 /* Constants for minimum and maximum testing in vm_table.
    We use these as one-element integer vectors. */
 static int zero;
+static int one = 1;
 static int two = 2;
 static int one_hundred = 100;
 

@@ -1066,6 +1067,19 @@
 		.extra1		= &zero,
 	},
 #endif
+#ifdef CONFIG_HIGHMEM
+	{
+		.ctl_name	= CTL_UNNUMBERED,
+		.procname	= "highmem_is_dirtyable",
+		.data		= &vm_highmem_is_dirtyable,
+		.maxlen		= sizeof(vm_highmem_is_dirtyable),
+		.mode		= 0644,
+		.proc_handler	= &proc_dointvec_minmax,
+		.strategy	= &sysctl_intvec,
+		.extra1		= &zero,
+		.extra2		= &one,
+	},
+#endif
 /*
  * NOTE: do not add new entries to this table unless you have read
  * Documentation/sysctl/ctl_unnumbered.txt

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/Documentation/filesystems/proc.txt
===================================================================
@@ -1253,6 +1253,21 @@
 Data which has been dirty in-memory for longer than this interval will be
 written out next time a pdflush daemon wakes up.
 
+highmem_is_dirtyable
+--------------------
+
+Only present if CONFIG_HIGHMEM is set.
+
+This defaults to 0 (false), meaning that the ratios set above are calculated
+as a percentage of lowmem only.  This protects against excessive scanning
+in page reclaim, swapping and general VM distress.
+
+Setting this to 1 can be useful on 32 bit machines where you want to make
+random changes within an MMAPed file that is larger than your available
+lowmem without causing large quantities of random IO.  Is is safe if the
+behavior of all programs running on the machine is known and memory will
+not be otherwise stressed.
+
 legacy_va_layout
 ----------------
 

Index: linux-2.6.23.8-reiserfix-fai-vmdirty/Documentation/sysctl/vm.txt
===================================================================
@@ -22,6 +22,7 @@
 - dirty_background_ratio
 - dirty_expire_centisecs
 - dirty_writeback_centisecs
+- highmem_is_dirtyable   (only if CONFIG_HIGHMEM set)
 - max_map_count
 - min_free_kbytes
 - laptop_mode

@@ -36,10 +37,10 @@
 
 ==============================================================
 
-dirty_ratio, dirty_background_ratio, dirty_expire_centisecs,
-dirty_writeback_centisecs, vfs_cache_pressure, laptop_mode,
-block_dump, swap_token_timeout, drop-caches,
-hugepages_treat_as_movable:
+dirty_ratio, dirty_background_ratio, dirty_expire_centisecs, 
+dirty_writeback_centisecs, highmem_is_dirtyable,
+vfs_cache_pressure, laptop_mode, block_dump, swap_token_timeout,
+drop-caches, hugepages_treat_as_movable:
 
 See Documentation/filesystems/proc.txt
 

mmap: mmap call failed: errno: 12 errmsg: Cannot allocate memory

Yep, that's "fixed" the problem alright! No way this puppy is dirtying 2Gb of memory any more.

http://linux.brong.fastmail.fm/2007-11-22/bmtest.pl

That said, pushing the size down to 1700 rather than 2000 in that file makes it run, and the behaviour matches the 2000 Mb case on 2.6.16.55 rather than 2.6.20.20 or 2.6.23.1 (my other test case kernels that happened to be pre-built on that machine)

[root@lb1 ~]$ free
total used free shared buffers cached

Mem:       4149836    2073056    2076780          0      22036    1846096
-/+ buffers/cache:     204924    3944912
Swap:      2096472          0    2096472

That's after running the 1700Mb version. You can see this machine is our one remaining 4Gb machine (it's not running any production services unlike the 6Gb machine, so it's better for testing)

Anyway - looks like this may be a "good enough" solution for out1 if it can manage an ~2Gb file with 6Gb of memory available. I'll test that later today - but I should drag myself into the office now...

Bron.

(patch left attached below for reference)

Alternatively perhaps I'm just a moron who used a config file with:
CONFIG_PAGE_OFFSET=0x80000000 set to build the new kernel (I hadn't committed it because it turned out not to solve the issue it was there for). That would explain a few things.

[root@lb1 perl]$ free
total used free shared buffers cached

Mem:       4150620    2272284    1878336          0      11212    2066536
-/+ buffers/cache:     194536    3956084
Swap:      2096472          0    2096472

That's more the usage I would expect to see.

Now for the downside. It works again, but it still runs slow. Seems to hit (and this is totally unscientific, I'm just watching the numbers scroll by) at about 120000 writes rather than 70000 writes, but that's still not fitting the while file dirty.

I notice that PF_LESS_THROTTLE gets set by nfsd to get an extra 25% bonus free space allocated. Potentially dcc could use similar tricks to claim extra space if that knob is available up in userspace. I'm happy to patch dcc as well if I have to, I'm already backporting it, so adding another little quilt directory and applying it is pretty trivial (must try guilt/stgit one of these days)

Bron.

rant on :) ... These aren't directed specifically at Andrew, but everyone who merges patches or is involved in the release process.

Not if you said their regression causing patches will get reverted unless it can be fixed for release. For everyone not involved, they'll be doing their own productive things and we don't want to stop that.

I don't know why this would be better than the above. If you are worried about perception of released kernels, then it would actually be worse, because the non-bugfix-only releases will get out there, and the numbering scheme gets yet another level of complexity.

Make 2.6.24-rc3 your bugfix only release. If you argue that would make the cycle too long, the solution is to merge less stuff in 2.6.24-rc1.

If you don't allow regressions to build up over releases in the first place, then this naturally gets done for you.

That's pretty annoying. Everybody tries not to introduce regressions. It's just a natural part of development. You just have to make the system work well within that constraint.

It's easy: "if you add a regression, you fix it. If it doesn't get fixed, we either don't release or we revert your patch."

That takes care of regressions. Now for actually improving quality. I think to do that you have to encourage code review and bug fixing.

- Take reviewers seriously, and don't allow patches to be merged if they have outstanding unaddressed comments (unaddressed doesn't have to mean changed completely to reviewers taste, but at least answered questions and provided rationale). Don't merge unreviewed patches.

- Prioritise bugfixes and regression fixes. I realise they can be very complex changes, but I am disappointed sometimes at how some of my bugfix attempts are received. Things like silent and un-reproduceable pagecache corruption, memory ordering bugs which will likewise cause silent and un-reproduceable data corruption are totally unacceptable IMO; worse than most simple (eg. fail-stop, performance) regressions. To see them struggle because the patch isn't perfect right off the bat, they're deemed too hard, or they clash with some pending "feature" is a pity to me. I don't expect thanks or even help, but at least having some priority would help me stay interested and the bug to get fixed quicker.

For a concrete example: when I did the remap_vmalloc_range() patch, I converted over all in-tree drivers to use the range checking and much safer remap_vmalloc_range. During this conversion I found several places where drivers were potentially inserting mappings wrongly or not checking user inputs properly (eg. corruption and/or security issues). And tried to fix them. And yet the whole patchset was dropped, despite being simpler and cleaner code, and attempting to fix bugs. Because I wasn't able to test the hardware (and/or maintainers didn't respond). It didn't even get a chance in -mm (and if it had, it probably would have been immediately dropped if I had a typo somewhere). Those patches are still rotting on my hdd somewhere... So it is actually far easier for me to shut my eyes and not bother reviewing any in-tree code when introducing a new API or something like that.

And yet something like CFS basically bypasses the review and test process entirely and gets upstream at least a release too soon. That isn't anything against the technical side of CFS development, which I'm a lot happier with than the old scheduler. It is a problem with the merging and release process.

- "if you added a regression, you should fix some other bug too" rule should be "if you want to add a new feature, you should review a number of other patches". Review points can be traded. If this could be done right, it would create a direct economic incentive to review.

OK, the last one is my pipedream that will never happen ;) But you should understand what can be achieved, what can't, and what will be the most productive way to change the system.

Well I don't think I need to agree that the kernel is getting worse in order to think there are ways we can change the system to help it get better. We can make changes for the better without making things more difficult for anybody, IMO.

But is the kernel getting worse? I wonder if it has anything to do with getting further away from large distros? (and thus will get better again during the next distro release cycle). Because I know SLES9 took a huge amount of work to get stable. Or do you think it is getting worse since 2.6.0?

To summarise, I think don't actually worry about how the developers spend their time. All you have to do is control what patches you take and release policy. Improve the system so it tolerates regressions and poor quality code (which are inevitable).

Review bandwidth and regression fix rate should just naturally throttle new features / code going in. And you encourage people to do good things rather than hit them when inevitable bad things happen.

Actually, I'm pretty happy reverting patches that cause regressions even if it *can* be "fixed for release". If there isn't a fix available within a day or two, it should get reverted.

The "fix" can then be re-applying the *fixed* patch - and at that point we should strive to require the person who re-submits the patch (with fixes) having to have an Ack from the person who found the problem in the first place, so that it's verified to actually fix things!

So I really would encourage people to send me emails like

Please revert commit xyz, because it breaks abc, and there is no fix available even though this was reported x days ago.

I have verified that revert just that change fixes the issue.

and just make the "because it breaks abc" be specific and clear enough that I go "Ahh, ok, I'd better revert it".

Also, please notice the latter part of the suggestion above: even if somebody has bisected down their problem to a specific commit, I really *do* want to hear that actually undoing the commit on top of the current tree acually fixes it again, because sometimes that just isn't the case - sometimes you end up having various interactions that means that reverting a commit might simply not even work.

I have no trouble at all with reverting commits in general. I think regressions are serious.

So the problematic cases are the cases where:

- the commit no longer reverts cleanly, or just otherwise introduces other infrastructure that other commits that already got merged depend on.

So sometimes you actually need a patch along with the revert (Andrew does that kind of thing anyway, since he works with patches regardless, so the "needs a patch" case is obviously not limited to just the problem cases)

- more commonly: it's not entirely clear which commit actually caused the problem.

but I *do* want to encourage people to revert (and ask other people to revert) much more aggressively. I personally try to revert things that people report regressions to me about very actively, unless I know somebody already has or is working on a fix.

Linus

Ok. drivers/net/skge has been broken for several weeks. I have manually fixed the driver at each rc* release since then.

Please revert skge changes, the commit that broke driver is 7fb7ac241162dc51ec0f7644d4a97b2855213c32

See

	http://bugzilla.kernel.org/show_bug.cgi?id=9321

for more information. I think Stephen Hemminger is working on the skge fix, but it has been several days since I've heard anything from him.

Heikki Orsila			Barbie's law:
[email protected]		"Math is hard, let's go shopping!"
http://www.iki.fi/shd

That's a pretty unhelpful thing. It doesn't describe the breakage at all, so there is hardly much "more info".

You've also apparently made all the attachements "octet-streams", so they are singularly painful to look at (ie no normal web-browser will show them, you have to save them to a file and look at them there).

That said, I think I'll revert it, since it certainly fits the bill, but that's a really quite unreadable bug-report.

I finally found the actual description of the problem (by following multiple links), but really, if people want me to revert things, I would *strongly* suggest they make it *obvious* what's going on in the email to me that says "please revert".

Because quite frankly, if that email doesn't explain why something should be reverted (and just points to other things), it doesn't really cut it.

Linus

Strongly agree.

This is exactly what happened to that ARM NO_HZ bug report. The report in bugzilla was rather lacking (and wrong) in ways that have already been described. HPET on ARM? 8)

Then on the morning of 6th November, someone reported on the mailing list that "pxa270 doesn't work with oneshot timer" and that was the trigger to getting the bug resolved - because it was a narrowly defined bug report.

Since it was a narrowly defined bug report, it became very easy to investigate and resolve. About half an hour of time for an initial patch.

There's another issue I want to raise concerning bugzilla. We have the classic case of "not enough people reading bugzilla bugs" - which is one of the biggest problems with bugzilla. Virtually no one in the ARM community looks for ARM bugs in bugzilla.

Let's not forget that it would be a waste of time for people to manually check bugzilla for ARM bugs. There's soo few people reporting ARM bugs into bugzilla that a weekly manual check by every maintainer would just return the same old boring results for months and months at a time.

It would be far more productive if the ARM category was deleted from bugzilla and the few people who use bugzilla reported their bugs on the mailing list. We've a couple of thousand people on the ARM kernel mailing list at the moment - that's 3 orders of magnitude more of eyes than look at bugzilla.

(I'm not saying that if the ARM NO_HZ bug as reported in bugzilla had been reported on the correct mailing list would've been solved earlier; I doubt there'd be much difference. However, the probability of a question being asked of the reporter would've been much higher, and _that_ might have led to an earlier resolution.)

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

What about having all ARM bugs in Bugzilla by default assigned to [email protected]? [1]

cu
Adrian

[1] Either directly or through a pseudo address, but that's just a technical detail.

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

That would also work, probably much better than setting up yet another list.

My experience of trying to get mbligh to do this when I stopped looking after PCMCIA stuff was *extremely* painful. Wonder if it's become any easier of late?

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

cpufreq (at least) does it this way. I don't know how well it is turning out in practice.

It's useful if the initial report makes it clear (ie; to me) that the report has already gone to a mailing list so I don't go and forward a duplicate.

But there are so few arm reports in bugzilla that this is all rather moot.

He's a bad, bad man ;)

But he's been turning these things around pretty rapidly lately.

What about having all ARM bugs in Bugzilla by default assigned to [email protected]? [1]

cu
Adrian

[1] Either directly or through a pseudo address, but that's just a technical detail.

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Nor should they.

I screen all bugzilla reports. 100% of them.

- I'll try to establish whether it is a regression

- I'll solicit any extra information which I believe the reveloper will need

- I'll ensure that an appropriate developer has seen the report

And yes, the number of arm-specific reports in there is very small.

Is that [email protected]?

If so, MANITAINERS claims that it is subscribers-only. That would cause some bug reporters to give up and go away.

So what you're saying is...

that just because you do this everyone in a select clique, who you include me in, should be doing this as well.

No. Thank. You.

Yes.

Find some other mailing list; I'm not hosting *nor* am I willing to run a non-subscribers only mailing list. Period. Not negotiable, so don't even try to change my mind.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

No, I don't mean that at all and this was very plainly obviously from my very clearly written email. Let me try again.

No, no subsystem developer needs to monitor new bugzilla reports. This is because *I do it for them*. I will actively make them aware of new reports which I believe are legitimate and which contain sufficient information for them to be able to take further action.

Making a list subscribers-only will cause some bug reports to be lost.

Tradeoffs are involved, against which decisions must be made. You have made yours.

If you screen all bugzilla reports then you'll know that bug #9356 arrived at about 1400 GMT yesterday. It's hardly surprising then that your utterly crappy responses to Natalie's message (which, incidentally, wasn't copied to me) sent within 24 hours of that report cause *great* annoyance.

On the whole you do an excellent job with feeding the bug reports to people, and while I recognise that you're only human, things do occasionally go wrong. For instance, sending clearly marked Samsung S3C bugs to me rather than Ben Dooks (who's in MAINTAINERS for those platforms.)

So how are they lost when they're held in a moderation queue and are either accepted, a useful response given to the original poster, or are forwarded to someone who can deal with the issue.

I don't think "subscribers only" describes my lists - we don't devnull stuff just because the poster is not a subscriber.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Well whatever, sorry. But this is in the noise floor. Point is: many bug reports aren't being attended to.

Oh, OK, as long as there really is a human paying attention to those things then that's fine. When one is on the sending end of these things one never knows how long it will take, not whether it will even happen.

From: Andrew Morton <[email protected]>
Date: Tue, 13 Nov 2007 14:32:01 -0800

Russell doesn't have to worry any more, he doesn't have to host it, and he doesn't have to be willing to run a non-subscribers-only mailing list.

Because I am.

I've created [email protected]

Enjoy.

Let me just say - I'm astonished at how little spam gets though the vger lists. Considering how many times those email addresses must have been added to spam databases.

It must be a lot of work, and whoever is doing it does it well.

I don't even know. Is it Matti? You?

<contemplates [email protected]. Shudders.>

From: Andrew Morton <[email protected]>
Date: Tue, 13 Nov 2007 18:27:00 -0800

Matti gets all the credit for setting up the bayesian et al. filters we have and training it as needed.

Yes, sourceforge is a complete joke.

Martin's changed the owner for ARM bugs last night to the mailing list so the whole issue is now redundant.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

By doing so you've just said (implicitly) that you can not tolerate someone having a different opinion from your own.

While I accept *your* right to run *your* lists how you please, you are unable to accept *my* right to run *my* lists how I see fit.

Time will tell which lists will survive. Whatever, I suspect that by doing what you've just done, you're going to create more confusion and problems. Instead of having one focused place for discussions and bug reports, they're going to be spread more thinly, meaning less people looking at such things, meaning more bugs get ignored. Thus making the issue worse.

So, when are you creating a replacement alsa-devel mailing list on vger? That's also subscribers-only.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

From: Russell King <[email protected]>
Date: Wed, 14 Nov 2007 09:55:07 +0000

I created a mailing list on a machine where I provide such services.

People can choose to use or not use the new list, it is their choice.

I didn't tell you to take your list down or to run it in some other way. I didn't tell you to unsubscribe everyone and move them over to the new list either.

I've provided an alternative, and people can pick and choose how they see fit. I'm letting natural selection run it's course. Are you able to cope with the fact that people might not want to use your list any longer? Perhaps that is what bugs you so much about my giving people a alternative choice.

The operative term is "alternative" rather than "replacement". Perhaps this misunderstanding is what you're so upset about.

And yes, that alsa list bugs the crap out of me too. I'm more than happy to provide an alternative for that one as well.

In fact, *poof*, there it is, [email protected] is there and available for anyone who wants to use it.

Have a nice day Russell.

On 14-11-07 11:07, David Miller wrote:

Added Jaroslav and Takashi to the already extensive CC....

[email protected] is not subscriber-only. Same as that arm list, it's _moderated_ for non-subscribers and given that I and other moderators have been doing our best to moderate quickly (I tend to stay logged in to the moderation interface all day for example) what specifically bugged the crap out of you? It's not something a poster needs to concern himself with.

Also for alsa-devel the moderators tend to add any valid non-subcribers to a whitelist after landing in the queue the first time meaning even a delay is just a one-time thing normally. So what's the trouble? Basically, noone need even notice...

Not that I think that moving alsa-devel over to vger wouldn't be a good idea mind you; when the list moved from sourceforge, asking you to host it was my preferred option. I do somewhat suspect that Jaroslav would like to keep the alsa-devel@ name (and I'd like to ask you to then also host alsa-user@) and would then rewrite mail to those lists @alsa-project.org to vger.

But what is the problem you speak of with the alsa-devel list? While I would not mind loosing it, moderation hasn't been overly laborious and I'm not aware of any serious problems.

Rene.

From: Rene Herman <[email protected]>
Date: Wed, 14 Nov 2007 12:46:24 +0100

The fact that it farts at me every time I post to this thread. That's rude and annoying.

That sucks for new people taking part in the conversation.

There is no reason for moderation at all, it isn't necessary for spam prevention and it does nothing but annoy new posters and make work for the moderator.

From: David Miller <[email protected]>
Date: Wed, 14 Nov 2007 03:56:57 -0800 (PST)

See? I got another one and I have received at least 10 of the following over the past 2 days.

That's rediculious.

And because a human adds the whitelist this is always going to happen to someone when they start posting to the alsa list for the first time.

/me gets ready for the 11th copy in response to this one...

--------------------
Subject: Your message to Alsa-devel awaits moderator approval
From: [email protected]
To: [email protected]
Date: Wed, 14 Nov 2007 12:57:06 +0100
Sender: [email protected]

Your mail to 'Alsa-devel' with the subject

Re: [alsa-devel] [BUG] New Kernel Bugs

Is being held until the list moderator can review it for approval.

The reason it is being held:

Too many recipients to the message

Either the message will get posted to the list, or you will receive notification of the moderator's decision. If you would like to cancel this posting, please visit the following URL:

    http://mailman.alsa-project.org/mailman/confirm/alsa-devel/12dd3bd077bbf9cd142f214beae6d22ae43b53aa

At Wed, 14 Nov 2007 04:01:31 -0800 (PST),

... if you give too many recipients in your post. That is often really annoying thing to me, together with keeping the unrelated subject line ;)

I personally don't care whether it's a moderated or open list. We chose it simply due to too bad S/N ratio at that time. So, if the current list annoys your or many others and the list management on vger is so good, it'd be basically a good move, of course. I'll appreciate it.

The only confusion would be the change of ML address, but we can do it slowly, too.

Takashi

I'd love the lists at vger. Amazing spam-filtering. I'd like to request the name [email protected] (and [email protected] if at all possible so we can open that one up as well) though.

There wouldn't need to be a forced ML address change if Jaroslov would then just rewrite alsa-{devel,user}@alsa-project.org to vger.kernel.org same as he did for alsa-devel and does for alsa-user to @lists.sf.net.

Rene.

At Wed, 14 Nov 2007 13:21:30 +0100,

I think alsa-user can stay as is. It's no place for dragging many other addresses like alsa-devel.

BTW, I also prefer keeping the name alsa-devel@. It's been so.

If it works, then I'm for it, too.

thanks,

Takashi

From: Takashi Iwai <[email protected]>
Date: Wed, 14 Nov 2007 10:47:39 +0100

That's fine with me, I've changed it [email protected]

Great, thanks. Jaroslav -- given that this list won't need moderation I'd consider it the main/only alsa-devel. The alsa-devel subscriber database was cleansed only a couple of months ago when moving from sourceforge so it should now be okay to just transfer all subscriptions.

Or maybe you're already moving things; mailman.alsa-project.org seems to be down at least....

Rene.

[Unrelated lists cut from Cc]

At Wed, 14 Nov 2007 15:23:15 -0800 (PST),

Great, thanks David!

I believe it'd be really better to move alsa-devel to vger. The spam filters on vger does obviously better (faster) job than human being. We'll be released from some labors at least.

Jaroslav, let us know your opinion. alsa-devel ML is being hosted on your machine right now, so it's your decision...

thanks,

Takashi

I would like to keep mailing list history in mailman, so I subscribed [email protected] to [email protected] mailing list.

David, could I send you the list of subscribers on the alsa-devel list (in a private e-mail) to move mailing list to vger without bothering all users to resubscribe again (we did it several months ago, so the subscriber list is quite clean)? I will redirect [email protected] address then to [email protected] (if it does not break any vger rules or spam filtering on vger) to preserve alsa-devel list e-mail.

Thanks,
Jaroslav

-----
Jaroslav Kysela <[email protected]>
Linux Kernel Sound Maintainer
ALSA Project

From: Jaroslav Kysela <[email protected]>
Date: Thu, 15 Nov 2007 14:56:38 +0100 (CET)

We don't do this because:

1) If the user doesn't do the subscribe, they won't know how to unsubscribe.

If you think this isn't an issue, after 10 years of co-running vger I can tell you it's a huge issue as they just spam the list owner, with requests to be removed.

2) Users generally get upset when you subscribe them to a site they did not get asked to be subscribed to. And yes this is true even if it's the same mailing list just at a new location.

I apply this rule unilaterally to all new mailing lists hosted at vger, I'm not just singling out this case, believe me :-)

So please ask the folks on the existing list to subscribe to the new one.

Thank you.

Not trying to sound like a jerk or an idiot, but if you require us to resubscribe, why don't you post the link in your email? It would make our lives that much easier.

Thanks,

Tobin

From: Tobin Davis <[email protected]>
Date: Thu, 15 Nov 2007 13:57:18 -0800

Sure thing:

http://vger.kernel.org/majordomo-info.html#subscription

Enjoy.

That is:

mailto:[email protected]?body=subscribe alsa-devel

and then follow instructions in (ie, reply to) the confirmation mail you'll be sent.

Rene.

Nah, in this case you are not even getting them to not being a non-subcriber but due to too many CCs. I got one as well. That just needs to be disabled, does not have anything to do with non-subscribers (and you're in the white list) but is just a retarted bit of list configuration...

(no, I can't personally change it, needs Jaroslav Kysela)

Rene.

It certainly is. I only experienced that now due to the "too many recipients to message" moderation notice that I got from my own message.

Jaroslav -- please disable that junk or if possible, make it a "at most once per address per month" thing or somesuch. This is complete crap.

Yes there is. It's necessary for lists that do not have the human and other resouces behind it that vger does. alsa-devel was drowning in spam and dying as a result back when it was at sourceforge. Upon moving, my preference was to ask the lists to be hosted at vger but given that (it seems) Jaroslav wanted to keep them locally, moderation was very necessary. I moderate out quite a bit of spam every day.

vger is doing an amazing job at spam filtering -- if it's an option to move to vger, than sure, no need. But otherwise, the "no need" needs a list admin with enough bandwidth and skill.

As to the "new people": it's not optimal, but (upto this thread I'll admit -- I woke up to a huge number of posts in the queue) it's not been a _real_ problem. alsa-devel is not high-volume enough for it to be.

Rene.

Totally unrelated - I sent something to the kolab mailing list a couple of days ago (it's moderated for non subscribers) informing them that I had found the cause of some Cyrus bugs that they had problems with in the past and providing a link to my post to the cyrus list with the patches attached.

It sat in the moderation queue and then was rejected with "non subscriber post to subscription only list". Not only was the reponse a day later when I had moved on to other things, but it got me really pissed off that I had put some effort into providing a good quality post that outlined the specific issues and how they applied to their project, and had been summarily dismissed, probably without the effort being put in.

There's no way for a non-subscriber to know in advance if the list they are trying to post to will do that to them, completely negating the effort put in to writing something worthwhile to inform that community. It's insular, and it sucks.

So yeah, my attitude now is that the Kolab folks can go screw themselves and track down the fix on their own or wait until I've convinced upstream to accept the fixes (likely) and they have moved to the new version (unlikely for a long time, and meanwhile they're missing out on the performance increases that having a more stable skiplist library would give them)

I'm sure if I had something that I considered worth informing the ALSA project of, I'd be wary of spending the same effort writing a good post knowing it may be dropped in between the by a list moderator just selecing all and bouncing them.

Bron.

Totally unrelated - I sent something to the kolab mailing list a couple of days ago (it's moderated for non subscribers) informing them that I had found the cause of some Cyrus bugs that they had problems with in the past and providing a link to my post to the cyrus list with the patches attached.

It sat in the moderation queue and then was rejected with "non subscriber post to subscription only list". Not only was the reponse a day later when I had moved on to other things, but it got me really pissed off that I had put some effort into providing a good quality post that outlined the specific issues and how they applied to their project, and had been summarily dismissed, probably without the effort being put in.

There's no way for a non-subscriber to know in advance if the list they are trying to post to will do that to them, completely negating the effort put in to writing something worthwhile to inform that community. It's insular, and it sucks.

So yeah, my attitude now is that the Kolab folks can go screw themselves and track down the fix on their own or wait until I've convinced upstream to accept the fixes (likely) and they have moved to the new version (unlikely for a long time, and meanwhile they're missing out on the performance increases that having a more stable skiplist library would give them)

I'm sure if I had something that I considered worth informing the ALSA project of, I'd be wary of spending the same effort writing a good post knowing it may be dropped in between the by a list moderator just selecing all and bouncing them.

Bron.

Totally unrelated - I sent something to the kolab mailing list a couple of days ago (it's moderated for non subscribers) informing them that I had found the cause of some Cyrus bugs that they had problems with in the past and providing a link to my post to the cyrus list with the patches attached.

It sat in the moderation queue and then was rejected with "non subscriber post to subscription only list". Not only was the reponse a day later when I had moved on to other things, but it got me really pissed off that I had put some effort into providing a good quality post that outlined the specific issues and how they applied to their project, and had been summarily dismissed, probably without the effort being put in.

There's no way for a non-subscriber to know in advance if the list they are trying to post to will do that to them, completely negating the effort put in to writing something worthwhile to inform that community. It's insular, and it sucks.

So yeah, my attitude now is that the Kolab folks can go screw themselves and track down the fix on their own or wait until I've convinced upstream to accept the fixes (likely) and they have moved to the new version (unlikely for a long time, and meanwhile they're missing out on the performance increases that having a more stable skiplist library would give them)

I'm sure if I had something that I considered worth informing the ALSA project of, I'd be wary of spending the same effort writing a good post knowing it may be dropped in between the by a list moderator just selecing all and bouncing them.

Bron.

Totally unrelated - I sent something to the kolab mailing list a couple of days ago (it's moderated for non subscribers) informing them that I had found the cause of some Cyrus bugs that they had problems with in the past and providing a link to my post to the cyrus list with the patches attached.

It sat in the moderation queue and then was rejected with "non subscriber post to subscription only list". Not only was the reponse a day later when I had moved on to other things, but it got me really pissed off that I had put some effort into providing a good quality post that outlined the specific issues and how they applied to their project, and had been summarily dismissed, probably without the effort being put in.

There's no way for a non-subscriber to know in advance if the list they are trying to post to will do that to them, completely negating the effort put in to writing something worthwhile to inform that community. It's insular, and it sucks.

So yeah, my attitude now is that the Kolab folks can go screw themselves and track down the fix on their own or wait until I've convinced upstream to accept the fixes (likely) and they have moved to the new version (unlikely for a long time, and meanwhile they're missing out on the performance increases that having a more stable skiplist library would give them)

I'm sure if I had something that I considered worth informing the ALSA project of, I'd be wary of spending the same effort writing a good post knowing it may be dropped in between the by a list moderator just selecing all and bouncing them.

Bron.

Totally unrelated - I sent something to the kolab mailing list a couple of days ago (it's moderated for non subscribers) informing them that I had found the cause of some Cyrus bugs that they had problems with in the past and providing a link to my post to the cyrus list with the patches attached.

It sat in the moderation queue and then was rejected with "non subscriber post to subscription only list". Not only was the reponse a day later when I had moved on to other things, but it got me really pissed off that I had put some effort into providing a good quality post that outlined the specific issues and how they applied to their project, and had been summarily dismissed, probably without the effort being put in.

There's no way for a non-subscriber to know in advance if the list they are trying to post to will do that to them, completely negating the effort put in to writing something worthwhile to inform that community. It's insular, and it sucks.

So yeah, my attitude now is that the Kolab folks can go screw themselves and track down the fix on their own or wait until I've convinced upstream to accept the fixes (likely) and they have moved to the new version (unlikely for a long time, and meanwhile they're missing out on the performance increases that having a more stable skiplist library would give them)

I'm sure if I had something that I considered worth informing the ALSA project of, I'd be wary of spending the same effort writing a good post knowing it may be dropped in between the by a list moderator just selecing all and bouncing them.

Bron.

Totally unrelated - I sent something to the kolab mailing list a couple of days ago (it's moderated for non subscribers) informing them that I had found the cause of some Cyrus bugs that they had problems with in the past and providing a link to my post to the cyrus list with the patches attached.

It sat in the moderation queue and then was rejected with "non subscriber post to subscription only list". Not only was the reponse a day later when I had moved on to other things, but it got me really pissed off that I had put some effort into providing a good quality post that outlined the specific issues and how they applied to their project, and had been summarily dismissed, probably without the effort being put in.

There's no way for a non-subscriber to know in advance if the list they are trying to post to will do that to them, completely negating the effort put in to writing something worthwhile to inform that community. It's insular, and it sucks.

So yeah, my attitude now is that the Kolab folks can go screw themselves and track down the fix on their own or wait until I've convinced upstream to accept the fixes (likely) and they have moved to the new version (unlikely for a long time, and meanwhile they're missing out on the performance increases that having a more stable skiplist library would give them)

I'm sure if I had something that I considered worth informing the ALSA project of, I'd be wary of spending the same effort writing a good post knowing it may be dropped in between the by a list moderator just selecing all and bouncing them.

Bron.

[ ... ]

Totally unrelated indeed so why are spouting crap? If the kohab list has a problem take it up with them but keep ALSA out of it. alsa-devel has only ever moderated out spam -- nothing else.

ene

That is incorrect. Hopefully it is the case now though, since my experience of the subject was years ago.

OG.

At Thu, 15 Nov 2007 14:17:27 +0100,

Yeah, it was really years ago that we once switched to the open list. Funny that people never forget such a thing :)

Takashi

At Thu, 15 Nov 2007 14:17:27 +0100,

Yeah, it was really years ago that we once switched to the open list. Funny that people never forget such a thing :)

Takashi

At Thu, 15 Nov 2007 14:17:27 +0100,

Yeah, it was really years ago that we once switched to the open list. Funny that people never forget such a thing :)

Takashi

At Thu, 15 Nov 2007 14:17:27 +0100,

Yeah, it was really years ago that we once switched to the open list. Funny that people never forget such a thing :)

Takashi

At Thu, 15 Nov 2007 14:17:27 +0100,

Yeah, it was really years ago that we once switched to the open list. Funny that people never forget such a thing :)

Takashi

At Thu, 15 Nov 2007 14:17:27 +0100,

Yeah, it was really years ago that we once switched to the open list. Funny that people never forget such a thing :)

Takashi

As an outsider to the list, how do I know what your policy will be other than "I've been rejected out of hand by someone else's list, so my experience is that member only lists aren't willing to listen to something I have to say unless I make the effort to sign up and have yet another folder accumulating unread messages". I don't.

Well, ok - maybe I do here since I've let myself be dragged in to the debate. Oops.

I get the same information from both project websites: "moderated for non-members, public archives" - no way of knowing that ALSA will accept me informing them of something they would be interested without committing to reading or bit-bucketing their list.

The alternative is to subscribe just long enough to send something and then unsubscribe again.... or cold-email a member and ask them to pass a message along. Or post and hope it doesn't get rejected, not even knowing for a day or so.

Bron.

Can you please just shelve this crap? You have a way of knowing that "ALSA will accept you" and that is knowing or assuming that the ALSA project doesn't consist of drooling retards.

When a project list goes to the difficulty of moderating non-subscribers it has made the explicit choice to _not_ become subscriber only. Then refusing valid non-subscribers after all makes no sense whatsoever. I'm sorry you got your feelings hurt by that other list but it was no doubt an accident; take it up with them.

Rene.

Well, my experience with moderation has been that moderated mails are stuck in some queue for weeks. Two seperate lists, neither of them was alsa. If also is doing a better job, great. But it still has to live with the general reputation of non-subscriber moderation.

Been there, done that. In spite of people not being drooling retards, the amount of time and effort they invest into either moderation or improving the ruleset is quite limited. Problems persist.

And even without mails being held hostage for weeks, every single moderation mail is annoying. Like the one I'm sure to receive after sending this out.

Jörn

Joern's library part 5:
http://www.faqs.org/faqs/compression-faq/part2/section-9.html

Certainly. Upto this thread I wasn't actually aware the list was doing that. While it might be informative once, getting it each time quickly gets old. Don't know if mailman can do anything like it but I'd suggest anyone running a non-subscriber-moderation list configure it to send such messages at most once a <time-period> per address or some such. And just disable the message if it cannot do that.

Fortunately, alsa-devel is (almost) no longer such a list anyway as it's moving to vger. Hurrah. David -- thanks.

Rene.

That is incorrect. Hopefully it is the case now though, since my experience of the subject was years ago.

OG.

That is incorrect. Hopefully it is the case now though, since my experience of the subject was years ago.

OG.

That is incorrect. Hopefully it is the case now though, since my experience of the subject was years ago.

OG.

That is incorrect. Hopefully it is the case now though, since my experience of the subject was years ago.

OG.

That is incorrect. Hopefully it is the case now though, since my experience of the subject was years ago.

OG.

I didn't say that you were.

Absolutely, and if you'd have read my message you'd have seen that I'd said effectively the same thing that you're saying here.

Having been flamed for not reading emails properly by AKPM shall I flame you for not reading my emails properly? Oh no, it's merely human to occasionally have such misunderstandings. Unless you're rmk.

Russell King
 Linux kernel    2.6 ARM Linux   - http://www.arm.linux.org.uk/
 maintainer of:

Hi Dave,

* David Miller <[email protected]> [071114 02:09]:

<snip>

Can you please use your *poof* trick one more time to set up [email protected]?

We've (as in linux-omap community) would like to move from subscriber only list at [email protected] to vger as we're starting to get more patches and comments on LKML. For related discussion on [email protected], see [1].

Regards,

Tony

[1] http://linux.omap.com/pipermail/linux-omap-open-source/2007-November/011980.html

From: Tony Lindgren <[email protected]>
Date: Fri, 16 Nov 2007 14:16:11 -0800

Done, enjoy.

Thanks!

Tony

The postmasters at vger is pretty good at running mailing lists. For linux-kbuild my effort so far has been to request it. Thats not a big deal.

So if they accept it you could have [email protected] for zero overhead for you.

Sam

And in a later mail I saw davem already created it.

Sam

And in a later mail I saw davem already created it.

Sam

From: Sam Ravnborg <[email protected]>
Date: Wed, 14 Nov 2007 06:56:06 +0100

I already did, get a little deeper in your mailbox before replying :-)

The postmasters at vger is pretty good at running mailing lists. For linux-kbuild my effort so far has been to request it. Thats not a big deal.

So if they accept it you could have [email protected] for zero overhead for you.

Sam

yes, yes, yes, and i agree with you that there is a problem. I tried to make this point at the 2007 KS: not only is degradation in quality not apparent for years, slow degradation in quality can give kernel developers the exact _opposite_ perception! (Fewer testers means fewer bugreports and that results in apparent "improved" quality and fewer reported regressions - while exactly the opposite is happening and testers are leaving us without giving us any indication that this is happening. We just dont notice.)

I'm not moaning about bugs that slip through - those are unavoidable facts of a high flux codebase. I'm moaning about reoccuring, avoidable bugs, i'm moaning about hostility towards testers, i'm moaning about hostility towards automated testing, i'm moaning about unnecessary hoops a willing (but unskilled) tester has to go through to help us out.

I tried to make the point that the only good approach is to remove our current subjective bias from quality metrics and to at least realize what a cavalier attitude we still have to QA. The moment we are able to _measure_ how bad we are, kernel developers will adopt in a second and will improve those metrics. Lets use more debug tools, both static and dynamic ones. Lets measure tester base and we need to measure _lost_ early adopters and the reasons why they are lost. Regression metrics are a very important first step too and i'm very happy about the increasing effort that is being spent on this.

This is all QA-101 that _cannot be argued against on a rational basis_, it's just that these sorts of things have been largely ignored for years, in favor of the all-too-easy "open source means many eyeballs and that is our QA" answer, which is a _good_ answer but by far not the most intelligent answer! Today "many eyeballs" is simply not good enough and nature (and other OS projects) will route us around if we dont change.

We kernel developers have been spoiled by years of abundance in testing resources. We squander tons of resources in this area, and we could be so much more economic about this without hindering our development model in any way. We could be so much better about QA and everyone would benefit without having to compromize on the incoming flux of changes - it's so much easier to write new features for a high quality kernel.

My current guesstimation is that we are utilizing our current testing resources at around 10% efficiency. (i.e. if we did an 'ideal' job we could fix 10 times as many bugs with the same size of tester effort!) It used to be around 5%. (and i mainly attribute the increase from 5% to 10% to Andrew and the many other people who do kernel QA - kudos!) 10% is still awful and we very much suck.

Paradoxically, the "end product" is still considerably good quality in absolute terms because other pieces of our infrastructure are so good and powerful, but QA is still a 'weak link' of our path to the user that reduces the quality of the end result. We could _really_ be so much better without any compromises that hurt.

(and this is in no way directed at the networking folks - it holds for all of us. I have one main complaint about networking: the separate netdev list is a bad idea - networking regressions should be discussed and fixed on lkml, like most other subsystems are. Any artificial split of the lk discussion space is bad.)

Ingo

Ingo Molnar wrote:
..

..

QA-101 and "many eyeballs" are not at all in opposition. The latter is how we find out about bugs on uncommon hardware, and the former is what we need to track them and overall quality.

A HUGE problem I have with current "efforts", is that once someone reports a bug, the onus seems to be 99% on the *reporter* to find the exact line of code or commit. Ghad what a repressive method.

And if the "developer" who broke the damn thing, or who at least "claims" to be supporting that code, cannot "reproduce" the bug, they drop it completely.

Contrast that flawed approach with how Linus does things.. he thinks through the symptoms, matches them to the code, and figures out what the few possibilities might be,
and feeds back some trial balloon patches for the bug reporter to try.

MUCH better.

Linus also asks for a git bisect, but doesn't insist upon the reporter learning an entire new (poorly documented) toolset just to to report a bug.

Blah!

And remember, *I'm* an old-time Linux kernel developer.. just think about the people reporting bugs who haven't been around here since 1992..

-ml

As a long time kernel tester, I see some problem with the newer "new development model". In the short merge windows, after to much time, there are to many patches.
So there are problem to bisect bugs, and to have attention of developers. My impression is that in a week there are many more messages in lkml and to much bugs to be handled in these few days.

I've two proposal:

- better patch quality. I would like that every commit would compile. So an automatic commit test and public blames could increase the quality of first commits. [bisecting with non compilable point it is not a trivial task]

- a slow down the patch inclusion on the merge windows (aka: not to much big changes in the first days). As tester I prefer that some big changes would be included in a "secondary window" (pre o rc release), in an other period as the big patch rush.

ciao
cate

I think the root issue there is that it's hard to get all testers to run a bisect, but easy to ask them to test snapshots. Right now the snapshots are generated nightly, but I think it would make more sense if they were generated every N patches, for some value of N...

Of course, for that to really work, we have to ensure that the result is always compilable, which has been getting better, but not perfect.

Ray

I don't see a point in doing that - that would be a more manual bisecting, and the result would not be one guilty commit.

Testers are not expected to be able to hack a kernel, but it's reasonable to expect testers to be able to build their own kernels (and your proposal wouldn't change that).

The small instruction below is enough for everyone who is able to build his own kernel to do a git bisect.

cu
Adrian

<-- snip -->

# install git

# clone Linus' tree:
git clone \ 
git://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux-2.6.git

# start bisecting:
cd linux-2.6
git bisect start
git bisect bad v2.6.21
git bisect good v2.6.20
cp /path/to/.config .

# start a round
make oldconfig
make
# install kernel, check whether it's good or bad, then:
git bisect [bad|good]
# start next round

After at about 10-15 reboots you'll have found the guilty commit ("... is first bad commit").

More information on git bisecting:
man git-bisect

I jump in this discussion hoping to have some more insight on git and to report my experience as a tester. I consider myself as half-literate in this (I am here since 1991, more or less, and I am able to compile a kernel and even hand-apply a patch, although I am in no way a kernel programmer).

This was what I did in my (in the end almost successful) bisecting when trying to find the mmc problem (see the thread named "2.6.24-rc1 eat my SD card"). This is true in theory, but it has some problem. The "this commit does not compile is the easiest and in man git-bisect it's explained how to solve it. The changes in .config options, added or removed, are another problem when jumping back and forth from version (I was bitten by the gadzillions new options added to hda-intel alsa driver, but well, that is solvable with a bit of attention).

The main problem I had, and that stopped me to arrive to a definite is this situation:

j version-bad
i
h
g unrelated (but similar) bug corrected
f
e
d unrelated (but similar) bug introduced
c
b
a version-good

(d was the series to change drivers to use sg helpers, and g was a "fix fallout from sg helpers" patch). Now I have a series of kernels (d, e,
f) that did not work at all and so I cannot mark them good or bad. With the number of patches added in the free-for-all week, this is a very probable scenario. There is a way out from this using bisect?

Romano

PS as a suggestion, I think that added a "Reported-by", or "Tested-by", or "Debugged-by" attribution in the repository, as happened to be in the MMC case, is a nice an d welcomed reward for the effort.

Sorry for the disclaimer --- ¡I cannot stop it!



--
La presente comunicación tiene carácter confidencial y es para el exclusivo uso del destinatario indicado en la misma. Si Ud. no es el destinatario indicado, le informamos que cualquier forma de distribución, reproducción o uso de esta comunicación y/o de la información contenida en la misma están estrictamente prohibidos por la ley. Si Ud. ha recibido esta comunicación por error, por favor, notifíquelo inmediatamente al remitente contestando a este mensaje y proceda a continuación a destruirlo. Gracias por su colaboración.

This communication contains confidential information. It is for the exclusive use of the intended addressee. If you are not the intended addressee, please note that any form of distribution, copying or use of this communication or the information in it is strictly prohibited by law. If you have received this communication in error, please immediately notify the sender by reply e-mail and destroy this message. Thank you for your cooperation. 

[...]

I think there are three strategies you can use in this case:
- create a kernel config that is as simple as possible, but still supports your hardware and reproduces your problem; a simpler config will often avoid compilation issues in parts of the kernel that you're not using anyway and has the benefit of speeding up the compiles too

- if you know/suspect in what part of the tree the bug is, first limit the bisection to that; you will have to verify that you did indeed find the correct (broken) change by doing a compile for the "last good commit + 1"

- if you find a broken commit, use 'git-reset --hard' to try to jump past the bad set of commits, but of course that does not help in the case:
g version-bad
f unrelated bug corrected
e
d the broken commit that caused your problem
c
b unrelated bug that breaks compilation or system introduced a version-good
in that case the best you can reasonably be expected to do is report that you narrowed it down to "between a and g" and leave the rest to the developers

Cheers,
FJP

Btw, I used to test every -mm kernel. But since I've switched distros (gentoo->ubuntu)
and I have less time, I feel it's harder to test -rc or -mm kernels (I know this isn't a lkml problem
but more a distro problem, but I would love having an ubuntu blessed repo with current dev kernel
for the latest stable ubuntu release).

For debugging, maybe it's time someone does an amazon ec2+s3 service to automate the bisecting
and create .deb/.rpm from git, I don't know how much it would cost though.

regards,

Benoit

a few months ago i estimated the costs of this and it's just a few terabytes so within arm's reach. As long as the .deb/.rpm's are built by tracking -git in a rolling fashion CPU time should not be a big issue. The only limit is download bandwidth - but even that problem might be solvable via a huge git repository of ready-to-boot .o's that are linked together on the tester's machine.

Ingo

There are two parts to this. One is a Ubuntu development kernel which we can give to large numbers of people to expand our testing pool. But if we don't do a better job of responding to bug reports that would be generated by expanded testing this won't necessarily help us.

The other an automated set of standard pre-built bisection points so that testers can more easily localize a bug down to a few hundred commits without needing to learn how to use "git bisect" (think Ubuntu users).

So for the first, I've actually been playing with some plans to put together an unofficial kernel that basically "what Ted is using on his laptop". It generally has emergency bug fixes that haven't made it into mainline, plus some other trees where I've been more aggressive since I want to latest in wireless and powersaving technology, etc. It has the property that "if it breaks, you get to keep both pieces --- and I've helpfully included the git ID in the package name so you can do the bisection yourself". If you want to try it, the first such kernel is here:

http://www.kernel.org/~tytso/tbek

I wasn't planning on talking about it until it was more fully baked, but if people want something vaguely stable based on 2.6.24-rc2, this might be interesting.

As for the second, I was just talking to Arjan over pizza and beer last night, and we reached the same conclusion as Ingo, which is this really isn't that hard. It wouldn't be that hard to set up infrastructure to do this, and it's just a matter of getting the disk space and the network bandwidth togehter in the right place, plus a relatively small amount of prgramming at least for the simplest iteration of the idea. (As is quite common when doing designs over beer, we talked about some more gradious web-based schemes to do custom built kernels that was tied to the kernel bugzilla, but first things first. :-)

- Ted

Before that you want a flowchart or instruction list of boot options to try. A lot of errors can be localised simply by asking the reported to boot with things like "iommu=off", "pci=routeirq", "apci=off" etc

That takes a lot less time to run through and can be very informative.

Alan

I'm very encouraged to read of your expanded testing efforts. As a bcm43xx developer, Ubuntu has been our problem distro, mostly because your standard kernels have debugging turned off for bcm43xx. When a Ubuntu user reports a problem and we ask for the relevant output from dmesg, they have no information. I ask two things of all distros: (1) Turn on debugging - we don't spam the logs that badly, and (2) forward any bugs found by your testing to the maintainer, and/or the bcm43xx mailing list.

Thanks,

Larry

Heh. I hadn't enabled CONFIG_BCM43XX_DEBUG myself, but I just changed it for my next kernel build. This is a slightly different issue, which is that sometimes _DEBUG options shouldn't be turned on by default (because they really trash performance and bloat log size), and sometimes they are painless to turn on and don't cost much.

If that is the case, I'd suggest removing the option and just making it compiled in by default with a run-time option to enable it.

- Ted

I am taking your suggestion and will produce the necessary patches for ssb, b43 and b43legacy. As bcm43xx is likely to be removed from 2.6.25, which is the earliest such a non-bug fix patch would be accepted, I hope that your future distribution and testing kernels will include the debug option.

Thanks,

Larry

I am taking your suggestion and will produce the necessary patches for ssb, b43 and b43legacy. As bcm43xx is likely to be removed from 2.6.25, which is the earliest such a non-bug fix patch would be accepted, I hope that your future distribution and testing kernels will include the debug option.

Thanks,

Larry

I am taking your suggestion and will produce the necessary patches for ssb, b43 and b43legacy. As bcm43xx is likely to be removed from 2.6.25, which is the earliest such a non-bug fix patch would be accepted, I hope that your future distribution and testing kernels will include the debug option.

Thanks,

Larry

I am taking your suggestion and will produce the necessary patches for ssb, b43 and b43legacy. As bcm43xx is likely to be removed from 2.6.25, which is the earliest such a non-bug fix patch would be accepted, I hope that your future distribution and testing kernels will include the debug option.

Thanks,

Larry

I'm very encouraged to read of your expanded testing efforts. As a bcm43xx developer, Ubuntu has been our problem distro, mostly because your standard kernels have debugging turned off for bcm43xx. When a Ubuntu user reports a problem and we ask for the relevant output from dmesg, they have no information. I ask two things of all distros: (1) Turn on debugging - we don't spam the logs that badly, and (2) forward any bugs found by your testing to the maintainer, and/or the bcm43xx mailing list.

Thanks,

Larry

I'm very encouraged to read of your expanded testing efforts. As a bcm43xx developer, Ubuntu has been our problem distro, mostly because your standard kernels have debugging turned off for bcm43xx. When a Ubuntu user reports a problem and we ask for the relevant output from dmesg, they have no information. I ask two things of all distros: (1) Turn on debugging - we don't spam the logs that badly, and (2) forward any bugs found by your testing to the maintainer, and/or the bcm43xx mailing list.

Thanks,

Larry

I'm very encouraged to read of your expanded testing efforts. As a bcm43xx developer, Ubuntu has been our problem distro, mostly because your standard kernels have debugging turned off for bcm43xx. When a Ubuntu user reports a problem and we ask for the relevant output from dmesg, they have no information. I ask two things of all distros: (1) Turn on debugging - we don't spam the logs that badly, and (2) forward any bugs found by your testing to the maintainer, and/or the bcm43xx mailing list.

Thanks,

Larry

The main problem aren't missing testers [1] - we already have relatively experienced people testing kernels and/or reporting bugs, and we slowly scare them away due to the many bug reports without any reaction.

The main problem is finding experienced developers who spend time on looking into bug reports.

Getting many relatively unexperienced users (who need more guidance for debugging issues) as additional testers is therefore IMHO not necessarily a good idea.

cu
Adrian

[1] and e.g. when Greg says he has a few hundred people who want to write drivers it would most likely be possible to find a few dozen additional -rc testers among them

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

There are already. IMO the problem is the development model.

There are tons new features in each new kernel release and 'tons new bugs' which are not fixed during the release cycle nor in the .XX stable kernels.

Maybe after XX kernel releases there should be one just with bug-fixes _without_ any new features , eg: cleaning bugs from bugzilla , know regressions , cleaning up code , removing broken drivers and the like.

Gabriel

Won't work. You cannot force people to work on things they don't find interesting, long-term.
--
vda

And where experienced developrs are coming from?
They are not born with Linux kernel skills.
They grow up from within user base.

Bigger user base -> more developers (eventually)
--
vda

You missed the following in my email:
"we slowly scare them away due to the many bug reports without any
reaction."

The problem is that bug reports take time. If you go away from easy things like compile errors then even things like describing what does no longer work, ideally producing a scenario where you can reproduce it and verifying whether it was present in previous kernels can easily take many hours that are spent before the initial bug report.

If the bug report then gets ignored we discourage the person who sent the bug report to do any work related to the kernel again.

cu
Adrian

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

Cannot agree more. I am in a similar position right now. My patch to aic7xxx driver was ubmitted four times with not much reaction from scsi guys.

Finally they replied and asked to rediff it against their git tree. I did that and sent patches back. No reply since then.

And mind you, the patch is not trying to do anything complex, it mostly moves code around, removes 'inline', adds 'const'. What should I think about it?
--
vda

I'm waiting for an ACK/NAK from Hannes, the maintainer. What should I do?

Intel are signing my paycheques ... these opinions are still mine
"Bill, look, we understand that you're interested in selling us this
operating system, but compare it to ours.  We can't possibly take such
a retrograde step."

I haven't actually been able to test it here (too busy, sorry). If someone else confirms it does it's job then

Cheers,

Hannes

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)

hi Matthew,

You could have informed me about this, and I would talk to Hannes myself. This would free up your mind from keeping track of this particular patch.
Parallelize development, prevent things from being forgotten.

Hi Hannes,

It's not in my mailbox on this machine, gladly we have lkml archived in the Net. Here is a positive tester report:

http://lkml.org/lkml/2007/10/15/168:

======================
Date Mon, 15 Oct 2007 15:53:08 +0200
From Gabriel C <>
Subject Re: [PATCH 0/3] debloat aic7xxx and aic79xx drivers

want.

Works fine for me tested on :

03:0e.0 SCSI storage controller [0100]: Adaptec AIC-7892P U160/m [9005:008f] (rev 02)

Gabriel
=======================

--
vda

I still run the patches on a box with 2.6.23 and one with 2.6.24-rc1 without any problems. Didn't tested rc2/current git but I can if is needed.

If you need an Tested-by.. let me know.

Gabriel

Hi Matthew, Hannes,

Any news on this? Where these patches stand now?
Are they accepted into linux-scsi?

vda

As it's been tested, you have my

Cheers,

Hannes

Dr. Hannes Reinecke		      zSeries & Storage
[email protected]			      +49 911 74053 688
SUSE LINUX Products GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: Markus Rex, HRB 16746 (AG Nürnberg)
-
To unsubscribe from this list: send the line "unsubscribe linux-scsi" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html

this has nothing to do with the bugs on bugzilla.

you're trying to send a janitor patch. It should be logical that the response to that is not heated or receiving a joyous reception :)

If you have a problem getting your cleanup patch to the driver maintainer, send it to the subsystem maintainer instead, or even the janitors, or even Adrian Bunk who will gladly push it to everyone. Or, even to Andrew Morton who will carry it in -mm for a while and then harrasses the subsystem maintainer to merge it for you!

Cheers,

Auke

Hum. If only each of those would squash one bug a week besides their own work...
I would expect he's got a handful that know IDE, another group that is into network drivers and so on.

I predict that pile of bugs to disappear in weeks (-;

Just my $0.02.

Jan Evert

The main problem aren't missing testers [1] - we already have relatively experienced people testing kernels and/or reporting bugs, and we slowly scare them away due to the many bug reports without any reaction.

The main problem is finding experienced developers who spend time on looking into bug reports.

Getting many relatively unexperienced users (who need more guidance for debugging issues) as additional testers is therefore IMHO not necessarily a good idea.

cu
Adrian

[1] and e.g. when Greg says he has a few hundred people who want to write drivers it would most likely be possible to find a few dozen additional -rc testers among them

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

The main problem aren't missing testers [1] - we already have relatively experienced people testing kernels and/or reporting bugs, and we slowly scare them away due to the many bug reports without any reaction.

The main problem is finding experienced developers who spend time on looking into bug reports.

Getting many relatively unexperienced users (who need more guidance for debugging issues) as additional testers is therefore IMHO not necessarily a good idea.

cu
Adrian

[1] and e.g. when Greg says he has a few hundred people who want to write drivers it would most likely be possible to find a few dozen additional -rc testers among them

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

The main problem aren't missing testers [1] - we already have relatively experienced people testing kernels and/or reporting bugs, and we slowly scare them away due to the many bug reports without any reaction.

The main problem is finding experienced developers who spend time on looking into bug reports.

Getting many relatively unexperienced users (who need more guidance for debugging issues) as additional testers is therefore IMHO not necessarily a good idea.

cu
Adrian

[1] and e.g. when Greg says he has a few hundred people who want to write drivers it would most likely be possible to find a few dozen additional -rc testers among them

       "Is there not promise of rain?" Ling Tan asked suddenly out
        of the darkness. There had been need of rain for many days.
       "Only a promise," Lao Er said.
                                       Pearl S. Buck - Dragon Seed

I don't see any reason that we couldn't have a tool accessible to Ubuntu users that does a real "git bisect". Git is really good at being scripted by fancy GUIs. It should be easy enough to have a drop down with all of the Ubuntu kernel package releases, where the user selects what works and what doesn't. Then the tool clones a git repository with flags to only get relevant parts, and then leads a bisect run, where it's also configuring, building, and installing the kernels (as a different grub entry), and providing instructions in general. Fundamentally, "git bisect" is a really low-interaction process: you tell it a couple of commits, and then it does stuff, and then you tell it "I tested, and it worked" or "I tested, and it had the problem" or "Something else went wrong", and it asks you something new. Other than that, it just takes time (and a build system hook, which this tool would handle for the kernel). Eventually, it tells you what to report, and you do so.

-Daniel
*This .sig left intentionally blank*

It's possible users who haven't yet downloaded a git repository have to surmount some obstacles that might cause them to lose interest. First, they have to download some 190 megs of git repository, and if they have a slow link, that can take a while, and then they have to build each kernel, which can take a while. A full kernel build with everything selected can take good 30 minutes or more, and that's on a fast dual-core machine with 4gigs of memory and 7200rpm disk drives. On a slower, memory limited laptop, doing a single kernel build can take more time than the user has patiences; multiply that by 7 or 8 build and test boots, and it starts to get tiresome.

And then on top of that there are the issues about whether there is enough support for dealing with hitting kernel revisions that fail due to other bugs getting merged in during the -rc1 process, etc.

I agree that a tool that automated the bisection process and walked the user through it would be helpful, but I believe it would be possible for us do better.

- Ted

It should be possible for it to clone only the portion that they actually care about based on where the known-good version is. It should also (in theory, anyway) be possible to put off some amount of the download until it's actually going to be relevant.

None of this is going to take as long, even on a slow link and a slow computer, as waiting for a response to a mailing list post. It'd annoy users who are specifically waiting for it, but if the interface is that the user says "kernel package X didn't work but the current kernel does", and it says "I'll let you know when I've got something to test", and the user watches a DVD, and afterward finds a message saying there's something to test, and tries it, and reports how it went, and the process repeats until it narrows it down to a single commit after a couple of days of the user getting occasional responses, it's not that different from asking for help online.

Could have a distro-provided mask of things that aren't worth testing and possibly back-ported fixes for revisions in particular ranges.

That would probably help for giving the user something to try right away. I still think that the main cost to the user is the number of times that the user has to stop doing stuff to reboot with a kernel to test, whether the test kernels are available quickly from the distro site, slowly built locally, or slowly as suggested by humans helping online.

-Daniel
*This .sig left intentionally blank*

It should be possible for it to clone only the portion that they actually care about based on where the known-good version is. It should also (in theory, anyway) be possible to put off some amount of the download until it's actually going to be relevant.

None of this is going to take as long, even on a slow link and a slow computer, as waiting for a response to a mailing list post. It'd annoy users who are specifically waiting for it, but if the interface is that the user says "kernel package X didn't work but the current kernel does", and it says "I'll let you know when I've got something to test", and the user watches a DVD, and afterward finds a message saying there's something to test, and tries it, and reports how it went, and the process repeats until it narrows it down to a single commit after a couple of days of the user getting occasional responses, it's not that different from asking for help online.

Could have a distro-provided mask of things that aren't worth testing and possibly back-ported fixes for revisions in particular ranges.

That would probably help for giving the user something to try right away. I still think that the main cost to the user is the number of times that the user has to stop doing stuff to reboot with a kernel to test, whether the test kernels are available quickly from the distro site, slowly built locally, or slowly as suggested by humans helping online.

-Daniel
*This .sig left intentionally blank*

It should be possible for it to clone only the portion that they actually care about based on where the known-good version is. It should also (in theory, anyway) be possible to put off some amount of the download until it's actually going to be relevant.

None of this is going to take as long, even on a slow link and a slow computer, as waiting for a response to a mailing list post. It'd annoy users who are specifically waiting for it, but if the interface is that the user says "kernel package X didn't work but the current kernel does", and it says "I'll let you know when I've got something to test", and the user watches a DVD, and afterward finds a message saying there's something to test, and tries it, and reports how it went, and the process repeats until it narrows it down to a single commit after a couple of days of the user getting occasional responses, it's not that different from asking for help online.

Could have a distro-provided mask of things that aren't worth testing and possibly back-ported fixes for revisions in particular ranges.

That would probably help for giving the user something to try right away. I still think that the main cost to the user is the number of times that the user has to stop doing stuff to reboot with a kernel to test, whether the test kernels are available quickly from the distro site, slowly built locally, or slowly as suggested by humans helping online.

-Daniel
*This .sig left intentionally blank*

It should be possible for it to clone only the portion that they actually care about based on where the known-good version is. It should also (in theory, anyway) be possible to put off some amount of the download until it's actually going to be relevant.

None of this is going to take as long, even on a slow link and a slow computer, as waiting for a response to a mailing list post. It'd annoy users who are specifically waiting for it, but if the interface is that the user says "kernel package X didn't work but the current kernel does", and it says "I'll let you know when I've got something to test", and the user watches a DVD, and afterward finds a message saying there's something to test, and tries it, and reports how it went, and the process repeats until it narrows it down to a single commit after a couple of days of the user getting occasional responses, it's not that different from asking for help online.

Could have a distro-provided mask of things that aren't worth testing and possibly back-ported fixes for revisions in particular ranges.

That would probably help for giving the user something to try right away. I still think that the main cost to the user is the number of times that the user has to stop doing stuff to reboot with a kernel to test, whether the test kernels are available quickly from the distro site, slowly built locally, or slowly as suggested by humans helping online.

-Daniel
*This .sig left intentionally blank*

It should be possible for it to clone only the portion that they actually care about based on where the known-good version is. It should also (in theory, anyway) be possible to put off some amount of the download until it's actually going to be relevant.

None of this is going to take as long, even on a slow link and a slow computer, as waiting for a response to a mailing list post. It'd annoy users who are specifically waiting for it, but if the interface is that the user says "kernel package X didn't work but the current kernel does", and it says "I'll let you know when I've got something to test", and the user watches a DVD, and afterward finds a message saying there's something to test, and tries it, and reports how it went, and the process repeats until it narrows it down to a single commit after a couple of days of the user getting occasional responses, it's not that different from asking for help online.

Could have a distro-provided mask of things that aren't worth testing and possibly back-ported fixes for revisions in particular ranges.

That would probably help for giving the user something to try right away. I still think that the main cost to the user is the number of times that the user has to stop doing stuff to reboot with a kernel to test, whether the test kernels are available quickly from the distro site, slowly built locally, or slowly as suggested by humans helping online.

-Daniel
*This .sig left intentionally blank*

It should be possible for it to clone only the portion that they actually care about based on where the known-good version is. It should also (in theory, anyway) be possible to put off some amount of the download until it's actually going to be relevant.

None of this is going to take as long, even on a slow link and a slow computer, as waiting for a response to a mailing list post. It'd annoy users who are specifically waiting for it, but if the interface is that the user says "kernel package X didn't work but the current kernel does", and it says "I'll let you know when I've got something to test", and the user watches a DVD, and afterward finds a message saying there's something to test, and tries it, and reports how it went, and the process repeats until it narrows it down to a single commit after a couple of days of the user getting occasional responses, it's not that different from asking for help online.

Could have a distro-provided mask of things that aren't worth testing and possibly back-ported fixes for revisions in particular ranges.

That would probably help for giving the user something to try right away. I still think that the main cost to the user is the number of times that the user has to stop doing stuff to reboot with a kernel to test, whether the test kernels are available quickly from the distro site, slowly built locally, or slowly as suggested by humans helping online.

-Daniel
*This .sig left intentionally blank*