2.4.10-pre11: alsaplayer skiping during kernel build (-pre10 did not)
Hello Andrea,
I gave your new vm a try and I have to report a problem. System is an Athlon 1200 with 256MB memory. Workload:
1. top refreshing every second reniced to -10
2. alsaplayer -n -q -r *.wav
3. make -j4 bzImage modules
The problem is that with 2.4.10-pre11 alsaplayer is skiping very much. Almost every ten seconds and then the break seems to be relatevily long (like >1s). With 2.4.10-pre10 I noticed alsaplayer skiping once or twice during the whole build (which takes about 6min) for a very short time (like a cd player skipping over a scratch). Here is the diff output of time -v of the make -j4 command:
planetzork:/usr/src/linux# diff -u time.out-2.4.10-pre1*
--- time.out-2.4.10-pre10-1000823801 Tue Sep 18 16:42:30 2001
+++ time.out-2.4.10-pre11-1000824952 Tue Sep 18 17:01:39 2001
@@ -1,16 +1,16 @@
- Command being timed: "sh -c make dep clean>logfile-2.4.10-pre10 2>&1 && make -j4 bzImage modules>>logfile-2.4.10-pre10 2>&1"
- User time (seconds): 241.28
- System time (seconds): 17.82
+ Command being timed: "sh -c make dep clean>logfile-2.4.10-pre11 2>&1 && make -j4 bzImage modules>>logfile-2.4.10-pre11 2>&1"
+ User time (seconds): 242.11
+ System time (seconds): 17.10
Percent of CPU this job got: 74%
- Elapsed (wall clock) time (h:mm:ss or m:ss): 5:49.57
+ Elapsed (wall clock) time (h:mm:ss or m:ss): 5:47.60
Average shared text size (kbytes): 0
Average unshared data size (kbytes): 0
Average stack size (kbytes): 0
Average total size (kbytes): 0
Maximum resident set size (kbytes): 0
Average resident set size (kbytes): 0
- Major (requiring I/O) page faults: 901054
- Minor (reclaiming a frame) page faults: 934971
+ Major (requiring I/O) page faults: 906843
+ Minor (reclaiming a frame) page faults: 936570
Voluntary context switches: 0
Involuntary context switches: 0
Swaps: 0Best regards,
Jochen
Signature · 3 lines
Well, yeah ... I suppose there's no point in getting greedy, is there?
<< Calvin & Hobbes >>Quoted 5 lines · On Tue, Sep 18, 2001 at 05:14:16PM +0200, [email protected] wrote:
Hello Andrea,
I gave your new vm a try and I have to report a problem. System is an Athlon 1200 with 256MB memory. Workload:
Quoted 7 lines
1. top refreshing every second reniced to -10
2. alsaplayer -n -q -r *.wav
3. make -j4 bzImage modulesThe problem is that with 2.4.10-pre11 alsaplayer is skiping very much. Almost every ten seconds and then the break seems to be relatevily long (like >1s). With 2.4.10-pre10 I noticed alsaplayer skiping once or twice
the skips shouldn't really be realated to vm changes, if something to the schedrt fix. but the real issue is that you should avoid to run top at -10 (or you meant +10?). Running top at -10 isn't a good idea, it is allowing it to get more cpu than the other tasks for no good reason.
Andrea
Quoted 4 lines · On Tue, Sep 18, 2001 at 05:14:16PM +0200, [email protected] wrote:
Hello Andrea,
I gave your new vm a try and I have to report a problem. System is an Athlon 1200 with 256MB memory. Workload:
Sorry to follow up on my own. But the problem seems to be worse than I first thought. Kernel build is the first thing I test when I try a new kernel :-)
So now I logged into X and even during starting of Mozilla or normal web browsing alsaplayer is skiping *lots*. The system is not into swap and has about 150MB cached. Just to let you know ...
I guess I will go back to -pre10 now. If you want to let me test some things just let me know.
Regards,
Jogi
Signature · 3 lines
Well, yeah ... I suppose there's no point in getting greedy, is there?
<< Calvin & Hobbes >>Quoted 10 lines · On Tue, Sep 18, 2001 at 05:25:29PM +0200, [email protected] wrote:
Quoted 4 lines · On Tue, Sep 18, 2001 at 05:14:16PM +0200, [email protected] wrote:
Hello Andrea,
I gave your new vm a try and I have to report a problem. System is an Athlon 1200 with 256MB memory. Workload:
Sorry to follow up on my own. But the problem seems to be worse than I first thought. Kernel build is the first thing I test when I try a new kernel :-)
Quoted 6 lines
So now I logged into X and even during starting of Mozilla or normal web browsing alsaplayer is skiping *lots*. The system is not into swap and has about 150MB cached. Just to let you know ...
I guess I will go back to -pre10 now. If you want to let me test some things just let me know.
just make sure not to renice top at -10. Are you sure you reniced top at -10 also with pre10? Those issues shouldn't really vm related. I also profiled the new vm well and it never showed up in the toplist so I suspect you changed something in userspace. and of course I run xmms all the time too and it didn't skept one sample yet.
Andrea
Quoted 10 lines · Andrea Arcangeli
Quoted 9 lines · On Tue, Sep 18, 2001 at 05:14:16PM +0200, [email protected] wrote:
Hello Andrea,
I gave your new vm a try and I have to report a problem. System is an Athlon 1200 with 256MB memory. Workload:
1. top refreshing every second reniced to -10
2. alsaplayer -n -q -r *.wav
3. make -j4 bzImage modules
Quoted 8 lines
Quoted 3 lines
The problem is that with 2.4.10-pre11 alsaplayer is skiping very much. Almost every ten seconds and then the break seems to be relatevily long (like >1s). With 2.4.10-pre10 I noticed alsaplayer skiping once or twice
the skips shouldn't really be realated to vm changes, if something to the schedrt fix. but the real issue is that you should avoid to run top at -10 (or you meant +10?). Running top at -10 isn't a good idea, it is allowing it to get more cpu than the other tasks for no good reason.
Ok, I just wanted to see the current load all the time :-) Btw. I don't think that just one top running should cause alsaplayer (which runs with real-time properties:
-r, --realtime
Enable realtime scheduling. To use this as a nor mal user, alsaplayer must be SUID root.
As I wrote in a seperate mail alsaplayer skips during other activities also, like starting mutt, mozilla, whatever ... And these are not started reniced.
Since I am not using md there are not that much changes left between -pre10 and -pre11. Or do you think that it is caused by the console locking changes?
Regards,
Jogi
Signature · 3 lines
Well, yeah ... I suppose there's no point in getting greedy, is there?
<< Calvin & Hobbes >>Quoted 3 lines · On Tue, Sep 18, 2001 at 05:35:15PM +0200, [email protected] wrote:
Since I am not using md there are not that much changes left between -pre10 and -pre11. Or do you think that it is caused by the console locking changes?
certainly not from the console locking changes. Can you just go back to pre10 and verify you don't get those skips to just to be 100% sure the userspace config is the same?
The only scheduler change in pre11 is this one:
ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.10pre10aa1/00_sched-rt-fix-1
which should be infact a bugfix for rt threads, also discussed on l-k recently, so it's not clear how this odd regression happened.
You can try to back it out and see if helps just in case.
Andrea
Quoted 15 lines · Andrea Arcangeli
Quoted 14 lines · On Tue, Sep 18, 2001 at 05:25:29PM +0200, [email protected] wrote:
Quoted 4 lines · On Tue, Sep 18, 2001 at 05:14:16PM +0200, [email protected] wrote:
Hello Andrea,
I gave your new vm a try and I have to report a problem. System is an Athlon 1200 with 256MB memory. Workload:
Sorry to follow up on my own. But the problem seems to be worse than I first thought. Kernel build is the first thing I test when I try a new kernel :-)
So now I logged into X and even during starting of Mozilla or normal web browsing alsaplayer is skiping *lots*. The system is not into swap and has about 150MB cached. Just to let you know ...
Quoted 8 lines
Quoted 2 lines
I guess I will go back to -pre10 now. If you want to let me test some things just let me know.
just make sure not to renice top at -10. Are you sure you reniced top at -10 also with pre10? Those issues shouldn't really vm related. I also profiled the new vm well and it never showed up in the toplist so I suspect you changed something in userspace. and of course I run xmms all the time too and it didn't skept one sample yet.
Ok, I just killed top. And it still is skiping lots although I do not have any reniced jobs running. And yes, I had top running reniced too on -pre10. I will boot -pre10 and check if it is skipping because alsaplayer is currently skipping as hell (running make -j4 in the background). Still no swap used. Strange. I will check xmms also and let you know.
Regards,
Jogi
Signature · 3 lines
Well, yeah ... I suppose there's no point in getting greedy, is there?
<< Calvin & Hobbes >>Quoted 9 lines · Andrea Arcangeli
Quoted 3 lines · On Tue, Sep 18, 2001 at 05:35:15PM +0200, [email protected] wrote:
Since I am not using md there are not that much changes left between -pre10 and -pre11. Or do you think that it is caused by the console locking changes?
certainly not from the console locking changes. Can you just go back to pre10 and verify you don't get those skips to just to be 100% sure the userspace config is the same?
Quoted 8 lines
The only scheduler change in pre11 is this one:
ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.10pre10aa1/00_sched-rt-fix-1which should be infact a bugfix for rt threads, also discussed on l-k recently, so it's not clear how this odd regression happened.
You can try to back it out and see if helps just in case.
Ok Andrea,
I will test and let you know about the findings but I am afraid I can not test this today. But I will let you know. Btw. xmms is skipping like mad too.
Regards,
Jogi
Signature · 3 lines
Well, yeah ... I suppose there's no point in getting greedy, is there?
<< Calvin & Hobbes >>Quoted 7 lines · On 18 Sep 2001 17:51:04 +0200 [email protected] wrote:
Quoted 1 line
You can try to back it out and see if helps just in case.
Ok Andrea,
I will test and let you know about the findings but I am afraid I can not test this today. But I will let you know. Btw. xmms is skipping like mad too.
Hello,
just my experience in this topic:
I worked on -pre11 for one complete day after massive nfs server usage, no problems so far.
current meminfo looks like:
total: used: free: shared: buffers: cached:
Mem: 923574272 865853440 57720832 0 66609152 629137408 Swap: 271392768 434176 270958592 MemTotal: 901928 kB MemFree: 56368 kB MemShared: 0 kB Buffers: 65048 kB Cached: 614240 kB SwapCached: 152 kB Active: 671956 kB Inactive: 7484 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 901928 kB LowFree: 56368 kB SwapTotal: 265032 kB SwapFree: 264608 kB
I currently compile kernel with -j 5 while playing mp3s via xmms without the slightest problem (xosview load is now 8). I tried xmms run as root and user, makes no difference. I tried starting applications (like mozilla or applixware), no problem. No mouse hangs or anything else negative. I am _not_ using alsa, but emu10k1 from kernel.
I couldn't even make it crash or skip in xmms while singing Torn together with Natalie - and listening to that _is_ real stress :-))
You only have to find some strategy to come around these formerly noticed alloc failures, perhaps by an idea to free up pages instantly if an alloc would fail otherwise (now). I like the overall performance very much, Andrea.
Regards,
Stephan
Quoted 8 lines · Andrea Arcangeli
Quoted 3 lines · On Tue, Sep 18, 2001 at 05:35:15PM +0200, [email protected] wrote:
Since I am not using md there are not that much changes left between -pre10 and -pre11. Or do you think that it is caused by the console locking changes?
certainly not from the console locking changes. Can you just go back to pre10 and verify you don't get those skips to just to be 100% sure the userspace config is the same?
Ok, -pre10 is ok. There xmms did hang once for about a second, and at that time I was building a kernel (make -j4), reniced top, starting mozilla, StarOffice, Netscape, mutt (reading in mailbox with >200 emails in qmail format).
With -pre11 much less is required.
Quoted 8 lines
The only scheduler change in pre11 is this one:
ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.10pre10aa1/00_sched-rt-fix-1which should be infact a bugfix for rt threads, also discussed on l-k recently, so it's not clear how this odd regression happened.
You can try to back it out and see if helps just in case.
I will test this tomorrow and let you know about the results.
JOgi
Signature · 3 lines
Well, yeah ... I suppose there's no point in getting greedy, is there?
<< Calvin & Hobbes >>Quoted 43 lines · Stephan von Krawczynski
Quoted 7 lines · On 18 Sep 2001 17:51:04 +0200 [email protected] wrote:
Quoted 1 line
You can try to back it out and see if helps just in case.
Ok Andrea,
I will test and let you know about the findings but I am afraid I can not test this today. But I will let you know. Btw. xmms is skipping like mad too.
Hello,
just my experience in this topic:
I worked on -pre11 for one complete day after massive nfs server usage, no problems so far.
current meminfo looks like:total: used: free: shared: buffers: cached:
Mem: 923574272 865853440 57720832 0 66609152 629137408 Swap: 271392768 434176 270958592 MemTotal: 901928 kB MemFree: 56368 kB MemShared: 0 kB Buffers: 65048 kB Cached: 614240 kB SwapCached: 152 kB Active: 671956 kB Inactive: 7484 kB HighTotal: 0 kB HighFree: 0 kB LowTotal: 901928 kB LowFree: 56368 kB SwapTotal: 265032 kB SwapFree: 264608 kBI currently compile kernel with -j 5 while playing mp3s via xmms without the slightest problem (xosview load is now 8). I tried xmms run as root and user, makes no difference. I tried starting applications (like mozilla or applixware), no problem. No mouse hangs or anything else negative. I am _not_ using alsa, but emu10k1 from kernel.
I couldn't even make it crash or skip in xmms while singing Torn together with Natalie - and listening to that _is_ real stress :-))
Quoted 3 lines
You only have to find some strategy to come around these formerly noticed alloc failures, perhaps by an idea to free up pages instantly if an alloc would fail otherwise (now). I like the overall performance very much, Andrea.
I'm very happy to hear this ;).
I now have an update ready for Linus to merge that should fix the few leftovers I had in the very first release of the vm rewrite but of course I will be interested to hear about any regression/progression about those changes, I'll post them in a few minutes in CC to l-k.
They're still untested on the >1G boxes but the oom problems weren't specific to highmem so it should fix the highmem troubles too.
Andrea
Quoted 6 lines · Andrea Arcangeli
I now have an update ready for Linus to merge that should fix the few leftovers I had in the very first release of the vm rewrite but of course I will be interested to hear about any regression/progression about those changes, I'll post them in a few minutes in CC to l-k.
Please include Andi's likely()/unlikely() change - it's nice.
I can't measure any obviously new causes of latency in your VM. It's nice that you've paid attention to this in various places.
The main culprits now are the file IO and dirty buffer writeout paths:
up to fifty milliseconds in each.
I suggest you stick scheduling points in generic_file_read(), generic_file_write() and write_locked_buffers() and then dispose of the copy-user-latency patch from -aa kernels.
With the above fixed, the main source of latency is /proc/meminfo->si_swapinfo(). It's about five milliseconds per gig of swap, which isn't too bad. But it's directly invokable by userspace (ie: /usr/bin/top) and really should be made less dumb.
Quoted 5 lines · Andrew Morton
With the above fixed, the main source of latency is /proc/meminfo->si_swapinfo(). It's about five milliseconds per gig of swap, which isn't too bad. But it's directly invokable by userspace (ie:
/usr/bin/top) and really should be made less dumb.
Don't worry about this one, fix is included in patch I posted last Saturday, and will be rebasing, breaking up and submitting to Linus in a couple(?) of days. Basically, Zach Brown's patch to use nr_swap_pages and total_swap_pages, but in rare case of concurrent swapoff, does need to scan that swap_map (with appropriate locking) to give sensible numbers without strange negatives at that time.
Hugh
Quoted 9 lines · Andrew Morton
Quoted 6 lines · Andrea Arcangeli
I now have an update ready for Linus to merge that should fix the few leftovers I had in the very first release of the vm rewrite but of course I will be interested to hear about any regression/progression about those changes, I'll post them in a few minutes in CC to l-k.
Please include Andi's likely()/unlikely() change - it's nice.
I guess I'll postpone the likely/unlikely after resynching with Linus since I'm basically ready to run rsync and I prefer to go sleep early today and to think about new things tomorrow ;).
I also have a few arguments about the likely/unlikely to solve before agreeing on it, in particular in all my usages the value will be either 0 or 1 so I don't see why should I tell gcc to do !!, probably it will be optimized away but I also don't see why should the left term matter for an "if", the "if" only cares about zero or non zero, so I should be able to define the fast path with an 1 even if my result is 2, otherwise it sounds like gcc is doing something strange.
Quoted 3 lines
I can't measure any obviously new causes of latency in your VM. It's nice that you've paid attention to this in various places.
thanks.
Quoted 6 lines
The main culprits now are the file IO and dirty buffer writeout paths:
up to fifty milliseconds in each.I suggest you stick scheduling points in
generic_file_read(),generic_file_write()andwrite_locked_buffers()and then dispose of the copy-user-latency patch from -aa kernels.
Yes, I pretty much agree on such change, I remeber you just pointed out once. It's postponed to tomorrow too ;).
At the moment I just care to post the vm fixes against pre11 plain in order to possibly get some feedback on it while I sleep :)
Quoted 4 lines
With the above fixed, the main source of latency is /proc/meminfo->si_swapinfo(). It's about five milliseconds per gig of swap, which isn't too bad. But it's directly invokable by userspace (ie:
/usr/bin/top) and really should be made less dumb.
ok.
Andrea
Quoted 9 lines · Andrea Arcangeli
Quoted 3 lines · On Tue, Sep 18, 2001 at 05:35:15PM +0200, [email protected] wrote:
Since I am not using md there are not that much changes left between -pre10 and -pre11. Or do you think that it is caused by the console locking changes?
certainly not from the console locking changes. Can you just go back to pre10 and verify you don't get those skips to just to be 100% sure the userspace config is the same?
Quoted 8 lines
The only scheduler change in pre11 is this one:
ftp://ftp.us.kernel.org/pub/linux/kernel/people/andrea/kernels/v2.4/2.4.10pre10aa1/00_sched-rt-fix-1which should be infact a bugfix for rt threads, also discussed on l-k recently, so it's not clear how this odd regression happened.
You can try to back it out and see if helps just in case.
Hello Andrea,
I backed out this patch and it did not help. But I checked with top and alsaplayer is in status D and hanging in waitchannel wait_on_p (probably wait_on_paige). Furthermore there are lots of other processes waiting in wait_on_p and wait_on_b. Since the problem might be I/O related here is the output of hdparm /dev/hda:
/dev/hda:
multcount = 16 (on) I/O support = 1 (32-bit) unmaskirq = 1 (on) using_dma = 1 (on) keepsettings = 0 (off) nowerr = 0 (off) readonly = 0 (off) readahead = 8 (on)
geometry = 3165/255/63, sectors = 50859648, start = 0
dmesg:
Uniform Multi-Platform E-IDE driver Revision: 6.31
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xxVP_IDE: IDE controller on PCI bus 00 dev 39VP_IDE: chipset revision 6VP_IDE: not 100% native mode: will probe irqs later
ide: Assuming 33MHz system bus speed for PIO modes; override with idebus=xxVP_IDE: VIA vt82c686b (rev 40) IDE UDMA100 controller on pci00:07.1
ide0: BM-DMA at 0xa400-0xa407, BIOS settings: hda:DMA, hdb:pio
ide1: BM-DMA at 0xa408-0xa40f, BIOS settings: hdc:DMA, hdd:piohda: QUANTUM FIREBALLlct08 26, ATA DISK drive
hdc: QUANTUM FIREBALL CR13.0A, ATA DISK drive
ide0 at 0x1f0-0x1f7,0x3f6 on irq 14
ide1 at 0x170-0x177,0x376 on irq 15
hda: 50859648 sectors (26040 MB) w/418KiB Cache, CHS=3165/255/63, (U)DMA
hdc: 25429824 sectors (13020 MB) w/418KiB Cache, CHS=25228/16/63, (U)DMA Partition check:
hda: hda1 hda2 hda3 hda4 < hda5 hda6 hda7 hda8 >
hdc: [PTBL] [1582/255/63] hdc1 hdc2 hdc3 hdc4 < hdc5 hdc6 >
mount:
/dev/hda3 on / type ext2 (rw,errors=remount-ro,errors=remount-ro) proc on /proc type proc (rw)
devpts on /dev/pts type devpts (rw,gid=5,mode=620) /dev/hda2 on /boot type ext2 (ro)/dev/hda5 on /var type reiserfs (rw)/dev/hda6 on /usr type reiserfs (rw)/dev/hda7 on /home type reiserfs (rw)
Hope this helps. If you need further testing just let me know.
Regards,
Jogi
Signature · 3 lines
Well, yeah ... I suppose there's no point in getting greedy, is there?
<< Calvin & Hobbes >>Quoted 3 lines · Andrea Arcangeli
certainly not from the console locking changes. Can you just go back to pre10 and verify you don't get those skips to just to be 100% sure the userspace config is the same?
Hello Andrea,
forget about the scheduler changes. I did the following change and the problem is almost solved. Before I had both kernel and wav files on
/dev/hda6 4200828 1745204 2455624 42% /usr/dev/hda6 on /usr type reiserfs (rw)
I copied kernel source and wav files to
/dev/hda8 8989695 2907606 5615804 34% /mnt/dev/hda8 on /mnt type ext2 (rw)
and it skipped only three (very short) times. So it seems to be a problem of reiserfs in combination with your memory management changes. Dang! Can someone of the reiserfs people comment on this?
Regards,
Jogi
Signature · 3 lines
Well, yeah ... I suppose there's no point in getting greedy, is there?
<< Calvin & Hobbes >>