Skip to contentSkip to conversation

LTP VM test slower under 2.6.3 than 2.4.20

DISCUSSION Kirk True 5 participants 14 messages Feb 20, 2004 – Feb 24, 2004 2 messages not in the archive Memory management

lore
Share & tools
lore

Kirk True

author
#0 lore raw

Hi all,

Executing the LTP "mem01" VM test shows a huge time discrepancy between 2.4.20 and 2.6.3. Under 2.4.20 the total time is around 5 seconds, while under 2.6.3 the system seems to hang for nearly a minute.

Where in particular should I start to look to see if it's a configuration/environment issue or a real problem? What other information would be helpful to know?

Thanks greatly!!!
Kirk

--------------

2.6.3:

# time /tmp/ltp-full-20040206/testcases/kernel/mem/mem/mem01
Free Mem:       749 Mb
Free Swap:      1992 Mb
Total Free:     2741 Mb
Total Tested:   1024 Mb
mem01       0  INFO  :  touching 1024MB of malloc'ed memory (linear)
mem01       1  PASS  :  malloc - alloc of 1024MB succeeded
real    0m53.134s
user    0m0.066s
sys     0m3.292s

2.4.20:

# time /tmp/ltp-full-20040206/testcases/kernel/mem/mem/mem01
Free Mem:       859 Mb
Free Swap:      1992 Mb
Total Free:     2852 Mb
Total Tested:   1024 Mb
mem01       0  INFO  :  touching 1024MB of malloc'ed memory (linear)
mem01       1  PASS  :  malloc - alloc of 1024MB succeeded
real    0m5.493s
user    0m0.090s
sys     0m1.260s

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#1 lore raw
Quoted 8 lines

Hi all,

Executing the LTP "mem01" VM test shows a huge time discrepancy between 2.4.20 and 2.6.3. Under 2.4.20 the total time is around 5 seconds, while under 2.6.3 the system seems to hang for nearly a minute.

Where in particular should I start to look to see if it's a configuration/environment issue or a real problem? What other information would be helpful to know?

Thanks greatly!!!
Kirk

A kernel profile might help.

M.

Quoted 33 lines

--------------

2.6.3:

# time /tmp/ltp-full-20040206/testcases/kernel/mem/mem/mem01
Free Mem:       749 Mb
Free Swap:      1992 Mb
Total Free:     2741 Mb
Total Tested:   1024 Mb
mem01       0  INFO  :  touching 1024MB of malloc'ed memory (linear)
mem01       1  PASS  :  malloc - alloc of 1024MB succeeded
real    0m53.134s
user    0m0.066s
sys     0m3.292s

2.4.20:

# time /tmp/ltp-full-20040206/testcases/kernel/mem/mem/mem01
Free Mem:       859 Mb
Free Swap:      1992 Mb
Total Free:     2852 Mb
Total Tested:   1024 Mb
mem01       0  INFO  :  touching 1024MB of malloc'ed memory (linear)
mem01       1  PASS  :  malloc - alloc of 1024MB succeeded
real    0m5.493s
user    0m0.090s
sys     0m1.260s
Quoted 7 lines

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#2 lore raw

Hi all,

Quoted 1 line

A kernel profile might help.

Here's the profiles comparing the two versions. I don't understand why no "do_page_fault" shows up under 2.4.20 or why "gunzip" does.

Kirk

-------------------------------------------------

2.6.3:

# readprofile -r;./mem01;readprofile -m /boot/System.map-2.6.3 \
      >> captured_profile2.6.3

1446 poll_idle 24.9310

     4 delay_tsc                                  0.1667
    71 do_page_fault                              0.0536
    41 schedule                                   0.0238
    13 __might_sleep                              0.0637
     1 prepare_to_wait                            0.0068
     1 put_files_struct                           0.0042

3229 do_softirq 16.3081
1 run_timer_softirq 0.0022
4807 total 0.0262

2.4.20:

# readprofile -r;./mem01;readprofile -m /boot/System.map-2.4.20 \
      >> captured_profile2.4.20
   347 gunzip                                     0.1559
     8 acpi_restore_state_mem                     0.0021
     2 proc_dostring                              0.0033
     3 access_process_vm                          0.0054
     2 __mod_timer                                0.0038
     1 del_timer                                  0.0065
     1 update_one_process                         0.0036
     1 notifier_chain_unregister                  0.0052
   365 total                                      0.0020

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#3 lore raw

OK, try this profile for 2.4.20 instead (helps to use the right System.map):

   221 default_idle                               4.6042
     2 error_code                                 0.0333
     1 restore_fpu                                0.0312
    12 apm_bios_call_simple                       0.0833
     6 do_page_fault                              0.0047
     1 do_anonymous_page                          0.0039
     1 pte_alloc                                  0.0057
     2 set_page_dirty                             0.0179
     1 add_to_page_cache_unique                   0.0078
     2 mark_page_accessed                         0.0417
     1 kmem_cache_free                            0.0208
     1 activate_page                              0.0078
     1 lru_cache_add                              0.0104
     2 __lru_cache_del                            0.0179
    14 shrink_cache                               0.0179
     3 swap_out_pmd                               0.0117
     2 try_to_swap_out                            0.0048
     2 __free_pages_ok                            0.0030
     7 rmqueue                                    0.0129
     1 try_to_free_buffers                        0.0039
     1 ide_dmaproc                                0.0012
    53 fast_clear_page                            0.6625
   337 total                                      0.0003

Kirk

Quoted 36 lines · Kirk True

Hi all,

Quoted 1 line

A kernel profile might help.

Here's the profiles comparing the two versions. I don't understand why no "do_page_fault" shows up under 2.4.20 or why "gunzip" does.

Kirk

-------------------------------------------------

2.6.3:

# readprofile -r;./mem01;readprofile -m /boot/System.map-2.6.3 \
       >> captured_profile2.6.3

1446 poll_idle 24.9310

      4 delay_tsc                                  0.1667
     71 do_page_fault                              0.0536
     41 schedule                                   0.0238
     13 __might_sleep                              0.0637
      1 prepare_to_wait                            0.0068
      1 put_files_struct                           0.0042

3229 do_softirq 16.3081
1 run_timer_softirq 0.0022
4807 total 0.0262

2.4.20:

# readprofile -r;./mem01;readprofile -m /boot/System.map-2.4.20 \
       >> captured_profile2.4.20
Quoted 16 lines
    347 gunzip                                     0.1559
      8 acpi_restore_state_mem                     0.0021
      2 proc_dostring                              0.0033
      3 access_process_vm                          0.0054
      2 __mod_timer                                0.0038
      1 del_timer                                  0.0065
      1 update_one_process                         0.0036
      1 notifier_chain_unregister                  0.0052
    365 total                                      0.0020
Signature · 4 lines
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 5760000.1077296989@[10.10.2.4]

#7 lore raw
In reply to Kirk True
Quoted 6 lines · Kirk True

Hi all,

Executing the LTP "mem01" VM test shows a huge time discrepancy between 2.4.20 and 2.6.3. Under 2.4.20 the total time is around 5 seconds, while under 2.6.3 the system seems to hang for nearly a minute.

Where in particular should I start to look to see if it's a configuration/environment issue or a real problem? What other information would be helpful to know?

2.6 must start writeout, does it?
Can you post vmstat 1 logs for each kernel?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#8 lore raw
Quoted 1 line · Nick Piggin

2.6 must start writeout, does it?

Sorry, but how can I tell?

Quoted 1 line

Can you post vmstat 1 logs for each kernel?

The 2.4.20 vmstat is attached (formatting inline is ugly) but I couldn't get a vmstat for 2.6.3. Running strace vmstat shows that it's dying when reading from /proc/stat with a SEGFAULT. I googled about this but didn't see anything.

Kirk

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#9 lore raw

OK, here's the attachment...

procs memory swap io system cpu

r  b  w swpd   free buff  cache si  so bi bo  in    cs  us  sy  id
0  0  0    0 886072 9132  66152  0   0  6  2 101    23   0   0 100
author
#10 lore raw
In reply to Nick Piggin
Quoted 1 line

Can you post vmstat 1 logs for each kernel?

Attached is the vmstat output. The CPU stats are pretty interesting.

Kirk

2.4.20:

procs memory swap io system cpu

 r  b    swpd    free  buff  cache  si     so   bi     bo   in    cs  us  sy wa   id
 0  0       0  886312  8612  66152   0      0    0      0  109    88   1   0  0   99
 0  0       0  886312  8612  66152   0      0    0      0  132   140   1   0  0   99
 2  0    1016    5272  4212  28188   0    368   36    368  115   380  13  65  0   22
 1  0   45116    5236  4192  27872   0  40912    4  40912  724  1889   0  22  0   78
 1  1   76604    5236  4192  27872   0  30912    0  31004  646  1416   0  17  0   83
 0  2  112316    5236  4200  27872   0  36876   12  36872  675   873   0  14  0   86
 0  0    3560  930120  4192  27872   0  21536    8  21540  492   487   1  13  0   86
 0  0    3304  930372  4192  27876   0      0    0      0  124    45   0   0  0  100
 0  0    3304  930372  4192  27876   0      0    0      0  107    24   0   0  0  100

2.6.3:

procs memory swap io system cpu

 r   b    swpd    free  buff cache    si     so    bi     bo     in   cs  us  sy   wa  id
 0   0   41884  862460  752  11916     0      0     0      0   1008   67   0   0    0 100
 1   0   41884  692660  756  11912     0      0     0      0   1028  115   9  38    0  53
 1   0   41884  254580  756  11912     0      0     0      0   1003   15   3  97    0   0
 0  12   52536    4148  128   2232  1124  74012  2128  74184  16321  930   2  60   38   0
 4  11  112492    4104  152   2460   180   1856   332   1856   5281   23   0   0  100   0
 4  11  131836    4936  152   2728   600  19344  1084  19348   5782  196   0  32   68   0
 6   8  151528    4512  172   2624   388  19692   672  19704   5650  134   0  33   67   0
 3  10  172484    4688  176   3056   436  20956  1188  20960   5718  200   0  30   70   0
 4  10  191760    4448  176   3036   220  19276   668  19288   5533  135   0  53   47   0
 5  11  212456    4580  180   3056   416  20696   960  20704   5656  155   0  33   67   0
 4  10  231376    4256  180   3176   124  18920   488  18924   5429  145   7  50   43   0
 0   7   41996  880800  208   4340   496    124  1740    136   1177  174   0  28   72   0
 0   4   41996  878752  216   5608   900      0  2180      0   1092  164   0   2   98   0
 0   4   41996  876576  224   6880   896      0  2192      0   1094  174   0   0  100   0
 0   2   41996  874280  236   8032  1148      0  2308      0   1138  241   0   2   98   0
 0   3   41996  871976  248   9220  1044      0  2236      4   1121  226   0   2   98   0
 0   0   41996  870696  264   9728   768      0  1284     48   1074  161   7   1   39  52
 0   0   41996  870696  264   9728     0      0     0      0   1024   42   0   0    0 100
 0   0   41996  870700  264   9728     0      0     0      0   1003   17   0   0    0 100
 0   0   41996  870700  264   9728     0      0     0      0   1023   40   0   0    0 100
#11 lore raw
In reply to Kirk True
Quoted 4 lines · Kirk True

Executing the LTP "mem01" VM test shows a huge time discrepancy between 2.4.20 and 2.6.3. Under 2.4.20 the total time is around 5 seconds, while under 2.6.3 the system seems to hang for nearly a minute.

I'd be wondering if your disk system is correctly running in DMA mode.

On my 256MB test box, mem01 takes 31 seconds under 2.4.25, 25 seconds under 2.6.3-mm3.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

author
#12 lore raw

Hi Andrew,

Quoted 1 line

I'd be wondering if your disk system is correctly running in DMA mode.

Apparently support for my hardware isn't magically preset in 2.6.3 as it was somehow in 2.4. After including it in the kernel the values returned by hdparm -Tt /dev/hda/ were sped up by a factor of 10! Thanks!

But...

Quoted 2 lines

On my 256MB test box, mem01 takes 31 seconds under 2.4.25, 25 seconds under 2.6.3-mm3.

...I'm still seeing a factor of two speed slowdown between 2.4.20 and 2.6.3. Would it help to do a vmstat log/graph for the new results?

Kirk

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#13 lore raw
Quoted 9 lines · Kirk True

Hi Andrew,

Quoted 1 line

I'd be wondering if your disk system is correctly running in DMA mode.

Apparently support for my hardware isn't magically preset in 2.6.3 as it was somehow in 2.4. After including it in the kernel the values returned by hdparm -Tt /dev/hda/ were sped up by a factor of 10! Thanks!

Quoted 8 lines

But...

Quoted 2 lines

On my 256MB test box, mem01 takes 31 seconds under 2.4.25, 25 seconds under 2.6.3-mm3.

...I'm still seeing a factor of two speed slowdown between 2.4.20 and 2.6.3. Would it help to do a vmstat log/graph for the new results?

Try 2.6.3-mm3. It's tons better at this.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#14 lore raw
In reply to Kirk True

Hi all,

I just upgraded to 2.6.3-mm2 but am still seeing a factor of two speed slowdown between 2.4.20 and 2.6.3-mm2 for both sequential and random memory accesses into 1024 MB allocated from malloc.

I'm not trying to whine, I'm looking to explain this behavior and maybe be of some help somehow :)

Kirk

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

#15 lore raw
Quoted 4 lines · Kirk True

I just upgraded to 2.6.3-mm2 but am still seeing a factor of two speed slowdown between 2.4.20 and 2.6.3-mm2 for both sequential and random memory accesses into 1024 MB allocated from malloc.

2.6 VM has problems, but is usually OK for single-task stuff.

You'd need to tell us a lot about your machine, and provide sufficient information for others to reproduce what you're seeing.

And run some other tests to verify that your disk system is achieving the same bandwidth under both kernels. Not `hdparm -t' please, it is crap. Something like

	time (dd if=/dev/zero of=/mnt/x/foo bs=1M count=2000 ; sync)
	umount /mnt/x
	mount /mnt/x
	time dd if=/mnt/x/foo of=/dev/null bs=1M

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>