2.4.5-rc1 shm paging blocker
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 5 lines · Andrea Arcangeli
One of those patchsets (extracted from kernel CVS) is screwing
completely the paging of shm in 2.6.5-rc1. The malfunction is nearly no-swapping of shm and machine hanging forever in a sort of live lock scenario and I've to click reboot.
Can't reproduce it, sorry. Stock 2.6.5-rc1 on 4-way with mem=1024M, shmmax=3e9:
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id 0 0 0 908480 15140 76796 0 0 0 0 1017 30 0 0 0 100 0 0 0 908480 15148 76788 0 0 0 76 1010 16 0 0 0 100 0 0 0 908480 15148 76788 0 0 28 0 1030 56 2 0 2 96 41 21 0 865368 15148 115344 0 0 0 0 1011 33978 2 43 0 55 41 31 0 782576 15148 189940 0 0 0 0 1006 64580 5 95 0 0 47 40 0 701688 15148 259368 0 0 0 0 1006 13025 5 95 0 0 69 29 0 656960 15148 291668 0 0 0 0 1005 1918 6 94 0 0 62 29 0 644864 15156 291728 0 0 0 72 1007 4269 6 94 0 0 67 28 0 560192 15156 367888 0 0 0 0 1005 40205 5 95 0 0 67 30 0 511432 15156 406376 0 0 0 0 1005 24601 5 95 0 0 57 39 0 476744 15156 429224 0 0 0 0 1004 12228 6 94 0 0 80 46 0 443264 15156 450916 0 0 0 0 1005 16476 6 94 0 0 87 53 0 387840 15156 493484 0 0 0 0 1005 10994 6 94 0 0 90 49 0 310136 15156 561688 0 0 0 0 1005 39578 5 95 0 0 72 60 0 254384 15156 604936 0 0 0 0 1006 13469 6 94 0 0 104 59 0 198144 15156 652128 0 0 0 0 1006 46466 5 95 0 0 85 71 0 150144 15160 688844 0 0 0 4 1007 38554 5 95 0 0 101 63 0 110080 15160 718356 0 0 0 0 1006 15811 6 94 0 0 84 59 0 72832 15160 743856 0 0 0 0 1005 8047 5 95 0 0 96 60 0 46464 15160 757796 0 0 0 0 1005 11904 6 94 0 0 112 56 0 2696 9332 796740 0 0 0 0 1005 18330 6 94 0 0 85 53 684 2104 1380 796188 0 0 0 116 1000 4171 5 95 0 0 98 48 6416 3412 240 787788 428 5284 428 5284 1524 7250 4 96 0 0 94 47 8140 2908 240 784648 192 1988 192 1988 1034 3397 3 97 0 0 67 58 21632 3132 240 771612 328 10532 328 10532 1206 11524 5 95 0 0 94 73 26796 3796 244 765696 628 4692 628 4696 1184 21272 6 94 0 0 97 89 52528 5960 244 734396 652 8672 652 8672 1122 32257 7 93 0 0 47 97 57584 2164 184 730240 1284 28208 1284 28208 1103 15508 6 94 0 0
34 109 64376 2520 184 724280 1256 2072 1256 2072 1078 37640 4 96 0 0
procs memory swap io system cpu
r b swpd free buff cache si so bi bo in cs us sy wa id
131 104 68964 2232 184 719776 716 10304 716 10304 1079 9494 5 95 0 0
Is there some info I'm missing?
it seems the testing I did on 2.6.5-rc1 was incorrect (after testing another 2 combinations of kernels were the problem exists I may have not waited long enough while testing 2.6.5-rc1 for the first time and I mistaken a sluggishness for a live lock and I clicked reboot too early, that's the only reasonable explanation), it's working correctly now (like it works fine with 2.6.5-rc1 - backout + objrmap + anon_vma so it's impossible that it was my changes triggering it). If it happens again I'll let you know. I still have to find what cause it in another combinations of vm patches. Andrew, now that you've the testcase could you test the -mm tree, and see if happens there, some -mm patch is one of the diffs between the working tree and the non-working one.
However now I'll repeat all tests, if they works all flawlessy today I will be very annoyed, since the thing was definitely not swapping at all yesterday, no matter how long I awaited.
Now I start to wonder if kbuild may have screwed my kernel, effectively it was a development tree, I don't run make distclean anymore in between the kernel compiles (I almost don't feel the need of ccache anymoe ;), but maybe I'm wrong trusting the buildsystem can be smart enough.
Quoted 1 line · Andrea Arcangeli
it seems the testing I did on 2.6.5-rc1 was incorrect (after testing
Ok I solved it. The testing I did was perfectly correct.
But the new changes were colliding especially with objrmap and I didn't expect a special collision breaking _only_ objrmap (this has nothing to do with the anon_vma work I did on top of objrmap, Martin's tree will have the same problem that I had). The new changes alone works fine, but in combination with objrmap they cause everything to go into taking the nap in blk_congestion_wait. Since I tested objrmap previously extensively (on 2.6.3) I excluded originally that it could be a problem that could trigger only in combination with objrmap, but apparently the new code threats the referenced pages differently. This is the fix, and I don't need this fix to avoid a system-hang-livelock with 2.6.4/2.6.3 (or with 2.6.5-rc1 with the patches backed out).
It wasn't very easy to debug since there are around 200 tasks all rescheduling and all cpus calling shrink_cache in a live lock loop and then going back to sleep, and it's not a deadlock, but the machine hangs forever and clicking reboot is the only way out.
mm/objrmap.c renamed from mm/objrmap.c.~1~ +3 -5
@@ -151,14 +151,12 @@ page_referenced_inode(struct page *page) {struct address_space *mapping = page->as.mapping;struct vm_area_struct *vma; -int referenced; +int referenced =0 ;BUG_ON (PageSwapCache (page));if (down_trylock (&mapping->i_shared_sem)) -return 1 ; - - referenced =0 ; + goto out;list_for_each_entry (vma, &mapping->i_mmap, shared) referenced +=page_referenced_one (vma, page);
@@ -167,7 +165,7 @@ page_referenced_inode(struct page *page) referenced +=page_referenced_one (vma, page);up (&mapping->i_shared_sem); - + out:return referenced; }
this is 2.6.5-rc1 + objrmap:
got shmid 32768
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 11 2 0 927636 10776 44168 0 0 7 1 270 12 0 0 100 0 25 3 0 704792 10776 263604 0 0 0 0 1087 28592 10 91 0 0 32 0 0 590800 10776 373492 0 0 0 0 1084 151 15 85 0 0 46 1 0 471056 10776 489024 0 0 0 220 1132 104 14 86 0 0 53 1 0 393480 10784 562660 0 0 0 16 1087 91 15 85 0 0 60 0 0 315264 10792 636704 0 0 0 24 1086 90 15 85 0 0 67 0 0 249208 10792 698788 0 0 0 0 1084 83 15 85 0 0 74 0 0 214112 10792 729796 0 0 0 0 1084 89 18 82 0 0 81 1 0 149384 10792 790452 0 0 0 0 1085 89 15 85 0 0 81 0 0 105992 10792 829960 0 0 0 0 1085 79 17 83 0 0 88 0 0 60736 10792 871100 0 0 0 0 1085 81 15 85 0 0 89 0 0 3880 10536 924532 0 0 0 0 1091 68 15 85 0 0 91 1 0 2248 5768 928348 0 0 0 28 1092 151 8 92 0 0 89 18 588 11864 660 920564 216 640 216 640 7108 756 2 98 0 0 92 20 908 13276 532 918532 88 376 88 376 1960 317 2 98 0 0 92 21 964 13188 532 918440 112 88 112 88 959 170 3 97 0 0 90 18 652 2068 540 930364 272 132 280 132 985 175 4 96 0 0 92 14 700 2092 532 930448 68 156 68 156 1733 248 2 98 0 0 92 17 656 2016 532 930340 156 120 156 120 1110 325 4 96 0 0 90 24 12040 2532 188 937616 772 11972 920 12012 13078 1857 1 99 0 0 89 33 11992 3184 184 938300 604 380 648 428 3211 851 1 99 0 0 91 33 11964 3012 184 938328 24 36 24 36 969 127 1 99 0 0 91 39 12272 2348 164 938756 272 416 304 416 1406 373 1 99 0 0 92 36 13952 2300 152 940492 1704 2868 2716 2868 4864 923 1 99 0 0
[hang - live lock]
this is 2.6.5-rc1 + objrmap + the above fix:
procs -----------memory---------- ---swap-- -----io---- --system-- ----cpu----
r b swpd free buff cache si so bi bo in cs us sy id wa 9 1 0 934448 5400 43832 0 0 128 9 270 111 1 4 91 3 27 1 0 728920 5400 245588 0 0 0 0 1089 18437 11 89 0 0 41 4 0 592296 5400 378120 0 0 0 0 1083 100 13 87 0 0 49 1 0 516944 5400 449384 0 0 0 0 1086 89 15 85 0 0 56 1 0 449224 5408 513092 0 0 0 44 1087 91 15 85 0 0 69 0 0 372944 5416 585028 0 0 0 16 1089 110 14 86 0 0 69 0 0 302080 5416 652212 0 0 0 0 1085 68 15 85 0 0 76 0 0 276664 5416 673428 0 0 0 0 1086 83 15 85 0 0 76 1 0 204984 5416 741360 0 0 0 0 1083 70 14 86 0 0 83 0 0 162400 5416 779848 0 0 0 0 1085 81 15 86 0 0 83 0 0 111728 5420 826764 0 0 0 472 1095 88 13 87 0 0 90 1 0 64288 5420 870012 0 0 0 0 1195 86 15 85 0 0 90 1 0 24224 5420 906256 0 0 0 0 1083 73 15 86 0 0 92 4 0 2064 2500 927944 0 0 0 856 1214 143 11 89 0 0 91 35 4124 3016 1532 925900 64 2304 64 2324 1298 1092 9 91 0 0 95 25 7520 1996 1428 926948 3560 5816 3560 5820 1356 6653 10 90 0 0 80 43 25236 2704 168 937332 2144 17916 2144 17916 1371 8706 6 94 0 0 93 42 33312 3308 172 938176 1680 9752 2428 9752 1417 7547 2 99 0 0 99 71 44388 2232 176 931668 3392 11536 3720 11536 1503 21226 5 95 0 0 99 67 49540 2316 160 934400 2940 6084 2940 6084 1424 4030 0 100 0 0 97 59 53808 3408 164 931584 3196 5068 3392 5068 1426 2600 0 100 0 0 57 72 62420 5112 152 923976 3824 8200 3824 8200 1438 12514 2 98 0 0 99 87 69792 2928 152 927704 3184 9288 3184 9288 1518 11193 1 99 0 0 90 74 74392 3984 152 925828 2904 5556 2904 5556 1389 2431 0 100 0 0 70 82 81580 2748 152 921060 5672 8156 5672 8156 1398 17712 1 99 0 0 94 80 88064 2232 152 921404 5008 7900 5008 7900 1453 17488 1 100 0 0 99 73 93668 5044 172 920308 3372 6340 3532 6340 1437 4039 0 100 0 0 97 69 99784 2232 152 920132 3900 6664 3900 6664 1380 15020 0 100 0 0 70 71 105624 4308 152 919736 4300 7356 4300 7356 1464 9876 0 100 0 0 97 71 112260 8492 148 916520 3988 7500 3988 7500 1444 22537 0 100 0 0 49 90 118828 3668 148 914696 6076 7208 6076 7208 1464 27088 0 100 0 0 99 86 125552 3216 148 919176 2616 7608 2616 7608 1423 9301 0 100 0 0 98 84 131004 5204 148 917896 3192 6396 3192 6396 1444 2116 0 100 0 0 78 76 136884 2768 148 917520 4696 6844 4696 6844 1403 12965 0 100 0 0 84 73 142328 8108 148 913836 3504 6140 3536 6140 1363 8199 0 100 0 0 81 81 149696 5040 148 913204 4000 8356 4024 8356 1532 32326 0 100 0 0 99 81 155940 2876 148 916620 3144 6996 3144 6996 1543 7558 0 100 0 0 99 80 161800 5312 148 912484 4348 6912 4348 6912 1412 8694 0 100 0 0 99 75 168108 4428 148 914340 3656 7280 3656 7280 1443 12129 0 100 0 0 88 79 174432 3120 152 914548 4952 6936 5020 6936 1368 15244 0 100 0 0
2.6.3 + objrmap (w/o the above) fix is also working like the above w/o any live lock condition.
So my testing was accurate, it was just my assumption that objrmap wouldn't especially collide with the new 2.6.5-rc1 changes that was wrong and that generated the confusion.
Now, if the new congestion-nap code is better or worse I don't know, probably it's safer to return 0 from a failure in taking the lock anyways, so I will apply the objrmap fix anyways (Martin you must pick it too if you want to use objrmap on 2.6.5-rc1+), but at least now things makes sense again ;).
So now the last bit I've to fix is bttv + anon_vma (bttv is causing a BUG_ON to trigger in xawtv context).