Regression caused by commit 4bdc33ed ("NFSDv4.2: Add NFS v4.2 support to the NFS server")
Subsystems (2)
DQpIaSBhbGwsDQoNCk15IGVtYmVkZGVkIGRldmVsb3BtZW50IGJveCBmYWlscyB0byBORlMtYm9v dCB3aXRoIE5GUyBzZXJ2ZXIgd2hpY2ggdXNlcyByZWNlbnQga2VybmVsLg0KDQpVc2luZyBnaXQg YmlzZWN0LCBJIGZvdW5kIGl0IGlzIGNhdXNlZCBieSBjb21taXQgNGJkYzMzZWQgKCJORlNEdjQu MjogQWRkIE5GUyB2NC4yIHN1cHBvcnQgdG8gdGhlIE5GUyBzZXJ2ZXIiKS4NCg0KDQoxLiBkbWVz ZyAoTkZTIGJvb3QgZmFpbHVyZSBjYXNlKQ0KDQouLi4NClsgICAgMi4wNDA4OTNdIEFERFJDT05G KE5FVERFVl9VUCk6IGV0aDA6IGxpbmsgaXMgbm90IHJlYWR5DQpbICAgIDIuMDQ2MjA3XSBlMTAw MDogZXRoMCBOSUMgTGluayBpcyBVcCAxMDAwIE1icHMgRnVsbCBEdXBsZXgsIEZsb3cgQ29udHJv bDogUlgNClsgICAgMi4wNTM1NzBdIEFERFJDT05GKE5FVERFVl9DSEFOR0UpOiBldGgwOiBsaW5r IGJlY29tZXMgcmVhZHkNClsgICAgMy4wNTUwMjNdIElQLUNvbmZpZzogR3Vlc3NpbmcgbmV0bWFz ayAyNTUuMjU1LjAuMA0KWyAgICAzLjA1OTk3OV0gSVAtQ29uZmlnOiBHYXRld2F5IG5vdCBvbiBk aXJlY3RseSBjb25uZWN0ZWQgbmV0d29yay4NClsgICAgMy4wNjYzMzBdIExvb2tpbmcgdXAgcG9y dCBvZiBSUEMgMTAwMDAzLzIgb24gMTY1LjIxMy44OC4yNDkNClsgICAgMy4wNzQwMDFdIExvb2tp bmcgdXAgcG9ydCBvZiBSUEMgMTAwMDA1LzEgb24gMTY1LjIxMy44OC4yNDkNClsgICAgMy4xMjI4 NzhdIFZGUzogVW5hYmxlIHRvIG1vdW50IHJvb3QgZnMgdmlhIE5GUywgdHJ5aW5nIGZsb3BweS4N ClsgICAgMy4xMjkxMzRdIFZGUzogQ2Fubm90IG9wZW4gcm9vdCBkZXZpY2UgIm5mcyIgb3IgdW5r bm93bi1ibG9jaygyLDApDQpbICAgIDMuMTM1NDc4XSBQbGVhc2UgYXBwZW5kIGEgY29ycmVjdCAi cm9vdD0iIGJvb3Qgb3B0aW9uOyBoZXJlIGFyZSB0aGUgYXZhaWxhYmxlIHBhcnRpdGlvbnM6DQpb ICAgIDMuMTQzODMxXSAxZjAwICAgICAgICAgICAgMzA3MiBtdGRibG9jazAgKGRyaXZlcj8pDQpb ICAgIDMuMTQ4Nzk4XSAxZjAxICAgICAgICAgICAgICA2NCBtdGRibG9jazEgKGRyaXZlcj8pDQpb ICAgIDMuMTUzNzU4XSAxZjAyICAgICAgICAgICAgICA2NCBtdGRibG9jazIgKGRyaXZlcj8pDQpb ICAgIDMuMTU4NzE5XSAxZjAzICAgICAgICAgICAgICA2NCBtdGRibG9jazMgKGRyaXZlcj8pDQpb ICAgIDMuMTYzNjgyXSAxZjA0ICAgICAgICAgICAgICA2NCBtdGRibG9jazQgKGRyaXZlcj8pDQpb ICAgIDMuMTY4NjQ0XSAxZjA1ICAgICAgICAgICAgICA2NCBtdGRibG9jazUgKGRyaXZlcj8pDQpb ICAgIDMuMTczNjA3XSAxZjA2ICAgICAgICAgICAgICA2NCBtdGRibG9jazYgKGRyaXZlcj8pDQpb ICAgIDMuMTc4NTY4XSAwODAwICAgICAgIDQ4ODM4NjU4NCBzZGEgZHJpdmVyOiBzZA0KWyAgICAz LjE4MzA5OV0gICAwODAxICAgICAgICAgIDUwNjAxNiBzZGExDQpbICAgIDMuMTg2OTI3XSAgIDA4 MDIgICAgICAgICA0MDA4MjE3IHNkYTINClsgICAgMy4xOTA3NTVdICAgMDgwMyAgICAgICA0ODM4 Njk3Njcgc2RhMw0KWyAgICAzLjE5NDU4NF0gYjMwMCAgICAgICAgIDE4ODAwNjQgbW1jYmxrMCBk cml2ZXI6IG1tY2Jsaw0KWyAgICAzLjE5OTgwMl0gICBiMzAxICAgICAgICAgICAgNDA5NiBtbWNi bGswcDENClsgICAgMy4yMDQwNjNdICAgYjMwMiAgICAgICAgICAxMDI0MDAgbW1jYmxrMHAyDQpb ICAgIDMuMjA4MzMwXSAgIGIzMDMgICAgICAgICAgICA0MDk2IG1tY2JsazBwMw0KWyAgICAzLjIx MjU5NF0gICBiMzA0ICAgICAgICAgICAgICAgMSBtbWNibGswcDQNClsgICAgMy4yMTY4NTVdICAg YjMwNSAgICAgICAgICAgIDIwNDggbW1jYmxrMHA1DQpbICAgIDMuMjIxMTE2XSAgIGIzMDYgICAg ICAgICAgICAyMDQ4IG1tY2JsazBwNg0KWyAgICAzLjIyNTM4Ml0gICBiMzA3ICAgICAgICAgICAg MjA0OCBtbWNibGswcDcNClsgICAgMy4yMjk2NDRdICAgYjMwOCAgICAgICAgICAgIDQwOTYgbW1j YmxrMHA4DQpbICAgIDMuMjMzOTA2XSAgIGIzMDkgICAgICAgICAgIDEyMjg4IG1tY2JsazBwOQ0K WyAgICAzLjIzODE3Nl0gICBiMzBhICAgICAgICAgICAxNjM4NCBtbWNibGswcDEwDQpbICAgIDMu MjQyNTI0XSAgIGIzMGIgICAgICAgICAgMTQyMzM2IG1tY2JsazBwMTENClsgICAgMy4yNDY4Njld ICAgYjMwYyAgICAgICAgIDE1NzI4NjQgbW1jYmxrMHAxMg0KWyAgICAzLjI1MTIxOV0gYjMyMCAg ICAgICAgICAgMTIyODggbW1jYmxrMGdwMSAoZHJpdmVyPykNClsgICAgMy4yNTYyNzJdIGIzMTAg ICAgICAgICAgIDEyMjg4IG1tY2JsazBncDAgKGRyaXZlcj8pDQpbICAgIDMuMjYxMzIwXSBLZXJu ZWwgcGFuaWMgLSBub3Qgc3luY2luZzogVkZTOiBVbmFibGUgdG8gbW91bnQgcm9vdCBmcyBvbiB1 bmtub3duLWJsb2NrKDIsMCkNClsgICAgMy4yNjk1NjZdIFBpZDogMSwgY29tbTogc3dhcHBlciBO b3QgdGFpbnRlZCAyLjYuMzUgIzENClsgICAgMy4yNzQ3NzZdIENhbGwgVHJhY2U6DQpbICAgIDMu Mjc3MjMyXSAgWzw4MGQwZGI1Yj5dID8gcHJpbnRrKzB4MWUvMHgyMA0KWyAgICAzLjI4MTQ5Ml0g IFs8ODBkMGRhZDE+XSBwYW5pYysweDY1LzB4ZDENClsgICAgMy4yODU0OTVdICBbPDgwZWI5Y2Uz Pl0gbW91bnRfYmxvY2tfcm9vdCsweDEyNS8weDFiZQ0KWyAgICAzLjI5MDYzMV0gIFs8ODA5ZDFm NmQ+XSA/IHN5c19ta25vZCsweDJkLzB4MzANClsgICAgMy4yOTUxNTZdICBbPDgwZWI5ZjZkPl0g bW91bnRfcm9vdCsweGQwLzB4ZjINClsgICAgMy4yOTk1OTFdICBbPDgwZWJhMGQ5Pl0gcHJlcGFy ZV9uYW1lc3BhY2UrMHgxNGEvMHgxODQNClsgICAgMy4zMDQ4MDNdICBbPDgwOWM0NGY2Pl0gPyBz eXNfYWNjZXNzKzB4MjYvMHgzMA0KWyAgICAzLjMwOTQxMV0gIFs8ODBlYjlhNGU+XSBrZXJuZWxf aW5pdCsweDI1ZS8weDI2ZQ0KWyAgICAzLjMxNDEwNV0gIFs8ODBlYjk3ZjA+XSA/IGtlcm5lbF9p bml0KzB4MC8weDI2ZQ0KWyAgICAzLjMxODgwMF0gIFs8ODA5MDMyNDI+XSBrZXJuZWxfdGhyZWFk X2hlbHBlcisweDYvMHgxMA0KDQoNCjIuIENsaWVudCAobXkgZW1iZWRkZWQgYm94KSBjb25maWd1 cmF0aW9uDQogIEl0J3Mga2VybmVsIDIuNi4zNSBiYXNlZCwgYW5kIGhhcyBmb2xsb3dpbmcgTkZT IGtlcm5lbCBjb25maWdzLg0KDQojIGdyZXAgTkZTIC5jb25maWcNCkNPTkZJR19ORlNfRlM9eQ0K Q09ORklHX05GU19WMz15DQpDT05GSUdfTkZTX1YzX0FDTD15DQpDT05GSUdfTkZTX1Y0PXkNCiMg Q09ORklHX05GU19WNF8xIGlzIG5vdCBzZXQNCkNPTkZJR19ST09UX05GUz15DQojIENPTkZJR19O RlNEIGlzIG5vdCBzZXQNCkNPTkZJR19ORlNfQUNMX1NVUFBPUlQ9eQ0KQ09ORklHX05GU19DT01N T049eQ0KDQoNCjMuIFNlcnZlciAoTkZTRCkgY29uZmlndXJhdGlvbg0KICAgRmVkb3JhIDE5ICsg bGF0ZXN0IGxpbnVzIGdpdCBrZXJuZWwgMy4xMi4wLXJjMisgKGNvbW1pdCAyMjM1NmY0NCwgbW06 IFBsYWNlIHByZWVtcHRpb24gcG9pbnQgaW4gZG9fbWxvY2thbGwoKSBsb29wKQ0KDQoNCjQuIHdv cmthcm91bmQNCg0KUmV2ZXJ0aW5nIHRoZSBjb21taXQgNGJkYzMzZWQgcmVzb2x2ZXMgbXkgaXNz dWUsIE5GUyBib290IGlzIHdvcmtpbmcgdGhlbi4NCkkndmUgZG9uZSBnaXQgYmlzZWN0LCBidXQg bG9zdCB0aGUgcmVzdWx0aW5nIGJpc2VjdCBsb2cgZHVlIHRvIHN1ZGRlbiBwb3dlciBsb3NzIDoo Lg0KDQpCZXN0IHJlZ2FyZHMsDQpKb25nbWFuIEhlbw0K
Hi Jongman,
Is the panic on your client or server? I don't see how the patch your bisect led you to could cause the problem, since all it does is expand the minor version array on the server. Your client doesn't have NFSD enabled, so this code shouldn't even be affecting it.
A few questions: what is your /etc/exports on the server? What version of NFS are you using for nfsroot?
Thanks!
Anna
Quoted 88 lines · Jongman Heo
Hi all,
My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
Using git bisect, I found it is caused by commit
4bdc33ed("NFSDv4.2: Add NFS v4.2 support to the NFS server").1. dmesg (NFS boot failure case)
...
[ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 3.055023] IP-Config: Guessing netmask 255.255.0.0 [ 3.059979] IP-Config: Gateway not on directly connected network. [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249 [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249 [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy. [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0) [ 3.135478] Please append a correct "root=" boot option; here are the available partitions: [ 3.143831] 1f00 3072 mtdblock0 (driver?) [ 3.148798] 1f01 64 mtdblock1 (driver?) [ 3.153758] 1f02 64 mtdblock2 (driver?) [ 3.158719] 1f03 64 mtdblock3 (driver?) [ 3.163682] 1f04 64 mtdblock4 (driver?) [ 3.168644] 1f05 64 mtdblock5 (driver?) [ 3.173607] 1f06 64 mtdblock6 (driver?) [ 3.178568] 0800 488386584 sda driver: sd [ 3.183099] 0801 506016 sda1 [ 3.186927] 0802 4008217 sda2 [ 3.190755] 0803 483869767 sda3 [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk [ 3.199802] b301 4096 mmcblk0p1 [ 3.204063] b302 102400 mmcblk0p2 [ 3.208330] b303 4096 mmcblk0p3 [ 3.212594] b304 1 mmcblk0p4 [ 3.216855] b305 2048 mmcblk0p5 [ 3.221116] b306 2048 mmcblk0p6 [ 3.225382] b307 2048 mmcblk0p7 [ 3.229644] b308 4096 mmcblk0p8 [ 3.233906] b309 12288 mmcblk0p9 [ 3.238176] b30a 16384 mmcblk0p10 [ 3.242524] b30b 142336 mmcblk0p11 [ 3.246869] b30c 1572864 mmcblk0p12 [ 3.251219] b320 12288 mmcblk0gp1 (driver?) [ 3.256272] b310 12288 mmcblk0gp0 (driver?) [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0) [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1 [ 3.274776] Call Trace: [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20 [ 3.281492] [<80d0dad1>] panic+0x65/0xd1 [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30 [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2 [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184 [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30 [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x102. Client (my embedded box) configuration
It's kernel 2.6.35 based, and has following NFS kernel configs.# grep NFS .config CONFIG_NFS_FS=y CONFIG_NFS_V3=y CONFIG_NFS_V3_ACL=y CONFIG_NFS_V4=y # CONFIG_NFS_V4_1 is not set CONFIG_ROOT_NFS=y # CONFIG_NFSD is not set CONFIG_NFS_ACL_SUPPORT=y CONFIG_NFS_COMMON=y3. Server (NFSD) configuration
Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit22356f44, mm: Place preemption point indo_mlockall()loop)4. workaround
Reverting the commit
4bdc33edresolves my issue, NFS boot is working then. I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.Best regards,
Jongman Heo
Quoted 74 lines · Jongman Heo
My embedded development box fails to NFS-boot with NFS server which uses recent kernel.
Using git bisect, I found it is caused by commit
4bdc33ed("NFSDv4.2: Add NFS v4.2 support to the NFS server").1. dmesg (NFS boot failure case)
...
[ 2.040893] ADDRCONF(NETDEV_UP): eth0: link is not ready [ 2.046207] e1000: eth0 NIC Link is Up 1000 Mbps Full Duplex, Flow Control: RX [ 2.053570] ADDRCONF(NETDEV_CHANGE): eth0: link becomes ready [ 3.055023] IP-Config: Guessing netmask 255.255.0.0 [ 3.059979] IP-Config: Gateway not on directly connected network. [ 3.066330] Looking up port of RPC 100003/2 on 165.213.88.249 [ 3.074001] Looking up port of RPC 100005/1 on 165.213.88.249 [ 3.122878] VFS: Unable to mount root fs via NFS, trying floppy. [ 3.129134] VFS: Cannot open root device "nfs" or unknown-block(2,0) [ 3.135478] Please append a correct "root=" boot option; here are the available partitions: [ 3.143831] 1f00 3072 mtdblock0 (driver?) [ 3.148798] 1f01 64 mtdblock1 (driver?) [ 3.153758] 1f02 64 mtdblock2 (driver?) [ 3.158719] 1f03 64 mtdblock3 (driver?) [ 3.163682] 1f04 64 mtdblock4 (driver?) [ 3.168644] 1f05 64 mtdblock5 (driver?) [ 3.173607] 1f06 64 mtdblock6 (driver?) [ 3.178568] 0800 488386584 sda driver: sd [ 3.183099] 0801 506016 sda1 [ 3.186927] 0802 4008217 sda2 [ 3.190755] 0803 483869767 sda3 [ 3.194584] b300 1880064 mmcblk0 driver: mmcblk [ 3.199802] b301 4096 mmcblk0p1 [ 3.204063] b302 102400 mmcblk0p2 [ 3.208330] b303 4096 mmcblk0p3 [ 3.212594] b304 1 mmcblk0p4 [ 3.216855] b305 2048 mmcblk0p5 [ 3.221116] b306 2048 mmcblk0p6 [ 3.225382] b307 2048 mmcblk0p7 [ 3.229644] b308 4096 mmcblk0p8 [ 3.233906] b309 12288 mmcblk0p9 [ 3.238176] b30a 16384 mmcblk0p10 [ 3.242524] b30b 142336 mmcblk0p11 [ 3.246869] b30c 1572864 mmcblk0p12 [ 3.251219] b320 12288 mmcblk0gp1 (driver?) [ 3.256272] b310 12288 mmcblk0gp0 (driver?) [ 3.261320] Kernel panic - not syncing: VFS: Unable to mount root fs on unknown-block(2,0) [ 3.269566] Pid: 1, comm: swapper Not tainted 2.6.35 #1 [ 3.274776] Call Trace: [ 3.277232] [<80d0db5b>] ? printk+0x1e/0x20 [ 3.281492] [<80d0dad1>] panic+0x65/0xd1 [ 3.285495] [<80eb9ce3>] mount_block_root+0x125/0x1be [ 3.290631] [<809d1f6d>] ? sys_mknod+0x2d/0x30 [ 3.295156] [<80eb9f6d>] mount_root+0xd0/0xf2 [ 3.299591] [<80eba0d9>] prepare_namespace+0x14a/0x184 [ 3.304803] [<809c44f6>] ? sys_access+0x26/0x30 [ 3.309411] [<80eb9a4e>] kernel_init+0x25e/0x26e [ 3.314105] [<80eb97f0>] ? kernel_init+0x0/0x26e [ 3.318800] [<80903242>] kernel_thread_helper+0x6/0x102. Client (my embedded box) configuration
It's kernel 2.6.35 based, and has following NFS kernel configs.# grep NFS .config CONFIG_NFS_FS=y CONFIG_NFS_V3=y CONFIG_NFS_V3_ACL=y CONFIG_NFS_V4=y # CONFIG_NFS_V4_1 is not set CONFIG_ROOT_NFS=y # CONFIG_NFSD is not set CONFIG_NFS_ACL_SUPPORT=y CONFIG_NFS_COMMON=y
Quoted 8 lines
3. Server (NFSD) configuration
Fedora 19 + latest linus git kernel 3.12.0-rc2+ (commit22356f44, mm: Place preemption point indo_mlockall()loop)4. workaround
Reverting the commit
4bdc33edresolves my issue, NFS boot is working then. I've done git bisect, but lost the resulting bisect log due to sudden power loss :(.
So when you say you revert that commit, you mean you revert it on your *server*, right? You're not changing the client at all throughout these tests?
A network trace might be interesting: so, on the server, run
tcpdump -s0 -wtmp.pcap -ieth0
(replace eth0 by the right network interface), then try booting the client and after the client fails, kill tcpdump and send us a copy of tmp.pcap.
(And also you might want to fire up "wireshark tmp.pcap" and take a look yourself--you'll probably see something like a version mismatch error in the network traffic.)
--b.