Skip to contentSkip to conversation

Regression in xen-netfront on v3.6.

DISCUSSION Konrad Rzeszutek Wilk 4 participants 6 messages Aug 1, 2012 – Aug 6, 2012 Networking

lore
Share & tools
lore
#0 lore raw

So I hadn't done a git bisection yet. But if I choose git commit:
4b24ff71108164e047cf2c95990b77651163e315
Merge tag 'for-v3.6' of git://git.infradead.org/battery-2.6

Pull battery updates from Anton Vorontsov:

everything works nicely. Anything past that, so these merges:

konrad@phenom:~/ssd/linux$ git log --oneline --merges 4b24ff71108164e047cf2c95990b77651163e315..linus/master
2d53492 Merge tag 'irqdomain-for-linus' of git://git.secretlab.ca/git/linux-2.6
ac694db Merge branch 'akpm' (Andrew's patch-bomb)
a40a1d3 Merge tag 'vfio-for-v3.6' of git://github.com/awilliam/linux-vfio
3e9a970 Merge tag 'random_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/random
941c872 Merge tag 'rdma-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband
8762541 Merge branch 'v4l_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
6dbb35b Merge tag 'nfs-for-3.6-2' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
fd37ce3 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
1da9b6b Merge branches 'cma', 'ipoib', 'ocrdma' and 'qib' into for-next
6aeea3e Merge remote-tracking branch 'origin' into irqdomain/next
931efdf Merge branch 'v4l_for_linus' into staging/for_v3.6
80c1834 Merge tag 'v3.5-rc6' into irqdomain/next

are the culprit. I think it might be the networking pull but not sure. Ian any thoughts?

Using config file "/test.xm".
Started domain latest (id=2)

[    0.000000] console [hvc0] enabled, bootconsole disabled
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 2294.530 MHz processor
[    0.000999] Calibrating delay loop (skipped), value calculated using timer frequency.. 4589.06 BogoMIPS (lpj=2294530)
[    0.000999] pid_max: default: 32768 minimum: 301
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] SELinux:  Starting in permissive mode
[    0.000999] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.001520] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001875] Mount-cache hash table entries: 256
[    0.002007] Initializing cgroup subsys cpuacct
[    0.002013] Initializing cgroup subsys freezer
[    0.002070] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.002070] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.002084] CPU: Physical Processor ID: 0
[    0.002087] CPU: Processor Core ID: 0
[    0.002094] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.002094] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[    0.002094] tlb_flushall_shift is 0x5
[    0.002164] SMP alternatives: switching to UP code
[    0.025291] Freeing SMP alternatives: 24k freed
[    0.025356] cpu 0 spinlock event irq 17
[    0.025383] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.
[    0.025551] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.025576] Brought up 1 CPUs
[    0.028642] kworker/u:0 (14) used greatest stack depth: 5936 bytes left
[    0.028675] Grant tables using version 2 layout.
[    0.028691] Grant table initialized
[    0.047616] RTC time: 165:165:165, date: 165/165/65
[    0.047661] NET: Registered protocol family 16
[    0.048184] dca service started, version 1.12.1
[    0.048545] PCI: setting up Xen PCI frontend stub
[    0.048552] PCI: pci_cache_line_size set to 64 bytes
[    0.049543] kworker/u:0 (51) used greatest stack depth: 5472 bytes left
[    0.054147] bio: create slab <bio-0> at 0
[    0.054240] ACPI: Interpreter disabled.
[    0.054288] xen/balloon: Initialising balloon driver.
[    0.055127] xen-balloon: Initialising balloon driver.
[    0.055127] vgaarb: loaded
[    0.056125] usbcore: registered new interface driver usbfs
[    0.056162] usbcore: registered new interface driver hub
[    0.056217] usbcore: registered new device driver usb
[    0.056425] PCI: System does not support PCI
[    0.056431] PCI: System does not support PCI
[    0.056617] NetLabel: Initializing
[    0.056624] NetLabel:  domain hash size = 128
[    0.056627] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.056642] NetLabel:  unlabeled traffic allowed by default
[    0.056725] Switching to clocksource xen
[    0.056795] pnp: PnP ACPI: disabled
[    0.058698] NET: Registered protocol family 2
[    0.059805] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    0.061110] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.061243] TCP: Hash tables configured (established 524288 bind 65536)
[    0.061281] TCP: reno registered
[    0.061304] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[    0.061341] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[    0.061425] NET: Registered protocol family 1
[    0.061492] RPC: Registered named UNIX socket transport module.
[    0.061498] RPC: Registered udp transport module.
[    0.061504] RPC: Registered tcp transport module.
[    0.061510] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.061518] PCI: CLS 0 bytes, default 64
[    0.061643] Trying to unpack rootfs image as initramfs...
[    0.382189] Freeing initrd memory: 362080k freed
[    0.499615] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.499831] Machine check injector initialized
[    0.500181] microcode: CPU0 sig=0x206d2, pf=0x1, revision=0x8000020c
[    0.500229] microcode: Microcode Update Driver: v2.00 <[email protected]>, Peter Oruba
[    0.500544] audit: initializing netlink socket (disabled)
[    0.500566] type=2000 audit(1343845740.901:1): initialized
[    0.515227] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.515358] VFS: Disk quotas dquot_6.5.2
[    0.515386] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.515525] NFS: Registering the id_resolver key type
[    0.515544] Key type id_resolver registered
[    0.515551] Key type id_legacy registered
[    0.515599] NTFS driver 2.1.30 [Flags: R/W].
[    0.515706] msgmni has been set to 8021
[    0.515765] SELinux:  Registering netfilter hooks
[    0.516222] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.516232] io scheduler noop registered
[    0.516236] io scheduler deadline registered
[    0.516243] io scheduler cfq registered (default)
[    0.516337] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.516442] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    0.532923] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.533399] Non-volatile memory driver v1.3
[    0.533406] Linux agpgart interface v0.103
[    0.533588] [drm] Initialized drm 1.1.0 20060810
[    0.535196] brd: module loaded
[    0.535992] loop: module loaded
[    0.536344] libphy: Fixed MDIO Bus: probed
[    0.536351] tun: Universal TUN/TAP device driver, 1.6
[    0.536354] tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
[    0.536419] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.6.0-k
[    0.536428] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.536721] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.536729] ehci_hcd: block sizes: qh 104 qtd 96 itd 192 sitd 96
[    0.536770] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.536776] ohci_hcd: block sizes: ed 80 td 96
[    0.536826] uhci_hcd: USB Universal Host Controller Interface driver
[    0.536929] usbcore: registered new interface driver usblp
[    0.536977] usbcore: registered new interface driver libusual
[    0.537164] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    0.538013] i8042: No controller found
[    0.538103] mousedev: PS/2 mouse device common for all mice
[    0.598349] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.598404] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.598559] EFI Variables Facility v0.08 2004-May-17
[    0.598701] Netfilter messages via NETLINK v0.30.
[    0.598719] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    0.598790] ctnetlink v0.93: registering with nfnetlink.
[    0.598971] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.599007] TCP: cubic registered
[    0.599014] Initializing XFRM netlink socket
[    0.599043] NET: Registered protocol family 10
[    0.599238] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    0.599374] sit: IPv6 over IPv4 tunneling driver
[    0.599589] NET: Registered protocol family 17
[    0.599618] Key type dns_resolver registered
[    0.599808] PM: Hibernation image not present or could not be loaded.
[    0.599824] registered taskstats version 1
[    0.599848] XENBUS: Device with no driver: device/vkbd/0
[    0.599852] XENBUS: Device with no driver: device/vfb/0
[    0.599856] XENBUS: Device with no driver: device/vbd/51712
[    0.599860] XENBUS: Device with no driver: device/vif/0
[    0.599886]   Magic number: 1:252:3141
[    0.600271] Freeing unused kernel memory: 704k freed
[    0.600500] Write protecting the kernel read-only data: 8192k
[    0.602437] Freeing unused kernel memory: 132k freed
[    0.602589] Freeing unused kernel memory: 340k freed
init started: BusyBox v1.14.3 (2012-08-01 13:52:44 EDT)
[    0.607489] consoletype (1056) used greatest stack depth: 5288 bytes left
Mounting directories  [  OK  ]
mount: mount point /proc/bus/usb does not exist
[    0.781044] modprobe (1085) used greatest stack depth: 5048 bytes left
mount: mount point /sys/kernel/config does not exist
[    0.785748] core_filesystem (1057) used greatest stack depth: 4968 bytes left
[    0.793721] input: Xen Virtual Keyboard as /devices/virtual/input/input0
[    0.793892] input: Xen Virtual Pointer as /devices/virtual/input/input1
[    1.010121] Initialising Xen virtual ethernet driver.
[    1.124604] blkfront: xvda: flush diskcache: enabled
[    1.126118]  xvda: xvda1 xvda2 xvda3 xvda4
[    1.239316] udevd (1128): /proc/1128/oom_adj is deprecated, please use /proc/1128/oom_score_adj instead.

udevd-work[1130]: error opening ATTR{/sys/devices/system/cpu/cpu0/online} for writing: No such file or directory

[    1.395080] ip (1408) used greatest stack depth: 3912 bytes left

Waiting for devices [ OK ]
Waiting for init.pre_custom [ OK ]
Waiting for fb [ OK ]
Starting..[/dev/fb0]
/dev/fb0: len:0
/dev/fb0: bits/pixel32
(7f44ddbc2000): Writting .. [800:600]
Done!
FATAL: Module agpgart_intel not found.

[    1.652131] [drm] radeon kernel modesetting enabled.

WARNING: Error inserting video (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/acpi/video.ko): No such device
WARNING: Error inserting mxm_wmi (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/platform/x86/mxm-wmi.ko): No such device
WARNING: Error inserting drm_kms_helper (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
WARNING: Error inserting ttm (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/ttm/ttm.ko): No such device
FATAL: Error inserting nouveau (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/nouveau/nouveau.ko): No such device

[    1.660288] Console: switching to colour frame buffer device 100x37

WARNING: Error inserting drm_kms_helper (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
FATAL: Error inserting i915 (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/i915/i915.ko): No such device Starting..[/dev/fb0]
/dev/fb0: len:0
/dev/fb0: bits/pixel32
(7fb8669cc000): Writting .. [800:600]
Done!
VGA: 0000:
Waiting for network [ OK ]

Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  [    1.908592] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[    1.908643] IP: [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.908703] PGD ea1df067 PUD e8ada067 PMD 0 
[    1.908774] Oops: 0000 [#1] SMP 
[    1.908797] Modules linked in: fbcon tileblit font radeon bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd
[    1.908938] CPU 0 
[    1.908950] Pid: 2165, comm: ip Not tainted 3.5.0upstream-08854-g444fa66 #1  
[    1.908983] RIP: e030:[<ffffffffa0037750>]  [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.909029] RSP: e02b:ffff8800ffc03db8  EFLAGS: 00010282
[    1.909055] RAX: ffff8800ea010140 RBX: ffff8800f00e86c0 RCX: 000000000000009a
[    1.909055] RDX: 0000000000000040 RSI: 000000000000005a RDI: ffff8800fa7dee80
[    1.909055] RBP: ffff8800ffc03ee8 R08: ffff8800f00e86d8 R09: ffff8800ea010000
[    1.909055] R10: dead000000200200 R11: dead000000100100 R12: ffff8800fa7dee80
[    1.909055] R13: 000000000000005a R14: ffff8800fa7dee80 R15: 0000000000000200
[    1.909055] FS:  00007fbafc188700(0000) GS:ffff8800ffc00000(0000) knlGS:0000000000000000
[    1.909055] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    1.909055] CR2: 0000000000000010 CR3: 00000000ea108000 CR4: 0000000000002660
[    1.909055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    1.909055] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    1.909055] Process ip (pid: 2165, threadinfo ffff8800ea0f2000, task ffff8800fa783040)
[    1.909055] Stack:
[    1.909055]  ffff8800e27e5040 ffff8800ffc03e88 ffff8800ffc03e68 ffff8800ffc03e48
[    1.909055]  7fffffffffffffff ffff8800ffc03e00 ffff8800e27e5040 ffff8800f00e86d8
[    1.909055]  ffff8800ffc03eb0 00000040ffffffff ffff8800f00e8000 00000000ffc03e30
[    1.909055] Call Trace:
[    1.909055]  <IRQ> 
[    1.909055]  [<ffffffff81066028>] ? pvclock_clocksource_read+0x58/0xd0
[    1.909055]  [<ffffffff81486352>] net_rx_action+0x112/0x240
[    1.909055]  [<ffffffff8107f319>] __do_softirq+0xb9/0x190
[    1.909055]  [<ffffffff815d8d7c>] call_softirq+0x1c/0x30
[    1.909055]  <EOI> 
[    1.909055]  [<ffffffff8103a435>] do_softirq+0x65/0xa0
[    1.909055]  [<ffffffff8107f834>] local_bh_enable_ip+0x94/0xa0
[    1.909055]  [<ffffffff815d11a4>] _raw_spin_unlock_bh+0x24/0x30
[    1.909055]  [<ffffffffa0036d44>] xennet_open+0x54/0xe0 [xen_netfront]
[    1.909055]  [<ffffffff81481dcf>] __dev_open+0xbf/0x120
[    1.909055]  [<ffffffff8148022c>] __dev_change_flags+0x9c/0x180
[    1.909055]  [<ffffffff81481cc3>] dev_change_flags+0x23/0x70
[    1.909055]  [<ffffffff81491062>] do_setlink+0x1c2/0xa10
[    1.909055]  [<ffffffff812b5d74>] ? nla_parse+0x34/0x110
[    1.909055]  [<ffffffff81494005>] rtnl_newlink+0x3a5/0x5c0
[    1.909055]  [<ffffffff812541c4>] ? selinux_capable+0x34/0x50
[    1.909055]  [<ffffffff81250223>] ? security_capable+0x13/0x20
[    1.909055]  [<ffffffff81491e07>] rtnetlink_rcv_msg+0x2c7/0x330
[    1.909055]  [<ffffffff810a18b7>] ? __might_sleep+0xe7/0x100
[    1.909055]  [<ffffffff81149a52>] ? kmem_cache_alloc_node+0x82/0x1d0
[    1.909055]  [<ffffffff8147a00c>] ? __skb_recv_datagram+0x11c/0x2f0
[    1.909055]  [<ffffffff81491b40>] ? rtnetlink_rcv+0x30/0x30
[    1.909055]  [<ffffffff814a9c89>] netlink_rcv_skb+0x99/0xc0
[    1.909055]  [<ffffffff81491b30>] rtnetlink_rcv+0x20/0x30
[    1.909055]  [<ffffffff814a9998>] netlink_unicast+0x1a8/0x220
[    1.909055]  [<ffffffff814aa535>] netlink_sendmsg+0x205/0x300
[    1.909055]  [<ffffffff8146ce19>] sock_sendmsg+0xb9/0xf0
[    1.909055]  [<ffffffff8146c51e>] ? copy_from_user+0x3e/0x50
[    1.909055]  [<ffffffff8146c576>] ? move_addr_to_kernel+0x46/0x80
[    1.909055]  [<ffffffff810a18b7>] ? __might_sleep+0xe7/0x100
[    1.909055]  [<ffffffff8146dd2d>] __sys_sendmsg+0x3dd/0x400
[    1.909055]  [<ffffffff8112c751>] ? handle_mm_fault+0x261/0x380
[    1.909055]  [<ffffffff815d4cd0>] ? do_page_fault+0x250/0x4f0
[    1.909055]  [<ffffffff8114a587>] ? kmem_cache_alloc+0x1a7/0x1f0
[    1.909055]  [<ffffffff811311a4>] ? do_brk+0x1b4/0x350
[    1.909055]  [<ffffffff8146df04>] sys_sendmsg+0x44/0x80
[    1.909055]  [<ffffffff815d7bf9>] system_call_fastpath+0x16/0x1b
[    1.909055] Code: 44 00 00 41 80 8c 24 a8 00 00 00 04 e9 2f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 41 8b 84 24 c8 00 00 00 49 03 84 24 d0 00 00 00 <80> 3c 25 10 00 00 00 00 48 8d 50 30 74 0f 48 83 3c 25 08 00 00 
[    1.909055] RIP  [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.909055]  RSP <ffff8800ffc03db8>
[    1.909055] CR2: 0000000000000010
[    1.947298] ---[ end trace 3f4ba742dffbe90d ]---
[    1.947824] Kernel panic - not syncing: Fatal exception in interrupt
#1 lore raw
Quoted 1 line · Konrad Rzeszutek Wilk
konrad@phenom:~/ssd/linux$ git log --oneline --merges 4b24ff71108164e047cf2c95990b77651163e315..linus/master

What is your linus/master right now? I get many more merges than you did.

git tells me that nothing changed in netfront (or netback) in this range, nor drivers/xen or arch/x86/xen

Quoted 2 lines
[    1.908592] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[    1.908643] IP: [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]

What is xennet_poll+0x980 in your symbols? There's a lot of inlining going on in this function so I can't make a sensible by-eye guess.

Nothing leaps out from DaveMs merge. I think you might have to keep bisecting.

Ian

#2 lore raw
In reply to Konrad Rzeszutek Wilk
Quoted 11 lines · Konrad Rzeszutek Wilk

So I hadn't done a git bisection yet. But if I choose git commit:
4b24ff71108164e047cf2c95990b77651163e315
Merge tag 'for-v3.6' of git://git.infradead.org/battery-2.6

Pull battery updates from Anton Vorontsov:

everything works nicely. Anything past that, so these merges:

konrad@phenom:~/ssd/linux$ git log --oneline --merges 4b24ff71108164e047cf2c95990b77651163e315..linus/master
2d53492 Merge tag 'irqdomain-for-linus' of git://git.secretlab.ca/git/linux-2.6

===> ac694db Merge branch 'akpm' (Andrew's patch-bomb)

Somewhere in there is the culprit. Hadn't done yet the full bisection (was just checking out in each merge to see when it stopped working)

Andrew CC-ing you here, the serial log is below.

Quoted 266 lines

a40a1d3 Merge tag 'vfio-for-v3.6' of git://github.com/awilliam/linux-vfio
3e9a970 Merge tag 'random_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/random
941c872 Merge tag 'rdma-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband
8762541 Merge branch 'v4l_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
6dbb35b Merge tag 'nfs-for-3.6-2' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
fd37ce3 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
1da9b6b Merge branches 'cma', 'ipoib', 'ocrdma' and 'qib' into for-next
6aeea3e Merge remote-tracking branch 'origin' into irqdomain/next
931efdf Merge branch 'v4l_for_linus' into staging/for_v3.6
80c1834 Merge tag 'v3.5-rc6' into irqdomain/next

are the culprit. I think it might be the networking pull but not sure. Ian any thoughts?

Using config file "/test.xm".
Started domain latest (id=2)

[    0.000000] console [hvc0] enabled, bootconsole disabled
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 2294.530 MHz processor
[    0.000999] Calibrating delay loop (skipped), value calculated using timer frequency.. 4589.06 BogoMIPS (lpj=2294530)
[    0.000999] pid_max: default: 32768 minimum: 301
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] SELinux:  Starting in permissive mode
[    0.000999] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.001520] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001875] Mount-cache hash table entries: 256
[    0.002007] Initializing cgroup subsys cpuacct
[    0.002013] Initializing cgroup subsys freezer
[    0.002070] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.002070] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.002084] CPU: Physical Processor ID: 0
[    0.002087] CPU: Processor Core ID: 0
[    0.002094] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.002094] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[    0.002094] tlb_flushall_shift is 0x5
[    0.002164] SMP alternatives: switching to UP code
[    0.025291] Freeing SMP alternatives: 24k freed
[    0.025356] cpu 0 spinlock event irq 17
[    0.025383] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.
[    0.025551] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.025576] Brought up 1 CPUs
[    0.028642] kworker/u:0 (14) used greatest stack depth: 5936 bytes left
[    0.028675] Grant tables using version 2 layout.
[    0.028691] Grant table initialized
[    0.047616] RTC time: 165:165:165, date: 165/165/65
[    0.047661] NET: Registered protocol family 16
[    0.048184] dca service started, version 1.12.1
[    0.048545] PCI: setting up Xen PCI frontend stub
[    0.048552] PCI: pci_cache_line_size set to 64 bytes
[    0.049543] kworker/u:0 (51) used greatest stack depth: 5472 bytes left
[    0.054147] bio: create slab <bio-0> at 0
[    0.054240] ACPI: Interpreter disabled.
[    0.054288] xen/balloon: Initialising balloon driver.
[    0.055127] xen-balloon: Initialising balloon driver.
[    0.055127] vgaarb: loaded
[    0.056125] usbcore: registered new interface driver usbfs
[    0.056162] usbcore: registered new interface driver hub
[    0.056217] usbcore: registered new device driver usb
[    0.056425] PCI: System does not support PCI
[    0.056431] PCI: System does not support PCI
[    0.056617] NetLabel: Initializing
[    0.056624] NetLabel:  domain hash size = 128
[    0.056627] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.056642] NetLabel:  unlabeled traffic allowed by default
[    0.056725] Switching to clocksource xen
[    0.056795] pnp: PnP ACPI: disabled
[    0.058698] NET: Registered protocol family 2
[    0.059805] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    0.061110] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.061243] TCP: Hash tables configured (established 524288 bind 65536)
[    0.061281] TCP: reno registered
[    0.061304] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[    0.061341] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[    0.061425] NET: Registered protocol family 1
[    0.061492] RPC: Registered named UNIX socket transport module.
[    0.061498] RPC: Registered udp transport module.
[    0.061504] RPC: Registered tcp transport module.
[    0.061510] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.061518] PCI: CLS 0 bytes, default 64
[    0.061643] Trying to unpack rootfs image as initramfs...
[    0.382189] Freeing initrd memory: 362080k freed
[    0.499615] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.499831] Machine check injector initialized
[    0.500181] microcode: CPU0 sig=0x206d2, pf=0x1, revision=0x8000020c
[    0.500229] microcode: Microcode Update Driver: v2.00 <[email protected]>, Peter Oruba
[    0.500544] audit: initializing netlink socket (disabled)
[    0.500566] type=2000 audit(1343845740.901:1): initialized
[    0.515227] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.515358] VFS: Disk quotas dquot_6.5.2
[    0.515386] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.515525] NFS: Registering the id_resolver key type
[    0.515544] Key type id_resolver registered
[    0.515551] Key type id_legacy registered
[    0.515599] NTFS driver 2.1.30 [Flags: R/W].
[    0.515706] msgmni has been set to 8021
[    0.515765] SELinux:  Registering netfilter hooks
[    0.516222] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.516232] io scheduler noop registered
[    0.516236] io scheduler deadline registered
[    0.516243] io scheduler cfq registered (default)
[    0.516337] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.516442] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    0.532923] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.533399] Non-volatile memory driver v1.3
[    0.533406] Linux agpgart interface v0.103
[    0.533588] [drm] Initialized drm 1.1.0 20060810
[    0.535196] brd: module loaded
[    0.535992] loop: module loaded
[    0.536344] libphy: Fixed MDIO Bus: probed
[    0.536351] tun: Universal TUN/TAP device driver, 1.6
[    0.536354] tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
[    0.536419] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.6.0-k
[    0.536428] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.536721] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.536729] ehci_hcd: block sizes: qh 104 qtd 96 itd 192 sitd 96
[    0.536770] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.536776] ohci_hcd: block sizes: ed 80 td 96
[    0.536826] uhci_hcd: USB Universal Host Controller Interface driver
[    0.536929] usbcore: registered new interface driver usblp
[    0.536977] usbcore: registered new interface driver libusual
[    0.537164] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    0.538013] i8042: No controller found
[    0.538103] mousedev: PS/2 mouse device common for all mice
[    0.598349] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.598404] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.598559] EFI Variables Facility v0.08 2004-May-17
[    0.598701] Netfilter messages via NETLINK v0.30.
[    0.598719] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    0.598790] ctnetlink v0.93: registering with nfnetlink.
[    0.598971] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.599007] TCP: cubic registered
[    0.599014] Initializing XFRM netlink socket
[    0.599043] NET: Registered protocol family 10
[    0.599238] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    0.599374] sit: IPv6 over IPv4 tunneling driver
[    0.599589] NET: Registered protocol family 17
[    0.599618] Key type dns_resolver registered
[    0.599808] PM: Hibernation image not present or could not be loaded.
[    0.599824] registered taskstats version 1
[    0.599848] XENBUS: Device with no driver: device/vkbd/0
[    0.599852] XENBUS: Device with no driver: device/vfb/0
[    0.599856] XENBUS: Device with no driver: device/vbd/51712
[    0.599860] XENBUS: Device with no driver: device/vif/0
[    0.599886]   Magic number: 1:252:3141
[    0.600271] Freeing unused kernel memory: 704k freed
[    0.600500] Write protecting the kernel read-only data: 8192k
[    0.602437] Freeing unused kernel memory: 132k freed
[    0.602589] Freeing unused kernel memory: 340k freed
init started: BusyBox v1.14.3 (2012-08-01 13:52:44 EDT)
[    0.607489] consoletype (1056) used greatest stack depth: 5288 bytes left
Mounting directories  [  OK  ]
mount: mount point /proc/bus/usb does not exist
[    0.781044] modprobe (1085) used greatest stack depth: 5048 bytes left
mount: mount point /sys/kernel/config does not exist
[    0.785748] core_filesystem (1057) used greatest stack depth: 4968 bytes left
[    0.793721] input: Xen Virtual Keyboard as /devices/virtual/input/input0
[    0.793892] input: Xen Virtual Pointer as /devices/virtual/input/input1
[    1.010121] Initialising Xen virtual ethernet driver.
[    1.124604] blkfront: xvda: flush diskcache: enabled
[    1.126118]  xvda: xvda1 xvda2 xvda3 xvda4
[    1.239316] udevd (1128): /proc/1128/oom_adj is deprecated, please use /proc/1128/oom_score_adj instead.

udevd-work[1130]: error opening ATTR{/sys/devices/system/cpu/cpu0/online} for writing: No such file or directory

[    1.395080] ip (1408) used greatest stack depth: 3912 bytes left

Waiting for devices [ OK ]
Waiting for init.pre_custom [ OK ]
Waiting for fb [ OK ]
Starting..[/dev/fb0]
/dev/fb0: len:0
/dev/fb0: bits/pixel32
(7f44ddbc2000): Writting .. [800:600]
Done!
FATAL: Module agpgart_intel not found.

[    1.652131] [drm] radeon kernel modesetting enabled.

WARNING: Error inserting video (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/acpi/video.ko): No such device
WARNING: Error inserting mxm_wmi (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/platform/x86/mxm-wmi.ko): No such device
WARNING: Error inserting drm_kms_helper (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
WARNING: Error inserting ttm (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/ttm/ttm.ko): No such device
FATAL: Error inserting nouveau (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/nouveau/nouveau.ko): No such device

[    1.660288] Console: switching to colour frame buffer device 100x37

WARNING: Error inserting drm_kms_helper (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
FATAL: Error inserting i915 (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/i915/i915.ko): No such device Starting..[/dev/fb0]
/dev/fb0: len:0
/dev/fb0: bits/pixel32
(7fb8669cc000): Writting .. [800:600]
Done!
VGA: 0000:
Waiting for network [ OK ]

Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  [    1.908592] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[    1.908643] IP: [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.908703] PGD ea1df067 PUD e8ada067 PMD 0 
[    1.908774] Oops: 0000 [#1] SMP 
[    1.908797] Modules linked in: fbcon tileblit font radeon bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd
[    1.908938] CPU 0 
[    1.908950] Pid: 2165, comm: ip Not tainted 3.5.0upstream-08854-g444fa66 #1  
[    1.908983] RIP: e030:[<ffffffffa0037750>]  [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.909029] RSP: e02b:ffff8800ffc03db8  EFLAGS: 00010282
[    1.909055] RAX: ffff8800ea010140 RBX: ffff8800f00e86c0 RCX: 000000000000009a
[    1.909055] RDX: 0000000000000040 RSI: 000000000000005a RDI: ffff8800fa7dee80
[    1.909055] RBP: ffff8800ffc03ee8 R08: ffff8800f00e86d8 R09: ffff8800ea010000
[    1.909055] R10: dead000000200200 R11: dead000000100100 R12: ffff8800fa7dee80
[    1.909055] R13: 000000000000005a R14: ffff8800fa7dee80 R15: 0000000000000200
[    1.909055] FS:  00007fbafc188700(0000) GS:ffff8800ffc00000(0000) knlGS:0000000000000000
[    1.909055] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    1.909055] CR2: 0000000000000010 CR3: 00000000ea108000 CR4: 0000000000002660
[    1.909055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    1.909055] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    1.909055] Process ip (pid: 2165, threadinfo ffff8800ea0f2000, task ffff8800fa783040)
[    1.909055] Stack:
[    1.909055]  ffff8800e27e5040 ffff8800ffc03e88 ffff8800ffc03e68 ffff8800ffc03e48
[    1.909055]  7fffffffffffffff ffff8800ffc03e00 ffff8800e27e5040 ffff8800f00e86d8
[    1.909055]  ffff8800ffc03eb0 00000040ffffffff ffff8800f00e8000 00000000ffc03e30
[    1.909055] Call Trace:
[    1.909055]  <IRQ> 
[    1.909055]  [<ffffffff81066028>] ? pvclock_clocksource_read+0x58/0xd0
[    1.909055]  [<ffffffff81486352>] net_rx_action+0x112/0x240
[    1.909055]  [<ffffffff8107f319>] __do_softirq+0xb9/0x190
[    1.909055]  [<ffffffff815d8d7c>] call_softirq+0x1c/0x30
[    1.909055]  <EOI> 
[    1.909055]  [<ffffffff8103a435>] do_softirq+0x65/0xa0
[    1.909055]  [<ffffffff8107f834>] local_bh_enable_ip+0x94/0xa0
[    1.909055]  [<ffffffff815d11a4>] _raw_spin_unlock_bh+0x24/0x30
[    1.909055]  [<ffffffffa0036d44>] xennet_open+0x54/0xe0 [xen_netfront]
[    1.909055]  [<ffffffff81481dcf>] __dev_open+0xbf/0x120
[    1.909055]  [<ffffffff8148022c>] __dev_change_flags+0x9c/0x180
[    1.909055]  [<ffffffff81481cc3>] dev_change_flags+0x23/0x70
[    1.909055]  [<ffffffff81491062>] do_setlink+0x1c2/0xa10
[    1.909055]  [<ffffffff812b5d74>] ? nla_parse+0x34/0x110
[    1.909055]  [<ffffffff81494005>] rtnl_newlink+0x3a5/0x5c0
[    1.909055]  [<ffffffff812541c4>] ? selinux_capable+0x34/0x50
[    1.909055]  [<ffffffff81250223>] ? security_capable+0x13/0x20
[    1.909055]  [<ffffffff81491e07>] rtnetlink_rcv_msg+0x2c7/0x330
[    1.909055]  [<ffffffff810a18b7>] ? __might_sleep+0xe7/0x100
[    1.909055]  [<ffffffff81149a52>] ? kmem_cache_alloc_node+0x82/0x1d0
[    1.909055]  [<ffffffff8147a00c>] ? __skb_recv_datagram+0x11c/0x2f0
[    1.909055]  [<ffffffff81491b40>] ? rtnetlink_rcv+0x30/0x30
[    1.909055]  [<ffffffff814a9c89>] netlink_rcv_skb+0x99/0xc0
[    1.909055]  [<ffffffff81491b30>] rtnetlink_rcv+0x20/0x30
[    1.909055]  [<ffffffff814a9998>] netlink_unicast+0x1a8/0x220
[    1.909055]  [<ffffffff814aa535>] netlink_sendmsg+0x205/0x300
[    1.909055]  [<ffffffff8146ce19>] sock_sendmsg+0xb9/0xf0
[    1.909055]  [<ffffffff8146c51e>] ? copy_from_user+0x3e/0x50
[    1.909055]  [<ffffffff8146c576>] ? move_addr_to_kernel+0x46/0x80
[    1.909055]  [<ffffffff810a18b7>] ? __might_sleep+0xe7/0x100
[    1.909055]  [<ffffffff8146dd2d>] __sys_sendmsg+0x3dd/0x400
[    1.909055]  [<ffffffff8112c751>] ? handle_mm_fault+0x261/0x380
[    1.909055]  [<ffffffff815d4cd0>] ? do_page_fault+0x250/0x4f0
[    1.909055]  [<ffffffff8114a587>] ? kmem_cache_alloc+0x1a7/0x1f0
[    1.909055]  [<ffffffff811311a4>] ? do_brk+0x1b4/0x350
[    1.909055]  [<ffffffff8146df04>] sys_sendmsg+0x44/0x80
[    1.909055]  [<ffffffff815d7bf9>] system_call_fastpath+0x16/0x1b
[    1.909055] Code: 44 00 00 41 80 8c 24 a8 00 00 00 04 e9 2f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 41 8b 84 24 c8 00 00 00 49 03 84 24 d0 00 00 00 <80> 3c 25 10 00 00 00 00 48 8d 50 30 74 0f 48 83 3c 25 08 00 00 
[    1.909055] RIP  [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.909055]  RSP <ffff8800ffc03db8>
[    1.909055] CR2: 0000000000000010
[    1.947298] ---[ end trace 3f4ba742dffbe90d ]---
[    1.947824] Kernel panic - not syncing: Fatal exception in interrupt

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#3 lore raw

Re: [Xen-devel] Regression in xen-netfront on v3.6 (git commit c48a11c7ad2623b99bbd6859b0b4234e7f11176f, netvm: propagate page->pfmemalloc to skb)

Quoted 8 lines · Konrad Rzeszutek Wilk
Quoted 7 lines · Konrad Rzeszutek Wilk

So I hadn't done a git bisection yet. But if I choose git commit:
4b24ff71108164e047cf2c95990b77651163e315
Merge tag 'for-v3.6' of git://git.infradead.org/battery-2.6

Pull battery updates from Anton Vorontsov:

Quoted 8 lines
Quoted 4 lines

everything works nicely. Anything past that, so these merges:

konrad@phenom:~/ssd/linux$ git log --oneline --merges 4b24ff71108164e047cf2c95990b77651163e315..linus/master
2d53492 Merge tag 'irqdomain-for-linus' of git://git.secretlab.ca/git/linux-2.6

===> ac694db Merge branch 'akpm' (Andrew's patch-bomb)

Somewhere in there is the culprit. Hadn't done yet the full bisection (was just checking out in each merge to see when it stopped working)

Mel, your:
commit c48a11c7ad2623b99bbd6859b0b4234e7f11176f

Author: Mel Gorman <[email protected]>
Date:   Tue Jul 31 16:44:23 2012 -0700
    netvm: propagate page->pfmemalloc to skb

is the culprit per git bisect. Any ideas - do the drivers need to do some extra processing? Here is the git bisect log

git bisect start
# good: [a40a1d3d0a2fd613fdec6d89d3c053268ced76ed] Merge tag # 'vfio-for-v3.6' of git://github.com/awilliam/linux-vfio
git bisect good a40a1d3d0a2fd613fdec6d89d3c053268ced76ed
# bad: [ac694dbdbc403c00e2c14d10bc7b8412cc378259] Merge branch 'akpm' # (Andrew's patch-bomb)
git bisect bad ac694dbdbc403c00e2c14d10bc7b8412cc378259
# good: [62ce1c706f817cb9defef3ac2dfdd815149f2968] mm, oom: move

# declaration for mem_cgroup_out_of_memory to oom.h
git bisect good 62ce1c706f817cb9defef3ac2dfdd815149f2968
# bad: [5a178119b0fbe37f7dfb602b37df9cc4b1dc9d71] mm: add support for
# direct_IO to highmem pages
git bisect bad 5a178119b0fbe37f7dfb602b37df9cc4b1dc9d71
# good: [7cb0240492caea2f6467f827313478f41877e6ef] netvm: allow the use
# of __GFP_MEMALLOC by specific sockets
git bisect good 7cb0240492caea2f6467f827313478f41877e6ef
# bad: [5515061d22f0f9976ae7815864bfd22042d36848] mm: throttle direct
# reclaimers if PF_MEMALLOC reserves are low and swap is backed by
# network storage
git bisect bad 5515061d22f0f9976ae7815864bfd22042d36848
# bad: [0614002bb5f7411e61ffa0dfe5be1f2c84df3da3] netvm: propagate
# page->pfmemalloc from skb_alloc_page to skb
git bisect bad 0614002bb5f7411e61ffa0dfe5be1f2c84df3da3
# bad: [c48a11c7ad2623b99bbd6859b0b4234e7f11176f] netvm: propagate
# page->pfmemalloc to skb
git bisect bad c48a11c7ad2623b99bbd6859b0b4234e7f11176f
# good: [c93bdd0e03e848555d144eb44a1f275b871a8dd5] netvm: allow skb
# allocation to use PFMEMALLOC reserves
git bisect good c93bdd0e03e848555d144eb44a1f275b871a8dd5
Quoted 273 lines

Andrew CC-ing you here, the serial log is below.

Quoted 262 lines

a40a1d3 Merge tag 'vfio-for-v3.6' of git://github.com/awilliam/linux-vfio
3e9a970 Merge tag 'random_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/random
941c872 Merge tag 'rdma-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/roland/infiniband
8762541 Merge branch 'v4l_for_linus' of git://git.kernel.org/pub/scm/linux/kernel/git/mchehab/linux-media
6dbb35b Merge tag 'nfs-for-3.6-2' of git://git.linux-nfs.org/projects/trondmy/linux-nfs
fd37ce3 Merge git://git.kernel.org/pub/scm/linux/kernel/git/davem/net
1da9b6b Merge branches 'cma', 'ipoib', 'ocrdma' and 'qib' into for-next
6aeea3e Merge remote-tracking branch 'origin' into irqdomain/next
931efdf Merge branch 'v4l_for_linus' into staging/for_v3.6
80c1834 Merge tag 'v3.5-rc6' into irqdomain/next

are the culprit. I think it might be the networking pull but not sure. Ian any thoughts?

Using config file "/test.xm".
Started domain latest (id=2)

[    0.000000] console [hvc0] enabled, bootconsole disabled
[    0.000000] Xen: using vcpuop timer interface
[    0.000000] installing Xen timer for CPU 0
[    0.000000] tsc: Detected 2294.530 MHz processor
[    0.000999] Calibrating delay loop (skipped), value calculated using timer frequency.. 4589.06 BogoMIPS (lpj=2294530)
[    0.000999] pid_max: default: 32768 minimum: 301
[    0.000999] Security Framework initialized
[    0.000999] SELinux:  Initializing.
[    0.000999] SELinux:  Starting in permissive mode
[    0.000999] Dentry cache hash table entries: 524288 (order: 10, 4194304 bytes)
[    0.001520] Inode-cache hash table entries: 262144 (order: 9, 2097152 bytes)
[    0.001875] Mount-cache hash table entries: 256
[    0.002007] Initializing cgroup subsys cpuacct
[    0.002013] Initializing cgroup subsys freezer
[    0.002070] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[    0.002070] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[    0.002084] CPU: Physical Processor ID: 0
[    0.002087] CPU: Processor Core ID: 0
[    0.002094] Last level iTLB entries: 4KB 512, 2MB 0, 4MB 0
[    0.002094] Last level dTLB entries: 4KB 512, 2MB 32, 4MB 32
[    0.002094] tlb_flushall_shift is 0x5
[    0.002164] SMP alternatives: switching to UP code
[    0.025291] Freeing SMP alternatives: 24k freed
[    0.025356] cpu 0 spinlock event irq 17
[    0.025383] Performance Events: unsupported p6 CPU model 45 no PMU driver, software events only.
[    0.025551] NMI watchdog: disabled (cpu0): hardware events not enabled
[    0.025576] Brought up 1 CPUs
[    0.028642] kworker/u:0 (14) used greatest stack depth: 5936 bytes left
[    0.028675] Grant tables using version 2 layout.
[    0.028691] Grant table initialized
[    0.047616] RTC time: 165:165:165, date: 165/165/65
[    0.047661] NET: Registered protocol family 16
[    0.048184] dca service started, version 1.12.1
[    0.048545] PCI: setting up Xen PCI frontend stub
[    0.048552] PCI: pci_cache_line_size set to 64 bytes
[    0.049543] kworker/u:0 (51) used greatest stack depth: 5472 bytes left
[    0.054147] bio: create slab <bio-0> at 0
[    0.054240] ACPI: Interpreter disabled.
[    0.054288] xen/balloon: Initialising balloon driver.
[    0.055127] xen-balloon: Initialising balloon driver.
[    0.055127] vgaarb: loaded
[    0.056125] usbcore: registered new interface driver usbfs
[    0.056162] usbcore: registered new interface driver hub
[    0.056217] usbcore: registered new device driver usb
[    0.056425] PCI: System does not support PCI
[    0.056431] PCI: System does not support PCI
[    0.056617] NetLabel: Initializing
[    0.056624] NetLabel:  domain hash size = 128
[    0.056627] NetLabel:  protocols = UNLABELED CIPSOv4
[    0.056642] NetLabel:  unlabeled traffic allowed by default
[    0.056725] Switching to clocksource xen
[    0.056795] pnp: PnP ACPI: disabled
[    0.058698] NET: Registered protocol family 2
[    0.059805] TCP established hash table entries: 524288 (order: 11, 8388608 bytes)
[    0.061110] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[    0.061243] TCP: Hash tables configured (established 524288 bind 65536)
[    0.061281] TCP: reno registered
[    0.061304] UDP hash table entries: 2048 (order: 4, 65536 bytes)
[    0.061341] UDP-Lite hash table entries: 2048 (order: 4, 65536 bytes)
[    0.061425] NET: Registered protocol family 1
[    0.061492] RPC: Registered named UNIX socket transport module.
[    0.061498] RPC: Registered udp transport module.
[    0.061504] RPC: Registered tcp transport module.
[    0.061510] RPC: Registered tcp NFSv4.1 backchannel transport module.
[    0.061518] PCI: CLS 0 bytes, default 64
[    0.061643] Trying to unpack rootfs image as initramfs...
[    0.382189] Freeing initrd memory: 362080k freed
[    0.499615] platform rtc_cmos: registered platform RTC device (no PNP device found)
[    0.499831] Machine check injector initialized
[    0.500181] microcode: CPU0 sig=0x206d2, pf=0x1, revision=0x8000020c
[    0.500229] microcode: Microcode Update Driver: v2.00 <[email protected]>, Peter Oruba
[    0.500544] audit: initializing netlink socket (disabled)
[    0.500566] type=2000 audit(1343845740.901:1): initialized
[    0.515227] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[    0.515358] VFS: Disk quotas dquot_6.5.2
[    0.515386] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[    0.515525] NFS: Registering the id_resolver key type
[    0.515544] Key type id_resolver registered
[    0.515551] Key type id_legacy registered
[    0.515599] NTFS driver 2.1.30 [Flags: R/W].
[    0.515706] msgmni has been set to 8021
[    0.515765] SELinux:  Registering netfilter hooks
[    0.516222] Block layer SCSI generic (bsg) driver version 0.4 loaded (major 253)
[    0.516232] io scheduler noop registered
[    0.516236] io scheduler deadline registered
[    0.516243] io scheduler cfq registered (default)
[    0.516337] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[    0.516442] ioatdma: Intel(R) QuickData Technology Driver 4.00
[    0.532923] Serial: 8250/16550 driver, 4 ports, IRQ sharing enabled
[    0.533399] Non-volatile memory driver v1.3
[    0.533406] Linux agpgart interface v0.103
[    0.533588] [drm] Initialized drm 1.1.0 20060810
[    0.535196] brd: module loaded
[    0.535992] loop: module loaded
[    0.536344] libphy: Fixed MDIO Bus: probed
[    0.536351] tun: Universal TUN/TAP device driver, 1.6
[    0.536354] tun: (C) 1999-2004 Max Krasnyansky <[email protected]>
[    0.536419] ixgbevf: Intel(R) 10 Gigabit PCI Express Virtual Function Network Driver - version 2.6.0-k
[    0.536428] ixgbevf: Copyright (c) 2009 - 2012 Intel Corporation.
[    0.536721] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[    0.536729] ehci_hcd: block sizes: qh 104 qtd 96 itd 192 sitd 96
[    0.536770] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[    0.536776] ohci_hcd: block sizes: ed 80 td 96
[    0.536826] uhci_hcd: USB Universal Host Controller Interface driver
[    0.536929] usbcore: registered new interface driver usblp
[    0.536977] usbcore: registered new interface driver libusual
[    0.537164] i8042: PNP: No PS/2 controller found. Probing ports directly.
[    0.538013] i8042: No controller found
[    0.538103] mousedev: PS/2 mouse device common for all mice
[    0.598349] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[    0.598404] rtc_cmos: probe of rtc_cmos failed with error -38
[    0.598559] EFI Variables Facility v0.08 2004-May-17
[    0.598701] Netfilter messages via NETLINK v0.30.
[    0.598719] nf_conntrack version 0.5.0 (16384 buckets, 65536 max)
[    0.598790] ctnetlink v0.93: registering with nfnetlink.
[    0.598971] ip_tables: (C) 2000-2006 Netfilter Core Team
[    0.599007] TCP: cubic registered
[    0.599014] Initializing XFRM netlink socket
[    0.599043] NET: Registered protocol family 10
[    0.599238] ip6_tables: (C) 2000-2006 Netfilter Core Team
[    0.599374] sit: IPv6 over IPv4 tunneling driver
[    0.599589] NET: Registered protocol family 17
[    0.599618] Key type dns_resolver registered
[    0.599808] PM: Hibernation image not present or could not be loaded.
[    0.599824] registered taskstats version 1
[    0.599848] XENBUS: Device with no driver: device/vkbd/0
[    0.599852] XENBUS: Device with no driver: device/vfb/0
[    0.599856] XENBUS: Device with no driver: device/vbd/51712
[    0.599860] XENBUS: Device with no driver: device/vif/0
[    0.599886]   Magic number: 1:252:3141
[    0.600271] Freeing unused kernel memory: 704k freed
[    0.600500] Write protecting the kernel read-only data: 8192k
[    0.602437] Freeing unused kernel memory: 132k freed
[    0.602589] Freeing unused kernel memory: 340k freed
init started: BusyBox v1.14.3 (2012-08-01 13:52:44 EDT)
[    0.607489] consoletype (1056) used greatest stack depth: 5288 bytes left
Mounting directories  [  OK  ]
mount: mount point /proc/bus/usb does not exist
[    0.781044] modprobe (1085) used greatest stack depth: 5048 bytes left
mount: mount point /sys/kernel/config does not exist
[    0.785748] core_filesystem (1057) used greatest stack depth: 4968 bytes left
[    0.793721] input: Xen Virtual Keyboard as /devices/virtual/input/input0
[    0.793892] input: Xen Virtual Pointer as /devices/virtual/input/input1
[    1.010121] Initialising Xen virtual ethernet driver.
[    1.124604] blkfront: xvda: flush diskcache: enabled
[    1.126118]  xvda: xvda1 xvda2 xvda3 xvda4
[    1.239316] udevd (1128): /proc/1128/oom_adj is deprecated, please use /proc/1128/oom_score_adj instead.

udevd-work[1130]: error opening ATTR{/sys/devices/system/cpu/cpu0/online} for writing: No such file or directory

[    1.395080] ip (1408) used greatest stack depth: 3912 bytes left

Waiting for devices [ OK ]
Waiting for init.pre_custom [ OK ]
Waiting for fb [ OK ]
Starting..[/dev/fb0]
/dev/fb0: len:0
/dev/fb0: bits/pixel32
(7f44ddbc2000): Writting .. [800:600]
Done!
FATAL: Module agpgart_intel not found.

[    1.652131] [drm] radeon kernel modesetting enabled.

WARNING: Error inserting video (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/acpi/video.ko): No such device
WARNING: Error inserting mxm_wmi (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/platform/x86/mxm-wmi.ko): No such device
WARNING: Error inserting drm_kms_helper (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
WARNING: Error inserting ttm (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/ttm/ttm.ko): No such device
FATAL: Error inserting nouveau (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/nouveau/nouveau.ko): No such device

[    1.660288] Console: switching to colour frame buffer device 100x37

WARNING: Error inserting drm_kms_helper (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/drm_kms_helper.ko): No such device
FATAL: Error inserting i915 (/lib/modules/3.5.0upstream-08854-g444fa66/kernel/drivers/gpu/drm/i915/i915.ko): No such device Starting..[/dev/fb0]
/dev/fb0: len:0
/dev/fb0: bits/pixel32
(7fb8669cc000): Writting .. [800:600]
Done!
VGA: 0000:
Waiting for network [ OK ]

Bringing up loopback interface:  [  OK  ]
Bringing up interface eth0:  [    1.908592] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[    1.908643] IP: [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.908703] PGD ea1df067 PUD e8ada067 PMD 0 
[    1.908774] Oops: 0000 [#1] SMP 
[    1.908797] Modules linked in: fbcon tileblit font radeon bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea xen_kbdfront xenfs xen_privcmd
[    1.908938] CPU 0 
[    1.908950] Pid: 2165, comm: ip Not tainted 3.5.0upstream-08854-g444fa66 #1  
[    1.908983] RIP: e030:[<ffffffffa0037750>]  [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.909029] RSP: e02b:ffff8800ffc03db8  EFLAGS: 00010282
[    1.909055] RAX: ffff8800ea010140 RBX: ffff8800f00e86c0 RCX: 000000000000009a
[    1.909055] RDX: 0000000000000040 RSI: 000000000000005a RDI: ffff8800fa7dee80
[    1.909055] RBP: ffff8800ffc03ee8 R08: ffff8800f00e86d8 R09: ffff8800ea010000
[    1.909055] R10: dead000000200200 R11: dead000000100100 R12: ffff8800fa7dee80
[    1.909055] R13: 000000000000005a R14: ffff8800fa7dee80 R15: 0000000000000200
[    1.909055] FS:  00007fbafc188700(0000) GS:ffff8800ffc00000(0000) knlGS:0000000000000000
[    1.909055] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    1.909055] CR2: 0000000000000010 CR3: 00000000ea108000 CR4: 0000000000002660
[    1.909055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    1.909055] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    1.909055] Process ip (pid: 2165, threadinfo ffff8800ea0f2000, task ffff8800fa783040)
[    1.909055] Stack:
[    1.909055]  ffff8800e27e5040 ffff8800ffc03e88 ffff8800ffc03e68 ffff8800ffc03e48
[    1.909055]  7fffffffffffffff ffff8800ffc03e00 ffff8800e27e5040 ffff8800f00e86d8
[    1.909055]  ffff8800ffc03eb0 00000040ffffffff ffff8800f00e8000 00000000ffc03e30
[    1.909055] Call Trace:
[    1.909055]  <IRQ> 
[    1.909055]  [<ffffffff81066028>] ? pvclock_clocksource_read+0x58/0xd0
[    1.909055]  [<ffffffff81486352>] net_rx_action+0x112/0x240
[    1.909055]  [<ffffffff8107f319>] __do_softirq+0xb9/0x190
[    1.909055]  [<ffffffff815d8d7c>] call_softirq+0x1c/0x30
[    1.909055]  <EOI> 
[    1.909055]  [<ffffffff8103a435>] do_softirq+0x65/0xa0
[    1.909055]  [<ffffffff8107f834>] local_bh_enable_ip+0x94/0xa0
[    1.909055]  [<ffffffff815d11a4>] _raw_spin_unlock_bh+0x24/0x30
[    1.909055]  [<ffffffffa0036d44>] xennet_open+0x54/0xe0 [xen_netfront]
[    1.909055]  [<ffffffff81481dcf>] __dev_open+0xbf/0x120
[    1.909055]  [<ffffffff8148022c>] __dev_change_flags+0x9c/0x180
[    1.909055]  [<ffffffff81481cc3>] dev_change_flags+0x23/0x70
[    1.909055]  [<ffffffff81491062>] do_setlink+0x1c2/0xa10
[    1.909055]  [<ffffffff812b5d74>] ? nla_parse+0x34/0x110
[    1.909055]  [<ffffffff81494005>] rtnl_newlink+0x3a5/0x5c0
[    1.909055]  [<ffffffff812541c4>] ? selinux_capable+0x34/0x50
[    1.909055]  [<ffffffff81250223>] ? security_capable+0x13/0x20
[    1.909055]  [<ffffffff81491e07>] rtnetlink_rcv_msg+0x2c7/0x330
[    1.909055]  [<ffffffff810a18b7>] ? __might_sleep+0xe7/0x100
[    1.909055]  [<ffffffff81149a52>] ? kmem_cache_alloc_node+0x82/0x1d0
[    1.909055]  [<ffffffff8147a00c>] ? __skb_recv_datagram+0x11c/0x2f0
[    1.909055]  [<ffffffff81491b40>] ? rtnetlink_rcv+0x30/0x30
[    1.909055]  [<ffffffff814a9c89>] netlink_rcv_skb+0x99/0xc0
[    1.909055]  [<ffffffff81491b30>] rtnetlink_rcv+0x20/0x30
[    1.909055]  [<ffffffff814a9998>] netlink_unicast+0x1a8/0x220
[    1.909055]  [<ffffffff814aa535>] netlink_sendmsg+0x205/0x300
[    1.909055]  [<ffffffff8146ce19>] sock_sendmsg+0xb9/0xf0
[    1.909055]  [<ffffffff8146c51e>] ? copy_from_user+0x3e/0x50
[    1.909055]  [<ffffffff8146c576>] ? move_addr_to_kernel+0x46/0x80
[    1.909055]  [<ffffffff810a18b7>] ? __might_sleep+0xe7/0x100
[    1.909055]  [<ffffffff8146dd2d>] __sys_sendmsg+0x3dd/0x400
[    1.909055]  [<ffffffff8112c751>] ? handle_mm_fault+0x261/0x380
[    1.909055]  [<ffffffff815d4cd0>] ? do_page_fault+0x250/0x4f0
[    1.909055]  [<ffffffff8114a587>] ? kmem_cache_alloc+0x1a7/0x1f0
[    1.909055]  [<ffffffff811311a4>] ? do_brk+0x1b4/0x350
[    1.909055]  [<ffffffff8146df04>] sys_sendmsg+0x44/0x80
[    1.909055]  [<ffffffff815d7bf9>] system_call_fastpath+0x16/0x1b
[    1.909055] Code: 44 00 00 41 80 8c 24 a8 00 00 00 04 e9 2f ff ff ff 66 2e 0f 1f 84 00 00 00 00 00 41 8b 84 24 c8 00 00 00 49 03 84 24 d0 00 00 00 <80> 3c 25 10 00 00 00 00 48 8d 50 30 74 0f 48 83 3c 25 08 00 00 
[    1.909055] RIP  [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.909055]  RSP <ffff8800ffc03db8>
[    1.909055] CR2: 0000000000000010
[    1.947298] ---[ end trace 3f4ba742dffbe90d ]---
[    1.947824] Kernel panic - not syncing: Fatal exception in interrupt
Quoted 4 lines

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

_______________________________________________
Xen-devel mailing list
[email protected]
http://lists.xen.org/xen-devel

#4 lore raw

Re: [Xen-devel] Regression in xen-netfront on v3.6 (git commit c48a11c7ad2623b99bbd6859b0b4234e7f11176f, netvm: propagate page->pfmemalloc to skb)

Quoted 20 lines · Konrad Rzeszutek Wilk
Quoted 8 lines · Konrad Rzeszutek Wilk
Quoted 7 lines · Konrad Rzeszutek Wilk

So I hadn't done a git bisection yet. But if I choose git commit:
4b24ff71108164e047cf2c95990b77651163e315
Merge tag 'for-v3.6' of git://git.infradead.org/battery-2.6

Pull battery updates from Anton Vorontsov:

Quoted 8 lines
Quoted 4 lines

everything works nicely. Anything past that, so these merges:

konrad@phenom:~/ssd/linux$ git log --oneline --merges 4b24ff71108164e047cf2c95990b77651163e315..linus/master
2d53492 Merge tag 'irqdomain-for-linus' of git://git.secretlab.ca/git/linux-2.6

===> ac694db Merge branch 'akpm' (Andrew's patch-bomb)

Somewhere in there is the culprit. Hadn't done yet the full bisection (was just checking out in each merge to see when it stopped working)

Mel, your:
commit c48a11c7ad2623b99bbd6859b0b4234e7f11176f

Quoted 8 lines
Author: Mel Gorman <[email protected]>
Date:   Tue Jul 31 16:44:23 2012 -0700
    netvm: propagate page->pfmemalloc to skb

is the culprit per git bisect. Any ideas - do the drivers need to do some extra processing? Here is the git bisect log

The problem appears to be at drivers/net/xen-netfront.c#973 where it calls __skb_fill_page_desc(skb, 0, NULL, 0, 0) . The driver does not have to do extra processing as such but I did not expect NULL to be passed in like this. Can you check if this fixes the bug please?

---8<---
netvm: check for page == NULL when propogating the skb->pfmemalloc flag

Commit [c48a11c7: netvm: propagate page->pfmemalloc to skb] is responsible for the following bug triggered by a xen network driver

[    1.908592] BUG: unable to handle kernel NULL pointer dereference at 0000000000000010
[    1.908643] IP: [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.908703] PGD ea1df067 PUD e8ada067 PMD 0
[    1.908774] Oops: 0000 [#1] SMP
[    1.908797] Modules linked in: fbcon tileblit font radeon bitblit softcursor ttm drm_kms_helper crc32c_intel xen_blkfront xen_netfront xen_fbfront fb_sys_fops sysimgblt sysfillrect syscopyarea +xen_kbdfront xenfs xen_privcmd
[    1.908938] CPU 0
[    1.908950] Pid: 2165, comm: ip Not tainted 3.5.0upstream-08854-g444fa66 #1
[    1.908983] RIP: e030:[<ffffffffa0037750>]  [<ffffffffa0037750>] xennet_poll+0x980/0xec0 [xen_netfront]
[    1.909029] RSP: e02b:ffff8800ffc03db8  EFLAGS: 00010282
[    1.909055] RAX: ffff8800ea010140 RBX: ffff8800f00e86c0 RCX: 000000000000009a
[    1.909055] RDX: 0000000000000040 RSI: 000000000000005a RDI: ffff8800fa7dee80
[    1.909055] RBP: ffff8800ffc03ee8 R08: ffff8800f00e86d8 R09: ffff8800ea010000
[    1.909055] R10: dead000000200200 R11: dead000000100100 R12: ffff8800fa7dee80
[    1.909055] R13: 000000000000005a R14: ffff8800fa7dee80 R15: 0000000000000200
[    1.909055] FS:  00007fbafc188700(0000) GS:ffff8800ffc00000(0000) knlGS:0000000000000000
[    1.909055] CS:  e033 DS: 0000 ES: 0000 CR0: 000000008005003b
[    1.909055] CR2: 0000000000000010 CR3: 00000000ea108000 CR4: 0000000000002660
[    1.909055] DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
[    1.909055] DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400
[    1.909055] Process ip (pid: 2165, threadinfo ffff8800ea0f2000, task ffff8800fa783040)
[    1.909055] Stack:
[    1.909055]  ffff8800e27e5040 ffff8800ffc03e88 ffff8800ffc03e68 ffff8800ffc03e48
[    1.909055]  7fffffffffffffff ffff8800ffc03e00 ffff8800e27e5040 ffff8800f00e86d8
[    1.909055]  ffff8800ffc03eb0 00000040ffffffff ffff8800f00e8000 00000000ffc03e30
[    1.909055] Call Trace:
[    1.909055]  <IRQ>
[    1.909055]  [<ffffffff81066028>] ?  pvclock_clocksource_read+0x58/0xd0
[    1.909055]  [<ffffffff81486352>] net_rx_action+0x112/0x240
[    1.909055]  [<ffffffff8107f319>] __do_softirq+0xb9/0x190
[    1.909055]  [<ffffffff815d8d7c>] call_softirq+0x1c/0x30

The problem is that the xenfront driver is passing a NULL page to __skb_fill_page_desc() which was unexpected. This patch checks that there is a page before dereferencing.

---
 include/linux/skbuff.h |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/skbuff.h +1 -1
index 7632c87..8857669 100644
@@ -1256,7 +1256,7 @@ static inline void __skb_fill_page_desc(struct sk_buff *skb, int i,
 	 * do not lose pfmemalloc information as the pages would not be
 	 * allocated using __GFP_MEMALLOC.
 	 */
-	if (page->pfmemalloc && !page->mapping)
+	if (page && page->pfmemalloc && !page->mapping)
 		skb->pfmemalloc	= true;
 	frag->page.p		  = page;
 	frag->page_offset	  = off;
#5 lore raw

Re: [Xen-devel] Regression in xen-netfront on v3.6 (git commit c48a11c7ad2623b99bbd6859b0b4234e7f11176f, netvm: propagate page->pfmemalloc to skb)

Quoted 26 lines · Mel Gorman
Quoted 25 lines · Konrad Rzeszutek Wilk
Quoted 8 lines · Konrad Rzeszutek Wilk
Quoted 7 lines · Konrad Rzeszutek Wilk

So I hadn't done a git bisection yet. But if I choose git commit:
4b24ff71108164e047cf2c95990b77651163e315
Merge tag 'for-v3.6' of git://git.infradead.org/battery-2.6

Pull battery updates from Anton Vorontsov:

Quoted 8 lines
Quoted 4 lines

everything works nicely. Anything past that, so these merges:

konrad@phenom:~/ssd/linux$ git log --oneline --merges 4b24ff71108164e047cf2c95990b77651163e315..linus/master
2d53492 Merge tag 'irqdomain-for-linus' of git://git.secretlab.ca/git/linux-2.6

===> ac694db Merge branch 'akpm' (Andrew's patch-bomb)

Somewhere in there is the culprit. Hadn't done yet the full bisection (was just checking out in each merge to see when it stopped working)

Mel, your:
commit c48a11c7ad2623b99bbd6859b0b4234e7f11176f

Author: Mel Gorman <[email protected]>
Date:   Tue Jul 31 16:44:23 2012 -0700
    netvm: propagate page->pfmemalloc to skb
Quoted 8 lines
Quoted 3 lines

is the culprit per git bisect. Any ideas - do the drivers need to do some extra processing? Here is the git bisect log

The problem appears to be at drivers/net/xen-netfront.c#973 where it calls __skb_fill_page_desc(skb, 0, NULL, 0, 0) . The driver does not have to do extra processing as such but I did not expect NULL to be passed in like this. Can you check if this fixes the bug please?

That does it!
.. snip..

Quoted 2 lines
Quoted 17 lines
---
 include/linux/skbuff.h |    2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
include/linux/skbuff.h +1 -1
index 7632c87..8857669 100644
@@ -1256,7 +1256,7 @@ static inline void __skb_fill_page_desc(struct sk_buff *skb, int i,
 	 * do not lose pfmemalloc information as the pages would not be
 	 * allocated using __GFP_MEMALLOC.
 	 */
-	if (page->pfmemalloc && !page->mapping)
+	if (page && page->pfmemalloc && !page->mapping)
 		skb->pfmemalloc	= true;
 	frag->page.p		  = page;
 	frag->page_offset	  = off;