problems with 2.6.12 and ioremap/iounmap
Hello,
I'm looking into a customer issue where they hit a kernel BUG when starting X with our driver (attached logfile). This only occurs with 2.6.12 kernels.
this appears to be the 'vmalloc guard page causing change_page_attr problems' bug. at one point, iounmap subtracted the guard page before calling change_page_attr, but I see this was reverted as part of a recent cleanup.
in this case, we're remapping a single page of the extended pci config space. note that in the log RAX is the the physical address 00000000e00001e3, but the stack indicates that __change_page_attr was called with address ffff8100e0001000 and pfn 00000000000e0001.
from looking at the implementation in 2.6.12-pre4, I'm not clear how the guard page is accounted for in iounmap. in vmalloc.c, the guard page is subtracted from the vm_struct in remove_vm_area (which calls unmap_vm_area). but iounmap in ioremap.c calls unmap_vm_area directly rather than calling remove_vm_area, so the guard page is never subtracted and the wrong size is passed to change_page_attr.
is the intent that iounmap should call remove_vm_area rather than unmap_vm_area (with additional changes to not unlink the vma itself)? or that the guard page should be removed by unmap_ rather than remove_?
when debugging this issue, I also ran into problems with iounmap using virt_to_page on a pci IO region. this problem went away when I tried calling change_page_attr_addr with the virtual address instead. but perhaps iounmap should be calling ioremap_change_attr rather than change_page_attr directly. I'll run some additional tests and send out a patch.
Thanks,
Terence
nvidia: module license 'NVIDIA' taints kernel.
ACPI: PCI Interrupt 0000:02:00.0[A] -> Link [LNK3] -> GSI 19 (level, high) -> IRQ 185
PCI: Setting latency timer of device 0000:02:00.0 to 64
NVRM: loading NVIDIA Linux x86_64 NVIDIA Kernel Module 1.0-7174 Tue Mar 22 06:45:40 PST 2005 ----------- [cut here ] --------- [please bite here ] ---------
Kernel BUG at "arch/x86_64/mm/pageattr.c":154
invalid operand: 0000 [1] PREEMPT SMP
CPU 3
Modules linked in: nvidia eeprom i2c_nforce2 smsc47b397 i2c_sensor i2c_isa i2c_core snd_pcm_oss snd_mixer_oss snd_emu10k1 snd_rawmidi snd_ac97_codec snd_pcm snd_page_alloc snd_util_mem snd_hwdep
Pid: 6087, comm: X Tainted: P 2.6.12-rc3
RIP: 0010:[<ffffffff80121844>] <ffffffff80121844>{__change_page_attr+724}
RSP: 0018:ffff81015c0ffad8 EFLAGS: 00010282
RAX: 00000000e00001e3 RBX: 8000000000000163 RCX: 0000000000000000
RDX: 0000000000000054 RSI: 00000000000e0001 RDI: ffff81000000f000
RBP: 8000000000000163 R08: 03fffffffffff000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000000 R12: ffff8100e0001000
R13: ffff8100010002a0 R14: ffff81000000c800 R15: 0000000000000002
FS: 00002aaaab493b80(0000) GS:ffffffff8049fac0(0000) knlGS:0000000000000000
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 00000000008d0540 CR3: 000000015c5b4000 CR4: 00000000000006e0
Process X (pid: 6087, threadinfo ffff81015c0fe000, task ffff81009fede330)
Stack: ffff8100e0001000 00000000000e0001 0000000000000000 0000000000000001
ffffffff7fffffff ffffffff80121a8c 0000000000000000 0000000000000163
8000000000000163 ffff810002c66c80
Call Trace:<ffffffff80121a8c>{change_page_attr_addr+140} <ffffffff8012149b>{iounmap+459}
<ffffffff882b8b0c>{:nvidia:os_unmap_kernel_space+9}
<ffffffff880abc02>{:nvidia:_nv001628rm+42} <ffffffff880a8fe2>{:nvidia:_nv002102rm+208}
<ffffffff880a8323>{:nvidia:_nv002113rm+255} <ffffffff880a84f8>{:nvidia:_nv002071rm+100}
<ffffffff881c6831>{:nvidia:_nv004369rm+371} <ffffffff880a85e0>{:nvidia:_nv002114rm+64}
<ffffffff880aeb69>{:nvidia:_nv003530rm+141} <ffffffff880aeaad>{:nvidia:_nv003486rm+275}
<ffffffff88233a1a>{:nvidia:_nv003113rm+126} <ffffffff881c8278>{:nvidia:_nv004360rm+100}
<ffffffff881c8064>{:nvidia:_nv004193rm+142} <ffffffff880ae8ba>{:nvidia:_nv001209rm+118}
<ffffffff880af1b1>{:nvidia:_nv001214rm+471} <ffffffff880b24f1>{:nvidia:rm_init_adapter+107}
<ffffffff882b2f90>{:nvidia:nv_kern_open+826} <ffffffff80181c49>{chrdev_open+457}
<ffffffff80177e5b>{dentry_open+315} <ffffffff80177fde>{filp_open+62}
<ffffffff801780cb>{get_unused_fd+219} <ffffffff8017821c>{sys_open+76}
<ffffffff8010e916>{system_call+126}
Code: 0f 0b 7c ab 36 80 ff ff ff ff 9a 00 41 8b 45 00 f6 c4 08 74 RIP <ffffffff80121844>{__change_page_attr+724} RSP <ffff81015c0ffad8>
<6>note: X[6087] exited with preempt_count 1
scheduling while atomic: X/0x00000001/6087
Call Trace:<ffffffff8034dd1a>{schedule+122} <ffffffff801456f3>{flush_cpu_workqueue+467}
<ffffffff8014a1c0>{autoremove_wake_function+0} <ffffffff8034dae8>{__down+152}
<ffffffff8012ffa0>{default_wake_function+0} <ffffffff8034f5d9>{__down_failed+53}
<ffffffff801a8700>{proc_destroy_inode+0} <ffffffff882b8cb1>{:nvidia:.text.lock.os_interface+5}
<ffffffff880abea6>{:nvidia:_nv001740rm+12} <ffffffff880b2d39>{:nvidia:rm_free_unused_clients+105}
<ffffffff882b30d8>{:nvidia:nv_kern_ctl_close+146} <ffffffff882b320d>{:nvidia:nv_kern_close+252}
<ffffffff80179992>{__fput+98} <ffffffff801782fe>{filp_close+126}
<ffffffff801360b3>{put_files_struct+115} <ffffffff80136a13>{do_exit+403}
<ffffffff80245b87>{do_unblank_screen+119} <ffffffff801101e5>{die+69}
<ffffffff80110631>{do_invalid_op+145} <ffffffff80121844>{__change_page_attr+724}
<ffffffff8011a8a0>{do_flush_tlb_all+0} <ffffffff8010f329>{error_exit+0}
<ffffffff80121844>{__change_page_attr+724} <ffffffff80121899>{__change_page_attr+809}
<ffffffff80121a8c>{change_page_attr_addr+140} <ffffffff8012149b>{iounmap+459}
<ffffffff882b8b0c>{:nvidia:os_unmap_kernel_space+9}
<ffffffff880abc02>{:nvidia:_nv001628rm+42} <ffffffff880a8fe2>{:nvidia:_nv002102rm+208}
<ffffffff880a8323>{:nvidia:_nv002113rm+255} <ffffffff880a84f8>{:nvidia:_nv002071rm+100}
<ffffffff881c6831>{:nvidia:_nv004369rm+371} <ffffffff880a85e0>{:nvidia:_nv002114rm+64}
<ffffffff880aeb69>{:nvidia:_nv003530rm+141} <ffffffff880aeaad>{:nvidia:_nv003486rm+275}
<ffffffff88233a1a>{:nvidia:_nv003113rm+126} <ffffffff881c8278>{:nvidia:_nv004360rm+100}
<ffffffff881c8064>{:nvidia:_nv004193rm+142} <ffffffff880ae8ba>{:nvidia:_nv001209rm+118}
<ffffffff880af1b1>{:nvidia:_nv001214rm+471} <ffffffff880b24f1>{:nvidia:rm_init_adapter+107}
<ffffffff882b2f90>{:nvidia:nv_kern_open+826} <ffffffff80181c49>{chrdev_open+457}
<ffffffff80177e5b>{dentry_open+315} <ffffffff80177fde>{filp_open+62}
<ffffffff801780cb>{get_unused_fd+219} <ffffffff8017821c>{sys_open+76}
<ffffffff8010e916>{system_call+126} Quoted 4 lines · Terence Ripperda
this appears to be the 'vmalloc guard page causing
change_page_attrproblems' bug. at one point, iounmap subtracted the guard page before callingchange_page_attr, but I see this was reverted as part of a recent cleanup.
Hmm, yes looks like it was reintroduced.
Quoted 1 line
from looking at the implementation in 2.6.12-pre4, I'm not clear how
I suppose you mean rc4, not pre4?
Quoted 5 lines
the guard page is accounted for in iounmap. in
vmalloc.c, the guard page is subtracted from thevm_structinremove_vm_area(which callsunmap_vm_area). but iounmap inioremap.ccallsunmap_vm_areadirectly rather than callingremove_vm_area, so the guard page is never subtracted and the wrong size is passed tochange_page_attr.
Ok obviously needs to be fixed.
Quoted 5 lines
is the intent that iounmap should call
remove_vm_arearather thanunmap_vm_area(with additional changes to not unlink the vma itself)? or that the guard page should be removed byunmap_rather thanremove_?
There doesn't seem to be a clear rule, that is where the confusion comes from I guess. I would consider it cleaner to handle it in the higher level vmalloc code.
Quoted 4 lines
when debugging this issue, I also ran into problems with iounmap using
virt_to_pageon a pci IO region. this problem went away when I tried callingchange_page_attr_addrwith the virtual address instead. but
A patch for that already went into mainline.
Quoted 1 line
perhaps iounmap should be calling
ioremap_change_attrrather than
What is ioremap_change_attr?
Quoted 2 lines
change_page_attrdirectly. I'll run some additional tests and send out a patch.
-Andi
Quoted 3 lines · On Thu, May 19, 2005 at 01:29:01AM +0200, [email protected] wrote:
Quoted 1 line
from looking at the implementation in 2.6.12-pre4, I'm not clear how
I suppose you mean rc4, not pre4?
yes, just confused.
Quoted 8 lines
Quoted 4 lines
is the intent that iounmap should call
remove_vm_arearather thanunmap_vm_area(with additional changes to not unlink the vma itself)? or that the guard page should be removed byunmap_rather thanremove_?There doesn't seem to be a clear rule, that is where the confusion comes from I guess. I would consider it cleaner to handle it in the higher level vmalloc code.
that certainly sounds reasonable. I'll try putting together a patch that does this and send it to our end user to see if it fixes his problem.
Quoted 5 lines
Quoted 3 lines
when debugging this issue, I also ran into problems with iounmap using
virt_to_pageon a pci IO region. this problem went away when I tried callingchange_page_attr_addrwith the virtual address instead. butA patch for that already went into mainline.
ok, great.
Quoted 3 lines
Quoted 1 line
perhaps iounmap should be calling
ioremap_change_attrrather thanWhat is
ioremap_change_attr?
a static function in ioremap.c that is called by __ioremap. it's a wrapper function around change_page_attr. I only see it in the x86_64 architecture, not i386.
Thanks,
Terence
Quoted 7 lines · Terence Ripperda
Quoted 3 lines
Quoted 1 line
perhaps iounmap should be calling
ioremap_change_attrrather thanWhat is
ioremap_change_attr?a static function in
ioremap.cthat is called by__ioremap. it's a wrapper function aroundchange_page_attr. I only see it in thex86_64architecture, not i386.
Oh yes, indeed. I forgot that I wrote it :) Calling global_flush_tlb in iounmap is indeed a very good idea.
-Andi