Skip to contentSkip to conversation

2.6.10 Kernel BUG at hugetlbpage:212 (x86_64 and i386)

DISCUSSION Mr. Berkley Shands 3 participants 5 messages Feb 3, 2005 – Feb 4, 2005 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

Reproducible BUG on 3GB hugetlbfs filesystem for opterons and xeons with either FC3 or RedHat ES3.0 and GCC 3.4.2. Details and code snippets in attachment. Executables to reproduce BUG are available on request.

berkley

On an 8GB dual cpu opteron (Tyan S2884) 2.6.10 kernel I can reproduce a crash within several minutes by creating/mapping/deleting/unmapping 512MB files using the code below. On a IA32 box (Xeon 2.4GHz, 3GB SuperMicro X5DA8) the crash is fatal (no /var/log/messages) and immediate. Executables for x86_64 and i386 from either FC3 or Redhat ES3.0 available on request. GCC is 3.4.2 on both machines and O/S releases.

Feb  2 11:28:49 noreaster kernel: ----------- [cut here ] --------- [please bite here ] ---------
Feb  2 11:28:49 noreaster kernel: Kernel BUG at hugetlbpage:212
Feb  2 11:28:49 noreaster kernel: invalid operand: 0000 [1] SMP
Feb  2 11:28:49 noreaster kernel: CPU 1
Feb  2 11:28:49 noreaster kernel: Modules linked in:
Feb  2 11:28:49 noreaster kernel: Pid: 15687, comm: DssiEPSearch Not tainted 2.6.10
Feb  2 11:28:49 noreaster kernel: RIP: 0010:[<ffffffff8011e3cb>] <ffffffff8011e3cb>{unmap_hugepage_range+75}
Feb  2 11:28:49 noreaster kernel: RSP: 0018:00000100d7701dd8  EFLAGS: 00010206
Feb  2 11:28:49 noreaster kernel: RAX: 000001012eee98e0 RBX: 000001001000a0c0 RCX: 0000000061001000
Feb  2 11:28:49 noreaster kernel: RDX: 0000000061001000 RSI: 0000000041000000 RDI: 000001012eee98e0
Feb  2 11:28:49 noreaster kernel: RBP: 0000000041000000 R08: 0000000061200000 R09: 00000100d7701f10
Feb  2 11:28:49 noreaster kernel: R10: 0000000000000000 R11: 0000000000000202 R12: 0000000061001000
Feb  2 11:28:49 noreaster kernel: R13: 00000101f6ee0040 R14: 0000000041000000 R15: 000001012eee98e0
Feb  2 11:28:49 noreaster kernel: FS:  0000002a96b88080(0000) GS:ffffffff80611c00(0000) knlGS:0000000000000000
Feb  2 11:28:49 noreaster kernel: CS:  0010 DS: 0000 ES: 0000 CR0: 000000008005003b
Feb  2 11:28:49 noreaster kernel: CR2: 0000002a9632e970 CR3: 000000001184a000 CR4: 00000000000006e0
Feb  2 11:28:49 noreaster kernel: Process DssiEPSearch (pid: 15687, threadinfo 00000100d7700000, task 00000101215c63f0)
Feb  2 11:28:49 noreaster kernel: Stack: 0000000000000206 000001001000a0c0 0000000041000000 000001012eee9838
Feb  2 11:28:49 noreaster kernel:        00000101f6ee0040 0000000061200000 000001012eee98e0 ffffffff8015d800
Feb  2 11:28:49 noreaster kernel:        0000000000000000 ffffffff802bf59b
Feb  2 11:28:49 noreaster kernel: Call Trace:<ffffffff8015d800>{unmap_vmas+320} <ffffffff802bf59b>{fpga_ioctl+2171}
Feb  2 11:28:49 noreaster kernel:        <ffffffff802c141b>{fpga_read+555} <ffffffff80161a23>{do_munmap+547}
Feb  2 11:28:49 noreaster kernel:        <ffffffff801627a8>{sys_munmap+72} <ffffffff8010d216>{system_call+126}
Feb  2 11:28:49 noreaster kernel:
Feb  2 11:28:49 noreaster kernel:
Feb  2 11:28:49 noreaster kernel: Code: 0f 0b 33 1f 49 80 ff ff ff ff d4 00 4c 89 f5 4d 39 e6 66 66
Feb  2 11:28:49 noreaster kernel: RIP <ffffffff8011e3cb>{unmap_hugepage_range+75} RSP <00000100d7701dd8>
Feb  2 11:32:32 noreaster kernel: Linux version 2.6.10 (root@noreaster) (gcc version 3.4.2) #47 SMP Wed Feb 2 08:01:40 CST 2005

// code to access a huge tlb filesystem

#if defined(__linux__) && defined(__x86_64__)
HugePageFileName_ = new char[64];
::sprintf(HugePageFileName_, "/mnt/huge/Silo_XXXXXX");
::mkstemp(HugePageFileName_); // randomize this name
Mmap_Fd_ = ::open(HugePageFileName_, O_CREAT | O_RDWR | O_LARGEFILE | O_TRUNC, 0755); if (Mmap_Fd_ != -1)

   {
      LONG64 MyBig = Big * sizeof(LONG64) + ((2UL * 1024UL * 1024UL) - 1);
      MyBig &= ~((2UL * 1024UL * 1024UL) - 1);
      BigBlock0_ = (LONG64 *) ::mmap(NULL, MyBig, 
      		(PROT_READ | PROT_WRITE), 
      		MAP_SHARED, Mmap_Fd_, 0);
      if (BigBlock0_ == MAP_FAILED || !BigBlock0_)
      {
         ::perror("mmap failed for huge pages");
	 ::cerr << "Asked for ";
	 printHRNumber(MyBig, cerr);
	 ::cerr << endl;
	 ::close(Mmap_Fd_);
	 ::unlink(HugePageFileName_);
	 delete [] HugePageFileName_;
	 HugePageFileName_ = NULL;
	 Mmap_Fd_ = -1;
	 BigBlock0_ = NULL;
	 BigBlock_ = NULL;
      }
      else
      {
      BigBlock_ = BigBlock0_;
      BigSize_ = MyBig;			// remember this size
      ::unlink(HugePageFileName_);	// pre-delete this file so segfaults
      					// do not leave huge pages tied up.
      }
   }
   else
   {
      if (Debug_)
      {
         ::perror("Sorry, no HUGE pages today");
      }

}
#endif

// snip from /etc/rc.local

#

# setup the huge file system space pages are 2MB each
#
if [-f /proc/fpga0]; then
   echo "Allocating huge file system"
   echo 1536 > /proc/sys/vm/nr_hugepages
   mount none /mnt/huge -t hugetlbfs -o size=3G,mode=0777
   fi
#1 lore raw
Quoted 4 lines · Mr. Berkley Shands

Reproducible BUG on 3GB hugetlbfs filesystem for opterons and xeons with either
FC3 or RedHat ES3.0 and GCC 3.4.2. Details and code snippets in attachment. Executables to reproduce BUG are available on request.

Patch below (against 2.6.11-rc3, applies at offset to 2.6.10) fixes the unmap_hugepage_range BUGs I could generate: does it fix yours?

Hugh

The hugetlb_page test in do_munmap is too permissive. It checks start vma, but forgets that end vma might be different and huge though start is not: so hits unmap_hugepage_range BUG if misaligned end was given.

And it's too restrictive: munmap has always succeeded on unmapped areas within its range, why should it behave differently near a hugepage vma?

And the additional checks in is_aligned_hugepage_range are irrelevant here, when the hugepage vma already exists. But the function is still required (on some arches), as the default for prepare_hugepage_range - leave renaming cleanup to another occasion.

mm/mmap.c +4 -7
@@ -1808,13 +1808,6 @@ int do_munmap(struct mm_struct *mm, unsi
 		return 0;
 	/* we have  start < mpnt->vm_end  */
 
-	if (is_vm_hugetlb_page(mpnt)) {
-		int ret = is_aligned_hugepage_range(start, len);
-
-		if (ret)
-			return ret;
-	}
-
 	/* if it doesn't overlap, we have nothing.. */
 	end = start + len;
 	if (mpnt->vm_start >= end)
@@ -1828,6 +1821,8 @@ int do_munmap(struct mm_struct *mm, unsi
 	 * places tmp vma above, and higher split_vma places tmp vma below.
 	 */
 	if (start > mpnt->vm_start) {
+		if (is_vm_hugetlb_page(mpnt) && (start & ~HPAGE_MASK))
+			return -EINVAL;
 		if (split_vma(mm, mpnt, start, 0))
 			return -ENOMEM;
 		prev = mpnt;
@@ -1836,6 +1831,8 @@ int do_munmap(struct mm_struct *mm, unsi
 	/* Does it split the last one? */
 	last = find_vma(mm, end);
 	if (last && end > last->vm_start) {
+		if (is_vm_hugetlb_page(last) && (end & ~HPAGE_MASK))
+			return -EINVAL;
 		if (split_vma(mm, last, end, 1))
 			return -ENOMEM;
 	}
#2 lore raw
Quoted 11 lines · Hugh Dickins

Patch below (against 2.6.11-rc3, applies at offset to 2.6.10) fixes the unmap_hugepage_range BUGs I could generate: does it fix yours? The hugetlb_page test in do_munmap is too permissive. It checks start vma, but forgets that end vma might be different and huge though start is not: so hits unmap_hugepage_range BUG if misaligned end was given. And it's too restrictive: munmap has always succeeded on unmapped areas within its range, why should it behave differently near a hugepage vma? And the additional checks in is_aligned_hugepage_range are irrelevant here, when the hugepage vma already exists. But the function is still required (on some arches), as the default for prepare_hugepage_range - leave renaming cleanup to another occasion.

Quoted 1 line

As usual, excellent work. Thanks for fixing this up.

-- wli

A message that is not in the archive (sent off-list or not yet received): [email protected]

#4 lore raw
Quoted 2 lines · Mr. Berkley Shands

Sorry, but I still crash. This time it hung the kernel so bad I had to powerfail to restart.

Well, that fix is needed anyway.

Quoted 2 lines · Mr. Berkley Shands
Feb  4 13:43:19 eclipse kernel: RIP: 0010:[<ffffffff8011e3cb>] 

<ffffffff8011e3cb>{unmap_hugepage_range+75}

Could you try this?

-- wli

arch/i386/mm/hugetlbpage.c +5 -2
Index: mm2-2.6.11-rc2/arch/i386/mm/hugetlbpage.c
===================================================================
@@ -209,14 +209,17 @@
 {
 	struct mm_struct *mm = vma->vm_mm;
 	unsigned long address;
-	pte_t pte;
+	pte_t pte, *ptep;
 	struct page *page;
 
 	BUG_ON(start & (HPAGE_SIZE - 1));
 	BUG_ON(end & (HPAGE_SIZE - 1));
 
 	for (address = start; address < end; address += HPAGE_SIZE) {
-		pte = ptep_get_and_clear(huge_pte_offset(mm, address));
+		ptep = huge_pte_offset(mm, address);
+		if (!ptep)
+			continue;
+		pte = ptep_get_and_clear(ptep);
 		if (pte_none(pte))
 			continue;
 		page = pte_page(pte);
#5 lore raw
In reply to a message not in the archive
Quoted 3 lines · Mr. Berkley Shands
Quoted 1 line

Sorry, but I still crash. This time it hung the kernel so bad I had to powerfail to restart.

Thanks for trying, okay, your problem is a different one from what my patch fixes: I thought that might turn out to be so.

Quoted 9 lines
Feb  4 13:43:19 eclipse kernel: Kernel BUG at hugetlbpage:212
Feb  4 13:43:19 eclipse kernel: invalid operand: 0000 [1] SMP
Feb  4 13:43:19 eclipse kernel: CPU 1
Feb  4 13:43:19 eclipse kernel: Modules linked in:
Feb  4 13:43:19 eclipse kernel: Pid: 1374, comm: DssiEPSearch Not tainted

2.6.10
<ffffffff8011e3cb>{unmap_hugepage_range+75} RSP <000001007f337dd8>

patch applied and rebooted (I made sure this time :-)

And kernel rebuilt, I trust!

Quoted 2 lines

unless of course I'm not functional today :-)
The patch I had was for mmap.c, not in hugetlb.c. Did I miss something?

No, that's right, it was only fixing mm/mmap.c.

Please point me privately to your app source, oh, it may be gigantic, or difficult for me to compile, I guess your binary as well or instead:
so I can try to reproduce on i386.

We notice fpga_read and fpga_ioctl in both your traces, not in our 2.6.10 kernel source: just leftover addresses on the stack, not part of the real backtrace, but interestingly there in both. Where are they from (FC3?), any idea what chance it's doing something bad?

Hugh