Skip to contentSkip to conversation

Latest AIO patchset

DISCUSSION Suparna Bhattacharya 8 participants 15 messages Feb 24, 2004 – Mar 8, 2004

lore
Share & tools
lore
#0 lore raw

The latest set of AIO patches are being maintained at:
http://www.kernel.org/pub/linux/kernel/people/suparna/aio/

The patches have been reduced to a minimal set that addresses the most relevant blocking points. Please let me know if you think there is a case for bringing in any of the additional patches.
(The patches need to be applied in the order mention in the 'series' file)

A new addition to the patchset is Chris Mason's nice and simple implementation of AIO support for pipes using the retry infrastructure. He also fixed some problems in the AIO cancel logic to make it play well with retries.

Besides this and some re-organization and cleaning up, there are a couple of changes since the last set of patches in -mm, that are worth a mention:

- Upfront readahead is now clipped to the readahead limit for the device (ra_pages) and happens only for AIO. This helps address the sendfile regression seen by Felix von Leitner. If your AIO read requests are likely to exceed the default readahead size, then use hdparm -a <new size> to tune it. The patchset also currently includes Ram Pai's adaptive lazy readahead code which is required for good streaming AIO read performance.
- David Brownell's suggestion of enabling the fops to set up their own retry methods. This should make his USB gadgetfs AIO patch co-exist smoothly with fsaio.

Some basic results are up comparing streaming non-cached random AIO read/write throughputs for a single ext3 file using aio-stress for various io sizes with and without these patches, and also comparsions with O_DIRECT AIO throughputs. The short summary has been a doubling of throughput using the fsaio patches, which is also close to the results seen with O_DIRECT AIO.

As usual feedback, bug fixes, test results etc are welcome.

Regards
Suparna

Signature · 3 lines
Suparna Bhattacharya ([email protected])
Linux Technology Center
IBM Software Lab, India
#1 lore raw

kexec "problem"

hi guys,

i have just compiled a kernel with the kexec patch. compiled kexec-tools and when i try to load a kernel, it gives me this:

# ./do-kexec.sh /boot/bzImage-2.6.2-g
kexec_load failed: Invalid argument
entry       = 0x91764
nr_segments = 2
segment[0].buf   = 0x80b3480
segment[0].bufsz = 1880
segment[0].mem   = 0x90000
segment[0].memsz = 1880
segment[1].buf   = 0x40001008
segment[1].bufsz = 19795a
segment[1].mem   = 0x100000
segment[1].memsz = 19795a

anyone tried to run kexec and actually did it? i'm trying with kernel 2.6.3

#2 lore raw

Re: kexec "problem"

On Tue, 24 Feb 2004 11:02:21 -0000 (WET) Carlos Silva wrote:

| hi guys,
| 
| i have just compiled a kernel with the kexec patch. compiled kexec-tools
| and when i try to load a kernel, it gives me this:
| # ./do-kexec.sh /boot/bzImage-2.6.2-g
| kexec_load failed: Invalid argument
| entry       = 0x91764
| nr_segments = 2
| segment[0].buf   = 0x80b3480
| segment[0].bufsz = 1880
| segment[0].mem   = 0x90000
| segment[0].memsz = 1880
| segment[1].buf   = 0x40001008
| segment[1].bufsz = 19795a
| segment[1].mem   = 0x100000
| segment[1].memsz = 19795a
| 
| anyone tried to run kexec and actually did it? i'm trying with kernel 2.6.3

I haven't updated for 2.6.3 yet, or even tested it... but I'll get to it soon.

--
~Randy

#3 lore raw

Re: kexec "problem"

Quoted 20 lines · On Tue, 24 Feb 2004 11:02:21 -0000 (WET) Carlos Silva wrote:
| hi guys,
|
| i have just compiled a kernel with the kexec patch. compiled kexec-tools
| and when i try to load a kernel, it gives me this:
| # ./do-kexec.sh /boot/bzImage-2.6.2-g
| kexec_load failed: Invalid argument
| entry       = 0x91764
| nr_segments = 2
| segment[0].buf   = 0x80b3480
| segment[0].bufsz = 1880
| segment[0].mem   = 0x90000
| segment[0].memsz = 1880
| segment[1].buf   = 0x40001008
| segment[1].bufsz = 19795a
| segment[1].mem   = 0x100000
| segment[1].memsz = 19795a
|
| anyone tried to run kexec and actually did it? i'm trying with kernel
Quoted 8 lines

2.6.3

I haven't updated for 2.6.3 yet, or even tested it... but I'll get to it soon.

--
~Randy

just changed line 13 on kexec.h from:
"long kexec_load(void *entry, unsigned long nr_segments, struct kexec_segment *segments, unsigned long);"
to:
"long kexec_load(void *entry, unsigned long nr_segments, struct kexec_segment *segments, unsigned long flags);"

and it gives me this error:

# ./do-kexec.sh /boot/bzImage-2.6.2-g
kexec_load failed: Function not implemented
entry       = 0x91764
nr_segments = 2
segment[0].buf   = 0x80b3480
segment[0].bufsz = 1880
segment[0].mem   = 0x90000
segment[0].memsz = 1880
segment[1].buf   = 0x40001008
segment[1].bufsz = 19795a
segment[1].mem   = 0x100000
segment[1].memsz = 19795a
#11 lore raw
In reply to Suparna Bhattacharya
Quoted 10 lines

Some basic results are up comparing streaming non-cached random AIO read/write throughputs for a single ext3 file using aio-stress for various io sizes with and without these patches, and also comparsions with O_DIRECT AIO throughputs. The short summary has been a doubling of throughput using the fsaio patches, which is also close to the results seen with O_DIRECT AIO.

As usual feedback, bug fixes, test results etc are welcome.

What exactly is the O_DIRECT flag? When I add this flag to the open func it fails.

More specificaly, this function fails
open("filename", O_RDWR | O_DIRECT | O_LARGEFILE | O_CREAT, S_IRWXU);

but this one succeeds
open("filename", O_RDWR | O_LARGEFILE | O_CREAT, S_IRWXU);

I'm running linux 2.6.0 with libaio 0.3.92.

Hayim.

#12 lore raw
Quoted 10 lines · Hayim Shaul

What exactly is the O_DIRECT flag? When I add this flag to the open func it fails.

More specificaly, this function fails
open("filename", O_RDWR | O_DIRECT | O_LARGEFILE | O_CREAT, S_IRWXU);

but this one succeeds
open("filename", O_RDWR | O_LARGEFILE | O_CREAT, S_IRWXU);

I'm running linux 2.6.0 with libaio 0.3.92.

Which filesystem? Not all support O_DIRECT.

-ben

#13 lore raw
Quoted 7 lines · Benjamin LaHaise
Quoted 6 lines · Hayim Shaul

What exactly is the O_DIRECT flag? When I add this flag to the open func it fails.

More specificaly, this function fails
open("filename", O_RDWR | O_DIRECT | O_LARGEFILE | O_CREAT, S_IRWXU);

Quoted 7 lines
Quoted 4 lines

but this one succeeds
open("filename", O_RDWR | O_LARGEFILE | O_CREAT, S_IRWXU);

I'm running linux 2.6.0 with libaio 0.3.92.

Which filesystem? Not all support O_DIRECT.

ext3
I'm think it does support ext3.

Actually, I was wrong. open does succeed. It return a valid fd but after writing and exiting, the file is still zero size.

removing the O_DIRECT with the same prog writes quite alot to the file.

Hayim.

#14 lore raw

Are you checking the return value of write()?

O_DIRECT has alignment requirements since it writes directly from user-space to disk by passing the page cache. Also the size of the write has to be a multiple of 512 (for 2.6).
Try using posix_memalign() with pagesize as the alignment arg to allocated the data buffer. O_DIRECT does work on ext3.

Daniel

Quoted 32 lines · Hayim Shaul
Quoted 7 lines · Benjamin LaHaise
Quoted 6 lines · Hayim Shaul

What exactly is the O_DIRECT flag? When I add this flag to the open func it fails.

More specificaly, this function fails
open("filename", O_RDWR | O_DIRECT | O_LARGEFILE | O_CREAT, S_IRWXU);

Quoted 7 lines
Quoted 4 lines

but this one succeeds
open("filename", O_RDWR | O_LARGEFILE | O_CREAT, S_IRWXU);

I'm running linux 2.6.0 with libaio 0.3.92.

Which filesystem? Not all support O_DIRECT.

ext3
I'm think it does support ext3.

Actually, I was wrong. open does succeed. It return a valid fd but after writing and exiting, the file is still zero size.

removing the O_DIRECT with the same prog writes quite alot to the file.

Hayim.

--
To unsubscribe, send a message with 'unsubscribe linux-aio' in the body to [email protected]. For more info on Linux AIO,
see: http://www.kvack.org/aio/
Don't email: <a href=mailto:"[email protected]">[email protected]</a>

#4 lore raw

Re: kexec "problem" [and patch updates]

In reply to Carlos Silva

On Tue, 24 Feb 2004 11:02:21 -0000 (WET) Carlos Silva wrote:

| hi guys,
| 
| i have just compiled a kernel with the kexec patch. compiled kexec-tools
| and when i try to load a kernel, it gives me this:
| # ./do-kexec.sh /boot/bzImage-2.6.2-g
| kexec_load failed: Invalid argument
| entry       = 0x91764
| nr_segments = 2
| segment[0].buf   = 0x80b3480
| segment[0].bufsz = 1880
| segment[0].mem   = 0x90000
| segment[0].memsz = 1880
| segment[1].buf   = 0x40001008
| segment[1].bufsz = 19795a
| segment[1].mem   = 0x100000
| segment[1].memsz = 19795a
| 
| anyone tried to run kexec and actually did it? i'm trying with kernel 2.6.3
| -

I updated the kexec patch for 2.6.2 and 2.6.3.
It works fine on 2.6.2. It works for me on 2.6.3 if not SMP. If the kernel is built for SMP, when running kexec, I get a BUG in arch/i386/kernel/smp.c at line 359.
I'm testing various workarounds for that BUG now.

--
~Randy

kexec updates are at:
http://developer.osdl.org/rddunlap/kexec/2.6.2/
and
http://developer.osdl.org/rddunlap/kexec/2.6.3/

#5 lore raw

Re: [Fastboot] Re: kexec "problem" [and patch updates]

Quoted 22 lines · Randy.Dunlap

On Tue, 24 Feb 2004 11:02:21 -0000 (WET) Carlos Silva wrote:

| hi guys,
| 
| i have just compiled a kernel with the kexec patch. compiled kexec-tools
| and when i try to load a kernel, it gives me this:
| # ./do-kexec.sh /boot/bzImage-2.6.2-g
| kexec_load failed: Invalid argument
| entry       = 0x91764
| nr_segments = 2
| segment[0].buf   = 0x80b3480
| segment[0].bufsz = 1880
| segment[0].mem   = 0x90000
| segment[0].memsz = 1880
| segment[1].buf   = 0x40001008
| segment[1].bufsz = 19795a
| segment[1].mem   = 0x100000
| segment[1].memsz = 19795a
| 
| anyone tried to run kexec and actually did it? i'm trying with kernel 2.6.3
| -
Quoted 5 lines

I updated the kexec patch for 2.6.2 and 2.6.3.
It works fine on 2.6.2. It works for me on 2.6.3 if not SMP. If the kernel is built for SMP, when running kexec, I get a BUG in arch/i386/kernel/smp.c at line 359.
I'm testing various workarounds for that BUG now.

I will eyeball it...

Is it the kernel that is shutting down, or the kernel that is being brought up that has problems?

The back trace from the BUG would be interesting.

As I see it flush_tlb_others is being called when we have shutdown cpus and the kernel still thinks we have the mm present on foreign cpus.

So it appears we simply have a case that was not anticipated by the authors of that code. So we need to adjust either the code we are calling or cpu_vm_mask so it does not list other cpus after we have shut them down.

At least that it what it looks like at first glance.

Eric

#6 lore raw

Re: [Fastboot] Re: kexec "problem" [and patch updates]

On 27 Feb 2004 01:00:04 -0700 Eric W. Biederman wrote:

| "Randy.Dunlap" <[email protected]> writes:
| 
| > On Tue, 24 Feb 2004 11:02:21 -0000 (WET) Carlos Silva wrote:
| > 
| > | hi guys,
| > | 
| > | i have just compiled a kernel with the kexec patch. compiled kexec-tools
| > | and when i try to load a kernel, it gives me this:
| > | # ./do-kexec.sh /boot/bzImage-2.6.2-g
| > | kexec_load failed: Invalid argument
| > | entry       = 0x91764
| > | nr_segments = 2
| > | segment[0].buf   = 0x80b3480
| > | segment[0].bufsz = 1880
| > | segment[0].mem   = 0x90000
| > | segment[0].memsz = 1880
| > | segment[1].buf   = 0x40001008
| > | segment[1].bufsz = 19795a
| > | segment[1].mem   = 0x100000
| > | segment[1].memsz = 19795a
| > | 
| > | anyone tried to run kexec and actually did it? i'm trying with kernel 2.6.3
| > | -
| > 
| > I updated the kexec patch for 2.6.2 and 2.6.3.
| > It works fine on 2.6.2.  It works for me on 2.6.3 if not SMP.
| > If the kernel is built for SMP, when running kexec, I get a
| > BUG in arch/i386/kernel/smp.c at line 359.
| > I'm testing various workarounds for that BUG now.
| 
| I will eyeball it...
| 
| Is it the kernel that is shutting down, or the kernel that is being
| brought up that has problems?

the kernel that is shutting down.

| The back trace from the BUG would be interesting.

see below. my bad. i should have included it.

| As I see it flush_tlb_others is being called when we have shutdown
| cpus and the kernel still thinks we have the mm present on foreign
| cpus.

Martin Bligh thinks that there is a tlb race here. I printed the 2 cpu masks on my dual-proc macine and saw 0 in one of them and 0xc in the other one.

| So it appears we simply have a case that was not anticipated
| by the authors of that code.  So we need to adjust either
| the code we are calling or cpu_vm_mask so it does not list
| other cpus after we have shut them down.
| 
| At least that it what it looks like at first glance.

Thanks,
--
~Randy

Feb 25 15:52:21 gargoyle kernel: ------------[ cut here ]------------
Feb 25 15:52:21 gargoyle kernel: kernel BUG at arch/i386/kernel/smp.c:359!
Feb 25 15:52:21 gargoyle kernel: invalid operand: 0000 [#1]
Feb 25 15:52:21 gargoyle kernel: CPU:    1
Feb 25 15:52:21 gargoyle kernel: EIP:    0060:[<c011673d>]    Not tainted
Feb 25 15:52:21 gargoyle kernel: EFLAGS: 00010206
Feb 25 15:52:21 gargoyle kernel: EIP is at flush_tlb_others+0x141/0x15c
Feb 25 15:52:21 gargoyle kernel: eax: 00000000   ebx: c043c9e0   ecx: c043c9e0   edx: 0000000c
Feb 25 15:52:21 gargoyle kernel: esi: f53effc4   edi: 00851da8   ebp: f5449ebc   esp: f5449ea8
Feb 25 15:52:21 gargoyle kernel: ds: 007b   es: 007b   ss: 0068
Feb 25 15:52:21 gargoyle kernel: Process kexec (pid: 1095, threadinfo=f5448000 task=f54719b0)
Feb 25 15:52:21 gargoyle kernel: Stack: f5449ed4 c014eae3 c1851d58 353f0000 00000000 f5449ed4 c01167e9 0000000c 
Feb 25 15:52:21 gargoyle kernel:        c043c9e0 ffffffff 0000000c f5449f20 c0150084 c043c9e0 f61d7610 003f1000 
Feb 25 15:52:21 gargoyle kernel:        003f1000 35000000 35000000 00400000 c0101354 c043ca14 c043c9e0 353f1000 
Feb 25 15:52:21 gargoyle kernel: Call Trace:
Feb 25 15:52:21 gargoyle kernel:  [<c014eae3>] pte_alloc_map+0xd9/0x12e
Feb 25 15:52:21 gargoyle kernel:  [<c01167e9>] flush_tlb_mm+0x47/0x8c
Feb 25 15:52:21 gargoyle kernel:  [<c0150084>] remap_page_range+0x1ae/0x218
Feb 25 15:52:21 gargoyle kernel:  [<c013dae7>] identity_map_pages+0xf7/0x130
Feb 25 15:52:21 gargoyle kernel:  [<c013dbd4>] kimage_alloc_reboot_code_pages+0xb4/0x164
Feb 25 15:52:21 gargoyle kernel:  [<c013d928>] kimage_alloc+0x100/0x186
Feb 25 15:52:21 gargoyle kernel:  [<c013e496>] sys_kexec_load+0x9c/0xff
Feb 25 15:52:21 gargoyle kernel:  [<c0109637>] syscall_call+0x7/0xb
Feb 25 15:52:21 gargoyle kernel: 
Feb 25 15:52:21 gargoyle kernel: Code: 0f 0b 67 01 ee df 3d c0 e9 d7 fe ff ff 0f 0b 64 01 ee df 3d 
#7 lore raw

Re: [Fastboot] Re: kexec "problem" [and patch updates]

Quoted 18 lines · Randy.Dunlap

On 27 Feb 2004 01:00:04 -0700 Eric W. Biederman wrote:

| > It works fine on 2.6.2.  It works for me on 2.6.3 if not SMP.
| > If the kernel is built for SMP, when running kexec, I get a
| > BUG in arch/i386/kernel/smp.c at line 359.
| > I'm testing various workarounds for that BUG now.
| 
| I will eyeball it...
| 
| Is it the kernel that is shutting down, or the kernel that is being
| brought up that has problems?

the kernel that is shutting down.

| The back trace from the BUG would be interesting.

see below. my bad. i should have included it.

Quoted 7 lines
| As I see it flush_tlb_others is being called when we have shutdown
| cpus and the kernel still thinks we have the mm present on foreign
| cpus.

Martin Bligh thinks that there is a tlb race here. I printed the 2 cpu masks on my dual-proc macine and saw 0 in one of them and 0xc in the other one.

Ouch we have both cpus running when this happens, and we have not started any shutdown whatsoever. This is the bit that sets up the page tables for later use...

I think identity_map_pages will have problems with a kernel that does the 4G/4G split, and it has known issues on some other architectures, because they treat init_mm specially. So the proper solution may be to simply rewrite identity_map_pages.

Before we do that in the short term we need to see if identity_map_pages is actually doing anything bad. You are not using the 4G/4G split so that is not the cause. So either init_mm is now special in some way, or we have hit a generic kernel bug.

So this may indeed be a tlb race. But it is init_mm->cpu_vm_mask and cpu_online map that are different. With the implication being that init_mm->cpu_vm_mask has cpus set that are not in cpu_online_map? Very weird especially on SMP.

Without attribution I have a hard time making sense of which cpumask is which so I can't draw any conclusions. But I find it very interesting that it is bits 2 and 3 that are set. I wonder if there is any mixup between logical cpu identities and apic ids.

Eric

#8 lore raw

Re: [Fastboot] Re: kexec "problem" [and patch updates]

Hello,

I recreated this on a UNI system running an SMP kernel as well.

The problem is because we now initialize cpu_vm_mask for init_mm with CPU_MASK_ALL (from 2.6.3 onwards) which makes all bits in cpumask 1. Hence BUG_ON(!cpus_equal(cpumask,tmp) fails. The change to set cpu_vm_mask to CPU_MASK_ALL was done to remove tlb flush optimizations for ppc64. On UNI kernels, CPU_MASK_ALL is 1 and hence the problem does not occur.

I made a small patch which fixes this problem. The change is, essentially, to use "tmp" instead of "cpumask". This ensures that only the (other) online cpus are sent the IPI.

I have done some testing with this patch. Kexec loads fine and I haven't seen anything untoward.

Comments please.

Regards, Hari

arch/i386/kernel/smp.c +5 -4
diff -Naur linux-2.6.3-before/arch/i386/kernel/smp.c linux-2.6.3/arch/i386/kernel/smp.c
@@ -356,7 +356,8 @@
 	BUG_ON(cpus_empty(cpumask));
 
 	cpus_and(tmp, cpumask, cpu_online_map);
-	BUG_ON(!cpus_equal(cpumask, tmp));
+	if(cpus_empty(tmp))
+		return;
 	BUG_ON(cpu_isset(smp_processor_id(), cpumask));
 	BUG_ON(!mm);
 
@@ -371,12 +372,12 @@
 	flush_mm = mm;
 	flush_va = va;
 #if NR_CPUS <= BITS_PER_LONG
-	atomic_set_mask(cpumask, &flush_cpumask);
+	atomic_set_mask(tmp, &flush_cpumask);
 #else
 	{
 		int k;
 		unsigned long *flush_mask = (unsigned long *)&flush_cpumask;
-		unsigned long *cpu_mask = (unsigned long *)&cpumask;
+		unsigned long *cpu_mask = (unsigned long *)&tmp;
 		for (k = 0; k < BITS_TO_LONGS(NR_CPUS); ++k)
 			atomic_set_mask(cpu_mask[k], &flush_mask[k]);
 	}
@@ -385,7 +386,7 @@
 	 * We have to send the IPI only to
 	 * CPUs affected.
 	 */
-	send_IPI_mask(cpumask, INVALIDATE_TLB_VECTOR);
+	send_IPI_mask(tmp, INVALIDATE_TLB_VECTOR);
 
 	while (!cpus_empty(flush_cpumask))
 		/* nothing. lockup detection does not belong here */
Quoted 58 lines · Eric W. Biederman
Quoted 18 lines · Randy.Dunlap

On 27 Feb 2004 01:00:04 -0700 Eric W. Biederman wrote:

| > It works fine on 2.6.2.  It works for me on 2.6.3 if not SMP.
| > If the kernel is built for SMP, when running kexec, I get a
| > BUG in arch/i386/kernel/smp.c at line 359.
| > I'm testing various workarounds for that BUG now.
| 
| I will eyeball it...
| 
| Is it the kernel that is shutting down, or the kernel that is being
| brought up that has problems?

the kernel that is shutting down.

| The back trace from the BUG would be interesting.

see below. my bad. i should have included it.

Quoted 7 lines
| As I see it flush_tlb_others is being called when we have shutdown
| cpus and the kernel still thinks we have the mm present on foreign
| cpus.

Martin Bligh thinks that there is a tlb race here. I printed the 2 cpu masks on my dual-proc macine and saw 0 in one of them and 0xc in the other one.

Ouch we have both cpus running when this happens, and we have not started any shutdown whatsoever. This is the bit that sets up the page tables for later use...

I think identity_map_pages will have problems with a kernel that does the 4G/4G split, and it has known issues on some other architectures, because they treat init_mm specially. So the proper solution may be to simply rewrite identity_map_pages.

Before we do that in the short term we need to see if identity_map_pages is actually doing anything bad. You are not using the 4G/4G split so that is not the cause. So either init_mm is now special in some way, or we have hit a generic kernel bug.

So this may indeed be a tlb race. But it is init_mm->cpu_vm_mask and cpu_online map that are different. With the implication being that init_mm->cpu_vm_mask has cpus set that are not in cpu_online_map? Very weird especially on SMP.

Without attribution I have a hard time making sense of which cpumask is which so I can't draw any conclusions. But I find it very interesting that it is bits 2 and 3 that are set. I wonder if there is any mixup between logical cpu identities and apic ids.

Eric
_______________________________________________
fastboot mailing list
[email protected]
http://lists.osdl.org/mailman/listinfo/fastboot

Signature · 4 lines
Hariprasad Nellitheertha
Linux Technology Center
India Software Labs
IBM India, Bangalore
#9 lore raw

Re: [Fastboot] Re: kexec "problem" [and patch updates]

Quoted 10 lines · Hariprasad Nellitheertha

Hello,

I recreated this on a UNI system running an SMP kernel as well.

The problem is because we now initialize cpu_vm_mask for init_mm with CPU_MASK_ALL (from 2.6.3 onwards) which makes all bits in cpumask 1. Hence BUG_ON(!cpus_equal(cpumask,tmp) fails. The change to set cpu_vm_mask to CPU_MASK_ALL was done to remove tlb flush optimizations for ppc64. On UNI kernels, CPU_MASK_ALL is 1 and hence the problem does not occur.

So the problem is that CPU_MASK_ALL includes cpus that are not currently online. So it has gone from being wrong by including too few cpus to being wrong by including too many cpus.

Quoted 8 lines

I made a small patch which fixes this problem. The change is, essentially, to use "tmp" instead of "cpumask". This ensures that only the (other) online cpus are sent the IPI.

I have done some testing with this patch. Kexec loads fine and I haven't seen anything untoward.

Comments please.

Any chance we can fix this right and get a proper value in cpu_vm_mask for init_mm? All that needs to happen is that each cpu as it is started up is included in cpu_vm_mask.

The reason kexec sees this is that it is possibly the only generic modifier of init_mm.

If fixing this needs to be kexec specific we need to simply remove using init_mm.

Eric

#10 lore raw

Re: [Fastboot] Re: kexec "problem" [and patch updates]

In reply to Hariprasad Nellitheertha

On Thu, 4 Mar 2004 18:33:10 +0530 Hariprasad Nellitheertha wrote:

| Hello,
| 
| I recreated this on a UNI system running an SMP kernel as well. 
| 
| The problem is because we now initialize cpu_vm_mask for init_mm with 
| CPU_MASK_ALL (from 2.6.3 onwards) which makes all bits in cpumask 1. 
| Hence BUG_ON(!cpus_equal(cpumask,tmp) fails. The change to set 
| cpu_vm_mask to CPU_MASK_ALL was done to remove tlb flush optimizations 
| for ppc64. On UNI kernels, CPU_MASK_ALL is 1 and hence the problem 
| does not occur.
| 
| I made a small patch which fixes this problem. The change is, essentially,
| to use "tmp" instead of "cpumask". This ensures that only the (other) online 
| cpus are sent the IPI. 
| 
| I have done some testing with this patch. Kexec loads fine and I haven't seen
| anything untoward. 

Yes, that does work well... Thanks for the patch.

Is this satisfactory for pushing into the mainline kernel, or should kexec use another method to solve this problem?

--
~Randy

| Comments please.
| 
| Regards, Hari
| 
| 
| diff -Naur linux-2.6.3-before/arch/i386/kernel/smp.c linux-2.6.3/arch/i386/kernel/smp.c
| --- linux-2.6.3-before/arch/i386/kernel/smp.c	2004-02-18 09:27:15.000000000 +0530
| +++ linux-2.6.3/arch/i386/kernel/smp.c	2004-03-04 14:16:43.000000000 +0530
| @@ -356,7 +356,8 @@
|  	BUG_ON(cpus_empty(cpumask));
|  
|  	cpus_and(tmp, cpumask, cpu_online_map);
| -	BUG_ON(!cpus_equal(cpumask, tmp));
| +	if(cpus_empty(tmp))
| +		return;
|  	BUG_ON(cpu_isset(smp_processor_id(), cpumask));
|  	BUG_ON(!mm);
|  
| @@ -371,12 +372,12 @@
|  	flush_mm = mm;
|  	flush_va = va;
|  #if NR_CPUS <= BITS_PER_LONG
| -	atomic_set_mask(cpumask, &flush_cpumask);
| +	atomic_set_mask(tmp, &flush_cpumask);
|  #else
|  	{
|  		int k;
|  		unsigned long *flush_mask = (unsigned long *)&flush_cpumask;
| -		unsigned long *cpu_mask = (unsigned long *)&cpumask;
| +		unsigned long *cpu_mask = (unsigned long *)&tmp;
|  		for (k = 0; k < BITS_TO_LONGS(NR_CPUS); ++k)
|  			atomic_set_mask(cpu_mask[k], &flush_mask[k]);
|  	}
| @@ -385,7 +386,7 @@
|  	 * We have to send the IPI only to
|  	 * CPUs affected.
|  	 */
| -	send_IPI_mask(cpumask, INVALIDATE_TLB_VECTOR);
| +	send_IPI_mask(tmp, INVALIDATE_TLB_VECTOR);
|  
|  	while (!cpus_empty(flush_cpumask))
|  		/* nothing. lockup detection does not belong here */
| 
| 
| On Sat, Feb 28, 2004 at 03:41:33AM -0700, Eric W. Biederman wrote:
| > "Randy.Dunlap" <[email protected]> writes:
| > 
| > > On 27 Feb 2004 01:00:04 -0700 Eric W. Biederman wrote:
| > > 
| > > | > It works fine on 2.6.2.  It works for me on 2.6.3 if not SMP.
| > > | > If the kernel is built for SMP, when running kexec, I get a
| > > | > BUG in arch/i386/kernel/smp.c at line 359.
| > > | > I'm testing various workarounds for that BUG now.
| > > | 
| > > | I will eyeball it...
| > > | 
| > > | Is it the kernel that is shutting down, or the kernel that is being
| > > | brought up that has problems?
| > > 
| > > the kernel that is shutting down.
| > > 
| > > | The back trace from the BUG would be interesting.
| > > 
| > > see below.  my bad.  i should have included it.
| > > 
| > > | As I see it flush_tlb_others is being called when we have shutdown
| > > | cpus and the kernel still thinks we have the mm present on foreign
| > > | cpus.
| > > 
| > > Martin Bligh thinks that there is a tlb race here.
| > > I printed the 2 cpu masks on my dual-proc macine and saw
| > > 0 in one of them and 0xc in the other one.
| > 
| > Ouch we have both cpus running when this happens, and we have not
| > started any shutdown whatsoever.  This is the bit that sets up
| > the page tables for later use...
| > 
| > I think identity_map_pages will have problems with a kernel that does
| > the 4G/4G split, and it has known issues on some other architectures,
| > because they treat init_mm specially.  So the proper solution may be
| > to simply rewrite identity_map_pages. 
| > 
| > Before we do that in the short term we need to see if
| > identity_map_pages is actually doing anything bad.  You are
| > not using the 4G/4G split so that is not the cause.  So either
| > init_mm is now special in some way, or we have hit a generic kernel
| > bug.
| > 
| > So this may indeed be a tlb race.  But it is init_mm->cpu_vm_mask and
| > cpu_online map that are different.  With the implication being
| > that init_mm->cpu_vm_mask has cpus set that are not in cpu_online_map?
| > Very weird especially on SMP.
| > 
| > Without attribution I have a hard time making sense of which cpumask
| > is which so I can't draw any conclusions.  But I find it very
| > interesting that it is bits 2 and 3 that are set.  I wonder if
| > there is any mixup between logical cpu identities and apic ids.
| > 
| > Eric
| > _______________________________________________
| 
| -- 
| Hariprasad Nellitheertha
| Linux Technology Center
| India Software Labs
| IBM India, Bangalore
| -