makedumpfile 1.5.0 takes much more time to dump

Hi Atsushi san,

We tried makedumpfile 1.5.0 on a 1TB machine and it seems to regress badly. We reserved 192MB of memory and following are test results.

#1. makedumpfile-1.4.2 -E --message-level 1 -d 31
real 3m47.520s
user 0m56.543s
sys 2m41.631s

#2. makedumpfile-1.5.0 -E --message-level 1 -d 31
real 52m25.262s
user 32m51.310s
sys 18m53.265s

#3. makedumpfile-1.4.2 -c --message-level 1 -d 31
real 8m49.107s
user 4m34.180s
sys 4m8.691s

#4. makedumpfile-1.5.0 -c --message-level 1 -d 31
real 46m48.985s
user 29m35.203s
sys 16m43.149s

From: Vivek Goyal <[email protected]>
Subject: makedumpfile 1.5.0 takes much more time to dump
Date: Thu, 20 Sep 2012 16:06:34 -0400

Quoted 20 lines

Hi Atsushi san,

We tried makedumpfile 1.5.0 on a 1TB machine and it seems to regress badly. We reserved 192MB of memory and following are test results.

#1. makedumpfile-1.4.2 -E --message-level 1 -d 31
real 3m47.520s
user 0m56.543s
sys 2m41.631s

#2. makedumpfile-1.5.0 -E --message-level 1 -d 31
real 52m25.262s
user 32m51.310s
sys 18m53.265s

#3. makedumpfile-1.4.2 -c --message-level 1 -d 31
real 8m49.107s
user 4m34.180s
sys 4m8.691s

Quoted 5 lines

#4. makedumpfile-1.5.0 -c --message-level 1 -d 31
real 46m48.985s
user 29m35.203s
sys 16m43.149s

Hello Vivek,

On v1.5.0 we cannot filter free pages in constant space becuase we have yet to test it. Instead, the existing method is used here, which repeats walking on a whole page frames the number of cycles times.

As Kumagai-san explains, the number of cycles can be calculated by the following expression:

N = physical memory size / (page size * bit per byte(8) * BUFSIZE_CYCLIC)

So,

N = 2TB / (4KB * 8 * 1MB) = 64 cycles.

I guess on this environment, it took about 50 seconds to filter free pages in one cycle.

From: Vivek Goyal <[email protected]>
Subject: makedumpfile 1.5.0 takes much more time to dump
Date: Thu, 20 Sep 2012 16:06:34 -0400

Quoted 20 lines

Hi Atsushi san,

We tried makedumpfile 1.5.0 on a 1TB machine and it seems to regress badly. We reserved 192MB of memory and following are test results.

#1. makedumpfile-1.4.2 -E --message-level 1 -d 31
real 3m47.520s
user 0m56.543s
sys 2m41.631s

#2. makedumpfile-1.5.0 -E --message-level 1 -d 31
real 52m25.262s
user 32m51.310s
sys 18m53.265s

#3. makedumpfile-1.4.2 -c --message-level 1 -d 31
real 8m49.107s
user 4m34.180s
sys 4m8.691s

Quoted 5 lines

#4. makedumpfile-1.5.0 -c --message-level 1 -d 31
real 46m48.985s
user 29m35.203s
sys 16m43.149s

Hello Vivek,

On v1.5.0 we cannot filter free pages in constant space becuase we have yet to test it. Instead, the existing method is used here, which repeats walking on a whole page frames the number of cycles times.

As Kumagai-san explains, the number of cycles can be calculated by the following expression:

N = physical memory size / (page size * bit per byte(8) * BUFSIZE_CYCLIC)

So,

N = 2TB / (4KB * 8 * 1MB) = 64 cycles.

I guess on this environment, it took about 50 seconds to filter free pages in one cycle.

Quoted 43 lines

From: Vivek Goyal <[email protected]>
Subject: makedumpfile 1.5.0 takes much more time to dump
Date: Thu, 20 Sep 2012 16:06:34 -0400

Quoted 20 lines

Hi Atsushi san,

We tried makedumpfile 1.5.0 on a 1TB machine and it seems to regress badly. We reserved 192MB of memory and following are test results.

#1. makedumpfile-1.4.2 -E --message-level 1 -d 31
real 3m47.520s
user 0m56.543s
sys 2m41.631s

#2. makedumpfile-1.5.0 -E --message-level 1 -d 31
real 52m25.262s
user 32m51.310s
sys 18m53.265s

#3. makedumpfile-1.4.2 -c --message-level 1 -d 31
real 8m49.107s
user 4m34.180s
sys 4m8.691s

Quoted 5 lines

#4. makedumpfile-1.5.0 -c --message-level 1 -d 31
real 46m48.985s
user 29m35.203s
sys 16m43.149s

Hello Vivek,

On v1.5.0 we cannot filter free pages in constant space becuase we have yet to test it. Instead, the existing method is used here, which repeats walking on a whole page frames the number of cycles times.

As Kumagai-san explains, the number of cycles can be calculated by the following expression:

N = physical memory size / (page size * bit per byte(8) * BUFSIZE_CYCLIC)

So,

Quoted 4 lines

N = 2TB / (4KB * 8 * 1MB) = 64 cycles.

I guess on this environment, it took about 50 seconds to filter free pages in one cycle.

Ok, so once we have your walking through page struct patches in, hopefully this problem will be gone?

If that's going to take time, can we make using of new logic conditional on a command line option. So that user has the option of using old logic.

From: Vivek Goyal <[email protected]>
Subject: Re: makedumpfile 1.5.0 takes much more time to dump
Date: Fri, 21 Sep 2012 09:32:32 -0400

Quoted 44 lines · HATAYAMA Daisuke

Quoted 43 lines

From: Vivek Goyal <[email protected]>
Subject: makedumpfile 1.5.0 takes much more time to dump
Date: Thu, 20 Sep 2012 16:06:34 -0400

Quoted 20 lines

Hi Atsushi san,

We tried makedumpfile 1.5.0 on a 1TB machine and it seems to regress badly. We reserved 192MB of memory and following are test results.

#1. makedumpfile-1.4.2 -E --message-level 1 -d 31
real 3m47.520s
user 0m56.543s
sys 2m41.631s

#2. makedumpfile-1.5.0 -E --message-level 1 -d 31
real 52m25.262s
user 32m51.310s
sys 18m53.265s

#3. makedumpfile-1.4.2 -c --message-level 1 -d 31
real 8m49.107s
user 4m34.180s
sys 4m8.691s

Quoted 5 lines

#4. makedumpfile-1.5.0 -c --message-level 1 -d 31
real 46m48.985s
user 29m35.203s
sys 16m43.149s

Hello Vivek,

On v1.5.0 we cannot filter free pages in constant space becuase we have yet to test it. Instead, the existing method is used here, which repeats walking on a whole page frames the number of cycles times.

As Kumagai-san explains, the number of cycles can be calculated by the following expression:

N = physical memory size / (page size * bit per byte(8) * BUFSIZE_CYCLIC)

So,

Quoted 8 lines

Quoted 4 lines

N = 2TB / (4KB * 8 * 1MB) = 64 cycles.

I guess on this environment, it took about 50 seconds to filter free pages in one cycle.

Ok, so once we have your walking through page struct patches in, hopefully this problem will be gone?

Yes, then free page filtering is integrated in other mem_map array logics. You can see how it takes from the following message.

Excluding unnecessary pages : [100 %] STEP [Excluding unnecessary pages] : 0.204574 seconds

New logic takes equal to or quicker than the total time indicated by the above kind of messages now.

Quoted 4 lines

If that's going to take time, can we make using of new logic conditional on a command line option. So that user has the option of using old logic.

Kumagai-san should decide this.

Quoted 39 lines

Quoted 39 lines

Quoted 20 lines

Hi Atsushi san,

We tried makedumpfile 1.5.0 on a 1TB machine and it seems to regress badly. We reserved 192MB of memory and following are test results.

#1. makedumpfile-1.4.2 -E --message-level 1 -d 31
real 3m47.520s
user 0m56.543s
sys 2m41.631s

#2. makedumpfile-1.5.0 -E --message-level 1 -d 31
real 52m25.262s
user 32m51.310s
sys 18m53.265s

#3. makedumpfile-1.4.2 -c --message-level 1 -d 31
real 8m49.107s
user 4m34.180s
sys 4m8.691s

Quoted 5 lines

#4. makedumpfile-1.5.0 -c --message-level 1 -d 31
real 46m48.985s
user 29m35.203s
sys 16m43.149s

Hello Vivek,

On v1.5.0 we cannot filter free pages in constant space becuase we have yet to test it. Instead, the existing method is used here, which repeats walking on a whole page frames the number of cycles times.

As Kumagai-san explains, the number of cycles can be calculated by the following expression:

N = physical memory size / (page size * bit per byte(8) * BUFSIZE_CYCLIC)

So,

Quoted 7 lines

Quoted 4 lines

N = 2TB / (4KB * 8 * 1MB) = 64 cycles.

I guess on this environment, it took about 50 seconds to filter free pages in one cycle.

Ok, so once we have your walking through page struct patches in, hopefully this problem will be gone?

Yes, then free page filtering is integrated in other mem_map array logics. You can see how it takes from the following message.

Excluding unnecessary pages : [100 %] STEP [Excluding unnecessary pages] : 0.204574 seconds

New logic takes equal to or quicker than the total time indicated by the above kind of messages now.

Quoted 4 lines

If that's going to take time, can we make using of new logic conditional on a command line option. So that user has the option of using old logic.

Kumagai-san should decide this.

BTW, Kumagai-san, in what version do you plan release of this new mem_map logic? Around v1.5.2? I want to re-send a new patch set to which I add small changes and comments for maintainability.

Quoted 6 lines

Quoted 4 lines

If that's going to take time, can we make using of new logic conditional on a command line option. So that user has the option of using old logic.

Kumagai-san should decide this.

I'm not planning to make the cyclic mode optional.
However, I think the performance issue should be improved even without mem_map array logic.

So, I will try to reduce the number of cycles as few times as possible for v1.5.1, the performance issue will be improved.
To make sure of it, would you re-test with --cyclic-buffer 32768 (32MB), Vivek ? Then the result of v1.5.0 is still too bad, I will consider using the old logic as default logic.

Quoted 3 lines

BTW, Kumagai-san, in what version do you plan release of this new mem_map logic? Around v1.5.2? I want to re-send a new patch set to which I add small changes and comments for maintainability.

Yes, I plan to merge it in v1.5.2.
Would you re-send the new patch set based on v1.5.1 ?

Additionally, I'm afraid that I haven't reviewed the most of patches for v1.5.1 yet. So I can't tell you when v1.5.2 will be released.

[..]

Quoted 11 lines

Quoted 6 lines

Quoted 4 lines

If that's going to take time, can we make using of new logic conditional on a command line option. So that user has the option of using old logic.

Kumagai-san should decide this.

I'm not planning to make the cyclic mode optional.
However, I think the performance issue should be improved even without mem_map array logic.

Quoted 5 lines

So, I will try to reduce the number of cycles as few times as possible for v1.5.1, the performance issue will be improved.
To make sure of it, would you re-test with --cyclic-buffer 32768 (32MB), Vivek ? Then the result of v1.5.0 is still too bad, I will consider using the old logic as default logic.

Actually chaowang did the testing. In the bug he provided data for 16MB buffer.

makedumpfile with 16M cyclic buffer,
#1. makedumpfile-1.5.0 -c --message-level 1 -d 31 --cyclic-buffer 16384

real     12m51.886s
user     6m30.710s
sys      6m11.642s
#2. makedumpfile-1.5.0 -E --message-level 1 -d 31 --cyclic-buffer 16384
real     11m24.141s
user     4m25.897s
sys      6m38.116s

Which looks much better than default numbers. Chao, can you please do the testing with 32MB buffer size and provide the data here.