numastats updates

DISCUSSION Stanislav Kholmanskikh 3 participants 10 messages Apr 3, 2014 – Apr 11, 2014 Memory management

Share & tools

Stanislav Kholmanskikh

author

2014-04-03 10:45 UTC #0 lore raw

Hello!

In LTP we have a test - numa01.sh (https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/numa/numa01.sh)

In general, this script does something like:
* numastat > /tmp/before
* starts a binary with the specified numa memory policy using numactl
(or a like):

    numactl --interleave=all get_some_memory_with_malloc_and_write_it

* sleep for few seconds
* numastat > /tmp/after
* compares /tmp/before and /tmp/after to check that the numa policy
was applied the right way

But the problem is that on a host with many NUMA nodes (8) the process of updating that numastats statistics takes some time. Even 10 seconds may be not enough. Therefore the test fails.

Is there a direct or indirect way to force the kernel to update the NUMA statistics?

Thank you.

Andi Kleen

2014-04-07 01:39 UTC #1 lore raw

Quoted 1 line · Stanislav Kholmanskikh

In LTP we have a test - numa01.sh (https://github.com/linux-test-project/ltp/blob/master/testcases/kernel/numa/numa01.sh)

numactl has similar tests

Quoted 11 lines

In general, this script does something like:
* numastat > /tmp/before
* starts a binary with the specified numa memory policy using
numactl (or a like):
    numactl --interleave=all get_some_memory_with_malloc_and_write_it
* sleep for few seconds
* numastat > /tmp/after
* compares /tmp/before and /tmp/after to check that the numa policy
was applied the right way

Quoted 6 lines

But the problem is that on a host with many NUMA nodes (8) the process of updating that numastats statistics takes some time. Even 10 seconds may be not enough. Therefore the test fails.
Is there a direct or indirect way to force the kernel to update the NUMA statistics?

Not currently. It depends on how much memory you have and subsequent operations. I guess would need to add one.

-Andi

Christoph Lameter

2014-04-07 15:47 UTC #2 lore raw

Quoted 12 lines · Andi Kleen

Quoted 12 lines
* starts a binary with the specified numa memory policy using
numactl (or a like):
    numactl --interleave=all get_some_memory_with_malloc_and_write_it
* sleep for few seconds
* numastat > /tmp/after
* compares /tmp/before and /tmp/after to check that the numa policy
was applied the right way
But the problem is that on a host with many NUMA nodes (8) the process of updating that numastats statistics takes some time. Even 10 seconds may be not enough. Therefore the test fails.

Quoted 5 lines

Quoted 2 lines
Is there a direct or indirect way to force the kernel to update the NUMA statistics?
Not currently. It depends on how much memory you have and subsequent operations. I guess would need to add one.

The kernel vm statistics are brought up to date with the default settings every 2 seconds.

The interval is controlled via /proc/sys/vm/stat_interval

Check the value that you have setup there.

Stanislav Kholmanskikh

author

2014-04-08 16:49 UTC #3 lore raw

Quoted 20 lines · Christoph Lameter

Quoted 12 lines · Andi Kleen
Quoted 12 lines
* starts a binary with the specified numa memory policy using
numactl (or a like):
     numactl --interleave=all get_some_memory_with_malloc_and_write_it
* sleep for few seconds
* numastat > /tmp/after
* compares /tmp/before and /tmp/after to check that the numa policy
was applied the right way
But the problem is that on a host with many NUMA nodes (8) the process of updating that numastats statistics takes some time. Even 10 seconds may be not enough. Therefore the test fails.
Quoted 5 lines
Quoted 2 lines
Is there a direct or indirect way to force the kernel to update the NUMA statistics?
Not currently. It depends on how much memory you have and subsequent operations. I guess would need to add one.

Quoted 7 lines

The kernel vm statistics are brought up to date with the default settings every 2 seconds.
The interval is controlled via /proc/sys/vm/stat_interval
Check the value that you have setup there.

Thank you, Andi, Christoph.

In my setup stat_interval is 1.

Please, look at this reproducer:
#!/bin/bash

sum_pages()

{
        local i

        ret=0
        for i in $@; do
                ret=$(( $ret + $i ))
        done
}

ret=0

for i in seq 20; do

        sum_pages $( numastat | grep interleav | cut -d ' ' -f 2-)
        val_before=$ret
        numactl --interleave=all support_numa 2
        sleep 2
        sum_pages $( numastat | grep interleav | cut -d ' ' -f 2-)
        val_after=$ret

        echo "$i: $(( $val_after - $val_before))"

done

On a two-node system it prints:
1: 294
2: 294
3: 295
4: 294
5: 294
6: 295
7: 294
8: 294
9: 293
10: 293
11: 294
12: 295
13: 296
14: 293
15: 295
16: 294
17: 295
18: 294
19: 294
20: 294

i.e. everything is ok.

But on an eight-node system:
1: 173
2: 0
3: 0
4: 173
5: 173
6: 0
7: 173
8: 173
9: 0
10: 0
11: 173
12: 0
13: 173
14: 0
15: 346
16: 0
17: 0
18: 89
19: 0
20: 173

So in general we can't rely on stat_interval value. Correct?

Christoph Lameter

2014-04-08 16:57 UTC #4 lore raw

Quoted 2 lines · Stanislav Kholmanskikh

On a two-node system it prints:
1: 294

Quoted 12 lines

20: 294
i.e. everything is ok.
But on an eight-node system:
1: 173
2: 0
3: 0
4: 173
5: 173

Quoted 1 line

So in general we can't rely on stat_interval value. Correct?

Owww... This could have something to do with the tick being disabled on idle without regard for pending vmstat updates. The processor goes to sleep and never updates the global counters. Once in awhile the processor wakes up to deal with the statistics.

The idle logic needs to check that all differentials are folded before going to slee. Frederic: Can we add such a check?

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Stanislav Kholmanskikh

author

2014-04-08 16:58 UTC #5 lore raw

In reply to Stanislav Kholmanskikh

Quoted 38 lines · Stanislav Kholmanskikh

On a two-node system it prints:
1: 294
2: 294
3: 295
4: 294
5: 294
6: 295
7: 294
8: 294
9: 293
10: 293
11: 294
12: 295
13: 296
14: 293
15: 295
16: 294
17: 295
18: 294
19: 294
20: 294
i.e. everything is ok.
But on an eight-node system:
1: 173
2: 0
3: 0
4: 173
5: 173
6: 0
7: 173
8: 173
9: 0
10: 0
11: 173
12: 0

Quoted 9 lines

13: 173
14: 0
15: 346
16: 0
17: 0
18: 89
19: 0
20: 173

Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.

Christoph Lameter

2014-04-08 18:03 UTC #6 lore raw

Quoted 1 line · Stanislav Kholmanskikh

Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.

So they are not x86? There are some counter races on !x86.

Stanislav Kholmanskikh

author

2014-04-10 05:44 UTC #7 lore raw

Quoted 5 lines · Christoph Lameter

Quoted 1 line · Stanislav Kholmanskikh
Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.
So they are not x86? There are some counter races on !x86.

Yes. This is Linux on SPARC.

Quoted 6 lines

--
To unsubscribe from this list: send the line "unsubscribe linux-numa" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html

Christoph Lameter

2014-04-11 01:48 UTC #8 lore raw

Quoted 9 lines · Stanislav Kholmanskikh

Quoted 6 lines · Christoph Lameter
Quoted 2 lines · Stanislav Kholmanskikh
Forgot to mention. This eight-node system has 8k pages, two-node - 4k pages.
So they are not x86? There are some counter races on !x86.
Yes. This is Linux on SPARC.

Hmmm... I just got the infrastructure merged to define fast counter updates. If we would define raw_cpu_ops for sparc then we could fix the issue. Sparc has atomics that can increment a register relative to a percpu base register right?

Christoph Lameter

2014-04-11 02:12 UTC #9 lore raw

Interrupts are disabled when the hit and miss statistics are incremented. SO no problem there. Weird situation.