Question: new bind_zonelist uses only one zone type

KAMEZAWA Hiroyuki

author

2006-01-17 08:46 UTC #0 lore raw

in -mm4 (in linus.patch)
==
static struct zonelist *bind_zonelist(nodemask_t *nodes)

{
        struct zonelist *zl;
        int num, max, nd;

        max = 1 + MAX_NR_ZONES * nodes_weight(*nodes);
        zl = kmalloc(sizeof(void *) * max, GFP_KERNEL);
        if (!zl)
                return NULL;
        num = 0;
        for_each_node_mask(nd, *nodes)
                zl->zones[num++] = &NODE_DATA(nd)->node_zones[policy_zone];
        zl->zones[num] = NULL;
        return zl;
}

==
policy_zone is ZONE_DMA, ZONE_NORMAL, ZONE_HIGHMEM, depends on system.

If policy_zone is ZONE_NORMAL, returned zonelist will be {Node(0)'s NORMAL, Node(1)'s NORMAL, Node(2)'s Normal.....}

If node0 has only DMA/DMA32 and Node1-NodeX has Normal, node0 will be ignored and zonelist will include not-populated zone.

Is this intended ?

-- Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andi Kleen

2006-01-17 14:29 UTC #1 lore raw

Quoted 9 lines · KAMEZAWA Hiroyuki

policy_zone is ZONE_DMA, ZONE_NORMAL, ZONE_HIGHMEM, depends on system.
If policy_zone is ZONE_NORMAL, returned zonelist will be {Node(0)'s NORMAL, Node(1)'s NORMAL, Node(2)'s Normal.....}
If node0 has only DMA/DMA32 and Node1-NodeX has Normal, node0 will be ignored and zonelist will include not-populated zone.
Is this intended ?

I was wondering when someone else would notice. Congratulations, you are the first ;-)

It was originally intended - back then either IA64 NUMA systems didn't have a ZONE_DMA and on x86-64 it was only 16MB and for i386 NUMA it was considered acceptable - and it made the code simpler and policies use less memory. But is now considered a bug because of the introduction of ZONE_DMA32 on x86-64 and I gather from your report your platform has NUMA and a 4GB ZONE_DMA too?

It is on my todo list to fix, but I haven't gotten around to it yet.

Fixing it will unfortunately increase the footprint of the policy structures, so likely it would only increase to two. If someone beats me to a patch that would be ok too.

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

KAMEZAWA Hiroyuki

author

2006-01-17 23:47 UTC #2 lore raw

Quoted 5 lines · Andi Kleen

It was originally intended - back then either IA64 NUMA systems didn't
have a ZONE_DMA and on x86-64 it was only 16MB and for i386 NUMA
it was considered acceptable - and it made the code simpler and policies use less memory. But is now considered a bug because of the introduction of ZONE_DMA32 on x86-64 and I gather from your report your platform has NUMA and a 4GB ZONE_DMA too?

on ia64, 0-4G area is ZONE_DMA.

Quoted 5 lines

It is on my todo list to fix, but I haven't gotten around to it yet.
Fixing it will unfortunately increase the footprint of the policy structures, so likely it would only increase to two. If someone beats me to a patch that would be ok too.

I don't have real problem now. It just looks curious.
And, anyway, mbind's list doesn't guarantee fair allocation among nodes.

Thanks,
-- Kame

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andi Kleen

2006-01-18 03:00 UTC #3 lore raw

Quoted 9 lines · KAMEZAWA Hiroyuki

Quoted 6 lines · Andi Kleen
It was originally intended - back then either IA64 NUMA systems didn't have a ZONE_DMA and on x86-64 it was only 16MB and for i386 NUMA it was considered acceptable - and it made the code simpler and policies use less memory. But is now considered a bug because of the introduction of ZONE_DMA32 on x86-64 and I gather from your report your platform has NUMA and a 4GB ZONE_DMA too?
on ia64, 0-4G area is ZONE_DMA.

On IA64/SN2 ZONE_DMA is empty and at least in the part SGI was the only IA64 vendor actively interested in NUMA policy.

I assume you have a NUMA platform too. Do your machines have a contiguous memory map where there could be one or more nodes which only have ZONE_DMA?

Quoted 7 lines

Quoted 5 lines
It is on my todo list to fix, but I haven't gotten around to it yet.
Fixing it will unfortunately increase the footprint of the policy structures, so likely it would only increase to two. If someone beats me to a patch that would be ok too.
I don't have real problem now. It just looks curious.

Well it's a bit nasty to not be able to policy 4GB of your memory. Maybe if you have a few TB of it you won't care, but on smaller machines it likely will make a difference.

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

KAMEZAWA Hiroyuki

author

2006-01-18 03:26 UTC #4 lore raw

Quoted 7 lines · Andi Kleen

On IA64/SN2 ZONE_DMA is empty and at least in the part SGI was the only IA64 vendor actively interested in NUMA policy.
I assume you have a NUMA platform too. Do your machines have a contiguous memory map where there could be one or more nodes which only have ZONE_DMA?

Fujitsu's PrimeQuest is NUMA and has memory in 0-4G areas in node 0. It depends on installed memory whether node 0 contains only ZONE_DMA or not.

When using SPARSEMEM, it uses NUMA config. This means one-node-NUMA. So, if I use SPARSEMEM on ia64 SMP machine with 5 Gbytes mem, I can allocate just 1G bytes on mbind area.
(*)Maybe using mempolicy on one-node-NUMA make no sense.

Quoted 11 lines

Quoted 6 lines
Quoted 5 lines
It is on my todo list to fix, but I haven't gotten around to it yet.
Fixing it will unfortunately increase the footprint of the policy structures, so likely it would only increase to two. If someone beats me to a patch that would be ok too.
I don't have real problem now. It just looks curious.
Well it's a bit nasty to not be able to policy 4GB of your memory. Maybe if you have a few TB of it you won't care, but on smaller machines it likely will make a difference.

It makes difference on my *numa emulation* environment, now ;) But it's just emulation.

-- Kame.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Jack Steiner

2006-01-18 03:40 UTC #5 lore raw

In reply to Andi Kleen

Quoted 9 lines · Andi Kleen

Quoted 8 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · Andi Kleen
It was originally intended - back then either IA64 NUMA systems didn't have a ZONE_DMA and on x86-64 it was only 16MB and for i386 NUMA it was considered acceptable - and it made the code simpler and policies use less memory. But is now considered a bug because of the introduction of ZONE_DMA32 on x86-64 and I gather from your report your platform has NUMA and a 4GB ZONE_DMA too?

Quoted 5 lines

Quoted 1 line
on ia64, 0-4G area is ZONE_DMA.
On IA64/SN2 ZONE_DMA is empty and at least in the part SGI was the only IA64 vendor actively interested in NUMA policy.

On the SN systems, ALL memory is in the DMA zone. The other zones are empty.

I think this is SN-specific - other IA64 platforms may be different & have memory in multiple zones.

Signature · 7 lines

Jack

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to [email protected].  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>

Andi Kleen

2006-01-18 03:49 UTC #6 lore raw

Quoted 10 lines · Jack Steiner

Quoted 9 lines · Andi Kleen
Quoted 8 lines · KAMEZAWA Hiroyuki
Quoted 6 lines · Andi Kleen
It was originally intended - back then either IA64 NUMA systems didn't have a ZONE_DMA and on x86-64 it was only 16MB and for i386 NUMA it was considered acceptable - and it made the code simpler and policies use less memory. But is now considered a bug because of the introduction of ZONE_DMA32 on x86-64 and I gather from your report your platform has NUMA and a 4GB ZONE_DMA too?

Quoted 7 lines

Quoted 5 lines
Quoted 1 line
on ia64, 0-4G area is ZONE_DMA.
On IA64/SN2 ZONE_DMA is empty and at least in the part SGI was the only IA64 vendor actively interested in NUMA policy.
On the SN systems, ALL memory is in the DMA zone. The other zones are empty.

Ah sorry Jack got that wrong. Thanks for the correction. For the NUMA policy makes no difference though because only the highest zone with pages populated counts.

Quoted 2 lines

I think this is SN-specific - other IA64 platforms may be different & have memory in multiple zones.

Yes, it is.

-Andi

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in the body to [email protected]. For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"[email protected]"> [email protected] </a>