Skip to contentSkip to conversation

Which of the virtualization approaches is more suitable for kernel?

DISCUSSION Kirill Korotaev 7 participants 27 messages Feb 20, 2006 – Mar 10, 2006

Subsystems (3)
lore
Share & tools
lore
#0 lore raw

Linus, Andrew,

We need your help on what virtualization approach you would accept to mainstream (if any) and where we should go.

If to drop VPID virtualization which caused many disputes, we actually have the one virtualization solution, but 2 approaches for it. Which one will go depends on the goals and your approval any way.

So what are the approaches?

1. namespaces for all types of resources (Eric W. Biederman)

The proposed solution is similar to fs namespaces. This approach introduces namespaces for IPC, IPv4 networking, IPv6 networking, pids etc. It also adds to task_struct a set of pointers to namespaces which task belongs to, i.e. current->ipv4_ns, current->ipc_ns etc.

Benefits:
- fine grained namespaces
- task_struct points directly to a structure describing the namespace and it's data (not to container)
- maybe a bit more logical/clean implementation, since no effective container is involved unlike a second approach.

Disadvantages:
- it is only proof of concept code right now. nothing more.
- such an approach requires adding of additional argument to many functions (e.g. Eric's patch for networking is 1.5 bigger than openvz). it also adds code for getting namespace from objects. e.g. skb->sk->namespace.
- so it increases stack usage
- it can't efficiently compile to the same not virtualized kernel, which can be undesired for embedded linux.
- fine grained namespaces are actually an obfuscation, since kernel subsystems are tightly interconnected. e.g. network -> sysctl -> proc, mqueues -> netlink, ipc -> fs and most often can be used only as a whole container.
- it involves a bit more complicated procedure of a container create/enter which requires exec or something like this, since there is no effective container which could be simply triggered.

2. containers (OpenVZ.org/linux-vserver.org)

Container solution was discussed before, and actually it is also namespace solution, but as a whole total namespace, with a single kernel structure describing it.
Every task has two cotnainer pointers: container and effective container. The later is used to temporarily switch to other contexts, e.g. when handling IRQs, TCP/IP etc.

Benefits:
- clear logical bounded container, it is clear when container is alive and when not.
- it doesn't introduce additional args for most functions, no additional stack usage.
- it compiles to old good kernel when virtualization if off, so doesn't disturb other configurations.
- Eric brought an interesting idea about introducing interface like DEFINE_CPU_VAR(), which could potentially allow to create virtualized variables automagically and access them via econtainer().
- mature working code exists which is used in production for years, so first working version can be done much quicker

Disadvantages:
- one additional pointer dereference when accessing virtualized resources, e.g. current->econtainer->shm_ids

Kirill

#1 lore raw
Quoted 25 lines · Kirill Korotaev

Linus, Andrew,

We need your help on what virtualization approach you would accept to mainstream (if any) and where we should go.

If to drop VPID virtualization which caused many disputes, we actually have the one virtualization solution, but 2 approaches for it. Which one will go depends on the goals and your approval any way.

So what are the approaches?

1. namespaces for all types of resources (Eric W. Biederman)

The proposed solution is similar to fs namespaces. This approach introduces namespaces for IPC, IPv4 networking, IPv6 networking, pids etc. It also adds to task_struct a set of pointers to namespaces which task belongs to, i.e. current->ipv4_ns, current->ipc_ns etc.

Benefits:
- fine grained namespaces
- task_struct points directly to a structure describing the namespace and it's data (not to container)
- maybe a bit more logical/clean implementation, since no effective container is involved unlike a second approach.

Quoted 2 lines

Disadvantages:
- it is only proof of concept code right now. nothing more.

sorry, but that is no disadvantage in my book ...

Quoted 2 lines

- such an approach requires adding of additional argument to many functions (e.g. Eric's patch for networking is 1.5 bigger than openvz).

hmm? last time I checked OpenVZ was quite bloated compared to Linux-VServer, and Eric's network part isn't even there yet ...

Quoted 7 lines

it also adds code for getting namespace from objects. e.g. skb->sk->namespace.

- so it increases stack usage

- it can't efficiently compile to the same not virtualized kernel, which can be undesired for embedded linux.

while OpenVZ does?

Quoted 4 lines

- fine grained namespaces are actually an obfuscation, since kernel subsystems are tightly interconnected. e.g. network -> sysctl -> proc, mqueues -> netlink, ipc -> fs and most often can be used only as a whole container.

I think a lot of _strange_ interconnects there could use some cleanup, and after that the interconenctions would be very small

Quoted 3 lines

- it involves a bit more complicated procedure of a container create/enter which requires exec or something like this, since there is no effective container which could be simply triggered.

I don't understand this argument ...

Quoted 1 line

2. containers (OpenVZ.org/linux-vserver.org)

please do not generalize here, Linux-VServer does not use a single container structure as you might think ...

Quoted 3 lines

Container solution was discussed before, and actually it is also namespace solution, but as a whole total namespace, with a single kernel structure describing it.

that might be true for OpenVZ, but it is not for Linux-VServer, as we have structures for network and process contexts as well as different ones for disk limits

Quoted 3 lines

Every task has two cotnainer pointers: container and effective container. The later is used to temporarily switch to other contexts, e.g. when handling IRQs, TCP/IP etc.

this doesn't look very cool to me, as IRQs should be handled in the host context and TCP/IP in the proper network space ...

Quoted 3 lines

Benefits:
- clear logical bounded container, it is clear when container is alive and when not.

how does that handle the issues you described with sockets in wait state which have very long timeouts?

Quoted 2 lines

- it doesn't introduce additional args for most functions, no additional stack usage.

a single additional arg here and there won't hurt, and I'm pretty sure most of them will be in inlined code, where it doesn't really matter

Quoted 2 lines

- it compiles to old good kernel when virtualization if off, so doesn't disturb other configurations.

the question here is, do we really want to turn it off at all? IMHO the design and implementation should be sufficiently good so that it does neither impose unnecessary overhead nor change the default behaviour ...

Quoted 3 lines

- Eric brought an interesting idea about introducing interface like DEFINE_CPU_VAR(), which could potentially allow to create virtualized variables automagically and access them via econtainer().

how is that an advantage of the container approach?

Quoted 2 lines

- mature working code exists which is used in production for years, so first working version can be done much quicker

from the OpenVZ/Virtuozzo(tm) page:

Specific benefits of Virtuozzo(tm) compared to OpenVZ can be found below:

- Higher VPS density. Virtuozzo(tm) provides efficient memory and file sharing mechanisms enabling higher VPS density and better performance of VPSs.

- Improved Stability, Scalability, and Performance. Virtuozzo(tm) is designed to run 24×7 environments with production workloads on hosts with up-to 32 CPUs.

so I conclude, OpenVZ does not contain the code which provides all this ...

Quoted 3 lines

Disadvantages:
- one additional pointer dereference when accessing virtualized resources, e.g. current->econtainer->shm_ids

best,
Herbert

Quoted 1 line

Kirill

#2 lore raw
Quoted 5 lines
Quoted 2 lines

- such an approach requires adding of additional argument to many functions (e.g. Eric's patch for networking is 1.5 bigger than openvz).

hmm? last time I checked OpenVZ was quite bloated compared to Linux-VServer, and Eric's network part isn't even there yet ...

This is rather subjective feeling.
I can tell the same about VServer.

Quoted 3 lines
Quoted 2 lines

- it can't efficiently compile to the same not virtualized kernel, which can be undesired for embedded linux.

while OpenVZ does?

yes. In _most_ cases does.
If Linus/Andrew/others feel this is not an issues at all I will be the first who will greet Eric's approach. I'm not against it, though it looks as a disadvantage for me.

Quoted 7 lines
Quoted 4 lines

- fine grained namespaces are actually an obfuscation, since kernel subsystems are tightly interconnected. e.g. network -> sysctl -> proc, mqueues -> netlink, ipc -> fs and most often can be used only as a whole container.

I think a lot of _strange_ interconnects there could use some cleanup, and after that the interconenctions would be very small

Why do you think they are strange!? Is it strange that networking exports it's sysctls and statictics via proc? Is it strange for you that IPC uses fs?
It is by _design_.

Quoted 4 lines
Quoted 3 lines

- it involves a bit more complicated procedure of a container create/enter which requires exec or something like this, since there is no effective container which could be simply triggered.

I don't understand this argument ...

- you need to track dependencies between namespaces (e.g. NAT requires conntracks, IPC requires FS etc.). this should be handled, otherwise one container being able to create nested container will be able to make oops.

Quoted 5 lines
Quoted 1 line

2. containers (OpenVZ.org/linux-vserver.org)

please do not generalize here, Linux-VServer does not use a single container structure as you might think ...

1.
This topic is not a question of single container only... also AFAIK you use it altogether only.
2.
Just to be clear: once again, I'm not against namespaces.

Quoted 7 lines
Quoted 2 lines

Container solution was discussed before, and actually it is also
namespace solution, but as a whole total namespace, with a single kernel structure describing it.

that might be true for OpenVZ, but it is not for Linux-VServer, as we have structures for network and process contexts as well as different ones for disk limits

do you have support for it in tools?
i.e. do you support namespaces somehow? can you create half virtualized container?

Quoted 7 lines
Quoted 2 lines

Every task has two cotnainer pointers: container and effective
container. The later is used to temporarily switch to other contexts, e.g. when handling IRQs, TCP/IP etc.

this doesn't look very cool to me, as IRQs should be handled in the host context and TCP/IP in the proper network space ...

this is exactly what it does.
on IRQ context is switched to host.
In TCP/IP to context of socket or network device.

Quoted 5 lines
Quoted 3 lines

Benefits:
- clear logical bounded container, it is clear when container is alive and when not.

how does that handle the issues you described with sockets in wait state which have very long timeouts?

easily.
we have clear logic of container lifetime - it is alive until last process is alive in it. When processes die, container is destroy and so does all it's sockets. from namespaces point of view, this means that lifetime of network namespace is limited to lifetime of pid_namespace.

Quoted 5 lines
Quoted 2 lines

- it doesn't introduce additional args for most functions, no additional stack usage.

a single additional arg here and there won't hurt, and I'm pretty sure most of them will be in inlined code, where it doesn't really matter

have you analyzed that before thinking about inlining?

Quoted 5 lines
Quoted 2 lines

- it compiles to old good kernel when virtualization if off, so doesn't disturb other configurations.

the question here is, do we really want to turn it off at all? IMHO the design and implementation should be sufficiently good so that it does neither impose unnecessary overhead nor change the default
behaviour ...

this is the question I want to get from Linus/Andrew.
I don't believe in low overhead. It starts from virtualization, then goes reource management etc.
These features _definetely_ introduce overhead and increase resource consumption. Not big, but why not configurable? You don't need CPUSETS on UP i386 machine, do you? Why may I want this stuff in my embedded Linux? The same for secured Linux distributions, it only opens the ways for possible security issues.

Quoted 4 lines
Quoted 3 lines

- Eric brought an interesting idea about introducing interface like DEFINE_CPU_VAR(), which could potentially allow to create virtualized variables automagically and access them via econtainer().

how is that an advantage of the container approach?

Such vars can automatically be defined to something like "(econtainer()->virtualized_variable)". This looks similar to percpu variable interfaces.

Quoted 13 lines
Quoted 2 lines

- mature working code exists which is used in production for years, so first working version can be done much quicker

from the OpenVZ/Virtuozzo(tm) page:
Specific benefits of Virtuozzo(tm) compared to OpenVZ can be found below:
- Higher VPS density. Virtuozzo(tm) provides efficient memory and file sharing mechanisms enabling higher VPS density and better performance of VPSs.
- Improved Stability, Scalability, and Performance. Virtuozzo(tm) is designed to run 24×7 environments with production workloads on hosts with up-to 32 CPUs.
so I conclude, OpenVZ does not contain the code which provides all this ..

:))))
Doesn't provide what? Stability?
Q/A process used for Virtuozzo end up in OpenVZ code eventually as well. This is more subject of support/QA. Performance? we optimize systems for customers, have HA/monitoring/tuning/management tools for it etc.

Seems, you are just trying to move from the topic. Great.

Thanks,
Kirill

#3 lore raw
Quoted 10 lines · Kirill Korotaev
Quoted 7 lines
Quoted 4 lines

- fine grained namespaces are actually an obfuscation, since kernel subsystems are tightly interconnected. e.g. network -> sysctl -> proc, mqueues -> netlink, ipc -> fs and most often can be used only as a whole container.

I think a lot of _strange_ interconnects there could use some cleanup, and after that the interconenctions would be very small

Why do you think they are strange!? Is it strange that networking exports it's sysctls and statictics via proc? Is it strange for you that IPC uses fs?
It is by _design_.

Great, and this kind of simple design also worked well for the first few iterations of Linux-VServer. However, some people need more flexibility as we are seeing by the wide range of virtualisation schemes being proposed. In the 2.1.x VServer patch the network and (process&IPC) isolation and virtualisation have been kept seperate, and can be managed with seperate utilities. There is also a syscall and utility to manage the existing kernel filesystem namespaces.

Eric's pspace work keeps the PID aspect seperate too, which I never envisioned possible.

I think that if we can keep as much seperation between systems as possible, then we will have a cleaner design. Also it will make life easier for the core team as we can more easily divide up the patches for consideration by the relevant subsystem maintainer.

Quoted 2 lines

- you need to track dependencies between namespaces (e.g. NAT requires
conntracks, IPC requires FS etc.). this should be handled, otherwise one container being able to create nested container will be able to make oops.

This is just normal refcounting. Yes, IPC requires filesystem code, but it doesn't care about the VFS, which is what filesystem namespaces abstract.

Quoted 3 lines

do you have support for it in tools?
i.e. do you support namespaces somehow? can you create half virtualized container?

See the util-vserver package, it comes with chbind and vnamespace which allow creation of 'half-virtualized' containers, though most of the rest of the functionality, such as per-vserver ulimits, disklimits, etc have been shoehorned into the general vx_info structure. As we merge into the mainstream we can review each of these decisions and decide whether it is an inherantly per-process decision, or more XX_info structures are warranted.

Quoted 6 lines
Quoted 3 lines

this doesn't look very cool to me, as IRQs should
be handled in the host context and TCP/IP in the
proper network space ...

this is exactly what it does.
on IRQ context is switched to host.
In TCP/IP to context of socket or network device.

That sounds like an interesting innovation, and we can compare our patches in this space once we have some common terms of reference and starting points.

Quoted 6 lines
Quoted 3 lines

the question here is, do we really want to turn it off at all? IMHO the design and implementation should be sufficiently good so that it does neither impose unnecessary overhead nor change the default
behaviour ...

this is the question I want to get from Linus/Andrew.
I don't believe in low overhead. It starts from virtualization, then goes reource management etc. These features _definetely_ introduce overhead and increase resource consumption. Not big, but why not configurable?

Obviously, our projects have different goals; Linux-VServer has very little performance overhead. Special provisions are made to achieve scalability on SMP and to avoid unnecessary cacheline issues. Once that is sorted out, it's very hard to measure any performance overhead of it, especially when the task_struct->vx_info pointer is null.

However I see nothing wrong with making all code disappear without the kernel config option enabled. I expect that as time goes on, you'd just as soon disable it as you would disable the open() system call. I think that's what Herbert was getting at with his comment.

Quoted 1 line

Seems, you are just trying to move from the topic. Great.

I always did want to be a Lumberjack!

Sam.

#4 lore raw
In reply to Kirill Korotaev
Quoted 6 lines · Kirill Korotaev
Quoted 5 lines
Quoted 2 lines

- such an approach requires adding of additional argument to many functions (e.g. Eric's patch for networking is 1.5 bigger than openvz).

hmm? last time I checked OpenVZ was quite bloated compared to Linux-VServer, and Eric's network part isn't even there yet ...

This is rather subjective feeling.

of course, of course ...

OpenVZ stable patches:

	1857829 patch-022stab032-core
	1886915 patch-022stab034-core
	7390511 patch-022stab045-combined
	7570326 patch-022stab050-combined
	8042889 patch-022stab056-combined
	8059201 patch-022stab064-combined

Linux-VServer stable releases:

	 100130 patch-2.4.20-vs1.00.diff
	 135068 patch-2.4.21-vs1.20.diff
	 587170 patch-2.6.12.4-vs2.0.diff
	 593052 patch-2.6.14.3-vs2.01.diff
	 619268 patch-2.6.15.4-vs2.0.2-rc6.diff
Quoted 1 line

I can tell the same about VServer.

really?

Quoted 43 lines
Quoted 3 lines
Quoted 2 lines

- it can't efficiently compile to the same not virtualized kernel, which can be undesired for embedded linux.

while OpenVZ does?

yes. In _most_ cases does.
If Linus/Andrew/others feel this is not an issues at all I will be the first who will greet Eric's approach. I'm not against it, though it looks as a disadvantage for me.

Quoted 7 lines
Quoted 4 lines

- fine grained namespaces are actually an obfuscation, since kernel subsystems are tightly interconnected. e.g. network -> sysctl -> proc, mqueues -> netlink, ipc -> fs and most often can be used only as a whole container.

I think a lot of _strange_ interconnects there could use some cleanup, and after that the interconenctions would be very small

Why do you think they are strange!? Is it strange that networking exports it's sysctls and statictics via proc?
Is it strange for you that IPC uses fs?
It is by _design_.

Quoted 4 lines
Quoted 3 lines

- it involves a bit more complicated procedure of a container create/enter which requires exec or something like this, since there is no effective container which could be simply triggered.

I don't understand this argument ...

- you need to track dependencies between namespaces (e.g. NAT requires conntracks, IPC requires FS etc.). this should be handled, otherwise one container being able to create nested container will be able to make oops.

Quoted 5 lines
Quoted 1 line

2. containers (OpenVZ.org/linux-vserver.org)

please do not generalize here, Linux-VServer does not use a single container structure as you might think ...

1.
This topic is not a question of single container only... also AFAIK you use it altogether only.
2.
Just to be clear: once again, I'm not against namespaces.

Quoted 4 lines
Quoted 3 lines

Container solution was discussed before, and actually it is also namespace solution, but as a whole total namespace, with a single kernel structure describing it.

Quoted 7 lines
Quoted 4 lines

that might be true for OpenVZ, but it is not for Linux-VServer, as we have structures for network and process contexts as well as different ones for disk limits

do you have support for it in tools?
i.e. do you support namespaces somehow? can you create half virtualized container?

sure, just get the tools and use vnamespace or if you prefer chcontext or chbind ...

Quoted 37 lines
Quoted 8 lines
Quoted 3 lines

Every task has two cotnainer pointers: container and effective container. The later is used to temporarily switch to other contexts, e.g. when handling IRQs, TCP/IP etc.

this doesn't look very cool to me, as IRQs should be handled in the host context and TCP/IP in the proper network space ...

this is exactly what it does.
on IRQ context is switched to host.
In TCP/IP to context of socket or network device.

Quoted 5 lines
Quoted 3 lines

Benefits:
- clear logical bounded container, it is clear when container is alive and when not.

how does that handle the issues you described with sockets in wait state which have very long timeouts?

easily.
we have clear logic of container lifetime - it is alive until last process is alive in it. When processes die, container is destroy and so does all it's sockets. from namespaces point of view, this means that lifetime of network namespace is limited to lifetime of pid_namespace.

Quoted 5 lines
Quoted 2 lines

- it doesn't introduce additional args for most functions, no additional stack usage.

a single additional arg here and there won't hurt, and I'm pretty sure most of them will be in inlined code, where it doesn't really matter

have you analyzed that before thinking about inlining?

Quoted 7 lines
Quoted 2 lines

- it compiles to old good kernel when virtualization if off, so doesn't disturb other configurations.

the question here is, do we really want to turn it off at all? IMHO the design and implementation should be sufficiently good so that it does neither impose unnecessary overhead nor change the default behaviour ...

Quoted 8 lines

this is the question I want to get from Linus/Andrew.
I don't believe in low overhead. It starts from virtualization, then goes reource management etc.
These features _definetely_ introduce overhead and increase resource consumption. Not big, but why not configurable?
You don't need CPUSETS on UP i386 machine, do you? Why may I want this stuff in my embedded Linux? The same for secured Linux distributions, it only opens the ways for possible security issues.

ah, you got me wrong there, of course embedded systems have other requirements, and it might turn out that some of those virtualizations require config options to disable them ...

but, I do not see a measurable overhead there and I do not consider it a problem to disable certain more expensive parts ...

Quoted 21 lines
Quoted 4 lines
Quoted 3 lines

- Eric brought an interesting idea about introducing interface like DEFINE_CPU_VAR(), which could potentially allow to create virtualized variables automagically and access them via econtainer().

how is that an advantage of the container approach?

Such vars can automatically be defined to something like "(econtainer()->virtualized_variable)".
This looks similar to percpu variable interfaces.

Quoted 13 lines
Quoted 2 lines

- mature working code exists which is used in production for years, so first working version can be done much quicker

from the OpenVZ/Virtuozzo(tm) page:
Specific benefits of Virtuozzo(tm) compared to OpenVZ can be found below:
- Higher VPS density. Virtuozzo(tm) provides efficient memory and file sharing mechanisms enabling higher VPS density and better performance of VPSs.
- Improved Stability, Scalability, and Performance. Virtuozzo(tm) is designed to run 24×7 environments with production workloads on hosts with up-to 32 CPUs.
so I conclude, OpenVZ does not contain the code which provides all this ..

Quoted 8 lines

:))))
Doesn't provide what? Stability?
Q/A process used for Virtuozzo end up in OpenVZ code eventually as well. This is more subject of support/QA.
Performance? we optimize systems for customers, have HA/monitoring/tuning/management tools for it etc.

Seems, you are just trying to move from the topic. Great.

I guess I was right on topic ...

best,
Herbert

Quoted 2 lines

Thanks,
Kirill

#5 lore raw
Quoted 26 lines · Herbert Poetzl
Quoted 4 lines · Kirill Korotaev
Quoted 4 lines
Quoted 4 lines

- such an approach requires adding of additional argument to many functions (e.g. Eric's patch for networking is 1.5 bigger than openvz).

Quoted 8 lines
Quoted 5 lines

hmm? last time I checked OpenVZ was quite bloated compared to Linux-VServer, and Eric's network part isn't even there yet ...

This is rather subjective feeling.

of course, of course ...

OpenVZ stable patches:

	1857829 patch-022stab032-core
	1886915 patch-022stab034-core
	7390511 patch-022stab045-combined
	7570326 patch-022stab050-combined
	8042889 patch-022stab056-combined
	8059201 patch-022stab064-combined
Quoted 8 lines

Linux-VServer stable releases:

	 100130 patch-2.4.20-vs1.00.diff
	 135068 patch-2.4.21-vs1.20.diff
	 587170 patch-2.6.12.4-vs2.0.diff
	 593052 patch-2.6.14.3-vs2.01.diff
	 619268 patch-2.6.15.4-vs2.0.2-rc6.diff

Herbert,

Please stop seeding, hmm, falseness. OpenVZ patches you mention are against 2.6.8 kernel, thus they contain tons of backported mainstream bugfixes and driver updates; so, most of this size is not virtualization, but general security/stability/drivers stuff. And yes, that size also indirectly tells how much work we do to keep our users happy.

Back to the topic. If you (or somebody else) wants to see the real size of things, take a look at broken-out patch set, available from
http://download.openvz.org/kernel/broken-out/. Here (2.6.15-025stab014.1 kernel) we see that it all boils down to:

Virtualization stuff:                    diff-vemix-20060120-core   817K
Resource management (User Beancounters): diff-ubc-20060120          377K
Two-level disk quota:                    diff-vzdq-20051219-2       154K
#6 lore raw
Quoted 5 lines · Kir Kolyshkin

Please stop seeding, hmm, falseness. OpenVZ patches you mention are against 2.6.8 kernel, thus they contain tons of backported mainstream bugfixes and driver updates; so, most of this size is not virtualization, but general security/stability/drivers stuff. And yes, that size also indirectly tells how much work we do to keep our users happy.

I think Herbert was trying to add some balance to the equation.

Quoted 4 lines

Back to the topic. If you (or somebody else) wants to see the real size of things, take a look at broken-out patch set, available from
http://download.openvz.org/kernel/broken-out/. Here (2.6.15-025stab014.1 kernel) we see that it all boils down to:

Thanks. This is the first indication I have seen that you even have broken-out patches. Why those aren't in your source rpms is beyond me. Everything seems to have been posted in a 2-3 day window at the end of January and the beginning of February. Is this something you are now providing?

Shakes head. You have a patch in broken-out that is 817K. Do you really maintain it this way as one giant patch?

Quoted 3 lines
Virtualization stuff:                    diff-vemix-20060120-core   817K
Resource management (User Beancounters): diff-ubc-20060120          377K
Two-level disk quota:                    diff-vzdq-20051219-2       154K

As for the size of my code, sure parts of it are big I haven't really measured. Primarily this is because I'm not afraid of doing the heavy lifting necessary for a clean long term maintainable solution.

Now while all of this is interesting. It really is beside the point because neither the current vserver nor the current openvz code are ready for mainstream kernel inclusion. Please let's not get side tracked playing whose patch is bigger.

Eric

#7 lore raw
Quoted 7 lines · Eric W. Biederman
Quoted 4 lines

Back to the topic. If you (or somebody else) wants to see the real size of things, take a look at broken-out patch set, available from
http://download.openvz.org/kernel/broken-out/. Here (2.6.15-025stab014.1 kernel) we see that it all boils down to:

Thanks. This is the first indication I have seen that you even have broken-out patches.

When Kirill Korovaev announced OpenVZ patch set on LKML (two times -- initially and for 2.6.15), he gave the links to the broken-out patch set, both times.

Quoted 1 line

Why those aren't in your source rpms is beyond me.

That reflects our internal organization: we have a core virtualization team which comes up with a core patch (combining all the stuff), and a maintenance team which can add some extra patches (driver updates, some bugfixes). So that extra patches comes up as a separate patches in src.rpms, while virtualization stuff comes up as a single patch. That way it is easier for our maintainters group.

Sure we understand this is not convenient for developers who want to look at our code -- and thus we provide broken-out kernel patch sets from time to time (not for every release, as it requires some effort from Kirill, who is really buzy anyway). So, if you want this for a specific kernel -- just ask.

I understand that this might look strange, but again, this reflects our internal development structure.

Quoted 3 lines

Everything
seems to have been posted in a 2-3 day window at the end of January and the beginning of February. Is this something you are now providing?

Again, yes, occasionally from time to time, or upon request.

Quoted 2 lines

Shakes head. You have a patch in broken-out that is 817K. Do you really maintain it this way as one giant patch?

In that version I took (025stab014) it was indeed as one big patch, and I believe Kirill maintains it that way.

Previous kernel version (025stab012) was more fine-grained, take a look at http://download.openvz.org/kernel/broken-out/2.6.15-025stab012.1

Quoted 1 line

Please let's not get side tracked playing whose patch is bigger.

Absolutely agree!

Regards,
Kir Kolyshkin, OpenVZ team.

#8 lore raw
Quoted 4 lines · Kir Kolyshkin
Quoted 3 lines · Eric W. Biederman
Quoted 3 lines

Back to the topic. If you (or somebody else) wants to see the real size of things, take a look at broken-out patch set, available from
http://download.openvz.org/kernel/broken-out/. Here (2.6.15-025stab014.1

Quoted 8 lines

kernel)

Quoted 3 lines
Quoted 1 line

we see that it all boils down to:

Thanks. This is the first indication I have seen that you even have broken-out patches.

When Kirill Korovaev announced OpenVZ patch set on LKML (two times -- initially and for 2.6.15), he gave the links to the broken-out patch set, both times.

Hmm. I guess I just missed it.

Quoted 9 lines
Quoted 1 line

Why those aren't in your source rpms is beyond me.

That reflects our internal organization: we have a core virtualization team which comes up with a core patch (combining all the stuff), and a maintenance team which can add some extra patches (driver updates, some bugfixes). So that extra patches comes up as a separate patches in src.rpms, while virtualization stuff comes up as a single patch. That way it is easier for our maintainters group.

Quoted 7 lines

Sure we understand this is not convenient for developers who want to look at our code -- and thus we provide broken-out kernel patch sets from time to time (not for every release, as it requires some effort from Kirill, who is really buzy anyway). So, if you want this for a specific kernel -- just ask.

I understand that this might look strange, but again, this reflects our internal development structure.

There is something this brings up. Currently OpenVZ seems to be a project where you guys do the work and release the source under the GPL. Making it technically an open source project. However at the development level it does not appear to be a community project, at least at the developer level.

There is nothing wrong with not doing involving the larger community in the development, but what it does mean is that largely as a developer OpenVZ is uninteresting to me.

Quoted 8 lines
Quoted 2 lines

Shakes head. You have a patch in broken-out that is 817K. Do you really maintain it this way as one giant patch?

In that version I took (025stab014) it was indeed as one big patch, and I believe Kirill maintains it that way.

Previous kernel version (025stab012) was more fine-grained, take a look at
http://download.openvz.org/kernel/broken-out/2.6.15-025stab012.1

Looks a little better yes. This actually surprises me because my past experience is that usually well focused patches are easier to forward port as they usually have fewer collisions and those collisions are usually easier to resolve.

Eric

#9 lore raw
Quoted 23 lines · Eric W. Biederman
Quoted 7 lines

That reflects our internal organization: we have a core virtualization team which comes up with a core patch (combining all the stuff), and a maintenance team which can add some extra patches (driver updates, some bugfixes). So that extra patches comes up as a separate patches in src.rpms, while virtualization stuff comes up as a single patch. That way it is easier for our maintainters group.

Quoted 9 lines

Sure we understand this is not convenient for developers who want to look at our code -- and thus we provide broken-out kernel patch sets from time to time (not for every release, as it requires some effort from Kirill, who is really buzy anyway). So, if you want this for a specific kernel -- just ask.

I understand that this might look strange, but again, this reflects our internal development structure.

There is something this brings up. Currently OpenVZ seems to be a project where you guys do the work and release the source under the GPL. Making it technically an open source project. However at the development level it does not appear to be a community project, at least at the developer level.

Quoted 5 lines

There is nothing wrong with not doing involving the larger community in the development, but what it does mean is that largely as a developer OpenVZ is uninteresting to me.

I though that first thing that makes particular technology interesting or otherwise appealing to developers is the technology itself, i.e. is it interesting, appealing, innovative and superior, is it tomorrow today and so on. From that point, OpenVZ is pretty much interesting. From the point of openness -- well, you might be right, there's still something we could do.

I understand it should work both ways -- we should provide easier ways to access the code, to contribute etc. Still, I see little to no interest of contributing to OpenVZ kernel. Probably this is because of high entry level, probably it is because we are not yet open enough or so.

Any way, I would love to hear any comments/suggestions of how we can improve this situation from our side (and let me express hope you will improve it from yours:)).

Regards,
Kir.

#10 lore raw
In reply to Kirill Korotaev
Quoted 8 lines · Kirill Korotaev

Linus, Andrew,

We need your help on what virtualization approach you would accept to mainstream (if any) and where we should go.

If to drop VPID virtualization which caused many disputes, we actually have the one virtualization solution, but 2 approaches for it. Which one will go depends on the goals and your approval any way.

My apologies for not replying sooner.

Quoted 1 line

From the looks of previous replies I think we have some valid commonalities

that we can focus on.

Largely we all agree that to applications things should look exactly as they do now. Currently we do not agree on management interfaces.

We seem to have much more agreement on everything except pids, so discussing some of the other pieces looks worth while.

So I propose we the patches to solve the problem into three categories.
- General cleanups that simplify or fix problems now, but have a major advantage for our work.
- The kernel internal implementation of the various namespaces without an interface to create new ones.
- The new interfaces for how we create and control containers/namesp aces.

This should allow the various approach to start sharing code, getting progressively closer to each other until we have an implementation we can agree is ready to go into Linus's kernel. Plus that will allow us to have our technical flame wars without totally stopping progress.

We can start on a broad front, looking at several different things. But I suggest the first thing we all look at is SYSVIPC. It is currently a clearly recognized namespace in the kernel so the scope is well defined. SYSVIPC is just complicated enough to have a non-trivial implementation while at the same time being simple enough that we can go through the code in exhausting detail. Getting the group dynamics working properly.

Then we can as a group look at networking, pids, and the other pieces.

But I do think it is important that we take the problem in pieces because otherwise it is simply to large to review properly.

Eric

#11 lore raw
Quoted 23 lines · Eric W. Biederman
Quoted 9 lines · Kirill Korotaev

Linus, Andrew,

We need your help on what virtualization approach you would accept to mainstream (if any) and where we should go.

If to drop VPID virtualization which caused many disputes, we actually have the one virtualization solution, but 2 approaches for it. Which one will go depends on the goals and your approval any way.

My apologies for not replying sooner.

Quoted 2 lines

From the looks of previous replies I think we have some valid commonalities that we can focus on.

Largely we all agree that to applications things should look exactly as they do now. Currently we do not agree on management interfaces.

We seem to have much more agreement on everything except pids, so discussing some of the other pieces looks worth while.

Quoted 6 lines

So I propose we the patches to solve the problem into three categories.
- General cleanups that simplify or fix problems now, but have a major advantage for our work.
- The kernel internal implementation of the various namespaces without an interface to create new ones.
- The new interfaces for how we create and control containers/namespaces.

proposal accepted on my side

Quoted 5 lines

This should allow the various approach to start sharing code, getting progressively closer to each other until we have an implementation we can agree is ready to go into Linus's kernel. Plus that will allow us to have our technical flame wars without totally stopping progress.

Quoted 7 lines

We can start on a broad front, looking at several different things. But I suggest the first thing we all look at is SYSVIPC. It is currently a clearly recognized namespace in the kernel so the scope is well defined. SYSVIPC is just complicated enough to have a non-trivial implementation while at the same time being simple enough that we can go through the code in exhausting detail. Getting the group dynamics working properly.

okay, sounds good ...

Quoted 4 lines

Then we can as a group look at networking, pids, and the other pieces.

But I do think it is important that we take the problem in pieces because otherwise it is simply to large to review properly.

definitely

best,
Herbert

Quoted 6 lines

Eric
-
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to [email protected]
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/

#12 lore raw
In reply to Eric W. Biederman
Quoted 7 lines · Eric W. Biederman

We can start on a broad front, looking at several different things. But I suggest the first thing we all look at is SYSVIPC. It is currently a clearly recognized namespace in the kernel so the scope is well defined. SYSVIPC is just complicated enough to have a non-trivial implementation while at the same time being simple enough that we can go through the code in exhausting detail. Getting the group dynamics working properly.

Here's a quick stab at the ipc/msg.c portion of this work. The basic approach was to move msg_ids, msg_bytes, and msg_hdrs into a structure, put a pointer to that structure in the task_struct and then dynamically allocate it.

There is still only one system-wide one of these for now. It can obviously be extended, though. :)

This is a very simple, brute-force, hack-until-it-compiles-and-boots approach. (I just realized that I didn't check the return of the alloc properly.)

Is this the form that we'd like these patches to take? Any comments about the naming? Do we want to keep the _namespace nomenclature, or does the "context" that I used here make more sense

-- Dave

 work-dave/include/linux/ipc.h   |   12 +++
 work-dave/include/linux/sched.h |    1 
 work-dave/ipc/msg.c             |  152 ++++++++++++++++++++++------------------
 work-dave/ipc/util.c            |    7 +
 work-dave/ipc/util.h            |    2 
 work-dave/kernel/fork.c         |    5 +
 6 files changed, 108 insertions(+), 71 deletions(-)
ipc/msg.c renamed from ipc/msg.c~sysv-container +84 -68
@@ -60,35 +60,44 @@ struct msg_sender {
 #define SEARCH_NOTEQUAL		3
 #define SEARCH_LESSEQUAL	4
 
-static atomic_t msg_bytes = ATOMIC_INIT(0);
-static atomic_t msg_hdrs = ATOMIC_INIT(0);
+#define msg_lock(ctx, id)	((struct msg_queue*)ipc_lock(&ctx->msg_ids,id))
+#define msg_unlock(ctx, msq)	ipc_unlock(&(msq)->q_perm)
+#define msg_rmid(ctx, id)	((struct msg_queue*)ipc_rmid(&ctx->msg_ids,id))
+#define msg_checkid(ctx, msq, msgid)	\
+	ipc_checkid(&ctx->msg_ids,&msq->q_perm,msgid)
+#define msg_buildid(ctx, id, seq) \
+	ipc_buildid(&ctx->msg_ids, id, seq)
 
-static struct ipc_ids msg_ids;
-
-#define msg_lock(id)	((struct msg_queue*)ipc_lock(&msg_ids,id))
-#define msg_unlock(msq)	ipc_unlock(&(msq)->q_perm)
-#define msg_rmid(id)	((struct msg_queue*)ipc_rmid(&msg_ids,id))
-#define msg_checkid(msq, msgid)	\
-	ipc_checkid(&msg_ids,&msq->q_perm,msgid)
-#define msg_buildid(id, seq) \
-	ipc_buildid(&msg_ids, id, seq)
-
-static void freeque (struct msg_queue *msq, int id);
-static int newque (key_t key, int msgflg);
+static void freeque (struct ipc_msg_context *, struct msg_queue *msq, int id);
+static int newque (struct ipc_msg_context *context, key_t key, int id);
 #ifdef CONFIG_PROC_FS
 static int sysvipc_msg_proc_show(struct seq_file *s, void *it);
 #endif
 
-void __init msg_init (void)
+struct ipc_msg_context *alloc_ipc_msg_context(gfp_t flags)
+{
+	struct ipc_msg_context *msg_context;
+
+	msg_context = kzalloc(sizeof(*msg_context), flags);
+	if (!msg_context)
+		return NULL;
+
+	atomic_set(&msg_context->msg_bytes, 0);
+	atomic_set(&msg_context->msg_hdrs, 0);
+
+	return msg_context;
+}
+
+void __init msg_init (struct ipc_msg_context *context)
 {
-	ipc_init_ids(&msg_ids,msg_ctlmni);
+	ipc_init_ids(&context->msg_ids,msg_ctlmni);
 	ipc_init_proc_interface("sysvipc/msg",
 				"       key      msqid perms      cbytes       qnum lspid lrpid   uid   gid  cuid  cgid      stime      rtime      ctime\n",
-				&msg_ids,
+				&context->msg_ids,
 				sysvipc_msg_proc_show);
 }
 
-static int newque (key_t key, int msgflg)
+static int newque (struct ipc_msg_context *context, key_t key, int msgflg)
 {
 	int id;
 	int retval;
@@ -108,14 +117,14 @@ static int newque (key_t key, int msgflg
 		return retval;
 	}
 
-	id = ipc_addid(&msg_ids, &msq->q_perm, msg_ctlmni);
+	id = ipc_addid(&context->msg_ids, &msq->q_perm, msg_ctlmni);
 	if(id == -1) {
 		security_msg_queue_free(msq);
 		ipc_rcu_putref(msq);
 		return -ENOSPC;
 	}
 
-	msq->q_id = msg_buildid(id,msq->q_perm.seq);
+	msq->q_id = msg_buildid(context,id,msq->q_perm.seq);
 	msq->q_stime = msq->q_rtime = 0;
 	msq->q_ctime = get_seconds();
 	msq->q_cbytes = msq->q_qnum = 0;
@@ -124,7 +133,7 @@ static int newque (key_t key, int msgflg
 	INIT_LIST_HEAD(&msq->q_messages);
 	INIT_LIST_HEAD(&msq->q_receivers);
 	INIT_LIST_HEAD(&msq->q_senders);
-	msg_unlock(msq);
+	msg_unlock(context, msq);
 
 	return msq->q_id;
 }
@@ -182,23 +191,24 @@ static void expunge_all(struct msg_queue
  * msg_ids.sem and the spinlock for this message queue is hold
  * before freeque() is called. msg_ids.sem remains locked on exit.
  */
-static void freeque (struct msg_queue *msq, int id)
+static void freeque (struct ipc_msg_context *context,
+		     struct msg_queue *msq, int id)
 {
 	struct list_head *tmp;
 
 	expunge_all(msq,-EIDRM);
 	ss_wakeup(&msq->q_senders,1);
-	msq = msg_rmid(id);
-	msg_unlock(msq);
+	msq = msg_rmid(context, id);
+	msg_unlock(context, msq);
 		
 	tmp = msq->q_messages.next;
 	while(tmp != &msq->q_messages) {
 		struct msg_msg* msg = list_entry(tmp,struct msg_msg,m_list);
 		tmp = tmp->next;
-		atomic_dec(&msg_hdrs);
+		atomic_dec(&context->msg_hdrs);
 		free_msg(msg);
 	}
-	atomic_sub(msq->q_cbytes, &msg_bytes);
+	atomic_sub(msq->q_cbytes, &context->msg_bytes);
 	security_msg_queue_free(msq);
 	ipc_rcu_putref(msq);
 }
@@ -207,32 +217,34 @@ asmlinkage long sys_msgget (key_t key, i
 {
 	int id, ret = -EPERM;
 	struct msg_queue *msq;
-	
-	down(&msg_ids.sem);
+	struct ipc_msg_context *context = current->ipc_msg_context;
+
+	down(&context->msg_ids.sem);
 	if (key == IPC_PRIVATE) 
-		ret = newque(key, msgflg);
-	else if ((id = ipc_findkey(&msg_ids, key)) == -1) { /* key not used */
+		ret = newque(context, key, msgflg);
+	else if ((id = ipc_findkey(&context->msg_ids, key)) == -1) {
+		/* key not used */
 		if (!(msgflg & IPC_CREAT))
 			ret = -ENOENT;
 		else
-			ret = newque(key, msgflg);
+			ret = newque(context, key, msgflg);
 	} else if (msgflg & IPC_CREAT && msgflg & IPC_EXCL) {
 		ret = -EEXIST;
 	} else {
-		msq = msg_lock(id);
+		msq = msg_lock(context, id);
 		if(msq==NULL)
 			BUG();
 		if (ipcperms(&msq->q_perm, msgflg))
 			ret = -EACCES;
 		else {
-			int qid = msg_buildid(id, msq->q_perm.seq);
+			int qid = msg_buildid(context, id, msq->q_perm.seq);
 		    	ret = security_msg_queue_associate(msq, msgflg);
 			if (!ret)
 				ret = qid;
 		}
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 	}
-	up(&msg_ids.sem);
+	up(&context->msg_ids.sem);
 	return ret;
 }
 
@@ -333,6 +345,7 @@ asmlinkage long sys_msgctl (int msqid, i
 	struct msg_queue *msq;
 	struct msq_setbuf setbuf;
 	struct kern_ipc_perm *ipcp;
+	struct ipc_msg_context *context = current->ipc_msg_context;
 	
 	if (msqid < 0 || cmd < 0)
 		return -EINVAL;
@@ -362,18 +375,18 @@ asmlinkage long sys_msgctl (int msqid, i
 		msginfo.msgmnb = msg_ctlmnb;
 		msginfo.msgssz = MSGSSZ;
 		msginfo.msgseg = MSGSEG;
-		down(&msg_ids.sem);
+		down(&context->msg_ids.sem);
 		if (cmd == MSG_INFO) {
-			msginfo.msgpool = msg_ids.in_use;
-			msginfo.msgmap = atomic_read(&msg_hdrs);
-			msginfo.msgtql = atomic_read(&msg_bytes);
+			msginfo.msgpool = context->msg_ids.in_use;
+			msginfo.msgmap = atomic_read(&context->msg_hdrs);
+			msginfo.msgtql = atomic_read(&context->msg_bytes);
 		} else {
 			msginfo.msgmap = MSGMAP;
 			msginfo.msgpool = MSGPOOL;
 			msginfo.msgtql = MSGTQL;
 		}
-		max_id = msg_ids.max_id;
-		up(&msg_ids.sem);
+		max_id = context->msg_ids.max_id;
+		up(&context->msg_ids.sem);
 		if (copy_to_user (buf, &msginfo, sizeof(struct msginfo)))
 			return -EFAULT;
 		return (max_id < 0) ? 0: max_id;
@@ -385,20 +398,21 @@ asmlinkage long sys_msgctl (int msqid, i
 		int success_return;
 		if (!buf)
 			return -EFAULT;
-		if(cmd == MSG_STAT && msqid >= msg_ids.entries->size)
+		if(cmd == MSG_STAT && msqid >= context->msg_ids.entries->size)
 			return -EINVAL;
 
 		memset(&tbuf,0,sizeof(tbuf));
 
-		msq = msg_lock(msqid);
+		msq = msg_lock(context, msqid);
 		if (msq == NULL)
 			return -EINVAL;
 
 		if(cmd == MSG_STAT) {
-			success_return = msg_buildid(msqid, msq->q_perm.seq);
+			success_return =
+				msg_buildid(context, msqid, msq->q_perm.seq);
 		} else {
 			err = -EIDRM;
-			if (msg_checkid(msq,msqid))
+			if (msg_checkid(context,msq,msqid))
 				goto out_unlock;
 			success_return = 0;
 		}
@@ -419,7 +433,7 @@ asmlinkage long sys_msgctl (int msqid, i
 		tbuf.msg_qbytes = msq->q_qbytes;
 		tbuf.msg_lspid  = msq->q_lspid;
 		tbuf.msg_lrpid  = msq->q_lrpid;
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 		if (copy_msqid_to_user(buf, &tbuf, version))
 			return -EFAULT;
 		return success_return;
@@ -438,14 +452,14 @@ asmlinkage long sys_msgctl (int msqid, i
 		return  -EINVAL;
 	}
 
-	down(&msg_ids.sem);
-	msq = msg_lock(msqid);
+	down(&context->msg_ids.sem);
+	msq = msg_lock(context, msqid);
 	err=-EINVAL;
 	if (msq == NULL)
 		goto out_up;
 
 	err = -EIDRM;
-	if (msg_checkid(msq,msqid))
+	if (msg_checkid(context,msq,msqid))
 		goto out_unlock_up;
 	ipcp = &msq->q_perm;
 	err = -EPERM;
@@ -480,22 +494,22 @@ asmlinkage long sys_msgctl (int msqid, i
 		 * due to a larger queue size.
 		 */
 		ss_wakeup(&msq->q_senders,0);
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 		break;
 	}
 	case IPC_RMID:
-		freeque (msq, msqid); 
+		freeque (context, msq, msqid);
 		break;
 	}
 	err = 0;
 out_up:
-	up(&msg_ids.sem);
+	up(&context->msg_ids.sem);
 	return err;
 out_unlock_up:
-	msg_unlock(msq);
+	msg_unlock(context, msq);
 	goto out_up;
 out_unlock:
-	msg_unlock(msq);
+	msg_unlock(context, msq);
 	return err;
 }
 
@@ -558,7 +572,8 @@ asmlinkage long sys_msgsnd (int msqid, s
 	struct msg_msg *msg;
 	long mtype;
 	int err;
-	
+	struct ipc_msg_context *context = current->ipc_msg_context;
+
 	if (msgsz > msg_ctlmax || (long) msgsz < 0 || msqid < 0)
 		return -EINVAL;
 	if (get_user(mtype, &msgp->mtype))
@@ -573,13 +588,13 @@ asmlinkage long sys_msgsnd (int msqid, s
 	msg->m_type = mtype;
 	msg->m_ts = msgsz;
 
-	msq = msg_lock(msqid);
+	msq = msg_lock(context, msqid);
 	err=-EINVAL;
 	if(msq==NULL)
 		goto out_free;
 
 	err= -EIDRM;
-	if (msg_checkid(msq,msqid))
+	if (msg_checkid(context,msq,msqid))
 		goto out_unlock_free;
 
 	for (;;) {
@@ -605,7 +620,7 @@ asmlinkage long sys_msgsnd (int msqid, s
 		}
 		ss_add(msq, &s);
 		ipc_rcu_getref(msq);
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 		schedule();
 
 		ipc_lock_by_ptr(&msq->q_perm);
@@ -630,15 +645,15 @@ asmlinkage long sys_msgsnd (int msqid, s
 		list_add_tail(&msg->m_list,&msq->q_messages);
 		msq->q_cbytes += msgsz;
 		msq->q_qnum++;
-		atomic_add(msgsz,&msg_bytes);
-		atomic_inc(&msg_hdrs);
+		atomic_add(msgsz,&context->msg_bytes);
+		atomic_inc(&context->msg_hdrs);
 	}
 	
 	err = 0;
 	msg = NULL;
 
 out_unlock_free:
-	msg_unlock(msq);
+	msg_unlock(context, msq);
 out_free:
 	if(msg!=NULL)
 		free_msg(msg);
@@ -670,17 +685,18 @@ asmlinkage long sys_msgrcv (int msqid, s
 	struct msg_queue *msq;
 	struct msg_msg *msg;
 	int mode;
+	struct ipc_msg_context *context = current->ipc_msg_context;
 
 	if (msqid < 0 || (long) msgsz < 0)
 		return -EINVAL;
 	mode = convert_mode(&msgtyp,msgflg);
 
-	msq = msg_lock(msqid);
+	msq = msg_lock(context, msqid);
 	if(msq==NULL)
 		return -EINVAL;
 
 	msg = ERR_PTR(-EIDRM);
-	if (msg_checkid(msq,msqid))
+	if (msg_checkid(context,msq,msqid))
 		goto out_unlock;
 
 	for (;;) {
@@ -720,10 +736,10 @@ asmlinkage long sys_msgrcv (int msqid, s
 			msq->q_rtime = get_seconds();
 			msq->q_lrpid = current->tgid;
 			msq->q_cbytes -= msg->m_ts;
-			atomic_sub(msg->m_ts,&msg_bytes);
-			atomic_dec(&msg_hdrs);
+			atomic_sub(msg->m_ts,&context->msg_bytes);
+			atomic_dec(&context->msg_hdrs);
 			ss_wakeup(&msq->q_senders,0);
-			msg_unlock(msq);
+			msg_unlock(context, msq);
 			break;
 		}
 		/* No message waiting. Wait for a message */
@@ -741,7 +757,7 @@ asmlinkage long sys_msgrcv (int msqid, s
 			msr_d.r_maxsize = msgsz;
 		msr_d.r_msg = ERR_PTR(-EAGAIN);
 		current->state = TASK_INTERRUPTIBLE;
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 
 		schedule();
 
@@ -794,7 +810,7 @@ asmlinkage long sys_msgrcv (int msqid, s
 		if (signal_pending(current)) {
 			msg = ERR_PTR(-ERESTARTNOHAND);
 out_unlock:
-			msg_unlock(msq);
+			msg_unlock(context, msq);
 			break;
 		}
 	}

diff -puN ipc/msgutil.c~sysv-container ipc/msgutil.c diff -puN ipc/sem.c~sysv-container ipc/sem.c diff -puN ipc/shm.c~sysv-container ipc/shm.c

ipc/util.c renamed from ipc/util.c~sysv-container +5 -2
diff -puN ipc/util.c~sysv-container ipc/util.c
@@ -45,11 +45,14 @@ struct ipc_proc_iface {
  *	The various system5 IPC resources (semaphores, messages and shared
  *	memory are initialised
  */
- 
+
+struct ipc_msg_context *ipc_msg_context;
 static int __init ipc_init(void)
 {
+	ipc_msg_context = alloc_ipc_msg_context(GFP_KERNEL);
+
 	sem_init();
-	msg_init();
+	msg_init(ipc_msg_context);
 	shm_init();
 	return 0;
 }
ipc/util.h renamed from ipc/util.h~sysv-container +1 -1
diff -puN ipc/util.h~sysv-container ipc/util.h
@@ -12,7 +12,7 @@
 #define SEQ_MULTIPLIER	(IPCMNI)
 
 void sem_init (void);
-void msg_init (void);
+void msg_init (struct ipc_msg_context *context);
 void shm_init (void);
 
 struct seq_file;
include/linux/sched.h renamed from include/linux/sched.h~sysv-container +1 -0
diff -puN include/linux/sched.h~sysv-container include/linux/sched.h
@@ -793,6 +793,7 @@ struct task_struct {
 	int link_count, total_link_count;
 /* ipc stuff */
 	struct sysv_sem sysvsem;
+	struct ipc_msg_context *ipc_msg_context;
 /* CPU-specific state of this task */
 	struct thread_struct thread;
 /* filesystem information */
include/linux/ipc.h renamed from include/linux/ipc.h~sysv-container +12 -0
diff -puN include/linux/ipc.h~sysv-container include/linux/ipc.h
@@ -2,6 +2,9 @@
 #define _LINUX_IPC_H
 
 #include <linux/types.h>
+#include <linux/kref.h>
+#include <asm/atomic.h>
+#include <asm/semaphore.h>
 
 #define IPC_PRIVATE ((__kernel_key_t) 0)  
 
@@ -83,6 +86,15 @@ struct ipc_ids {
 	struct ipc_id_ary* entries;
 };
 
+struct ipc_msg_context {
+	atomic_t msg_bytes;
+	atomic_t msg_hdrs;
+
+	struct ipc_ids msg_ids;
+	struct kref count;
+};
+
+extern struct ipc_msg_context *alloc_ipc_msg_context(gfp_t flags);
 #endif /* __KERNEL__ */
 
 #endif /* _LINUX_IPC_H */
kernel/fork.c renamed from kernel/fork.c~sysv-container +5 -0
@@ -1184,6 +1184,11 @@ static task_t *copy_process(unsigned lon
 	}
 	attach_pid(p, PIDTYPE_TGID, p->tgid);
 	attach_pid(p, PIDTYPE_PID, p->pid);
+	{ // this extern will go away when we start to dynamically
+	  // allocate these, nothing to see here
+		extern struct ipc_msg_context ipc_msg_context;
+		p->ipc_msg_context = current->ipc_msg_context;
+	}
 
 	nr_threads++;
 	total_forks++;

_

#13 lore raw
Quoted 17 lines · Dave Hansen
Quoted 7 lines · Eric W. Biederman

We can start on a broad front, looking at several different things. But I suggest the first thing we all look at is SYSVIPC. It is currently a clearly recognized namespace in the kernel so the scope is well defined. SYSVIPC is just complicated enough to have a non-trivial implementation while at the same time being simple enough that we can go through the code in exhausting detail. Getting the group dynamics working properly.

Here's a quick stab at the ipc/msg.c portion of this work. The basic approach was to move msg_ids, msg_bytes, and msg_hdrs into a structure, put a pointer to that structure in the task_struct and then dynamically allocate it.

There is still only one system-wide one of these for now. It can obviously be extended, though. :)

Quoted 7 lines

This is a very simple, brute-force, hack-until-it-compiles-and-boots approach. (I just realized that I didn't check the return of the alloc properly.)

Is this the form that we'd like these patches to take? Any comments about the naming? Do we want to keep the _namespace nomenclature, or does the "context" that I used here make more sense

I think from 10,000 feet the form is about right.

I like the namespace nomenclature. (It can be shorted to _space or _ns). In part because it shortens well, and in part because it emphasizes that we are *just* dealing with the names.

You split the resolution at just ipc_msgs. When I really think it should be everything ipcs deals with.

Performing the assignment inside the tasklist_lock is not something we want to do in do_fork().

So it looks like a good start. There are a lot of details yet to be filled in, proc, sysctl, cleanup on namespace release. (We can still provide the create destroy methods even if we don't hook the up).

I think in this case I would put the actual namespace structure definition in util.h, and just put a struct ipc_ns in sched.h. sysvipc is isolated enough that nothing outside of the ipc/ directory needs to know the implementation details.

It probably makes sense to have a statically structure and to set the pointer initially in init_task.h

Until we reach the point where we can multiple instances that even removes the need to have a pointer copy in do_fork() as that happens already as part of the structure copy.

Eric

#14 lore raw
Quoted 3 lines · Eric W. Biederman

I like the namespace nomenclature. (It can be shorted to _space or _ns). In part because it shortens well, and in part because it emphasizes that we are *just* dealing with the names.

When I was looking at this, I was pretending to be just somebody looking at sysv code, with no knowledge of containers or namespaces.

For a person like that, I think names like _space or _ns are pretty much not an option, unless those terms become as integral to the kernel as things like kobjects.

Quoted 2 lines

You split the resolution at just ipc_msgs. When I really think it should be everything ipcs deals with.

This was just the first patch. :)

Quoted 2 lines

Performing the assignment inside the tasklist_lock is not something we want to do in do_fork().

Any particular reason why? There seem to be a number of things done in there that aren't _strictly_ needed under the tasklist_lock. Where would you do it?

Quoted 3 lines

So it looks like a good start. There are a lot of details yet to be filled in, proc, sysctl, cleanup on namespace release. (We can still provide the create destroy methods even if we don't hook the up).

Yeah, I saved shm for last because it has the largest number of outside interactions. My current thoughts are that we'll need _contexts or _namespaces associated with /proc mounts as well.

Quoted 2 lines

I think in this case I would put the actual namespace structure definition in util.h, and just put a struct ipc_ns in sched.h.

Ahhh, as in

	struct ipc_ns;

And just keep a pointer from the task? Yeah, that does keep it quite isolated.

-- Dave

#15 lore raw
Quoted 11 lines · Dave Hansen
Quoted 3 lines · Eric W. Biederman

I like the namespace nomenclature. (It can be shorted to _space or _ns). In part because it shortens well, and in part because it emphasizes that we are *just* dealing with the names.

When I was looking at this, I was pretending to be just somebody looking at sysv code, with no knowledge of containers or namespaces.

For a person like that, I think names like _space or _ns are pretty much not an option, unless those terms become as integral to the kernel as things like kobjects.

To be clear I was talking name suffixes. So ipc_space certainly conveys something, and even ipc_ns may be ok.

Quoted 4 lines
Quoted 2 lines

You split the resolution at just ipc_msgs. When I really think it should be everything ipcs deals with.

This was just the first patch. :)

:)

Just wanted to make certain we agreed on the scope.

Quoted 6 lines
Quoted 2 lines

Performing the assignment inside the tasklist_lock is not something we want to do in do_fork().

Any particular reason why? There seem to be a number of things done in there that aren't _strictly_ needed under the tasklist_lock. Where would you do it?

Well all of the other things we can share or not share are already outside of the tasklist_lock.

We may not be quite minimal but we actually are fairly close to minimal inside the tasklist_lock.

Quoted 7 lines
Quoted 3 lines

So it looks like a good start. There are a lot of details yet to be filled in, proc, sysctl, cleanup on namespace release. (We can still provide the create destroy methods even if we don't hook the up).

Yeah, I saved shm for last because it has the largest number of outside interactions. My current thoughts are that we'll need _contexts or _namespaces associated with /proc mounts as well.

Yes. I think the easy way to handle this is to have a symlink from /proc/sysvipc to /proc/self/sysvipc. And then we have a per process reporting area.

That preserves all of the old programs but enables us to get the information out.

Quoted 9 lines
Quoted 2 lines

I think in this case I would put the actual namespace structure definition in util.h, and just put a struct ipc_ns in sched.h.

Ahhh, as in

	struct ipc_ns;

And just keep a pointer from the task? Yeah, that does keep it quite isolated.

Yep.

Eric

#16 lore raw

sysctls inside containers

In reply to Eric W. Biederman

Trimming the cc list down, because the scope has narrowed significantly...

Quoted 3 lines · Eric W. Biederman

So it looks like a good start. There are a lot of details yet to be filled in, proc, sysctl, cleanup on namespace release. (We can still provide the create destroy methods even if we don't hook the up).

Well, I at least got to the point of seeing how the sysctls interact when I tried to containerize them. Eric, I think the idea of the sysv code being nicely and completely isolated is pretty much gone, due to their connection to sysctls. I think I'll go back and just isolate the "struct ipc_ids" portion. We can do the accounting bits later.

The patches I have will isolate the IDs, but I'm not sure how much sense that makes without doing the things like the shm_tot variable. Does anybody think we need to go after sysctls first, perhaps? Or, is this a problem graph with cycles in it? :)

I don't see an immediately clear solution on how to containerize sysctls properly. The entire construct seems to be built around getting data from in and out of global variables and into /proc files.

We obviously want to be rid of many of these global variables. So, does it make sense to introduce different classes of sysctls, at least internally? There are probably just two types: global, writable only from the root container and container-private. Does it make sense to have _both_? Perhaps a sysadmin

Eric, can you think of how you would represent these in the hierarchical container model? How would they work?

On another note, after messing with putting data in the init_task for these things, I'm a little more convinced that we aren't going to want to clutter up the task_struct with all kinds of containerized resources, _plus_ make all of the interfaces to share or unshare each of those. That global 'struct container' is looking a bit more attractive.

-- Dave

#17 lore raw

Re: sysctls inside containers

Quoted 2 lines · Dave Hansen

Trimming the cc list down, because the scope has narrowed significantly...

Undoing some of the trimming to include the vserver guys at least. Herbert at least has a better handle on how to do resource limits than I do.

Quoted 12 lines · Eric W. Biederman
Quoted 3 lines

So it looks like a good start. There are a lot of details yet to be filled in, proc, sysctl, cleanup on namespace release. (We can still provide the create destroy methods even if we don't hook the up).

Well, I at least got to the point of seeing how the sysctls interact when I tried to containerize them. Eric, I think the idea of the sysv code being nicely and completely isolated is pretty much gone, due to their connection to sysctls. I think I'll go back and just isolate the "struct ipc_ids" portion. We can do the accounting bits later.

Quoted 4 lines

The patches I have will isolate the IDs, but I'm not sure how much sense that makes without doing the things like the shm_tot variable. Does anybody think we need to go after sysctls first, perhaps? Or, is this a problem graph with cycles in it? :)

There is a reason I'm cleaning up /proc at the moment :)

But the only real gotcha I see is how do we do per namespace resource limits. As opposed to global resource limits. For the moment just having truly global limits seems reasonable.

Quoted 3 lines

I don't see an immediately clear solution on how to containerize sysctls properly. The entire construct seems to be built around getting data from in and out of global variables and into /proc files.

I successfully handled pid_max. So while it is awkward you can get per task values in and out if sysctl.

Quoted 5 lines

We obviously want to be rid of many of these global variables. So, does it make sense to introduce different classes of sysctls, at least internally? There are probably just two types: global, writable only from the root container and container-private. Does it make sense to have _both_? Perhaps a sysadmin

So having both is doable. Although most of that goes to resource limits, and resource limits are something we clearly need to discuss but they really are a separate problem.

Quoted 2 lines

Eric, can you think of how you would represent these in the hierarchical container model? How would they work?

There is nothing in the implementation of sysvipc that is inherently hierarchical. So it would simply be disjoin flat namespaces. That are only hierarchical in the sense that they are connected to processes which form a hierarchical process tree.

As for the resource limit problem that is the domain of CKRM and the bean counter patches. The isolation work we are doing may put a new spin on the problem and help it get solved but it isn't something we need to solve.

Herbert Poetzl has suggest in the past is that we may need some kind of namespace off to the side that is the essence of a container and has all of the container resource limits.

Quoted 5 lines

On another note, after messing with putting data in the init_task for these things, I'm a little more convinced that we aren't going to want to clutter up the task_struct with all kinds of containerized resources, _plus_ make all of the interfaces to share or unshare each of those. That global 'struct container' is looking a bit more attractive.

There aren't that many of them, and what different groups want to share/unshare is different. The biggest difference is that for migration restart it is not necessary to solve the problem of having the same UID map to multiple user_structs depending on the context and all of the weird things that will do to permission checking in the kernel.

There are also a few items migration cares about that people doing virtual private servers don't. Like the ability to isolate monotonic timers, so even after migration the monotonic properties are maintained. If you always remain on the same kernel that is a problem that simply does not come up.

Eric

#18 lore raw

Re: sysctls inside containers

Quoted 6 lines · Eric W. Biederman
Quoted 3 lines

I don't see an immediately clear solution on how to containerize sysctls properly. The entire construct seems to be built around getting data from in and out of global variables and into /proc files.

I successfully handled pid_max. So while it is awkward you can get per task values in and out if sysctl.

This:

http://www.kernel.org/git/gitweb.cgi?p=linux/kernel/git/ebiederm/linux-2.6-ns.git;a=commitdiff;h=1150082e0bae41a3621043b4c5ce15e9112884fa

sir, is a hack :)

We can't possibly do that for each and every sysctl variable. It would mean about fourteen billion duplicated _conv() functions.

I'm wondering if, instead of having the .data field in that table, we can have a function pointer which, when called, gives a pointer to the data. I'll give it a shot.

-- Dave

#19 lore raw

Re: sysctls inside containers

Quoted 12 lines · Dave Hansen
Quoted 6 lines · Eric W. Biederman
Quoted 3 lines

I don't see an immediately clear solution on how to containerize sysctls properly. The entire construct seems to be built around getting data from in and out of global variables and into /proc files.

I successfully handled pid_max. So while it is awkward you can get per task values in and out if sysctl.

This:

http://www.kernel.org/git/gitweb.cgi?p=linux/kernel/git/ebiederm/linux-2.6-ns.git;a=commitdiff;h=1150082e0bae41a3621043b4c5ce15e9112884fa

Quoted 8 lines

sir, is a hack :)

We can't possibly do that for each and every sysctl variable. It would mean about fourteen billion duplicated _conv() functions.

I'm wondering if, instead of having the .data field in that table, we can have a function pointer which, when called, gives a pointer to the data. I'll give it a shot.

something similar to this?

http://www.13thfloor.at/vserver/d_rel26/v2.1.0/split-2.6.14.4-vs2.1.0/15_2.6.14.4_virt.diff.hl

(look for virt_handler() for sysctl ops)

best,
Herbert

Quoted 1 line

-- Dave

#20 lore raw

Re: sysctls inside containers

Quoted 9 lines · Herbert Poetzl
Quoted 3 lines

I'm wondering if, instead of having the .data field in that table, we can have a function pointer which, when called, gives a pointer to the data. I'll give it a shot.

something similar to this?

http://www.13thfloor.at/vserver/d_rel26/v2.1.0/split-2.6.14.4-vs2.1.0/15_2.6.14.4_virt.diff.hl

(look for virt_handler() for sysctl ops)

Yeah, that's the right general idea. However, I think for mainline we probably want to make it a little less of an add-on feature. I was thinking that we could replace all of the current, flat sysctl .data pointers with some simple helper functions. Something like this:

#define SYSCTL_HELPER_FOR(name) sysctl_return_var_named##name

#define DECLARE_HELPER_FOR(type, name)          \
void *SYSCTL_HELPER_FOR(name)(void)             \
{                                               \
        extern type name;                       \
        return &name;                           \

}

And, the declaration for max_threads:

DECLARE_HELPER_FOR(int, max_threads));

and the ctl_table entry:

{
.ctl_name = KERN_MAX_THREADS,
.procname = "threads-max",
.data_access = &SYSCTL_HELPER_FOR(max_threads),
.maxlen = sizeof(int),
.mode = 0644,
.proc_handler = &proc_dointvec,
},

Note that the

	.data	= &max_threads,

line is gone.

I hate multi-line #defines as much as the next guy, but it might be worth it. I've attached what I have so far, but it is a major work in progress.

-- Dave

---

 work-dave/include/linux/ipc.h   |   12 +++
 work-dave/include/linux/sched.h |    1 
 work-dave/ipc/msg.c             |  152 ++++++++++++++++++++++------------------
 work-dave/ipc/util.c            |    7 +
 work-dave/ipc/util.h            |    2 
 work-dave/kernel/fork.c         |    5 +
 6 files changed, 108 insertions(+), 71 deletions(-)

diff -puN ipc/compat.c~sysv-container ipc/compat.c diff -puN ipc/compat_mq.c~sysv-container ipc/compat_mq.c diff -puN ipc/mqueue.c~sysv-container ipc/mqueue.c

ipc/msg.c renamed from ipc/msg.c~sysv-container +84 -68
diff -puN ipc/msg.c~sysv-container ipc/msg.c
@@ -60,35 +60,44 @@ struct msg_sender {
 #define SEARCH_NOTEQUAL		3
 #define SEARCH_LESSEQUAL	4
 
-static atomic_t msg_bytes = ATOMIC_INIT(0);
-static atomic_t msg_hdrs = ATOMIC_INIT(0);
+#define msg_lock(ctx, id)	((struct msg_queue*)ipc_lock(&ctx->msg_ids,id))
+#define msg_unlock(ctx, msq)	ipc_unlock(&(msq)->q_perm)
+#define msg_rmid(ctx, id)	((struct msg_queue*)ipc_rmid(&ctx->msg_ids,id))
+#define msg_checkid(ctx, msq, msgid)	\
+	ipc_checkid(&ctx->msg_ids,&msq->q_perm,msgid)
+#define msg_buildid(ctx, id, seq) \
+	ipc_buildid(&ctx->msg_ids, id, seq)
 
-static struct ipc_ids msg_ids;
-
-#define msg_lock(id)	((struct msg_queue*)ipc_lock(&msg_ids,id))
-#define msg_unlock(msq)	ipc_unlock(&(msq)->q_perm)
-#define msg_rmid(id)	((struct msg_queue*)ipc_rmid(&msg_ids,id))
-#define msg_checkid(msq, msgid)	\
-	ipc_checkid(&msg_ids,&msq->q_perm,msgid)
-#define msg_buildid(id, seq) \
-	ipc_buildid(&msg_ids, id, seq)
-
-static void freeque (struct msg_queue *msq, int id);
-static int newque (key_t key, int msgflg);
+static void freeque (struct ipc_msg_context *, struct msg_queue *msq, int id);
+static int newque (struct ipc_msg_context *context, key_t key, int id);
 #ifdef CONFIG_PROC_FS
 static int sysvipc_msg_proc_show(struct seq_file *s, void *it);
 #endif
 
-void __init msg_init (void)
+struct ipc_msg_context *alloc_ipc_msg_context(gfp_t flags)
+{
+	struct ipc_msg_context *msg_context;
+
+	msg_context = kzalloc(sizeof(*msg_context), flags);
+	if (!msg_context)
+		return NULL;
+
+	atomic_set(&msg_context->msg_bytes, 0);
+	atomic_set(&msg_context->msg_hdrs, 0);
+
+	return msg_context;
+}
+
+void __init msg_init (struct ipc_msg_context *context)
 {
-	ipc_init_ids(&msg_ids,msg_ctlmni);
+	ipc_init_ids(&context->msg_ids,msg_ctlmni);
 	ipc_init_proc_interface("sysvipc/msg",
 				"       key      msqid perms      cbytes       qnum lspid lrpid   uid   gid  cuid  cgid      stime      rtime      ctime\n",
-				&msg_ids,
+				&context->msg_ids,
 				sysvipc_msg_proc_show);
 }
 
-static int newque (key_t key, int msgflg)
+static int newque (struct ipc_msg_context *context, key_t key, int msgflg)
 {
 	int id;
 	int retval;
@@ -108,14 +117,14 @@ static int newque (key_t key, int msgflg
 		return retval;
 	}
 
-	id = ipc_addid(&msg_ids, &msq->q_perm, msg_ctlmni);
+	id = ipc_addid(&context->msg_ids, &msq->q_perm, msg_ctlmni);
 	if(id == -1) {
 		security_msg_queue_free(msq);
 		ipc_rcu_putref(msq);
 		return -ENOSPC;
 	}
 
-	msq->q_id = msg_buildid(id,msq->q_perm.seq);
+	msq->q_id = msg_buildid(context,id,msq->q_perm.seq);
 	msq->q_stime = msq->q_rtime = 0;
 	msq->q_ctime = get_seconds();
 	msq->q_cbytes = msq->q_qnum = 0;
@@ -124,7 +133,7 @@ static int newque (key_t key, int msgflg
 	INIT_LIST_HEAD(&msq->q_messages);
 	INIT_LIST_HEAD(&msq->q_receivers);
 	INIT_LIST_HEAD(&msq->q_senders);
-	msg_unlock(msq);
+	msg_unlock(context, msq);
 
 	return msq->q_id;
 }
@@ -182,23 +191,24 @@ static void expunge_all(struct msg_queue
  * msg_ids.sem and the spinlock for this message queue is hold
  * before freeque() is called. msg_ids.sem remains locked on exit.
  */
-static void freeque (struct msg_queue *msq, int id)
+static void freeque (struct ipc_msg_context *context,
+		     struct msg_queue *msq, int id)
 {
 	struct list_head *tmp;
 
 	expunge_all(msq,-EIDRM);
 	ss_wakeup(&msq->q_senders,1);
-	msq = msg_rmid(id);
-	msg_unlock(msq);
+	msq = msg_rmid(context, id);
+	msg_unlock(context, msq);
 		
 	tmp = msq->q_messages.next;
 	while(tmp != &msq->q_messages) {
 		struct msg_msg* msg = list_entry(tmp,struct msg_msg,m_list);
 		tmp = tmp->next;
-		atomic_dec(&msg_hdrs);
+		atomic_dec(&context->msg_hdrs);
 		free_msg(msg);
 	}
-	atomic_sub(msq->q_cbytes, &msg_bytes);
+	atomic_sub(msq->q_cbytes, &context->msg_bytes);
 	security_msg_queue_free(msq);
 	ipc_rcu_putref(msq);
 }
@@ -207,32 +217,34 @@ asmlinkage long sys_msgget (key_t key, i
 {
 	int id, ret = -EPERM;
 	struct msg_queue *msq;
-	
-	down(&msg_ids.sem);
+	struct ipc_msg_context *context = current->ipc_msg_context;
+
+	down(&context->msg_ids.sem);
 	if (key == IPC_PRIVATE) 
-		ret = newque(key, msgflg);
-	else if ((id = ipc_findkey(&msg_ids, key)) == -1) { /* key not used */
+		ret = newque(context, key, msgflg);
+	else if ((id = ipc_findkey(&context->msg_ids, key)) == -1) {
+		/* key not used */
 		if (!(msgflg & IPC_CREAT))
 			ret = -ENOENT;
 		else
-			ret = newque(key, msgflg);
+			ret = newque(context, key, msgflg);
 	} else if (msgflg & IPC_CREAT && msgflg & IPC_EXCL) {
 		ret = -EEXIST;
 	} else {
-		msq = msg_lock(id);
+		msq = msg_lock(context, id);
 		if(msq==NULL)
 			BUG();
 		if (ipcperms(&msq->q_perm, msgflg))
 			ret = -EACCES;
 		else {
-			int qid = msg_buildid(id, msq->q_perm.seq);
+			int qid = msg_buildid(context, id, msq->q_perm.seq);
 		    	ret = security_msg_queue_associate(msq, msgflg);
 			if (!ret)
 				ret = qid;
 		}
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 	}
-	up(&msg_ids.sem);
+	up(&context->msg_ids.sem);
 	return ret;
 }
 
@@ -333,6 +345,7 @@ asmlinkage long sys_msgctl (int msqid, i
 	struct msg_queue *msq;
 	struct msq_setbuf setbuf;
 	struct kern_ipc_perm *ipcp;
+	struct ipc_msg_context *context = current->ipc_msg_context;
 	
 	if (msqid < 0 || cmd < 0)
 		return -EINVAL;
@@ -362,18 +375,18 @@ asmlinkage long sys_msgctl (int msqid, i
 		msginfo.msgmnb = msg_ctlmnb;
 		msginfo.msgssz = MSGSSZ;
 		msginfo.msgseg = MSGSEG;
-		down(&msg_ids.sem);
+		down(&context->msg_ids.sem);
 		if (cmd == MSG_INFO) {
-			msginfo.msgpool = msg_ids.in_use;
-			msginfo.msgmap = atomic_read(&msg_hdrs);
-			msginfo.msgtql = atomic_read(&msg_bytes);
+			msginfo.msgpool = context->msg_ids.in_use;
+			msginfo.msgmap = atomic_read(&context->msg_hdrs);
+			msginfo.msgtql = atomic_read(&context->msg_bytes);
 		} else {
 			msginfo.msgmap = MSGMAP;
 			msginfo.msgpool = MSGPOOL;
 			msginfo.msgtql = MSGTQL;
 		}
-		max_id = msg_ids.max_id;
-		up(&msg_ids.sem);
+		max_id = context->msg_ids.max_id;
+		up(&context->msg_ids.sem);
 		if (copy_to_user (buf, &msginfo, sizeof(struct msginfo)))
 			return -EFAULT;
 		return (max_id < 0) ? 0: max_id;
@@ -385,20 +398,21 @@ asmlinkage long sys_msgctl (int msqid, i
 		int success_return;
 		if (!buf)
 			return -EFAULT;
-		if(cmd == MSG_STAT && msqid >= msg_ids.entries->size)
+		if(cmd == MSG_STAT && msqid >= context->msg_ids.entries->size)
 			return -EINVAL;
 
 		memset(&tbuf,0,sizeof(tbuf));
 
-		msq = msg_lock(msqid);
+		msq = msg_lock(context, msqid);
 		if (msq == NULL)
 			return -EINVAL;
 
 		if(cmd == MSG_STAT) {
-			success_return = msg_buildid(msqid, msq->q_perm.seq);
+			success_return =
+				msg_buildid(context, msqid, msq->q_perm.seq);
 		} else {
 			err = -EIDRM;
-			if (msg_checkid(msq,msqid))
+			if (msg_checkid(context,msq,msqid))
 				goto out_unlock;
 			success_return = 0;
 		}
@@ -419,7 +433,7 @@ asmlinkage long sys_msgctl (int msqid, i
 		tbuf.msg_qbytes = msq->q_qbytes;
 		tbuf.msg_lspid  = msq->q_lspid;
 		tbuf.msg_lrpid  = msq->q_lrpid;
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 		if (copy_msqid_to_user(buf, &tbuf, version))
 			return -EFAULT;
 		return success_return;
@@ -438,14 +452,14 @@ asmlinkage long sys_msgctl (int msqid, i
 		return  -EINVAL;
 	}
 
-	down(&msg_ids.sem);
-	msq = msg_lock(msqid);
+	down(&context->msg_ids.sem);
+	msq = msg_lock(context, msqid);
 	err=-EINVAL;
 	if (msq == NULL)
 		goto out_up;
 
 	err = -EIDRM;
-	if (msg_checkid(msq,msqid))
+	if (msg_checkid(context,msq,msqid))
 		goto out_unlock_up;
 	ipcp = &msq->q_perm;
 	err = -EPERM;
@@ -480,22 +494,22 @@ asmlinkage long sys_msgctl (int msqid, i
 		 * due to a larger queue size.
 		 */
 		ss_wakeup(&msq->q_senders,0);
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 		break;
 	}
 	case IPC_RMID:
-		freeque (msq, msqid); 
+		freeque (context, msq, msqid);
 		break;
 	}
 	err = 0;
 out_up:
-	up(&msg_ids.sem);
+	up(&context->msg_ids.sem);
 	return err;
 out_unlock_up:
-	msg_unlock(msq);
+	msg_unlock(context, msq);
 	goto out_up;
 out_unlock:
-	msg_unlock(msq);
+	msg_unlock(context, msq);
 	return err;
 }
 
@@ -558,7 +572,8 @@ asmlinkage long sys_msgsnd (int msqid, s
 	struct msg_msg *msg;
 	long mtype;
 	int err;
-	
+	struct ipc_msg_context *context = current->ipc_msg_context;
+
 	if (msgsz > msg_ctlmax || (long) msgsz < 0 || msqid < 0)
 		return -EINVAL;
 	if (get_user(mtype, &msgp->mtype))
@@ -573,13 +588,13 @@ asmlinkage long sys_msgsnd (int msqid, s
 	msg->m_type = mtype;
 	msg->m_ts = msgsz;
 
-	msq = msg_lock(msqid);
+	msq = msg_lock(context, msqid);
 	err=-EINVAL;
 	if(msq==NULL)
 		goto out_free;
 
 	err= -EIDRM;
-	if (msg_checkid(msq,msqid))
+	if (msg_checkid(context,msq,msqid))
 		goto out_unlock_free;
 
 	for (;;) {
@@ -605,7 +620,7 @@ asmlinkage long sys_msgsnd (int msqid, s
 		}
 		ss_add(msq, &s);
 		ipc_rcu_getref(msq);
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 		schedule();
 
 		ipc_lock_by_ptr(&msq->q_perm);
@@ -630,15 +645,15 @@ asmlinkage long sys_msgsnd (int msqid, s
 		list_add_tail(&msg->m_list,&msq->q_messages);
 		msq->q_cbytes += msgsz;
 		msq->q_qnum++;
-		atomic_add(msgsz,&msg_bytes);
-		atomic_inc(&msg_hdrs);
+		atomic_add(msgsz,&context->msg_bytes);
+		atomic_inc(&context->msg_hdrs);
 	}
 	
 	err = 0;
 	msg = NULL;
 
 out_unlock_free:
-	msg_unlock(msq);
+	msg_unlock(context, msq);
 out_free:
 	if(msg!=NULL)
 		free_msg(msg);
@@ -670,17 +685,18 @@ asmlinkage long sys_msgrcv (int msqid, s
 	struct msg_queue *msq;
 	struct msg_msg *msg;
 	int mode;
+	struct ipc_msg_context *context = current->ipc_msg_context;
 
 	if (msqid < 0 || (long) msgsz < 0)
 		return -EINVAL;
 	mode = convert_mode(&msgtyp,msgflg);
 
-	msq = msg_lock(msqid);
+	msq = msg_lock(context, msqid);
 	if(msq==NULL)
 		return -EINVAL;
 
 	msg = ERR_PTR(-EIDRM);
-	if (msg_checkid(msq,msqid))
+	if (msg_checkid(context,msq,msqid))
 		goto out_unlock;
 
 	for (;;) {
@@ -720,10 +736,10 @@ asmlinkage long sys_msgrcv (int msqid, s
 			msq->q_rtime = get_seconds();
 			msq->q_lrpid = current->tgid;
 			msq->q_cbytes -= msg->m_ts;
-			atomic_sub(msg->m_ts,&msg_bytes);
-			atomic_dec(&msg_hdrs);
+			atomic_sub(msg->m_ts,&context->msg_bytes);
+			atomic_dec(&context->msg_hdrs);
 			ss_wakeup(&msq->q_senders,0);
-			msg_unlock(msq);
+			msg_unlock(context, msq);
 			break;
 		}
 		/* No message waiting. Wait for a message */
@@ -741,7 +757,7 @@ asmlinkage long sys_msgrcv (int msqid, s
 			msr_d.r_maxsize = msgsz;
 		msr_d.r_msg = ERR_PTR(-EAGAIN);
 		current->state = TASK_INTERRUPTIBLE;
-		msg_unlock(msq);
+		msg_unlock(context, msq);
 
 		schedule();
 
@@ -794,7 +810,7 @@ asmlinkage long sys_msgrcv (int msqid, s
 		if (signal_pending(current)) {
 			msg = ERR_PTR(-ERESTARTNOHAND);
 out_unlock:
-			msg_unlock(msq);
+			msg_unlock(context, msq);
 			break;
 		}
 	}

diff -puN ipc/msgutil.c~sysv-container ipc/msgutil.c diff -puN ipc/sem.c~sysv-container ipc/sem.c diff -puN ipc/shm.c~sysv-container ipc/shm.c

ipc/util.c renamed from ipc/util.c~sysv-container +5 -2
diff -puN ipc/util.c~sysv-container ipc/util.c
@@ -45,11 +45,14 @@ struct ipc_proc_iface {
  *	The various system5 IPC resources (semaphores, messages and shared
  *	memory are initialised
  */
- 
+
+struct ipc_msg_context *ipc_msg_context;
 static int __init ipc_init(void)
 {
+	ipc_msg_context = alloc_ipc_msg_context(GFP_KERNEL);
+
 	sem_init();
-	msg_init();
+	msg_init(ipc_msg_context);
 	shm_init();
 	return 0;
 }
ipc/util.h renamed from ipc/util.h~sysv-container +1 -1
diff -puN ipc/util.h~sysv-container ipc/util.h
@@ -12,7 +12,7 @@
 #define SEQ_MULTIPLIER	(IPCMNI)
 
 void sem_init (void);
-void msg_init (void);
+void msg_init (struct ipc_msg_context *context);
 void shm_init (void);
 
 struct seq_file;
include/linux/sched.h renamed from include/linux/sched.h~sysv-container +1 -0
diff -puN include/linux/sched.h~sysv-container include/linux/sched.h
@@ -793,6 +793,7 @@ struct task_struct {
 	int link_count, total_link_count;
 /* ipc stuff */
 	struct sysv_sem sysvsem;
+	struct ipc_msg_context *ipc_msg_context;
 /* CPU-specific state of this task */
 	struct thread_struct thread;
 /* filesystem information */
include/linux/ipc.h renamed from include/linux/ipc.h~sysv-container +12 -0
diff -puN include/linux/ipc.h~sysv-container include/linux/ipc.h
@@ -2,6 +2,9 @@
 #define _LINUX_IPC_H
 
 #include <linux/types.h>
+#include <linux/kref.h>
+#include <asm/atomic.h>
+#include <asm/semaphore.h>
 
 #define IPC_PRIVATE ((__kernel_key_t) 0)  
 
@@ -83,6 +86,15 @@ struct ipc_ids {
 	struct ipc_id_ary* entries;
 };
 
+struct ipc_msg_context {
+	atomic_t msg_bytes;
+	atomic_t msg_hdrs;
+
+	struct ipc_ids msg_ids;
+	struct kref count;
+};
+
+extern struct ipc_msg_context *alloc_ipc_msg_context(gfp_t flags);
 #endif /* __KERNEL__ */
 
 #endif /* _LINUX_IPC_H */

diff -puN fs/proc/kmsg.c~sysv-container fs/proc/kmsg.c
diff -puN include/linux/tipc_config.h~sysv-container include/linux/tipc_config.h diff -puN fs/ufs/util.c~sysv-container fs/ufs/util.c
diff -puN fs/ufs/util.h~sysv-container fs/ufs/util.h
diff -puN include/linux/cn_proc.h~sysv-container include/linux/cn_proc.h

kernel/fork.c renamed from kernel/fork.c~sysv-container +5 -0
diff -puN kernel/fork.c~sysv-container kernel/fork.c
@@ -1184,6 +1184,11 @@ static task_t *copy_process(unsigned lon
 	}
 	attach_pid(p, PIDTYPE_TGID, p->tgid);
 	attach_pid(p, PIDTYPE_PID, p->pid);
+	{ // this extern will go away when we start to dynamically
+	  // allocate these
+		extern struct ipc_msg_context ipc_msg_context;
+		p->ipc_msg_context = current->ipc_msg_context;
+	}
 
 	nr_threads++;
 	total_forks++;

_

#21 lore raw

Re: sysctls inside containers

In reply to Dave Hansen
Quoted 5 lines · Dave Hansen
Quoted 4 lines · Eric W. Biederman
Quoted 3 lines

I don't see an immediately clear solution on how to containerize sysctls properly. The entire construct seems to be built around getting data from in and out of global variables and into /proc files.

Quoted 8 lines
Quoted 2 lines

I successfully handled pid_max. So while it is awkward you can get per task values in and out if sysctl.

This:

http://www.kernel.org/git/gitweb.cgi?p=linux/kernel/git/ebiederm/linux-2.6-ns.git;a=commitdiff;h=1150082e0bae41a3621043b4c5ce15e9112884fa

sir, is a hack :)

I did mention awkward!

Quoted 6 lines

We can't possibly do that for each and every sysctl variable. It would mean about fourteen billion duplicated _conv() functions.

I'm wondering if, instead of having the .data field in that table, we can have a function pointer which, when called, gives a pointer to the data. I'll give it a shot.

Sounds like a step in the right direction. In the context of _conv() functions it would also be nice if the limit checking between the sys_sysctl path and the proc path were the same.

We definitely need some infrastructure cleanups if we are going to do much with sysctl.

Eric

#22 lore raw

Re: sysctls inside containers

In reply to Dave Hansen
Quoted 21 lines

Well, I at least got to the point of seeing how the sysctls interact when I tried to containerize them. Eric, I think the idea of the sysv code being nicely and completely isolated is pretty much gone, due to their connection to sysctls. I think I'll go back and just isolate the "struct ipc_ids" portion. We can do the accounting bits later.

The patches I have will isolate the IDs, but I'm not sure how much sense that makes without doing the things like the shm_tot variable. Does anybody think we need to go after sysctls first, perhaps? Or, is this a problem graph with cycles in it? :)

I don't see an immediately clear solution on how to containerize sysctls properly. The entire construct seems to be built around getting data from in and out of global variables and into /proc files.

We obviously want to be rid of many of these global variables. So, does it make sense to introduce different classes of sysctls, at least internally? There are probably just two types: global, writable only from the root container and container-private. Does it make sense to have _both_? Perhaps a sysadmin

Quoted 8 lines

Eric, can you think of how you would represent these in the hierarchical container model? How would they work?

On another note, after messing with putting data in the init_task for these things, I'm a little more convinced that we aren't going to want to clutter up the task_struct with all kinds of containerized resources, _plus_ make all of the interfaces to share or unshare each of those. That global 'struct container' is looking a bit more attractive.

After checking proposed yours, Eric and vserver solutions, I must say that these all are hacks.
If we want to virtualize sysctl we need to do it in honest way:
multiple sysctl trees, which can be different in different namespaces.
For example, one namespace can see /proc/sys/net/route and the other one not. Introducing helpers/handlers etc. doesn't fully solve the problem of visibility of different parts of sysctl tree and it's access rights. Another example, the same network device can present in 2 namespaces and these are dynamically(!) created entries in sysctl.

So we need to address actually 2 issues:
- ability to limit parts of sysctl tree visibility to namespace
- ability to limit/change sysctl access rights in namespace

You can check OpenVZ for cloning sysctl tree code. It is not clean, nor elegant, but can be cleanuped.

Thanks,
Kirill

#23 lore raw

Re: sysctls inside containers

Quoted 6 lines · Kirill Korotaev

After checking proposed yours, Eric and vserver solutions, I must say that these all are hacks.
If we want to virtualize sysctl we need to do it in honest way:
multiple sysctl trees, which can be different in different namespaces. For example, one namespace can see /proc/sys/net/route and the other one not.

At least a different copy of /proc/sys/net/route :)

Quoted 2 lines

Introducing helpers/handlers etc. doesn't fully solve the problem of visibility of different parts of sysctl tree and it's access rights.

I need to look a little deeper but I think if we add two helper
functions: One that returns the address of a value based upon our state, and another that returns a subdirectory based upon our state I think we should be ok.

Both of them taking a struct task_struct argument so we can make the decision what to show based upon the calling process.

Quoted 9 lines

Another example, the same network device can present in 2 namespaces and these are dynamically(!) created entries in sysctl.

So we need to address actually 2 issues:
- ability to limit parts of sysctl tree visibility to namespace
- ability to limit/change sysctl access rights in namespace

You can check OpenVZ for cloning sysctl tree code. It is not clean, nor elegant, but can be cleanuped.

Sounds like a decent idea.

What I have found so far with access rights is that if you dig deep enough you don't need magic to make it safe. But this may only be because I have not hit something that is fundamentally different.

Eric

#24 lore raw

Re: sysctls inside containers

In reply to Dave Hansen
Quoted 5 lines

On another note, after messing with putting data in the init_task for these things, I'm a little more convinced that we aren't going to want to clutter up the task_struct with all kinds of containerized resources, _plus_ make all of the interfaces to share or unshare each of those. That global 'struct container' is looking a bit more attractive.

BTW, Dave,

have you noticed that ipc/mqueue.c uses netlink to send messages? This essentially means that they are tied as well...

Thanks,
Kirill

#25 lore raw

Re: sysctls inside containers

Quoted 9 lines · Kirill Korotaev
Quoted 5 lines

On another note, after messing with putting data in the init_task for these things, I'm a little more convinced that we aren't going to want to clutter up the task_struct with all kinds of containerized resources, _plus_ make all of the interfaces to share or unshare each of those. That global 'struct container' is looking a bit more attractive.

BTW, Dave,

have you noticed that ipc/mqueue.c uses netlink to send messages? This essentially means that they are tied as well...

Yes, netlink is something to be considered in the great untangling.

However for a sysvipc namespace ipc/mqueue.c is something that doesn't need to be handled because that is the implementation of posix message queues not sysv ipc.

I think I succeeded in untagling the worst of netlink in my proof of concept implementation, but certainly there is more todo.

Eric

#26 lore raw

Re: sysctls inside containers

In reply to Kirill Korotaev
Quoted 8 lines · Kirill Korotaev
Quoted 5 lines

On another note, after messing with putting data in the init_task for these things, I'm a little more convinced that we aren't going to want to clutter up the task_struct with all kinds of containerized resources, _plus_ make all of the interfaces to share or unshare each of those. That global 'struct container' is looking a bit more attractive.

have you noticed that ipc/mqueue.c uses netlink to send messages? This essentially means that they are tied as well...

Nope, I missed that.

But, netlink is probably a completely separate issue, at least for now. I'm sure we're going to have "container leaks" and boundary violations for a long time, but I'll add netlink to the list.

-- Dave