DRM cgroup controller

Since many parts of the DRM subsystem has vendor-specific implementations, we introduce mechanisms for vendor to register their specific resources and control files to the DRM cgroup subsystem. A vendor will register itself with the DRM cgroup subsystem first before registering individual DRM devices to the cgroup subsystem.

In addition to the cgroup_subsys_state that is common to all DRM devices, a device-specific state is introduced and it is allocated according to the vendor of the device.

So, I'm still pretty negative about adding drm controller at this point. There isn't enough of common resource model defined yet and until that gets sorted out I think it's in the best interest of everyone involved to keep it inside drm or specific driver proper.

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution? When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup?  Let say a user

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution?  When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Thanks Tejun,Eric and Christian for your replies.

We want GPUs resource management to work seamlessly with containers and container orchestration. With the Intel / bpf based approach this is not possible.

From your response we gather the following. GPU resources need to be abstracted. We will send a new proposal in same vein. Our current thinking is to start with a single abstracted resource and build a framework that can be expanded to include additional resources. We plan to start with “GPU cores”. We believe all GPUs have some concept of cores or compute unit.

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup?  Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation.  AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution?  When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing?  It's not difficult to hook into cgroup for identification purposes.

Thanks.

Thanks Tejun,Eric and Christian for your replies.

We want GPUs resource management to work seamlessly with containers and container orchestration. With the Intel / bpf based approach this is not possible.

From your response we gather the following. GPU resources need to be abstracted. We will send a new proposal in same vein. Our current thinking is to start with a single abstracted resource and build a framework that can be expanded to include additional resources. We plan to start with “GPU cores”. We believe all GPUs have some concept of cores or compute unit.

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup?  Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation.  AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution?  When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing?  It's not difficult to hook into cgroup for identification purposes.

Thanks.

Signature · 15 lines

tejun
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx


amd-gfx Info Page - freedesktop.org
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives.. Using amd-gfx: To post a message to all the list members, send email to [email protected]. You can subscribe to the list, or change your existing subscription, in the sections below.
    
_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

I think a more abstract property "% of GPU (processing power)" might be a more universal approach. One can then implement that through subdividing the resources or timeslicing them, depending on the GPU topology.

Leasing 1/8th, 1/4th or 1/2 of the GPU would probably be the most applicable to cloud provider usecases, too. At least that's what I see done for the CPUs today.

That combined with the "GPU memory usable" property should be a good starting point to start subdividing the GPU resources for multiple users.

Regards, Joonas

Quoted 61 lines

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution? When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

Thanks.

--
tejun
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

amd-gfx Info Page - freedesktop.org
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives.. Using amd-gfx: To post a message to all the list members, send email to [email protected]. You can subscribe to the list, or change your existing subscription, in the sections below.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Quoted 8 lines

I think a more abstract property "% of GPU (processing power)" might be a more universal approach. One can then implement that through subdividing the resources or timeslicing them, depending on the GPU topology.

Leasing 1/8th, 1/4th or 1/2 of the GPU would probably be the most applicable to cloud provider usecases, too. At least that's what I see done for the CPUs today.

I think there are opportunities to slice the gpu in more than one way (similar to the way it is done for cpu.) We can potentially frame resources as continuous or discrete. Percentage definitely fits well for continuous measurements such as time/time slices but I think there are places for discrete units such as core counts as well.

Regards,
Kenny

Quoted 67 lines

That combined with the "GPU memory usable" property should be a good starting point to start subdividing the GPU resources for multiple users.

Regards, Joonas

Quoted 61 lines

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution? When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

Thanks.

--
tejun
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

amd-gfx Info Page - freedesktop.org
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives.. Using amd-gfx: To post a message to all the list members, send email to [email protected]. You can subscribe to the list, or change your existing subscription, in the sections below.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

So we can only choose the lowest common denominator, right?

Any core count out of total core count should translate nicely into a fraction, so what would be the problem with percentage amounts?

Quoted 59 lines

Quoted 59 lines

That combined with the "GPU memory usable" property should be a good starting point to start subdividing the GPU resources for multiple users.

Regards, Joonas

Quoted 53 lines

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution? When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

Thanks.

--
tejun
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Quoted 8 lines

Quoted 8 lines

Quoted 8 lines

amd-gfx Info Page - freedesktop.org
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives.. Using amd-gfx: To post a message to all the list members, send email to [email protected]. You can subscribe to the list, or change your existing subscription, in the sections below.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

Quoted 4 lines · Joonas Lahtinen

So we can only choose the lowest common denominator, right?

Any core count out of total core count should translate nicely into a fraction, so what would be the problem with percentage amounts?

I don't think having an abstracted resource necessarily equate 'lowest'. The issue with percentage is the lack of precision. If you look at cpuset cgroup, you can see the specification can be very precise:

# /bin/echo 1-4,6 > cpuset.cpus -> set cpus list to cpus 1,2,3,4,6
(https://www.kernel.org/doc/Documentation/cgroup-v1/cpusets.txt)

The driver can translate something like this to core count and then to percentage and handle accordingly while the reverse is not possible. (You can't tell which set of CUs/EUs a user want from a percentage request.) It's also not clear to me, from user/application/admin/resource management perspective, how the base core counts of a GPU is relevant to the workload (since percentage is a 'relative' quantity.) For example, let say a workload wants to use 256 'cores', does it matter if that workload is put on a GPU with 1024 cores or a GPU with 4096 cores total?

I am not dismissing the possible need for percentage. I just think there should be a way to accommodate more than just the 'lowest'.

Regards,
Kennny

Quoted 67 lines

Quoted 67 lines

Quoted 67 lines

That combined with the "GPU memory usable" property should be a good starting point to start subdividing the GPU resources for multiple users.

Regards, Joonas

Quoted 61 lines

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution? When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

Thanks.

--
tejun
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

amd-gfx Info Page - freedesktop.org
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives.. Using amd-gfx: To post a message to all the list members, send email to [email protected]. You can subscribe to the list, or change your existing subscription, in the sections below.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

I think Joonas is describing something closer in
design to the cgroup-v2 "cpu" controller, which partitions the general time/usage allocated to via cgroup; afaiu, "cpu" doesn't really care which specific core the tasks run on, just the relative weights that determine how much time they get to run on any of the cores.

It sounds like with your hardware, your kernel driver is able to specify exactly which subset of GPU EU's a specific GPU context winds up running on. However I think there are a lot of platforms that don't allow that kind of low-level control. E.g., I don't think we can do that on Intel hardware; we have a handful of high-level GPU engines that we can submit different types of batchbuffers to (render, blitter, media, etc.). What we can do is use GPU preemption to limit how much time specific GPU contexts get to run on the render engine before the engine is reclaimed for use by a different context.

Using a %gputime approach like Joonas is suggesting could be handled in a driver by reserving specific subsets of EU's on hardware like yours that's capable of doing that, whereas it could be mostly handled on other types of hardware via GPU engine preemption.

I think either approach "gpu_euset" or "%gputime" should map well to a cgroup controller implementation. Granted, neither one solves the specific use case I was working on earlier this year where we need unfair (starvation-okay) scheduling that will run contexts strictly according to priority (i.e., lower priority contexts will never run at all unless all higher priority contexts have completed all of their submitted work), but that's a pretty specialized use case that we'll probably need to handle in a different manner anyway.

Matt

Quoted 71 lines

Regards,
Kennny

Quoted 67 lines

Quoted 67 lines

Quoted 67 lines

That combined with the "GPU memory usable" property should be a good starting point to start subdividing the GPU resources for multiple users.

Regards, Joonas

Quoted 61 lines

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution? When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

Thanks.

--
tejun
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

amd-gfx Info Page - freedesktop.org
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives.. Using amd-gfx: To post a message to all the list members, send email to [email protected]. You can subscribe to the list, or change your existing subscription, in the sections below.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

--
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795

Hey Matt,

Quoted 5 lines · Matt Roper

I think Joonas is describing something closer in
design to the cgroup-v2 "cpu" controller, which partitions the general time/usage allocated to via cgroup; afaiu, "cpu" doesn't really care which specific core the tasks run on, just the relative weights that determine how much time they get to run on any of the cores.

Depending on the level of optimization one wants to do, I think people care about which cpu core a task runs on. Modern processors are no longer a monolithic 'thing'. At least for AMD, there are multiple cpus on a core complex (CCX), multiple CCX on a die, and multiple dies on a processor. A task running on cpu 0 and cpu 1 on die 0 will behave very differently from a task running on core 0s on die 0 and die 1 on the same socket.
(https://en.wikichip.org/wiki/amd/microarchitectures/zen#Die-die_memory_latencies)

It's not just an AMD thing either. Here is an open issue on Intel's architecture:
https://github.com/kubernetes/kubernetes/issues/67355

and a proposed solution using cpu affinity
https://github.com/kubernetes/community/blob/630acc487c80e4981a232cdd8400eb8207119788/keps/sig-node/0030-qos-class-cpu-affinity.md#proposal
(by one of your colleagues.)

The time-based sharing below is also something we are thinking about, but it's personally not as exciting as the resource-based sharing for me because the time-share use case has already been addressed by our SRIOV/virtualization products. We can potentially have different level of time sharing using cgroup though (in addition to SRIOV), potentially trading efficiency against isolation. That said, I think the time-based approach maybe orthogonal to the resource-based approach (orthogonal in the sense that both are needed depending on the usage.)

Regards,
Kenny

Quoted 106 lines

It sounds like with your hardware, your kernel driver is able to specify exactly which subset of GPU EU's a specific GPU context winds up running on. However I think there are a lot of platforms that don't allow that kind of low-level control. E.g., I don't think we can do that on Intel hardware; we have a handful of high-level GPU engines that we can submit different types of batchbuffers to (render, blitter, media, etc.). What we can do is use GPU preemption to limit how much time specific GPU contexts get to run on the render engine before the engine is reclaimed for use by a different context.

Using a %gputime approach like Joonas is suggesting could be handled in a driver by reserving specific subsets of EU's on hardware like yours that's capable of doing that, whereas it could be mostly handled on other types of hardware via GPU engine preemption.

I think either approach "gpu_euset" or "%gputime" should map well to a cgroup controller implementation. Granted, neither one solves the specific use case I was working on earlier this year where we need unfair (starvation-okay) scheduling that will run contexts strictly according to priority (i.e., lower priority contexts will never run at all unless all higher priority contexts have completed all of their submitted work), but that's a pretty specialized use case that we'll probably need to handle in a different manner anyway.

Matt

Quoted 71 lines

Regards,
Kennny

Quoted 67 lines

Quoted 67 lines

Quoted 67 lines

That combined with the "GPU memory usable" property should be a good starting point to start subdividing the GPU resources for multiple users.

Regards, Joonas

Quoted 61 lines

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution? When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

Thanks.

--
tejun
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

amd-gfx Info Page - freedesktop.org
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives.. Using amd-gfx: To post a message to all the list members, send email to [email protected]. You can subscribe to the list, or change your existing subscription, in the sections below.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

--
Matt Roper
Graphics Software Engineer
IoTG Platform Enabling & Development
Intel Corporation
(916) 356-2795

Quoting Ho, Kenny (2018-11-27 17:41:17)

Quoted 9 lines · Joonas Lahtinen

Quoted 8 lines

I think a more abstract property "% of GPU (processing power)" might be a more universal approach. One can then implement that through subdividing the resources or timeslicing them, depending on the GPU topology.

Leasing 1/8th, 1/4th or 1/2 of the GPU would probably be the most applicable to cloud provider usecases, too. At least that's what I see done for the CPUs today.

Quoted 1 line

I think there are opportunities to slice the gpu in more than one way (similar to the way it is done for cpu.) We can potentially frame resources as continuous or discrete. Percentage definitely fits well for continuous measurements such as time/time slices but I think there are places for discrete units such as core counts as well.

I think the ask in return to the early series from Intal was to agree on the variables that could be common to all of DRM subsystem.

So we can only choose the lowest common denominator, right?

Any core count out of total core count should translate nicely into a fraction, so what would be the problem with percentage amounts?

Regards, Joonas

Quoted 63 lines

Regards,
Kenny

Quoted 60 lines

That combined with the "GPU memory usable" property should be a good starting point to start subdividing the GPU resources for multiple users.

Regards, Joonas

Quoted 54 lines

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution? When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

Thanks.

--
tejun
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

amd-gfx Info Page - freedesktop.org
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives.. Using amd-gfx: To post a message to all the list members, send email to [email protected]. You can subscribe to the list, or change your existing subscription, in the sections below.

Quoted 4 lines

Quoted 4 lines

Quoted 4 lines

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Quoted 14 lines

Quoting Ho, Kenny (2018-11-27 17:41:17)

Quoted 10 lines · Joonas Lahtinen

Quoted 8 lines

I think a more abstract property "% of GPU (processing power)" might be a more universal approach. One can then implement that through subdividing the resources or timeslicing them, depending on the GPU topology.

Leasing 1/8th, 1/4th or 1/2 of the GPU would probably be the most applicable to cloud provider usecases, too. At least that's what I see done for the CPUs today.

I think there are opportunities to slice the gpu in more than one way (similar to the way it is done for cpu.) We can potentially frame resources as continuous or discrete. Percentage definitely fits well for continuous measurements such as time/time slices but I think there are places for discrete units such as core counts as well.

I think the ask in return to the early series from Intal was to agree on the variables that could be common to all of DRM subsystem.

Quoted 4 lines

So we can only choose the lowest common denominator, right?

Any core count out of total core count should translate nicely into a fraction, so what would be the problem with percentage amounts?

How would you handle overcommitment with a percentage? That is, more than 100% of the GPU cores assigned to cgroups. Which cgroups end up sharing cores would be up to chance.

If we allow specifying a set of GPU cores, we can be more specific in assigning and sharing resources between cgroups.

Regards,
  Felix

Quoted 74 lines

Regards, Joonas

Quoted 67 lines

Regards,
Kenny

Quoted 64 lines

That combined with the "GPU memory usable" property should be a good starting point to start subdividing the GPU resources for multiple users.

Regards, Joonas

Quoted 58 lines

Your feedback is highly appreciated.

Best Regards,
Harish

From: amd-gfx <[email protected]> on behalf of Tejun Heo <[email protected]>
Sent: Tuesday, November 20, 2018 5:30 PM
To: Ho, Kenny
Cc: [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [PATCH RFC 2/5] cgroup: Add mechanism to register vendor specific DRM devices

Hello,

Quoted 2 lines · Kenny

By this reply, are you suggesting that vendor specific resources will never be acceptable to be managed under cgroup? Let say a user

I wouldn't say never but whatever which gets included as a cgroup controller should have clearly defined resource abstractions and the control schemes around them including support for delegation. AFAICS, gpu side still seems to have a long way to go (and it's not clear whether that's somewhere it will or needs to end up).

Quoted 8 lines

want to have similar functionality as what cgroup is offering but to manage vendor specific resources, what would you suggest as a solution? When you say keeping vendor specific resource regulation inside drm or specific drivers, do you mean we should replicate the cgroup infrastructure there or do you mean either drm or specific driver should query existing hierarchy (such as device or perhaps cpu) for the process organization information?

Quoted 4 lines

To put the questions in more concrete terms, let say a user wants to expose certain part of a gpu to a particular cgroup similar to the way selective cpu cores are exposed to a cgroup via cpuset, how should we go about enabling such functionality?

Do what the intel driver or bpf is doing? It's not difficult to hook into cgroup for identification purposes.

Thanks.

--
tejun
_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

amd-gfx Info Page - freedesktop.org
lists.freedesktop.org
To see the collection of prior postings to the list, visit the amd-gfx Archives.. Using amd-gfx: To post a message to all the list members, send email to [email protected]. You can subscribe to the list, or change your existing subscription, in the sections below.

_______________________________________________
Intel-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/intel-gfx

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Since many parts of the DRM subsystem has vendor-specific implementations, we introduce mechanisms for vendor to register their specific resources and control files to the DRM cgroup subsystem. A vendor will register itself with the DRM cgroup subsystem first before registering individual DRM devices to the cgroup subsystem.

In addition to the cgroup_subsys_state that is common to all DRM devices, a device-specific state is introduced and it is allocated according to the vendor of the device.

Change-Id I908ee6975ea0585e4c30eafde4599f87094d8c65

Signed-off-by Kenny Ho <[email protected]>

---
  include/drm/drm_cgroup.h      | 39 ++++++++++++++++
  include/drm/drmcgrp_vendors.h |  7 +++
  include/linux/cgroup_drm.h    | 26 +++++++++++
  kernel/cgroup/drm.c           | 84 +++++++++++++++++++++++++++++++++++
  4 files changed, 156 insertions(+)

create mode 100644 include/drm/drm_cgroup.h create mode 100644 include/drm/drmcgrp_vendors.h

include/drm/drm_cgroup.h new file +39 -0

new file mode 100644
index 000000000000..26cbea7059a6
@@ -0,0 +1,39 @@
+/* SPDX-License-Identifier: MIT
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ */
+#ifndef __DRM_CGROUP_H__
+#define __DRM_CGROUP_H__
+
+#define DRMCGRP_VENDOR(_x) _x ## _drmcgrp_vendor_id,
+enum drmcgrp_vendor_id {
+#include <drm/drmcgrp_vendors.h>
+	DRMCGRP_VENDOR_COUNT,
+};
+#undef DRMCGRP_VENDOR
+
+#define DRMCGRP_VENDOR(_x) extern struct drmcgrp_vendor _x ## _drmcgrp_vendor;
+#include <drm/drmcgrp_vendors.h>
+#undef DRMCGRP_VENDOR
+
+
+
+#ifdef CONFIG_CGROUP_DRM
+
+extern struct drmcgrp_vendor *drmcgrp_vendors[];
+
+int drmcgrp_register_vendor(struct drmcgrp_vendor *vendor, enum drmcgrp_vendor_id id);
+int drmcgrp_register_device(struct drm_device *device, enum drmcgrp_vendor_id id);
+
+#else
+static int drmcgrp_register_vendor(struct drmcgrp_vendor *vendor, enum drmcgrp_vendor_id id)
+{
+	return 0;
+}
+
+static int drmcgrp_register_device(struct drm_device *device, enum drmcgrp_vendor_id id)
+{
+	return 0;
+}
+
+#endif /* CONFIG_CGROUP_DRM */
+#endif /* __DRM_CGROUP_H__ */

include/drm/drmcgrp_vendors.h new file +7 -0

new file mode 100644
index 000000000000..b04d8649851b
@@ -0,0 +1,7 @@
+/* SPDX-License-Identifier: MIT
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ */
+#if IS_ENABLED(CONFIG_CGROUP_DRM)
+
+
+#endif

include/linux/cgroup_drm.h +26 -0

index 79ab38b0f46d..a776662d9593 100644
@@ -6,10 +6,36 @@
 
  #ifdef CONFIG_CGROUP_DRM
 
+#include <linux/mutex.h>
  #include <linux/cgroup.h>
+#include <drm/drm_file.h>
+#include <drm/drm_cgroup.h>
+
+/* limit defined per the way drm_minor_alloc operates */
+#define MAX_DRM_DEV (64 * DRM_MINOR_RENDER)
+
+struct drmcgrp_device {
+	enum drmcgrp_vendor_id	vid;
+	struct drm_device	*dev;
+	struct mutex		mutex;
+};
+
+/* vendor-common resource counting goes here */
+/* this struct should be included in the vendor specific resource */
+struct drmcgrp_device_resource {
+	struct drmcgrp_device	*ddev;
+};
+
+struct drmcgrp_vendor {
+	struct cftype *(*get_cftypes)(void);
+	struct drmcgrp_device_resource *(*alloc_dev_resource)(void);
+	void (*free_dev_resource)(struct drmcgrp_device_resource *dev_resource);
+};
+
 
  struct drmcgrp {
  	struct cgroup_subsys_state	css;
+	struct drmcgrp_device_resource	*dev_resources[MAX_DRM_DEV];
  };
 
  static inline struct drmcgrp *css_drmcgrp(struct cgroup_subsys_state *css)

kernel/cgroup/drm.c +74 -0

index d9e194b9aead..f9630cc389bc 100644
@@ -1,8 +1,30 @@
  // SPDX-License-Identifier: MIT
  // Copyright 2018 Advanced Micro Devices, Inc.
+#include <linux/export.h>
  #include <linux/slab.h>
  #include <linux/cgroup.h>
+#include <linux/fs.h>
+#include <linux/seq_file.h>
+#include <linux/mutex.h>
  #include <linux/cgroup_drm.h>
+#include <drm/drm_device.h>
+#include <drm/drm_cgroup.h>
+
+/* generate an array of drm cgroup vendor pointers */
+#define DRMCGRP_VENDOR(_x)[_x ## _drmcgrp_vendor_id] = NULL,
+struct drmcgrp_vendor *drmcgrp_vendors[] = {
+#include <drm/drmcgrp_vendors.h>
+};
+#undef DRMCGRP_VENDOR
+EXPORT_SYMBOL(drmcgrp_vendors);
+
+static DEFINE_MUTEX(drmcgrp_mutex);
+
+/* indexed by drm_minor for access speed */
+static struct drmcgrp_device	*known_drmcgrp_devs[MAX_DRM_DEV];
+
+static int max_minor;
+
 
  static u64 drmcgrp_test_read(struct cgroup_subsys_state *css,
  					struct cftype *cft)

@@ -13,6 +35,12 @@ static u64 drmcgrp_test_read(struct cgroup_subsys_state *css,
  static void drmcgrp_css_free(struct cgroup_subsys_state *css)
  {
  	struct drmcgrp *drmcgrp = css_drmcgrp(css);
+	int i;
+
+	for (i = 0; i <= max_minor; i++) {
+		if (drmcgrp->dev_resources[i] != NULL)
+			drmcgrp_vendors[known_drmcgrp_devs[i]->vid]->free_dev_resource(drmcgrp->dev_resources[i]);
+	}
 
  	kfree(css_drmcgrp(css));
  }

@@ -21,11 +49,27 @@ static struct cgroup_subsys_state *
  drmcgrp_css_alloc(struct cgroup_subsys_state *parent_css)
  {
  	struct drmcgrp *drmcgrp;
+	int i;
 
  	drmcgrp = kzalloc(sizeof(struct drmcgrp), GFP_KERNEL);
  	if (!drmcgrp)
  		return ERR_PTR(-ENOMEM);
 
+	for (i = 0; i <= max_minor; i++) {
+		if (known_drmcgrp_devs[i] != NULL) {
+			struct drmcgrp_device_resource *ddr =
+				drmcgrp_vendors[known_drmcgrp_devs[i]->vid]->alloc_dev_resource();
+
+			if (IS_ERR(ddr)) {
+				drmcgrp_css_free(&drmcgrp->css);
+				return ERR_PTR(-ENOMEM);
+			}
+
+			drmcgrp->dev_resources[i] = ddr;
+			drmcgrp->dev_resources[i]->ddev = known_drmcgrp_devs[i];
+		}
+	}
+
  	return &drmcgrp->css;
  }
 

@@ -44,3 +88,43 @@ struct cgroup_subsys drm_cgrp_subsys = {
  	.legacy_cftypes	= files,
  	.dfl_cftypes	= files,
  };
+
+int drmcgrp_register_vendor(struct drmcgrp_vendor *vendor, enum drmcgrp_vendor_id id)
+{
+	int rc = 0;
+	struct cftype *cfts;
+
+	// TODO: root css created before any registration
+	if (drmcgrp_vendors[id] == NULL) {
+		drmcgrp_vendors[id] = vendor;
+		cfts = vendor->get_cftypes();
+		if (cfts != NULL)
+			rc = cgroup_add_legacy_cftypes(&drm_cgrp_subsys, cfts);
+	}
+	return rc;
+}
+EXPORT_SYMBOL(drmcgrp_register_vendor);
+
+
+int drmcgrp_register_device(struct drm_device *dev, enum drmcgrp_vendor_id id)
+{
+	struct drmcgrp_device *ddev;
+
+	ddev = kzalloc(sizeof(struct drmcgrp_device), GFP_KERNEL);
+	if (!ddev)
+		return -ENOMEM;
+
+	mutex_lock(&drmcgrp_mutex);
+
+	ddev->vid = id;
+	ddev->dev = dev;

Change-Id Ib66c44ac1b1c367659e362a2fc05b6fbb3805876

Signed-off-by Kenny Ho <[email protected]>

---
  drivers/gpu/drm/amd/amdgpu/Makefile         |  3 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_device.c  |  7 ++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.c | 37 +++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.h | 19 +++++++++++
  include/drm/drmcgrp_vendors.h               |  1 +
  5 files changed, 67 insertions(+)

create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.c create mode 100644 drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.h

drivers/gpu/drm/amd/amdgpu/Makefile +3 -0

index 138cb787d27e..5cf8048f2d75 100644
@@ -186,4 +186,7 @@ amdgpu-y += $(AMD_DISPLAY_FILES)
 
  endif
 
+#DRM cgroup controller
+amdgpu-y += amdgpu_drmcgrp.o
+
  obj-$(CONFIG_DRM_AMDGPU)+= amdgpu.o

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +1 -0

index 30bc345d6fdf..ad0373f83ed3 100644
@@ -33,6 +33,7 @@
  #include <drm/drm_crtc_helper.h>
  #include <drm/drm_atomic_helper.h>
  #include <drm/amdgpu_drm.h>
+#include <drm/drm_cgroup.h>
  #include <linux/vgaarb.h>
  #include <linux/vga_switcheroo.h>
  #include <linux/efi.h>

drivers/gpu/drm/amd/amdgpu/amdgpu_device.c +0 -0

  	return 0;
 
  failed:

drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.c new file +37 -0

new file mode 100644
index 000000000000..ed8aac17769c
@@ -0,0 +1,37 @@
+// SPDX-License-Identifier: MIT
+// Copyright 2018 Advanced Micro Devices, Inc.
+#include <linux/slab.h>
+#include <linux/cgroup_drm.h>
+#include <drm/drm_device.h>
+#include "amdgpu_drmcgrp.h"
+
+struct cftype files[] = {
+	{ } /* terminate */
+};
+
+struct cftype *drmcgrp_amd_get_cftypes(void)
+{
+	return files;
+}
+
+struct drmcgrp_device_resource *amd_drmcgrp_alloc_dev_resource(void)
+{
+	struct amd_drmcgrp_dev_resource *a_ddr;
+
+	a_ddr = kzalloc(sizeof(struct amd_drmcgrp_dev_resource), GFP_KERNEL);
+	if (!a_ddr)
+		return ERR_PTR(-ENOMEM);
+
+	return &a_ddr->ddr;
+}
+
+void amd_drmcgrp_free_dev_resource(struct drmcgrp_device_resource *ddr)
+{
+	kfree(ddr_amdddr(ddr));
+}
+
+struct drmcgrp_vendor amd_drmcgrp_vendor = {
+	.get_cftypes = drmcgrp_amd_get_cftypes,
+	.alloc_dev_resource = amd_drmcgrp_alloc_dev_resource,
+	.free_dev_resource = amd_drmcgrp_free_dev_resource,
+};

drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.h new file +19 -0

new file mode 100644
index 000000000000..e2934b7a49f5
@@ -0,0 +1,19 @@
+/* SPDX-License-Identifier: MIT
+ * Copyright 2018 Advanced Micro Devices, Inc.
+ */
+#ifndef _AMDGPU_DRMCGRP_H
+#define _AMDGPU_DRMCGRP_H
+
+#include <linux/cgroup_drm.h>
+
+/* for AMD specific DRM resources */
+struct amd_drmcgrp_dev_resource {
+	struct drmcgrp_device_resource ddr;
+};
+
+static inline struct amd_drmcgrp_dev_resource *ddr_amdddr(struct drmcgrp_device_resource *ddr)
+{
+	return ddr ? container_of(ddr, struct amd_drmcgrp_dev_resource, ddr) : NULL;
+}
+
+#endif	/* _AMDGPU_DRMCGRP_H */

include/drm/drmcgrp_vendors.h +0 -0

index b04d8649851b..6cfbf1825344 100644

Account for the number of command submitted to amdgpu by type on a per cgroup basis, for the purpose of profiling/monitoring applications.

Quoted 2 lines · Kenny Ho

Account for the number of command submitted to amdgpu by type on a per cgroup basis, for the purpose of profiling/monitoring applications.

For profiling other drivers, I've used perf tracepoints, which let you get useful timelines of multiple events in the driver. Have you made use of this stat for productive profiling?

_______________________________________________
amd-gfx mailing list [email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Am 20.11.18 um 21:57 schrieb Eric Anholt:

Quoted 4 lines

Quoted 2 lines · Kenny Ho

Account for the number of command submitted to amdgpu by type on a per cgroup basis, for the purpose of profiling/monitoring applications.

Quoted 3 lines

For profiling other drivers, I've used perf tracepoints, which let you get useful timelines of multiple events in the driver. Have you made use of this stat for productive profiling?

Yes, but this is not related to profiling at all.

What we want to do is to limit the resource usage of processes.

Quoted 8 lines · Christian König

Am 20.11.18 um 21:57 schrieb Eric Anholt:

Quoted 7 lines

Quoted 2 lines · Kenny Ho

Account for the number of command submitted to amdgpu by type on a per cgroup basis, for the purpose of profiling/monitoring applications.

For profiling other drivers, I've used perf tracepoints, which let you get useful timelines of multiple events in the driver. Have you made use of this stat for productive profiling?

Quoted 3 lines

Yes, but this is not related to profiling at all.

What we want to do is to limit the resource usage of processes.

That sounds great, and something I'd be interested in for vc4. However, as far as I saw explained here, this patch doesn't let you limit resource usage of a process and is only useful for "profiling/monitoring" so I'm wondering how it is useful for that purpose.

Am 23.11.18 um 18:36 schrieb Eric Anholt:

Quoted 11 lines

Quoted 10 lines · Christian König

Am 20.11.18 um 21:57 schrieb Eric Anholt:

Quoted 6 lines

Quoted 2 lines · Kenny Ho

Account for the number of command submitted to amdgpu by type on a per cgroup basis, for the purpose of profiling/monitoring applications.

For profiling other drivers, I've used perf tracepoints, which let you get useful timelines of multiple events in the driver. Have you made use of this stat for productive profiling?

Yes, but this is not related to profiling at all.

What we want to do is to limit the resource usage of processes.

Quoted 5 lines

That sounds great, and something I'd be interested in for vc4. However, as far as I saw explained here, this patch doesn't let you limit resource usage of a process and is only useful for "profiling/monitoring" so I'm wondering how it is useful for that purpose.

Ok, good to know. I haven't looked at this in deep, but if this is just for accounting that would certainly be missing the goal.

Account for the number of command submitted to amdgpu by type on a per cgroup basis, for the purpose of profiling/monitoring applications.

x prefix in the control file name x.cmd_submitted.amd.stat signify experimental.

Change-Id Ibc22e5bda600f54fe820fe0af5400ca348691550

Signed-off-by Kenny Ho <[email protected]>

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c      |  5 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.c | 54 +++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.h |  5 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c    | 15 ++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h    |  5 +-
  5 files changed, 83 insertions(+), 1 deletion(-)

drivers/gpu/drm/amd/amdgpu/amdgpu_cs.c +5 -0

index 663043c8f0f5..b448160aed89 100644
@@ -33,6 +33,7 @@
  #include "amdgpu_trace.h"
  #include "amdgpu_gmc.h"
  #include "amdgpu_gem.h"
+#include "amdgpu_drmcgrp.h"
 
  static int amdgpu_cs_user_fence_chunk(struct amdgpu_cs_parser *p,
  				      struct drm_amdgpu_cs_chunk_fence *data,

@@ -1275,6 +1276,7 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
  	union drm_amdgpu_cs *cs = data;
  	struct amdgpu_cs_parser parser = {};
  	bool reserved_buffers = false;
+	struct amdgpu_ring *ring;
  	int i, r;
 
  	if (!adev->accel_working)

@@ -1317,6 +1319,9 @@ int amdgpu_cs_ioctl(struct drm_device *dev, void *data, struct drm_file *filp)
  	if (r)
  		goto out;
 
+	ring = to_amdgpu_ring(parser.entity->rq->sched);
+	amdgpu_drmcgrp_count_cs(current, dev, ring->funcs->type);
+
  	r = amdgpu_cs_submit(&parser, cs);
 
  out:

drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.c +54 -0

index ed8aac17769c..853b77532428 100644
@@ -1,11 +1,65 @@
  // SPDX-License-Identifier: MIT
  // Copyright 2018 Advanced Micro Devices, Inc.
  #include <linux/slab.h>
+#include <linux/mutex.h>
  #include <linux/cgroup_drm.h>
  #include <drm/drm_device.h>
+#include "amdgpu_ring.h"
  #include "amdgpu_drmcgrp.h"
 
+void amdgpu_drmcgrp_count_cs(struct task_struct *task, struct drm_device *dev,
+		enum amdgpu_ring_type r_type)
+{
+	struct drmcgrp *drmcgrp = get_drmcgrp(task);
+	struct drmcgrp_device_resource *ddr;
+	struct drmcgrp *p;
+	struct amd_drmcgrp_dev_resource *a_ddr;
+
+	if (drmcgrp == NULL)
+		return;
+
+	ddr = drmcgrp->dev_resources[dev->primary->index];
+
+	mutex_lock(&ddr->ddev->mutex);
+	for (p = drmcgrp; p != NULL; p = parent_drmcgrp(drmcgrp)) {
+		a_ddr = ddr_amdddr(p->dev_resources[dev->primary->index]);
+
+		a_ddr->cs_count[r_type]++;
+	}
+	mutex_unlock(&ddr->ddev->mutex);
+}
+
+int amd_drmcgrp_cmd_submit_accounting_read(struct seq_file *sf, void *v)
+{
+	struct drmcgrp *drmcgrp = css_drmcgrp(seq_css(sf));
+	struct drmcgrp_device_resource *ddr = NULL;
+	struct amd_drmcgrp_dev_resource *a_ddr = NULL;
+	int i, j;
+
+	seq_puts(sf, "---\n");
+	for (i = 0; i < MAX_DRM_DEV; i++) {
+		ddr = drmcgrp->dev_resources[i];
+
+		if (ddr == NULL || ddr->ddev->vid != amd_drmcgrp_vendor_id)
+			continue;
+
+		a_ddr = ddr_amdddr(ddr);
+
+		seq_printf(sf, "card%d:\n", i);
+		for (j = 0; j < __MAX_AMDGPU_RING_TYPE; j++)
+			seq_printf(sf, "  %s: %llu\n", amdgpu_ring_names[j], a_ddr->cs_count[j]);
+	}
+
+	return 0;
+}
+
+
  struct cftype files[] = {
+	{
+		.name = "x.cmd_submitted.amd.stat",
+		.seq_show = amd_drmcgrp_cmd_submit_accounting_read,
+		.flags = CFTYPE_NOT_ON_ROOT,
+	},
  	{ } /* terminate */
  };
 

drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.h +5 -0

index e2934b7a49f5..f894a9a1059f 100644
@@ -5,12 +5,17 @@
  #define _AMDGPU_DRMCGRP_H
 
  #include <linux/cgroup_drm.h>
+#include "amdgpu_ring.h"
 
  /* for AMD specific DRM resources */
  struct amd_drmcgrp_dev_resource {
  	struct drmcgrp_device_resource ddr;
+	u64 cs_count[__MAX_AMDGPU_RING_TYPE];
  };
 
+void amdgpu_drmcgrp_count_cs(struct task_struct *task, struct drm_device *dev,
+		enum amdgpu_ring_type r_type);
+
  static inline struct amd_drmcgrp_dev_resource *ddr_amdddr(struct drmcgrp_device_resource *ddr)
  {
  	return ddr ? container_of(ddr, struct amd_drmcgrp_dev_resource, ddr) : NULL;

drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +5 -0

index b70e85ec147d..1606f84d2334 100644
@@ -34,6 +34,21 @@
  #include "amdgpu.h"
  #include "atom.h"
 
+char const *amdgpu_ring_names[] = {
+	[AMDGPU_RING_TYPE_GFX]		= "gfx",
+	[AMDGPU_RING_TYPE_COMPUTE]	= "compute",
+	[AMDGPU_RING_TYPE_SDMA]		= "sdma",
+	[AMDGPU_RING_TYPE_UVD]		= "uvd",

drivers/gpu/drm/amd/amdgpu/amdgpu_ring.c +0 -0

  /*
   * Rings
   * Most engines on the GPU are fed via ring buffers.  Ring

drivers/gpu/drm/amd/amdgpu/amdgpu_ring.h +0 -0

index 4caa301ce454..e292b7e89c6a 100644
@@ -56,9 +56,12 @@ enum amdgpu_ring_type {
  	AMDGPU_RING_TYPE_UVD_ENC,
  	AMDGPU_RING_TYPE_VCN_DEC,
  	AMDGPU_RING_TYPE_VCN_ENC,

Account for the total size of buffer object requested to amdgpu by buffer type on a per cgroup basis.

x prefix in the control file name x.bo_requested.amd.stat signify experimental.

Account for the total size of buffer object requested to amdgpu by buffer type on a per cgroup basis.

x prefix in the control file name x.bo_requested.amd.stat signify experimental.

Change-Id Ifb680c4bcf3652879a7a659510e25680c2465cf6

Signed-off-by Kenny Ho <[email protected]>

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.c | 56 +++++++++++++++++++++
  drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.h |  3 ++
  drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c     | 13 +++++
  include/uapi/drm/amdgpu_drm.h               | 24 ++++++---
  4 files changed, 90 insertions(+), 6 deletions(-)

drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.c +56 -0

index 853b77532428..e3d98ed01b79 100644
@@ -7,6 +7,57 @@
  #include "amdgpu_ring.h"
  #include "amdgpu_drmcgrp.h"
 
+void amdgpu_drmcgrp_count_bo_req(struct task_struct *task, struct drm_device *dev,
+		u32 domain, unsigned long size)
+{
+	struct drmcgrp *drmcgrp = get_drmcgrp(task);
+	struct drmcgrp_device_resource *ddr;
+	struct drmcgrp *p;
+	struct amd_drmcgrp_dev_resource *a_ddr;
+        int i;
+
+	if (drmcgrp == NULL)
+		return;
+
+	ddr = drmcgrp->dev_resources[dev->primary->index];
+
+	mutex_lock(&ddr->ddev->mutex);
+	for (p = drmcgrp; p != NULL; p = parent_drmcgrp(drmcgrp)) {
+		a_ddr = ddr_amdddr(p->dev_resources[dev->primary->index]);
+
+		for (i = 0; i < __MAX_AMDGPU_MEM_DOMAIN; i++)
+			if ( (1 << i) & domain)
+				a_ddr->bo_req_count[i] += size;
+	}
+	mutex_unlock(&ddr->ddev->mutex);
+}
+
+int amd_drmcgrp_bo_req_stat_read(struct seq_file *sf, void *v)
+{
+	struct drmcgrp *drmcgrp = css_drmcgrp(seq_css(sf));
+	struct drmcgrp_device_resource *ddr = NULL;
+	struct amd_drmcgrp_dev_resource *a_ddr = NULL;
+	int i, j;
+
+	seq_puts(sf, "---\n");
+	for (i = 0; i < MAX_DRM_DEV; i++) {
+		ddr = drmcgrp->dev_resources[i];
+
+		if (ddr == NULL || ddr->ddev->vid != amd_drmcgrp_vendor_id)
+			continue;
+
+		a_ddr = ddr_amdddr(ddr);
+
+		seq_printf(sf, "card%d:\n", i);
+		for (j = 0; j < __MAX_AMDGPU_MEM_DOMAIN; j++)
+			seq_printf(sf, "  %s: %llu\n", amdgpu_mem_domain_names[j], a_ddr->bo_req_count[j]);
+	}
+
+	return 0;
+}
+
+
+
  void amdgpu_drmcgrp_count_cs(struct task_struct *task, struct drm_device *dev,
  		enum amdgpu_ring_type r_type)
  {

@@ -55,6 +106,11 @@ int amd_drmcgrp_cmd_submit_accounting_read(struct seq_file *sf, void *v)
 
 
  struct cftype files[] = {
+	{
+		.name = "x.bo_requested.amd.stat",
+		.seq_show = amd_drmcgrp_bo_req_stat_read,
+		.flags = CFTYPE_NOT_ON_ROOT,
+	},
  	{
  		.name = "x.cmd_submitted.amd.stat",
  		.seq_show = amd_drmcgrp_cmd_submit_accounting_read,

drivers/gpu/drm/amd/amdgpu/amdgpu_drmcgrp.h +3 -0

index f894a9a1059f..8b9d61e47dde 100644
@@ -11,10 +11,13 @@
  struct amd_drmcgrp_dev_resource {
  	struct drmcgrp_device_resource ddr;
  	u64 cs_count[__MAX_AMDGPU_RING_TYPE];
+	u64 bo_req_count[__MAX_AMDGPU_MEM_DOMAIN];
  };
 
  void amdgpu_drmcgrp_count_cs(struct task_struct *task, struct drm_device *dev,
  		enum amdgpu_ring_type r_type);
+void amdgpu_drmcgrp_count_bo_req(struct task_struct *task, struct drm_device *dev,
+		u32 domain, unsigned long size);
 
  static inline struct amd_drmcgrp_dev_resource *ddr_amdddr(struct drmcgrp_device_resource *ddr)
  {

drivers/gpu/drm/amd/amdgpu/amdgpu_gem.c +13 -0

index 7b3d1ebda9df..339e1d3edad8 100644
@@ -31,6 +31,17 @@
  #include <drm/amdgpu_drm.h>
  #include "amdgpu.h"
  #include "amdgpu_display.h"
+#include "amdgpu_drmcgrp.h"
+
+char const *amdgpu_mem_domain_names[] = {
+	[AMDGPU_MEM_DOMAIN_CPU]		= "cpu",
+	[AMDGPU_MEM_DOMAIN_GTT]		= "gtt",
+	[AMDGPU_MEM_DOMAIN_VRAM]	= "vram",
+	[AMDGPU_MEM_DOMAIN_GDS]		= "gds",
+	[AMDGPU_MEM_DOMAIN_GWS]		= "gws",
+	[AMDGPU_MEM_DOMAIN_OA]		= "oa",
+	[__MAX_AMDGPU_MEM_DOMAIN]	= "_max"
+};
 
  void amdgpu_gem_object_free(struct drm_gem_object *gobj)
  {

@@ -52,6 +63,8 @@ int amdgpu_gem_object_create(struct amdgpu_device *adev, unsigned long size,
  	struct amdgpu_bo_param bp;
  	int r;
 
+	amdgpu_drmcgrp_count_bo_req(current, adev->ddev, initial_domain, size);
+
  	memset(&bp, 0, sizeof(bp));
  	*obj = NULL;
  	/* At least align on page size */

include/uapi/drm/amdgpu_drm.h +0 -0

index 370e9a5536ef..531726443104 100644
@@ -72,6 +72,18 @@ extern "C" {
  #define DRM_IOCTL_AMDGPU_FENCE_TO_HANDLE DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_FENCE_TO_HANDLE, union drm_amdgpu_fence_to_handle)
  #define DRM_IOCTL_AMDGPU_SCHED		DRM_IOW(DRM_COMMAND_BASE + DRM_AMDGPU_SCHED, union drm_amdgpu_sched)

include/uapi/drm/amdgpu_drm.h +3 -5

+
+extern char const *amdgpu_mem_domain_names[];
+
  /**
   * DOC: memory domains
   *

@@ -95,12 +107,12 @@ extern "C" {
   * %AMDGPU_GEM_DOMAIN_OA	Ordered append, used by 3D or Compute engines
   * for appending data.
   */
-#define AMDGPU_GEM_DOMAIN_CPU		0x1
-#define AMDGPU_GEM_DOMAIN_GTT		0x2
-#define AMDGPU_GEM_DOMAIN_VRAM		0x4
-#define AMDGPU_GEM_DOMAIN_GDS		0x8
-#define AMDGPU_GEM_DOMAIN_GWS		0x10

Quoted 8 lines

include/uapi/drm/amdgpu_drm.h +0 -0

index 370e9a5536ef..531726443104 100644
@@ -72,6 +72,18 @@ extern "C" {
  #define DRM_IOCTL_AMDGPU_FENCE_TO_HANDLE DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_FENCE_TO_HANDLE, union drm_amdgpu_fence_to_handle)
  #define DRM_IOCTL_AMDGPU_SCHED              DRM_IOW(DRM_COMMAND_BASE + DRM_AMDGPU_SCHED, union drm_amdgpu_sched)
 

include/uapi/drm/amdgpu_drm.h extern "C" { · line 83

Quoted 9 lines

include/uapi/drm/amdgpu_drm.h +9 -0

+enum amdgpu_mem_domain {
+     AMDGPU_MEM_DOMAIN_CPU,
+     AMDGPU_MEM_DOMAIN_GTT,
+     AMDGPU_MEM_DOMAIN_VRAM,
+     AMDGPU_MEM_DOMAIN_GDS,
+     AMDGPU_MEM_DOMAIN_GWS,
+     AMDGPU_MEM_DOMAIN_OA,
+     __MAX_AMDGPU_MEM_DOMAIN
+};

Well that is a clear NAK since it duplicates the TTM defines. Please use that one instead and don't make this UAPI.

Quoted 17 lines

include/uapi/drm/amdgpu_drm.h +4 -6

+
+extern char const *amdgpu_mem_domain_names[];
+
  /**
   * DOC: memory domains
   *

@@ -95,12 +107,12 @@ extern "C" {
   * %AMDGPU_GEM_DOMAIN_OA    Ordered append, used by 3D or Compute engines
   * for appending data.
   */
-#define AMDGPU_GEM_DOMAIN_CPU                0x1
-#define AMDGPU_GEM_DOMAIN_GTT                0x2
-#define AMDGPU_GEM_DOMAIN_VRAM               0x4
-#define AMDGPU_GEM_DOMAIN_GDS                0x8
-#define AMDGPU_GEM_DOMAIN_GWS                0x10
-#define AMDGPU_GEM_DOMAIN_OA         0x20
+#define AMDGPU_GEM_DOMAIN_CPU                (1 << AMDGPU_MEM_DOMAIN_CPU)

include/uapi/drm/amdgpu_drm.h extern "C" { · line 118

Quoted 8 lines

include/uapi/drm/amdgpu_drm.h +5 -0

+#define AMDGPU_GEM_DOMAIN_GTT                (1 << AMDGPU_MEM_DOMAIN_GTT)
+#define AMDGPU_GEM_DOMAIN_VRAM               (1 << AMDGPU_MEM_DOMAIN_VRAM)
+#define AMDGPU_GEM_DOMAIN_GDS                (1 << AMDGPU_MEM_DOMAIN_GDS)
+#define AMDGPU_GEM_DOMAIN_GWS                (1 << AMDGPU_MEM_DOMAIN_GWS)
+#define AMDGPU_GEM_DOMAIN_OA         (1 << AMDGPU_MEM_DOMAIN_OA)
  #define AMDGPU_GEM_DOMAIN_MASK              (AMDGPU_GEM_DOMAIN_CPU | \
                                       AMDGPU_GEM_DOMAIN_GTT | \
                                       AMDGPU_GEM_DOMAIN_VRAM | \

Hey Christian,

Sorry for the late reply, I missed this for some reason.

Quoted 8 lines · Christian König

Quoted 8 lines

include/uapi/drm/amdgpu_drm.h +0 -0

index 370e9a5536ef..531726443104 100644
@@ -72,6 +72,18 @@ extern "C" {
   #define DRM_IOCTL_AMDGPU_FENCE_TO_HANDLE DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_FENCE_TO_HANDLE, union drm_amdgpu_fence_to_handle)
   #define DRM_IOCTL_AMDGPU_SCHED              DRM_IOW(DRM_COMMAND_BASE + DRM_AMDGPU_SCHED, union drm_amdgpu_sched)
 

Quoted 11 lines

include/uapi/drm/amdgpu_drm.h extern "C" { · line 83

Quoted 9 lines

include/uapi/drm/amdgpu_drm.h +9 -0

+enum amdgpu_mem_domain {
+     AMDGPU_MEM_DOMAIN_CPU,
+     AMDGPU_MEM_DOMAIN_GTT,
+     AMDGPU_MEM_DOMAIN_VRAM,
+     AMDGPU_MEM_DOMAIN_GDS,
+     AMDGPU_MEM_DOMAIN_GWS,
+     AMDGPU_MEM_DOMAIN_OA,
+     __MAX_AMDGPU_MEM_DOMAIN
+};

Well that is a clear NAK since it duplicates the TTM defines. Please use that one instead and don't make this UAPI.

This is defined to help with the chunk of changes below. The AMDGPU_GEM_DOMAIN* already exists and this is similar to how TTM has TTM_PL_* to help with the creation of TTM_PL_FLAG_*:
https://elixir.bootlin.com/linux/v4.20-rc4/source/include/drm/ttm/ttm_placement.h#L36

I don't disagree that there is a duplication here but it's pre-existing so if you can help clarify my confusion that would be much appreciated.

Reards,
Kenny

Quoted 21 lines

Quoted 21 lines

include/uapi/drm/amdgpu_drm.h +8 -6

+
+extern char const *amdgpu_mem_domain_names[];
+
   /**
    * DOC: memory domains
    *

@@ -95,12 +107,12 @@ extern "C" {
    * %AMDGPU_GEM_DOMAIN_OA    Ordered append, used by 3D or Compute engines
    * for appending data.
    */
-#define AMDGPU_GEM_DOMAIN_CPU                0x1
-#define AMDGPU_GEM_DOMAIN_GTT                0x2
-#define AMDGPU_GEM_DOMAIN_VRAM               0x4
-#define AMDGPU_GEM_DOMAIN_GDS                0x8
-#define AMDGPU_GEM_DOMAIN_GWS                0x10
-#define AMDGPU_GEM_DOMAIN_OA         0x20
+#define AMDGPU_GEM_DOMAIN_CPU                (1 << AMDGPU_MEM_DOMAIN_CPU)
+#define AMDGPU_GEM_DOMAIN_GTT                (1 << AMDGPU_MEM_DOMAIN_GTT)
+#define AMDGPU_GEM_DOMAIN_VRAM               (1 << AMDGPU_MEM_DOMAIN_VRAM)
+#define AMDGPU_GEM_DOMAIN_GDS                (1 << AMDGPU_MEM_DOMAIN_GDS)
+#define AMDGPU_GEM_DOMAIN_GWS                (1 << AMDGPU_MEM_DOMAIN_GWS)

Quoted 4 lines

include/uapi/drm/amdgpu_drm.h extern "C" { · line 118

Quoted 4 lines

include/uapi/drm/amdgpu_drm.h +1 -0

+#define AMDGPU_GEM_DOMAIN_OA         (1 << AMDGPU_MEM_DOMAIN_OA)
   #define AMDGPU_GEM_DOMAIN_MASK              (AMDGPU_GEM_DOMAIN_CPU | \
                                        AMDGPU_GEM_DOMAIN_GTT | \
                                        AMDGPU_GEM_DOMAIN_VRAM | \

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

Am 27.11.18 um 19:15 schrieb Kenny Ho:

Quoted 25 lines

Hey Christian,

Sorry for the late reply, I missed this for some reason.

Quoted 8 lines · Christian König

Quoted 8 lines

include/uapi/drm/amdgpu_drm.h +0 -0

index 370e9a5536ef..531726443104 100644
@@ -72,6 +72,18 @@ extern "C" {
   #define DRM_IOCTL_AMDGPU_FENCE_TO_HANDLE DRM_IOWR(DRM_COMMAND_BASE + DRM_AMDGPU_FENCE_TO_HANDLE, union drm_amdgpu_fence_to_handle)
   #define DRM_IOCTL_AMDGPU_SCHED              DRM_IOW(DRM_COMMAND_BASE + DRM_AMDGPU_SCHED, union drm_amdgpu_sched)
 

Quoted 11 lines

include/uapi/drm/amdgpu_drm.h extern "C" { · line 83

Quoted 9 lines

include/uapi/drm/amdgpu_drm.h +9 -0

+enum amdgpu_mem_domain {
+     AMDGPU_MEM_DOMAIN_CPU,
+     AMDGPU_MEM_DOMAIN_GTT,
+     AMDGPU_MEM_DOMAIN_VRAM,
+     AMDGPU_MEM_DOMAIN_GDS,
+     AMDGPU_MEM_DOMAIN_GWS,
+     AMDGPU_MEM_DOMAIN_OA,
+     __MAX_AMDGPU_MEM_DOMAIN
+};

Well that is a clear NAK since it duplicates the TTM defines. Please use that one instead and don't make this UAPI.

Quoted 8 lines

This is defined to help with the chunk of changes below. The AMDGPU_GEM_DOMAIN* already exists and this is similar to how TTM has TTM_PL_* to help with the creation of TTM_PL_FLAG_*:
https://elixir.bootlin.com/linux/v4.20-rc4/source/include/drm/ttm/ttm_placement.h#L36

I don't disagree that there is a duplication here but it's pre-existing so if you can help clarify my confusion that would be much appreciated.

The AMDGPU_GEM_DOMAIN are masks which are used in the frontend IOCTL interface to create BOs.

TTM defines the backend pools where the memory is then allocated from to fill the BOs.

So you are mixing frontend and backend here.

In other words for the whole cgroup interface you should not make a single change to amdgpu_drm.h or otherwise you are doing something wrong.

Regards,
Christian.

Quoted 33 lines

Reards,
Kenny

Quoted 17 lines

Quoted 17 lines

include/uapi/drm/amdgpu_drm.h +4 -6

+
+extern char const *amdgpu_mem_domain_names[];
+
   /**
    * DOC: memory domains
    *

@@ -95,12 +107,12 @@ extern "C" {
    * %AMDGPU_GEM_DOMAIN_OA    Ordered append, used by 3D or Compute engines
    * for appending data.
    */
-#define AMDGPU_GEM_DOMAIN_CPU                0x1
-#define AMDGPU_GEM_DOMAIN_GTT                0x2
-#define AMDGPU_GEM_DOMAIN_VRAM               0x4
-#define AMDGPU_GEM_DOMAIN_GDS                0x8
-#define AMDGPU_GEM_DOMAIN_GWS                0x10
-#define AMDGPU_GEM_DOMAIN_OA         0x20
+#define AMDGPU_GEM_DOMAIN_CPU                (1 << AMDGPU_MEM_DOMAIN_CPU)

Quoted 8 lines

include/uapi/drm/amdgpu_drm.h extern "C" { · line 118

Quoted 8 lines

include/uapi/drm/amdgpu_drm.h +5 -0

+#define AMDGPU_GEM_DOMAIN_GTT                (1 << AMDGPU_MEM_DOMAIN_GTT)
+#define AMDGPU_GEM_DOMAIN_VRAM               (1 << AMDGPU_MEM_DOMAIN_VRAM)
+#define AMDGPU_GEM_DOMAIN_GDS                (1 << AMDGPU_MEM_DOMAIN_GDS)
+#define AMDGPU_GEM_DOMAIN_GWS                (1 << AMDGPU_MEM_DOMAIN_GWS)
+#define AMDGPU_GEM_DOMAIN_OA         (1 << AMDGPU_MEM_DOMAIN_OA)
   #define AMDGPU_GEM_DOMAIN_MASK              (AMDGPU_GEM_DOMAIN_CPU | \
                                        AMDGPU_GEM_DOMAIN_GTT | \
                                        AMDGPU_GEM_DOMAIN_VRAM | \

_______________________________________________
amd-gfx mailing list
[email protected]
https://lists.freedesktop.org/mailman/listinfo/amd-gfx

This new drmcgrp resource limits the largest GEM buffer that can be allocated in a cgroup.

Change-Id I0830d56775568e1cf215b56cc892d5e7945e9f25

Signed-off-by Kenny Ho <[email protected]>

---
  include/linux/cgroup_drm.h |  2 ++
  kernel/cgroup/drm.c        | 59 ++++++++++++++++++++++++++++++++++++++
  2 files changed, 61 insertions(+)

include/linux/cgroup_drm.h +0 -0

index fe14ba7bb1cf..57c07a148975 100644

include/linux/cgroup_drm.h +0 -0

  };
 
  struct drmcgrp {

kernel/cgroup/drm.c +56 -0

index bc3abff09113..5c7e1b8059ce 100644
@@ -17,6 +17,7 @@ struct drmcgrp_device {
  	struct mutex		mutex;
 
  	s64			bo_limits_total_allocated_default;
+	size_t			bo_limits_peak_allocated_default;
  };
 
  #define DRMCG_CTF_PRIV_SIZE 3

@@ -24,6 +25,7 @@ struct drmcgrp_device {
 
  enum drmcgrp_res_type {
  	DRMCGRP_TYPE_BO_TOTAL,
+	DRMCGRP_TYPE_BO_PEAK,
  };
 
  enum drmcgrp_file_type {

@@ -72,6 +74,9 @@ static inline int init_drmcgrp_single(struct drmcgrp *drmcgrp, int i)
  	if (known_drmcgrp_devs[i] != NULL) {
  		ddr->bo_limits_total_allocated =
  		  known_drmcgrp_devs[i]->bo_limits_total_allocated_default;
+
+		ddr->bo_limits_peak_allocated =
+		  known_drmcgrp_devs[i]->bo_limits_peak_allocated_default;
  	}
 
  	return 0;

@@ -131,6 +136,9 @@ static inline void drmcgrp_print_stats(struct drmcgrp_device_resource *ddr,
  	case DRMCGRP_TYPE_BO_TOTAL:
  		seq_printf(sf, "%lld\n", ddr->bo_stats_total_allocated);
  		break;
+	case DRMCGRP_TYPE_BO_PEAK:
+		seq_printf(sf, "%zu\n", ddr->bo_stats_peak_allocated);
+		break;
  	default:
  		seq_puts(sf, "\n");
  		break;

@@ -149,6 +157,9 @@ static inline void drmcgrp_print_limits(struct drmcgrp_device_resource *ddr,
  	case DRMCGRP_TYPE_BO_TOTAL:
  		seq_printf(sf, "%lld\n", ddr->bo_limits_total_allocated);
  		break;
+	case DRMCGRP_TYPE_BO_PEAK:
+		seq_printf(sf, "%zu\n", ddr->bo_limits_peak_allocated);
+		break;
  	default:
  		seq_puts(sf, "\n");
  		break;

@@ -167,6 +178,9 @@ static inline void drmcgrp_print_default(struct drmcgrp_device *ddev,
  	case DRMCGRP_TYPE_BO_TOTAL:
  		seq_printf(sf, "%lld\n", ddev->bo_limits_total_allocated_default);
  		break;
+	case DRMCGRP_TYPE_BO_PEAK:
+		seq_printf(sf, "%zu\n", ddev->bo_limits_peak_allocated_default);
+		break;
  	default:
  		seq_puts(sf, "\n");
  		break;

@@ -182,6 +196,11 @@ static inline void drmcgrp_print_help(int cardNum, struct seq_file *sf,
  		"Total amount of buffer allocation in bytes for card%d\n",
  		cardNum);
  		break;
+	case DRMCGRP_TYPE_BO_PEAK:
+		seq_printf(sf,
+		"Largest buffer allocation in bytes for card%d\n",
+		cardNum);
+		break;
  	default:
  		seq_puts(sf, "\n");
  		break;

@@ -254,6 +273,10 @@ ssize_t drmcgrp_bo_limit_write(struct kernfs_open_file *of, char *buf,
                                  if (val < 0) continue;
  				ddr->bo_limits_total_allocated = val;
  				break;
+			case DRMCGRP_TYPE_BO_PEAK:
+                                if (val < 0) continue;
+				ddr->bo_limits_peak_allocated = val;
+				break;
  			default:
  				break;
  			}

@@ -300,6 +323,33 @@ struct cftype files[] = {
  		.private = (DRMCGRP_TYPE_BO_TOTAL << DRMCG_CTF_PRIV_SIZE) |
  			DRMCGRP_FTYPE_MAX,
  	},
+	{
+		.name = "buffer.peak.stats",
+		.seq_show = drmcgrp_bo_show,
+		.private = (DRMCGRP_TYPE_BO_PEAK << DRMCG_CTF_PRIV_SIZE) |
+			DRMCGRP_FTYPE_STATS,
+	},
+	{
+		.name = "buffer.peak.default",
+		.seq_show = drmcgrp_bo_show,
+		.flags = CFTYPE_ONLY_ON_ROOT,
+		.private = (DRMCGRP_TYPE_BO_PEAK << DRMCG_CTF_PRIV_SIZE) |
+			DRMCGRP_FTYPE_DEFAULT,
+	},
+	{
+		.name = "buffer.peak.help",
+		.seq_show = drmcgrp_bo_show,
+		.flags = CFTYPE_ONLY_ON_ROOT,
+		.private = (DRMCGRP_TYPE_BO_PEAK << DRMCG_CTF_PRIV_SIZE) |
+			DRMCGRP_FTYPE_HELP,
+	},
+	{
+		.name = "buffer.peak.max",
+		.write = drmcgrp_bo_limit_write,
+		.seq_show = drmcgrp_bo_show,
+		.private = (DRMCGRP_TYPE_BO_PEAK << DRMCG_CTF_PRIV_SIZE) |
+			DRMCGRP_FTYPE_MAX,
+	},
  	{ }	/* terminate */
  };
 

@@ -323,6 +373,7 @@ int drmcgrp_register_device(struct drm_device *dev)
 
  	ddev->dev = dev;
  	ddev->bo_limits_total_allocated_default = S64_MAX;
+	ddev->bo_limits_peak_allocated_default = SIZE_MAX;
 
  	mutex_init(&ddev->mutex);
 

@@ -393,6 +444,11 @@ bool drmcgrp_bo_can_allocate(struct task_struct *task, struct drm_device *dev,
  			result = false;
  			break;
  		}
+
+		if (d->bo_limits_peak_allocated < size) {
+			result = false;
+			break;
+		}
  	}
  	mutex_unlock(&known_drmcgrp_devs[devIdx]->mutex);
 

This is a follow up to the RFC I made last november to introduce a cgroup controller for the GPU/DRM subsystem [a]. The goal is to be able to provide resource management to GPU resources using things like container. The cover letter from v1 is copied below for reference.

Usage examples:
// set limit for card1 to 1GB
sed -i '2s/.*/1073741824/' /sys/fs/cgroup/<cgroup>/drm.buffer.total.max

// set limit for card0 to 512MB
sed -i '1s/.*/536870912/' /sys/fs/cgroup/<cgroup>/drm.buffer.total.max

v2:
* Removed the vendoring concepts
* Add limit to total buffer allocation
* Add limit to the maximum size of a buffer allocation

TODO: process migration
TODO: documentations

[a]: https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html

v1: cover letter

The purpose of this patch series is to start a discussion for a generic cgroup controller for the drm subsystem. The design proposed here is a very early one. We are hoping to engage the community as we develop the idea.

Backgrounds
==========
Control Groups/cgroup provide a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour, such as accounting/limiting the resources which processes in a cgroup can access[1]. Weights, limits, protections, allocations are the main resource distribution models. Existing cgroup controllers includes cpu, memory, io, rdma, and more. cgroup is one of the foundational technologies that enables the popular container application deployment and management method.

Direct Rendering Manager/drm contains code intended to support the needs of complex graphics devices. Graphics drivers in the kernel may make use of DRM functions to make tasks like memory management, interrupt handling and DMA easier, and provide a uniform interface to applications. The DRM has also developed beyond traditional graphics applications to support compute/GPGPU applications.

Motivations
=========
As GPU grow beyond the realm of desktop/workstation graphics into areas like data center clusters and IoT, there are increasing needs to monitor and regulate GPU as a resource like cpu, memory and io.

Matt Roper from Intel began working on similar idea in early 2018 [2] for the purpose of managing GPU priority using the cgroup hierarchy. While that particular use case may not warrant a standalone drm cgroup controller, there are other use cases where having one can be useful [3]. Monitoring GPU resources such as VRAM and buffers, CU (compute unit [AMD's nomenclature])/EU (execution unit [Intel's nomenclature]), GPU job scheduling [4] can help sysadmins get a better understanding of the applications usage profile. Further usage regulations of the aforementioned resources can also help sysadmins optimize workload deployment on limited GPU resources.

With the increased importance of machine learning, data science and other cloud-based applications, GPUs are already in production use in data centers today [5,6,7]. Existing GPU resource management is very course grain, however, as sysadmins are only able to distribute workload on a per-GPU basis [8]. An alternative is to use GPU virtualization (with or without SRIOV) but it generally acts on the entire GPU instead of the specific resources in a GPU. With a drm cgroup controller, we can enable alternate, fine-grain, sub-GPU resource management (in addition to what may be available via GPU virtualization.)

In addition to production use, the DRM cgroup can also help with testing graphics application robustness by providing a mean to artificially limit DRM resources availble to the applications.

Challenges
========
While there are common infrastructure in DRM that is shared across many vendors (the scheduler [4] for example), there are also aspects of DRM that are vendor specific. To accommodate this, we borrowed the mechanism used by the cgroup to handle different kinds of cgroup controller.

Resources for DRM are also often device (GPU) specific instead of system specific and a system may contain more than one GPU. For this, we borrowed some of the ideas from RDMA cgroup controller.

Approach
=======
To experiment with the idea of a DRM cgroup, we would like to start with basic accounting and statistics, then continue to iterate and add regulating mechanisms into the driver.

[1] https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
[2] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
[3] https://www.spinics.net/lists/cgroups/msg20720.html
[4] https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/scheduler
[5] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
[6] https://blog.openshift.com/gpu-accelerated-sql-queries-with-postgresql-pg-strom-in-openshift-3-10/
[7] https://github.com/RadeonOpenCompute/k8s-device-plugin
[8] https://github.com/kubernetes/kubernetes/issues/52757

Kenny Ho (5):
cgroup: Introduce cgroup for drm subsystem
cgroup: Add mechanism to register DRM devices drm/amdgpu: Register AMD devices for DRM cgroup drm, cgroup: Add total GEM buffer allocation limit drm, cgroup: Add peak GEM buffer allocation limit

  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c    |   4 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |   4 +
  drivers/gpu/drm/drm_gem.c                  |   7 +
  drivers/gpu/drm/drm_prime.c                |   9 +
  include/drm/drm_cgroup.h                   |  54 +++
  include/drm/drm_gem.h                      |  11 +
  include/linux/cgroup_drm.h                 |  47 ++
  include/linux/cgroup_subsys.h              |   4 +
  init/Kconfig                               |   5 +
  kernel/cgroup/Makefile                     |   1 +
  kernel/cgroup/drm.c                        | 497 +++++++++++++++++++++
  11 files changed, 643 insertions(+)

create mode 100644 include/drm/drm_cgroup.h
create mode 100644 include/linux/cgroup_drm.h
create mode 100644 kernel/cgroup/drm.c

I think it is a good approach to try to add a global limit first and when that's working go ahead with limiting device specific resources.

The only major issue I can see is on patch #4, see there for further details.

Christian.

Am 09.05.19 um 23:04 schrieb Kenny Ho:

Quoted 124 lines

This is a follow up to the RFC I made last november to introduce a cgroup controller for the GPU/DRM subsystem [a]. The goal is to be able to provide resource management to GPU resources using things like container. The cover letter from v1 is copied below for reference.

Usage examples:
// set limit for card1 to 1GB
sed -i '2s/.*/1073741824/' /sys/fs/cgroup/<cgroup>/drm.buffer.total.max

// set limit for card0 to 512MB
sed -i '1s/.*/536870912/' /sys/fs/cgroup/<cgroup>/drm.buffer.total.max

v2:
* Removed the vendoring concepts
* Add limit to total buffer allocation
* Add limit to the maximum size of a buffer allocation

TODO: process migration
TODO: documentations

[a]: https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html

v1: cover letter

The purpose of this patch series is to start a discussion for a generic cgroup controller for the drm subsystem. The design proposed here is a very early one. We are hoping to engage the community as we develop the idea.

Backgrounds
==========
Control Groups/cgroup provide a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour, such as accounting/limiting the resources which processes in a cgroup can access[1]. Weights, limits, protections, allocations are the main resource distribution models. Existing cgroup controllers includes cpu, memory, io, rdma, and more. cgroup is one of the foundational technologies that enables the popular container application deployment and management method.

Direct Rendering Manager/drm contains code intended to support the needs of complex graphics devices. Graphics drivers in the kernel may make use of DRM functions to make tasks like memory management, interrupt handling and DMA easier, and provide a uniform interface to applications. The DRM has also developed beyond traditional graphics applications to support compute/GPGPU applications.

Motivations
=========
As GPU grow beyond the realm of desktop/workstation graphics into areas like data center clusters and IoT, there are increasing needs to monitor and regulate GPU as a resource like cpu, memory and io.

Matt Roper from Intel began working on similar idea in early 2018 [2] for the purpose of managing GPU priority using the cgroup hierarchy. While that particular use case may not warrant a standalone drm cgroup controller, there are other use cases where having one can be useful [3]. Monitoring GPU resources such as VRAM and buffers, CU (compute unit [AMD's nomenclature])/EU (execution unit [Intel's nomenclature]), GPU job scheduling [4] can help sysadmins get a better understanding of the applications usage profile. Further usage regulations of the aforementioned resources can also help sysadmins optimize workload deployment on limited GPU resources.

With the increased importance of machine learning, data science and other cloud-based applications, GPUs are already in production use in data centers today [5,6,7]. Existing GPU resource management is very course grain, however, as sysadmins are only able to distribute workload on a per-GPU basis [8]. An alternative is to use GPU virtualization (with or without SRIOV) but it generally acts on the entire GPU instead of the specific resources in a GPU. With a drm cgroup controller, we can enable alternate, fine-grain, sub-GPU resource management (in addition to what may be available via GPU virtualization.)

In addition to production use, the DRM cgroup can also help with testing graphics application robustness by providing a mean to artificially limit DRM resources availble to the applications.

Challenges
========
While there are common infrastructure in DRM that is shared across many vendors (the scheduler [4] for example), there are also aspects of DRM that are vendor specific. To accommodate this, we borrowed the mechanism used by the cgroup to handle different kinds of cgroup controller.

Resources for DRM are also often device (GPU) specific instead of system specific and a system may contain more than one GPU. For this, we borrowed some of the ideas from RDMA cgroup controller.

Approach
=======
To experiment with the idea of a DRM cgroup, we would like to start with basic accounting and statistics, then continue to iterate and add regulating mechanisms into the driver.

[1] https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
[2] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
[3] https://www.spinics.net/lists/cgroups/msg20720.html
[4] https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/scheduler
[5] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
[6] https://blog.openshift.com/gpu-accelerated-sql-queries-with-postgresql-pg-strom-in-openshift-3-10/
[7] https://github.com/RadeonOpenCompute/k8s-device-plugin
[8] https://github.com/kubernetes/kubernetes/issues/52757

Kenny Ho (5):
cgroup: Introduce cgroup for drm subsystem
cgroup: Add mechanism to register DRM devices drm/amdgpu: Register AMD devices for DRM cgroup drm, cgroup: Add total GEM buffer allocation limit drm, cgroup: Add peak GEM buffer allocation limit

  drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c    |   4 +
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |   4 +
  drivers/gpu/drm/drm_gem.c                  |   7 +
  drivers/gpu/drm/drm_prime.c                |   9 +
  include/drm/drm_cgroup.h                   |  54 +++
  include/drm/drm_gem.h                      |  11 +
  include/linux/cgroup_drm.h                 |  47 ++
  include/linux/cgroup_subsys.h              |   4 +
  init/Kconfig                               |   5 +
  kernel/cgroup/Makefile                     |   1 +
  kernel/cgroup/drm.c                        | 497 +++++++++++++++++++++
  11 files changed, 643 insertions(+)

create mode 100644 include/drm/drm_cgroup.h
create mode 100644 include/linux/cgroup_drm.h
create mode 100644 kernel/cgroup/drm.c

[CAUTION: External Email]

Quoted 2 lines · Christian König

I think it is a good approach to try to add a global limit first and when that's working go ahead with limiting device specific resources.

What are some of the global drm resource limit/allocation that would be useful to implement? I would be happy to dig into those.

Regards,
Kenny

Quoted 130 lines

The only major issue I can see is on patch #4, see there for further details.

Christian.

Am 09.05.19 um 23:04 schrieb Kenny Ho:

Quoted 124 lines

This is a follow up to the RFC I made last november to introduce a cgroup controller for the GPU/DRM subsystem [a]. The goal is to be able to provide resource management to GPU resources using things like container. The cover letter from v1 is copied below for reference.

Usage examples:
// set limit for card1 to 1GB
sed -i '2s/.*/1073741824/' /sys/fs/cgroup/<cgroup>/drm.buffer.total.max

// set limit for card0 to 512MB
sed -i '1s/.*/536870912/' /sys/fs/cgroup/<cgroup>/drm.buffer.total.max

v2:
* Removed the vendoring concepts
* Add limit to total buffer allocation
* Add limit to the maximum size of a buffer allocation

TODO: process migration
TODO: documentations

[a]: https://lists.freedesktop.org/archives/dri-devel/2018-November/197106.html

v1: cover letter

The purpose of this patch series is to start a discussion for a generic cgroup controller for the drm subsystem. The design proposed here is a very early one. We are hoping to engage the community as we develop the idea.

Backgrounds
==========
Control Groups/cgroup provide a mechanism for aggregating/partitioning sets of tasks, and all their future children, into hierarchical groups with specialized behaviour, such as accounting/limiting the resources which processes in a cgroup can access[1]. Weights, limits, protections, allocations are the main resource distribution models. Existing cgroup controllers includes cpu, memory, io, rdma, and more. cgroup is one of the foundational technologies that enables the popular container application deployment and management method.

Direct Rendering Manager/drm contains code intended to support the needs of complex graphics devices. Graphics drivers in the kernel may make use of DRM functions to make tasks like memory management, interrupt handling and DMA easier, and provide a uniform interface to applications. The DRM has also developed beyond traditional graphics applications to support compute/GPGPU applications.

Motivations
=========
As GPU grow beyond the realm of desktop/workstation graphics into areas like data center clusters and IoT, there are increasing needs to monitor and regulate GPU as a resource like cpu, memory and io.

Matt Roper from Intel began working on similar idea in early 2018 [2] for the purpose of managing GPU priority using the cgroup hierarchy. While that particular use case may not warrant a standalone drm cgroup controller, there are other use cases where having one can be useful [3]. Monitoring GPU resources such as VRAM and buffers, CU (compute unit [AMD's nomenclature])/EU (execution unit [Intel's nomenclature]), GPU job scheduling [4] can help sysadmins get a better understanding of the applications usage profile. Further usage regulations of the aforementioned resources can also help sysadmins optimize workload deployment on limited GPU resources.

With the increased importance of machine learning, data science and other cloud-based applications, GPUs are already in production use in data centers today [5,6,7]. Existing GPU resource management is very course grain, however, as sysadmins are only able to distribute workload on a per-GPU basis [8]. An alternative is to use GPU virtualization (with or without SRIOV) but it generally acts on the entire GPU instead of the specific resources in a GPU. With a drm cgroup controller, we can enable alternate, fine-grain, sub-GPU resource management (in addition to what may be available via GPU virtualization.)

In addition to production use, the DRM cgroup can also help with testing graphics application robustness by providing a mean to artificially limit DRM resources availble to the applications.

Challenges
========
While there are common infrastructure in DRM that is shared across many vendors (the scheduler [4] for example), there are also aspects of DRM that are vendor specific. To accommodate this, we borrowed the mechanism used by the cgroup to handle different kinds of cgroup controller.

Resources for DRM are also often device (GPU) specific instead of system specific and a system may contain more than one GPU. For this, we borrowed some of the ideas from RDMA cgroup controller.

Approach
=======
To experiment with the idea of a DRM cgroup, we would like to start with basic accounting and statistics, then continue to iterate and add regulating mechanisms into the driver.

[1] https://www.kernel.org/doc/Documentation/cgroup-v1/cgroups.txt
[2] https://lists.freedesktop.org/archives/intel-gfx/2018-January/153156.html
[3] https://www.spinics.net/lists/cgroups/msg20720.html
[4] https://elixir.bootlin.com/linux/latest/source/drivers/gpu/drm/scheduler
[5] https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
[6] https://blog.openshift.com/gpu-accelerated-sql-queries-with-postgresql-pg-strom-in-openshift-3-10/
[7] https://github.com/RadeonOpenCompute/k8s-device-plugin
[8] https://github.com/kubernetes/kubernetes/issues/52757

Kenny Ho (5):
cgroup: Introduce cgroup for drm subsystem
cgroup: Add mechanism to register DRM devices drm/amdgpu: Register AMD devices for DRM cgroup drm, cgroup: Add total GEM buffer allocation limit drm, cgroup: Add peak GEM buffer allocation limit

   drivers/gpu/drm/amd/amdgpu/amdgpu_kms.c    |   4 +
   drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |   4 +
   drivers/gpu/drm/drm_gem.c                  |   7 +
   drivers/gpu/drm/drm_prime.c                |   9 +
   include/drm/drm_cgroup.h                   |  54 +++
   include/drm/drm_gem.h                      |  11 +
   include/linux/cgroup_drm.h                 |  47 ++
   include/linux/cgroup_subsys.h              |   4 +
   init/Kconfig                               |   5 +
   kernel/cgroup/Makefile                     |   1 +
   kernel/cgroup/drm.c                        | 497 +++++++++++++++++++++
   11 files changed, 643 insertions(+)

create mode 100644 include/drm/drm_cgroup.h
create mode 100644 include/linux/cgroup_drm.h
create mode 100644 kernel/cgroup/drm.c

The drm resource being measured and limited here is the GEM buffer objects. User applications allocate and free these buffers. In addition, a process can allocate a buffer and share it with another process. The consumer of a shared buffer can also outlive the allocator of the buffer.

For the purpose of cgroup accounting and limiting, ownership of the buffer is deemed to be the cgroup for which the allocating process belongs to. There is one limit per drm device.

In order to prevent the buffer outliving the cgroup that owns it, a process is prevented from importing buffers that are not own by the process' cgroup or the ancestors of the process' cgroup.

For this resource, the control files are prefixed with drm.buffer.total.

There are four control file types,
stats (ro) - display current measured values for a resource max (rw) - limits for a resource
default (ro, root cgroup only) - default values for a resource help (ro, root cgroup only) - help string for a resource

Each file is multi-lined with one entry/line per drm device.

Usage examples:
// set limit for card1 to 1GB
sed -i '2s/.*/1073741824/' /sys/fs/cgroup/<cgroup>/drm.buffer.total.max

// set limit for card0 to 512MB
sed -i '1s/.*/536870912/' /sys/fs/cgroup/<cgroup>/drm.buffer.total.max

Change-Id I4c249d06d45ec709d6481d4cbe87c5168545c5d0

Signed-off-by Kenny Ho <[email protected]>

---
  drivers/gpu/drm/amd/amdgpu/amdgpu_object.c |   4 +
  drivers/gpu/drm/drm_gem.c                  |   7 +
  drivers/gpu/drm/drm_prime.c                |   9 +
  include/drm/drm_cgroup.h                   |  34 ++-
  include/drm/drm_gem.h                      |  11 +
  include/linux/cgroup_drm.h                 |   3 +
  kernel/cgroup/drm.c                        | 280 +++++++++++++++++++++
  7 files changed, 346 insertions(+), 2 deletions(-)

drivers/gpu/drm/amd/amdgpu/amdgpu_object.c +4 -0

index 93b2c5a48a71..b4c078b7ad63 100644
@@ -34,6 +34,7 @@
  #include <drm/drmP.h>
  #include <drm/amdgpu_drm.h>
  #include <drm/drm_cache.h>
+#include <drm/drm_cgroup.h>
  #include "amdgpu.h"
  #include "amdgpu_trace.h"
  #include "amdgpu_amdkfd.h"

@@ -446,6 +447,9 @@ static int amdgpu_bo_do_create(struct amdgpu_device *adev,
  	if (!amdgpu_bo_validate_size(adev, size, bp->domain))
  		return -ENOMEM;
 
+	if (!drmcgrp_bo_can_allocate(current, adev->ddev, size))
+		return -ENOMEM;
+
  	*bo_ptr = NULL;
 
  	acc_size = ttm_bo_dma_acc_size(&adev->mman.bdev, size,

drivers/gpu/drm/drm_gem.c +7 -0

index 6a80db077dc6..cbd49bf34dcf 100644
@@ -37,10 +37,12 @@
  #include <linux/shmem_fs.h>
  #include <linux/dma-buf.h>
  #include <linux/mem_encrypt.h>
+#include <linux/cgroup_drm.h>
  #include <drm/drmP.h>
  #include <drm/drm_vma_manager.h>
  #include <drm/drm_gem.h>
  #include <drm/drm_print.h>
+#include <drm/drm_cgroup.h>
  #include "drm_internal.h"
 
  /** @file drm_gem.c

@@ -154,6 +156,9 @@ void drm_gem_private_object_init(struct drm_device *dev,
  	obj->handle_count = 0;
  	obj->size = size;
  	drm_vma_node_reset(&obj->vma_node);
+
+	obj->drmcgrp = get_drmcgrp(current);
+	drmcgrp_chg_bo_alloc(obj->drmcgrp, dev, size);
  }
  EXPORT_SYMBOL(drm_gem_private_object_init);
 

@@ -804,6 +809,8 @@ drm_gem_object_release(struct drm_gem_object *obj)
  	if (obj->filp)
  		fput(obj->filp);
 
+	drmcgrp_unchg_bo_alloc(obj->drmcgrp, obj->dev, obj->size);
+
  	drm_gem_free_mmap_offset(obj);
  }
  EXPORT_SYMBOL(drm_gem_object_release);

drivers/gpu/drm/drm_prime.c +2 -0

index 231e3f6d5f41..faed5611a1c6 100644
@@ -32,6 +32,7 @@
  #include <drm/drm_prime.h>
  #include <drm/drm_gem.h>
  #include <drm/drmP.h>
+#include <drm/drm_cgroup.h>
 
  #include "drm_internal.h"
 

@@ -794,6 +795,7 @@ int drm_gem_prime_fd_to_handle(struct drm_device *dev,
  {
  	struct dma_buf *dma_buf;
  	struct drm_gem_object *obj;
+	struct drmcgrp *drmcgrp = get_drmcgrp(current);
  	int ret;
 
  	dma_buf = dma_buf_get(prime_fd);

drivers/gpu/drm/drm_prime.c +0 -0

  	if (obj->dma_buf) {
  		WARN_ON(obj->dma_buf != dma_buf);
  	} else {

include/drm/drm_cgroup.h +32 -2

index ddb9eab64360..8711b7c5f7bf 100644
@@ -4,12 +4,20 @@
  #ifndef __DRM_CGROUP_H__
  #define __DRM_CGROUP_H__
 
+#include <linux/cgroup_drm.h>
+
  #ifdef CONFIG_CGROUP_DRM
 
  int drmcgrp_register_device(struct drm_device *device);
-
  int drmcgrp_unregister_device(struct drm_device *device);
-
+bool drmcgrp_is_self_or_ancestor(struct drmcgrp *self,
+		struct drmcgrp *relative);
+void drmcgrp_chg_bo_alloc(struct drmcgrp *drmcgrp, struct drm_device *dev,
+		size_t size);
+void drmcgrp_unchg_bo_alloc(struct drmcgrp *drmcgrp, struct drm_device *dev,
+		size_t size);
+bool drmcgrp_bo_can_allocate(struct task_struct *task, struct drm_device *dev,
+		size_t size);
  #else
  static inline int drmcgrp_register_device(struct drm_device *device)
  {

@@ -20,5 +28,27 @@ static inline int drmcgrp_unregister_device(struct drm_device *device)
  {
  	return 0;
  }
+
+static inline bool drmcgrp_is_self_or_ancestor(struct drmcgrp *self,
+		struct drmcgrp *relative)
+{
+	return false;
+}
+
+static inline void drmcgrp_chg_bo_alloc(struct drmcgrp *drmcgrp,
+		struct drm_device *dev,	size_t size)
+{
+}
+
+static inline void drmcgrp_unchg_bo_alloc(struct drmcgrp *drmcgrp,
+		struct drm_device *dev,	size_t size)
+{
+}
+
+static inline bool drmcgrp_bo_can_allocate(struct task_struct *task,
+		struct drm_device *dev,	size_t size)
+{
+	return true;
+}
  #endif /* CONFIG_CGROUP_DRM */
  #endif /* __DRM_CGROUP_H__ */

include/drm/drm_gem.h +11 -0

index c95727425284..02854c674b5c 100644
@@ -272,6 +272,17 @@ struct drm_gem_object {
  	 *
  	 */
  	const struct drm_gem_object_funcs *funcs;
+
+	/**
+	 * @drmcgrp:
+	 *
+	 * DRM cgroup this GEM object belongs to.
+         *
+         * This is used to track and limit the amount of GEM objects a user
+         * can allocate.  Since GEM objects can be shared, this is also used
+         * to ensure GEM objects are only shared within the same cgroup.
+	 */
+	struct drmcgrp *drmcgrp;
  };
 
  /**

include/linux/cgroup_drm.h +3 -0

index d7ccf434ca6b..fe14ba7bb1cf 100644
@@ -15,6 +15,9 @@
 
  struct drmcgrp_device_resource {
  	/* for per device stats */
+	s64			bo_stats_total_allocated;
+
+	s64			bo_limits_total_allocated;
  };
 
  struct drmcgrp {

kernel/cgroup/drm.c +270 -0

index f9ef4bf042d8..bc3abff09113 100644
@@ -15,6 +15,22 @@ static DEFINE_MUTEX(drmcgrp_mutex);
  struct drmcgrp_device {
  	struct drm_device	*dev;
  	struct mutex		mutex;
+
+	s64			bo_limits_total_allocated_default;
+};
+
+#define DRMCG_CTF_PRIV_SIZE 3
+#define DRMCG_CTF_PRIV_MASK GENMASK((DRMCG_CTF_PRIV_SIZE - 1), 0)
+
+enum drmcgrp_res_type {
+	DRMCGRP_TYPE_BO_TOTAL,
+};
+
+enum drmcgrp_file_type {
+	DRMCGRP_FTYPE_STATS,
+	DRMCGRP_FTYPE_MAX,
+	DRMCGRP_FTYPE_DEFAULT,
+	DRMCGRP_FTYPE_HELP,
  };
 
  /* indexed by drm_minor for access speed */

@@ -53,6 +69,10 @@ static inline int init_drmcgrp_single(struct drmcgrp *drmcgrp, int i)
  	}
 
  	/* set defaults here */
+	if (known_drmcgrp_devs[i] != NULL) {
+		ddr->bo_limits_total_allocated =
+		  known_drmcgrp_devs[i]->bo_limits_total_allocated_default;
+	}
 
  	return 0;
  }

@@ -99,7 +119,187 @@ drmcgrp_css_alloc(struct cgroup_subsys_state *parent_css)
  	return &drmcgrp->css;
  }
 
+static inline void drmcgrp_print_stats(struct drmcgrp_device_resource *ddr,
+		struct seq_file *sf, enum drmcgrp_res_type type)
+{
+	if (ddr == NULL) {
+		seq_puts(sf, "\n");
+		return;
+	}
+
+	switch (type) {
+	case DRMCGRP_TYPE_BO_TOTAL:
+		seq_printf(sf, "%lld\n", ddr->bo_stats_total_allocated);
+		break;
+	default:
+		seq_puts(sf, "\n");
+		break;
+	}
+}
+
+static inline void drmcgrp_print_limits(struct drmcgrp_device_resource *ddr,
+		struct seq_file *sf, enum drmcgrp_res_type type)
+{
+	if (ddr == NULL) {
+		seq_puts(sf, "\n");
+		return;
+	}
+
+	switch (type) {
+	case DRMCGRP_TYPE_BO_TOTAL:
+		seq_printf(sf, "%lld\n", ddr->bo_limits_total_allocated);
+		break;
+	default:
+		seq_puts(sf, "\n");
+		break;
+	}
+}
+
+static inline void drmcgrp_print_default(struct drmcgrp_device *ddev,
+		struct seq_file *sf, enum drmcgrp_res_type type)
+{
+	if (ddev == NULL) {
+		seq_puts(sf, "\n");
+		return;
+	}
+
+	switch (type) {
+	case DRMCGRP_TYPE_BO_TOTAL:
+		seq_printf(sf, "%lld\n", ddev->bo_limits_total_allocated_default);
+		break;
+	default:
+		seq_puts(sf, "\n");
+		break;
+	}
+}
+
+static inline void drmcgrp_print_help(int cardNum, struct seq_file *sf,
+		enum drmcgrp_res_type type)
+{
+	switch (type) {
+	case DRMCGRP_TYPE_BO_TOTAL:
+		seq_printf(sf,
+		"Total amount of buffer allocation in bytes for card%d\n",
+		cardNum);
+		break;
+	default:
+		seq_puts(sf, "\n");
+		break;
+	}
+}
+
+int drmcgrp_bo_show(struct seq_file *sf, void *v)
+{
+	struct drmcgrp *drmcgrp = css_drmcgrp(seq_css(sf));
+	struct drmcgrp_device_resource *ddr = NULL;
+	enum drmcgrp_file_type f_type = seq_cft(sf)->
+		private & DRMCG_CTF_PRIV_MASK;
+	enum drmcgrp_res_type type = seq_cft(sf)->
+		private >> DRMCG_CTF_PRIV_SIZE;
+	struct drmcgrp_device *ddev;
+	int i;
+
+	for (i = 0; i <= max_minor; i++) {
+		ddr = drmcgrp->dev_resources[i];
+		ddev = known_drmcgrp_devs[i];
+
+		switch (f_type) {
+		case DRMCGRP_FTYPE_STATS:
+			drmcgrp_print_stats(ddr, sf, type);
+			break;
+		case DRMCGRP_FTYPE_MAX:
+			drmcgrp_print_limits(ddr, sf, type);
+			break;
+		case DRMCGRP_FTYPE_DEFAULT:
+			drmcgrp_print_default(ddev, sf, type);
+			break;
+		case DRMCGRP_FTYPE_HELP:
+			drmcgrp_print_help(i, sf, type);
+			break;
+		default:
+			seq_puts(sf, "\n");
+			break;
+		}
+	}
+
+	return 0;
+}
+
+ssize_t drmcgrp_bo_limit_write(struct kernfs_open_file *of, char *buf,
+		size_t nbytes, loff_t off)
+{
+	struct drmcgrp *drmcgrp = css_drmcgrp(of_css(of));
+	enum drmcgrp_res_type type = of_cft(of)->private >> DRMCG_CTF_PRIV_SIZE;
+	char *cft_name = of_cft(of)->name;
+	char *limits = strstrip(buf);
+	struct drmcgrp_device_resource *ddr;
+	char *sval;
+	s64 val;
+	int i = 0;
+	int rc;
+
+	while (i <= max_minor && limits != NULL) {
+		sval =  strsep(&limits, "\n");
+		rc = kstrtoll(sval, 0, &val);
+
+		if (rc) {
+			pr_err("drmcgrp: %s: minor %d, err %d. ",
+				cft_name, i, rc);
+			pr_cont_cgroup_name(drmcgrp->css.cgroup);
+			pr_cont("\n");
+		} else {
+			ddr = drmcgrp->dev_resources[i];
+			switch (type) {
+			case DRMCGRP_TYPE_BO_TOTAL:
+                                if (val < 0) continue;
+				ddr->bo_limits_total_allocated = val;
+				break;
+			default:
+				break;
+			}
+		}
+
+		i++;
+	}
+
+	if (i <= max_minor) {
+		pr_err("drmcgrp: %s: less entries than # of drm devices. ",
+				cft_name);
+		pr_cont_cgroup_name(drmcgrp->css.cgroup);
+		pr_cont("\n");
+	}
+
+	return nbytes;
+}
+
  struct cftype files[] = {
+	{
+		.name = "buffer.total.stats",
+		.seq_show = drmcgrp_bo_show,
+		.private = (DRMCGRP_TYPE_BO_TOTAL << DRMCG_CTF_PRIV_SIZE) |
+			DRMCGRP_FTYPE_STATS,
+	},
+	{
+		.name = "buffer.total.default",
+		.seq_show = drmcgrp_bo_show,
+		.flags = CFTYPE_ONLY_ON_ROOT,
+		.private = (DRMCGRP_TYPE_BO_TOTAL << DRMCG_CTF_PRIV_SIZE) |
+			DRMCGRP_FTYPE_DEFAULT,
+	},
+	{
+		.name = "buffer.total.help",
+		.seq_show = drmcgrp_bo_show,
+		.flags = CFTYPE_ONLY_ON_ROOT,
+		.private = (DRMCGRP_TYPE_BO_TOTAL << DRMCG_CTF_PRIV_SIZE) |
+			DRMCGRP_FTYPE_HELP,
+	},
+	{
+		.name = "buffer.total.max",
+		.write = drmcgrp_bo_limit_write,
+		.seq_show = drmcgrp_bo_show,
+		.private = (DRMCGRP_TYPE_BO_TOTAL << DRMCG_CTF_PRIV_SIZE) |
+			DRMCGRP_FTYPE_MAX,
+	},
  	{ }	/* terminate */
  };
 

@@ -122,6 +322,8 @@ int drmcgrp_register_device(struct drm_device *dev)
  		return -ENOMEM;
 
  	ddev->dev = dev;
+	ddev->bo_limits_total_allocated_default = S64_MAX;
+
  	mutex_init(&ddev->mutex);
 
  	mutex_lock(&drmcgrp_mutex);

@@ -156,3 +358,81 @@ int drmcgrp_unregister_device(struct drm_device *dev)
  	return 0;
  }
  EXPORT_SYMBOL(drmcgrp_unregister_device);
+
+bool drmcgrp_is_self_or_ancestor(struct drmcgrp *self, struct drmcgrp *relative)
+{
+	for (; self != NULL; self = parent_drmcgrp(self))
+		if (self == relative)
+			return true;
+
+	return false;
+}
+EXPORT_SYMBOL(drmcgrp_is_self_or_ancestor);
+
+bool drmcgrp_bo_can_allocate(struct task_struct *task, struct drm_device *dev,
+		size_t size)
+{
+	struct drmcgrp *drmcgrp = get_drmcgrp(task);
+	struct drmcgrp_device_resource *ddr;
+	struct drmcgrp_device_resource *d;
+	int devIdx = dev->primary->index;
+	bool result = true;
+	s64 delta = 0;
+
+	if (drmcgrp == NULL || drmcgrp == root_drmcgrp)
+		return true;
+
+	ddr = drmcgrp->dev_resources[devIdx];
+	mutex_lock(&known_drmcgrp_devs[devIdx]->mutex);
+	for ( ; drmcgrp != root_drmcgrp; drmcgrp = parent_drmcgrp(drmcgrp)) {
+		d = drmcgrp->dev_resources[devIdx];
+		delta = d->bo_limits_total_allocated -
+				d->bo_stats_total_allocated;
+
+		if (delta <= 0 || size > delta) {
+			result = false;
+			break;
+		}
+	}
+	mutex_unlock(&known_drmcgrp_devs[devIdx]->mutex);
+
+	return result;
+}
+EXPORT_SYMBOL(drmcgrp_bo_can_allocate);
+
+void drmcgrp_chg_bo_alloc(struct drmcgrp *drmcgrp, struct drm_device *dev,
+		size_t size)
+{
+	struct drmcgrp_device_resource *ddr;
+	int devIdx = dev->primary->index;
+
+	if (drmcgrp == NULL || known_drmcgrp_devs[devIdx] == NULL)
+		return;
+
+	mutex_lock(&known_drmcgrp_devs[devIdx]->mutex);
+	for ( ; drmcgrp != NULL; drmcgrp = parent_drmcgrp(drmcgrp)) {
+		ddr = drmcgrp->dev_resources[devIdx];
+
+		ddr->bo_stats_total_allocated += (s64)size;
+	}
+	mutex_unlock(&known_drmcgrp_devs[devIdx]->mutex);
+}
+EXPORT_SYMBOL(drmcgrp_chg_bo_alloc);
+
+void drmcgrp_unchg_bo_alloc(struct drmcgrp *drmcgrp, struct drm_device *dev,
+		size_t size)
+{
+	struct drmcgrp_device_resource *ddr;
+	int devIdx = dev->primary->index;
+
+	if (drmcgrp == NULL || known_drmcgrp_devs[devIdx] == NULL)