x86,fs/resctrl: Support for Global Bandwidth Enforcement and Priviledge Level Zero Association

On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QOS domain. However, multiple QOS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, user must statically partition the available system bandwidth between the active QOS domains. This typically

results in system memory being under-utilized since not all QOS domains are using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS Domains. This collection of QOS domains is referred to as GLBE control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service(COS) across every QoS domain managed by the GLBE control domain.

Hi Babu,

Quoted 5 lines · Babu Moger

On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QOS domain. However, multiple QOS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, user must statically partition the available system bandwidth between the active QOS domains. This typically

How do you define "active" QoS Domain?

Quoted 10 lines

results in system memory being under-utilized since not all QOS domains are using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS Domains. This collection of QOS domains is referred to as GLBE control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service(COS) across every QoS domain managed by the GLBE control domain.

How does this bandwidth allocation limit impact existing MBA? For example, if a system has two domains (A and B) that user space separately sets MBA allocations for while also placing both domains within a "GLBE control domain" with a different allocation, does the individual MBA allocations still matter?

Quoted 1 line

From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

Hi Reinette,

Quoted 9 lines · Reinette Chatre

Hi Babu,

Quoted 5 lines · Babu Moger

On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QOS domain. However, multiple QOS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, user must statically partition the available system bandwidth between the active QOS domains. This typically

Quoted 1 line

How do you define "active" QoS Domain?

Some domains may not have any CPUs associated with that CLOSID. Active meant, I'm referring to domains that have CPUs assigned to the CLOSID.

Quoted 12 lines

Quoted 11 lines

results in system memory being under-utilized since not all QOS domains are using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS Domains. This collection of QOS domains is referred to as GLBE control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service(COS) across every QoS domain managed by the GLBE control domain.

Quoted 4 lines

How does this bandwidth allocation limit impact existing MBA? For example, if a system has two domains (A and B) that user space separately sets MBA allocations for while also placing both domains within a "GLBE control domain" with a different allocation, does the individual MBA allocations still matter?

Yes. Both ceilings are enforced at their respective levels.
The MBA ceiling is applied at the QoS domain level.
The GLBE ceiling is applied at the GLBE control  domain level.
If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling.

Quoted 4 lines

Quoted 1 line

From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

Quoted 3 lines

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

Quoted 4 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

Hi Babu,

Quoted 12 lines · Babu

Hi Reinette,

Quoted 8 lines · Reinette Chatre

Hi Babu,

Quoted 5 lines · Babu Moger

On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QOS domain. However, multiple QOS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, user must statically partition the available system bandwidth between the active QOS domains. This typically

Quoted 2 lines

Quoted 1 line

How do you define "active" QoS Domain?

Some domains may not have any CPUs associated with that CLOSID. Active meant, I'm referring to domains that have CPUs assigned to the CLOSID.

To confirm, is this then specific to assigning CPUs to resource groups via the cpus/cpus_list files? This refers to how a user needs to partition available bandwidth so I am still trying to understand the message here since users still need to do this even when CPUs are not assigned to resource groups.

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

results in system memory being under-utilized since not all QOS domains are using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS Domains. This collection of QOS domains is referred to as GLBE control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service(COS) across every QoS domain managed by the GLBE control domain.

Quoted 8 lines

Quoted 4 lines

How does this bandwidth allocation limit impact existing MBA? For example, if a system has two domains (A and B) that user space separately sets MBA allocations for while also placing both domains within a "GLBE control domain" with a different allocation, does the individual MBA allocations still matter?

Yes. Both ceilings are enforced at their respective levels.
The MBA ceiling is applied at the QoS domain level.
The GLBE ceiling is applied at the GLBE control  domain level.
If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling.

It sounds as though MBA and GMBA/GLBE operates within the same parameters wrt the limits but in examples in this series they have different limits. For example, in the documentation patch [1] there is this:

  # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=4096;1=4096;2=4096;3=4096
     L3:0=ffff;1=ffff;2=ffff;3=ffff

followed up with what it will look like in new generation [2]:

    GMB:0=4096;1=4096;2=4096;3=4096
     MB:0=8192;1=8192;2=8192;3=8192
      L3:0=ffff;1=ffff;2=ffff;3=ffff

In both examples the per-domain MB ceiling is higher than the global GMB ceiling. With above showing defaults and you state "If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling." - does this mean that MB ceiling can never be higher than GMB ceiling as shown in the examples?

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=2048;1=2048;2=2048;3=2048

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:

   # echo "GMB:0=8;2=8" > schemata
   # cat schemata
   GMB:0=8;1=2048;2=8;3=2048
   MB:0=8;1=2048;2=8;3=2048

... and then when user space resets GMB the MB can reset like ...

   # echo "GMB:0=2048;2=2048" > schemata
   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=8;1=2048;2=8;3=2048

   # echo "GMB:0=8;2=8" > schemata
   # cat schemata
   GMB:0=8;1=2048;2=8;3=2048
   MB:0=8;1=2048;2=8;3=2048

   # echo "GMB:0=2048;2=2048" > schemata
   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=8;1=2048;2=8;3=2048

What would be most intuitive way for user to interact with the interfaces?

Quoted 8 lines

Quoted 8 lines

Quoted 2 lines

From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

Quoted 4 lines

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

No. When enumerating the features the number of CLOSID supported by each is enumerated separately. That means GMBA and MBA may support different number of CLOSID. My question is: "under which scenario will GMBA and MBA support different CLOSID?"

Because of a possible difference in number of CLOSIDs it seems the feature supports possible scenarios where some resource groups can support global AND per-domain limits while other resource groups can just support global or just support per-domain limits. Is this correct?

Quoted 6 lines

Quoted 5 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so that the features you are enabling can be supported well.

Reinette

[1] https://lore.kernel.org/lkml/d58f70592a4ce89e744e7378e49d5a36be3fd05e.1769029977.git.babu.moger@amd.com/
[2] https://lore.kernel.org/lkml/[email protected]/

Quoted 8 lines · Reinette Chatre

Quoted 7 lines · Babu

Quoted 6 lines · Reinette Chatre

Quoted 5 lines · Babu Moger

On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QOS domain. However, multiple QOS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, user must statically partition the available system bandwidth between the active QOS domains. This typically

Quoted 8 lines

Quoted 2 lines

Quoted 1 line

How do you define "active" QoS Domain?

Some domains may not have any CPUs associated with that CLOSID. Active meant, I'm referring to domains that have CPUs assigned to the CLOSID.

To confirm, is this then specific to assigning CPUs to resource groups via the cpus/cpus_list files? This refers to how a user needs to partition available bandwidth so I am still trying to understand the message here since users still need to do this even when CPUs are not assigned to resource groups.

It is not specific to CPU assignment. It applies to task assignment also.
For example:  We have 4 domains;

# cat schemata
  MB:0=8192;1=8192;2=8192;3=8192

If this group has the CPUs assigned to only first two domains. Then the group has only two active domains. Then we will only update the first two domains. The MB values in other domains does not matter.

#echo "MB:0=8;1=8" > schemata

# cat schemata
  MB:0=8;1=8;2=8192;3=8192

The combined bandwidth can go up to 16(8+8) units. Each unit is 1/8 GB.

With GMBA, we can set the combined limit higher level and total bandwidth will not exceed GMBA limit.

Quoted 33 lines

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

results in system memory being under-utilized since not all QOS domains are using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS Domains. This collection of QOS domains is referred to as GLBE control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service(COS) across every QoS domain managed by the GLBE control domain.

Quoted 8 lines

Quoted 4 lines

How does this bandwidth allocation limit impact existing MBA? For example, if a system has two domains (A and B) that user space separately sets MBA allocations for while also placing both domains within a "GLBE control domain" with a different allocation, does the individual MBA allocations still matter?

Yes. Both ceilings are enforced at their respective levels.
The MBA ceiling is applied at the QoS domain level.
The GLBE ceiling is applied at the GLBE control  domain level.
If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling.

It sounds as though MBA and GMBA/GLBE operates within the same parameters wrt the limits but in examples in this series they have different limits. For example, in the documentation patch [1] there is this:

  # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=4096;1=4096;2=4096;3=4096
     L3:0=ffff;1=ffff;2=ffff;3=ffff

followed up with what it will look like in new generation [2]:

    GMB:0=4096;1=4096;2=4096;3=4096
     MB:0=8192;1=8192;2=8192;3=8192
      L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 4 lines

In both examples the per-domain MB ceiling is higher than the global GMB ceiling. With above showing defaults and you state "If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling." - does this mean that MB ceiling can never be higher than GMB ceiling as shown in the examples?

That is correct.  There is one more information here.   The MB unit is in 1/8 GB and GMB unit is 1GB.  I have added that in documentation in patch 4.

The GMB limit defaults to max value 4096 (bit 12 set) when the new group is created.  Meaning GMB limit does not apply by default.

When setting the limits, it should be set to same value in all the domains in GMB control domain.  Having different value in each domain results in unexpected behavior.

Quoted 12 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=2048;1=2048;2=2048;3=2048

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:

      # echo "GMB:0=8;2=8" > schemata
   # cat schemata
   GMB:0=8;1=2048;2=8;3=2048
   MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Quoted 21 lines

... and then when user space resets GMB the MB can reset like ...

   # echo "GMB:0=2048;2=2048" > schemata
   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=8;1=2048;2=8;3=2048

   # echo "GMB:0=8;2=8" > schemata
   # cat schemata
   GMB:0=8;1=2048;2=8;3=2048
   MB:0=8;1=2048;2=8;3=2048

Quoted 6 lines

   # echo "GMB:0=2048;2=2048" > schemata
   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=8;1=2048;2=8;3=2048

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

We also need to track the previous settings so we can revert to the earlier value when needed. The best approach is to document this behavior clearly.

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

 From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

Quoted 7 lines

Quoted 4 lines

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

No. When enumerating the features the number of CLOSID supported by each is enumerated separately. That means GMBA and MBA may support different number of CLOSID. My question is: "under which scenario will GMBA and MBA support different CLOSID?"

No. There is not such scenario.

Quoted 4 lines

Because of a possible difference in number of CLOSIDs it seems the feature supports possible scenarios where some resource groups can support global AND per-domain limits while other resource groups can just support global or just support per-domain limits. Is this correct?

System can support up to 16 CLOSIDs. All of them support all the features LLC, MB, GMB, SMBA.   Yes. We have separate enumeration for  each feature.  Are you suggesting to change it ?

Quoted 7 lines

Quoted 5 lines

Quoted 4 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so that the features you are enabling can be supported well.

Yes, I am trying. I noticed that the proposal appears to affect how the schemata information is displayed(in info directory). It seems to introduce additional resource information. I don't see any harm in displaying it if it benefits certain architecture.

Thanks

Babu

Quoted 6 lines

Reinette

[1] https://lore.kernel.org/lkml/d58f70592a4ce89e744e7378e49d5a36be3fd05e.1769029977.git.babu.moger@amd.com/
[2] https://lore.kernel.org/lkml/[email protected]/

Hi Babu,

Quoted 24 lines · Babu Moger

Quoted 8 lines · Reinette Chatre

Quoted 7 lines · Babu

Quoted 6 lines · Reinette Chatre

Quoted 5 lines · Babu Moger

On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QOS domain. However, multiple QOS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, user must statically partition the available system bandwidth between the active QOS domains. This typically

Quoted 8 lines

Quoted 2 lines

Quoted 1 line

How do you define "active" QoS Domain?

Some domains may not have any CPUs associated with that CLOSID. Active meant, I'm referring to domains that have CPUs assigned to the CLOSID.

To confirm, is this then specific to assigning CPUs to resource groups via the cpus/cpus_list files? This refers to how a user needs to partition available bandwidth so I am still trying to understand the message here since users still need to do this even when CPUs are not assigned to resource groups.

It is not specific to CPU assignment. It applies to task assignment also.

For example:  We have 4 domains;

# cat schemata
   MB:0=8192;1=8192;2=8192;3=8192

Quoted 1 line

If this group has the CPUs assigned to only first two domains. Then the group has only two active domains. Then we will only update the first two domains. The MB values in other domains does not matter.

I see, thank you. As I understand an "active QoS domain" is something only user space can designate. It may be possible for resctrl to get a sense of which QoS domains are "active" when only CPUs are assigned to a resource group but when it comes to task assignment it is user space that controls where tasks belonging to a group can be scheduled and thus which QoS domains are "active" or not.

Quoted 8 lines

#echo"MB:0=8;1=8" > schemata

# cat schemata
   MB:0=8;1=8;2=8192;3=8192

The combined bandwidth can go up to 16(8+8) units. Each unit is 1/8 GB.

With GMBA, we can set the combined limit higher level and total bandwidth will not exceed GMBA limit.

Thank you for the confirmation.

Quoted 32 lines

Quoted 32 lines

Quoted 9 lines

Quoted 9 lines

Quoted 9 lines

results in system memory being under-utilized since not all QOS domains are using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS Domains. This collection of QOS domains is referred to as GLBE control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service(COS) across every QoS domain managed by the GLBE control domain.

Quoted 8 lines

Quoted 4 lines

How does this bandwidth allocation limit impact existing MBA? For example, if a system has two domains (A and B) that user space separately sets MBA allocations for while also placing both domains within a "GLBE control domain" with a different allocation, does the individual MBA allocations still matter?

Yes. Both ceilings are enforced at their respective levels.
The MBA ceiling is applied at the QoS domain level.
The GLBE ceiling is applied at the GLBE control  domain level.
If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling.

It sounds as though MBA and GMBA/GLBE operates within the same parameters wrt the limits but in examples in this series they have different limits. For example, in the documentation patch [1] there is this:

   # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=4096;1=4096;2=4096;3=4096
      L3:0=ffff;1=ffff;2=ffff;3=ffff

followed up with what it will look like in new generation [2]:

     GMB:0=4096;1=4096;2=4096;3=4096
      MB:0=8192;1=8192;2=8192;3=8192
       L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 5 lines

Quoted 4 lines

In both examples the per-domain MB ceiling is higher than the global GMB ceiling. With above showing defaults and you state "If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling." - does this mean that MB ceiling can never be higher than GMB ceiling as shown in the examples?

That is correct.  There is one more information here.   The MB unit is in 1/8 GB and GMB unit is 1GB.  I have added that in documentation in patch 4.

ah - right. I did not take the different units into account.

Quoted 15 lines

The GMB limit defaults to max value 4096 (bit 12 set) when the new group is created.  Meaning GMB limit does not apply by default.

When setting the limits, it should be set to same value in all the domains in GMB control domain.  Having different value in each domain results in unexpected behavior.

Quoted 11 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=2048;1=2048;2=2048;3=2048

Quoted 7 lines

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:
      # echo"GMB:0=8;2=8" > schemata

    # cat schemata
    GMB:0=8;1=2048;2=8;3=2048
    MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Thank you for confirming.

Quoted 24 lines

Quoted 24 lines

... and then when user space resets GMB the MB can reset like ...

   # echo"GMB:0=2048;2=2048" > schemata

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=8;1=2048;2=8;3=2048

   # echo"GMB:0=8;2=8" > schemata

    # cat schemata
    GMB:0=8;1=2048;2=8;3=2048
    MB:0=8;1=2048;2=8;3=2048

   # echo"GMB:0=2048;2=2048" > schemata

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=8;1=2048;2=8;3=2048

Quoted 2 lines

Quoted 1 line

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Indeed. My goal is to get an idea how user space may interact with the new interfaces and what would be a reasonable expectation from resctrl be during these interactions.

Quoted 1 line

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

hmmm ... this may be subjective. Could you please elaborate how presenting the effective settings may cause confusion?

Quoted 1 line

We also need to track the previous settings so we can revert to the earlier value when needed. The best approach is to document this behavior clearly.

Yes, this will require resctrl to maintain more state.

Documenting behavior is an option but I think we should first consider if there are things resctrl can do to make the interface intuitive to use.

Quoted 18 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

 From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

Quoted 7 lines

Quoted 4 lines

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

No. When enumerating the features the number of CLOSID supported by each is enumerated separately. That means GMBA and MBA may support different number of CLOSID. My question is: "under which scenario will GMBA and MBA support different CLOSID?"

No. There is not such scenario.

Quoted 3 lines

Because of a possible difference in number of CLOSIDs it seems the feature supports possible scenarios where some resource groups can support global AND per-domain limits while other resource groups can just support global or just support per-domain limits. Is this correct?

Quoted 1 line

System can support up to 16 CLOSIDs. All of them support all the features LLC, MB, GMB, SMBA.   Yes. We have separate enumeration for  each feature.  Are you suggesting to change it ?

It is not a concern to have different CLOSIDs between resources that are actually different, for example, having LLC or MB support different number of CLOSIDs. Having the possibility to allocate the *same* resource (memory bandwidth) with varying number of CLOSIDs does present a challenge though. Would it be possible to have a snippet in the spec that explicitly states that MB and GMB will always enumerate with the same number of CLOSIDs?

Please see below where I will try to support this request more clearly and you can decide if it is reasonable.

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Quoted 4 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so that the features you are enabling can be supported well.

Yes, I am trying. I noticed that the proposal appears to affect how the schemata information is displayed(in info directory). It seems to introduce additional resource information. I don't see any harm in displaying it if it benefits certain architecture.

It benefits all architectures.

There are two parts to the current proposals.

Part 1: Generic schema description
I believe there is consensus on this approach. This is actually something that is long overdue and something like this would have been a great to have with the initial AMD enabling. With the generic schema description forming part of resctrl the user can learn from resctrl how to interact with the schemata file instead of relying on external information and documentation.

For example, on an Intel system that uses percentage based proportional allocation for memory bandwidth the new resctrl files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:all
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:100 info/MB/resource_schemata/MB/tolerance:0
info/MB/resource_schemata/MB/max:100
info/MB/resource_schemata/MB/min:10

On an AMD system that uses absolute allocation with 1/8 GBps steps the files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:GBps
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:8 info/MB/resource_schemata/MB/tolerance:0 info/MB/resource_schemata/MB/max:2048
info/MB/resource_schemata/MB/min:1

Having such interface will be helpful today. Users do not need to first figure out whether they are on an AMD or Intel system, and then read the docs to learn the AMD units, before interacting with resctrl. resctrl will be the generic interface it intends to be.

Part 2: Supporting multiple controls for a single resource
This is a new feature on which there also appears to be consensus that is needed by MPAM and Intel RDT where it is possible to use different controls for the same resource. For example, there can be a minimum and maximum control associated with the memory bandwidth resource.

For example,
info/

  └─ MB/
      └─ resource_schemata/
          ├─ MB/
          ├─ MB_MIN/
          ├─ MB_MAX/
          ┆

Here is where the big question comes in for GLBE - is this actually a new resource for which resctrl needs to add interfaces to manage its allocation, or is it instead an additional control associated with the existing memory bandwith resource?

For me things are actually pointing to GLBE not being a new resource but instead being a new control for the existing memory bandwidth resource.

I understand that for a PoC it is simplest to add support for GLBE as a new resource as is done in this series but when considering it as an actual unique resource does not seem appropriate since resctrl already has a "memory bandwidth" resource. User space expects to find all the resources that it can allocate in info/ - I do not think it is correct to have two separate directories/resources for memory bandwidth here.

What if, instead, it looks something like:

info/

└── MB/
     └── resource_schemata/
         ├── GMB/
         │   ├──max:4096
         │   ├──min:1
         │   ├──resolution:1
         │   ├──scale:1
         │   ├──tolerance:0
         │   ├──type:scalar linear
         │   └──unit:GBps

         └── MB/
             ├──max:8192
             ├──min:1
             ├──resolution:8
             ├──scale:1
             ├──tolerance:0
             ├──type:scalar linear
             └──unit:GBps

With an interface like above GMB is just another control/schema used to allocate the existing memory bandwidth resource. With the planned files it is possible to express the different maximums and units used by the MB and GMB schema. Users no longer need to dig for the unit information in the docs, it is available in the interface.

Doing something like this does depend on GLBE supporting the same number of CLOSIDs as MB, which seems to be how this will be implemented. If there is indeed a confirmation of this from AMD architecture then we can do something like this in resctrl.

Hi Reinette,

Quoted 7 lines · Reinette Chatre

Hi Babu,

Quoted 4 lines · Babu Moger

Quoted 3 lines · Reinette Chatre

Quoted 2 lines · Babu

Quoted 1 line · Reinette Chatre

On 1/21/26 1:12 PM, Babu Moger wrote:

Quoted 49 lines

Quoted 11 lines

Quoted 11 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=2048;1=2048;2=2048;3=2048

Quoted 7 lines

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:
       # echo"GMB:0=8;2=8" > schemata

    # cat schemata
    GMB:0=8;1=2048;2=8;3=2048
    MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Thank you for confirming.

Quoted 19 lines

Quoted 19 lines

... and then when user space resets GMB the MB can reset like ...

    # echo"GMB:0=2048;2=2048" > schemata

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=8;1=2048;2=8;3=2048

    # echo"GMB:0=8;2=8" > schemata

    # cat schemata
    GMB:0=8;1=2048;2=8;3=2048
    MB:0=8;1=2048;2=8;3=2048

Quoted 7 lines

Quoted 6 lines

    # echo"GMB:0=2048;2=2048" > schemata

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=8;1=2048;2=8;3=2048

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Indeed. My goal is to get an idea how user space may interact with the new interfaces and what would be a reasonable expectation from resctrl be during these interactions.

Quoted 3 lines

Quoted 1 line

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

hmmm ... this may be subjective. Could you please elaborate how presenting the effective settings may cause confusion?

I mean in many cases, we cannot determine the effective settings correctly. It depends on benchmarks or applications running on the system.

Even with MB (without GMB support), even though we set the limit to 10GB, it may not use the whole 10GB.  Memory is shared resource. So, the effective bandwidth usage depends on other applications running on the system.

Quoted 24 lines

Quoted 1 line

We also need to track the previous settings so we can revert to the earlier value when needed. The best approach is to document this behavior clearly.

Yes, this will require resctrl to maintain more state.

Documenting behavior is an option but I think we should first consider if there are things resctrl can do to make the interface intuitive to use.

Quoted 18 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

  From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

Quoted 7 lines

Quoted 4 lines

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

No. When enumerating the features the number of CLOSID supported by each is enumerated separately. That means GMBA and MBA may support different number of CLOSID. My question is: "under which scenario will GMBA and MBA support different CLOSID?"

No. There is not such scenario.

Quoted 3 lines

Because of a possible difference in number of CLOSIDs it seems the feature supports possible scenarios where some resource groups can support global AND per-domain limits while other resource groups can just support global or just support per-domain limits. Is this correct?

Quoted 6 lines

Quoted 1 line

System can support up to 16 CLOSIDs. All of them support all the features LLC, MB, GMB, SMBA.   Yes. We have separate enumeration for  each feature.  Are you suggesting to change it ?

It is not a concern to have different CLOSIDs between resources that are actually different, for example, having LLC or MB support different number of CLOSIDs. Having the possibility to allocate the *same* resource (memory bandwidth) with varying number of CLOSIDs does present a challenge though. Would it be possible to have a snippet in the spec that explicitly states that MB and GMB will always enumerate with the same number of CLOSIDs?

I have confirmed that is the case always.  All current and planned implementations, MB and GMB will have the same number of CLOSIDs.

Quoted 13 lines

Please see below where I will try to support this request more clearly and you can decide if it is reasonable.

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Quoted 4 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so that the features you are enabling can be supported well.

Yes, I am trying. I noticed that the proposal appears to affect how the schemata information is displayed(in info directory). It seems to introduce additional resource information. I don't see any harm in displaying it if it benefits certain architecture.

It benefits all architectures.

Quoted 8 lines

There are two parts to the current proposals.

Part 1: Generic schema description
I believe there is consensus on this approach. This is actually something that is long overdue and something like this would have been a great to have with the initial AMD enabling. With the generic schema description forming part of resctrl the user can learn from resctrl how to interact with the schemata file instead of relying on external information and documentation.

ok.

Quoted 20 lines

For example, on an Intel system that uses percentage based proportional allocation for memory bandwidth the new resctrl files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:all
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:100 info/MB/resource_schemata/MB/tolerance:0
info/MB/resource_schemata/MB/max:100
info/MB/resource_schemata/MB/min:10

On an AMD system that uses absolute allocation with 1/8 GBps steps the files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:GBps
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:8 info/MB/resource_schemata/MB/tolerance:0 info/MB/resource_schemata/MB/max:2048
info/MB/resource_schemata/MB/min:1

Quoted 3 lines

Having such interface will be helpful today. Users do not need to first figure out whether they are on an AMD or Intel system, and then read the docs to learn the AMD units, before interacting with resctrl. resctrl will be the generic interface it intends to be.

Yes. That is a good point.

Quoted 15 lines

Part 2: Supporting multiple controls for a single resource
This is a new feature on which there also appears to be consensus that is needed by MPAM and Intel RDT where it is possible to use different controls for the same resource. For example, there can be a minimum and maximum control associated with the memory bandwidth resource.

For example,
info/

  └─ MB/
      └─ resource_schemata/
          ├─ MB/
          ├─ MB_MIN/
          ├─ MB_MAX/
          ┆

Quoted 3 lines

Here is where the big question comes in for GLBE - is this actually a new resource for which resctrl needs to add interfaces to manage its allocation, or is it instead an additional control associated with the existing memory bandwith resource?

It is not a new resource. It is new control mechanism to address limitation with memory bandwidth resource.

So, it is a new control for the existing memory bandwidth resource.

Quoted 23 lines

For me things are actually pointing to GLBE not being a new resource but instead being a new control for the existing memory bandwidth resource.

I understand that for a PoC it is simplest to add support for GLBE as a new resource as is done in this series but when considering it as an actual unique resource does not seem appropriate since resctrl already has a "memory bandwidth" resource. User space expects to find all the resources that it can allocate in info/ - I do not think it is correct to have two separate directories/resources for memory bandwidth here.

What if, instead, it looks something like:

info/

└── MB/
     └── resource_schemata/
         ├── GMB/
         │   ├──max:4096
         │   ├──min:1
         │   ├──resolution:1
         │   ├──scale:1
         │   ├──tolerance:0
         │   ├──type:scalar linear
         │   └──unit:GBps

Quoted 8 lines

         └── MB/
             ├──max:8192
             ├──min:1
             ├──resolution:8
             ├──scale:1
             ├──tolerance:0
             ├──type:scalar linear
             └──unit:GBps

Yes. It definitely looks very clean.

Quoted 4 lines

With an interface like above GMB is just another control/schema used to allocate the existing memory bandwidth resource. With the planned files it is possible to express the different maximums and units used by the MB and GMB schema. Users no longer need to dig for the unit information in the docs, it is available in the interface.

Yes. That is reasonable.

Is the plan to just update the resource information in /sys/fs/resctrl/info/<resource_name>  ?

Also, will the display of /sys/fs/resctrl/schemata change ?

Current display:

 GMB:0=4096;1=4096;2=4096;3=4096
  MB:0=8192;1=8192;2=8192;3=8192

Quoted 3 lines

Doing something like this does depend on GLBE supporting the same number of CLOSIDs as MB, which seems to be how this will be implemented. If there is indeed a confirmation of this from AMD architecture then we can do something like this in resctrl.

I don't see this being an issue. I will get consensus on it.

I am wondering about the time frame and who is leading this change. Not sure if that is been discussed already. I can definitely help.

Thanks

Babu

Hi Babu,

Quoted 10 lines · Babu Moger

Hi Reinette,

Quoted 7 lines · Reinette Chatre

Hi Babu,

Quoted 4 lines · Babu Moger

Quoted 3 lines · Reinette Chatre

Quoted 2 lines · Babu

Quoted 1 line · Reinette Chatre

On 1/21/26 1:12 PM, Babu Moger wrote:

...

Quoted 53 lines

Quoted 52 lines

Quoted 11 lines

Quoted 11 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

     # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=2048;1=2048;2=2048;3=2048

Quoted 7 lines

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:
       # echo"GMB:0=8;2=8" > schemata

     # cat schemata
     GMB:0=8;1=2048;2=8;3=2048
     MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Thank you for confirming.

Quoted 19 lines

Quoted 19 lines

... and then when user space resets GMB the MB can reset like ...

    # echo"GMB:0=2048;2=2048" > schemata

     # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

     # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=8;1=2048;2=8;3=2048

    # echo"GMB:0=8;2=8" > schemata

     # cat schemata
     GMB:0=8;1=2048;2=8;3=2048
     MB:0=8;1=2048;2=8;3=2048

Quoted 7 lines

Quoted 6 lines

    # echo"GMB:0=2048;2=2048" > schemata

     # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=8;1=2048;2=8;3=2048

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Indeed. My goal is to get an idea how user space may interact with the new interfaces and what would be a reasonable expectation from resctrl be during these interactions.

Quoted 1 line

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

hmmm ... this may be subjective. Could you please elaborate how presenting the effective settings may cause confusion?

Quoted 3 lines

I mean in many cases, we cannot determine the effective settings correctly. It depends on benchmarks or applications running on the system.

Even with MB (without GMB support), even though we set the limit to 10GB, it may not use the whole 10GB.  Memory is shared resource. So, the effective bandwidth usage depends on other applications running on the system.

Sounds like we interpret "effective limits" differently. To me the limits(*) are deterministic. If I understand correctly, if the GMB limit for domains A and B is set to x GB then that places an x GB limit on MB for domains A and B also. Displaying any MB limit in the schemata that is larger than x GB for domain A or domain B would be inaccurate, no?

When considering your example where the MB limit is 10GB.

Consider an example where there are two domains in this example with a configuration like below. (I am using a different syntax from schemata file that will hopefully make it easier to exchange ideas when not having to interpret the different GMB and MB units):

	MB:0=10GB;1=10GB

If user space can create a GMB domain that limits shared bandwidth to 10GB that can be displayed as below and will be accurate:

	MB:0=10GB;1=10GB
	GMB:0=10GB;1=10GB

If user space then reduces the combined bandwidth to 2GB then the MB limit is wrong since it is actually capped by the GMB limit:

	MB:0=10GB;1=10GB <==== Does reflect possible per-domain memory bandwidth which is now capped by GMB
	GMB:0=2GB;1=2GB

Would something like below not be more accurate that reflects that the maximum average bandwidth each domain could achieve is 2GB?

	MB:0=2GB;1=2GB <==== Reflects accurate possible per-domain memory bandwidth
	GMB:0=2GB;1=2GB

(*) As a side-note we may have to start being careful with how we use "limits" because of the planned introduction of a "MAX" as a bandwidth control that is an actual limit as opposed to the current control that is approximate.

Quoted 31 lines

Quoted 30 lines

Quoted 1 line

We also need to track the previous settings so we can revert to the earlier value when needed. The best approach is to document this behavior clearly.

Yes, this will require resctrl to maintain more state.

Documenting behavior is an option but I think we should first consider if there are things resctrl can do to make the interface intuitive to use.

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Quoted 7 lines

Quoted 1 line

  From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

Quoted 8 lines

Quoted 3 lines

No. When enumerating the features the number of CLOSID supported by each is enumerated separately. That means GMBA and MBA may support different number of CLOSID. My question is: "under which scenario will GMBA and MBA support different CLOSID?"

No. There is not such scenario.

Quoted 3 lines

Because of a possible difference in number of CLOSIDs it seems the feature supports possible scenarios where some resource groups can support global AND per-domain limits while other resource groups can just support global or just support per-domain limits. Is this correct?

System can support up to 16 CLOSIDs. All of them support all the features LLC, MB, GMB, SMBA.   Yes. We have separate enumeration for  each feature.  Are you suggesting to change it ?

It is not a concern to have different CLOSIDs between resources that are actually different, for example, having LLC or MB support different number of CLOSIDs. Having the possibility to allocate the *same* resource (memory bandwidth) with varying number of CLOSIDs does present a challenge though. Would it be possible to have a snippet in the spec that explicitly states that MB and GMB will always enumerate with the same number of CLOSIDs?

Quoted 1 line

I have confirmed that is the case always.  All current and planned implementations, MB and GMB will have the same number of CLOSIDs.

Thank you very much for confirming. Is this something the architects would be willing to commit to with a snippet in the PQoS spec?

Quoted 66 lines

Quoted 14 lines

Please see below where I will try to support this request more clearly and you can decide if it is reasonable.

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Quoted 4 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so that the features you are enabling can be supported well.

Yes, I am trying. I noticed that the proposal appears to affect how the schemata information is displayed(in info directory). It seems to introduce additional resource information. I don't see any harm in displaying it if it benefits certain architecture.

It benefits all architectures.

Quoted 8 lines

There are two parts to the current proposals.

Part 1: Generic schema description
I believe there is consensus on this approach. This is actually something that is long overdue and something like this would have been a great to have with the initial AMD enabling. With the generic schema description forming part of resctrl the user can learn from resctrl how to interact with the schemata file instead of relying on external information and documentation.

ok.

Quoted 20 lines

For example, on an Intel system that uses percentage based proportional allocation for memory bandwidth the new resctrl files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:all
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:100 info/MB/resource_schemata/MB/tolerance:0
info/MB/resource_schemata/MB/max:100
info/MB/resource_schemata/MB/min:10

On an AMD system that uses absolute allocation with 1/8 GBps steps the files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:GBps
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:8 info/MB/resource_schemata/MB/tolerance:0 info/MB/resource_schemata/MB/max:2048
info/MB/resource_schemata/MB/min:1

Quoted 3 lines

Having such interface will be helpful today. Users do not need to first figure out whether they are on an AMD or Intel system, and then read the docs to learn the AMD units, before interacting with resctrl. resctrl will be the generic interface it intends to be.

Yes. That is a good point.

Quoted 15 lines

Part 2: Supporting multiple controls for a single resource
This is a new feature on which there also appears to be consensus that is needed by MPAM and Intel RDT where it is possible to use different controls for the same resource. For example, there can be a minimum and maximum control associated with the memory bandwidth resource.

For example,
info/

   └─ MB/
       └─ resource_schemata/
           ├─ MB/
           ├─ MB_MIN/
           ├─ MB_MAX/
           ┆

Quoted 7 lines

Quoted 3 lines

Here is where the big question comes in for GLBE - is this actually a new resource for which resctrl needs to add interfaces to manage its allocation, or is it instead an additional control associated with the existing memory bandwith resource?

It is not a new resource. It is new control mechanism to address limitation with memory bandwidth resource.

So, it is a new control for the existing memory bandwidth resource.

Thank you for confirming.

Quoted 40 lines

Quoted 23 lines

For me things are actually pointing to GLBE not being a new resource but instead being a new control for the existing memory bandwidth resource.

I understand that for a PoC it is simplest to add support for GLBE as a new resource as is done in this series but when considering it as an actual unique resource does not seem appropriate since resctrl already has a "memory bandwidth" resource. User space expects to find all the resources that it can allocate in info/ - I do not think it is correct to have two separate directories/resources for memory bandwidth here.

What if, instead, it looks something like:

info/

└── MB/
      └── resource_schemata/
          ├── GMB/
          │   ├──max:4096
          │   ├──min:1
          │   ├──resolution:1
          │   ├──scale:1
          │   ├──tolerance:0
          │   ├──type:scalar linear
          │   └──unit:GBps

Quoted 8 lines

          └── MB/
              ├──max:8192
              ├──min:1
              ├──resolution:8
              ├──scale:1
              ├──tolerance:0
              ├──type:scalar linear
              └──unit:GBps

Yes. It definitely looks very clean.

Quoted 4 lines

With an interface like above GMB is just another control/schema used to allocate the existing memory bandwidth resource. With the planned files it is possible to express the different maximums and units used by the MB and GMB schema. Users no longer need to dig for the unit information in the docs, it is available in the interface.

Quoted 3 lines

Yes. That is reasonable.

Is the plan to just update the resource information in /sys/fs/resctrl/info/<resource_name>  ?

I do not see any resource information that needs to change. As you confirmed, MB and GMB have the same number of CLOSIDs and looking at the rest of the enumeration done in patch #2 all other properties exposed in top level of /sys/fs/resctrl/info/MB is the same for MB and GMB. Specifically, thread_throttle_mode, delay_linear, min_bandwidth, and bandwidth_gran have the same values for MB and GMB. All other content in /sys/fs/resctrl/info/MB would be new as part of the new "resource_schemata" sub-directory.

Even so, I believe we could expect that a user using any new schemata file entry introduced after the "resource_schemata" directory is introduced is aware of how the properties are exposed and will not use the top level files in /sys/fs/resctrl/info/MB (for example min_bandwidth and bandwidth_gran) to understand how to interact with the new schema.

Quoted 2 lines

Also, will the display of /sys/fs/resctrl/schemata change ?

There are no plans to change any of the existing schemata file entries.

Quoted 2 lines

Current display:

When viewing "current" as what this series does in schemata file ...

Quoted 3 lines

 GMB:0=4096;1=4096;2=4096;3=4096
  MB:0=8192;1=8192;2=8192;3=8192

yes, the schemata file should look like this on boot when all is done. All other user facing changes are to the info/ directory where user space learns about the new control for the resource and how to interact with the control.

Quoted 8 lines

Quoted 3 lines

Doing something like this does depend on GLBE supporting the same number of CLOSIDs as MB, which seems to be how this will be implemented. If there is indeed a confirmation of this from AMD architecture then we can do something like this in resctrl.

I don't see this being an issue. I will get consensus on it.

I am wondering about the time frame and who is leading this change. Not sure if that is been discussed already. I can definitely help.

A couple of features depend on the new schema descriptions as well as support for multiple
controls: min/max bandwidth controls on the MPAM side, region aware MBA and MBM on the Intel side, and GLBE on the AMD side. I am hoping that the folks working on these features can collaborate on the needed foundation. Since there are no patches for this yet I cannot say if there is a leader for this work yet, at this time this role appears to be available if you would like to see this moving forward in order to meet your goals.

Quoted 65 lines · Reinette Chatre

Quoted 6 lines · Babu Moger

Quoted 5 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

Quoted 3 lines · Reinette Chatre

Quoted 2 lines · Babu

Quoted 1 line · Reinette Chatre

On 1/21/26 1:12 PM, Babu Moger wrote:

...

Quoted 55 lines

Quoted 44 lines

Quoted 11 lines

Quoted 11 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

     # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=2048;1=2048;2=2048;3=2048

Quoted 7 lines

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:
        # echo"GMB:0=8;2=8" > schemata

     # cat schemata
     GMB:0=8;1=2048;2=8;3=2048
     MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Thank you for confirming.

Quoted 24 lines

Quoted 24 lines

... and then when user space resets GMB the MB can reset like ...

     # echo"GMB:0=2048;2=2048" > schemata

     # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

     # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=8;1=2048;2=8;3=2048

     # echo"GMB:0=8;2=8" > schemata

     # cat schemata
     GMB:0=8;1=2048;2=8;3=2048
     MB:0=8;1=2048;2=8;3=2048

     # echo"GMB:0=2048;2=2048" > schemata

     # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=8;1=2048;2=8;3=2048

Quoted 8 lines

Quoted 2 lines

Quoted 1 line

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Indeed. My goal is to get an idea how user space may interact with the new interfaces and what would be a reasonable expectation from resctrl be during these interactions.

Quoted 1 line

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

hmmm ... this may be subjective. Could you please elaborate how presenting the effective settings may cause confusion?

I mean in many cases, we cannot determine the effective settings correctly. It depends on benchmarks or applications running on the system.

Quoted 6 lines

Quoted 1 line

Even with MB (without GMB support), even though we set the limit to 10GB, it may not use the whole 10GB.  Memory is shared resource. So, the effective bandwidth usage depends on other applications running on the system.

Sounds like we interpret "effective limits" differently. To me the limits(*) are deterministic. If I understand correctly, if the GMB limit for domains A and B is set to x GB then that places an x GB limit on MB for domains A and B also. Displaying any MB limit in the schemata that is larger than x GB for domain A or domain B would be inaccurate, no?

Yea. But, I was thinking not to mess with values written at registers.

Quoted 20 lines

When considering your example where the MB limit is 10GB.

Consider an example where there are two domains in this example with a configuration like below. (I am using a different syntax from schemata file that will hopefully make it easier to exchange ideas when not having to interpret the different GMB and MB units):

    MB:0=10GB;1=10GB

If user space can create a GMB domain that limits shared bandwidth to 10GB that can be displayed as below and will be accurate:

    MB:0=10GB;1=10GB
    GMB:0=10GB;1=10GB

If user space then reduces the combined bandwidth to 2GB then the MB limit is wrong since it is actually capped by the GMB limit:

    MB:0=10GB;1=10GB <==== Does reflect possible per-domain memory bandwidth which is now capped by GMB GMB:0=2GB;1=2GB

Quoted 5 lines

Would something like below not be more accurate that reflects that the maximum average bandwidth each domain could achieve is 2GB?

    MB:0=2GB;1=2GB <==== Reflects accurate possible per-domain memory bandwidth GMB:0=2GB;1=2GB

That is reasonable. Will check how we can accommodate that.

Quoted 37 lines

(*) As a side-note we may have to start being careful with how we use "limits" because of the planned introduction of a "MAX" as a bandwidth control that is an actual limit as opposed to the current control that is approximate.

Quoted 31 lines

Quoted 30 lines

Quoted 1 line

We also need to track the previous settings so we can revert to the earlier value when needed. The best approach is to document this behavior clearly.

Yes, this will require resctrl to maintain more state.

Documenting behavior is an option but I think we should first consider if there are things resctrl can do to make the interface intuitive to use.

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Quoted 7 lines

Quoted 1 line

   From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

Quoted 8 lines

Quoted 3 lines

No. When enumerating the features the number of CLOSID supported by each is enumerated separately. That means GMBA and MBA may support different number of CLOSID. My question is: "under which scenario will GMBA and MBA support different CLOSID?"

No. There is not such scenario.

Quoted 3 lines

Because of a possible difference in number of CLOSIDs it seems the feature supports possible scenarios where some resource groups can support global AND per-domain limits while other resource groups can just support global or just support per-domain limits. Is this correct?

System can support up to 16 CLOSIDs. All of them support all the features LLC, MB, GMB, SMBA.   Yes. We have separate enumeration for  each feature.  Are you suggesting to change it ?

It is not a concern to have different CLOSIDs between resources that are actually different, for example, having LLC or MB support different number of CLOSIDs. Having the possibility to allocate the *same* resource (memory bandwidth) with varying number of CLOSIDs does present a challenge though. Would it be possible to have a snippet in the spec that explicitly states that MB and GMB will always enumerate with the same number of CLOSIDs?

Quoted 4 lines

Quoted 1 line

I have confirmed that is the case always.  All current and planned implementations, MB and GMB will have the same number of CLOSIDs.

Thank you very much for confirming. Is this something the architects would be willing to commit to with a snippet in the PQoS spec?

I checked on that. Here is the response.

"I do not plan to add a statement like that to the spec.  The CPUID enumeration allows for them to have different number of CLOS's supported for each.  However, it is true that for all current and planned implementations, MB and GMB will have the same number of CLOS."

Quoted 162 lines

Quoted 65 lines

Quoted 13 lines

Please see below where I will try to support this request more clearly and you can decide if it is reasonable.

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Quoted 4 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so that the features you are enabling can be supported well.

Yes, I am trying. I noticed that the proposal appears to affect how the schemata information is displayed(in info directory). It seems to introduce additional resource information. I don't see any harm in displaying it if it benefits certain architecture.

It benefits all architectures.

Quoted 8 lines

There are two parts to the current proposals.

Part 1: Generic schema description
I believe there is consensus on this approach. This is actually something that is long overdue and something like this would have been a great to have with the initial AMD enabling. With the generic schema description forming part of resctrl the user can learn from resctrl how to interact with the schemata file instead of relying on external information and documentation.

ok.

Quoted 20 lines

For example, on an Intel system that uses percentage based proportional allocation for memory bandwidth the new resctrl files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:all
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:100 info/MB/resource_schemata/MB/tolerance:0
info/MB/resource_schemata/MB/max:100
info/MB/resource_schemata/MB/min:10

On an AMD system that uses absolute allocation with 1/8 GBps steps the files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:GBps
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:8 info/MB/resource_schemata/MB/tolerance:0 info/MB/resource_schemata/MB/max:2048
info/MB/resource_schemata/MB/min:1

Quoted 3 lines

Having such interface will be helpful today. Users do not need to first figure out whether they are on an AMD or Intel system, and then read the docs to learn the AMD units, before interacting with resctrl. resctrl will be the generic interface it intends to be.

Yes. That is a good point.

Quoted 15 lines

Part 2: Supporting multiple controls for a single resource
This is a new feature on which there also appears to be consensus that is needed by MPAM and Intel RDT where it is possible to use different controls for the same resource. For example, there can be a minimum and maximum control associated with the memory bandwidth resource.

For example,
info/

   └─ MB/
       └─ resource_schemata/
           ├─ MB/
           ├─ MB_MIN/
           ├─ MB_MAX/
           ┆

Quoted 7 lines

Quoted 3 lines

Here is where the big question comes in for GLBE - is this actually a new resource for which resctrl needs to add interfaces to manage its allocation, or is it instead an additional control associated with the existing memory bandwith resource?

It is not a new resource. It is new control mechanism to address limitation with memory bandwidth resource.

So, it is a new control for the existing memory bandwidth resource.

Thank you for confirming.

Quoted 40 lines

Quoted 23 lines

For me things are actually pointing to GLBE not being a new resource but instead being a new control for the existing memory bandwidth resource.

I understand that for a PoC it is simplest to add support for GLBE as a new resource as is done in this series but when considering it as an actual unique resource does not seem appropriate since resctrl already has a "memory bandwidth" resource. User space expects to find all the resources that it can allocate in info/ - I do not think it is correct to have two separate directories/resources for memory bandwidth here.

What if, instead, it looks something like:

info/

└── MB/
      └── resource_schemata/
          ├── GMB/
          │   ├──max:4096
          │   ├──min:1
          │   ├──resolution:1
          │   ├──scale:1
          │   ├──tolerance:0
          │   ├──type:scalar linear
          │   └──unit:GBps

Quoted 8 lines

          └── MB/
              ├──max:8192
              ├──min:1
              ├──resolution:8
              ├──scale:1
              ├──tolerance:0
              ├──type:scalar linear
              └──unit:GBps

Yes. It definitely looks very clean.

Quoted 4 lines

With an interface like above GMB is just another control/schema used to allocate the existing memory bandwidth resource. With the planned files it is possible to express the different maximums and units used by the MB and GMB schema. Users no longer need to dig for the unit information in the docs, it is available in the interface.

Quoted 3 lines

Yes. That is reasonable.

Is the plan to just update the resource information in /sys/fs/resctrl/info/<resource_name>  ?

I do not see any resource information that needs to change. As you confirmed, MB and GMB have the same number of CLOSIDs and looking at the rest of the enumeration done in patch #2 all other properties exposed in top level of /sys/fs/resctrl/info/MB is the same for MB and GMB. Specifically, thread_throttle_mode, delay_linear, min_bandwidth, and bandwidth_gran have the same values for MB and GMB. All other content in /sys/fs/resctrl/info/MB would be new as part of the new "resource_schemata" sub-directory.

Even so, I believe we could expect that a user using any new schemata file entry introduced after the "resource_schemata" directory is introduced is aware of how the properties are exposed and will not use the top level files in /sys/fs/resctrl/info/MB (for example min_bandwidth and bandwidth_gran) to understand how to interact with the new schema.

Quoted 2 lines

Also, will the display of /sys/fs/resctrl/schemata change ?

There are no plans to change any of the existing schemata file entries.

Quoted 2 lines

Current display:

When viewing "current" as what this series does in schemata file ...

Quoted 3 lines

  GMB:0=4096;1=4096;2=4096;3=4096
   MB:0=8192;1=8192;2=8192;3=8192

yes, the schemata file should look like this on boot when all is done. All other user facing changes are to the info/ directory where user space learns about the new control for the resource and how to interact with the control.

Quoted 8 lines

Quoted 3 lines

Doing something like this does depend on GLBE supporting the same number of CLOSIDs as MB, which seems to be how this will be implemented. If there is indeed a confirmation of this from AMD architecture then we can do something like this in resctrl.

I don't see this being an issue. I will get consensus on it.

I am wondering about the time frame and who is leading this change. Not sure if that is been discussed already. I can definitely help.

Quoted 6 lines

A couple of features depend on the new schema descriptions as well as support for multiple
controls: min/max bandwidth controls on the MPAM side, region aware MBA and MBM on the Intel side, and GLBE on the AMD side. I am hoping that the folks working on these features can collaborate on the needed foundation. Since there are no patches for this yet I cannot say if there is a leader for this work yet, at this time this role appears to be available if you would like to see this moving forward in order to meet your goals.

I joined this feature effort a bit later, so I may not yet have full context on the MPAM and region‑aware requirements. I’m happy to provide all the necessary information for GMB and MB from the AMD side, and I’m also available to help with reviews and testing.

Hi Babu,

Quoted 66 lines · Babu

Quoted 65 lines · Reinette Chatre

Quoted 6 lines · Babu Moger

Quoted 5 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

Quoted 3 lines · Reinette Chatre

Quoted 2 lines · Babu

Quoted 1 line · Reinette Chatre

On 1/21/26 1:12 PM, Babu Moger wrote:

...

Quoted 55 lines

Quoted 44 lines

Quoted 11 lines

Quoted 11 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

      # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=2048;1=2048;2=2048;3=2048

Quoted 7 lines

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:
        # echo"GMB:0=8;2=8" > schemata

      # cat schemata
      GMB:0=8;1=2048;2=8;3=2048
      MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Thank you for confirming.

Quoted 24 lines

Quoted 24 lines

... and then when user space resets GMB the MB can reset like ...

     # echo"GMB:0=2048;2=2048" > schemata

      # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

      # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=8;1=2048;2=8;3=2048

     # echo"GMB:0=8;2=8" > schemata

      # cat schemata
      GMB:0=8;1=2048;2=8;3=2048
      MB:0=8;1=2048;2=8;3=2048

     # echo"GMB:0=2048;2=2048" > schemata

      # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=8;1=2048;2=8;3=2048

Quoted 8 lines

Quoted 2 lines

Quoted 1 line

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Indeed. My goal is to get an idea how user space may interact with the new interfaces and what would be a reasonable expectation from resctrl be during these interactions.

Quoted 1 line

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

hmmm ... this may be subjective. Could you please elaborate how presenting the effective settings may cause confusion?

I mean in many cases, we cannot determine the effective settings correctly. It depends on benchmarks or applications running on the system.

Quoted 8 lines

Quoted 6 lines

Quoted 1 line

Even with MB (without GMB support), even though we set the limit to 10GB, it may not use the whole 10GB.  Memory is shared resource. So, the effective bandwidth usage depends on other applications running on the system.

Sounds like we interpret "effective limits" differently. To me the limits(*) are deterministic. If I understand correctly, if the GMB limit for domains A and B is set to x GB then that places an x GB limit on MB for domains A and B also. Displaying any MB limit in the schemata that is larger than x GB for domain A or domain B would be inaccurate, no?

Yea. But, I was thinking not to mess with values written at registers.

This is not about what is written to the registers but how the combined values written to registers control system behavior and how to accurately reflect the resulting system behavior to user space.

Quoted 23 lines

Quoted 23 lines

When considering your example where the MB limit is 10GB.

Consider an example where there are two domains in this example with a configuration like below. (I am using a different syntax from schemata file that will hopefully make it easier to exchange ideas when not having to interpret the different GMB and MB units):

    MB:0=10GB;1=10GB

If user space can create a GMB domain that limits shared bandwidth to 10GB that can be displayed as below and will be accurate:

    MB:0=10GB;1=10GB
    GMB:0=10GB;1=10GB

If user space then reduces the combined bandwidth to 2GB then the MB limit is wrong since it is actually capped by the GMB limit:

    MB:0=10GB;1=10GB <==== Does reflect possible per-domain memory bandwidth which is now capped by GMB GMB:0=2GB;1=2GB

Would something like below not be more accurate that reflects that the maximum average bandwidth each domain could achieve is 2GB?

Quoted 4 lines

Quoted 2 lines

    MB:0=2GB;1=2GB <==== Reflects accurate possible per-domain memory bandwidth GMB:0=2GB;1=2GB

That is reasonable. Will check how we can accommodate that.

Right, this is not about the values in the L3BE registers but instead how those values are impacted by GLBE registers and how to most accurately present the resulting system configuration to user space. Thank you for considering.

Quoted 38 lines

Quoted 38 lines

(*) As a side-note we may have to start being careful with how we use "limits" because of the planned introduction of a "MAX" as a bandwidth control that is an actual limit as opposed to the current control that is approximate.

Quoted 31 lines

Quoted 30 lines

Quoted 1 line

We also need to track the previous settings so we can revert to the earlier value when needed. The best approach is to document this behavior clearly.

Yes, this will require resctrl to maintain more state.

Documenting behavior is an option but I think we should first consider if there are things resctrl can do to make the interface intuitive to use.

Quoted 11 lines

Quoted 11 lines

Quoted 11 lines

Quoted 7 lines

Quoted 1 line

   From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

Quoted 8 lines

Quoted 3 lines

No. When enumerating the features the number of CLOSID supported by each is enumerated separately. That means GMBA and MBA may support different number of CLOSID. My question is: "under which scenario will GMBA and MBA support different CLOSID?"

No. There is not such scenario.

Quoted 3 lines

Because of a possible difference in number of CLOSIDs it seems the feature supports possible scenarios where some resource groups can support global AND per-domain limits while other resource groups can just support global or just support per-domain limits. Is this correct?

System can support up to 16 CLOSIDs. All of them support all the features LLC, MB, GMB, SMBA.   Yes. We have separate enumeration for  each feature.  Are you suggesting to change it ?

It is not a concern to have different CLOSIDs between resources that are actually different, for example, having LLC or MB support different number of CLOSIDs. Having the possibility to allocate the *same* resource (memory bandwidth) with varying number of CLOSIDs does present a challenge though. Would it be possible to have a snippet in the spec that explicitly states that MB and GMB will always enumerate with the same number of CLOSIDs?

Quoted 8 lines

Quoted 4 lines

Quoted 1 line

I have confirmed that is the case always.  All current and planned implementations, MB and GMB will have the same number of CLOSIDs.

Thank you very much for confirming. Is this something the architects would be willing to commit to with a snippet in the PQoS spec?

I checked on that. Here is the response.

"I do not plan to add a statement like that to the spec.  The CPUID enumeration allows for them to have different number of CLOS's supported for each.  However, it is true that for all current and planned implementations, MB and GMB will have the same number of CLOS."

Thank you for asking. At this time the definition of a resource's "num_closids" is:

"num_closids":
The number of CLOSIDs which are valid for this resource. The kernel uses the smallest number of CLOSIDs of all enabled resources as limit.

Without commitment from architecture we could expand definition of "num_closids" when adding multiple controls to indicate that it is the smallest number of CLOSIDs supported by all controls.

Quoted 171 lines

Quoted 163 lines

Quoted 66 lines

Quoted 14 lines

Please see below where I will try to support this request more clearly and you can decide if it is reasonable.

Quoted 8 lines

Quoted 7 lines

Quoted 5 lines

Quoted 4 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so that the features you are enabling can be supported well.

Yes, I am trying. I noticed that the proposal appears to affect how the schemata information is displayed(in info directory). It seems to introduce additional resource information. I don't see any harm in displaying it if it benefits certain architecture.

It benefits all architectures.

Quoted 8 lines

There are two parts to the current proposals.

Part 1: Generic schema description
I believe there is consensus on this approach. This is actually something that is long overdue and something like this would have been a great to have with the initial AMD enabling. With the generic schema description forming part of resctrl the user can learn from resctrl how to interact with the schemata file instead of relying on external information and documentation.

ok.

Quoted 20 lines

For example, on an Intel system that uses percentage based proportional allocation for memory bandwidth the new resctrl files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:all
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:100 info/MB/resource_schemata/MB/tolerance:0
info/MB/resource_schemata/MB/max:100
info/MB/resource_schemata/MB/min:10

On an AMD system that uses absolute allocation with 1/8 GBps steps the files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:GBps
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:8 info/MB/resource_schemata/MB/tolerance:0 info/MB/resource_schemata/MB/max:2048
info/MB/resource_schemata/MB/min:1

Quoted 3 lines

Having such interface will be helpful today. Users do not need to first figure out whether they are on an AMD or Intel system, and then read the docs to learn the AMD units, before interacting with resctrl. resctrl will be the generic interface it intends to be.

Yes. That is a good point.

Quoted 15 lines

Part 2: Supporting multiple controls for a single resource
This is a new feature on which there also appears to be consensus that is needed by MPAM and Intel RDT where it is possible to use different controls for the same resource. For example, there can be a minimum and maximum control associated with the memory bandwidth resource.

For example,
info/

    └─ MB/
        └─ resource_schemata/
            ├─ MB/
            ├─ MB_MIN/
            ├─ MB_MAX/
            ┆

Quoted 7 lines

Quoted 3 lines

Here is where the big question comes in for GLBE - is this actually a new resource for which resctrl needs to add interfaces to manage its allocation, or is it instead an additional control associated with the existing memory bandwith resource?

It is not a new resource. It is new control mechanism to address limitation with memory bandwidth resource.

So, it is a new control for the existing memory bandwidth resource.

Thank you for confirming.

Quoted 40 lines

Quoted 23 lines

For me things are actually pointing to GLBE not being a new resource but instead being a new control for the existing memory bandwidth resource.

I understand that for a PoC it is simplest to add support for GLBE as a new resource as is done in this series but when considering it as an actual unique resource does not seem appropriate since resctrl already has a "memory bandwidth" resource. User space expects to find all the resources that it can allocate in info/ - I do not think it is correct to have two separate directories/resources for memory bandwidth here.

What if, instead, it looks something like:

info/

└── MB/
       └── resource_schemata/
           ├── GMB/
           │   ├──max:4096
           │   ├──min:1
           │   ├──resolution:1
           │   ├──scale:1
           │   ├──tolerance:0
           │   ├──type:scalar linear
           │   └──unit:GBps

Quoted 8 lines

           └── MB/
               ├──max:8192
               ├──min:1
               ├──resolution:8
               ├──scale:1
               ├──tolerance:0
               ├──type:scalar linear
               └──unit:GBps

Yes. It definitely looks very clean.

Quoted 4 lines

With an interface like above GMB is just another control/schema used to allocate the existing memory bandwidth resource. With the planned files it is possible to express the different maximums and units used by the MB and GMB schema. Users no longer need to dig for the unit information in the docs, it is available in the interface.

Quoted 3 lines

Yes. That is reasonable.

Is the plan to just update the resource information in /sys/fs/resctrl/info/<resource_name>  ?

I do not see any resource information that needs to change. As you confirmed, MB and GMB have the same number of CLOSIDs and looking at the rest of the enumeration done in patch #2 all other properties exposed in top level of /sys/fs/resctrl/info/MB is the same for MB and GMB. Specifically, thread_throttle_mode, delay_linear, min_bandwidth, and bandwidth_gran have the same values for MB and GMB. All other content in /sys/fs/resctrl/info/MB would be new as part of the new "resource_schemata" sub-directory.

Even so, I believe we could expect that a user using any new schemata file entry introduced after the "resource_schemata" directory is introduced is aware of how the properties are exposed and will not use the top level files in /sys/fs/resctrl/info/MB (for example min_bandwidth and bandwidth_gran) to understand how to interact with the new schema.

Quoted 2 lines

Also, will the display of /sys/fs/resctrl/schemata change ?

There are no plans to change any of the existing schemata file entries.

Quoted 2 lines

Current display:

When viewing "current" as what this series does in schemata file ...

Quoted 3 lines

  GMB:0=4096;1=4096;2=4096;3=4096
   MB:0=8192;1=8192;2=8192;3=8192

yes, the schemata file should look like this on boot when all is done. All other user facing changes are to the info/ directory where user space learns about the new control for the resource and how to interact with the control.

Quoted 8 lines

Quoted 3 lines

Doing something like this does depend on GLBE supporting the same number of CLOSIDs as MB, which seems to be how this will be implemented. If there is indeed a confirmation of this from AMD architecture then we can do something like this in resctrl.

I don't see this being an issue. I will get consensus on it.

I am wondering about the time frame and who is leading this change. Not sure if that is been discussed already. I can definitely help.

Quoted 6 lines

A couple of features depend on the new schema descriptions as well as support for multiple
controls: min/max bandwidth controls on the MPAM side, region aware MBA and MBM on the Intel side, and GLBE on the AMD side. I am hoping that the folks working on these features can collaborate on the needed foundation. Since there are no patches for this yet I cannot say if there is a leader for this work yet, at this time this role appears to be available if you would like to see this moving forward in order to meet your goals.

Quoted 1 line

I joined this feature effort a bit later, so I may not yet have full context on the MPAM and region‑aware requirements. I’m happy to provide all the necessary information for GMB and MB from the AMD side, and I’m also available to help with reviews and testing.

I understand there is a lot involved. With so many folks dependent on this work I anticipate that any effort will get support from the various content experts. Your knowledge of resctrl fs will be valuable in this effort.

Reinette

Hi Reinette,

Quoted 105 lines · Reinette Chatre

Hi Babu,

Quoted 66 lines · Babu

Quoted 65 lines · Reinette Chatre

Quoted 6 lines · Babu Moger

Quoted 5 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

Quoted 3 lines · Reinette Chatre

Quoted 2 lines · Babu

Quoted 1 line · Reinette Chatre

On 1/21/26 1:12 PM, Babu Moger wrote:

...

Quoted 55 lines

Quoted 44 lines

Quoted 11 lines

Quoted 11 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

      # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=2048;1=2048;2=2048;3=2048

Quoted 7 lines

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:
         # echo"GMB:0=8;2=8" > schemata

      # cat schemata
      GMB:0=8;1=2048;2=8;3=2048
      MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Thank you for confirming.

Quoted 24 lines

Quoted 24 lines

... and then when user space resets GMB the MB can reset like ...

      # echo"GMB:0=2048;2=2048" > schemata

      # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

      # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=8;1=2048;2=8;3=2048

      # echo"GMB:0=8;2=8" > schemata

      # cat schemata
      GMB:0=8;1=2048;2=8;3=2048
      MB:0=8;1=2048;2=8;3=2048

      # echo"GMB:0=2048;2=2048" > schemata

      # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=8;1=2048;2=8;3=2048

Quoted 8 lines

Quoted 2 lines

Quoted 1 line

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Indeed. My goal is to get an idea how user space may interact with the new interfaces and what would be a reasonable expectation from resctrl be during these interactions.

Quoted 1 line

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

hmmm ... this may be subjective. Could you please elaborate how presenting the effective settings may cause confusion?

I mean in many cases, we cannot determine the effective settings correctly. It depends on benchmarks or applications running on the system.

Quoted 8 lines

Quoted 6 lines

Quoted 1 line

Even with MB (without GMB support), even though we set the limit to 10GB, it may not use the whole 10GB.  Memory is shared resource. So, the effective bandwidth usage depends on other applications running on the system.

Sounds like we interpret "effective limits" differently. To me the limits(*) are deterministic. If I understand correctly, if the GMB limit for domains A and B is set to x GB then that places an x GB limit on MB for domains A and B also. Displaying any MB limit in the schemata that is larger than x GB for domain A or domain B would be inaccurate, no?

Yea. But, I was thinking not to mess with values written at registers.

This is not about what is written to the registers but how the combined values written to registers control system behavior and how to accurately reflect the resulting system behavior to user space.

Quoted 23 lines

Quoted 23 lines

When considering your example where the MB limit is 10GB.

Consider an example where there are two domains in this example with a configuration like below. (I am using a different syntax from schemata file that will hopefully make it easier to exchange ideas when not having to interpret the different GMB and MB units):

     MB:0=10GB;1=10GB

If user space can create a GMB domain that limits shared bandwidth to 10GB that can be displayed as below and will be accurate:

     MB:0=10GB;1=10GB
     GMB:0=10GB;1=10GB

If user space then reduces the combined bandwidth to 2GB then the MB limit is wrong since it is actually capped by the GMB limit:

     MB:0=10GB;1=10GB <==== Does reflect possible per-domain memory bandwidth which is now capped by GMB GMB:0=2GB;1=2GB

Would something like below not be more accurate that reflects that the maximum average bandwidth each domain could achieve is 2GB?

Quoted 8 lines

Quoted 4 lines

Quoted 2 lines

     MB:0=2GB;1=2GB <==== Reflects accurate possible per-domain memory bandwidth GMB:0=2GB;1=2GB

That is reasonable. Will check how we can accommodate that.

Right, this is not about the values in the L3BE registers but instead how those values are impacted by GLBE registers and how to most accurately present the resulting system configuration to user space. Thank you for considering.

I responded too quickly earlier—an internal discussion surfaced several concerns with this approach.

schemata represents what user space explicitly configured and what the hardware registers contain, not a derived “effective” value that depends on runtime conditions. Combining configured limits (MB/GMB) with effective bandwidth—which is inherently workload‑dependent—blurs semantics, breaks existing assumptions, and makes debugging more difficult.

MB and GMB use different units and encodings, so auto‑deriving values can introduce rounding issues and loss of precision.

I’ll revisit this and come back with a refined proposal.

Hi Babu,

Quoted 116 lines · Babu

Hi Reinette,

Quoted 100 lines · Reinette Chatre

Hi Babu,

Quoted 62 lines · Babu

Quoted 61 lines · Reinette Chatre

Quoted 6 lines · Babu Moger

Quoted 5 lines · Reinette Chatre

Quoted 4 lines · Babu Moger

Quoted 3 lines · Reinette Chatre

Quoted 2 lines · Babu

Quoted 1 line · Reinette Chatre

On 1/21/26 1:12 PM, Babu Moger wrote:

...

Quoted 52 lines

Quoted 52 lines

Quoted 11 lines

Quoted 11 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

       # cat schemata
       GMB:0=2048;1=2048;2=2048;3=2048
       MB:0=2048;1=2048;2=2048;3=2048

Quoted 7 lines

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:
         # echo"GMB:0=8;2=8" > schemata

       # cat schemata
       GMB:0=8;1=2048;2=8;3=2048
       MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Thank you for confirming.

Quoted 19 lines

Quoted 19 lines

... and then when user space resets GMB the MB can reset like ...

      # echo"GMB:0=2048;2=2048" > schemata

       # cat schemata
       GMB:0=2048;1=2048;2=2048;3=2048
       MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

       # cat schemata
       GMB:0=2048;1=2048;2=2048;3=2048
       MB:0=8;1=2048;2=8;3=2048

      # echo"GMB:0=8;2=8" > schemata

       # cat schemata
       GMB:0=8;1=2048;2=8;3=2048
       MB:0=8;1=2048;2=8;3=2048

Quoted 7 lines

Quoted 6 lines

      # echo"GMB:0=2048;2=2048" > schemata

       # cat schemata
       GMB:0=2048;1=2048;2=2048;3=2048
       MB:0=8;1=2048;2=8;3=2048

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Indeed. My goal is to get an idea how user space may interact with the new interfaces and what would be a reasonable expectation from resctrl be during these interactions.

Quoted 1 line

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

hmmm ... this may be subjective. Could you please elaborate how presenting the effective settings may cause confusion?

Quoted 8 lines

Quoted 7 lines

Quoted 3 lines

I mean in many cases, we cannot determine the effective settings correctly. It depends on benchmarks or applications running on the system.

Even with MB (without GMB support), even though we set the limit to 10GB, it may not use the whole 10GB.  Memory is shared resource. So, the effective bandwidth usage depends on other applications running on the system.

Sounds like we interpret "effective limits" differently. To me the limits(*) are deterministic. If I understand correctly, if the GMB limit for domains A and B is set to x GB then that places an x GB limit on MB for domains A and B also. Displaying any MB limit in the schemata that is larger than x GB for domain A or domain B would be inaccurate, no?

Yea. But, I was thinking not to mess with values written at registers.

This is not about what is written to the registers but how the combined values written to registers control system behavior and how to accurately reflect the resulting system behavior to user space.

Quoted 23 lines

Quoted 23 lines

When considering your example where the MB limit is 10GB.

Consider an example where there are two domains in this example with a configuration like below. (I am using a different syntax from schemata file that will hopefully make it easier to exchange ideas when not having to interpret the different GMB and MB units):

     MB:0=10GB;1=10GB

If user space can create a GMB domain that limits shared bandwidth to 10GB that can be displayed as below and will be accurate:

     MB:0=10GB;1=10GB
     GMB:0=10GB;1=10GB

If user space then reduces the combined bandwidth to 2GB then the MB limit is wrong since it is actually capped by the GMB limit:

     MB:0=10GB;1=10GB <==== Does reflect possible per-domain memory bandwidth which is now capped by GMB GMB:0=2GB;1=2GB

Would something like below not be more accurate that reflects that the maximum average bandwidth each domain could achieve is 2GB?

Quoted 6 lines

Quoted 3 lines

Quoted 2 lines

     MB:0=2GB;1=2GB <==== Reflects accurate possible per-domain memory bandwidth GMB:0=2GB;1=2GB

That is reasonable. Will check how we can accommodate that.

Right, this is not about the values in the L3BE registers but instead how those values are impacted by GLBE registers and how to most accurately present the resulting system configuration to user space. Thank you for considering.

I responded too quickly earlier—an internal discussion surfaced several concerns with this approach.

schemata represents what user space explicitly configured and what the hardware registers contain, not a derived “effective” value that depends on runtime conditions. Combining configured limits (MB/GMB) with effective bandwidth—which is inherently workload‑dependent—blurs semantics, breaks existing assumptions, and makes debugging more difficult.

Quoted 3 lines

MB and GMB use different units and encodings, so auto‑deriving values can introduce rounding issues and loss of precision.

I’ll revisit this and come back with a refined proposal.

Are we still talking about below copied from https://lore.kernel.org/lkml/[email protected]/ ?

The MBA ceiling is applied at the QoS domain level.
The GLBE ceiling is applied at the GLBE control domain level.
If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling.

Hi Babu,

Quoted 17 lines · Babu Moger

Quoted 8 lines · Reinette Chatre

Quoted 7 lines · Babu

Quoted 6 lines · Reinette Chatre

Quoted 5 lines · Babu Moger

On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QOS domain. However, multiple QOS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, user must statically partition the available system bandwidth between the active QOS domains. This typically

Quoted 8 lines

Quoted 2 lines

Quoted 1 line

How do you define "active" QoS Domain?

Some domains may not have any CPUs associated with that CLOSID. Active meant, I'm referring to domains that have CPUs assigned to the CLOSID.

To confirm, is this then specific to assigning CPUs to resource groups via the cpus/cpus_list files? This refers to how a user needs to partition available bandwidth so I am still trying to understand the message here since users still need to do this even when CPUs are not assigned to resource groups.

Quoted 8 lines

It is not specific to CPU assignment. It applies to task assignment also.
For example:  We have 4 domains;

# cat schemata
  MB:0=8192;1=8192;2=8192;3=8192

If this group has the CPUs assigned to only first two domains. Then the group has only two active domains. Then we will only update the first two domains. The MB values in other domains does not matter.

I see, thank you. As I understand an "active QoS domain" is something only user space can designate. It may be possible for resctrl to get a sense of which QoS domains are "active" when only CPUs are assigned to a resource group but when it comes to task assignment it is user space that controls where tasks belonging to a group can be scheduled and thus which QoS domains are "active" or not.

Quoted 9 lines

#echo "MB:0=8;1=8" > schemata

# cat schemata
  MB:0=8;1=8;2=8192;3=8192

The combined bandwidth can go up to 16(8+8) units. Each unit is 1/8 GB.

With GMBA, we can set the combined limit higher level and total bandwidth will not exceed GMBA limit.

Thank you for the confirmation.

Quoted 33 lines

Quoted 33 lines

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

results in system memory being under-utilized since not all QOS domains are using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS Domains. This collection of QOS domains is referred to as GLBE control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service(COS) across every QoS domain managed by the GLBE control domain.

Quoted 8 lines

Quoted 4 lines

How does this bandwidth allocation limit impact existing MBA? For example, if a system has two domains (A and B) that user space separately sets MBA allocations for while also placing both domains within a "GLBE control domain" with a different allocation, does the individual MBA allocations still matter?

Yes. Both ceilings are enforced at their respective levels.
The MBA ceiling is applied at the QoS domain level.
The GLBE ceiling is applied at the GLBE control  domain level.
If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling.

It sounds as though MBA and GMBA/GLBE operates within the same parameters wrt the limits but in examples in this series they have different limits. For example, in the documentation patch [1] there is this:

  # cat schemata
     GMB:0=2048;1=2048;2=2048;3=2048
     MB:0=4096;1=4096;2=4096;3=4096
     L3:0=ffff;1=ffff;2=ffff;3=ffff

followed up with what it will look like in new generation [2]:

    GMB:0=4096;1=4096;2=4096;3=4096
     MB:0=8192;1=8192;2=8192;3=8192
      L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 6 lines

Quoted 4 lines

In both examples the per-domain MB ceiling is higher than the global GMB ceiling. With above showing defaults and you state "If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling." - does this mean that MB ceiling can never be higher than GMB ceiling as shown in the examples?

That is correct.  There is one more information here.   The MB unit is in 1/8 GB and GMB unit is 1GB.  I have added that in documentation in patch 4.

ah - right. I did not take the different units into account.

Quoted 17 lines

The GMB limit defaults to max value 4096 (bit 12 set) when the new group is created.  Meaning GMB limit does not apply by default.

When setting the limits, it should be set to same value in all the domains in GMB control domain.  Having different value in each domain results in unexpected behavior.

Quoted 12 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=2048;1=2048;2=2048;3=2048

Quoted 8 lines

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:

      # echo "GMB:0=8;2=8" > schemata
   # cat schemata
   GMB:0=8;1=2048;2=8;3=2048
   MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Thank you for confirming.

Quoted 21 lines

Quoted 21 lines

... and then when user space resets GMB the MB can reset like ...

   # echo "GMB:0=2048;2=2048" > schemata
   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=8;1=2048;2=8;3=2048

   # echo "GMB:0=8;2=8" > schemata
   # cat schemata
   GMB:0=8;1=2048;2=8;3=2048
   MB:0=8;1=2048;2=8;3=2048

Quoted 8 lines

Quoted 6 lines

   # echo "GMB:0=2048;2=2048" > schemata
   # cat schemata
   GMB:0=2048;1=2048;2=2048;3=2048
   MB:0=8;1=2048;2=8;3=2048

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Indeed. My goal is to get an idea how user space may interact with the new interfaces and what would be a reasonable expectation from resctrl be during these interactions.

Quoted 2 lines

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

hmmm ... this may be subjective. Could you please elaborate how presenting the effective settings may cause confusion?

Quoted 2 lines

We also need to track the previous settings so we can revert to the earlier value when needed. The best approach is to document this behavior clearly.

Yes, this will require resctrl to maintain more state.

Documenting behavior is an option but I think we should first consider if there are things resctrl can do to make the interface intuitive to use.

Quoted 14 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

 From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

Quoted 7 lines

Quoted 4 lines

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

No. When enumerating the features the number of CLOSID supported by each is enumerated separately. That means GMBA and MBA may support different number of CLOSID. My question is: "under which scenario will GMBA and MBA support different CLOSID?"

Quoted 7 lines

No. There is not such scenario.

Quoted 4 lines

Because of a possible difference in number of CLOSIDs it seems the feature supports possible scenarios where some resource groups can support global AND per-domain limits while other resource groups can just support global or just support per-domain limits. Is this correct?

System can support up to 16 CLOSIDs. All of them support all the features LLC, MB, GMB, SMBA.   Yes. We have separate enumeration for  each feature.  Are you suggesting to change it ?

It is not a concern to have different CLOSIDs between resources that are actually different, for example, having LLC or MB support different number of CLOSIDs. Having the possibility to allocate the *same* resource (memory bandwidth) with varying number of CLOSIDs does present a challenge though. Would it be possible to have a snippet in the spec that explicitly states that MB and GMB will always enumerate with the same number of CLOSIDs?

Please see below where I will try to support this request more clearly and you can decide if it is reasonable.

Quoted 9 lines

Quoted 7 lines

Quoted 5 lines

Quoted 4 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so that the features you are enabling can be supported well.

Yes, I am trying. I noticed that the proposal appears to affect how the schemata information is displayed(in info directory). It seems to introduce additional resource information. I don't see any harm in displaying it if it benefits certain architecture.

It benefits all architectures.

There are two parts to the current proposals.

Part 1: Generic schema description
I believe there is consensus on this approach. This is actually something that is long overdue and something like this would have been a great to have with the initial AMD enabling. With the generic schema description forming part of resctrl the user can learn from resctrl how to interact with the schemata file instead of relying on external information and documentation.

For example, on an Intel system that uses percentage based proportional allocation for memory bandwidth the new resctrl files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:all
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:100 info/MB/resource_schemata/MB/tolerance:0
info/MB/resource_schemata/MB/max:100
info/MB/resource_schemata/MB/min:10

On an AMD system that uses absolute allocation with 1/8 GBps steps the files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:GBps
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:8 info/MB/resource_schemata/MB/tolerance:0 info/MB/resource_schemata/MB/max:2048
info/MB/resource_schemata/MB/min:1

Having such interface will be helpful today. Users do not need to first figure out whether they are on an AMD or Intel system, and then read the docs to learn the AMD units, before interacting with resctrl. resctrl will be the generic interface it intends to be.

Part 2: Supporting multiple controls for a single resource
This is a new feature on which there also appears to be consensus that is needed by MPAM and Intel RDT where it is possible to use different controls for the same resource. For example, there can be a minimum and maximum control associated with the memory bandwidth resource.

For example,
info/

 └─ MB/
     └─ resource_schemata/
         ├─ MB/
         ├─ MB_MIN/
         ├─ MB_MAX/
         ┆

Here is where the big question comes in for GLBE - is this actually a new resource for which resctrl needs to add interfaces to manage its allocation, or is it instead an additional control associated with the existing memory bandwith resource?

For me things are actually pointing to GLBE not being a new resource but instead being a new control for the existing memory bandwidth resource.

I understand that for a PoC it is simplest to add support for GLBE as a new resource as is done in this series but when considering it as an actual unique resource does not seem appropriate since resctrl already has a "memory bandwidth" resource. User space expects to find all the resources that it can allocate in info/ - I do not think it is correct to have two separate directories/resources for memory bandwidth here.

What if, instead, it looks something like:

info/

└── MB/
    └── resource_schemata/
        ├── GMB/
        │   ├── max:4096
        │   ├── min:1
        │   ├── resolution:1
        │   ├── scale:1
        │   ├── tolerance:0
        │   ├── type:scalar linear
        │   └── unit:GBps
        └── MB/
            ├── max:8192
            ├── min:1
            ├── resolution:8
            ├── scale:1
            ├── tolerance:0
            ├── type:scalar linear
            └── unit:GBps

With an interface like above GMB is just another control/schema used to allocate the existing memory bandwidth resource. With the planned files it is possible to express the different maximums and units used by the MB and GMB schema. Users no longer need to dig for the unit information in the docs, it is available in the interface.

Doing something like this does depend on GLBE supporting the same number of CLOSIDs as MB, which seems to be how this will be implemented. If there is indeed a confirmation of this from AMD architecture then we can do something like this in resctrl.

There is a "part 3" to the proposals that attempts to address the new requirement where some of the controls allocate at a different scope while also requiring monitoring at that new scope. After learning more about GLBE this does not seem relevant to GLBE but is something to return to for the "MPAM CPU-less" work. We could already prepare for this by adding the new "scope" schema property though.

Reinette

Quoted 12 lines

Thanks

Babu

Quoted 6 lines

Reinette

[1] https://lore.kernel.org/lkml/d58f70592a4ce89e744e7378e49d5a36be3fd05e.1769029977.git.babu.moger@amd.com/
[2] https://lore.kernel.org/lkml/[email protected]/

I haven't fully understood what GLBE is but in MPAM we have an optional feature in MSC (MPAM devices) called partid narrowing. For some MSC there are limited controls and the incoming partid is mapped to an effective partid using a mapping. This mapping is software controllable. Dave (with Shaopeng and Zeng) has a proposal to use this to use partid bits as pmg bits, [1]. This usage would have to be opt-in as it changes the number of closid/rmid that MPAM presents to resctrl. If however, the user doesn't use that scheme then the controls could be presented as controls for groups of closid in resctrl. Is this similar/usable with the same interface as GLBE or have I misunderstood?

[1]
https://lore.kernel.org/linux-arm-kernel/[email protected]/

Hi Ben,

Quoted 12 lines · Ben Horgan

I haven't fully understood what GLBE is but in MPAM we have an optional feature in MSC (MPAM devices) called partid narrowing. For some MSC there are limited controls and the incoming partid is mapped to an effective partid using a mapping. This mapping is software controllable. Dave (with Shaopeng and Zeng) has a proposal to use this to use partid bits as pmg bits, [1]. This usage would have to be opt-in as it changes the number of closid/rmid that MPAM presents to resctrl. If however, the user doesn't use that scheme then the controls could be presented as controls for groups of closid in resctrl. Is this similar/usable with the same interface as GLBE or have I misunderstood?

Quoted 2 lines

[1]
https://lore.kernel.org/linux-arm-kernel/[email protected]/

On a high level these look like different capabilities to me but I look forward to hear from others to understand where I may be wrong.

As I understand the feature you refer to is a way in which MPAM can increase the number of hardware monitoring IDs available(*). It does so by using the PARTID narrowing feature while taking advantage of the fact that PARTID for filtering resource monitors is always a "request PARTID". In itself I understand the PARTID narrowing feature to manage how resource allocation of a *single* "MPAM component" is managed.

On the other hand I see GLBE as a feature that essentially allows the scope of allocation to span multiple domains/components.

As I see it, applying GLBE to MPAM would need the capability to, for example, set a memory bandwidth MAX that is shared across multiple MPAM components.

Reinette

* as a sidenote it is not clear to me why this would require an opt-in since there only seems benefits to this.

Hi Reinette, Babu,

Quoted 290 lines · Reinette Chatre

Hi Babu,

Quoted 17 lines · Babu Moger

Quoted 8 lines · Reinette Chatre

Quoted 7 lines · Babu

Quoted 6 lines · Reinette Chatre

Quoted 5 lines · Babu Moger

On AMD systems, the existing MBA feature allows the user to set a bandwidth limit for each QOS domain. However, multiple QOS domains share system memory bandwidth as a resource. In order to ensure that system memory bandwidth is not over-utilized, user must statically partition the available system bandwidth between the active QOS domains. This typically

Quoted 8 lines

Quoted 2 lines

Quoted 1 line

How do you define "active" QoS Domain?

Some domains may not have any CPUs associated with that CLOSID. Active meant, I'm referring to domains that have CPUs assigned to the CLOSID.

To confirm, is this then specific to assigning CPUs to resource groups via the cpus/cpus_list files? This refers to how a user needs to partition available bandwidth so I am still trying to understand the message here since users still need to do this even when CPUs are not assigned to resource groups.

Quoted 8 lines

It is not specific to CPU assignment. It applies to task assignment also.

For example:  We have 4 domains;

# cat schemata
   MB:0=8192;1=8192;2=8192;3=8192

If this group has the CPUs assigned to only first two domains. Then the group has only two active domains. Then we will only update the first two domains. The MB values in other domains does not matter.

I see, thank you. As I understand an "active QoS domain" is something only user space can designate. It may be possible for resctrl to get a sense of which QoS domains are "active" when only CPUs are assigned to a resource group but when it comes to task assignment it is user space that controls where tasks belonging to a group can be scheduled and thus which QoS domains are "active" or not.

Quoted 9 lines

#echo "MB:0=8;1=8" > schemata

# cat schemata
   MB:0=8;1=8;2=8192;3=8192

The combined bandwidth can go up to 16(8+8) units. Each unit is 1/8 GB.

With GMBA, we can set the combined limit higher level and total bandwidth will not exceed GMBA limit.

Thank you for the confirmation.

Quoted 33 lines

Quoted 33 lines

Quoted 10 lines

Quoted 10 lines

Quoted 10 lines

results in system memory being under-utilized since not all QOS domains are using their full bandwidth Allocation.

AMD PQoS Global Bandwidth Enforcement(GLBE) provides a mechanism for software to specify bandwidth limits for groups of threads that span multiple QoS Domains. This collection of QOS domains is referred to as GLBE control domain. The GLBE ceiling sets a maximum limit on a memory bandwidth in GLBE control domain. Bandwidth is shared by all threads in a Class of Service(COS) across every QoS domain managed by the GLBE control domain.

Quoted 8 lines

Quoted 4 lines

How does this bandwidth allocation limit impact existing MBA? For example, if a system has two domains (A and B) that user space separately sets MBA allocations for while also placing both domains within a "GLBE control domain" with a different allocation, does the individual MBA allocations still matter?

Yes. Both ceilings are enforced at their respective levels.
The MBA ceiling is applied at the QoS domain level.
The GLBE ceiling is applied at the GLBE control  domain level.
If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling.

It sounds as though MBA and GMBA/GLBE operates within the same parameters wrt the limits but in examples in this series they have different limits. For example, in the documentation patch [1] there is this:

   # cat schemata
      GMB:0=2048;1=2048;2=2048;3=2048
      MB:0=4096;1=4096;2=4096;3=4096
      L3:0=ffff;1=ffff;2=ffff;3=ffff

followed up with what it will look like in new generation [2]:

    GMB:0=4096;1=4096;2=4096;3=4096
     MB:0=8192;1=8192;2=8192;3=8192
      L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 6 lines

Quoted 4 lines

In both examples the per-domain MB ceiling is higher than the global GMB ceiling. With above showing defaults and you state "If the MBA ceiling exceeds the GLBE ceiling, the effective MBA limit will be capped by the GLBE ceiling." - does this mean that MB ceiling can never be higher than GMB ceiling as shown in the examples?

That is correct.  There is one more information here.   The MB unit is in 1/8 GB and GMB unit is 1GB.  I have added that in documentation in patch 4.

ah - right. I did not take the different units into account.

Quoted 17 lines

The GMB limit defaults to max value 4096 (bit 12 set) when the new group is created.  Meaning GMB limit does not apply by default.

When setting the limits, it should be set to same value in all the domains in GMB control domain.  Having different value in each domain results in unexpected behavior.

Quoted 12 lines

Another question, when setting aside possible differences between MB and GMB.

I am trying to understand how user may expect to interact with these interfaces ...

Consider the starting state example as below where the MB and GMB ceilings are the same:

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=2048;1=2048;2=2048;3=2048

Quoted 8 lines

Quoted 6 lines

Would something like below be accurate? Specifically, showing how the GMB limit impacts the MB limit:

       # echo "GMB:0=8;2=8" > schemata
    # cat schemata
    GMB:0=8;1=2048;2=8;3=2048
    MB:0=8;1=2048;2=8;3=2048

Yes. That is correct.  It will cap the MB setting to  8.   Note that we are talking about unit differences to make it simple.

Thank you for confirming.

Quoted 21 lines

Quoted 21 lines

... and then when user space resets GMB the MB can reset like ...

    # echo "GMB:0=2048;2=2048" > schemata
    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=2048;1=2048;2=2048;3=2048

if I understand correctly this will only apply if the MB limit was never set so another scenario may be to keep a previous MB setting after a GMB change:

    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=8;1=2048;2=8;3=2048

    # echo "GMB:0=8;2=8" > schemata
    # cat schemata
    GMB:0=8;1=2048;2=8;3=2048
    MB:0=8;1=2048;2=8;3=2048

Quoted 8 lines

Quoted 6 lines

    # echo "GMB:0=2048;2=2048" > schemata
    # cat schemata
    GMB:0=2048;1=2048;2=2048;3=2048
    MB:0=8;1=2048;2=8;3=2048

What would be most intuitive way for user to interact with the interfaces?

I see that you are trying to display the effective behaviors above.

Indeed. My goal is to get an idea how user space may interact with the new interfaces and what would be a reasonable expectation from resctrl be during these interactions.

Quoted 2 lines

Please keep in mind that MB and GMB units differ. I recommend showing only the values the user has explicitly configured, rather than the effective settings, as displaying both may cause confusion.

hmmm ... this may be subjective. Could you please elaborate how presenting the effective settings may cause confusion?

Quoted 2 lines

We also need to track the previous settings so we can revert to the earlier value when needed. The best approach is to document this behavior clearly.

Yes, this will require resctrl to maintain more state.

Documenting behavior is an option but I think we should first consider if there are things resctrl can do to make the interface intuitive to use.

Quoted 14 lines

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

Quoted 1 line

 From the description it sounds as though there is a new "memory bandwidth

ceiling/limit" that seems to imply that MBA allocations are limited by GMBA allocations while the proposed user interface present them as independent.

If there is indeed some dependency here ... while MBA and GMBA CLOSID are enumerated separately, under which scenario will GMBA and MBA support different CLOSID? As I mentioned in [1] from user space perspective "memory bandwidth"

Quoted 7 lines

Quoted 4 lines

I can see the following scenarios where MBA and GMBA can operate independently:
1. If the GMBA limit is set to ‘unlimited’, then MBA functions as an independent CLOS.
2. If the MBA limit is set to ‘unlimited’, then GMBA functions as an independent CLOS. I hope this clarifies your question.

No. When enumerating the features the number of CLOSID supported by each is enumerated separately. That means GMBA and MBA may support different number of CLOSID. My question is: "under which scenario will GMBA and MBA support different CLOSID?"

Quoted 7 lines

No. There is not such scenario.

Quoted 4 lines

Because of a possible difference in number of CLOSIDs it seems the feature supports possible scenarios where some resource groups can support global AND per-domain limits while other resource groups can just support global or just support per-domain limits. Is this correct?

System can support up to 16 CLOSIDs. All of them support all the features LLC, MB, GMB, SMBA.   Yes. We have separate enumeration for  each feature.  Are you suggesting to change it ?

It is not a concern to have different CLOSIDs between resources that are actually different, for example, having LLC or MB support different number of CLOSIDs. Having the possibility to allocate the *same* resource (memory bandwidth) with varying number of CLOSIDs does present a challenge though. Would it be possible to have a snippet in the spec that explicitly states that MB and GMB will always enumerate with the same number of CLOSIDs?

Please see below where I will try to support this request more clearly and you can decide if it is reasonable.

Quoted 9 lines

Quoted 7 lines

Quoted 5 lines

Quoted 4 lines

can be seen as a single "resource" that can be allocated differently based on the various schemata associated with that resource. This currently has a dependency on the various schemata supporting the same number of CLOSID which may be something that we can reconsider?

After reviewing the new proposal again, I’m still unsure how all the pieces will fit together. MBA and GMBA share the same scope and have inter-dependencies. Without the full implementation details, it’s difficult for me to provide meaningful feedback on new approach.

The new approach is not final so please provide feedback to help improve it so that the features you are enabling can be supported well.

Yes, I am trying. I noticed that the proposal appears to affect how the schemata information is displayed(in info directory). It seems to introduce additional resource information. I don't see any harm in displaying it if it benefits certain architecture.

It benefits all architectures.

There are two parts to the current proposals.

Part 1: Generic schema description
I believe there is consensus on this approach. This is actually something that is long overdue and something like this would have been a great to have with the initial AMD enabling. With the generic schema description forming part of resctrl the user can learn from resctrl how to interact with the schemata file instead of relying on external information and documentation.

For example, on an Intel system that uses percentage based proportional allocation for memory bandwidth the new resctrl files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:all
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:100 info/MB/resource_schemata/MB/tolerance:0
info/MB/resource_schemata/MB/max:100
info/MB/resource_schemata/MB/min:10

On an AMD system that uses absolute allocation with 1/8 GBps steps the files will display:
info/MB/resource_schemata/MB/type:scalar linear info/MB/resource_schemata/MB/unit:GBps
info/MB/resource_schemata/MB/scale:1 info/MB/resource_schemata/MB/resolution:8 info/MB/resource_schemata/MB/tolerance:0 info/MB/resource_schemata/MB/max:2048
info/MB/resource_schemata/MB/min:1

Having such interface will be helpful today. Users do not need to first figure out whether they are on an AMD or Intel system, and then read the docs to learn the AMD units, before interacting with resctrl. resctrl will be the generic interface it intends to be.

Part 2: Supporting multiple controls for a single resource
This is a new feature on which there also appears to be consensus that is needed by MPAM and Intel RDT where it is possible to use different controls for the same resource. For example, there can be a minimum and maximum control associated with the memory bandwidth resource.

For example,
info/

  └─ MB/
      └─ resource_schemata/
          ├─ MB/
          ├─ MB_MIN/
          ├─ MB_MAX/
          ┆

Here is where the big question comes in for GLBE - is this actually a new resource for which resctrl needs to add interfaces to manage its allocation, or is it instead an additional control associated with the existing memory bandwith resource?

For me things are actually pointing to GLBE not being a new resource but instead being a new control for the existing memory bandwidth resource.

I understand that for a PoC it is simplest to add support for GLBE as a new resource as is done in this series but when considering it as an actual unique resource does not seem appropriate since resctrl already has a "memory bandwidth" resource. User space expects to find all the resources that it can allocate in info/ - I do not think it is correct to have two separate directories/resources for memory bandwidth here.

What if, instead, it looks something like:

info/

└── MB/
     └── resource_schemata/
         ├── GMB/
         │   ├── max:4096
         │   ├── min:1
         │   ├── resolution:1
         │   ├── scale:1
         │   ├── tolerance:0
         │   ├── type:scalar linear
         │   └── unit:GBps
         └── MB/
             ├── max:8192
             ├── min:1
             ├── resolution:8
             ├── scale:1
             ├── tolerance:0
             ├── type:scalar linear
             └── unit:GBps

Quoted 8 lines

With an interface like above GMB is just another control/schema used to allocate the existing memory bandwidth resource. With the planned files it is possible to express the different maximums and units used by the MB and GMB schema. Users no longer need to dig for the unit information in the docs, it is available in the interface.

Doing something like this does depend on GLBE supporting the same number of CLOSIDs as MB, which seems to be how this will be implemented. If there is indeed a confirmation of this from AMD architecture then we can do something like this in resctrl.

I haven't fully understood what GLBE is but in MPAM we have an optional feature in MSC (MPAM devices) called partid narrowing. For some MSC there are limited controls and the incoming partid is mapped to an effective partid using a mapping. This mapping is software controllable. Dave (with Shaopeng and Zeng) has a proposal to use this to use partid bits as pmg bits, [1]. This usage would have to be opt-in as it changes the number of closid/rmid that MPAM presents to resctrl. If however, the user doesn't use that scheme then the controls could be presented as controls for groups of closid in resctrl. Is this similar/usable with the same interface as GLBE or have I misunderstood?

[1]
https://lore.kernel.org/linux-arm-kernel/[email protected]/

Quoted 24 lines

There is a "part 3" to the proposals that attempts to address the new requirement where some of the controls allocate at a different scope while also requiring monitoring at that new scope. After learning more about GLBE this does not seem relevant to GLBE but is something to return to for the "MPAM CPU-less" work. We could already prepare for this by adding the new "scope" schema property though.

Reinette

Quoted 12 lines

Thanks

Babu

Quoted 6 lines

Reinette

[1] https://lore.kernel.org/lkml/d58f70592a4ce89e744e7378e49d5a36be3fd05e.1769029977.git.babu.moger@amd.com/
[2] https://lore.kernel.org/lkml/[email protected]/

Thanks,

Ben

What if, instead, it looks something like:

info/

└── MB/
   └── resource_schemata/
       ├── GMB/
       │   ├── max:4096
       │   ├── min:1
       │   ├── resolution:1
       │   ├── scale:1
       │   ├── tolerance:0
       │   ├── type:scalar linear
       │   └── unit:GBps

       └── MB/
           ├── max:8192
           ├── min:1
           ├── resolution:8
           ├── scale:1
           ├── tolerance:0
           ├── type:scalar linear
           └── unit:GBps

Hi, Reinette,

Quoted 13 lines

What if, instead, it looks something like:

info/

└── MB/
    └── resource_schemata/
        ├── GMB/
        │   ├── max:4096
        │   ├── min:1
        │   ├── resolution:1
        │   ├── scale:1
        │   ├── tolerance:0
        │   ├── type:scalar linear
        │   └── unit:GBps

Quoted 8 lines

        └── MB/
            ├── max:8192
            ├── min:1
            ├── resolution:8
            ├── scale:1
            ├── tolerance:0
            ├── type:scalar linear
            └── unit:GBps

May I have 2 comments?

1. This directory is for both info and control, right?

"info" is a read-only directory:
dr-xr-xr-x 8 root root 0 Feb 23 12:50 info

And its name suggests it's for info only as well.

Instead of mixing legal info and control together, is it better to add a new "control" or "config" directory in /sys/fs/resctrl for this control and info purpose?

2. This control method seems only handles global control for resources. But what if a control is per domain and per closid/partid?

For example, MPAM has a hardlimit control per mem bandwidth allocation domain per partid. When hardlimit is enabled, MPAM hardware enforces hard limit of MBW max. This can not be controlled globally.

For this kind of per partid per domain control, propose config_schemata/control_schemata file:

partition X/
    control_schemata (or config_schemata):
        MB_hardlimit: 0=0/1;1=0/1;...

Is this reasonable?

What if, instead, it looks something like:

info/

└── MB/
   └── resource_schemata/
       ├── GMB/
       │   ├── max:4096
       │   ├── min:1
       │   ├── resolution:1
       │   ├── scale:1
       │   ├── tolerance:0
       │   ├── type:scalar linear
       │   └── unit:GBps

       └── MB/
           ├── max:8192
           ├── min:1
           ├── resolution:8
           ├── scale:1
           ├── tolerance:0
           ├── type:scalar linear
           └── unit:GBps

While min_bandwidth is exposed for each resource under /sys/fs/resctrl, the maximum supported bandwidth is not currently shown.

Add max_bandwidth to report the maximum bandwidth permitted for a resource. This helps users understand the limits of the associated resource control group.

Signed-off-by Babu Moger <[email protected]>

---

Hi Babu,

Quoted 9 lines · Babu Moger

While min_bandwidth is exposed for each resource under /sys/fs/resctrl, the maximum supported bandwidth is not currently shown.

Add max_bandwidth to report the maximum bandwidth permitted for a resource. This helps users understand the limits of the associated resource control group.

Signed-off-by Babu Moger <[email protected]>

---

With resctrl fs being used by several architectures we should take care that interface changes take all planned usages into account.

As shared at LPC [1] and email [2] we are already trying to create an interface that works for everybody and it already contains a way to expose the maximum bandwidth to user space. You attended that LPC session and [2] directed to you received no response. This submission with a different interface is unexpected.

Reinette

[1] https://lpc.events/event/19/contributions/2093/attachments/1958/4172/resctrl%20Microconference%20LPC%202025%20Tokyo.pdf
[2] https://lore.kernel.org/lkml/[email protected]/

Documentation/filesystems/resctrl.rst Memory bandwidth domain is L3 cache. · line 972

Quoted 9 lines · Babu Moger

Documentation/filesystems/resctrl.rst +9 -0

+Global Memory bandwidth Allocation
+-----------------------------------
+
+AMD hardware supports Global Memory Bandwidth Allocation (GMBA) provides
+a mechanism for software to specify bandwidth limits for groups of threads
+that span across multiple QoS domains. This collection of QOS domains is
+referred to as GMBA control domain. The GMBA control domain is created by
+setting the same GMBA limits in one or more QoS domains. Setting the default
+max_bandwidth excludes the QoS domain from being part of GMBA control domain.

I don't see any checks that the user sets the *SAME* GMBA limits.

What happens if the user ignores the dosumentation and sets different limits?

... snip ...

Documentation/filesystems/resctrl.rst +10 -0

+  # cat schemata
+    GMB:0=2048;1=2048;2=2048;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff
+
+  # echo "GMB:0=8;2=8" > schemata
+  # cat schemata
+    GMB:0=   8;1=2048;2=   8;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff

Can the user go on to set:

    # echo "GMB:1=10;3=10" > schemata

and have domains 0 & 2 with a combined 8GB limit, while domains 1 & 3 run with a combined 10GB limit? Or is there a single "GMBA domain"?

Will using "2048" as the "this domain isn't limited by GMBA" value come back to haunt you when some system has much more than 2TB bandwidth to divide up?

Should resctrl have a non-numeric "unlimited" value in the schemata file for this?

The "mba_MBps" feature used U32_MAX as the unlimited value. But it looks somewhat ugly in the schemata file:

Hi Tony,

Quoted 12 lines · Tony

Documentation/filesystems/resctrl.rst Memory bandwidth domain is L3 cache. · line 972

Quoted 9 lines · Babu Moger

Documentation/filesystems/resctrl.rst +9 -0

+Global Memory bandwidth Allocation
+-----------------------------------
+
+AMD hardware supports Global Memory Bandwidth Allocation (GMBA) provides
+a mechanism for software to specify bandwidth limits for groups of threads
+that span across multiple QoS domains. This collection of QOS domains is
+referred to as GMBA control domain. The GMBA control domain is created by
+setting the same GMBA limits in one or more QoS domains. Setting the default
+max_bandwidth excludes the QoS domain from being part of GMBA control domain.

I don't see any checks that the user sets the *SAME* GMBA limits.

Quoted 2 lines

What happens if the user ignores the dosumentation and sets different limits?

Good point. Adding checks could be challenging when users update each schema individually with different values. We don't know which one value is the one he is intending to keep.

Quoted 13 lines

... snip ...

+  # cat schemata
+    GMB:0=2048;1=2048;2=2048;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff
+
+  # echo "GMB:0=8;2=8" > schemata
+  # cat schemata
+    GMB:0=   8;1=2048;2=   8;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 7 lines

Can the user go on to set:

    # echo "GMB:1=10;3=10" > schemata

and have domains 0 & 2 with a combined 8GB limit, while domains 1 & 3 run with a combined 10GB limit? Or is there a single "GMBA domain"?

In that case, it  is still treated as a single GMBA domain, but the behavior becomes unpredictable. The hardware expert mentioned that it will default to the lowest value among all inputs in this case, 8GB.

Quoted 3 lines

Will using "2048" as the "this domain isn't limited by GMBA" value come back to haunt you when some system has much more than 2TB bandwidth to divide up?

It is actually 4096 (4TB). I made a mistake in the example.  I am assuming it may not an issue in the current generation.

It is expected to go up in next generation.

GMB:0=4096;1=4096;2=4096;3=4096;
   MB:0=8192;1=8192;2=8192;3=8192;
    L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 3 lines

Should resctrl have a non-numeric "unlimited" value in the schemata file for this?

The value 4096 corresponds to 12th bit set.  It is called U-bit. If the U bit is set then that domain is not part of the GMBA domain.

I was thinking of displaying the "U" in those cases.  It may be good idea to do something like this.

GMB:0=      8;1=      U;2=     8 ;3=      U;
   MB:0=8192;1=8192;2=8192;3=8192;
    L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 1 line

Quoted 3 lines

The "mba_MBps" feature used U32_MAX as the unlimited value. But it looks somewhat ugly in the schemata file:

Yes, I agree. Non-numeric would have been better.

Hi Babu and Tony,

Quoted 65 lines · Babu Moger

Hi Tony,

Quoted 12 lines · Tony

Documentation/filesystems/resctrl.rst Memory bandwidth domain is L3 cache. · line 972

Quoted 9 lines · Babu Moger

Documentation/filesystems/resctrl.rst +9 -0

+Global Memory bandwidth Allocation
+-----------------------------------
+
+AMD hardware supports Global Memory Bandwidth Allocation (GMBA) provides
+a mechanism for software to specify bandwidth limits for groups of threads
+that span across multiple QoS domains. This collection of QOS domains is
+referred to as GMBA control domain. The GMBA control domain is created by
+setting the same GMBA limits in one or more QoS domains. Setting the default
+max_bandwidth excludes the QoS domain from being part of GMBA control domain.

I don't see any checks that the user sets the *SAME* GMBA limits.

Quoted 2 lines

What happens if the user ignores the dosumentation and sets different limits?

Good point. Adding checks could be challenging when users update each schema individually with different values. We don't know which one value is the one he is intending to keep.

Quoted 13 lines

... snip ...

+  # cat schemata
+    GMB:0=2048;1=2048;2=2048;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff
+
+  # echo "GMB:0=8;2=8" > schemata
+  # cat schemata
+    GMB:0=   8;1=2048;2=   8;3=2048
+     MB:0=4096;1=4096;2=4096;3=4096
+     L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 7 lines

Can the user go on to set:

     # echo "GMB:1=10;3=10" > schemata

and have domains 0 & 2 with a combined 8GB limit, while domains 1 & 3 run with a combined 10GB limit? Or is there a single "GMBA domain"?

In that case, it  is still treated as a single GMBA domain, but the behavior becomes unpredictable. The hardware expert mentioned that it will default to the lowest value among all inputs in this case, 8GB.

Quoted 3 lines

Will using "2048" as the "this domain isn't limited by GMBA" value come back to haunt you when some system has much more than 2TB bandwidth to divide up?

It is actually 4096 (4TB). I made a mistake in the example.  I am assuming it may not an issue in the current generation.

It is expected to go up in next generation.

GMB:0=4096;1=4096;2=4096;3=4096;
   MB:0=8192;1=8192;2=8192;3=8192;
    L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 2 lines

Should resctrl have a non-numeric "unlimited" value in the schemata file for this?

The value 4096 corresponds to 12th bit set.  It is called U-bit. If the U bit is set then that domain is not part of the GMBA domain.

I was thinking of displaying the "U" in those cases.  It may be good idea to do something like this.

GMB:0=      8;1=      U;2=     8 ;3=      U;
   MB:0=8192;1=8192;2=8192;3=8192;
    L3:0=ffff;1=ffff;2=ffff;3=ffff

Quoted 4 lines

Quoted 3 lines

The "mba_MBps" feature used U32_MAX as the unlimited value. But it looks somewhat ugly in the schemata file:

Yes, I agree. Non-numeric would have been better.

How would such a value be described in a generic way as part of the new schema description format?

Since the proposed format contains a maximum I think just using that value may be simplest while matching what is currently displayed for "unlimited" MB, no?

Add plza_capable field to the rdt_resource structure to indicate whether Privilege Level Zero Association (PLZA) is supported for that resource type.

Signed-off-by Babu Moger <[email protected]>

---
 arch/x86/kernel/cpu/resctrl/core.c     | 6 ++++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 5 +++++
 include/linux/resctrl.h                | 3 +++
 3 files changed, 14 insertions(+)

arch/x86/kernel/cpu/resctrl/core.c +6 -0

index 2de3140dd6d1..e41fe5fa3f30 100644
@@ -295,6 +295,9 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 	r->alloc_capable = true;
 
+	if (rdt_cpu_has(X86_FEATURE_PLZA))
+		r->plza_capable = true;
+
 	return true;
 }
 

@@ -314,6 +317,9 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
 		r->cache.arch_has_sparse_bitmasks = ecx.split.noncont;
 	r->alloc_capable = true;
+
+	if (rdt_cpu_has(X86_FEATURE_PLZA))
+		r->plza_capable = true;
 }
 
 static void rdt_get_cdp_config(int level)

arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -0

index 885026468440..540e1e719d7f 100644
@@ -229,6 +229,11 @@ bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
 	return rdt_resources_all[l].cdp_enabled;
 }
 
+bool resctrl_arch_get_plza_capable(enum resctrl_res_level l)
+{
+	return rdt_resources_all[l].r_resctrl.plza_capable;
+}
+
 void resctrl_arch_reset_all_ctrls(struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);

include/linux/resctrl.h +1 -0

index 63d74c0dbb8f..ae252a0e6d92 100644
@@ -319,6 +319,7 @@ struct resctrl_mon {
  * @name:		Name to use in "schemata" file.
  * @schema_fmt:		Which format string and parser is used for this schema.
  * @cdp_capable:	Is the CDP feature available on this resource
+ * @plza_capable:	Is Privilege Level Zero Association capable?
  */
 struct rdt_resource {
 	int			rid;

Hi Babu,

Quoted 64 lines · Babu Moger

Add plza_capable field to the rdt_resource structure to indicate whether Privilege Level Zero Association (PLZA) is supported for that resource type.

Signed-off-by Babu Moger <[email protected]>

---
 arch/x86/kernel/cpu/resctrl/core.c     | 6 ++++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 5 +++++
 include/linux/resctrl.h                | 3 +++
 3 files changed, 14 insertions(+)

arch/x86/kernel/cpu/resctrl/core.c +6 -0

index 2de3140dd6d1..e41fe5fa3f30 100644
@@ -295,6 +295,9 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 	r->alloc_capable = true;
 
+	if (rdt_cpu_has(X86_FEATURE_PLZA))
+		r->plza_capable = true;
+
 	return true;
 }
 

@@ -314,6 +317,9 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
 		r->cache.arch_has_sparse_bitmasks = ecx.split.noncont;
 	r->alloc_capable = true;
+
+	if (rdt_cpu_has(X86_FEATURE_PLZA))
+		r->plza_capable = true;
 }
 
 static void rdt_get_cdp_config(int level)

arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -0

index 885026468440..540e1e719d7f 100644
@@ -229,6 +229,11 @@ bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
 	return rdt_resources_all[l].cdp_enabled;
 }
 
+bool resctrl_arch_get_plza_capable(enum resctrl_res_level l)
+{
+	return rdt_resources_all[l].r_resctrl.plza_capable;
+}
+
 void resctrl_arch_reset_all_ctrls(struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);

include/linux/resctrl.h +1 -0

index 63d74c0dbb8f..ae252a0e6d92 100644
@@ -319,6 +319,7 @@ struct resctrl_mon {
  * @name:		Name to use in "schemata" file.
  * @schema_fmt:		Which format string and parser is used for this schema.
  * @cdp_capable:	Is the CDP feature available on this resource
+ * @plza_capable:	Is Privilege Level Zero Association capable?
  */
 struct rdt_resource {
 	int			rid;

@@ -334,6 +335,7 @@ struct rdt_resource {

include/linux/resctrl.h struct rdt_resource { · line 338

Quoted 4 lines

include/linux/resctrl.h +1 -0

 	char			*name;
 	enum resctrl_schema_fmt	schema_fmt;
 	bool			cdp_capable;
+	bool			plza_capable;

Why are you making plza a resource property? Certainly for MPAM we'd want this to be global across resources and I see above that you are just checking a cpu property rather then anything per resource.

Hi Ben,

Quoted 67 lines · Ben Horgan

Hi Babu,

Quoted 64 lines · Babu Moger

Add plza_capable field to the rdt_resource structure to indicate whether Privilege Level Zero Association (PLZA) is supported for that resource type.

Signed-off-by Babu Moger <[email protected]>

---
 arch/x86/kernel/cpu/resctrl/core.c     | 6 ++++++
 arch/x86/kernel/cpu/resctrl/rdtgroup.c | 5 +++++
 include/linux/resctrl.h                | 3 +++
 3 files changed, 14 insertions(+)

arch/x86/kernel/cpu/resctrl/core.c +6 -0

index 2de3140dd6d1..e41fe5fa3f30 100644
@@ -295,6 +295,9 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
 	r->alloc_capable = true;
 
+	if (rdt_cpu_has(X86_FEATURE_PLZA))
+		r->plza_capable = true;
+
 	return true;
 }
 

@@ -314,6 +317,9 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
 	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
 		r->cache.arch_has_sparse_bitmasks = ecx.split.noncont;
 	r->alloc_capable = true;
+
+	if (rdt_cpu_has(X86_FEATURE_PLZA))
+		r->plza_capable = true;
 }
 
 static void rdt_get_cdp_config(int level)

arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -0

index 885026468440..540e1e719d7f 100644
@@ -229,6 +229,11 @@ bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
 	return rdt_resources_all[l].cdp_enabled;
 }
 
+bool resctrl_arch_get_plza_capable(enum resctrl_res_level l)
+{
+	return rdt_resources_all[l].r_resctrl.plza_capable;
+}
+
 void resctrl_arch_reset_all_ctrls(struct rdt_resource *r)
 {
 	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);

include/linux/resctrl.h +1 -0

index 63d74c0dbb8f..ae252a0e6d92 100644
@@ -319,6 +319,7 @@ struct resctrl_mon {
  * @name:		Name to use in "schemata" file.
  * @schema_fmt:		Which format string and parser is used for this schema.
  * @cdp_capable:	Is the CDP feature available on this resource
+ * @plza_capable:	Is Privilege Level Zero Association capable?
  */
 struct rdt_resource {
 	int			rid;

@@ -334,6 +335,7 @@ struct rdt_resource {

Quoted 8 lines

include/linux/resctrl.h struct rdt_resource { · line 338

Quoted 4 lines

include/linux/resctrl.h +1 -0

 	char			*name;
 	enum resctrl_schema_fmt	schema_fmt;
 	bool			cdp_capable;
+	bool			plza_capable;

Why are you making plza a resource property? Certainly for MPAM we'd want this to be global across resources and I see above that you are just checking a cpu property rather then anything per resource.

I agree. For reference: https://lore.kernel.org/lkml/[email protected]/

Quoted 1 line

One possible concern for MPAM related to this caught my eye. From

https://lore.kernel.org/lkml/[email protected]/ :

If an SMCU is not shared with other cpus then it is implementation defined whether the configuration from MPAMSM_EL1 is used or that from the appropriate MPAMy_ELx. As we set the same, PMG_D and PARTID_D, configuration for MPAM0_EL1, MPAM1_EL1 and MPAMSM_EL1 the resulting configuration is the same regardless.

I admit that I am not yet comfortable with the MPAM register usages ... but from above it sounds to me as though if resctrl associates different CLOSID/PARTID and RMID/PMG with a task to be used at different privilege levels as planned with this work then the mapping to MPAM0_EL1 and MPAM1_EL1 may be easy but MPAMSM_EL1 may be difficult?

Reinette

Hi Babu,

Quoted 64 lines · Babu Moger

Add plza_capable field to the rdt_resource structure to indicate whether Privilege Level Zero Association (PLZA) is supported for that resource type.

Signed-off-by Babu Moger <[email protected]>

---
  arch/x86/kernel/cpu/resctrl/core.c     | 6 ++++++
  arch/x86/kernel/cpu/resctrl/rdtgroup.c | 5 +++++
  include/linux/resctrl.h                | 3 +++
  3 files changed, 14 insertions(+)

arch/x86/kernel/cpu/resctrl/core.c +6 -0

index 2de3140dd6d1..e41fe5fa3f30 100644
@@ -295,6 +295,9 @@ static __init bool __rdt_get_mem_config_amd(struct rdt_resource *r)
 
  	r->alloc_capable = true;
 
+	if (rdt_cpu_has(X86_FEATURE_PLZA))
+		r->plza_capable = true;
+
  	return true;
  }
 

@@ -314,6 +317,9 @@ static void rdt_get_cache_alloc_cfg(int idx, struct rdt_resource *r)
  	if (boot_cpu_data.x86_vendor == X86_VENDOR_INTEL)
  		r->cache.arch_has_sparse_bitmasks = ecx.split.noncont;
  	r->alloc_capable = true;
+
+	if (rdt_cpu_has(X86_FEATURE_PLZA))
+		r->plza_capable = true;
  }
 
  static void rdt_get_cdp_config(int level)

arch/x86/kernel/cpu/resctrl/rdtgroup.c +5 -0

index 885026468440..540e1e719d7f 100644
@@ -229,6 +229,11 @@ bool resctrl_arch_get_cdp_enabled(enum resctrl_res_level l)
  	return rdt_resources_all[l].cdp_enabled;
  }
 
+bool resctrl_arch_get_plza_capable(enum resctrl_res_level l)
+{
+	return rdt_resources_all[l].r_resctrl.plza_capable;
+}
+
  void resctrl_arch_reset_all_ctrls(struct rdt_resource *r)
  {
  	struct rdt_hw_resource *hw_res = resctrl_to_arch_res(r);

include/linux/resctrl.h +1 -0

index 63d74c0dbb8f..ae252a0e6d92 100644
@@ -319,6 +319,7 @@ struct resctrl_mon {
   * @name:		Name to use in "schemata" file.
   * @schema_fmt:		Which format string and parser is used for this schema.
   * @cdp_capable:	Is the CDP feature available on this resource
+ * @plza_capable:	Is Privilege Level Zero Association capable?
   */
  struct rdt_resource {
  	int			rid;

@@ -334,6 +335,7 @@ struct rdt_resource {

include/linux/resctrl.h struct rdt_resource { · line 338

Quoted 4 lines

include/linux/resctrl.h +1 -0

  	char			*name;
  	enum resctrl_schema_fmt	schema_fmt;
  	bool			cdp_capable;
+	bool			plza_capable;

Why are you making plza a resource property? Certainly for MPAM we'd want this to be global across resources and I see above that you are just checking a cpu property rather then anything per resource.

arch/x86/include/asm/resctrl.h +4 -0

@@ -138,6 +143,20 @@ static inline void __resctrl_sched_in(struct task_struct *tsk)
 		state->cur_rmid = rmid;
 		wrmsr(MSR_IA32_PQR_ASSOC, rmid, closid);
 	}
+
+	if (static_branch_likely(&rdt_plza_enable_key)) {
+		tmp = READ_ONCE(tsk->plza);
+		if (tmp)

Quoted 8 lines · Babu Moger

arch/x86/include/asm/resctrl.h +4 -0

@@ -138,6 +143,20 @@ static inline void __resctrl_sched_in(struct task_struct *tsk)
  		state->cur_rmid = rmid;
  		wrmsr(MSR_IA32_PQR_ASSOC, rmid, closid);
  	}
+
+	if (static_branch_likely(&rdt_plza_enable_key)) {
+		tmp = READ_ONCE(tsk->plza);
+		if (tmp)

arch/x86/include/asm/resctrl.h static inline void __resctrl_sched_in(struct task_struct *tsk) · line 159

Quoted 10 lines

arch/x86/include/asm/resctrl.h +10 -0

+			plza = tmp;
+
+		if (plza != state->cur_plza) {
+			state->cur_plza = plza;
+			wrmsr(MSR_IA32_PQR_PLZA_ASSOC,
+			      RMID_EN | state->plza_rmid,
+			      (plza ? PLZA_EN : 0) | CLOSID_EN | state->plza_closid);
+		}
+	}
+

Babu,

This addition to the context switch code surprised me. After your talk at LPC I had imagined that PLZA would be a single global setting so that every syscall/page-fault/interrupt would run with a different CLOSID (presumably one configured with more cache and memory bandwidth).

But this patch series looks like things are more flexible with the ability to set different values (of RMID as well as CLOSID) per group.

It looks like it is possible to have some resctrl group with very limited resources just bump up a bit when in ring0, while other groups may get some different amount.

The additions for plza to the Documentation aren't helping me understand how users will apply this.

Do you have some more examples?

Hi Tony,

Thanks for the comment.

Quoted 22 lines · Tony

Quoted 8 lines · Babu Moger

arch/x86/include/asm/resctrl.h +4 -0

@@ -138,6 +143,20 @@ static inline void __resctrl_sched_in(struct task_struct *tsk)
  		state->cur_rmid = rmid;
  		wrmsr(MSR_IA32_PQR_ASSOC, rmid, closid);
  	}
+
+	if (static_branch_likely(&rdt_plza_enable_key)) {
+		tmp = READ_ONCE(tsk->plza);
+		if (tmp)

arch/x86/include/asm/resctrl.h static inline void __resctrl_sched_in(struct task_struct *tsk) · line 159

Quoted 10 lines

arch/x86/include/asm/resctrl.h +10 -0

+			plza = tmp;
+
+		if (plza != state->cur_plza) {
+			state->cur_plza = plza;
+			wrmsr(MSR_IA32_PQR_PLZA_ASSOC,
+			      RMID_EN | state->plza_rmid,
+			      (plza ? PLZA_EN : 0) | CLOSID_EN | state->plza_closid);
+		}
+	}
+

Babu,

Quoted 7 lines

This addition to the context switch code surprised me. After your talk at LPC I had imagined that PLZA would be a single global setting so that every syscall/page-fault/interrupt would run with a different CLOSID (presumably one configured with more cache and memory bandwidth).

But this patch series looks like things are more flexible with the ability to set different values (of RMID as well as CLOSID) per group.

Yes. this similar what we have with MSR_IA32_PQR_ASSOC. The association can be done either thru CPUs (just one MSR write) or task based association(more MSR write as task moves around).

Quoted 9 lines

It looks like it is possible to have some resctrl group with very limited resources just bump up a bit when in ring0, while other groups may get some different amount.

The additions for plza to the Documentation aren't helping me understand how users will apply this.

Do you have some more examples?

Group creation is similar to what we have currently.

1. create a regular group and setup the limits.

   # mkdir /sys/fs/resctrl/group

2. Assign tasks or CPUs.

   # echo 1234 > /sys/fs/resctrl/group/tasks

This is a regular group.

3. Now you figured that you need to change things in CPL0 for this task.

4. Now create a PLZA group now and tweek the limits,

   # mkdir /sys/fs/resctrl/group1

   # echo 1 > /sys/fs/resctrl/group1/plza

   # echo "MB:0=100" > /sys/fs/resctrl/group1/schemata

5. Assign the same task to the plza group.

   # echo 1234 > /sys/fs/resctrl/group1/tasks

Now the task 1234 will be using the limits from group1 when running in CPL0.

I will add few more details in my next revision.

-----Original Message-----
From: Luck, Tony <[email protected]>
Sent: Wednesday, January 28, 2026 11:12 AM
To: Moger, Babu <[email protected]>
Cc: Babu Moger <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; akpm@linux- foundation.org; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 13/19] x86/resctrl: Add PLZA state tracking and context switch handling

Quoted 7 lines · Babu

Hi Tony,

Thanks for the comment.

Quoted 2 lines · Tony

Review static inline void __resctrl_sched_in(struct

Quoted 1 line · Babu Moger

(quoted diff) +0 -0

@@ -138,6 +143,20 @@ static inline void __resctrl_sched_in(struct

task_struct *tsk)

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

arch/x86/include/asm/resctrl.h +4 -0

                state->cur_rmid = rmid;
                wrmsr(MSR_IA32_PQR_ASSOC, rmid, closid);
        }
+
+       if (static_branch_likely(&rdt_plza_enable_key)) {
+               tmp = READ_ONCE(tsk->plza);
+               if (tmp)

Quoted 18 lines

Quoted 7 lines

arch/x86/include/asm/resctrl.h static inline void __resctrl_sched_in(struct task_struct *tsk)

Quoted 7 lines

arch/x86/include/asm/resctrl.h +7 -0

+                       plza = tmp;
+
+               if (plza != state->cur_plza) {
+                       state->cur_plza = plza;
+                       wrmsr(MSR_IA32_PQR_PLZA_ASSOC,
+                             RMID_EN | state->plza_rmid,
+                             (plza ? PLZA_EN : 0) | CLOSID_EN | state-

plza_closid);

Quoted 10 lines

arch/x86/include/asm/resctrl.h static inline void __resctrl_sched_in(struct task_struct *tsk) · line 159

Quoted 3 lines

arch/x86/include/asm/resctrl.h +3 -0

+               }
+       }
+

Babu,

This addition to the context switch code surprised me. After your talk at LPC I had imagined that PLZA would be a single global setting so that every syscall/page-fault/interrupt would run with a different CLOSID (presumably one configured with more cache and memory

bandwidth).

Quoted 37 lines

Quoted 3 lines

But this patch series looks like things are more flexible with the ability to set different values (of RMID as well as CLOSID) per group.

Yes. this similar what we have with MSR_IA32_PQR_ASSOC. The association can be done either thru CPUs (just one MSR write) or task based association(more MSR write as task moves around).

Quoted 9 lines

It looks like it is possible to have some resctrl group with very limited resources just bump up a bit when in ring0, while other groups may get some different amount.

The additions for plza to the Documentation aren't helping me understand how users will apply this.

Do you have some more examples?

Group creation is similar to what we have currently.

1. create a regular group and setup the limits.

   # mkdir /sys/fs/resctrl/group

2. Assign tasks or CPUs.

   # echo 1234 > /sys/fs/resctrl/group/tasks

This is a regular group.

3. Now you figured that you need to change things in CPL0 for this task.

4. Now create a PLZA group now and tweek the limits,

   # mkdir /sys/fs/resctrl/group1

   # echo 1 > /sys/fs/resctrl/group1/plza

   # echo "MB:0=100" > /sys/fs/resctrl/group1/schemata

Quoted 9 lines

5. Assign the same task to the plza group.

   # echo 1234 > /sys/fs/resctrl/group1/tasks

Now the task 1234 will be using the limits from group1 when running in CPL0.

I will add few more details in my next revision.

Babu,

I've read a bit more of the code now and I think I understand more.

Some useful additions to your explanation.

1) Only one CTRL group can be marked as PLZA

2) It can't be the root/default group

3) It can't have sub monitor groups
4) It can't be pseudo-locked

Would a potential use case involve putting *all* tasks into the PLZA group? That would avoid any additional context switch overhead as the PLZA MSR would never need to change.

If that is the case, maybe for the PLZA group we should allow user to do:

# echo '*' > tasks

Quoted 103 lines

-----Original Message-----
From: Luck, Tony <[email protected]>
Sent: Wednesday, January 28, 2026 11:12 AM
To: Moger, Babu <[email protected]>
Cc: Babu Moger <[email protected]>; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; akpm@linux- foundation.org; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; linux- [email protected]; [email protected]; [email protected]; [email protected]; [email protected]; [email protected]
Subject: Re: [RFC PATCH 13/19] x86/resctrl: Add PLZA state tracking and context switch handling

Quoted 7 lines · Babu

Hi Tony,

Thanks for the comment.

Quoted 2 lines · Tony

Review static inline void __resctrl_sched_in(struct

Quoted 1 line · Babu Moger

(quoted diff) +0 -0

@@ -138,6 +143,20 @@ static inline void __resctrl_sched_in(struct

task_struct *tsk)

Quoted 7 lines

Quoted 7 lines

Quoted 7 lines

arch/x86/include/asm/resctrl.h +4 -0

                 state->cur_rmid = rmid;
                 wrmsr(MSR_IA32_PQR_ASSOC, rmid, closid);
         }
+
+       if (static_branch_likely(&rdt_plza_enable_key)) {
+               tmp = READ_ONCE(tsk->plza);
+               if (tmp)

Quoted 18 lines

Quoted 7 lines

arch/x86/include/asm/resctrl.h static inline void __resctrl_sched_in(struct task_struct *tsk)

Quoted 7 lines

arch/x86/include/asm/resctrl.h +7 -0

+                       plza = tmp;
+
+               if (plza != state->cur_plza) {
+                       state->cur_plza = plza;
+                       wrmsr(MSR_IA32_PQR_PLZA_ASSOC,
+                             RMID_EN | state->plza_rmid,
+                             (plza ? PLZA_EN : 0) | CLOSID_EN | state-

plza_closid);

Quoted 10 lines

arch/x86/include/asm/resctrl.h static inline void __resctrl_sched_in(struct task_struct *tsk) · line 159

Quoted 3 lines

arch/x86/include/asm/resctrl.h +3 -0

+               }
+       }
+

Babu,

This addition to the context switch code surprised me. After your talk at LPC I had imagined that PLZA would be a single global setting so that every syscall/page-fault/interrupt would run with a different CLOSID (presumably one configured with more cache and memory

bandwidth).

Quoted 37 lines

Quoted 3 lines

But this patch series looks like things are more flexible with the ability to set different values (of RMID as well as CLOSID) per group.

Yes. this similar what we have with MSR_IA32_PQR_ASSOC. The association can be done either thru CPUs (just one MSR write) or task based association(more MSR write as task moves around).

Quoted 9 lines

It looks like it is possible to have some resctrl group with very limited resources just bump up a bit when in ring0, while other groups may get some different amount.

The additions for plza to the Documentation aren't helping me understand how users will apply this.

Do you have some more examples?

Group creation is similar to what we have currently.

1. create a regular group and setup the limits.

    # mkdir /sys/fs/resctrl/group

2. Assign tasks or CPUs.

    # echo 1234 > /sys/fs/resctrl/group/tasks

This is a regular group.

3. Now you figured that you need to change things in CPL0 for this task.

4. Now create a PLZA group now and tweek the limits,

    # mkdir /sys/fs/resctrl/group1

    # echo 1 > /sys/fs/resctrl/group1/plza

    # echo "MB:0=100" > /sys/fs/resctrl/group1/schemata

Quoted 9 lines

5. Assign the same task to the plza group.

    # echo 1234 > /sys/fs/resctrl/group1/tasks

Now the task 1234 will be using the limits from group1 when running in CPL0.

I will add few more details in my next revision.

Quoted 7 lines

Babu,

I've read a bit more of the code now and I think I understand more.

Some useful additions to your explanation.

1) Only one CTRL group can be marked as PLZA

Yes. Correct.

Quoted 1 line

2) It can't be the root/default group

This is something I added to keep the default group in a un-disturbed,

Quoted 2 lines

3) It can't have sub monitor groups
4) It can't be pseudo-locked

Yes.

Quoted 4 lines

Would a potential use case involve putting *all* tasks into the PLZA group? That would avoid any additional context switch overhead as the PLZA MSR would never need to change.

Yes. That can be one use case.

Quoted 5 lines

If that is the case, maybe for the PLZA group we should allow user to do:

# echo '*' > tasks

Yea. It can be done.

Thanks
Babu

Quoted 5 lines

Quoted 3 lines

Would a potential use case involve putting *all* tasks into the PLZA group? That would avoid any additional context switch overhead as the PLZA MSR would never need to change.

Yes. That can be one use case.

Quoted 4 lines · Babu

Quoted 4 lines · Babu

Quoted 1 line · Tony

Babu,

Quoted 7 lines

Quoted 5 lines

I've read a bit more of the code now and I think I understand more.

Some useful additions to your explanation.

1) Only one CTRL group can be marked as PLZA

Yes. Correct.