Nginx refcount scalability issue with Apparmor enabled and potential solutions
Subsystems (5)
cover Nginx refcount scalability issue with Apparmor enabled and potential solutions
Problem Statement
=================
Nginx performance testing with Apparmor enabled (with nginx running in unconfined profile), on kernel versions 6.1 and 6.5 show significant drop in throughput scalability, when Nginx workers are scaled to higher number of CPUs across various L3 cache domains.
Below is one sample data on the throughput scalability loss, based on results on AMD Zen4 system with 96 CPUs with SMT core count 2; so, overall, 192 CPUs:
Config Cache Domains apparmor=off apparmor=on
scaling eff (%) scaling eff (%)
8C16T 1 100% 100% 16C32T 2 95% 94% 24C48T 3 94% 93% 48C96T 6 92% 88% 96C192T 12 85% 68%
If we look at above data, there is a significant drop in scaling efficiency, when we move to 96 CPUs/192 SMT threads.
Perf tool shows most of the contention coming from below
6.56% nginx [kernel.vmlinux] [k] apparmor_current_getsecid_subj 6.22% nginx [kernel.vmlinux] [k] apparmor_file_open
The majority of the CPU cycles is found to be due to memory contention in atomic_fetch_add and atomic_fetch_sub operations from kref_get() and kref_put() operations on label.
Commit 2516fde1fa00 ("apparmor: Optimize retrieving current task secid"), from 6.7 alleviates the issue to an extent, but not completely:
Config Cache Domains apparmor=on apparmor=on (patched)
scaling eff (%) scaling eff (%)
8C16T 1 100% 100% 16C32T 2 97% 93% 24C48T 3 94% 92% 48C96T 6 88% 88% 96C192T 12 65% 79%
This adverse impact gets more pronounced when we move to >192 CPUs. The memory contention and impact increases with high frequency label update operations and labels are marked stale more frequently.
Label Refcount Management
=========================
Apparmor uses label objects (struct aa_label) to manage refcounts for below set of objects:
- Applicable profiles
- Namespaces (unconfined profile)
- Other non-profile references
These label references are acquired on various apparmor lsm hooks, on operations such as file open, task kill operations, socket bind, and other file, socket, misc operations which use current task's cred, when the label for the current cred, has been marked stale. This is done to check these operations against the set of allowed operations for the task performing them.
Use Percpu refcount for ref management?
=======================================
The ref put operations (percpu_ref_put()) in percpu refcount, in active mode, do not check whether ref count has dropped to
0. The users of the percpu_ref need to explicitly invoke a percpu_ref_kill() operation, to drop the initial reference, at shutdown paths. After the percpu_ref_kill() operation, ref switches to atomic mode and any new percpu_ref_put() operation checks for the drop to 0 case and invokes the release operation on that label.
Labels are marked stale is situations like profile removal, profile updates. For a namespace, the unconfined label reference is dropped when the namespace is destroyed. These points are potential shutdown points for labels. However, killing the percpu ref from these points has few issues:
- The label could still be referenced by tasks, which are still holding the reference to the now stale label. Killing the label ref while these operations are in progress will make all subsequent ref-put operations on the stale label to be atomic, which would still result in memory contention. Also, any new reference to the stale label, which is acquired with the elevated refcount will have atomic op contention.
- The label is marked stale using a non-atomic write operation. It is possible that new operations do not observe this flag and still reference it for quite some time.
- Explicitly tracking the shutdown points might not be maintainable at a per label granularity, as there can be various paths where label reference could get dropped, such as, before the label has gone live - object initialization error paths. Also, tracking the shutdown points for labels which reference other labels - subprofiles, merged labels requires careful analysis, and adds heavy burden on ensuring the memory contention is not introduced by these ref kill points.
Proposed Solution
=================
One potential solution to the refcount scalability problem is to convert the label refcount to a percpu refcount, and manage the initial reference from kworker context. The kworker keeps an extra reference to the label and periodically scans labels and release them if their refcount drops to 0.
Below is the sequence of operations, which shows the refcount management with this approach:
1. During label initialization, the percpu ref is initialized in atomic mode. This is done to ensure that, for cases where the label hasn't gone live (->ns isn't assigned), mostly during initialization error paths.
2. Labels are switched to percpu mode at various points - when a label is added to labelset tree, when a unconfined profile has been assigned a namespace.
3. As part of the initial prototype, only the in tree labels are managed by the kworker. These labels are added to a lockless list. The unconfined labels invoke a percpu_ref_kill() operation when the namespace is destroyed.
4. The kworker does a periodic scan of all the labels in the llist. It does below sequence of operations:
a. Enqueue a dummy node to mark the start of scan. This dummy
node is used as start point of scan and ensures that we there is no additional synchronization required with new label node additions to the llist. Any new labels will be processed in next run of the kworker.
SCAN START PTR
|
v
+----------+ +------+ +------+ +------+
| | | | | | | |
| head ------> dummy|--->|label |--->| label|--->NULL
| | | node | | | | |
+----------+ +------+ +------+ +------+
New label addition:
SCAN START PTR
|
v
+----------+ +------+ +------+ +------+ +------+
| | | | | | | | | |
| head |--> label|--> dummy|--->|label |--->| label|--->NULL
| | | | | node | | | | |
+----------+ +------+ +------+ +------+ +------+
b. Traverse through the llist, starting from dummy->next.
If the node is a dummy node, mark it free. If the node is a label node, do,
i) Switch the label ref to atomic mode. The ref switch wait for the existing percpu_ref_get() and percpu_ref_put() operations to complete, by waiting for a RCU grace period.
Once the switch is complete, from this point onwards, any percpu_ref_get(), percpu_ref_put() operations use atomic operations.
ii) Drop the initial reference, which was taken while adding
the label node to the llist.
iii) Use a percpu_ref_tryget() increment operation on the
ref, to see if we dropped the last ref count. if we dropped the last count, we remove the node from the llist.
All of these operations are done inside a RCU critical section, to avoid race with the release operations, which can potentially trigger, as soon as we drop the initial ref count.
iv) If we didn't drop the last ref, switch back the counter
to percpu mode.
Using this approach, to move the atomic refcount manipulation out of the contended paths, there is a significant scalability improvement seen on nginx test, and scalability efficiency is close to apparmor-off case.
Config Cache Domains apparmor=on (percpuref)
scaling eff (%)
8C16T 1 100% 16C32T 2 96% 24C48T 3 94% 48C96T 6 93% 96C192T 12 90%
Limitations
===========
1. Switching to percpu refcount increases memory size overhead, as percpu memory is allocated for all labels.
2. Deferring labels reclaim could potentially result in memory pressure, when there are high frequency of label update operations.
3. Percpu refcount uses call_rcu_hurry() to complete switch operations. These can impact energy efficiency, due to back to back hurry callbacks. Using deferrable workqueue partly mitigates this. However, deferring kworker can delay reclaims.
4. Back to back label switches can delay other percpu users, as there is a single global switch spinlock used by percpu refcount lib.
5. Long running kworker can delay other use cases like system suspend. This is mitigated using freezable workqueue and litming node scans to a max count.
6. There is a window where label operates is atomic mode, when its counter is being checked for zero. This can potentially result in high memory contention, during this window which spans RCU grace period (plus callback execution). For example, when scanning label corresponding to unconfined profile, all applications which use unconfined profile would be using atomic ref increment and decrement operations.
There are a few apparoaches which were tried to mitigate this issue:
a. At a lower time interval, check if scanned label's counter
has changed since the start of label scan. If there is a change
in count, terminate the switch to atomic mode. Below shows the
apparoch using rcuwait.
static void aa_label_switch_atomic_confirm(struct percpu_ref *label_ref)
{
WRITE_ONCE(aa_atomic_switch_complete, true);
rcuwait_wake_up(&aa_reclaim_rcuwait);
}
rcuwait_init(&aa_reclaim_rcuwait);
percpu_ref_switch_to_atomic(&label->count, aa_label_switch_atomic_confirm);
atomic_count = percpu_ref_count_read(&label->count);
do {
rcuwait_wait_event_timeout(&aa_reclaim_rcuwait,
(switch_complete = READ_ONCE(aa_atomic_switch_complete)),
TASK_IDLE,
msecs_to_jiffies(5));
if (percpu_ref_count_read(&label->count) != atomic_count)
break;
} while (!READ_ONCE(switch_complete));
However, this approach does not work, as percpu refcount lib does not
allow termination of an ongoing switch operation. Also, the counter
can return to the original value with set of get() and put() operations
before we check the current value. b. Approaches to notify the reclaim kworker from ref get and put operations
can potentially disturb cache line state between the various CPU contexts, which are referncing the label, and can potentially impact scalability again.
c. Swith the label to an immortal percpu ref, while the scan operates
on the current counter.
Below is the sequence of operations to do this:
1. Ensure that both immortal ref and label ref are in percpu mode. Reinit the immortal ref in percpu mode.
Swap percpu and atomic counters of label refcount and immortal ref
percpu-ref
+-------------------+
+-------+ | percpu-ctr-addr1 |
| label | --------->|-------------------| +----------------+
+-------+ | data |--->| Atomic counter1|
+-------------------+ +----------------+
+-------+ +-------------------+
|ImmLbl |---------->| percpu-ctr-addr2 | +----------------+
+-------+ |-------------------|--->| Atomic counter2|
| data | +----------------+
+-------------------+
label ->percpu-ctr-addr = percpu-ctr-addr2 ImmLbl ->percpu-ctr-addr = percpu-ctr-addr1
label ->data->count = Atomic counter2
ImmLbl ->data->count = Atomic counter1
2. Check the counters collected in immortal label, by switch it to atomic mode.
3. If the count is 0, do,
a. Switch immortal counter to percpu again, giving it an
initial count of 1.
b. Swap the label and immortal counters again. The immortal
ref now has the counter values from new percpu ref get and get operations on the label ref, from the point when we did the initial swap operation.
c. Transfer the percpu counts in immortal ref to atomic
counter of label percpu refcount.
d. Kill immortal ref, for reinit on next iteration. e. Switch label percpu ref to atomic mode.
f. If the counter is 1, drop the initial ref.
4. If the count is not 0, re-swap the counters.
a. Switch immortal counter to percpu again, giving it an
initial count of 1.
b. Swap the label and immortal counters again. The immortal
ref now has the counter values from new percpu ref get and get operations on the label ref, from the point when we did the initial swap operation.
c. Transfer the percpu counts in immortal ref to atomic
counter of label percpu refcount.
d. Kill immortal ref, for reinit on next iteration.
Using this approach, we ensure that, label ref users do not switch to atomic mode, while there are active references on the label. However, this approach requires multiple percpu ref mode switches and adds high overhead and complexity to the scanning code.
Extended/Future Work
====================
1. Look for ways to fix the limitations, as described in the "Limitations" section.
2. Generalize the approach to percpu rcuref, which is used for contexts where release path uses RCU grace period for release operations. Patch 7 creates an initial prototype for this.
3. Explore hazard pointers for scalable refcounting of labels.
Highly appreciate any feedback/suggestions on the design approach.
The patches of this patchset introduce following changes:
1. Documentation of Apparmor Refcount management.
2. Switch labels to percpu refcount in atomic mode.
Use percpu refcount for apparmor labels. Initial patch to init the percpu ref in atomic mode, to evaluate the potential impact of percpuref on top of kref based implementation.
3. Switch unconfined namespaces refcount to percpu mode.
Switch unconfined ns labels to percpu mode, and kill the initial refcount from namespace destroy path.
4. Add infrastructure to reclaim percpu labels.
Add a label reclaim infrastructure for labels which are in percpu mode, for managing their inital refcount.
5. Switch intree labels to percpu mode.
Use label reclaim infrastruture to manage intree labels.
6. Initial prototype for optimizing ref switch.
Prototype for reducing the time window when a label scan switches the label ref to atomic mode.
7. percpu-rcuref: Add basic infrastructure.
Prototype for Percpu refcounts for objects, which protect their object reclaims using RCU grace period.
8. Switch labels to percpu rcurefcount in unmanaged mode.
Use percpu rcuref for labels. Start with unmanaged/atomic mode.
9. Switch unconfined and in tree labels to managed ref mode.
Use percpu mode with manager worker for unconfined and intree labels.
------------------------------------------------------------------------
b/Documentation/admin-guide/LSM/ApparmorRefcount.rst | 351 ++++++++++++++++++++++++++++++++++++++++++++++++++ b/Documentation/admin-guide/LSM/index.rst | 1 b/Documentation/admin-guide/kernel-parameters.txt | 8 + b/include/linux/percpu-rcurefcount.h | 115 ++++++++++++++++ b/include/linux/percpu-refcount.h | 2 b/lib/Makefile | 2 b/lib/percpu-rcurefcount.c | 336 +++++++++++++++++++++++++++++++++++++++++++++++ b/lib/percpu-refcount.c | 93 +++++++++++++ b/security/apparmor/include/label.h | 16 +- b/security/apparmor/include/policy.h | 8 - b/security/apparmor/include/policy_ns.h | 24 +++ b/security/apparmor/label.c | 11 + b/security/apparmor/lsm.c | 145 ++++++++++++++++++++ b/security/apparmor/policy_ns.c | 6 include/linux/percpu-refcount.h | 2 lib/percpu-refcount.c | 93 ------------- security/apparmor/include/label.h | 17 +- security/apparmor/include/policy.h | 56 +++---- security/apparmor/include/policy_ns.h | 24 --- security/apparmor/label.c | 11 - security/apparmor/lsm.c | 325 ++++++++++++---------------------------------- security/apparmor/policy_ns.c | 8 - 22 files changed, 1237 insertions(+), 417 deletions(-)
base-commit: ab27740f7665
RFC 1/9 doc: Add document for apparmor refcount management
Add a document to describe refcount management of AppArmor kernel objects.
--- .../admin-guide/LSM/ApparmorRefcount.rst | 351 ++++++++++++++++++ Documentation/admin-guide/LSM/index.rst | 1 + 2 files changed, 352 insertions(+) create mode 100644 Documentation/admin-guide/LSM/ApparmorRefcount.rst
Documentation/admin-guide/LSM/ApparmorRefcount.rst new file +351 -0
@@ -0,0 +1,351 @@ +============================ +AppArmor Refcount Management +============================ + +Introduction +============ + +AppArmor task confinement is based on task profiles. Task profiles +specify the access rules - list of files which the task is allowed +to open/read/write, socket bind, mount, signal, ptrace and other +capabilities of the task. + +A sample raw task profile (typically present in /etc/apparmor.d/) +would look like this: + +:: + + 1. profile test_app /usr/bin/test_app { + 2. @{sys}/devices/** r, + 3. /var/log/testapp/access.log w, + 4. /var/log/testapp/error.log w, + 5. + 6. /lib/ld-*.so* mrix, + 7. + 8. ^hat { + 9. /dev/pts/* rw, + 10. } + 11. + 12. change_profile -> restricted_access_profile, + 13. + 14. /usr/*bash cx -> local_profile, + 15. + 16. profile local_profile { + 17. ... + 18. } + 19. } + + +Above example shows a sample profile. A quick description of +each line is given below: + +1 Defines a profile with name ``test_app`` and attachment specification + ``/usr/bin/test_app``. The attachment specification is used for + associating the application with a profile, during launch. + +2,3, 4 + Specifies read and write access to various paths. + +6 Read access for the so and inherit profile transition specification. + +8 Hat profile. Used for running a portion of the program with different + permissions, compared to the other portions of the program. For example, + to run unauthenticated traffic and authenticated traffic in separate + profiles in OpenSSH; running user supplied CGI scripts in separate + profile in Apache. + +12 Change profile rules, to switch child process to a profile, different + from the parent process, on exec. + +14 Profile transition for processes started from the current procees. For + example, transition to a different profile for ``ls``, which is invoked + from a shell program. + + +Objects and their Refcount Management +===================================== + +There are various object resources within AppArmor + +- Namespaces + + There is a root namespace associated for apparmorfs. This is the default + namespace, to which all profiles are associated with. + + Profiles can be associated with a different namespaces (for chroot, + containers env). + + Namespaces are represented using ``struct aa_ns``. Some of the relevant + fields are:: + + struct aa_policy base + struct aa_profile *unconfined + struct list_head sub_ns + struct aa_ns *parent + + ``struct aa_policy`` contains a list of profiles associated with this ns. + + ``unconfined`` profile manages refcount for this namespace. It is also + used as the default profile for tasks in this namespace and a proxy label, + when profiles are removed. + + ``sub_ns`` is the list of child namespaces. + + ``parent`` Parent namespace, for this namespace. + + A parent and its child sub namespaces keep reference to each other:: + + +---------------------+ + | | + | root_ns | + | | + +---------------------+ + ^ / \ ^ + / / \ \ + parent / / \ \ parent + / / sub_ns \ \ + / / \ \ + / / \ \ + / v v \ + +-----------+ +-----------+ + | | | | + | ns1 | | ns2 | + | | | | + +-----------+ +-----------+ + + Here, ``root_ns`` is the root apparmor namespace. It maintains a + reference to all child namespace which are present in ``->sub_ns``. + The child namespaces ``ns1``, ``ns2`` keep a reference to their + ``parent``, which is the ``root_ns``. + + +- Profiles + + Profiles are represented as ``struct aa_profile`` + + Some of the fields of interest are:: + + struct aa_policy base + struct aa_profile __rcu *parent + struct aa_ns *ns + struct aa_loaddata *rawdata + struct aa_label label + + ``base`` - Maintains the list of child subprofiles - hats + + ``parent`` - If subprofile, pointer to the parent profile + + ``ns`` - Parent namespace + + ``rawdata`` - Used for matching profile data, for profile updates + + ``label`` - Refcount object + + A profile keeps a reference to the namespace it is associated with. + In addition, there is a reference kept for all profiles in + ``base.profiles`` of a namespace:: + + +-----------------------------+ + | | + | root_ns | + | | + +-----------------------------+ + ^ / ^ | + / / ns | | + parent / / | | + / / sub_ns | |base.profiles + / / | | + / / | | + / v | v + +-----------+ +-----------+ + | | | | + | ns1 | | A | + | | | | + +-----------+ +-----------+ + base | ^ + .profiles| | parent + v | + +-----------+ + | | + | P | + | | + +-----------+ + + For subprofiles, a refcount is kept for the ``->parent`` profile. + For each child in ``->base.profiles``, a refcount is kept:: + + +--------------+ + | | + | root_ns | + | | + +-------^------+ + base. | | + profiles v |ns + +---------------+ + | | + ^| A |^ + parent / | | \parent + / +---------------+ \ + / / base.profiles\ \ + / / v \ + +------v-+ +----\---+ + | | | | + | B | | C | + | | | | + +--------+ +--------+ + + +- Labels + + Label manages refcount for the ``aa_profile`` objects. It is + represented as ``struct aa_label``. Some of the fields are:: + + struct kref count + struct aa_proxy *proxy + long flags + int size + struct aa_profile *vec[] + + ``count`` - Refcounting for the enclosing object. + ``proxy`` - Redirection of stale profiles + ``flags`` - state (STALE), type (NS, PROFILE) + ``vec`` - if ``size`` > 1, for compound labels (for stacked profiles) + + + For compound/stack labels, there is a reference kept, for all + the stack profiles:: + + +--------+ +---------+ +-------+ + | A | | B | | C | + | | | | | | + +-----^--+ +---------+ +-------+ + ^ \ ^ ^ + \ \ | | + \ \ +---------------+ | + \ \ | A//&:ns1:B | | + \ \| | | + \ +---------------+ | + \ | + \ | + \ +-------------------+ + \|A//&:ns1:B//&:ns2:C| + | | + +-------------------+ + +- Proxy + + A proxy is associated with a label, and is used for redirecting + running tasks to new profile on profile change. Proxies are + represented as ``struct aa_proxy``:: + + struct aa_label __rcu *label + + ``label`` - Redirect to the latest profile label. + + While a label is not stale, its proxy points to the same label. + On profile updates, the proxy, the label is marked as stale, + and label is redirected to the new profile label:: + + +------------+ +-----------+ + | | | | + | old | -------->| P1 | + | | <--------| | + +------------+ +-----------+ + + + +------------+ +------------+ + | | | | + | old |-------->| P1 | + | | | | + +------------+ +--^---------+ + | | + +------------+ | | + | |-----------/ | + | new |<-------------/ + | | + +------------+ + +Lifecycle of the Apparmor Kernel Objects +======================================== + +#. Init + + #. During AppArmor init, root ns (RNS:1) and its unconfined + profile are created (RNS:1). If initialization completes + successfully, the ``root_ns`` initial ref is never destroyed + (?). + + #. Usespace init scripts load the current set of defined profiles + into kernel, typically through ``apparmor_parser`` command. + + The loaded profiles have an initial refcount (P1:1 , P2:1). + A profile P1, which is in default namespace keeps a reference + to root ns (RNS:2). If a profile P2 is in a different namespace, + NS1, that namespace object is allocated (NS1:1) and the namespace + is added to ``sub_ns`` of ``root_ns`` (NS1:2). The child namespace + NS1 keeps a reference to parent ``root_ns`` (RNS:3). P2 keeps a + reference to NS1 (NS1:2). The root ns keeps a reference to P1 in + ``->profiles`` list (P1:2). NS1 keeps a reference to P2 in its + ``->profiles`` list (P2:2). In addition, label proxies keep + reference to P1 and P2 (P1:3, P2:3). + +#. Runtime + + #. As part of the bprm cred updates (``apparmor_bprm_creds_for_exec()``), + the current task T1 is attached to a profile (P1), based on the best + attachment match rule. T1 keeps a refcount for P1, while the current + ``cred`` is active (P1:4). + + #. If P1 is replaced with a new profile P3, P1 is removed from the root + ns profiles list (P1:3), proxy is redirected to P3 (P1:2), and the + initial label is released (P1:1) and P1's label is marked stale. + + #. Any T1 accesses, which have a apparmor hook, would reference the + current task's cred label:: + + __begin_current_label_crit_section() + struct aa_label *label = cred_label(cred); + + if (label_is_stale(label)) + label = aa_get_newest_label(label); + + return label; + + aa_get_newest_label(struct aa_label __rcu **l) + return aa_get_label_rcu(&l->proxy->label); + + aa_get_label_rcu(struct aa_label __rcu **l) + rcu_read_lock(); + do { + c = rcu_dereference(*l); + } while (c && !kref_get_unless_zero(&c->count)); + rcu_read_unlock(); + + #. On task exit and cref freeing, the last reference for P1 is + released (P1:0). + +#. Release + + Below is the set of release operations, based on the label's + parent object type. + + #. If ns is not assigned (early init error exit), do not wait for + RCU grace period. Otherwise use ``call_rcu()`` + + #. If label is associated with a namespace (unconfined label) + #. Drop Parent ns reference. + + #. If label is associated with a profile + #. Drop parent profile reference. + #. Drop ns reference. + + #. Drop all vector profile references for stacked profiles. + + +Links +===== + +Userspace tool - https://gitlab.com/apparmor/apparmor + Profile syntax - parser/apparmor.d.pod + Sample change hats - changehat/ + Other documentation - libraries/libapparmor/doc
Documentation/admin-guide/LSM/index.rst +1 -0
@@ -41,6 +41,7 @@ subdirectories. :maxdepth: 1 apparmor + ApparmorRefcount LoadPin SELinux Smack
Signature · 1 line
2.34.1
RFC 2/9 apparmor: Switch labels to percpu refcount in atomic mode
In reply to Neeraj UpadhyayIn preparation of using percpu refcount for labels, this patch replaces label kref with percpu refcount. The percpu ref is initialized to atomic mode, as using percpu mode, requires tracking ref kill points. As the atomic counter is in a different cacheline now, rearrange some of the fields - flags, proxy; to optimize some of the fast paths for unconfined labels.
In addition to the requirement to cleanup the percpu ref using percpu_ref_exit() in label destruction path, other potential impact from this patch could be:
- Increase in memory requirement (for per cpu counters) for each label.
- Displacement of aa_label struct members to different cacheline, as percpu ref takes 2 pointers space.
- Moving of the atomic counter outside of the cacheline of the aa_label struct.
--- security/apparmor/include/label.h | 16 ++++++++-------- security/apparmor/include/policy.h | 8 ++++---- security/apparmor/label.c | 11 ++++++++--- 3 files changed, 20 insertions(+), 15 deletions(-)
security/apparmor/include/label.h +8 -8
@@ -121,12 +121,12 @@ struct label_it {* @ent: set of profiles for label, actual size determined by @size */ struct aa_label { -struct kref count; +struct percpu_ref count; +long flags; +struct aa_proxy *proxy;struct rb_node node;struct rcu_head rcu; -struct aa_proxy *proxy; __counted char *hname; -long flags;u32 secid;int size;struct aa_profile *vec[];
@@ -276,7 +276,7 @@ void __aa_labelset_update_subtree(struct aa_ns *ns);void aa_label_destroy (struct aa_label *label);void aa_label_free (struct aa_label *label); -void aa_label_kref (struct kref *kref); +void aa_label_percpu_ref (struct percpu_ref *ref);bool aa_label_init (struct aa_label *label,int size,gfp_t gfp);struct aa_label *aa_label_alloc (int size,struct aa_proxy *proxy,gfp_t gfp);
@@ -373,7 +373,7 @@ int aa_label_match(struct aa_profile *profile, struct aa_ruleset *rules,*/ static inline struct aa_label *__aa_get_label (struct aa_label *l) { -if (l &&kref_get_unless_zero (&l->count)) +if (l &&percpu_ref_tryget (&l->count))return l;return NULL ;
@@ -382,7 +382,7 @@ static inline struct aa_label *__aa_get_label(struct aa_label *l)static inline struct aa_label *aa_get_label (struct aa_label *l) {if (l) -kref_get (&(l->count)); +percpu_ref_get (&(l->count));return l; }
@@ -402,7 +402,7 @@ static inline struct aa_label *aa_get_label_rcu(struct aa_label __rcu **l)rcu_read_lock ();do { c =rcu_dereference (*l); - }while (c && !kref_get_unless_zero (&c->count)); + }while (c && !percpu_ref_tryget (&c->count));rcu_read_unlock ();return c;
@@ -442,7 +442,7 @@ static inline struct aa_label *aa_get_newest_label(struct aa_label *l)static inline void aa_put_label (struct aa_label *l) {if (l) -kref_put (&l->count, aa_label_kref); +percpu_ref_put (&l->count); }
security/apparmor/include/policy.h +4 -4
@@ -329,7 +329,7 @@ static inline aa_state_t ANY_RULE_MEDIATES(struct list_head *head,static inline struct aa_profile *aa_get_profile (struct aa_profile *p) {if (p) -kref_get (&(p->label.count)); +percpu_ref_get (&(p->label.count));return p; }
@@ -343,7 +343,7 @@ static inline struct aa_profile *aa_get_profile(struct aa_profile *p)*/ static inline struct aa_profile *aa_get_profile_not0 (struct aa_profile *p) { -if (p &&kref_get_unless_zero (&p->label.count)) +if (p &&percpu_ref_tryget (&p->label.count))return p;return NULL ;
@@ -363,7 +363,7 @@ static inline struct aa_profile *aa_get_profile_rcu(struct aa_profile __rcu **p)rcu_read_lock ();do { c =rcu_dereference (*p); - }while (c && !kref_get_unless_zero (&c->label.count)); + }while (c && !percpu_ref_tryget (&c->label.count));rcu_read_unlock ();return c;
@@ -376,7 +376,7 @@ static inline struct aa_profile *aa_get_profile_rcu(struct aa_profile __rcu **p)static inline void aa_put_profile (struct aa_profile *p) {if (p) -kref_put (&p->label.count, aa_label_kref); +percpu_ref_put (&p->label.count); }static inline int AUDIT_MODE (struct aa_profile *profile)
security/apparmor/label.c +8 -3
@@ -336,6 +336,7 @@ void aa_label_destroy(struct aa_label *label)rcu_assign_pointer (label->proxy->label,NULL );aa_put_proxy (label->proxy); } +percpu_ref_exit (&label->count);aa_free_secid (label->secid); label->proxy = (struct aa_proxy *)PROXY_POISON +1 ;
@@ -369,9 +370,9 @@ static void label_free_rcu(struct rcu_head *head)label_free_switch (label); } -void aa_label_kref (struct kref *kref) +void aa_label_percpu_ref (struct percpu_ref *ref) { - struct aa_label *label = container_of(kref, struct aa_label, count); + struct aa_label *label = container_of(ref, struct aa_label, count);struct aa_ns *ns =labels_ns (label);if (!ns) {
@@ -408,7 +409,11 @@ bool aa_label_init(struct aa_label *label, int size, gfp_t gfp) label->size = size;/* doesn't include null */ label->vec[size] =NULL ;/* null terminate */ -kref_init (&label->count); +if (percpu_ref_init (&label->count, aa_label_percpu_ref,PERCPU_REF_INIT_ATOMIC , gfp)) { +aa_free_secid (label->secid); +return false; + } +RB_CLEAR_NODE (&label->node);return true;
Signature · 1 line
2.34.1
RFC 3/9 apparmor: Switch unconfined namespaces refcount to percpu mode
In reply to Neeraj UpadhyaySwitch unconfined labels to percpu mode, to avoid high memory contention on refcount get and put operations, when multiple cpus try to perform these operations
at the same time. Unconfined label for root and sub namespaces are killed at the point of aa_free_root_ns(). Though labels/profiles in various namespaces could potentially still be active after this point, aa_free_root_ns() is not typically called when apparmor enforcement is enabled.
--- security/apparmor/include/policy.h | 24 ++++++++++++++++++++++++ security/apparmor/include/policy_ns.h | 24 ++++++++++++++++++++++++ security/apparmor/policy_ns.c | 6 ++++-- 3 files changed, 52 insertions(+), 2 deletions(-)
security/apparmor/include/policy.h +24 -0
@@ -379,6 +379,30 @@ static inline void aa_put_profile(struct aa_profile *p)percpu_ref_put (&p->label.count); } +/** +* aa_switch_ref_profile - switch percpu-ref mode for profile @p +* @p: profile (MAYBE NULL) +*/ +static inline void aa_switch_ref_profile (struct aa_profile *p,bool percpu) +{ +if (p) { +if (percpu) +percpu_ref_switch_to_percpu (&p->label.count); +else +percpu_ref_switch_to_atomic_sync (&p->label.count); + } +} + +/** +* aa_kill_ref_profile - percpu-ref kill for profile @p +* @p: profile (MAYBE NULL) +*/ +static inline void aa_kill_ref_profile (struct aa_profile *p) +{ +if (p) +percpu_ref_kill (&p->label.count); +} +static inline int AUDIT_MODE (struct aa_profile *profile) {if (aa_g_audit !=AUDIT_NORMAL )
security/apparmor/include/policy_ns.h +24 -0
@@ -127,6 +127,30 @@ static inline void aa_put_ns(struct aa_ns *ns)aa_put_profile (ns->unconfined); } +/** +* aa_switch_ref_ns - switch percpu-ref mode for @ns +* @ns: namespace to switch percpu-ref mode of +* +* Switch percpu-ref mode of @ns between percpu and atomic +*/ +static inline void aa_switch_ref_ns (struct aa_ns *ns,bool percpu) +{ +if (ns) +aa_switch_ref_profile (ns->unconfined, percpu); +} + +/** +* aa_kill_ref_ns - do percpu-ref kill for @ns +* @ns: namespace to perform percpu-ref kill for +* +* Do percpu-ref kill of @ns refcount +*/ +static inline void aa_kill_ref_ns (struct aa_ns *ns) +{ +if (ns) +aa_kill_ref_profile (ns->unconfined); +} +/** * __aa_findn_ns - find a namespace on a list by @name * @head: list to search for namespace on (NOT NULL)
security/apparmor/policy_ns.c +4 -2
@@ -124,6 +124,7 @@ static struct aa_ns *alloc_ns(const char *prefix, const char *name) goto fail_unconfined;/* ns and ns->unconfined share ns->unconfined refcount */ ns->unconfined->ns = ns; +aa_switch_ref_ns (ns, true);atomic_set (&ns->uniq_null,0 );
@@ -336,7 +337,7 @@ void __aa_remove_ns(struct aa_ns *ns)/* remove ns from namespace list */ list_del_rcu (&ns->base.list);destroy_ns (ns); -aa_put_ns (ns); +aa_kill_ref_ns (ns); }/**
@@ -377,6 +378,7 @@ int __init aa_alloc_root_ns(void) } kernel_t = &kernel_p->label; root_ns->unconfined->ns =aa_get_ns (root_ns); +aa_switch_ref_ns (root_ns, true);return 0 ; }
@@ -392,5 +394,5 @@ void __init aa_free_root_ns(void)aa_label_free (kernel_t);destroy_ns (ns); -aa_put_ns (ns); +aa_kill_ref_ns (ns); }
Signature · 1 line
2.34.1
RFC 4/9 apparmor: Add infrastructure to reclaim percpu labels
In reply to Neeraj UpadhyayNginx performance testing with Apparmor enabled (with nginx running in unconfined profile), on kernel versions 6.1 and 6.5 show significant drop in throughput scalability, when Nginx workers are scaled to higher number of CPUs across various L3 cache domains.
Below is one sample data on the throughput scalability loss, based on results on AMD Zen4 system with 96 CPUs with SMT core count 2; so, overall, 192 CPUs:
Config Cache Domains apparmor=off apparmor=on
scaling eff (%) scaling eff (%)
8C16T 1 100% 100% 16C32T 2 95% 94% 24C48T 3 94% 93% 48C96T 6 92% 88% 96C192T 12 85% 68%
There is a significant drop in scaling efficiency, when we move to 96 CPUs/192 SMT threads.
Perf tool shows most of the contention coming from below
6.56% nginx [kernel.vmlinux] [k] apparmor_current_getsecid_subj 6.22% nginx [kernel.vmlinux] [k] apparmor_file_open
The majority of the CPU cycles is found to be due to memory contention in atomic_fetch_add and atomic_fetch_sub operations from kref_get() and kref_put() operations on label.
A part of the contention was remove with commit 2516fde1fa00 ("apparmor: Optimize retrieving current task secid"), which is part of 6.7-rc1 release. After including this commit, the scaling efficiency improved as shown below:
Config Cache Domains apparmor=on apparmor=on (patched)
scaling eff (%) scaling eff (%)
8C16T 1 100% 100% 16C32T 2 97% 93% 24C48T 3 94% 92% 48C96T 6 88% 88% 96C192T 12 65% 79%
However, the scaling efficiency impact is still significant even after including the commit. In addition, the performance impact is even higher when we move to >192 CPUs. In addition, the memory contention impact would increase whem there is a high frequency of label update operations and labels are marked stale more frequently.
This patch adds a mechanism to manage reclaim of apparmor labels, when they are working in percpu mode. Using percpu refcount for apparmor label refcount helps solve the throughput scalability drop problem seen on nginx.
Config Cache Domains apparmor=on (percpuref)
scaling eff (%)
8C16T 1 100% 16C32T 2 96% 24C48T 3 94% 48C96T 6 93% 96C192T 12 90%
Below is the sequence of operations, which shows the refcount management with this approach:
1. During label initialization, the percpu ref is initialized in atomic mode. This is done to ensure that, for cases where the label hasn't gone live (->ns isn't assigned), mostly during initialization error paths.
2. Labels are switched to percpu mode at various points - when a label is added to labelset tree, when a unconfined profile has been assigned a namespace.
3. As part of the initial prototype, only the in tree labels are managed by the kworker. These labels are added to a lockless list. The unconfined labels invoke a percpu_ref_kill() operation when the namespace is destroyed.
4. The kworker does a periodic scan of all the labels in the llist. It does below sequence of operations:
a. Enqueue a dummy node to mark the start of scan. This dummy
node is used as start point of scan and ensures that we there is no additional synchronization required with new label node additions to the llist. Any new labels will be processed in next run of the kworker.
SCAN START PTR
|
v
+----------+ +------+ +------+ +------+
| | | | | | | |
| head ------> dummy|--->|label |--->| label|--->NULL
| | | node | | | | |
+----------+ +------+ +------+ +------+
New label addition:
SCAN START PTR
|
v
+----------+ +------+ +------+ +------+ +------+
| | | | | | | | | |
| head |--> label|--> dummy|--->|label |--->| label|--->NULL
| | | | | node | | | | |
+----------+ +------+ +------+ +------+ +------+
b. Traverse through the llist, starting from dummy->next.
If the node is a dummy node, mark it free. If the node is a label node, do,
i) Switch the label ref to atomic mode. The ref switch wait for the existing percpu_ref_get() and percpu_ref_put() operations to complete, by waiting for a RCU grace period.
Once the switch is complete, from this point onwards, any percpu_ref_get(), percpu_ref_put() operations use atomic operations.
ii) Drop the initial reference, which was taken while adding
the label node to the llist.
iii) Use a percpu_ref_tryget() increment operation on the
ref, to see if we dropped the last ref count. if we dropped the last count, we remove the node from the llist.
All of these operations are done inside a RCU critical section, to avoid race with the release operations, which can potentially trigger, as soon as we drop the initial ref count.
iv) If we didn't drop the last ref, switch back the counter
to percpu mode.
--- security/apparmor/include/label.h | 3 + security/apparmor/lsm.c | 145 ++++++++++++++++++++++++++++++ 2 files changed, 148 insertions(+)
security/apparmor/include/label.h +3 -0
@@ -125,6 +125,7 @@ struct aa_label {long flags;struct aa_proxy *proxy;struct rb_node node; +struct llist_node reclaim_node;struct rcu_head rcu; __counted char *hname;u32 secid;
@@ -465,4 +466,6 @@ static inline void aa_put_proxy(struct aa_proxy *proxy)void __aa_proxy_redirect (struct aa_label *orig,struct aa_label *new); +void aa_label_reclaim_add_label (struct aa_label *label); +#endif /* __AA_LABEL_H */
security/apparmor/lsm.c +145 -0
157 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC 5/9 apparmor: Switch intree labels to percpu mode
In reply to Neeraj UpadhyayNow that we have an infrastructure to reclaim
the percpu labels, switch intree labels to
percpu mode, to improve the ref scalability.
--- security/apparmor/label.c | 2 ++ 1 file changed, 2 insertions(+)
security/apparmor/label.c +2 -0
@@ -710,6 +710,8 @@ static struct aa_label *__label_insert(struct aa_labelset *ls,rb_link_node (&label->node, parent, new);rb_insert_color (&label->node, &ls->root); label->flags |=FLAG_IN_TREE ; +percpu_ref_switch_to_percpu (&label->count); +aa_label_reclaim_add_label (label);return aa_get_label (label); }
Signature · 1 line
2.34.1
RFC 6/9 apparmor: Initial prototype for optimizing ref switch
In reply to Neeraj UpadhyayThis patches adds a prototype for optimizing the atomic window, during label scan by switching to an immortal percpu ref.
Below is the sequence of operations to do this:
1. Ensure that both immortal ref and label ref are in percpu mode. Reinit the immortal ref in percpu mode.
Swap percpu and atomic counters of label refcount and immortal ref
percpu-ref
+-------------------+
+-------+ | percpu-ctr-addr1 |
| label | --------->|-------------------| +----------------+
+-------+ | data |--->| Atomic counter1|
+-------------------+ +----------------+
+-------+ +-------------------+
|ImmLbl |---------->| percpu-ctr-addr2 | +----------------+
+-------+ |-------------------|--->| Atomic counter2|
| data | +----------------+
+-------------------+
label ->percpu-ctr-addr = percpu-ctr-addr2 ImmLbl ->percpu-ctr-addr = percpu-ctr-addr1
label ->data->count = Atomic counter2
ImmLbl ->data->count = Atomic counter1
2. Check the counters collected in immortal label, by switch it to atomic mode.
3. If the count is 0, do,
a. Switch immortal counter to percpu again, giving it an
initial count of 1.
b. Swap the label and immortal counters again. The immortal
ref now has the counter values from new percpu ref get and get operations on the label ref, from the point when we did the initial swap operation.
c. Transfer the percpu counts in immortal ref to atomic
counter of label percpu refcount.
d. Kill immortal ref, for reinit on next iteration. e. Switch label percpu ref to atomic mode.
f. If the counter is 1, drop the initial ref.
4. If the count is not 0, terminate the operations and re-swap the counters.
a. Switch immortal counter to percpu again, giving it an
initial count of 1.
b. Swap the label and immortal counters again. The immortal
ref now has the counter values from new percpu ref get and get operations on the label ref, from the point when we did the initial swap operation.
c. Transfer the percpu counts in immortal ref to atomic
counter of label percpu refcount.
d. Kill immortal ref, for reinit on next iteration.
Using this approach, we ensure that, label ref users do not switch to atomic mode, while there are active references on the label. However, this approach requires multiple percpu ref mode switches and adds high overhead and complexity to the scanning code.
--- include/linux/percpu-refcount.h | 2 + lib/percpu-refcount.c | 93 +++++++++++++++++++++++++++++ security/apparmor/lsm.c | 101 ++++++++++++++++++++++++++++---- 3 files changed, 185 insertions(+), 11 deletions(-)
include/linux/percpu-refcount.h +2 -0
@@ -131,6 +131,8 @@ void percpu_ref_kill_and_confirm(struct percpu_ref *ref,void percpu_ref_resurrect (struct percpu_ref *ref);void percpu_ref_reinit (struct percpu_ref *ref);bool percpu_ref_is_zero (struct percpu_ref *ref); +void percpu_ref_swap_percpu_sync (struct percpu_ref *ref1,struct percpu_ref *ref2); +void percpu_ref_transfer_percpu_count (struct percpu_ref *ref1,struct percpu_ref *ref2);/** * percpu_ref_kill - drop the initial ref
lib/percpu-refcount.c +93 -0
96 lines not on this page · raw message
security/apparmor/lsm.c +90 -11
138 lines not on this page · raw message
Signature · 1 line
2.34.1
RFC 7/9 percpu-rcuref: Add basic infrastructure
In reply to Neeraj UpadhyayAdd infrastructure for managing reclaims of percpu refs, which use RCU grace period before reclaiming the referenced objects.
The refcount management of percpu rcuref is same as normal percpu refcount, with the only difference that, instead of a explicit shutdown percpu_ref_kill() operation by the user, the initial ref is managed by a kworker.
The ref can be initialized to start either in managed or unmanaged mode. In managed mode, the ref is a set of percpu counters. There is an extra reference acquired for the llist node and provides the notion of initial ref in percpu refcount.
During normal operation, users ref get() and put() operations increment/decrement the percpu counters. There is no check for drop-to-zero while in percpu mode.
Periodically, the manager kworker thread scans all percpu rcurefs. It switches ref to centralized atomic counter mode and checks whether the object has no references left. The ref is dropped if there are no references. Otherwise, the ref is switched back to percpu mode again. During this ref scan, there is a window where ref operates in atomic mode. This window spans one RCU grace period.
There is a provision to start a percpu rcuref in unmanaged mode. This is provided for cases, where there is a need to avoid dependency on kworker and RCU grace period. In addition, unmanaged mode can be used for a ref, for which the release function initially does not wait for RCU grace period, for example when the enclosing object initialization fails, and there is a rollback operation in error paths. Later, when object initialization is complete, ref can be switched to percpu managed mode.
--- .../admin-guide/kernel-parameters.txt | 8 + include/linux/percpu-rcurefcount.h | 115 ++++++ lib/Makefile | 2 +- lib/percpu-rcurefcount.c | 336 ++++++++++++++++++ 4 files changed, 460 insertions(+), 1 deletion(-) create mode 100644 include/linux/percpu-rcurefcount.h create mode 100644 lib/percpu-rcurefcount.c
Documentation/admin-guide/kernel-parameters.txt +8 -0
@@ -4576,6 +4576,14 @@ allocator. This parameter is primarily for debugging and performance comparison. + percpu-rcurefcount.ref_scan_interval= [KNL] + Interval (in ms) between 2 scans of percpu rcu ref + managed refs. + + percpu-rcurefcount.max_ref_scan_count= [KNL] + Count of the maximum number of pcpu refs scanned during + one scan of managed refs. + pirq= [SMP,APIC] Manual mp-table setup See Documentation/arch/x86/i386/IO-APIC.rst.
include/linux/percpu-rcurefcount.h new file +115 -0
115 lines not on this page · raw message
lib/Makefile +1 -1
@@ -46,7 +46,7 @@ obj-y += bcd.o sort.o parser.o debug_locks.o random32.o \ bust_spinlocks.o kasprintf.o bitmap.o scatterlist.o \ list_sort.o uuid.o iov_iter.o clz_ctz.o \ bsearch.o find_bit.o llist.o lwq.o memweight.o kfifo.o \ - percpu-refcount.o rhashtable.o base64.o \ + percpu-refcount.o percpu-rcurefcount.o rhashtable.o base64.o \ once.o refcount.o rcuref.o usercopy.o errseq.o bucket_locks.o \ generic-radix-tree.o bitmap-str.o obj-$(CONFIG_STRING_SELFTEST) += test_string.o
lib/percpu-rcurefcount.c new file +336 -0
336 lines not on this page · raw message
Signature · 1 line
2.34.1
kernel test robot
build botRe: [RFC 7/9] percpu-rcuref: Add basic infrastructure
kernel test robot noticed the following build warnings:
Hi Neeraj,
[This is a private test report for your RFC patch.] kernel test robot noticed the following build warnings:
[auto build test WARNING on ab27740f76654ed58dd32ac0ba0031c18a6dea3b]
url: https://github.com/intel-lab-lkp/linux/commits/Neeraj-Upadhyay/doc-Add-document-for-apparmor-refcount-management/20240110-192850 base: ab27740f76654ed58dd32ac0ba0031c18a6dea3b patch link: https://lore.kernel.org/r/20240110111856.87370-7-Neeraj.Upadhyay%40amd.com patch subject: [RFC 7/9] percpu-rcuref: Add basic infrastructure config: i386-buildonly-randconfig-003-20240113 (https://download.01.org/0day-ci/archive/20240114/[email protected]/config) compiler: gcc-12 (Debian 12.2.0-14) 12.2.0 reproduce (this is a W=1 build): (https://download.01.org/0day-ci/archive/20240114/[email protected]/reproduce)
If you fix the issue in a separate patch/commit (i.e. not just a new version of the same patch/commit), kindly add following tags
| Reported-by: kernel test robot <[email protected]>
| Closes: https://lore.kernel.org/oe-kbuild-all/[email protected]/
All warnings (new ones prefixed by >>):
>> lib/percpu-rcurefcount.c:112: warning: Excess function parameter 'release' description in 'percpu_rcuref_manage' >> lib/percpu-rcurefcount.c:112: warning: Excess function parameter 'gfp' description in 'percpu_rcuref_manage'
vim +112 lib/percpu-rcurefcount.c
102
103 /**
104 * percpu_rcuref_manage - Switch an unmanaged ref to percpu mode.
105 *
106 * @rcuref: percpu_rcuref to initialize
107 * @release: function which will be called when refcount hits 0
108 * @gfp: allocation mask to use
109 *
110 */
111 int percpu_rcuref_manage(struct percpu_rcuref *rcuref)
> 112 { 113 if (WARN_ONCE(!percpu_rcuref_tryget(rcuref), "Percpu rcuref is not active\n"))
114 return -1;
115 if (WARN_ONCE(llist_on_list(&rcuref->node), "Percpu rcuref already managed\n")) {
116 percpu_rcuref_put(rcuref);
117 return -2;
118 }
119 percpu_ref_switch_to_percpu(&rcuref->pcpu_ref);
120 /* Ensure ordering of percpu mode switch and node scan */
121 smp_mb();
122 llist_add(&rcuref->node, &pcpu_rcuref_head);
123 return 0;
124 }
125 EXPORT_SYMBOL_GPL(percpu_rcuref_manage);
126
Signature · 2 lines
0-DAY CI Kernel Test Service https://github.com/intel/lkp-tests/wiki
RFC 8/9 apparmor: Switch labels to percpu rcurefcount in unmanaged mode
In reply to Neeraj UpadhyayReplaces label kref with percpu rcurefcount.
The percpu rcuref is initialized in unmanaged/atomic mode, as labels do not use RCU grace period based release for labels which do not have a namespace associated with them yet. Subsequent patch moves the managed/percpu mode, at points where rcu grace period based cleanup is guaranteed.
--- include/linux/percpu-refcount.h | 2 - lib/percpu-refcount.c | 93 ----------- security/apparmor/include/label.h | 14 +- security/apparmor/include/policy.h | 32 +--- security/apparmor/include/policy_ns.h | 24 --- security/apparmor/label.c | 8 +- security/apparmor/lsm.c | 224 -------------------------- security/apparmor/policy_ns.c | 6 +- 8 files changed, 15 insertions(+), 388 deletions(-)
include/linux/percpu-refcount.h +0 -2
@@ -131,8 +131,6 @@ void percpu_ref_kill_and_confirm(struct percpu_ref *ref,void percpu_ref_resurrect (struct percpu_ref *ref);void percpu_ref_reinit (struct percpu_ref *ref);bool percpu_ref_is_zero (struct percpu_ref *ref); -void percpu_ref_swap_percpu_sync (struct percpu_ref *ref1,struct percpu_ref *ref2); -void percpu_ref_transfer_percpu_count (struct percpu_ref *ref1,struct percpu_ref *ref2);/** * percpu_ref_kill - drop the initial ref
lib/percpu-refcount.c +0 -93
96 lines not on this page · raw message
security/apparmor/include/label.h +6 -8
@@ -14,6 +14,7 @@#include <linux/audit.h> #include <linux/rbtree.h> #include <linux/rcupdate.h> +#include <linux/percpu-rcurefcount.h> #include "apparmor.h" #include "lib.h"
@@ -121,11 +122,10 @@ struct label_it {* @ent: set of profiles for label, actual size determined by @size */ struct aa_label { -struct percpu_ref count; +struct percpu_rcuref count;long flags;struct aa_proxy *proxy;struct rb_node node; -struct llist_node reclaim_node;struct rcu_head rcu; __counted char *hname;u32 secid;
@@ -374,7 +374,7 @@ int aa_label_match(struct aa_profile *profile, struct aa_ruleset *rules,*/ static inline struct aa_label *__aa_get_label (struct aa_label *l) { -if (l &&percpu_ref_tryget (&l->count)) +if (l &&percpu_rcuref_tryget (&l->count))return l;return NULL ;
@@ -383,7 +383,7 @@ static inline struct aa_label *__aa_get_label(struct aa_label *l)static inline struct aa_label *aa_get_label (struct aa_label *l) {if (l) -percpu_ref_get (&(l->count)); +percpu_rcuref_get (&(l->count));return l; }
@@ -403,7 +403,7 @@ static inline struct aa_label *aa_get_label_rcu(struct aa_label __rcu **l)rcu_read_lock ();do { c =rcu_dereference (*l); - }while (c && !percpu_ref_tryget (&c->count)); + }while (c && !percpu_rcuref_tryget (&c->count));rcu_read_unlock ();return c;
@@ -443,7 +443,7 @@ static inline struct aa_label *aa_get_newest_label(struct aa_label *l)static inline void aa_put_label (struct aa_label *l) {if (l) -percpu_ref_put (&l->count); +percpu_rcuref_put (&l->count); }
@@ -466,6 +466,4 @@ static inline void aa_put_proxy(struct aa_proxy *proxy)void __aa_proxy_redirect (struct aa_label *orig,struct aa_label *new); -void aa_label_reclaim_add_label (struct aa_label *label); -#endif /* __AA_LABEL_H */
security/apparmor/include/policy.h +4 -28
@@ -329,7 +329,7 @@ static inline aa_state_t ANY_RULE_MEDIATES(struct list_head *head,static inline struct aa_profile *aa_get_profile (struct aa_profile *p) {if (p) -percpu_ref_get (&(p->label.count)); +percpu_rcuref_get (&(p->label.count));return p; }
@@ -343,7 +343,7 @@ static inline struct aa_profile *aa_get_profile(struct aa_profile *p)*/ static inline struct aa_profile *aa_get_profile_not0 (struct aa_profile *p) { -if (p &&percpu_ref_tryget (&p->label.count)) +if (p &&percpu_rcuref_tryget (&p->label.count))return p;return NULL ;
@@ -363,7 +363,7 @@ static inline struct aa_profile *aa_get_profile_rcu(struct aa_profile __rcu **p)rcu_read_lock ();do { c =rcu_dereference (*p); - }while (c && !percpu_ref_tryget (&c->label.count)); + }while (c && !percpu_rcuref_tryget (&c->label.count));rcu_read_unlock ();return c;
@@ -376,31 +376,7 @@ static inline struct aa_profile *aa_get_profile_rcu(struct aa_profile __rcu **p)static inline void aa_put_profile (struct aa_profile *p) {if (p) -percpu_ref_put (&p->label.count); -} - -/** -* aa_switch_ref_profile - switch percpu-ref mode for profile @p -* @p: profile (MAYBE NULL) -*/ -static inline void aa_switch_ref_profile (struct aa_profile *p,bool percpu) -{ -if (p) { -if (percpu) -percpu_ref_switch_to_percpu (&p->label.count); -else -percpu_ref_switch_to_atomic_sync (&p->label.count); - } -} - -/** -* aa_kill_ref_profile - percpu-ref kill for profile @p -* @p: profile (MAYBE NULL) -*/ -static inline void aa_kill_ref_profile (struct aa_profile *p) -{ -if (p) -percpu_ref_kill (&p->label.count); +percpu_rcuref_put (&p->label.count); }static inline int AUDIT_MODE (struct aa_profile *profile)
security/apparmor/include/policy_ns.h +0 -24
@@ -127,30 +127,6 @@ static inline void aa_put_ns(struct aa_ns *ns)aa_put_profile (ns->unconfined); } -/** -* aa_switch_ref_ns - switch percpu-ref mode for @ns -* @ns: namespace to switch percpu-ref mode of -* -* Switch percpu-ref mode of @ns between percpu and atomic -*/ -static inline void aa_switch_ref_ns (struct aa_ns *ns,bool percpu) -{ -if (ns) -aa_switch_ref_profile (ns->unconfined, percpu); -} - -/** -* aa_kill_ref_ns - do percpu-ref kill for @ns -* @ns: namespace to perform percpu-ref kill for -* -* Do percpu-ref kill of @ns refcount -*/ -static inline void aa_kill_ref_ns (struct aa_ns *ns) -{ -if (ns) -aa_kill_ref_profile (ns->unconfined); -} -/** * __aa_findn_ns - find a namespace on a list by @name * @head: list to search for namespace on (NOT NULL)
security/apparmor/label.c +3 -5
@@ -336,7 +336,7 @@ void aa_label_destroy(struct aa_label *label)rcu_assign_pointer (label->proxy->label,NULL );aa_put_proxy (label->proxy); } -percpu_ref_exit (&label->count); +percpu_rcuref_exit (&label->count);aa_free_secid (label->secid); label->proxy = (struct aa_proxy *)PROXY_POISON +1 ;
@@ -372,7 +372,7 @@ static void label_free_rcu(struct rcu_head *head)void aa_label_percpu_ref (struct percpu_ref *ref) { - struct aa_label *label = container_of(ref, struct aa_label, count); + struct aa_label *label = container_of(ref, struct aa_label, count.pcpu_ref);struct aa_ns *ns =labels_ns (label);if (!ns) {
@@ -409,7 +409,7 @@ bool aa_label_init(struct aa_label *label, int size, gfp_t gfp) label->size = size;/* doesn't include null */ label->vec[size] =NULL ;/* null terminate */ -if (percpu_ref_init (&label->count, aa_label_percpu_ref,PERCPU_REF_INIT_ATOMIC , gfp)) { +if (percpu_rcuref_init_unmanaged (&label->count, aa_label_percpu_ref, gfp)) {aa_free_secid (label->secid);return false; }
@@ -710,8 +710,6 @@ static struct aa_label *__label_insert(struct aa_labelset *ls,rb_link_node (&label->node, parent, new);rb_insert_color (&label->node, &ls->root); label->flags |=FLAG_IN_TREE ; -percpu_ref_switch_to_percpu (&label->count); -aa_label_reclaim_add_label (label);return aa_get_label (label); }
security/apparmor/lsm.c +0 -224
242 lines not on this page · raw message
security/apparmor/policy_ns.c +2 -4
@@ -124,7 +124,6 @@ static struct aa_ns *alloc_ns(const char *prefix, const char *name) goto fail_unconfined;/* ns and ns->unconfined share ns->unconfined refcount */ ns->unconfined->ns = ns; -aa_switch_ref_ns (ns, true);atomic_set (&ns->uniq_null,0 );
@@ -337,7 +336,7 @@ void __aa_remove_ns(struct aa_ns *ns)/* remove ns from namespace list */ list_del_rcu (&ns->base.list);destroy_ns (ns); -aa_kill_ref_ns (ns); +aa_put_ns (ns); }/**
@@ -378,7 +377,6 @@ int __init aa_alloc_root_ns(void) } kernel_t = &kernel_p->label; root_ns->unconfined->ns =aa_get_ns (root_ns); -aa_switch_ref_ns (root_ns, true);return 0 ; }
@@ -394,5 +392,5 @@ void __init aa_free_root_ns(void)aa_label_free (kernel_t);destroy_ns (ns); -aa_kill_ref_ns (ns); +aa_put_ns (ns); }
Signature · 1 line
2.34.1
RFC 9/9 apparmor: Switch unconfined and in tree labels to managed ref mode
In reply to Neeraj UpadhyaySwitch unconfined and in-tree labels to percpu managed mode of percpu rcuref. This helps avoid memory contention in ref get and put operations.
--- security/apparmor/label.c | 1 + security/apparmor/policy_ns.c | 2 ++ 2 files changed, 3 insertions(+)
security/apparmor/label.c +1 -0
@@ -710,6 +710,7 @@ static struct aa_label *__label_insert(struct aa_labelset *ls,rb_link_node (&label->node, parent, new);rb_insert_color (&label->node, &ls->root); label->flags |=FLAG_IN_TREE ; +percpu_rcuref_manage (&label->count);return aa_get_label (label); }
security/apparmor/policy_ns.c +2 -0
@@ -124,6 +124,7 @@ static struct aa_ns *alloc_ns(const char *prefix, const char *name) goto fail_unconfined;/* ns and ns->unconfined share ns->unconfined refcount */ ns->unconfined->ns = ns; +percpu_rcuref_manage (&ns->unconfined->label.count);atomic_set (&ns->uniq_null,0 );
@@ -377,6 +378,7 @@ int __init aa_alloc_root_ns(void) } kernel_t = &kernel_p->label; root_ns->unconfined->ns =aa_get_ns (root_ns); +percpu_rcuref_manage (&root_ns->unconfined->label.count);return 0 ; }
Signature · 1 line
2.34.1
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Thanks
Neeraj
Quoted 412 lines · Neeraj Upadhyay
Problem Statement
=================Nginx performance testing with Apparmor enabled (with nginx running in unconfined profile), on kernel versions 6.1 and 6.5 show significant drop in throughput scalability, when Nginx workers are scaled to higher number of CPUs across various L3 cache domains.
Below is one sample data on the throughput scalability loss, based on results on AMD Zen4 system with 96 CPUs with SMT core count 2; so, overall, 192 CPUs:
Config Cache Domains apparmor=off apparmor=on
scaling eff (%) scaling eff (%)8C16T 1 100% 100% 16C32T 2 95% 94% 24C48T 3 94% 93% 48C96T 6 92% 88% 96C192T 12 85% 68%If we look at above data, there is a significant drop in scaling efficiency, when we move to 96 CPUs/192 SMT threads.
Perf tool shows most of the contention coming from below
6.56% nginx [kernel.vmlinux] [k] apparmor_current_getsecid_subj 6.22% nginx [kernel.vmlinux] [k] apparmor_file_openThe majority of the CPU cycles is found to be due to memory contention in
atomic_fetch_addandatomic_fetch_suboperations fromkref_get()andkref_put()operations on label.Commit
2516fde1fa00("apparmor: Optimize retrieving current task secid"), from 6.7 alleviates the issue to an extent, but not completely:Config Cache Domains apparmor=on apparmor=on (patched)
scaling eff (%) scaling eff (%)8C16T 1 100% 100% 16C32T 2 97% 93% 24C48T 3 94% 92% 48C96T 6 88% 88% 96C192T 12 65% 79%This adverse impact gets more pronounced when we move to >192 CPUs. The memory contention and impact increases with high frequency label update operations and labels are marked stale more frequently.
Label Refcount Management
=========================Apparmor uses label objects (struct
aa_label) to manage refcounts for below set of objects:- Applicable profiles
- Namespaces (unconfined profile)
- Other non-profile referencesThese label references are acquired on various apparmor lsm hooks, on operations such as file open, task kill operations, socket bind, and other file, socket, misc operations which use current task's cred, when the label for the current cred, has been marked stale. This is done to check these operations against the set of allowed operations for the task performing them.
Use Percpu refcount for ref management?
=======================================The ref put operations (
percpu_ref_put()) in percpu refcount, in active mode, do not check whether ref count has dropped to
0. The users of thepercpu_refneed to explicitly invoke apercpu_ref_kill()operation, to drop the initial reference, at shutdown paths. After thepercpu_ref_kill()operation, ref switches to atomic mode and any newpercpu_ref_put()operation checks for the drop to 0 case and invokes the release operation on that label.Labels are marked stale is situations like profile removal, profile updates. For a namespace, the unconfined label reference is dropped when the namespace is destroyed. These points are potential shutdown points for labels. However, killing the percpu ref from these points has few issues:
- The label could still be referenced by tasks, which are still holding the reference to the now stale label. Killing the label ref while these operations are in progress will make all subsequent ref-put operations on the stale label to be atomic, which would still result in memory contention. Also, any new reference to the stale label, which is acquired with the elevated refcount will have atomic op contention.
- The label is marked stale using a non-atomic write operation. It is possible that new operations do not observe this flag and still reference it for quite some time.
- Explicitly tracking the shutdown points might not be maintainable at a per label granularity, as there can be various paths where label reference could get dropped, such as, before the label has gone live - object initialization error paths. Also, tracking the shutdown points for labels which reference other labels - subprofiles, merged labels requires careful analysis, and adds heavy burden on ensuring the memory contention is not introduced by these ref kill points.
Proposed Solution
=================One potential solution to the refcount scalability problem is to convert the label refcount to a percpu refcount, and manage the initial reference from kworker context. The kworker keeps an extra reference to the label and periodically scans labels and release them if their refcount drops to 0.
Below is the sequence of operations, which shows the refcount management with this approach:
1. During label initialization, the percpu ref is initialized in atomic mode. This is done to ensure that, for cases where the label hasn't gone live (->ns isn't assigned), mostly during initialization error paths.
2. Labels are switched to percpu mode at various points - when a label is added to labelset tree, when a unconfined profile has been assigned a namespace.
3. As part of the initial prototype, only the in tree labels are managed by the kworker. These labels are added to a lockless list. The unconfined labels invoke a
percpu_ref_kill()operation when the namespace is destroyed.4. The kworker does a periodic scan of all the labels in the llist. It does below sequence of operations:
a. Enqueue a dummy node to mark the start of scan. This dummy
node is used as start point of scan and ensures that we there is no additional synchronization required with new label node additions to the llist. Any new labels will be processed in next run of the kworker.SCAN START PTR
|
v
+----------+ +------+ +------+ +------+
| | | | | | | |
| head ------> dummy|--->|label |--->| label|--->NULL
| | | node | | | | |
+----------+ +------+ +------+ +------+New label addition:
SCAN START PTR
|
v
+----------+ +------+ +------+ +------+ +------+
| | | | | | | | | |
| head |--> label|--> dummy|--->|label |--->| label|--->NULL
| | | | | node | | | | |
+----------+ +------+ +------+ +------+ +------+b. Traverse through the llist, starting from
dummy->next.
If the node is a dummy node, mark it free. If the node is a label node, do,i) Switch the label ref to atomic mode. The ref switch wait for the existing
percpu_ref_get()andpercpu_ref_put()operations to complete, by waiting for a RCU grace period.Once the switch is complete, from this point onwards, any
percpu_ref_get(),percpu_ref_put()operations use atomic operations.ii) Drop the initial reference, which was taken while adding
the label node to the llist.iii) Use a
percpu_ref_tryget()increment operation on the
ref, to see if we dropped the last ref count. if we dropped the last count, we remove the node from the llist.All of these operations are done inside a RCU critical section, to avoid race with the release operations, which can potentially trigger, as soon as we drop the initial ref count.
iv) If we didn't drop the last ref, switch back the counter
to percpu mode.Using this approach, to move the atomic refcount manipulation out of the contended paths, there is a significant scalability improvement seen on nginx test, and scalability efficiency is close to apparmor-off case.
Config Cache Domains apparmor=on (percpuref)
scaling eff (%)8C16T 1 100% 16C32T 2 96% 24C48T 3 94% 48C96T 6 93% 96C192T 12 90%Limitations
===========1. Switching to percpu refcount increases memory size overhead, as percpu memory is allocated for all labels.
2. Deferring labels reclaim could potentially result in memory pressure, when there are high frequency of label update operations.
3. Percpu refcount uses
call_rcu_hurry()to complete switch operations. These can impact energy efficiency, due to back to back hurry callbacks. Using deferrable workqueue partly mitigates this. However, deferring kworker can delay reclaims.4. Back to back label switches can delay other percpu users, as there is a single global switch spinlock used by percpu refcount lib.
5. Long running kworker can delay other use cases like system suspend. This is mitigated using freezable workqueue and litming node scans to a max count.
6. There is a window where label operates is atomic mode, when its counter is being checked for zero. This can potentially result in high memory contention, during this window which spans RCU grace period (plus callback execution). For example, when scanning label corresponding to unconfined profile, all applications which use unconfined profile would be using atomic ref increment and decrement operations.
There are a few apparoaches which were tried to mitigate this issue:
a. At a lower time interval, check if scanned label's counter
has changed since the start of label scan. If there is a change in count, terminate the switch to atomic mode. Below shows the apparoch using rcuwait. static void aa_label_switch_atomic_confirm(struct percpu_ref *label_ref) { WRITE_ONCE(aa_atomic_switch_complete, true); rcuwait_wake_up(&aa_reclaim_rcuwait); } rcuwait_init(&aa_reclaim_rcuwait); percpu_ref_switch_to_atomic(&label->count, aa_label_switch_atomic_confirm); atomic_count = percpu_ref_count_read(&label->count); do { rcuwait_wait_event_timeout(&aa_reclaim_rcuwait, (switch_complete = READ_ONCE(aa_atomic_switch_complete)), TASK_IDLE, msecs_to_jiffies(5)); if (percpu_ref_count_read(&label->count) != atomic_count) break; } while (!READ_ONCE(switch_complete)); However, this approach does not work, as percpu refcount lib does not allow termination of an ongoing switch operation. Also, the counter can return to the original value with set of get() and put() operations before we check the current value.b. Approaches to notify the reclaim kworker from ref get and put operations
can potentially disturb cache line state between the various CPU contexts, which are referncing the label, and can potentially impact scalability again.c. Swith the label to an immortal percpu ref, while the scan operates
on the current counter.Below is the sequence of operations to do this:
1. Ensure that both immortal ref and label ref are in percpu mode. Reinit the immortal ref in percpu mode.
Swap percpu and atomic counters of label refcount and immortal ref
percpu-ref
+-------------------+
+-------+ | percpu-ctr-addr1 |
| label | --------->|-------------------| +----------------+
+-------+ | data |--->| Atomic counter1|
+-------------------+ +----------------+
+-------+ +-------------------+
|ImmLbl |---------->| percpu-ctr-addr2 | +----------------+
+-------+ |-------------------|--->| Atomic counter2|
| data | +----------------+
+-------------------+label ->percpu-ctr-addr = percpu-ctr-addr2 ImmLbl ->percpu-ctr-addr = percpu-ctr-addr1
label ->data->count = Atomic counter2
ImmLbl ->data->count = Atomic counter12. Check the counters collected in immortal label, by switch it to atomic mode.
3. If the count is 0, do,
a. Switch immortal counter to percpu again, giving it an
initial count of 1.
b. Swap the label and immortal counters again. The immortal
ref now has the counter values from new percpu ref get and get operations on the label ref, from the point when we did the initial swap operation.
c. Transfer the percpu counts in immortal ref to atomic
counter of label percpu refcount.
d. Kill immortal ref, for reinit on next iteration. e. Switch label percpu ref to atomic mode.
f. If the counter is 1, drop the initial ref.4. If the count is not 0, re-swap the counters.
a. Switch immortal counter to percpu again, giving it an
initial count of 1.
b. Swap the label and immortal counters again. The immortal
ref now has the counter values from new percpu ref get and get operations on the label ref, from the point when we did the initial swap operation.
c. Transfer the percpu counts in immortal ref to atomic
counter of label percpu refcount.
d. Kill immortal ref, for reinit on next iteration.Using this approach, we ensure that, label ref users do not switch to atomic mode, while there are active references on the label. However, this approach requires multiple percpu ref mode switches and adds high overhead and complexity to the scanning code.
Extended/Future Work
====================1. Look for ways to fix the limitations, as described in the "Limitations" section.
2. Generalize the approach to percpu rcuref, which is used for contexts where release path uses RCU grace period for release operations. Patch 7 creates an initial prototype for this.
3. Explore hazard pointers for scalable refcounting of labels.
Highly appreciate any feedback/suggestions on the design approach.
The patches of this patchset introduce following changes:
1. Documentation of Apparmor Refcount management.
2. Switch labels to percpu refcount in atomic mode.
Use percpu refcount for apparmor labels. Initial patch to init the percpu ref in atomic mode, to evaluate the potential impact of percpuref on top of kref based implementation.
3. Switch unconfined namespaces refcount to percpu mode.
Switch unconfined ns labels to percpu mode, and kill the initial refcount from namespace destroy path.
4. Add infrastructure to reclaim percpu labels.
Add a label reclaim infrastructure for labels which are in percpu mode, for managing their inital refcount.
5. Switch intree labels to percpu mode.
Use label reclaim infrastruture to manage intree labels.
6. Initial prototype for optimizing ref switch.
Prototype for reducing the time window when a label scan switches the label ref to atomic mode.
7. percpu-rcuref: Add basic infrastructure.
Prototype for Percpu refcounts for objects, which protect their object reclaims using RCU grace period.
8. Switch labels to percpu rcurefcount in unmanaged mode.
Use percpu rcuref for labels. Start with unmanaged/atomic mode.
9. Switch unconfined and in tree labels to managed ref mode.
Use percpu mode with manager worker for unconfined and intree labels.
------------------------------------------------------------------------
b/Documentation/admin-guide/LSM/ApparmorRefcount.rst | 351 ++++++++++++++++++++++++++++++++++++++++++++++++++ b/Documentation/admin-guide/LSM/index.rst | 1 b/Documentation/admin-guide/kernel-parameters.txt | 8 + b/include/linux/percpu-rcurefcount.h | 115 ++++++++++++++++ b/include/linux/percpu-refcount.h | 2 b/lib/Makefile | 2 b/lib/percpu-rcurefcount.c | 336 +++++++++++++++++++++++++++++++++++++++++++++++ b/lib/percpu-refcount.c | 93 +++++++++++++ b/security/apparmor/include/label.h | 16 +- b/security/apparmor/include/policy.h | 8 - b/security/apparmor/include/policy_ns.h | 24 +++ b/security/apparmor/label.c | 11 + b/security/apparmor/lsm.c | 145 ++++++++++++++++++++ b/security/apparmor/policy_ns.c | 6 include/linux/percpu-refcount.h | 2 lib/percpu-refcount.c | 93 ------------- security/apparmor/include/label.h | 17 +- security/apparmor/include/policy.h | 56 +++---- security/apparmor/include/policy_ns.h | 24 --- security/apparmor/label.c | 11 - security/apparmor/lsm.c | 325 ++++++++++++---------------------------------- security/apparmor/policy_ns.c | 8 - 22 files changed, 1237 insertions(+), 417 deletions(-)base-commit:
ab27740f7665
John Johansen
maintainerQuoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
john
Quoted 414 lines · Neeraj Upadhyay
Quoted 412 lines
Problem Statement
=================Nginx performance testing with Apparmor enabled (with nginx running in unconfined profile), on kernel versions 6.1 and 6.5 show significant drop in throughput scalability, when Nginx workers are scaled to higher number of CPUs across various L3 cache domains.
Below is one sample data on the throughput scalability loss, based on results on AMD Zen4 system with 96 CPUs with SMT core count 2; so, overall, 192 CPUs:
Config Cache Domains apparmor=off apparmor=on
scaling eff (%) scaling eff (%)8C16T 1 100% 100% 16C32T 2 95% 94% 24C48T 3 94% 93% 48C96T 6 92% 88% 96C192T 12 85% 68%If we look at above data, there is a significant drop in scaling efficiency, when we move to 96 CPUs/192 SMT threads.
Perf tool shows most of the contention coming from below
6.56% nginx [kernel.vmlinux] [k] apparmor_current_getsecid_subj 6.22% nginx [kernel.vmlinux] [k] apparmor_file_openThe majority of the CPU cycles is found to be due to memory contention in
atomic_fetch_addandatomic_fetch_suboperations fromkref_get()andkref_put()operations on label.Commit
2516fde1fa00("apparmor: Optimize retrieving current task secid"), from 6.7 alleviates the issue to an extent, but not completely:Config Cache Domains apparmor=on apparmor=on (patched)
scaling eff (%) scaling eff (%)8C16T 1 100% 100% 16C32T 2 97% 93% 24C48T 3 94% 92% 48C96T 6 88% 88% 96C192T 12 65% 79%This adverse impact gets more pronounced when we move to >192 CPUs. The memory contention and impact increases with high frequency label update operations and labels are marked stale more frequently.
Label Refcount Management
=========================Apparmor uses label objects (struct
aa_label) to manage refcounts for below set of objects:- Applicable profiles
- Namespaces (unconfined profile)
- Other non-profile referencesThese label references are acquired on various apparmor lsm hooks, on operations such as file open, task kill operations, socket bind, and other file, socket, misc operations which use current task's cred, when the label for the current cred, has been marked stale. This is done to check these operations against the set of allowed operations for the task performing them.
Use Percpu refcount for ref management?
=======================================The ref put operations (
percpu_ref_put()) in percpu refcount, in active mode, do not check whether ref count has dropped to
0. The users of thepercpu_refneed to explicitly invoke apercpu_ref_kill()operation, to drop the initial reference, at shutdown paths. After thepercpu_ref_kill()operation, ref switches to atomic mode and any newpercpu_ref_put()operation checks for the drop to 0 case and invokes the release operation on that label.Labels are marked stale is situations like profile removal, profile updates. For a namespace, the unconfined label reference is dropped when the namespace is destroyed. These points are potential shutdown points for labels. However, killing the percpu ref from these points has few issues:
- The label could still be referenced by tasks, which are still holding the reference to the now stale label. Killing the label ref while these operations are in progress will make all subsequent ref-put operations on the stale label to be atomic, which would still result in memory contention. Also, any new reference to the stale label, which is acquired with the elevated refcount will have atomic op contention.
- The label is marked stale using a non-atomic write operation. It is possible that new operations do not observe this flag and still reference it for quite some time.
- Explicitly tracking the shutdown points might not be maintainable at a per label granularity, as there can be various paths where label reference could get dropped, such as, before the label has gone live - object initialization error paths. Also, tracking the shutdown points for labels which reference other labels - subprofiles, merged labels requires careful analysis, and adds heavy burden on ensuring the memory contention is not introduced by these ref kill points.
Proposed Solution
=================One potential solution to the refcount scalability problem is to convert the label refcount to a percpu refcount, and manage the initial reference from kworker context. The kworker keeps an extra reference to the label and periodically scans labels and release them if their refcount drops to 0.
Below is the sequence of operations, which shows the refcount management with this approach:
1. During label initialization, the percpu ref is initialized in atomic mode. This is done to ensure that, for cases where the label hasn't gone live (->ns isn't assigned), mostly during initialization error paths.
2. Labels are switched to percpu mode at various points - when a label is added to labelset tree, when a unconfined profile has been assigned a namespace.
3. As part of the initial prototype, only the in tree labels are managed by the kworker. These labels are added to a lockless list. The unconfined labels invoke a
percpu_ref_kill()operation when the namespace is destroyed.4. The kworker does a periodic scan of all the labels in the llist. It does below sequence of operations:
a. Enqueue a dummy node to mark the start of scan. This dummy
node is used as start point of scan and ensures that we there is no additional synchronization required with new label node additions to the llist. Any new labels will be processed in next run of the kworker.SCAN START PTR
|
v
+----------+ +------+ +------+ +------+
| | | | | | | |
| head ------> dummy|--->|label |--->| label|--->NULL
| | | node | | | | |
+----------+ +------+ +------+ +------+New label addition:
SCAN START PTR
|
v
+----------+ +------+ +------+ +------+ +------+
| | | | | | | | | |
| head |--> label|--> dummy|--->|label |--->| label|--->NULL
| | | | | node | | | | |
+----------+ +------+ +------+ +------+ +------+b. Traverse through the llist, starting from
dummy->next.
If the node is a dummy node, mark it free. If the node is a label node, do,i) Switch the label ref to atomic mode. The ref switch wait for the existing
percpu_ref_get()andpercpu_ref_put()operations to complete, by waiting for a RCU grace period.Once the switch is complete, from this point onwards, any
percpu_ref_get(),percpu_ref_put()operations use atomic operations.ii) Drop the initial reference, which was taken while adding
the label node to the llist.iii) Use a
percpu_ref_tryget()increment operation on the
ref, to see if we dropped the last ref count. if we dropped the last count, we remove the node from the llist.All of these operations are done inside a RCU critical section, to avoid race with the release operations, which can potentially trigger, as soon as we drop the initial ref count.
iv) If we didn't drop the last ref, switch back the counter
to percpu mode.Using this approach, to move the atomic refcount manipulation out of the contended paths, there is a significant scalability improvement seen on nginx test, and scalability efficiency is close to apparmor-off case.
Config Cache Domains apparmor=on (percpuref)
scaling eff (%)8C16T 1 100% 16C32T 2 96% 24C48T 3 94% 48C96T 6 93% 96C192T 12 90%Limitations
===========1. Switching to percpu refcount increases memory size overhead, as percpu memory is allocated for all labels.
2. Deferring labels reclaim could potentially result in memory pressure, when there are high frequency of label update operations.
3. Percpu refcount uses
call_rcu_hurry()to complete switch operations. These can impact energy efficiency, due to back to back hurry callbacks. Using deferrable workqueue partly mitigates this. However, deferring kworker can delay reclaims.4. Back to back label switches can delay other percpu users, as there is a single global switch spinlock used by percpu refcount lib.
5. Long running kworker can delay other use cases like system suspend. This is mitigated using freezable workqueue and litming node scans to a max count.
6. There is a window where label operates is atomic mode, when its counter is being checked for zero. This can potentially result in high memory contention, during this window which spans RCU grace period (plus callback execution). For example, when scanning label corresponding to unconfined profile, all applications which use unconfined profile would be using atomic ref increment and decrement operations.
There are a few apparoaches which were tried to mitigate this issue:
a. At a lower time interval, check if scanned label's counter
has changed since the start of label scan. If there is a change in count, terminate the switch to atomic mode. Below shows the apparoch using rcuwait.static void
aa_label_switch_atomic_confirm(struct percpu_ref *label_ref){
WRITE_ONCE(aa_atomic_switch_complete, true);rcuwait_wake_up(&aa_reclaim_rcuwait);
}
rcuwait_init(&aa_reclaim_rcuwait);percpu_ref_switch_to_atomic(&label->count, aa_label_switch_atomic_confirm);
atomic_count=percpu_ref_count_read(&label->count);
do {
rcuwait_wait_event_timeout(&aa_reclaim_rcuwait,
(switch_complete=READ_ONCE(aa_atomic_switch_complete)),TASK_IDLE,
msecs_to_jiffies(5));
if (percpu_ref_count_read(&label->count)!=atomic_count)
break;
} while (!READ_ONCE(switch_complete));However, this approach does not work, as percpu refcount lib does not allow termination of an ongoing switch operation. Also, the counter can return to the original value with set of
get()andput()operations before we check the current value.b. Approaches to notify the reclaim kworker from ref get and put operations
can potentially disturb cache line state between the various CPU contexts, which are referncing the label, and can potentially impact scalability again.c. Swith the label to an immortal percpu ref, while the scan operates
on the current counter.Below is the sequence of operations to do this:
1. Ensure that both immortal ref and label ref are in percpu mode. Reinit the immortal ref in percpu mode.
Swap percpu and atomic counters of label refcount and immortal ref
percpu-ref
+-------------------+
+-------+ | percpu-ctr-addr1 |
| label | --------->|-------------------| +----------------+
+-------+ | data |--->| Atomic counter1|
+-------------------+ +----------------+
+-------+ +-------------------+
|ImmLbl |---------->| percpu-ctr-addr2 | +----------------+
+-------+ |-------------------|--->| Atomic counter2|
| data | +----------------+
+-------------------+label ->percpu-ctr-addr = percpu-ctr-addr2 ImmLbl ->percpu-ctr-addr = percpu-ctr-addr1
label ->data->count = Atomic counter2
ImmLbl ->data->count = Atomic counter12. Check the counters collected in immortal label, by switch it to atomic mode.
3. If the count is 0, do,
a. Switch immortal counter to percpu again, giving it an
initial count of 1.
b. Swap the label and immortal counters again. The immortal
ref now has the counter values from new percpu ref get and get operations on the label ref, from the point when we did the initial swap operation.
c. Transfer the percpu counts in immortal ref to atomic
counter of label percpu refcount.
d. Kill immortal ref, for reinit on next iteration. e. Switch label percpu ref to atomic mode.
f. If the counter is 1, drop the initial ref.4. If the count is not 0, re-swap the counters.
a. Switch immortal counter to percpu again, giving it an
initial count of 1.
b. Swap the label and immortal counters again. The immortal
ref now has the counter values from new percpu ref get and get operations on the label ref, from the point when we did the initial swap operation.
c. Transfer the percpu counts in immortal ref to atomic
counter of label percpu refcount.
d. Kill immortal ref, for reinit on next iteration.Using this approach, we ensure that, label ref users do not switch to atomic mode, while there are active references on the label. However, this approach requires multiple percpu ref mode switches and adds high overhead and complexity to the scanning code.
Extended/Future Work
====================1. Look for ways to fix the limitations, as described in the "Limitations" section.
2. Generalize the approach to percpu rcuref, which is used for contexts where release path uses RCU grace period for release operations. Patch 7 creates an initial prototype for this.
3. Explore hazard pointers for scalable refcounting of labels.
Highly appreciate any feedback/suggestions on the design approach.
The patches of this patchset introduce following changes:
1. Documentation of Apparmor Refcount management.
2. Switch labels to percpu refcount in atomic mode.
Use percpu refcount for apparmor labels. Initial patch to init the percpu ref in atomic mode, to evaluate the potential impact of percpuref on top of kref based implementation.
3. Switch unconfined namespaces refcount to percpu mode.
Switch unconfined ns labels to percpu mode, and kill the initial refcount from namespace destroy path.
4. Add infrastructure to reclaim percpu labels.
Add a label reclaim infrastructure for labels which are in percpu mode, for managing their inital refcount.
5. Switch intree labels to percpu mode.
Use label reclaim infrastruture to manage intree labels.
6. Initial prototype for optimizing ref switch.
Prototype for reducing the time window when a label scan switches the label ref to atomic mode.
7. percpu-rcuref: Add basic infrastructure.
Prototype for Percpu refcounts for objects, which protect their object reclaims using RCU grace period.
8. Switch labels to percpu rcurefcount in unmanaged mode.
Use percpu rcuref for labels. Start with unmanaged/atomic mode.
9. Switch unconfined and in tree labels to managed ref mode.
Use percpu mode with manager worker for unconfined and intree labels.
------------------------------------------------------------------------
b/Documentation/admin-guide/LSM/ApparmorRefcount.rst | 351 ++++++++++++++++++++++++++++++++++++++++++++++++++ b/Documentation/admin-guide/LSM/index.rst | 1 b/Documentation/admin-guide/kernel-parameters.txt | 8 + b/include/linux/percpu-rcurefcount.h | 115 ++++++++++++++++ b/include/linux/percpu-refcount.h | 2 b/lib/Makefile | 2 b/lib/percpu-rcurefcount.c | 336 +++++++++++++++++++++++++++++++++++++++++++++++ b/lib/percpu-refcount.c | 93 +++++++++++++ b/security/apparmor/include/label.h | 16 +- b/security/apparmor/include/policy.h | 8 - b/security/apparmor/include/policy_ns.h | 24 +++ b/security/apparmor/label.c | 11 + b/security/apparmor/lsm.c | 145 ++++++++++++++++++++ b/security/apparmor/policy_ns.c | 6 include/linux/percpu-refcount.h | 2 lib/percpu-refcount.c | 93 ------------- security/apparmor/include/label.h | 17 +- security/apparmor/include/policy.h | 56 +++---- security/apparmor/include/policy_ns.h | 24 --- security/apparmor/label.c | 11 - security/apparmor/lsm.c | 325 ++++++++++++---------------------------------- security/apparmor/policy_ns.c | 8 - 22 files changed, 1237 insertions(+), 417 deletions(-)base-commit:
ab27740f7665
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Gentle prod.
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
John Johansen
maintainerQuoted 20 lines · Mateusz Guzik
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Quoted 5 lines
Gentle prod.
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
yes, sorry I have been unavailable for the last couple of weeks. I am now back, I have a rather large backlog to try catching up on but this is has an entry on the list.
Quoted 24 lines · John Johansen
Quoted 22 lines · Mateusz Guzik
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Gentle prod.
Quoted 8 lines
Quoted 3 lines
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
yes, sorry I have been unavailable for the last couple of weeks. I am now back, I have a rather large backlog to try catching up on but this is has an entry on the list.
So where do we stand here?
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
John Johansen
maintainerQuoted 30 lines · Mateusz Guzik
Quoted 28 lines · John Johansen
Quoted 25 lines · Mateusz Guzik
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Gentle prod.
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
Quoted 7 lines
Quoted 4 lines
yes, sorry I have been unavailable for the last couple of weeks. I am now back, I have a rather large backlog to try catching up on but this is has an entry on the list.
So where do we stand here?
sorry I am still trying to dig out of my backlog, I will look at this, this weekend.
Quoted 37 lines · John Johansen
Quoted 35 lines · Mateusz Guzik
Quoted 32 lines · John Johansen
Quoted 20 lines · Mateusz Guzik
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Quoted 5 lines
Gentle prod.
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
yes, sorry I have been unavailable for the last couple of weeks. I am now back, I have a rather large backlog to try catching up on but this is has an entry on the list.
Quoted 5 lines
Quoted 2 lines
So where do we stand here?
sorry I am still trying to dig out of my backlog, I will look at this, this weekend.
How was the weekend? ;)
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
John Johansen
maintainerQuoted 39 lines · Mateusz Guzik
Quoted 37 lines · John Johansen
Quoted 35 lines · Mateusz Guzik
Quoted 32 lines · John Johansen
Quoted 20 lines · Mateusz Guzik
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Quoted 5 lines
Gentle prod.
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
yes, sorry I have been unavailable for the last couple of weeks. I am now back, I have a rather large backlog to try catching up on but this is has an entry on the list.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
So where do we stand here?
sorry I am still trying to dig out of my backlog, I will look at this, this weekend.
How was the weekend? ;)
lets say it was busy. Have I looked at this, yes. I am still digesting it. I don't have objections to moving towards percpu refcounts, but the overhead of a percpu stuct per label is a problem when we have thousands of labels on the system. That is to say, this would have to be a config option. We moved buffers from kmalloc to percpu to reduce memory overhead to reduce contention. The to percpu, to a global pool because the percpu overhead was too high for some machines, and then from a global pool to a hybrid scheme because of global lock contention. I don't see a way of doing that with the label, which means a config would be the next best thing.
Not part of your patch but something to be considered is that the label tree needs a rework, its locking needs to move to read side a read side lock less scheme, and the plan was to make it also use a linked list such that new labels are always queued at the end, allowing dynamically created labels to be lazily added to the tree.
I see the use of the kworker as problematic as well, especially if we are talking using kconfig to switch reference counting modes. I am futzing with some ideas, on how to deal with this.
Like I said I am still digesting.
Quoted 9 lines · John Johansen
I don't have objections to moving towards percpu refcounts, but the overhead of a percpu stuct per label is a problem when we have thousands of labels on the system. That is to say, this would have to be a config option. We moved buffers from kmalloc to percpu to reduce memory overhead to reduce contention. The to percpu, to a global pool because the percpu overhead was too high for some machines, and then from a global pool to a hybrid scheme because of global lock contention. I don't see a way of doing that with the label, which means a config would be the next best thing.
There was a patchset somewhere which adds counters starting as atomic and automagically converting themselves per-cpu if there as enough load applied to them. General point being it is plausible this may autotune itself.
Another option would be a boot-time tunable.
Quoted 6 lines
Not part of your patch but something to be considered is that the label tree needs a rework, its locking needs to move to read side a read side lock less scheme, and the plan was to make it also use a linked list such that new labels are always queued at the end, allowing dynamically created labels to be lazily added to the tree.
It's not *my* patchset. ;)
Quoted 4 lines
I see the use of the kworker as problematic as well, especially if we are talking using kconfig to switch reference counting modes. I am futzing with some ideas, on how to deal with this.
Thanks for the update. Hopefully this is going to get sorted out in the foreseeable future.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
Hi John,
Thanks for taking a look at the series!
Quoted 49 lines · John Johansen
Quoted 39 lines · Mateusz Guzik
Quoted 37 lines · John Johansen
Quoted 35 lines · Mateusz Guzik
Quoted 32 lines · John Johansen
Quoted 20 lines · Mateusz Guzik
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Quoted 5 lines
Gentle prod.
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
yes, sorry I have been unavailable for the last couple of weeks. I am now back, I have a rather large backlog to try catching up on but this is has an entry on the list.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
So where do we stand here?
sorry I am still trying to dig out of my backlog, I will look at this, this weekend.
How was the weekend? ;)
Quoted 10 lines
lets say it was busy. Have I looked at this, yes. I am still digesting it. I don't have objections to moving towards percpu refcounts, but the overhead of a percpu stuct per label is a problem when we have thousands of labels on the system. That is to say, this would have to be a config option. We moved buffers from kmalloc to percpu to reduce memory overhead to reduce contention. The to percpu, to a global pool because the percpu overhead was too high for some machines, and then from a global pool to a hybrid scheme because of global lock contention. I don't see a way of doing that with the label, which means a config would be the next best thing.
For the buffers, what was the percpu overhead roughly? For thousands of labels, I think, the extra memory overhead roughly would be in the range of few MBs (need to be profiled though). This extra label overhead would be considered high for the machines where percpu buffer overhead was considered high?
Please correct me here, so you are proposing that we use a kconfig to use either 'struct percpu_ref' or a 'struct kref' (using a union maybe) inside the 'struct aa_label' and update the refcount operations accordingly? If yes, I will work on a patch with this kconfig based selection of refcounting mode to see how it pans out.
@Mateusz can you share the dynamic switching counter mode patch series please?
In addition, for long term, there is an ongoing work (by Paul, Boqun and myself) on implementing hazard pointers as a scalable refcounting scheme [1] in kernel, which would not have memory usage overhead as in percpu refcount. At this point the API design/implementation is in early prototype stage.
[1] https://docs.google.com/document/d/113WFjGlAW4m72xNbZWHUSE-yU2HIJnWpiXp91ShtgeE/edit?usp=sharing
Quoted 6 lines
Not part of your patch but something to be considered is that the label tree needs a rework, its locking needs to move to read side a read side lock less scheme, and the plan was to make it also use a linked list such that new labels are always queued at the end, allowing dynamically created labels to be lazily added to the tree.
Read side would be rcu read lock protected in this scheme?
The linked list would store the dynamically created compound labels? What is the advantage of using this lazy addition to the tree? We optimize on the label search, addition/deletion for dynamic labels? The lazy addition to the tree is done when a label find operation on the list succeeds?
Quoted 4 lines
I see the use of the kworker as problematic as well, especially if we are talking using kconfig to switch reference counting modes. I am futzing with some ideas, on how to deal with this.
We can disable queuing of label reclaim work for non-percpu case?
Quoted 2 lines
Like I said I am still digesting.
Thank you!
Thanks
Neeraj
John Johansen
maintainerQuoted 65 lines · Neeraj Upadhyay
Hi John,
Thanks for taking a look at the series!
Quoted 49 lines · John Johansen
Quoted 39 lines · Mateusz Guzik
Quoted 37 lines · John Johansen
Quoted 35 lines · Mateusz Guzik
Quoted 32 lines · John Johansen
Quoted 20 lines · Mateusz Guzik
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Quoted 5 lines
Gentle prod.
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
yes, sorry I have been unavailable for the last couple of weeks. I am now back, I have a rather large backlog to try catching up on but this is has an entry on the list.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
So where do we stand here?
sorry I am still trying to dig out of my backlog, I will look at this, this weekend.
How was the weekend? ;)
Quoted 10 lines
lets say it was busy. Have I looked at this, yes. I am still digesting it. I don't have objections to moving towards percpu refcounts, but the overhead of a percpu stuct per label is a problem when we have thousands of labels on the system. That is to say, this would have to be a config option. We moved buffers from kmalloc to percpu to reduce memory overhead to reduce contention. The to percpu, to a global pool because the percpu overhead was too high for some machines, and then from a global pool to a hybrid scheme because of global lock contention. I don't see a way of doing that with the label, which means a config would be the next best thing.
Quoted 6 lines
For the buffers, what was the percpu overhead roughly? For thousands of labels, I think, the extra memory overhead roughly would be in the range of few MBs (need to be profiled though). This extra label overhead would be considered high for the machines where percpu buffer overhead was considered high?
It of course varies. It was fixed at 2-8K per cpu core depending on the buffer size. So on a 192 cpu machine you we are talking a couple MBs. Obviously more on bigger machines. The problem here is say the percpu refcount while smaller per label, will be more in situations with lots of cpus. Which is fine if that is what it needs to be, but for other use cases tuning it to be smaller would be nice.
Quoted 6 lines
Please correct me here, so you are proposing that we use a kconfig to use either 'struct
percpu_ref' or a 'struct kref' (using a union maybe) inside the 'structaa_label' and update the refcount operations accordingly? If yes, I will work on a patch with this kconfig based selection of refcounting mode to see how it pans out.
possibly, I am still mulling over how we want to approach this
Quoted 2 lines
@Mateusz can you share the dynamic switching counter mode patch series please?
yes I am interested in looking at this as well.
Quoted 7 lines
In addition, for long term, there is an ongoing work (by Paul, Boqun and myself) on implementing hazard pointers as a scalable refcounting scheme [1] in kernel, which would not have memory usage overhead as in percpu refcount. At this point the API design/implementation is in early prototype stage.
[1] https://docs.google.com/document/d/113WFjGlAW4m72xNbZWHUSE-yU2HIJnWpiXp91ShtgeE/edit?usp=sharing
okay, I will take a look
Quoted 8 lines
Quoted 7 lines
Not part of your patch but something to be considered is that the label tree needs a rework, its locking needs to move to read side a read side lock less scheme, and the plan was to make it also use a linked list such that new labels are always queued at the end, allowing dynamically created labels to be lazily added to the tree.
Quoted 6 lines
Read side would be rcu read lock protected in this scheme?
The linked list would store the dynamically created compound labels? What is the advantage of using this lazy addition to the tree? We optimize on the label search, addition/deletion for dynamic labels? The lazy addition to the tree is done when a label find operation on the list succeeds?
there are contexts where we are creating labels, and do not want to wait on some of the longer tree walk profile updates/replacements. If a replacement is on going the idea is to just add the label to the end of a list and let the process that is doing the tree update take the hit of inserting and rebalancing the tree.
Quoted 7 lines
Quoted 4 lines
I see the use of the kworker as problematic as well, especially if we are talking using kconfig to switch reference counting modes. I am futzing with some ideas, on how to deal with this.
We can disable queuing of label reclaim work for non-percpu case?
maybe, I am pondering ways we can deal with this. I have been pondering the if we might be able to leverage a seqlock here, but I will also take a look at hazard pointers.
Quoted 9 lines
Quoted 2 lines
Like I said I am still digesting.
Thank you!
Thanks
Neeraj
Quoted 88 lines · John Johansen
Quoted 65 lines · Neeraj Upadhyay
Hi John,
Thanks for taking a look at the series!
Quoted 49 lines · John Johansen
Quoted 39 lines · Mateusz Guzik
Quoted 37 lines · John Johansen
Quoted 35 lines · Mateusz Guzik
Quoted 32 lines · John Johansen
Quoted 20 lines · Mateusz Guzik
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Quoted 5 lines
Gentle prod.
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
yes, sorry I have been unavailable for the last couple of weeks. I am now back, I have a rather large backlog to try catching up on but this is has an entry on the list.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
So where do we stand here?
sorry I am still trying to dig out of my backlog, I will look at this, this weekend.
How was the weekend? ;)
Quoted 10 lines
lets say it was busy. Have I looked at this, yes. I am still digesting it. I don't have objections to moving towards percpu refcounts, but the overhead of a percpu stuct per label is a problem when we have thousands of labels on the system. That is to say, this would have to be a config option. We moved buffers from kmalloc to percpu to reduce memory overhead to reduce contention. The to percpu, to a global pool because the percpu overhead was too high for some machines, and then from a global pool to a hybrid scheme because of global lock contention. I don't see a way of doing that with the label, which means a config would be the next best thing.
Quoted 6 lines
For the buffers, what was the percpu overhead roughly? For thousands of labels, I think, the extra memory overhead roughly would be in the range of few MBs (need to be profiled though). This extra label overhead would be considered high for the machines where percpu buffer overhead was considered high?
It of course varies. It was fixed at 2-8K per cpu core depending on the buffer size. So on a 192 cpu machine you we are talking a couple MBs. Obviously more on bigger machines. The problem here is say the percpu refcount while smaller per label, will be more in situations with lots of cpus. Which is fine if that is what it needs to be, but for other use cases tuning it to be smaller would be nice.
Quoted 6 lines
Please correct me here, so you are proposing that we use a kconfig to use either 'struct
percpu_ref' or a 'struct kref' (using a union maybe) inside the 'structaa_label' and update the refcount operations accordingly? If yes, I will work on a patch with this kconfig based selection of refcounting mode to see how it pans out.
Quoted 6 lines
possibly, I am still mulling over how we want to approach this
Quoted 2 lines
@Mateusz can you share the dynamic switching counter mode patch series please?
yes I am interested in looking at this as well.
https://lore.kernel.org/lkml/[email protected]/
Quoted 36 lines
Quoted 7 lines
In addition, for long term, there is an ongoing work (by Paul, Boqun and myself) on implementing hazard pointers as a scalable refcounting scheme [1] in kernel, which would not have memory usage overhead as in percpu refcount. At this point the API design/implementation is in early prototype stage.
[1] https://docs.google.com/document/d/113WFjGlAW4m72xNbZWHUSE-yU2HIJnWpiXp91ShtgeE/edit?usp=sharing
okay, I will take a look
Quoted 8 lines
Quoted 7 lines
Not part of your patch but something to be considered is that the label tree needs a rework, its locking needs to move to read side a read side lock less scheme, and the plan was to make it also use a linked list such that new labels are always queued at the end, allowing dynamically created labels to be lazily added to the tree.
Quoted 6 lines
Read side would be rcu read lock protected in this scheme?
The linked list would store the dynamically created compound labels? What is the advantage of using this lazy addition to the tree? We optimize on the label search, addition/deletion for dynamic labels? The lazy addition to the tree is done when a label find operation on the list succeeds?there are contexts where we are creating labels, and do not want to wait on some of the longer tree walk profile updates/replacements. If a replacement is on going the idea is to just add the label to the end of a list and let the process that is doing the tree update take the hit of inserting and rebalancing the tree.
Quoted 5 lines
Quoted 4 lines
I see the use of the kworker as problematic as well, especially if we are talking using kconfig to switch reference counting modes. I am futzing with some ideas, on how to deal with this.
Quoted 6 lines
Quoted 2 lines
We can disable queuing of label reclaim work for non-percpu case?
maybe, I am pondering ways we can deal with this. I have been pondering the if we might be able to leverage a seqlock here, but I will also take a look at hazard pointers.
Since there is some elaborate talk going about this, let me throw in my own $0,03 -- I may happen to have a simple solution which will sort it out and it boils down to storing local ref updates in task_struct.
Here is the context: creds are always refed and unrefed when creating and destroying a file object. Should you have one instance of credentials for a given user across different processes they would serialize on updating the ref. Linux mostly dodges the problem by always creating a copy on fork, thus only serializing within threads of a given process. Even then that induces avoidable overhead if only from single-threaded standpoint -- that's 2 atomics slapped for every open/close cycle.
$elsewhere I devised an idea where cred ref updates to creds matching current->cred only modify a local counter. They get rolled up when current->creds is changed. That is to say there is 0 atomics or modifying memory seen by other cpus as long as the process does not change creds, which almost never happens compared to how often refing and unrefing is implemented.
In struct cred apart from regular refs you would have "user" counter and "distributed" counter. switching user to > 0 grabs a normal ref on creds, the value of the "distributed" counter is arbitrary as long as user > 0. users going back to 0 means we can release the special ref held for that purpose.
I was considering implementing this for Linux. In my original code all cred handling is augmented like this, but for Linux one could add a dedicated get_cred_localref or similar machinery.
Skimming through apparmor suggests the bit which does cause performance problems can be sorted out in the same manner.
Maybe i'll hack it up as a demo just for apparmor.
This would come with some extra space usage in task_struct which on the surface may sounds like a non-starter. However, should you take a look at the struct with pahole you will find it is riddles with padding. If I wanted to I could add all fields I need to the struct and not grow it on LP64.
Signature · 1 line
Mateusz Guzik <mjguzik gmail.com>
John Johansen
maintainerQuoted 146 lines · Mateusz Guzik
Quoted 88 lines · John Johansen
Quoted 65 lines · Neeraj Upadhyay
Hi John,
Thanks for taking a look at the series!
Quoted 49 lines · John Johansen
Quoted 39 lines · Mateusz Guzik
Quoted 37 lines · John Johansen
Quoted 35 lines · Mateusz Guzik
Quoted 32 lines · John Johansen
Quoted 20 lines · Mateusz Guzik
Quoted 11 lines · John Johansen
Quoted 9 lines · Neeraj Upadhyay
Gentle ping.
John,
Could you please confirm that:
a. The AppArmor refcount usage described in the RFC is correct? b. Approach taken to fix the scalability issue is valid/correct?
Quoted 7 lines
Hi Neeraj,
I know your patchset has been waiting on review for a long time. Unfortunately I have been very, very busy lately. I will try to get to it this weekend, but I can't promise that I will be able to get the review fully done.
Quoted 5 lines
Gentle prod.
Any chances of this getting reviewed in the foreseeable future? Would be a real bummer if the patchset fell through the cracks.
yes, sorry I have been unavailable for the last couple of weeks. I am now back, I have a rather large backlog to try catching up on but this is has an entry on the list.
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
So where do we stand here?
sorry I am still trying to dig out of my backlog, I will look at this, this weekend.
How was the weekend? ;)
Quoted 10 lines
lets say it was busy. Have I looked at this, yes. I am still digesting it. I don't have objections to moving towards percpu refcounts, but the overhead of a percpu stuct per label is a problem when we have thousands of labels on the system. That is to say, this would have to be a config option. We moved buffers from kmalloc to percpu to reduce memory overhead to reduce contention. The to percpu, to a global pool because the percpu overhead was too high for some machines, and then from a global pool to a hybrid scheme because of global lock contention. I don't see a way of doing that with the label, which means a config would be the next best thing.
Quoted 6 lines
For the buffers, what was the percpu overhead roughly? For thousands of labels, I think, the extra memory overhead roughly would be in the range of few MBs (need to be profiled though). This extra label overhead would be considered high for the machines where percpu buffer overhead was considered high?
It of course varies. It was fixed at 2-8K per cpu core depending on the buffer size. So on a 192 cpu machine you we are talking a couple MBs. Obviously more on bigger machines. The problem here is say the percpu refcount while smaller per label, will be more in situations with lots of cpus. Which is fine if that is what it needs to be, but for other use cases tuning it to be smaller would be nice.
Quoted 6 lines
Please correct me here, so you are proposing that we use a kconfig to use either 'struct
percpu_ref' or a 'struct kref' (using a union maybe) inside the 'structaa_label' and update the refcount operations accordingly? If yes, I will work on a patch with this kconfig based selection of refcounting mode to see how it pans out.Quoted 6 lines
possibly, I am still mulling over how we want to approach this
Quoted 2 lines
@Mateusz can you share the dynamic switching counter mode patch series please?
yes I am interested in looking at this as well.
https://lore.kernel.org/lkml/[email protected]/
Quoted 36 lines
Quoted 7 lines
In addition, for long term, there is an ongoing work (by Paul, Boqun and myself) on implementing hazard pointers as a scalable refcounting scheme [1] in kernel, which would not have memory usage overhead as in percpu refcount. At this point the API design/implementation is in early prototype stage.
[1] https://docs.google.com/document/d/113WFjGlAW4m72xNbZWHUSE-yU2HIJnWpiXp91ShtgeE/edit?usp=sharing
okay, I will take a look
Quoted 8 lines
Quoted 7 lines
Not part of your patch but something to be considered is that the label tree needs a rework, its locking needs to move to read side a read side lock less scheme, and the plan was to make it also use a linked list such that new labels are always queued at the end, allowing dynamically created labels to be lazily added to the tree.
Quoted 6 lines
Read side would be rcu read lock protected in this scheme?
The linked list would store the dynamically created compound labels? What is the advantage of using this lazy addition to the tree? We optimize on the label search, addition/deletion for dynamic labels? The lazy addition to the tree is done when a label find operation on the list succeeds?there are contexts where we are creating labels, and do not want to wait on some of the longer tree walk profile updates/replacements. If a replacement is on going the idea is to just add the label to the end of a list and let the process that is doing the tree update take the hit of inserting and rebalancing the tree.
Quoted 5 lines
Quoted 4 lines
I see the use of the kworker as problematic as well, especially if we are talking using kconfig to switch reference counting modes. I am futzing with some ideas, on how to deal with this.
Quoted 6 lines
Quoted 2 lines
We can disable queuing of label reclaim work for non-percpu case?
maybe, I am pondering ways we can deal with this. I have been pondering the if we might be able to leverage a seqlock here, but I will also take a look at hazard pointers.
Since there is some elaborate talk going about this, let me throw in my own $0,03 -- I may happen to have a simple solution which will sort it out and it boils down to storing local ref updates in
task_struct.
Quoted 9 lines
Here is the context: creds are always refed and unrefed when creating and destroying a file object. Should you have one instance of credentials for a given user across different processes they would serialize on updating the ref. Linux mostly dodges the problem by always creating a copy on fork, thus only serializing within threads of a given process. Even then that induces avoidable overhead if only from single-threaded standpoint -- that's 2 atomics slapped for every open/close cycle.
so the apparmor label can and will update beyond the open/close cycle. Yes they are used in the cred as well but, for more than that. The apparmor label on file can be updated by other tasks, for various reasons.
Quoted 7 lines
$elsewhere I devised an idea where cred ref updates to creds matching
current->credonly modify a local counter. They get rolled up whencurrent->credsis changed. That is to say there is 0 atomics or modifying memory seen by other cpus as long as the process does not change creds, which almost never happens compared to how often refing and unrefing is implemented.
right, we do something like this for the task cred with a crit section marked out by
label = begin_current_label_crit_section()
end_current_label_crit_section(label);
if everything works out, no reference counts are taken. The purpose of the fns is to deal with the cases where for one reason or another a refcount needs to be taken (generally because of live policy replacement, and the task hasn't been able to update its cred yet).
Quoted 6 lines
In struct cred apart from regular refs you would have "user" counter and "distributed" counter. switching user to > 0 grabs a normal ref on creds, the value of the "distributed" counter is arbitrary as long as user > 0. users going back to 0 means we can release the special ref held for that purpose.
So I don't see how this will generally help for labels which exist on many different objects.
Quoted 4 lines
I was considering implementing this for Linux. In my original code all cred handling is augmented like this, but for Linux one could add a dedicated
get_cred_localrefor similar machinery.
sure, but I am not sure its needed. The rules for task creds is only task can update its cred. The task can look at its cred and do most things without having to take a count. Most cred refs should just be being taken for objects.
Quoted 3 lines
Skimming through apparmor suggests the bit which does cause performance problems can be sorted out in the same manner.
I don't see it. The file cred is very much updated live, async to the task cred. And while currently it always starts as the task cred, that won't even be true much longer.
Quoted 8 lines
Maybe i'll hack it up as a demo just for apparmor.
This would come with some extra space usage in
task_structwhich on the surface may sounds like a non-starter. However, should you take a look at the struct with pahole you will find it is riddles with padding. If I wanted to I could add all fields I need to the struct and not grow it on LP64.