Skip to contentSkip to conversation

jump_label: mitigate dynamic-debug induced IPI storms

PATCH New Jim Cromie 1 participant 6 messages Apr 3

Subsystems (4)
lore
Share & tools
lore

Jim Cromie

author maintainer
#0 lore raw

cover jump_label: mitigate dynamic-debug induced IPI storms

This series adds batched toggling of jump_labels to address IPI storms triggered by bulk toggles of dynamic-debug pr_debug()s.

patches 1-3 are for submission. 4-5 are for demonstration only.

The Scaling Problem:

In testing on 7.1-rc6 guests (virtme-ng) hosted on a stock Fedora kernel, I measured the cost of toggling ~1,900 dyndbg callsites. The current one-at-a-time patching approach creates a massive synchronization bottleneck that scales linearly with both key count and core count:

Baseline IPIs (Total CAL Interrupts)
Successive query pairs (+p ; -p ;) toggling ~3,792 keys per pair. Pairs | -p 2 | -p 3 | -p 4 | -p 5 | Notes
-----------------------------------------------------------

  1   | 10,738 |  21,468 |  32,148 |  42,837 | ~11.3 IPI/key/CPU
  2   | 21,269 |  42,976 |  64,416 |  85,880 | Linear scaling (K*M)
  3   | 32,125 |  64,434 |  96,604 | 128,793 | 
  4   | 42,831 |  85,886 | 128,776 | 171,697 | 
  5   | 53,540 | 107,353 | 160,985 | 214,636 | 
  6   | 64,174 | 128,800 | 193,159 | 257,562 | > 250k IPI storm

On a 5-VCPU system, a sequence of 6 query pairs triggers over a quarter-million interrupts. This overhead is a direct result of invoking the full patching machinery for every individual key toggle.

The Solution:

While x86 already has a 256-entry queue for jump_label, 2 issues currently block its use:

1. dyndbg uses the synchronous API; an IPI is issued for every static-key. patch-1 provides a fix for this with a _queued api extension. patch-3 uses the new api in dyndbg.

2. The queue flushes if current_patch_addr < previous_patch_addr. dyndbg has consecutive descriptors, but not consecutive patch_addrs. Using the API with just patch-1 yields ~1/2 the IPIs. Patch 2 sorts the patch_addrs, allowing the queue to fill to its 256-entry capacity before flushing, for a bigger win.

Results Summary:

Jump-Batch IPIs (Total CAL Interrupts)
Pairs | -p 2 | -p 3 | -p 4 | -p 5 |
---------------------------------------

  1   |  135  |  202  |  239  |  290  |
  2   |  145  |  256  |  347  |  445  |
  3   |  198  |  352  |  479  |  640  |
  4   |  246  |  445  |  630  |  840  |
  5   |  293  |  545  |  774  | 1,033 |
  6   |  341  |  642  |  915  | 1,222 |

Marginal Cost Comparison (IPIs added per '+p;-p;' query pair) VCPUs | Baseline (m) | Jump-Batch (m) | Efficiency Gain
-------------------------------------------------------

  2   |    10,696    |      42.4      |      252x
  3   |    21,466    |      89.3      |      240x
  4   |    32,207    |     138.1      |      233x
  5   |    42,945    |     190.5      |      225x

Why not static_key_deferred?

static_key_deferred is a rate-limiter for single, high-frequency keys. For a bulk update of 1,900 distinct keys, a deferred timer would still fire 1,900 times, resulting in the same IPI storm spread across a longer window. This will make it harder to use, particularly for tools which are fast enough to perceive the asynchrony. In contrast, jump-batch just amortizes the IPI cost, eliminating the need for rate-throttling.

Why wasnt this done previously ?

The presence of the queue for x86 suggests it was done, minimally, for the cases where the patch-addresses were ordered, and where performance mattered more (ftrace etc). In contrast, pr_debug()s are pretty basic, for direct use by humans, who dont toggle them on and off at high rates, and who are far more selective than "+p" the 2nd time they use dyndbg.

Model Summary:

The baseline cost is driven by (Keys * CPUs), resulting in 43k IPIs per transaction on a 5-VCPU VM. The jump-batch solution reduces this to ~190 IPIs, moving the bottleneck from key density (K) to architectural rendezvous cost (M).

Detailed Data:

NB: most tests below used the " !module virtio* +p ; -p ; " query-pair, relying on patch-5 for the "!module virtio*" parsing to avoid flooding the guest/VM logs with messages. This is less important when both on;off are sent together, as console flooding impedes submitting a separate off command.

Baseline:

jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 0
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 0
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 0
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10387
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 10111
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 161
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 28
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 10162
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10418
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10390
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10379
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 10467
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 9812
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20856
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 20277
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 163
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 97
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 20259
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20856
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20861
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20818
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 20848
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 20275
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31237
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 30349
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 171
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 101
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 30367
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31224
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31233
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31223
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 31219
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 30346
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41593
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 40447
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 174
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 91
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 40451
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41546
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41595
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41622
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1897 Delta-CAL (IPI): 41607
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 40442

Patched:

jimc@frodo:~/projects/lx/wk-A$ . ../farmit.sh 
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 0
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 0
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 0
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 0
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 131
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 127
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 130
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 78
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 124
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 116
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 134
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 141
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 153
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 124
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 359
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 200
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 175
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 100
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 191
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 186
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 186
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 171
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 181
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 182
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 235
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 225
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 173
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 93
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 227
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 294
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 250
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 273
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 307
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 266
jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis
b0-dd:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 277
b0-dd-drm:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 265
b0-defcon:
 doing: dd_ipis
../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory
cat: /tmp/dd_err: No such file or directory
ERROR (0): 
wc: /proc/dynamic_debug/control: No such file or directory
 key-ct:  Delta-CAL (IPI): 189
b0-ddc:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1 Delta-CAL (IPI): 96
b0-dd-drm-all:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 267
b0-dd-tym:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 278
b0-dd-tyy:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 272
b0-ksan:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 285
b0-ksan-leak:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1896 Delta-CAL (IPI): 280
b0-ftrace:
 doing: dd_ipis
SUCCESS: !module virtio* +p ; -p
 key-ct: 1849 Delta-CAL (IPI): 265

LINEAR PROGRESSION

both progressions are linear, but patched has much lower slope and intercept (y=mx+b)

Baseline:

jimc@frodo:~/projects/lx/linux.git$ cd b0-dd
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 10738
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 21269
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 32125
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 42831
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 53540
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 64174
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 21468
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 42976
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 64434
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 85886
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 107353
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 128800
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 32148
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 64416
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 96604
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 128776
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 160985
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 193159
jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 42837
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 85880
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 128793
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 171697
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 214636
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1897 Delta-CAL (IPI): 257562
jimc@frodo:~/projects/lx/linux.git/b0-dd$ 

PATCHED:

jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 135
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 145
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 198
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 246
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 293
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 341
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 202
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 256
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 352
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 445
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 545
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 642
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 239
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 347
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 479
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 630
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 774
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 915
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear
virtme-ng 1.40
doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear
 doing: dd_linear
SUCCESS: +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 290
SUCCESS: +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 445
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 640
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 840
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 1033
SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ;
 key-ct: 1896 Delta-CAL (IPI): 1222
jimc@frodo:~/projects/lx/wk-A/b0-dd$ 

Acknowldegement:

I used google-gemini extensively to discuss and review this work, and to read the detailed data, and prepare the summary tables and prose above. It is patient with my pedantry.

Jim Cromie (5):
jump_label: add queueing API for batched static key updates x86/alternative.c: sort text-pokes before flushing the queue
dyndbg: use static-key queueing API in dynamic-debug
dyndbg: factor ddebug_match_desc out from ddebug_change (for
demonstration only)
lib/dynamic_debug: add negation support to queries

 arch/x86/kernel/alternative.c |  42 +++++------
 include/linux/jump_label.h    |  18 +++++
 kernel/jump_label.c           | 104 +++++++++++++++++++------
 lib/dynamic_debug.c           | 138 ++++++++++++++++++++++------------
 4 files changed, 211 insertions(+), 91 deletions(-)

base-commit: 7aaa8047eafd0bd628065b15757d9b48c5f9c07d

Signature · 1 line
2.53.0
author maintainer
#1 lore raw

PATCH 1/5 jump_label: add queueing API for batched static key updates

Introduce a new queueing API for static keys to allow batching of instruction updates. This is provided for subsystems like dynamic-debug, which may toggle thousands of static keys in a single operation. Currently, this suffers from (N)$ overhead due to repeated machine-wide stop_machine or text_poke_sync synchronizations.

The new API includes:
- static_key_enable_queued(key)
- static_key_disable_queued(key)
- static_key_apply_queued() (the global synchronization barrier)
- Corresponding static_branch_*_queued(x) macros.

The _queued functionality is achieved by adding a 'bool apply' parameter to 3 existing midlayer static functions, and using it to decide whether or not to call arch_jump_label_transform_apply(). By not explicitly calling it, we allow the transform queue to fill 1st. This amortizes the cost of the IPI sync to 1/N (256 on x86).

1. __jump_label_update(struct static_key *key, bool apply) this implements the if (apply) arch_jump_label_transform_apply();

2. __jump_label_mod_update(struct static_key *key, bool apply) extends _queued feature to modules' static-keys

3. rename jump_label_update(struct static_key *key)
to __jump_label_update_key(struct static_key *key, bool apply)
and wrap it with 1-liners:
jump_label_update() & jump_label_update_queued()

On architectures supporting HAVE_JUMP_LABEL_BATCH, the _queued variants lazily fill the architecture's patch queue.

It is safe to mix queued and immediate updates; any non-queued static_key_enable/disable call will effectively "flush" any previously queued changes as a side effect. Or just call static_key_apply_queued().

On architectures that do not need a patch queue (e.g., those with atomic patching) or that do not support batching, the API transparently falls back to immediate patching.

---
 include/linux/jump_label.h |  18 +++++++
 kernel/jump_label.c        | 104 +++++++++++++++++++++++++++++--------
 2 files changed, 100 insertions(+), 22 deletions(-)
include/linux/jump_label.h +18 -0
index fdb79dd1ebd8..283ee9360026 100644
@@ -234,6 +234,9 @@ extern void static_key_slow_dec_cpuslocked(struct static_key *key);
 extern int static_key_count(struct static_key *key);
 extern void static_key_enable(struct static_key *key);
 extern void static_key_disable(struct static_key *key);
+extern void static_key_enable_queued(struct static_key *key);
+extern void static_key_disable_queued(struct static_key *key);
+extern void static_key_apply_queued(void);
 extern void static_key_enable_cpuslocked(struct static_key *key);
 extern void static_key_disable_cpuslocked(struct static_key *key);
 extern enum jump_label_type jump_label_init_type(struct jump_entry *entry);
@@ -340,6 +343,18 @@ static inline void static_key_disable(struct static_key *key)
 	atomic_set(&key->enabled, 0);
 }
 
+static inline void static_key_enable_queued(struct static_key *key)
+{
+	static_key_enable(key);
+}
+
+static inline void static_key_disable_queued(struct static_key *key)
+{
+	static_key_disable(key);
+}
+
+static inline void static_key_apply_queued(void) {}
+
 #define static_key_enable_cpuslocked(k)		static_key_enable((k))
 #define static_key_disable_cpuslocked(k)	static_key_disable((k))
 
@@ -535,6 +550,9 @@ extern bool ____wrong_branch_error(void);
 
 #define static_branch_enable(x)			static_key_enable(&(x)->key)
 #define static_branch_disable(x)		static_key_disable(&(x)->key)
+#define static_branch_enable_queued(x)		static_key_enable_queued(&(x)->key)
+#define static_branch_disable_queued(x)		static_key_disable_queued(&(x)->key)
+#define static_branch_apply_queued()		static_key_apply_queued()
 #define static_branch_enable_cpuslocked(x)	static_key_enable_cpuslocked(&(x)->key)
 #define static_branch_disable_cpuslocked(x)	static_key_disable_cpuslocked(&(x)->key)
 
kernel/jump_label.c +82 -22

178 lines not on this page · raw message

Signature · 1 line
2.53.0
author maintainer
#2 lore raw

PATCH 2/5 x86/alternative.c: sort text-pokes before flushing the queue

In reply to Jim Cromie

Until now, x86 can queue jump_label text-pokes as long as the poke-addr is monotonically increasing, but flushes the queue when a new poke-addr is less than the previous.

Dynamic-debug now uses the queued static-key API, but the advantage is limited; we see a ~2x reduction in IPIs. Although the pr_debug descriptors are ordered, the patch-address in them are not; about 1/2 violate the ordering constraint.

So this patch drops that requirement, and sorts the text-pokes by their address before applying them. Doing so lets us fill the queue before sorting then flushing, giving a dramatic ~125x reduction in IPIs over the traditional single IPI per pr_debug.

Other arches don't need a queue, and so have nothing to sort.

#> dd_ipis

   [   23.100381] dyndbg: query 0: "module !virtio* +p " mod:*
   [   23.103432] dyndbg: query 1: "-p" mod:*
   Delta-CAL (IPI): 242
---
 arch/x86/kernel/alternative.c | 42 ++++++++++++++++-------------------
 lib/dynamic_debug.c           |  1 +
 2 files changed, 20 insertions(+), 23 deletions(-)
arch/x86/kernel/alternative.c +19 -23
index e87da25d1236..92987954b8aa 100644
@@ -2,6 +2,7 @@
 #define pr_fmt(fmt) "SMP alternatives: " fmt
 
 #include <linux/mmu_context.h>
+#include <linux/sort.h>
 #include <linux/perf_event.h>
 #include <linux/vmalloc.h>
 #include <linux/memory.h>
@@ -2823,6 +2824,18 @@ static __always_inline int patch_cmp(const void *tpl_a, const void *tpl_b)
 	return 0;
 }
 
+static int text_poke_loc_cmp(const void *a, const void *b)
+{
+	const struct smp_text_poke_loc *tpl_a = a;
+	const struct smp_text_poke_loc *tpl_b = b;
+
+	if (tpl_a->rel_addr < tpl_b->rel_addr)
+		return -1;
+	if (tpl_a->rel_addr > tpl_b->rel_addr)
+		return 1;
+	return 0;
+}
+
 noinstr int smp_text_poke_int3_handler(struct pt_regs *regs)
 {
 	struct smp_text_poke_loc *tpl;
@@ -2935,6 +2948,10 @@ void smp_text_poke_batch_finish(void)
 	if (!text_poke_array.nr_entries)
 		return;
 
+	if (text_poke_array.nr_entries > 1)
+		sort(text_poke_array.vec, text_poke_array.nr_entries,
+		     sizeof(struct smp_text_poke_loc), text_poke_loc_cmp, NULL);
+
 	lockdep_assert_held(&text_mutex);
 
 	/*
@@ -3151,28 +3168,6 @@ static void __smp_text_poke_batch_add(void *addr, const void *opcode, size_t len
 	}
 }
 
-/*
- * We hard rely on the text_poke_array.vec being ordered; ensure this is so by flushing
- * early if needed.
- */
-static bool text_poke_addr_ordered(void *addr)
-{
-	WARN_ON_ONCE(!addr);
-
-	if (!text_poke_array.nr_entries)
-		return true;
-
-	/*
-	 * If the last current entry's address is higher than the
-	 * new entry's address we'd like to add, then ordering
-	 * is violated and we must first flush all pending patching
-	 * requests:
-	 */
-	if (text_poke_addr(text_poke_array.vec + text_poke_array.nr_entries-1) > addr)
-		return false;
-
-	return true;
-}
 
 /**
  * smp_text_poke_batch_add() -- update instruction on live kernel on SMP, batched
@@ -3189,8 +3184,9 @@ static bool text_poke_addr_ordered(void *addr)
  */
 void __ref smp_text_poke_batch_add(void *addr, const void *opcode, size_t len, const void *emulate)
 {
-	if (text_poke_array.nr_entries == TEXT_POKE_ARRAY_MAX || !text_poke_addr_ordered(addr))
+	if (text_poke_array.nr_entries == TEXT_POKE_ARRAY_MAX)
 		smp_text_poke_batch_finish();
+
 	__smp_text_poke_batch_add(addr, opcode, len, emulate);
 }
 
lib/dynamic_debug.c +1 -0
index 18a71a9108d3..b5060749464e 100644
@@ -264,6 +264,7 @@ static int ddebug_change(const struct ddebug_query *query,
 		}
 	}
 	mutex_unlock(&ddebug_lock);
+	v2pr_info("applied %d queued updates to sites in total\n", nfound);
 
 	if (!nfound && verbose)
 		pr_info("no matches for query\n");
Signature · 1 line
2.53.0
author maintainer
#3 lore raw

PATCH 3/5 dyndbg: use static-key queueing API in dynamic-debug

In reply to Jim Cromie

Use the new static-key queueing API in dynamic-debug to reduce IPIs.

this replaces the 2 calls to static_branch_{enable.disable}() in ddebug_change() with their _queued() counterparts, and adds a call to static_branch_apply_queued() at the bottom.

This gives a theoretical reduction of IPIs up to 256x, and a practical reduction by 120-180x.

---
 lib/dynamic_debug.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
lib/dynamic_debug.c +3 -2
index b5060749464e..9bee7dde5148 100644
@@ -250,9 +250,9 @@ static int ddebug_change(const struct ddebug_query *query,
 #ifdef CONFIG_JUMP_LABEL
 			if (dp->flags & _DPRINTK_FLAGS_PRINT) {
 				if (!(newflags & _DPRINTK_FLAGS_PRINT))
-					static_branch_disable(&dp->key.dd_key_true);
+					static_branch_disable_queued(&dp->key.dd_key_true);
 			} else if (newflags & _DPRINTK_FLAGS_PRINT) {
-				static_branch_enable(&dp->key.dd_key_true);
+				static_branch_enable_queued(&dp->key.dd_key_true);
 			}
 #endif
 			v4pr_info("changed %s:%d [%s]%s %s => %s\n",
@@ -263,6 +263,7 @@ static int ddebug_change(const struct ddebug_query *query,
 			dp->flags = newflags;
 		}
 	}
+	static_branch_apply_queued();
 	mutex_unlock(&ddebug_lock);
 	v2pr_info("applied %d queued updates to sites in total\n", nfound);
 
Signature · 1 line
2.53.0
author maintainer
#4 lore raw

PATCH 4/5 dyndbg: factor ddebug_match_desc out from ddebug_change (for demonstration only)

In reply to Jim Cromie

ddebug_change() is a big (~100 lines) function with a nested for loop.

The outer loop walks the per-module ddebug_tables list, and does module stuff: it filters on a query's "module FOO*" and "class BAR", failures here skip the entire inner loop.

The inner loop (60 lines) scans a module's descriptors. It starts with a long block of filters on function, line, format, and the validated "BAR" class (or the legacy/_DPRINTK_CLASS_DFLT).

These filters "continue" past pr_debugs that don't match the query criteria, before it falls through the code below that counts matches, then adjusts the flags and static-keys. This is unnecessarily hard to think about.

So move the per-descriptor filter-block into a boolean function:
ddebug_match_desc(desc), and change each "continue" to "return false". This puts a clear interface in place, so any future changes are either inside, outside, or across this interface.

also fix checkpatch complaints about spaces and braces.

---
 lib/dynamic_debug.c | 83 +++++++++++++++++++++++++--------------------
 1 file changed, 47 insertions(+), 36 deletions(-)
lib/dynamic_debug.c +47 -36
index 9bee7dde5148..0b13810b03db 100644
@@ -172,6 +172,52 @@ static struct ddebug_class_map *ddebug_find_valid_class(struct ddebug_table cons
  * callsites, normally the same as number of changes.  If verbose,
  * logs the changes.  Takes ddebug_lock.
  */
+static bool ddebug_match_desc(const struct ddebug_query *query,
+			      struct _ddebug *dp,
+			      int valid_class)
+{
+	/* match site against query-class */
+	if (dp->class_id != valid_class)
+		return false;
+
+	/* match against the source filename */
+	if (query->filename &&
+	    !match_wildcard(query->filename, dp->filename) &&
+	    !match_wildcard(query->filename,
+			    kbasename(dp->filename)) &&
+	    !match_wildcard(query->filename,
+			    trim_prefix(dp->filename)))
+		return false;
+
+	/* match against the function */
+	if (query->function &&
+	    !match_wildcard(query->function, dp->function))
+		return false;
+
+	/* match against the format */
+	if (query->format) {
+		if (*query->format == '^') {
+			char *p;
+			/* anchored search. match must be at beginning */
+			p = strstr(dp->format, query->format + 1);
+			if (p != dp->format)
+				return false;
+		} else if (!strstr(dp->format, query->format)) {
+			return false;
+		}
+	}
+
+	/* match against the line number range */
+	if (query->first_lineno &&
+	    dp->lineno < query->first_lineno)
+		return false;
+	if (query->last_lineno &&
+	    dp->lineno > query->last_lineno)
+		return false;
+
+	return true;
+}
+
 static int ddebug_change(const struct ddebug_query *query,
 			 struct flag_settings *modifiers)
 {
@@ -204,42 +250,7 @@ static int ddebug_change(const struct ddebug_query *query,
 		for (i = 0; i < dt->num_ddebugs; i++) {
 			struct _ddebug *dp = &dt->ddebugs[i];
 
-			/* match site against query-class */
-			if (dp->class_id != valid_class)
-				continue;
-
-			/* match against the source filename */
-			if (query->filename &&
-			    !match_wildcard(query->filename, dp->filename) &&
-			    !match_wildcard(query->filename,
-					   kbasename(dp->filename)) &&
-			    !match_wildcard(query->filename,
-					   trim_prefix(dp->filename)))
-				continue;
-
-			/* match against the function */
-			if (query->function &&
-			    !match_wildcard(query->function, dp->function))
-				continue;
-
-			/* match against the format */
-			if (query->format) {
-				if (*query->format == '^') {
-					char *p;
-					/* anchored search. match must be at beginning */
-					p = strstr(dp->format, query->format+1);
-					if (p != dp->format)
-						continue;
-				} else if (!strstr(dp->format, query->format))
-					continue;
-			}
-
-			/* match against the line number range */
-			if (query->first_lineno &&
-			    dp->lineno < query->first_lineno)
-				continue;
-			if (query->last_lineno &&
-			    dp->lineno > query->last_lineno)
+			if (!ddebug_match_desc(query, dp, valid_class))
 				continue;
 
 			nfound++;
Signature · 1 line
2.53.0
author maintainer
#5 lore raw

PATCH 5/5 lib/dynamic_debug: add negation support to queries

In reply to Jim Cromie

This allow users to invert the selection of most keywords, excluding lineno and class. For example:

echo "!module virtio* +p" > /proc/dynamic_debug/control

When testing a kernel running in virtme-ng, this cmd prevents flooding the logs with virtio activity, and spoiling the the test scenario.

NOTE:

This is convenient, but not perfect; it cannot also avoid flooding from pr_debugs in serial_core or other potential sources.

A more robust command is:

echo "!module virtio* +p % module serial -p" > /proc/dynamic_debug/control

It also "interferes" with evolution of the grammar; particularly around and/or combinations. For example this could become meaningful, though perhaps inadvisable:

echo " !module virtio* !module serial +p "

lineno is currently excluded, pending review of its usefulness vs ambiguities, for example " !lineno 1-20 " has straightforward meaning.

!class FOO is unlikely to be justifiable; it defeats the "protection thru specificity" that classes are designed for.

---
 lib/dynamic_debug.c | 81 +++++++++++++++++++++++++++++++--------------
 1 file changed, 57 insertions(+), 24 deletions(-)
lib/dynamic_debug.c +57 -24
index 0b13810b03db..3addfa06e54d 100644
@@ -58,6 +58,10 @@ struct ddebug_query {
 	const char *format;
 	const char *class_string;
 	unsigned int first_lineno, last_lineno;
+	unsigned int filename_neg:1;
+	unsigned int module_neg:1;
+	unsigned int function_neg:1;
+	unsigned int format_neg:1;
 };
 
 struct ddebug_iter {
@@ -139,11 +143,12 @@ static void vpr_info_dq(const struct ddebug_query *query, const char *msg)
 			fmtlen--;
 	}
 
-	v3pr_info("%s: func=\"%s\" file=\"%s\" module=\"%s\" format=\"%.*s\" lineno=%u-%u class=%s\n",
+	v3pr_info("%s: func%s=\"%s\" file%s=\"%s\" module%s=\"%s\" format%s=\"%.*s\" lineno=%u-%u class=%s\n",
 		  msg,
-		  query->function ?: "",
-		  query->filename ?: "",
-		  query->module ?: "",
+		  query->function_neg ? "!" : "", query->function ?: "",
+		  query->filename_neg ? "!" : "", query->filename ?: "",
+		  query->module_neg ? "!" : "", query->module ?: "",
+		  query->format_neg ? "!" : "",
 		  fmtlen, query->format ?: "",
 		  query->first_lineno, query->last_lineno, query->class_string);
 }
@@ -176,35 +181,38 @@ static bool ddebug_match_desc(const struct ddebug_query *query,
 			      struct _ddebug *dp,
 			      int valid_class)
 {
+	bool match;
+
 	/* match site against query-class */
 	if (dp->class_id != valid_class)
 		return false;
 
 	/* match against the source filename */
-	if (query->filename &&
-	    !match_wildcard(query->filename, dp->filename) &&
-	    !match_wildcard(query->filename,
-			    kbasename(dp->filename)) &&
-	    !match_wildcard(query->filename,
-			    trim_prefix(dp->filename)))
-		return false;
+	if (query->filename) {
+		match = match_wildcard(query->filename, dp->filename) ||
+			match_wildcard(query->filename, kbasename(dp->filename)) ||
+			match_wildcard(query->filename, trim_prefix(dp->filename));
+		if (match == query->filename_neg)
+			return false;
+	}
 
 	/* match against the function */
-	if (query->function &&
-	    !match_wildcard(query->function, dp->function))
-		return false;
+	if (query->function) {
+		match = match_wildcard(query->function, dp->function);
+		if (match == query->function_neg)
+			return false;
+	}
 
 	/* match against the format */
 	if (query->format) {
 		if (*query->format == '^') {
-			char *p;
 			/* anchored search. match must be at beginning */
-			p = strstr(dp->format, query->format + 1);
-			if (p != dp->format)
-				return false;
-		} else if (!strstr(dp->format, query->format)) {
-			return false;
+			match = (strstr(dp->format, query->format + 1) == dp->format);
+		} else {
+			match = !!strstr(dp->format, query->format);
 		}
+		if (match == query->format_neg)
+			return false;
 	}
 
 	/* match against the line number range */
@@ -234,9 +242,11 @@ static int ddebug_change(const struct ddebug_query *query,
 	list_for_each_entry(dt, &ddebug_tables, link) {
 
 		/* match against the module name */
-		if (query->module &&
-		    !match_wildcard(query->module, dt->mod_name))
-			continue;
+		if (query->module) {
+			bool match = match_wildcard(query->module, dt->mod_name);
+			if (match == query->module_neg)
+				continue;
+		}
 
 		if (query->class_string) {
 			map = ddebug_find_valid_class(dt, query->class_string, &valid_class);
@@ -399,6 +409,16 @@ static int parse_linerange(struct ddebug_query *query, const char *first)
 	return 0;
 }
 
+static char *check_neg(char *src, unsigned int *neg)
+{
+	if (*src == '!') {
+		*neg = 1;
+		return src + 1;
+	}
+	*neg = 0;
+	return src;
+}
+
 static int check_set(const char **dest, char *src, char *name)
 {
 	int rc = 0;
@@ -441,12 +461,15 @@ static int ddebug_parse_query(char *words[], int nwords,
 	}
 
 	for (i = 0; i < nwords; i += 2) {
-		char *keyword = words[i];
+		unsigned int neg;
+		char *keyword = check_neg(words[i], &neg);
 		char *arg = words[i+1];
 
 		if (!strcmp(keyword, "func")) {
+			query->function_neg = neg;
 			rc = check_set(&query->function, arg, "func");
 		} else if (!strcmp(keyword, "file")) {
+			query->filename_neg = neg;
 			if (check_set(&query->filename, arg, "file"))
 				return -EINVAL;
 
@@ -464,16 +487,26 @@ static int ddebug_parse_query(char *words[], int nwords,
 					return -EINVAL;
 			}
 		} else if (!strcmp(keyword, "module")) {
+			query->module_neg = neg;
 			rc = check_set(&query->module, arg, "module");
 		} else if (!strcmp(keyword, "format")) {
 			string_unescape_inplace(arg, UNESCAPE_SPACE |
 							    UNESCAPE_OCTAL |
 							    UNESCAPE_SPECIAL);
+			query->format_neg = neg;
 			rc = check_set(&query->format, arg, "format");
 		} else if (!strcmp(keyword, "line")) {
+			if (neg) {
+				pr_err("negation not supported for \"line\"\n");
+				return -EINVAL;
+			}
 			if (parse_linerange(query, arg))
 				return -EINVAL;
 		} else if (!strcmp(keyword, "class")) {
+			if (neg) {
+				pr_err("negation not supported for \"class\"\n");
+				return -EINVAL;
+			}
 			rc = check_set(&query->class_string, arg, "class");
 		} else {
 			pr_err("unknown keyword \"%s\"\n", keyword);
Signature · 1 line
2.53.0