jump_label: mitigate dynamic-debug induced IPI storms
Subsystems (4)
Jim Cromie
maintainercover jump_label: mitigate dynamic-debug induced IPI storms
This series adds batched toggling of jump_labels to address IPI storms triggered by bulk toggles of dynamic-debug pr_debug()s.
patches 1-3 are for submission. 4-5 are for demonstration only.
The Scaling Problem:
In testing on 7.1-rc6 guests (virtme-ng) hosted on a stock Fedora kernel, I measured the cost of toggling ~1,900 dyndbg callsites. The current one-at-a-time patching approach creates a massive synchronization bottleneck that scales linearly with both key count and core count:
Baseline IPIs (Total CAL Interrupts)
Successive query pairs (+p ; -p ;) toggling ~3,792 keys per pair. Pairs | -p 2 | -p 3 | -p 4 | -p 5 | Notes
-----------------------------------------------------------
1 | 10,738 | 21,468 | 32,148 | 42,837 | ~11.3 IPI/key/CPU 2 | 21,269 | 42,976 | 64,416 | 85,880 | Linear scaling (K*M) 3 | 32,125 | 64,434 | 96,604 | 128,793 | 4 | 42,831 | 85,886 | 128,776 | 171,697 | 5 | 53,540 | 107,353 | 160,985 | 214,636 | 6 | 64,174 | 128,800 | 193,159 | 257,562 | > 250k IPI storm
On a 5-VCPU system, a sequence of 6 query pairs triggers over a quarter-million interrupts. This overhead is a direct result of invoking the full patching machinery for every individual key toggle.
The Solution:
While x86 already has a 256-entry queue for jump_label, 2 issues currently block its use:
1. dyndbg uses the synchronous API; an IPI is issued for every static-key. patch-1 provides a fix for this with a _queued api extension. patch-3 uses the new api in dyndbg.
2. The queue flushes if current_patch_addr < previous_patch_addr. dyndbg has consecutive descriptors, but not consecutive patch_addrs. Using the API with just patch-1 yields ~1/2 the IPIs. Patch 2 sorts the patch_addrs, allowing the queue to fill to its 256-entry capacity before flushing, for a bigger win.
Results Summary:
Jump-Batch IPIs (Total CAL Interrupts)
Pairs | -p 2 | -p 3 | -p 4 | -p 5 |
---------------------------------------
1 | 135 | 202 | 239 | 290 | 2 | 145 | 256 | 347 | 445 | 3 | 198 | 352 | 479 | 640 | 4 | 246 | 445 | 630 | 840 | 5 | 293 | 545 | 774 | 1,033 | 6 | 341 | 642 | 915 | 1,222 |
Marginal Cost Comparison (IPIs added per '+p;-p;' query pair) VCPUs | Baseline (m) | Jump-Batch (m) | Efficiency Gain
-------------------------------------------------------
2 | 10,696 | 42.4 | 252x 3 | 21,466 | 89.3 | 240x 4 | 32,207 | 138.1 | 233x 5 | 42,945 | 190.5 | 225x
Why not static_key_deferred?
static_key_deferred is a rate-limiter for single, high-frequency keys. For a bulk update of 1,900 distinct keys, a deferred timer would still fire 1,900 times, resulting in the same IPI storm spread across a longer window. This will make it harder to use, particularly for tools which are fast enough to perceive the asynchrony. In contrast, jump-batch just amortizes the IPI cost, eliminating the need for rate-throttling.
Why wasnt this done previously ?
The presence of the queue for x86 suggests it was done, minimally, for the cases where the patch-addresses were ordered, and where performance mattered more (ftrace etc). In contrast, pr_debug()s are pretty basic, for direct use by humans, who dont toggle them on and off at high rates, and who are far more selective than "+p" the 2nd time they use dyndbg.
Model Summary:
The baseline cost is driven by (Keys * CPUs), resulting in 43k IPIs per transaction on a 5-VCPU VM. The jump-batch solution reduces this to ~190 IPIs, moving the bottleneck from key density (K) to architectural rendezvous cost (M).
Detailed Data:
NB: most tests below used the " !module virtio* +p ; -p ; " query-pair, relying on patch-5 for the "!module virtio*" parsing to avoid flooding the guest/VM logs with messages. This is less important when both on;off are sent together, as console flooding impedes submitting a separate off command.
Baseline:
jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 0 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 0 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 0 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10387 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 10111 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 161 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 28 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 10162 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10418 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10390 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10379 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 10467 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 9812 jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20856 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 20277 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 163 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 97 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 20259 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20856 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20861 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20818 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 20848 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 20275 jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31237 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 30349 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 171 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 101 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 30367 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31224 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31233 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31223 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 31219 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 30346 jimc@frodo:~/projects/lx/linux.git$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41593 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 40447 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 174 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 91 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 40451 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41546 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41595 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41622 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1897 Delta-CAL (IPI): 41607 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 40442
Patched:
jimc@frodo:~/projects/lx/wk-A$ . ../farmit.sh jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 1 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 0 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 0 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 0 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 0 jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 2 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 131 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 127 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 130 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 78 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 124 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 116 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 134 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 141 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 153 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 124 jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 3 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 359 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 200 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 175 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 100 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 191 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 186 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 186 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 171 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 181 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 182 jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 4 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 235 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 225 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 173 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 93 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 227 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 294 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 250 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 273 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 307 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 266 jimc@frodo:~/projects/lx/wk-A$ for_bo_ -p 5 ../../dd-tools.rc dd_ipis b0-dd: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 277 b0-dd-drm: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 265 b0-defcon: doing: dd_ipis ../../dd-tools.rc: line 79: /proc/dynamic_debug/control: No such file or directory cat: /tmp/dd_err: No such file or directory ERROR (0): wc: /proc/dynamic_debug/control: No such file or directory key-ct: Delta-CAL (IPI): 189 b0-ddc: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1 Delta-CAL (IPI): 96 b0-dd-drm-all: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 267 b0-dd-tym: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 278 b0-dd-tyy: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 272 b0-ksan: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 285 b0-ksan-leak: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1896 Delta-CAL (IPI): 280 b0-ftrace: doing: dd_ipis SUCCESS: !module virtio* +p ; -p key-ct: 1849 Delta-CAL (IPI): 265
LINEAR PROGRESSION
both progressions are linear, but patched has much lower slope and intercept (y=mx+b)
Baseline:
jimc@frodo:~/projects/lx/linux.git$ cd b0-dd jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 10738 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 21269 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 32125 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 42831 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 53540 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 64174 jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 21468 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 42976 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 64434 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 85886 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 107353 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 128800 jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 32148 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 64416 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 96604 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 128776 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 160985 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 193159 jimc@frodo:~/projects/lx/linux.git/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-71-g347c80eb3607 --user root --rwdir=/home/jimc/projects/lx/linux.git/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 42837 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 85880 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 128793 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 171697 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 214636 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1897 Delta-CAL (IPI): 257562 jimc@frodo:~/projects/lx/linux.git/b0-dd$
PATCHED:
jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 2 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 2 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 135 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 145 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 198 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 246 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 293 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 341 jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 3 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 3 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 202 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 256 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 352 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 445 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 545 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 642 jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 4 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 4 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 239 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 347 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 479 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 630 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 774 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 915 jimc@frodo:~/projects/lx/wk-A/b0-dd$ vrun_ -p 5 ../../dd-tools.rc dd_linear virtme-ng 1.40 doing: vng --name v7.0-rc6-5-g6a8736c5bd10 --user root --rwdir=/home/jimc/projects/lx/wk-A/b0-dd -a dynamic_debug.verbose=1 -p 4 -p 5 ../../dd-tools.rc dd_linear doing: dd_linear SUCCESS: +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 290 SUCCESS: +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 445 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 640 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 840 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 1033 SUCCESS: +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; +p ; -p ; key-ct: 1896 Delta-CAL (IPI): 1222 jimc@frodo:~/projects/lx/wk-A/b0-dd$
Acknowldegement:
I used google-gemini extensively to discuss and review this work, and to read the detailed data, and prepare the summary tables and prose above. It is patient with my pedantry.
Jim Cromie (5):
jump_label: add queueing API for batched static key updates x86/alternative.c: sort text-pokes before flushing the queue
dyndbg: use static-key queueing API in dynamic-debug
dyndbg: factor ddebug_match_desc out from ddebug_change (for
demonstration only)
lib/dynamic_debug: add negation support to queries
arch/x86/kernel/alternative.c | 42 +++++------ include/linux/jump_label.h | 18 +++++ kernel/jump_label.c | 104 +++++++++++++++++++------ lib/dynamic_debug.c | 138 ++++++++++++++++++++++------------ 4 files changed, 211 insertions(+), 91 deletions(-)
base-commit: 7aaa8047eafd0bd628065b15757d9b48c5f9c07d
Signature · 1 line
2.53.0
Jim Cromie
maintainerPATCH 1/5 jump_label: add queueing API for batched static key updates
Introduce a new queueing API for static keys to allow batching of instruction updates. This is provided for subsystems like dynamic-debug, which may toggle thousands of static keys in a single operation. Currently, this suffers from (N)$ overhead due to repeated machine-wide stop_machine or text_poke_sync synchronizations.
The new API includes:
- static_key_enable_queued(key)
- static_key_disable_queued(key)
- static_key_apply_queued() (the global synchronization barrier)
- Corresponding static_branch_*_queued(x) macros.
The _queued functionality is achieved by adding a 'bool apply' parameter to 3 existing midlayer static functions, and using it to decide whether or not to call arch_jump_label_transform_apply(). By not explicitly calling it, we allow the transform queue to fill 1st. This amortizes the cost of the IPI sync to 1/N (256 on x86).
1. __jump_label_update(struct static_key *key, bool apply) this implements the if (apply) arch_jump_label_transform_apply();
2. __jump_label_mod_update(struct static_key *key, bool apply) extends _queued feature to modules' static-keys
3. rename jump_label_update(struct static_key *key)
to __jump_label_update_key(struct static_key *key, bool apply)
and wrap it with 1-liners:
jump_label_update() & jump_label_update_queued()
On architectures supporting HAVE_JUMP_LABEL_BATCH, the _queued variants lazily fill the architecture's patch queue.
It is safe to mix queued and immediate updates; any non-queued static_key_enable/disable call will effectively "flush" any previously queued changes as a side effect. Or just call static_key_apply_queued().
On architectures that do not need a patch queue (e.g., those with atomic patching) or that do not support batching, the API transparently falls back to immediate patching.
--- include/linux/jump_label.h | 18 +++++++ kernel/jump_label.c | 104 +++++++++++++++++++++++++++++-------- 2 files changed, 100 insertions(+), 22 deletions(-)
include/linux/jump_label.h +18 -0
@@ -234,6 +234,9 @@ extern void static_key_slow_dec_cpuslocked(struct static_key *key);extern int static_key_count (struct static_key *key);extern void static_key_enable (struct static_key *key);extern void static_key_disable (struct static_key *key); +extern void static_key_enable_queued (struct static_key *key); +extern void static_key_disable_queued (struct static_key *key); +extern void static_key_apply_queued (void );extern void static_key_enable_cpuslocked (struct static_key *key);extern void static_key_disable_cpuslocked (struct static_key *key);extern enum jump_label_type jump_label_init_type (struct jump_entry *entry);
@@ -340,6 +343,18 @@ static inline void static_key_disable(struct static_key *key)atomic_set (&key->enabled,0 ); } +static inline void static_key_enable_queued (struct static_key *key) +{ +static_key_enable (key); +} + +static inline void static_key_disable_queued (struct static_key *key) +{ +static_key_disable (key); +} + +static inline void static_key_apply_queued (void ) {} +#define static_key_enable_cpuslocked (k) static_key_enable((k))#define static_key_disable_cpuslocked (k) static_key_disable((k))
@@ -535,6 +550,9 @@ extern bool ____wrong_branch_error(void);#define static_branch_enable (x) static_key_enable(&(x)->key)#define static_branch_disable (x) static_key_disable(&(x)->key) +#define static_branch_enable_queued (x) static_key_enable_queued(&(x)->key) +#define static_branch_disable_queued (x) static_key_disable_queued(&(x)->key) +#define static_branch_apply_queued () static_key_apply_queued()#define static_branch_enable_cpuslocked (x) static_key_enable_cpuslocked(&(x)->key)#define static_branch_disable_cpuslocked (x) static_key_disable_cpuslocked(&(x)->key)
kernel/jump_label.c +82 -22
178 lines not on this page · raw message
Signature · 1 line
2.53.0
Jim Cromie
maintainerPATCH 2/5 x86/alternative.c: sort text-pokes before flushing the queue
In reply to Jim CromieUntil now, x86 can queue jump_label text-pokes as long as the poke-addr is monotonically increasing, but flushes the queue when a new poke-addr is less than the previous.
Dynamic-debug now uses the queued static-key API, but the advantage is limited; we see a ~2x reduction in IPIs. Although the pr_debug descriptors are ordered, the patch-address in them are not; about 1/2 violate the ordering constraint.
So this patch drops that requirement, and sorts the text-pokes by their address before applying them. Doing so lets us fill the queue before sorting then flushing, giving a dramatic ~125x reduction in IPIs over the traditional single IPI per pr_debug.
Other arches don't need a queue, and so have nothing to sort.
#> dd_ipis
[ 23.100381] dyndbg: query 0: "module !virtio* +p " mod:* [ 23.103432] dyndbg: query 1: "-p" mod:* Delta-CAL (IPI): 242
--- arch/x86/kernel/alternative.c | 42 ++++++++++++++++------------------- lib/dynamic_debug.c | 1 + 2 files changed, 20 insertions(+), 23 deletions(-)
arch/x86/kernel/alternative.c +19 -23
@@ -2,6 +2,7 @@#define pr_fmt (fmt) "SMP alternatives: " fmt#include <linux/mmu_context.h> +#include <linux/sort.h> #include <linux/perf_event.h> #include <linux/vmalloc.h> #include <linux/memory.h>
@@ -2823,6 +2824,18 @@ static __always_inline int patch_cmp(const void *tpl_a, const void *tpl_b)return 0 ; } +static int text_poke_loc_cmp (const void *a,const void *b) +{ +const struct smp_text_poke_loc *tpl_a = a; +const struct smp_text_poke_loc *tpl_b = b; + +if (tpl_a->rel_addr < tpl_b->rel_addr) +return -1 ; +if (tpl_a->rel_addr > tpl_b->rel_addr) +return 1 ; +return 0 ; +} + noinstr int smp_text_poke_int3_handler(struct pt_regs *regs) {struct smp_text_poke_loc *tpl;
@@ -2935,6 +2948,10 @@ void smp_text_poke_batch_finish(void)if (!text_poke_array.nr_entries)return ; +if (text_poke_array.nr_entries >1 ) +sort (text_poke_array.vec, text_poke_array.nr_entries, +sizeof (struct smp_text_poke_loc ), text_poke_loc_cmp,NULL ); +lockdep_assert_held (&text_mutex);/*
@@ -3151,28 +3168,6 @@ static void __smp_text_poke_batch_add(void *addr, const void *opcode, size_t len } } -/* -* We hard rely on the text_poke_array.vec being ordered; ensure this is so by flushing -* early if needed. -*/ -static bool text_poke_addr_ordered (void *addr) -{ -WARN_ON_ONCE (!addr); - -if (!text_poke_array.nr_entries) -return true; - -/* -* If the last current entry's address is higher than the -* new entry's address we'd like to add, then ordering -* is violated and we must first flush all pending patching -* requests: -*/ -if (text_poke_addr (text_poke_array.vec + text_poke_array.nr_entries-1 ) > addr) -return false; - -return true; -}/** * smp_text_poke_batch_add() -- update instruction on live kernel on SMP, batched
@@ -3189,8 +3184,9 @@ static bool text_poke_addr_ordered(void *addr)*/ void __ref smp_text_poke_batch_add (void *addr,const void *opcode,size_t len,const void *emulate) { -if (text_poke_array.nr_entries ==TEXT_POKE_ARRAY_MAX || !text_poke_addr_ordered (addr)) +if (text_poke_array.nr_entries ==TEXT_POKE_ARRAY_MAX )smp_text_poke_batch_finish (); +__smp_text_poke_batch_add (addr, opcode, len, emulate); }
lib/dynamic_debug.c +1 -0
@@ -264,6 +264,7 @@ static int ddebug_change(const struct ddebug_query *query, } }mutex_unlock (&ddebug_lock); +v2pr_info ("applied %d queued updates to sites in total\n" , nfound);if (!nfound && verbose)pr_info ("no matches for query\n" );
Signature · 1 line
2.53.0
Jim Cromie
maintainerPATCH 3/5 dyndbg: use static-key queueing API in dynamic-debug
In reply to Jim CromieUse the new static-key queueing API in dynamic-debug to reduce IPIs.
this replaces the 2 calls to static_branch_{enable.disable}() in ddebug_change() with their _queued() counterparts, and adds a call to static_branch_apply_queued() at the bottom.
This gives a theoretical reduction of IPIs up to 256x, and a practical reduction by 120-180x.
--- lib/dynamic_debug.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
lib/dynamic_debug.c +3 -2
@@ -250,9 +250,9 @@ static int ddebug_change(const struct ddebug_query *query,#ifdef CONFIG_JUMP_LABEL if (dp->flags & _DPRINTK_FLAGS_PRINT) {if (!(newflags & _DPRINTK_FLAGS_PRINT)) -static_branch_disable (&dp->key.dd_key_true); +static_branch_disable_queued (&dp->key.dd_key_true); }else if (newflags & _DPRINTK_FLAGS_PRINT) { -static_branch_enable (&dp->key.dd_key_true); +static_branch_enable_queued (&dp->key.dd_key_true); }#endif v4pr_info("changed %s:%d [%s]%s %s => %s\n",
@@ -263,6 +263,7 @@ static int ddebug_change(const struct ddebug_query *query, dp->flags = newflags; } } +static_branch_apply_queued ();mutex_unlock (&ddebug_lock);v2pr_info ("applied %d queued updates to sites in total\n" , nfound);
Signature · 1 line
2.53.0
Jim Cromie
maintainerPATCH 4/5 dyndbg: factor ddebug_match_desc out from ddebug_change (for demonstration only)
In reply to Jim Cromieddebug_change() is a big (~100 lines) function with a nested for loop.
The outer loop walks the per-module ddebug_tables list, and does module stuff: it filters on a query's "module FOO*" and "class BAR", failures here skip the entire inner loop.
The inner loop (60 lines) scans a module's descriptors. It starts with a long block of filters on function, line, format, and the validated "BAR" class (or the legacy/_DPRINTK_CLASS_DFLT).
These filters "continue" past pr_debugs that don't match the query criteria, before it falls through the code below that counts matches, then adjusts the flags and static-keys. This is unnecessarily hard to think about.
So move the per-descriptor filter-block into a boolean function:ddebug_match_desc(desc), and change each "continue" to "return false". This puts a clear interface in place, so any future changes are either inside, outside, or across this interface.
also fix checkpatch complaints about spaces and braces.
--- lib/dynamic_debug.c | 83 +++++++++++++++++++++++++-------------------- 1 file changed, 47 insertions(+), 36 deletions(-)
lib/dynamic_debug.c +47 -36
@@ -172,6 +172,52 @@ static struct ddebug_class_map *ddebug_find_valid_class(struct ddebug_table cons* callsites, normally the same as number of changes. If verbose, * logs the changes. Takes ddebug_lock. */ +static bool ddebug_match_desc (const struct ddebug_query *query, +struct _ddebug *dp, +int valid_class) +{ +/* match site against query-class */ +if (dp->class_id != valid_class) +return false; + +/* match against the source filename */ +if (query->filename && + !match_wildcard (query->filename, dp->filename) && + !match_wildcard (query->filename, +kbasename (dp->filename)) && + !match_wildcard (query->filename, +trim_prefix (dp->filename))) +return false; + +/* match against the function */ +if (query->function && + !match_wildcard (query->function, dp->function)) +return false; + +/* match against the format */ +if (query->format) { +if (*query->format =='^' ) { +char *p; +/* anchored search. match must be at beginning */ + p =strstr (dp->format, query->format +1 ); +if (p != dp->format) +return false; + }else if (!strstr (dp->format, query->format)) { +return false; + } + } + +/* match against the line number range */ +if (query->first_lineno && + dp->lineno < query->first_lineno) +return false; +if (query->last_lineno && + dp->lineno > query->last_lineno) +return false; + +return true; +} +static int ddebug_change (const struct ddebug_query *query,struct flag_settings *modifiers) {
@@ -204,42 +250,7 @@ static int ddebug_change(const struct ddebug_query *query,for (i =0 ; i < dt->num_ddebugs; i++) {struct _ddebug *dp = &dt->ddebugs[i]; -/* match site against query-class */ -if (dp->class_id != valid_class) -continue ; - -/* match against the source filename */ -if (query->filename && - !match_wildcard (query->filename, dp->filename) && - !match_wildcard (query->filename, -kbasename (dp->filename)) && - !match_wildcard (query->filename, -trim_prefix (dp->filename))) -continue ; - -/* match against the function */ -if (query->function && - !match_wildcard (query->function, dp->function)) -continue ; - -/* match against the format */ -if (query->format) { -if (*query->format =='^' ) { -char *p; -/* anchored search. match must be at beginning */ - p =strstr (dp->format, query->format+1 ); -if (p != dp->format) -continue ; - }else if (!strstr (dp->format, query->format)) -continue ; - } - -/* match against the line number range */ -if (query->first_lineno && - dp->lineno < query->first_lineno) -continue ; -if (query->last_lineno && - dp->lineno > query->last_lineno) +if (!ddebug_match_desc (query, dp, valid_class))continue ; nfound++;
Signature · 1 line
2.53.0
Jim Cromie
maintainerPATCH 5/5 lib/dynamic_debug: add negation support to queries
In reply to Jim CromieThis allow users to invert the selection of most keywords, excluding lineno and class. For example:
echo "!module virtio* +p" > /proc/dynamic_debug/control
When testing a kernel running in virtme-ng, this cmd prevents flooding the logs with virtio activity, and spoiling the the test scenario.
NOTE:
This is convenient, but not perfect; it cannot also avoid flooding from pr_debugs in serial_core or other potential sources.
A more robust command is:
echo "!module virtio* +p % module serial -p" > /proc/dynamic_debug/control
It also "interferes" with evolution of the grammar; particularly around and/or combinations. For example this could become meaningful, though perhaps inadvisable:
echo " !module virtio* !module serial +p "
lineno is currently excluded, pending review of its usefulness vs ambiguities, for example " !lineno 1-20 " has straightforward meaning.
!class FOO is unlikely to be justifiable; it defeats the "protection thru specificity" that classes are designed for.
--- lib/dynamic_debug.c | 81 +++++++++++++++++++++++++++++++-------------- 1 file changed, 57 insertions(+), 24 deletions(-)
lib/dynamic_debug.c +57 -24
@@ -58,6 +58,10 @@ struct ddebug_query {const char *format;const char *class_string;unsigned int first_lineno, last_lineno; + unsigned int filename_neg:1; + unsigned int module_neg:1; + unsigned int function_neg:1; + unsigned int format_neg:1; };struct ddebug_iter {
@@ -139,11 +143,12 @@ static void vpr_info_dq(const struct ddebug_query *query, const char *msg) fmtlen--; } -v3pr_info ("%s: func=\"%s\" file=\"%s\" module=\"%s\" format=\"%.*s\" lineno=%u-%u class=%s\n" , +v3pr_info ("%s: func%s=\"%s\" file%s=\"%s\" module%s=\"%s\" format%s=\"%.*s\" lineno=%u-%u class=%s\n" , msg, - query->function ?:"" , - query->filename ?:"" , - query->module ?:"" , + query->function_neg ?"!" :"" , query->function ?:"" , + query->filename_neg ?"!" :"" , query->filename ?:"" , + query->module_neg ?"!" :"" , query->module ?:"" , + query->format_neg ?"!" :"" , fmtlen, query->format ?:"" , query->first_lineno, query->last_lineno, query->class_string); }
@@ -176,35 +181,38 @@ static bool ddebug_match_desc(const struct ddebug_query *query,struct _ddebug *dp, int valid_class) { + bool match; +/* match site against query-class */ if (dp->class_id != valid_class)return false;/* match against the source filename */ -if (query->filename && - !match_wildcard (query->filename, dp->filename) && - !match_wildcard (query->filename, -kbasename (dp->filename)) && - !match_wildcard (query->filename, -trim_prefix (dp->filename))) -return false; +if (query->filename) { + match =match_wildcard (query->filename, dp->filename) || +match_wildcard (query->filename,kbasename (dp->filename)) || +match_wildcard (query->filename,trim_prefix (dp->filename)); +if (match == query->filename_neg) +return false; + }/* match against the function */ -if (query->function && - !match_wildcard (query->function, dp->function)) -return false; +if (query->function) { + match =match_wildcard (query->function, dp->function); +if (match == query->function_neg) +return false; + }/* match against the format */ if (query->format) {if (*query->format =='^' ) { -char *p;/* anchored search. match must be at beginning */ - p =strstr (dp->format, query->format +1 ); -if (p != dp->format) -return false; - }else if (!strstr (dp->format, query->format)) { -return false; + match = (strstr (dp->format, query->format +1 ) == dp->format); + }else { + match = !!strstr (dp->format, query->format); } +if (match == query->format_neg) +return false; }/* match against the line number range */
@@ -234,9 +242,11 @@ static int ddebug_change(const struct ddebug_query *query,list_for_each_entry (dt, &ddebug_tables, link) {/* match against the module name */ -if (query->module && - !match_wildcard (query->module, dt->mod_name)) -continue ; +if (query->module) { +bool match =match_wildcard (query->module, dt->mod_name); +if (match == query->module_neg) +continue ; + }if (query->class_string) { map =ddebug_find_valid_class (dt, query->class_string, &valid_class);
@@ -399,6 +409,16 @@ static int parse_linerange(struct ddebug_query *query, const char *first)return 0 ; } +static char *check_neg (char *src,unsigned int *neg) +{ +if (*src =='!' ) { + *neg =1 ; +return src +1 ; + } + *neg =0 ; +return src; +} +static int check_set (const char **dest,char *src,char *name) {int rc =0 ;
@@ -441,12 +461,15 @@ static int ddebug_parse_query(char *words[], int nwords, }for (i =0 ; i < nwords; i +=2 ) { -char *keyword = words[i]; +unsigned int neg; +char *keyword =check_neg (words[i], &neg);char *arg = words[i+1 ];if (!strcmp (keyword,"func" )) { + query->function_neg = neg; rc =check_set (&query->function, arg,"func" ); }else if (!strcmp (keyword,"file" )) { + query->filename_neg = neg;if (check_set (&query->filename, arg,"file" ))return -EINVAL ;
@@ -464,16 +487,26 @@ static int ddebug_parse_query(char *words[], int nwords,return -EINVAL ; } }else if (!strcmp (keyword,"module" )) { + query->module_neg = neg; rc =check_set (&query->module, arg,"module" ); }else if (!strcmp (keyword,"format" )) {string_unescape_inplace (arg,UNESCAPE_SPACE |UNESCAPE_OCTAL |UNESCAPE_SPECIAL ); + query->format_neg = neg; rc =check_set (&query->format, arg,"format" ); }else if (!strcmp (keyword,"line" )) { +if (neg) { +pr_err ("negation not supported for \"line\"\n" ); +return -EINVAL ; + }if (parse_linerange (query, arg))return -EINVAL ; }else if (!strcmp (keyword,"class" )) { +if (neg) { +pr_err ("negation not supported for \"class\"\n" ); +return -EINVAL ; + } rc =check_set (&query->class_string, arg,"class" ); }else {pr_err ("unknown keyword \"%s\"\n" , keyword);
Signature · 1 line
2.53.0