Fix DTrace on modern kernels and in containers

Without this fix, running any DTrace commands results in:

```
dtrace: invalid probe specifier ::::
"/usr/lib64/dtrace/6.12.0/procfs.d", line 131: operator -> cannot be
applied to pointer to type "void"; must be applied to a struct or
union pointer
```

The issue occurs on kernels which have BTF with type tags. The fix is to treat BTF_KIND_TYPE_TAG as a transparent modifier, resolving straight to type->type.

Signed-off-by Jacob Carlborg <[email protected]>

---
 libdtrace/dt_btf.c | 20 +++++++++++++++++++-
 1 file changed, 19 insertions(+), 1 deletion(-)

libdtrace/dt_btf.c +19 -2

index 1f793397..1414dd1e 100644
@@ -672,10 +672,28 @@ dt_btf_add_to_ctf(dtrace_hdl_t *dtp, dt_btf_t *btf, ctf_dict_t *ctf,
 
                return ctfid == CTF_ERR ? dt_ctf_error(dtp, ctf) : ctfid;
        }
+       case BTF_KIND_TYPE_TAG: {
+               /*
+                * A type tag (e.g. __rcu, __user, __percpu) is a transparent
+                * annotation on the type it wraps.  CTF has no equivalent, so
+                * we resolve straight through to the underlying type, just as
+                * we do for const/volatile/restrict.  Treating it as ignored
+                * (mapping it to void) would leave every tagged member -- such
+                * as task_struct.real_parent (__rcu) -- resolving to "void",
+                * which breaks translator compilation on kernels whose BTF
+                * carries type tags.
+                */
+               ctfid = dt_btf_add_to_ctf(dtp, btf, ctf, type->type);
+               if (ctfid == CTF_ERR)
+                       return CTF_ERR;         /* errno already set */
+
+               btf->ctfids[type_id] = ctfid;
+
+               return ctfid;
+       }
        case BTF_KIND_VAR:
        case BTF_KIND_DATASEC:
        case BTF_KIND_DECL_TAG:
-       case BTF_KIND_TYPE_TAG:
        case BTF_KIND_ENUM64:
        case BTF_KIND_FUNC:
                return btf->ctfids[0];          /* Ignored for CTF */
--

2.50.1 (Apple Git-155)

Hi, I've created three patches for the issues I've encountered when trying to get DTrace to work in my environments. One environment is on my local computer running macOS ARM64 using Docker through OrbStack. The other is our Kubernetes cluster

Hi, I've created three patches for the issues I've encountered when trying to get DTrace to work in my environments. One environment is on my local computer running macOS ARM64 using Docker through OrbStack. The other is our Kubernetes cluster in the cloud. The first two issues are differences in the Linux kernel compared to what DTrace expects. The third issue is container related.

The OrbStack environment is running version 7.0.11 of the Linux kernel. The Kubernetes cluster is running 6.12.85.

I'll be honest, the analysis of the issues and the patches are all done by an AI. I've verified everything manually myself.

Jacob Carlborg (3):
libdtrace: resolve BTF type tags when importing BTF into CTF
libdtrace: derive io provider dev_name from the device model
libdtrace: match BEGIN/END PID in the tracer's own PID namespace

 libdtrace/dt_btf.c         | 20 ++++++++++++-
 libdtrace/dt_prov_dtrace.c | 58 ++++++++++++++++++++++++++++++++++----
 libdtrace/io.d.in          | 17 +++++++++--
 3 files changed, 86 insertions(+), 9 deletions(-)

--
2.50.1 (Apple Git-155)

Thanks for reporting these issues and sending patches. It ceetainly highlights an area where our testing is lacking, i.e. containers and related namespace issues. That will need some attention as well.

Quoted 32 lines · Jacob Carlborg

Hi, I've created three patches for the issues I've encountered when trying to get DTrace to work in my environments. One environment is on my local computer running macOS ARM64 using Docker through OrbStack. The other is our Kubernetes cluster

Hi, I've created three patches for the issues I've encountered when trying to get DTrace to work in my environments. One environment is on my local computer running macOS ARM64 using Docker through OrbStack. The other is our Kubernetes cluster in the cloud. The first two issues are differences in the Linux kernel compared to what DTrace expects. The third issue is container related.

The OrbStack environment is running version 7.0.11 of the Linux kernel. The Kubernetes cluster is running 6.12.85.

I'll be honest, the analysis of the issues and the patches are all done by an AI. I've verified everything manually myself.

Jacob Carlborg (3):
libdtrace: resolve BTF type tags when importing BTF into CTF
libdtrace: derive io provider dev_name from the device model
libdtrace: match BEGIN/END PID in the tracer's own PID namespace

libdtrace/dt_btf.c         | 20 ++++++++++++-
libdtrace/dt_prov_dtrace.c | 58 ++++++++++++++++++++++++++++++++++----
libdtrace/io.d.in          | 17 +++++++++--
3 files changed, 86 insertions(+), 9 deletions(-)

--
2.50.1 (Apple Git-155)

Without this fix, running any probe hangs with no output, like dtrace -n 'BEGIN { trace("hi"); exit(0); }'.

The issues is that the BEGIN/END probes are implemented as uprobes on DTrace's own BEGIN_probe()/END_probe() functions in dt_work.c. bpf_get_current_pid_tgid() reports the TGID in the initial PID namespace, which does not match getpid() when DTrace runs inside a PID namespace. Using the --pid=host flag when running OrbStack does not help, because it shares the daemon's namespace, not the kernel init namespace. OrbStack's own supervisor runs in a nested PID namespace. On bare metal --pid=host does reach the init namespaces and the bug wouldn't appear.

This fix is when DTrace is in a non-initial PID namespace and the kernel is at least 5.7, use bpf_get_ns_current_pid_tgid instead of bpf_get_current_pid_tgid. bpf_get_ns_current_pid_tgid is namespace aware.

Signed-off-by Jacob Carlborg <[email protected]>

---
 libdtrace/dt_prov_dtrace.c | 58 ++++++++++++++++++++++++++++++++++----
 1 file changed, 53 insertions(+), 5 deletions(-)

libdtrace/dt_prov_dtrace.c +53 -6

index 4b788507..179ee4d6 100644
@@ -9,6 +9,7 @@
 #include <assert.h>
 #include <errno.h>
 #include <string.h>
+#include <sys/stat.h>
 
 #include <bpf_asm.h>
 

@@ -129,12 +130,59 @@ static int trampoline(dt_pcb_t *pcb, uint_t exitlbl)
         * the trampoline to minimize the cost of pointless firings in other
         * tracers, even though this means preserving the context in %r1 around
         * the call.
+        *
+        * bpf_get_current_pid_tgid() reports the TGID in the initial PID
+        * namespace, which does not match getpid() when DTrace runs inside a
+        * PID namespace (e.g. in a container) -- so the BEGIN/END probes would
+        * never recognise their own firing and tracing would never activate.
+        * When we are in a non-initial PID namespace on a kernel that provides
+        * it (5.7+), use bpf_get_ns_current_pid_tgid() with DTrace's own PID
+        * namespace so the value is reported in the same namespace as getpid().
+        * Otherwise fall back to bpf_get_current_pid_tgid(), leaving the
+        * initial-namespace case (and kernels < 5.7) exactly as before.
         */
-       emit(dlp, BPF_MOV_REG(BPF_REG_6, BPF_REG_1));
-       emit(dlp, BPF_CALL_HELPER(BPF_FUNC_get_current_pid_tgid));
-       emit(dlp, BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32));
-       emit(dlp, BPF_BRANCH_IMM(BPF_JNE, BPF_REG_0, getpid(), pcb->pcb_fastlbl));
-       emit(dlp, BPF_MOV_REG(BPF_REG_1, BPF_REG_6));
+       {
+               struct stat     st;
+               int             use_ns = 0;
+               uint64_t        dev = 0, ino = 0;
+
+               /*
+                * The initial PID namespace has a fixed inode number
+                * (PROC_PID_INIT_INO, 0xEFFFFFFC).  If we are in it, getpid()
+                * already agrees with bpf_get_current_pid_tgid() and no
+                * namespace lookup is needed.
+                */
+               if (stat("/proc/self/ns/pid", &st) == 0 &&
+                   st.st_ino != 0xEFFFFFFCULL &&
+                   pcb->pcb_hdl->dt_kernver >= DT_VERSION_NUMBER(5, 7, 0)) {
+                       use_ns = 1;
+                       dev = st.st_dev;
+                       ino = st.st_ino;
+               }
+
+               emit(dlp, BPF_MOV_REG(BPF_REG_6, BPF_REG_1));
+               if (use_ns) {
+                       /*
+                        * bpf_get_ns_current_pid_tgid(dev, ino, &nsinfo, sz)
+                        * fills a struct bpf_pidns_info { u32 pid; u32 tgid; };
+                        * the tgid is at offset 4.
+                        */
+                       dt_cg_xsetx(dlp, NULL, DT_LBL_NONE, BPF_REG_1, dev);
+                       dt_cg_xsetx(dlp, NULL, DT_LBL_NONE, BPF_REG_2, ino);
+                       emit(dlp, BPF_MOV_REG(BPF_REG_3, BPF_REG_FP));
+                       emit(dlp, BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, DT_TRAMP_SP_SLOT(0)));
+                       emit(dlp, BPF_MOV_IMM(BPF_REG_4, 8));
+                       emit(dlp, BPF_CALL_HELPER(BPF_FUNC_get_ns_current_pid_tgid));
+                       emit(dlp, BPF_LOAD(BPF_W, BPF_REG_0, BPF_REG_FP,
+                                          DT_TRAMP_SP_SLOT(0) + 4));
+               } else {
+                       emit(dlp, BPF_CALL_HELPER(BPF_FUNC_get_current_pid_tgid));
+                       emit(dlp, BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32));
+               }
+               emit(dlp, BPF_BRANCH_IMM(BPF_JNE, BPF_REG_0, getpid(),
+                                        pcb->pcb_fastlbl));
+               emit(dlp, BPF_MOV_REG(BPF_REG_1, BPF_REG_6));
+       }
 
        dt_cg_tramp_prologue_act(pcb, act);
 
--

2.50.1 (Apple Git-155)

Thank you for this patch. I actually implemented the code to get the dev/ino pair for the PID namespace in dt_vopen() so that we can store the values in dtrace_hdl_t. That way it is available to all places that need to retrieve pidns-aware PID and TGID values. I also added a function dt_cg_ns_pid_tgid() to encapsulate the code to use the correct BPF helper based on whether we are in the initial PID nameaspace or not, and I therefore modified your patch to simply call that function rather than doing the work here.

I'm updating the other places in the code that need to be pidns-aware, and will post the entire series once I am done with that.

I am also adding tests so that our testsuite going forward can ensure that DTrace works in a PID namespace.

Kris

Quoted 105 lines · Jacob Carlborg

Without this fix, running any probe hangs with no output, like dtrace -n 'BEGIN { trace("hi"); exit(0); }'.

The issues is that the BEGIN/END probes are implemented as uprobes on DTrace's own BEGIN_probe()/END_probe() functions in dt_work.c. bpf_get_current_pid_tgid() reports the TGID in the initial PID namespace, which does not match getpid() when DTrace runs inside a PID namespace. Using the --pid=host flag when running OrbStack does not help, because it shares the daemon's namespace, not the kernel init namespace. OrbStack's own supervisor runs in a nested PID namespace. On bare metal --pid=host does reach the init namespaces and the bug wouldn't appear.

This fix is when DTrace is in a non-initial PID namespace and the kernel is at least 5.7, use bpf_get_ns_current_pid_tgid instead of bpf_get_current_pid_tgid. bpf_get_ns_current_pid_tgid is namespace aware.

Signed-off-by Jacob Carlborg <[email protected]>

---
libdtrace/dt_prov_dtrace.c | 58 ++++++++++++++++++++++++++++++++++----

1 file changed, 53 insertions(+), 5 deletions(-)

libdtrace/dt_prov_dtrace.c +53 -6

index 4b788507..179ee4d6 100644
@@ -9,6 +9,7 @@
 #include <assert.h>
 #include <errno.h>
 #include <string.h>
+#include <sys/stat.h>
 
 #include <bpf_asm.h>
 

@@ -129,12 +130,59 @@ static int trampoline(dt_pcb_t *pcb, uint_t exitlbl)
        * the trampoline to minimize the cost of pointless firings in other
        * tracers, even though this means preserving the context in %r1 around
        * the call.
+        *
+        * bpf_get_current_pid_tgid() reports the TGID in the initial PID
+        * namespace, which does not match getpid() when DTrace runs inside a
+        * PID namespace (e.g. in a container) -- so the BEGIN/END probes would
+        * never recognise their own firing and tracing would never activate.
+        * When we are in a non-initial PID namespace on a kernel that provides
+        * it (5.7+), use bpf_get_ns_current_pid_tgid() with DTrace's own PID
+        * namespace so the value is reported in the same namespace as getpid().
+        * Otherwise fall back to bpf_get_current_pid_tgid(), leaving the
+        * initial-namespace case (and kernels < 5.7) exactly as before.
        */
-       emit(dlp, BPF_MOV_REG(BPF_REG_6, BPF_REG_1));
-       emit(dlp, BPF_CALL_HELPER(BPF_FUNC_get_current_pid_tgid));
-       emit(dlp, BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32));
-       emit(dlp, BPF_BRANCH_IMM(BPF_JNE, BPF_REG_0, getpid(), pcb->pcb_fastlbl));
-       emit(dlp, BPF_MOV_REG(BPF_REG_1, BPF_REG_6));
+       {
+               struct stat     st;
+               int             use_ns = 0;
+               uint64_t        dev = 0, ino = 0;
+
+               /*
+                * The initial PID namespace has a fixed inode number
+                * (PROC_PID_INIT_INO, 0xEFFFFFFC).  If we are in it, getpid()
+                * already agrees with bpf_get_current_pid_tgid() and no
+                * namespace lookup is needed.
+                */
+               if (stat("/proc/self/ns/pid", &st) == 0 &&
+                   st.st_ino != 0xEFFFFFFCULL &&
+                   pcb->pcb_hdl->dt_kernver >= DT_VERSION_NUMBER(5, 7, 0)) {
+                       use_ns = 1;
+                       dev = st.st_dev;
+                       ino = st.st_ino;
+               }
+
+               emit(dlp, BPF_MOV_REG(BPF_REG_6, BPF_REG_1));
+               if (use_ns) {
+                       /*
+                        * bpf_get_ns_current_pid_tgid(dev, ino, &nsinfo, sz)
+                        * fills a struct bpf_pidns_info { u32 pid; u32 tgid; };
+                        * the tgid is at offset 4.
+                        */
+                       dt_cg_xsetx(dlp, NULL, DT_LBL_NONE, BPF_REG_1, dev);
+                       dt_cg_xsetx(dlp, NULL, DT_LBL_NONE, BPF_REG_2, ino);
+                       emit(dlp, BPF_MOV_REG(BPF_REG_3, BPF_REG_FP));
+                       emit(dlp, BPF_ALU64_IMM(BPF_ADD, BPF_REG_3, DT_TRAMP_SP_SLOT(0)));
+                       emit(dlp, BPF_MOV_IMM(BPF_REG_4, 8));
+                       emit(dlp, BPF_CALL_HELPER(BPF_FUNC_get_ns_current_pid_tgid));
+                       emit(dlp, BPF_LOAD(BPF_W, BPF_REG_0, BPF_REG_FP,
+                                          DT_TRAMP_SP_SLOT(0) + 4));
+               } else {
+                       emit(dlp, BPF_CALL_HELPER(BPF_FUNC_get_current_pid_tgid));
+                       emit(dlp, BPF_ALU64_IMM(BPF_RSH, BPF_REG_0, 32));
+               }
+               emit(dlp, BPF_BRANCH_IMM(BPF_JNE, BPF_REG_0, getpid(),
+                                        pcb->pcb_fastlbl));
+               emit(dlp, BPF_MOV_REG(BPF_REG_1, BPF_REG_6));
+       }
 
       dt_cg_tramp_prologue_act(pcb, act);
 
--

2.50.1 (Apple Git-155)