bpf: Fix d_path test after last fs update
Jiri Olsa
reviewerRecent commit [1] broken d_path test, because now filp_close is not called directly from sys_close, but eventually later when the file is finally released.
I can't see any other solution than to hook filp_flush function and that also means we need to add it to btf_allowlist_d_path list, so it can use the d_path helper.
But it's probably not very stable because filp_flush is static so it could be potentially inlined.
Also if we'd keep the current filp_close hook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks like d_path might not work properly when the task is no longer around.
thoughts?
jirka
[1] 021a160abf62 ("fs: use __fput_sync in close(2)")
--- kernel/trace/bpf_trace.c | 1 + tools/testing/selftests/bpf/progs/test_d_path.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)
kernel/trace/bpf_trace.c +1 -0
@@ -941,6 +941,7 @@ BTF_ID(func, vfs_fallocate) BTF_ID(func, dentry_open) BTF_ID(func, vfs_getattr) BTF_ID(func, filp_close) +BTF_ID(func, filp_flush)BTF_SET_END (btf_allowlist_d_path) static bool bpf_d_path_allowed(const struct bpf_prog *prog)
tools/testing/selftests/bpf/progs/test_d_path.c +2 -2
@@ -40,8 +40,8 @@ int BPF_PROG(prog_stat, struct path *path, struct kstat *stat,return 0 ; } -SEC("fentry/filp_close") -int BPF_PROG (prog_close ,struct file *file,void *id) +SEC("fentry/filp_flush") +int BPF_PROG (prog_close ,struct file *file) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;__u32 cnt = cnt_close;
Signature · 1 line
2.41.0
Quoted 10 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.
looks like llvm makes it inlined (from CI)
Error: #68/1 d_path/basic
libbpf: prog 'prog_close': failed to find kernel BTF type ID of 'filp_flush': -3
jirka
Quoted 45 lines
Also if we'd keep the current
filp_closehook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks liked_pathmight not work properly when the task is no longer around.thoughts?
jirka[1]
021a160abf62("fs: use__fput_syncin close(2)")--- kernel/trace/bpf_trace.c | 1 + tools/testing/selftests/bpf/progs/test_d_path.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)kernel/trace/bpf_trace.c +1
-0@@ -941,6 +941,7 @@ BTF_ID(func, vfs_fallocate) BTF_ID(func, dentry_open) BTF_ID(func, vfs_getattr) BTF_ID(func, filp_close) +BTF_ID(func, filp_flush)BTF_SET_END (btf_allowlist_d_path) static bool bpf_d_path_allowed(const struct bpf_prog *prog)tools/testing/selftests/bpf/progs/test_d_path.c +2
-2@@ -40,8 +40,8 @@ int BPF_PROG(prog_stat, struct path *path, struct kstat *stat,return 0 ; } -SEC("fentry/filp_close") -int BPF_PROG (prog_close ,struct file *file,void *id) +SEC("fentry/filp_flush") +int BPF_PROG (prog_close ,struct file *file) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;__u32 cnt = cnt_close;Signature · 1 line
2.41.0
Song Liu
maintainerQuoted 13 lines · Jiri Olsa
Quoted 10 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.
Quoted 6 lines
looks like llvm makes it inlined (from CI)
Error: #68/1 d_path/basic
libbpf: prog 'prog_close': failed to find kernel BTF type ID of 'filp_flush': -3jirka
I played with it for a bit, but haven't got a good solution. Maybe we should just remove the test for close()?
Thanks,
Song
Quoted 5 lines
Quoted 5 lines
Also if we'd keep the current
filp_closehook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks liked_pathmight not work properly when the task is no longer around.
[...]
Quoted 16 lines · Song Liu
Quoted 15 lines · Jiri Olsa
Quoted 10 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.looks like llvm makes it inlined (from CI)
Quoted 7 lines
Quoted 4 lines
Error: #68/1 d_path/basic
libbpf: prog 'prog_close': failed to find kernel BTF type ID of 'filp_flush': -3jirka
I played with it for a bit, but haven't got a good solution. Maybe we should just remove the test for
close()?
I was thinking the same.. also we have some example with filp_close in bpftrace docs, I think we'll need to add some note with explanation in there
jirka
Quoted 10 lines
Thanks,
SongQuoted 5 lines
Quoted 5 lines
Also if we'd keep the current
filp_closehook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks liked_pathmight not work properly when the task is no longer around.[...]
Quoted 21 lines · Jiri Olsa
Quoted 19 lines · Song Liu
Quoted 18 lines · Jiri Olsa
Quoted 10 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.looks like llvm makes it inlined (from CI)
Error: #68/1 d_path/basic
libbpf: prog 'prog_close': failed to find kernel BTF type ID of 'filp_flush': -3
Quoted 7 lines
Quoted 4 lines
Quoted 1 line
jirka
I played with it for a bit, but haven't got a good solution. Maybe we should just remove the test for
close()?I was thinking the same.. also we have some example with
filp_closein bpftrace docs, I think we'll need to add some note with explanation in there
Maybe use __x64_sys_close in the test and recommend bpftrace scripts to do the same?
Quoted 24 lines · Alexei Starovoitov
Quoted 23 lines · Jiri Olsa
Quoted 21 lines · Song Liu
Quoted 19 lines · Jiri Olsa
Quoted 10 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.looks like llvm makes it inlined (from CI)
Error: #68/1 d_path/basic
libbpf: prog 'prog_close': failed to find kernel BTF type ID of 'filp_flush': -3jirka
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
I played with it for a bit, but haven't got a good solution. Maybe we should just remove the test for
close()?I was thinking the same.. also we have some example with
filp_closein bpftrace docs, I think we'll need to add some note with explanation in thereMaybe use
__x64_sys_closein the test and recommend bpftrace scripts to do the same?
we need struct file pointer as an argument, __x64_sys_close has pt_regs pointer
jirka
Yonghong Song
reviewerQuoted 12 lines · Jiri Olsa
Quoted 10 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.
Quoted 13 lines
looks like llvm makes it inlined (from CI)
Error: #68/1 d_path/basic
libbpf: prog 'prog_close': failed to find kernel BTF type ID of 'filp_flush': -3jirka
Quoted 6 lines
Also if we'd keep the current
filp_closehook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks liked_pathmight not work properly when the task is no longer around.thoughts?
Jiri,
The following patch works fine for me:
$ git diff
kernel/trace/bpf_trace.c +1 -0
@@ -941,6 +941,7 @@ BTF_ID(func, vfs_fallocate) BTF_ID(func, dentry_open) BTF_ID(func, vfs_getattr) BTF_ID(func, filp_close) +BTF_ID(func, __fput_sync)BTF_SET_END (btf_allowlist_d_path) static bool bpf_d_path_allowed(const struct bpf_prog *prog)
tools/testing/selftests/bpf/progs/test_d_path.c +2 -2
@@ -40,8 +40,8 @@ int BPF_PROG(prog_stat, struct path *path, struct kstat *stat,return 0 ; } -SEC("fentry/filp_close") -int BPF_PROG (prog_close ,struct file *file,void *id) +SEC("fentry/__fput_sync") +int BPF_PROG (prog_close ,struct file *file) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;__u32 cnt = cnt_close;
Quoted 40 lines
Quoted 39 lines
jirka
[1]
021a160abf62("fs: use__fput_syncin close(2)")--- kernel/trace/bpf_trace.c | 1 + tools/testing/selftests/bpf/progs/test_d_path.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)kernel/trace/bpf_trace.c +1
-0@@ -941,6 +941,7 @@ BTF_ID(func, vfs_fallocate) BTF_ID(func, dentry_open) BTF_ID(func, vfs_getattr) BTF_ID(func, filp_close) +BTF_ID(func, filp_flush)BTF_SET_END (btf_allowlist_d_path) static bool bpf_d_path_allowed(const struct bpf_prog *prog)tools/testing/selftests/bpf/progs/test_d_path.c +2
-2@@ -40,8 +40,8 @@ int BPF_PROG(prog_stat, struct path *path, struct kstat *stat,return 0 ; } -SEC("fentry/filp_close") -int BPF_PROG (prog_close ,struct file *file,void *id) +SEC("fentry/filp_flush") +int BPF_PROG (prog_close ,struct file *file) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;__u32 cnt = cnt_close;Signature · 1 line
2.41.0
Song Liu
maintainerQuoted 63 lines · Yonghong Song
Quoted 12 lines · Jiri Olsa
Quoted 10 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.Quoted 13 lines
looks like llvm makes it inlined (from CI)
Error: #68/1 d_path/basic
libbpf: prog 'prog_close': failed to find kernel BTF type ID of 'filp_flush': -3jirka
Quoted 6 lines
Also if we'd keep the current
filp_closehook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks liked_pathmight not work properly when the task is no longer around.thoughts?
Jiri,
The following patch works fine for me:
$ git diffkernel/trace/bpf_trace.c +1
-0@@ -941,6 +941,7 @@ BTF_ID(func, vfs_fallocate) BTF_ID(func, dentry_open) BTF_ID(func, vfs_getattr) BTF_ID(func, filp_close) +BTF_ID(func, __fput_sync)BTF_SET_END (btf_allowlist_d_path) static bool bpf_d_path_allowed(const struct bpf_prog *prog)tools/testing/selftests/bpf/progs/test_d_path.c +2
-2@@ -40,8 +40,8 @@ int BPF_PROG(prog_stat, struct path *path, structkstat *stat, return 0; } -SEC("fentry/filp_close") -int BPF_PROG (prog_close ,struct file *file,void *id) +SEC("fentry/__fput_sync") +int BPF_PROG (prog_close ,struct file *file) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;
Quoted 1 line
__u32 cnt = cnt_close;
Yeah, I guess this is the easiest fix at the moment.
Related, shall we have resolve_btfids fail for missing ID? Something like:
scripts/link-vmlinux.sh +4 -1
@@ -274,7 +274,10 @@ vmlinux_link vmlinux "${kallsymso}" ${btf_vmlinux_bin_o} # fill in BTF IDs if is_enabled CONFIG_DEBUG_INFO_BTF && is_enabled CONFIG_BPF; then info BTFIDS vmlinux - ${RESOLVE_BTFIDS} vmlinux + if ! ${RESOLVE_BTFIDS} vmlinux ; then + echo >&2 Failed to resolve BTF IDs + exit 1 + fi fi mksysmap vmlinux System.map ${kallsymso}
tools/bpf/resolve_btfids/main.c +0 -2
@@ -599,8 +599,10 @@ static int id_patch(struct object *obj, struct btf_id *id)int i;/* For set, set8, id->id may be 0 */ -if (!id->id && !id->is_set && !id->is_set8) - pr_err("WARN: resolve_btfids: unresolved symbol %s\n",
id->name);
+ if (!id->id && !id->is_set && !id->is_set8) {
+ pr_err("FAILED resolve_btfids: unresolved symbol
%s\n", id->name);
+ return -1;
+ }
for (i = 0; i < id->addr_cnt; i++) {
unsigned long addr = id->addr[i];Thanks,
Song
Quoted 76 lines · Song Liu
Quoted 63 lines · Yonghong Song
Quoted 12 lines · Jiri Olsa
Quoted 10 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.Quoted 13 lines
looks like llvm makes it inlined (from CI)
Error: #68/1 d_path/basic
libbpf: prog 'prog_close': failed to find kernel BTF type ID of 'filp_flush': -3jirka
Quoted 6 lines
Also if we'd keep the current
filp_closehook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks liked_pathmight not work properly when the task is no longer around.thoughts?
Jiri,
The following patch works fine for me:
$ git diffkernel/trace/bpf_trace.c +1
-0@@ -941,6 +941,7 @@ BTF_ID(func, vfs_fallocate) BTF_ID(func, dentry_open) BTF_ID(func, vfs_getattr) BTF_ID(func, filp_close) +BTF_ID(func, __fput_sync)BTF_SET_END (btf_allowlist_d_path) static bool bpf_d_path_allowed(const struct bpf_prog *prog)tools/testing/selftests/bpf/progs/test_d_path.c +2
-2@@ -40,8 +40,8 @@ int BPF_PROG(prog_stat, struct path *path, structkstat *stat, return 0; } -SEC("fentry/filp_close") -int BPF_PROG (prog_close ,struct file *file,void *id) +SEC("fentry/__fput_sync") +int BPF_PROG (prog_close ,struct file *file) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;Quoted 1 line
__u32 cnt = cnt_close;Yeah, I guess this is the easiest fix at the moment.
Related, shall we have
resolve_btfidsfail for missing ID? Something like:scripts/link-vmlinux.sh +0
-0
scripts/link-vmlinux.sh vmlinux_link vmlinux "${kallsymso}" ${btf_vmlinux_bin_o} · line 280
Quoted 8 lines
scripts/link-vmlinux.sh +4
-1@@ -274,7 +274,10 @@ vmlinux_link vmlinux "${kallsymso}" ${btf_vmlinux_bin_o} # fill in BTF IDs if is_enabled CONFIG_DEBUG_INFO_BTF && is_enabled CONFIG_BPF; then info BTFIDS vmlinux - ${RESOLVE_BTFIDS} vmlinux + if ! ${RESOLVE_BTFIDS} vmlinux ; then + echo >&2 Failed to resolve BTF IDs + exit 1 + fi
IIUC link-vmlinux.sh will fail when ${RESOLVE_BTFIDS} returns != 0, and now the unresolved symbol is just a warning
we used to have that but we decided to just warn:
5aad03685185 tools/resolve_btfids: Emit warnings and patch zero id for missing symbols
jirka
Quoted 39 lines · Yonghong Song
Quoted 12 lines · Jiri Olsa
Quoted 10 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.Quoted 13 lines
looks like llvm makes it inlined (from CI)
Error: #68/1 d_path/basic
libbpf: prog 'prog_close': failed to find kernel BTF type ID of 'filp_flush': -3jirka
Quoted 6 lines
Also if we'd keep the current
filp_closehook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks liked_pathmight not work properly when the task is no longer around.thoughts?
Jiri,
The following patch works fine for me:
$ git diffkernel/trace/bpf_trace.c +0
-0
kernel/trace/bpf_trace.c BTF_ID(func, vfs_fallocate) · line 944
Quoted 4 lines
kernel/trace/bpf_trace.c +1
-0@@ -941,6 +941,7 @@ BTF_ID(func, vfs_fallocate) BTF_ID(func, dentry_open) BTF_ID(func, vfs_getattr) BTF_ID(func, filp_close) +BTF_ID (func, __fput_sync)
hum right, that should work.. there are several callers of that, but AFAICT they all seem ok (not holding mount_lock or rename_lock) to run d_path helper
I'll send patch with the change
thanks,
jirka
Quoted 59 lines
kernel/trace/bpf_trace.c +0
-0BTF_SET_END (btf_allowlist_d_path) static bool bpf_d_path_allowed(const struct bpf_prog *prog)tools/testing/selftests/bpf/progs/test_d_path.c +2
-2@@ -40,8 +40,8 @@ int BPF_PROG(prog_stat, struct path *path, struct kstat *stat, return 0; } -SEC("fentry/filp_close") -int BPF_PROG (prog_close ,struct file *file,void *id) +SEC("fentry/__fput_sync") +int BPF_PROG (prog_close ,struct file *file) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;__u32 cnt = cnt_close;Quoted 38 lines
Quoted 37 lines
jirka
[1]
021a160abf62("fs: use__fput_syncin close(2)")--- kernel/trace/bpf_trace.c | 1 + tools/testing/selftests/bpf/progs/test_d_path.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)kernel/trace/bpf_trace.c +1
-0@@ -941,6 +941,7 @@ BTF_ID(func, vfs_fallocate) BTF_ID(func, dentry_open) BTF_ID(func, vfs_getattr) BTF_ID(func, filp_close) +BTF_ID(func, filp_flush)BTF_SET_END (btf_allowlist_d_path) static bool bpf_d_path_allowed(const struct bpf_prog *prog)tools/testing/selftests/bpf/progs/test_d_path.c +2
-2@@ -40,8 +40,8 @@ int BPF_PROG(prog_stat, struct path *path, struct kstat *stat,return 0 ; } -SEC("fentry/filp_close") -int BPF_PROG (prog_close ,struct file *file,void *id) +SEC("fentry/filp_flush") +int BPF_PROG (prog_close ,struct file *file) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;__u32 cnt = cnt_close;Signature · 1 line
2.41.0
Hi
Quoted 3 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.
To make test_d_path self-test pass, beside attaching to a different function (e.g., __fput_sync or filp_flush), we could also use close_range() or even dup2() to close the created fd, because these syscalls still use filp_close() to close the opened file.
Quoted 11 lines
I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.But it's probably not very stable because
filp_flushis static so it could be potentially inlined.Also if we'd keep the current
filp_closehook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks liked_pathmight not work properly when the task is no longer around.
It seems there is no need to wait for it to be called, because filp_close() is still called synchronously by some syscall (e.g., close_range or io_uring). So if the bpf program tries to collect many close event as possible, it should be attach to both filp_close() and __fput_sync(), right ?
Quoted 38 lines
thoughts?
jirka[1]
021a160abf62("fs: use__fput_syncin close(2)")--- kernel/trace/bpf_trace.c | 1 + tools/testing/selftests/bpf/progs/test_d_path.c | 4 ++-- 2 files changed, 3 insertions(+), 2 deletions(-)kernel/trace/bpf_trace.c +1
-0@@ -941,6 +941,7 @@ BTF_ID(func, vfs_fallocate) BTF_ID(func, dentry_open) BTF_ID(func, vfs_getattr) BTF_ID(func, filp_close) +BTF_ID(func, filp_flush)BTF_SET_END (btf_allowlist_d_path) static bool bpf_d_path_allowed(const struct bpf_prog *prog)tools/testing/selftests/bpf/progs/test_d_path.c +2
-2@@ -40,8 +40,8 @@ int BPF_PROG(prog_stat, struct path *path, struct kstat *stat,return 0 ; } -SEC("fentry/filp_close") -int BPF_PROG (prog_close ,struct file *file,void *id) +SEC("fentry/filp_flush") +int BPF_PROG (prog_close ,struct file *file) {pid_t pid =bpf_get_current_pid_tgid () >>32 ;__u32 cnt = cnt_close;
Quoted 11 lines · Hou Tao
Hi
Quoted 3 lines · Jiri Olsa
Recent commit [1] broken
d_pathtest, because nowfilp_closeis not called directly fromsys_close, but eventually later when the file is finally released.To make
test_d_pathself-test pass, beside attaching to a different function (e.g.,__fput_syncorfilp_flush), we could also useclose_range()or evendup2()to close the created fd, because these syscalls still usefilp_close()to close the opened file.
nice, I like the close_range solution ;-) patch below fixes the test
Quoted 13 lines
Quoted 6 lines
I can't see any other solution than to hook
filp_flushfunction and that also means we need to add it tobtf_allowlist_d_pathlist, so it can use thed_pathhelper.Quoted 6 lines
But it's probably not very stable because
filp_flushis static so it could be potentially inlined.Also if we'd keep the current
filp_closehook and find a way how to 'wait' for it to be called so user space can go with checks, then it looks liked_pathmight not work properly when the task is no longer around.
Quoted 6 lines
It seems there is no need to wait for it to be called, because
filp_close()is still called synchronously by some syscall (e.g.,close_rangeorio_uring). So if the bpf program tries to collect many close event as possible, it should be attach to bothfilp_close()and__fput_sync(), right ?
right, when we hook to close_range it's still synchronous
thanks,
jirka
---
tools/testing/selftests/bpf/prog_tests/d_path.c +2 -0
@@ -91,6 +91,7 @@ static int trigger_fstat_events(pid_t pid) out_close: /* triggers filp_close */ +#define close (fd) close_range(fd, fd, 0)close (pipefd[0 ]);close (pipefd[1 ]);close (sockfd);
@@ -98,6 +99,7 @@ static int trigger_fstat_events(pid_t pid)close (devfd);close (localfd);close (indicatorfd); +#undef closereturn ret; }