Getting better/supplementary error info back to userspace
Whilst undertaking a foray into container space and, related to that, looking at overhauling the mounting API, it occurred to me that I could make use of the mount context (now fs_context) that I was creating to allow the filesystem driver to pass supplementary error information back to the userspace program that was driving it in the form of textual messages:
int fd = fsopen("ext4");
write(fd, "d /dev/sda2");
write(fd, "o user_xattr");
if (fsmount(fd, "/mnt") == -1) {
/* Something went wrong, read back any error info */
size = read(fd, buffer, sizeof(buffer));
/* Now print the supplementary error message */
fprintf(stderr, "%*.*s\n", size, size, buffer);
}This would be particularly useful in the case of mounting a filesystem where so many things can go wrong that a small number is insufficient to represent them all. Yes, you have the dmesg log, but that's not necessarily available to you and is potentially intermixed with other things. Further, it's more user-friendly if the mount command or your GUI gives you the errors directly.
However, it occurred to me that this feature might be useful in other cases, not just mounting, and there are cases where it's not easy or not possible to get the message back to userspace because there's no user-accessible context (eg. automounting), or because the context is buried several levels down the stack (eg. NFS mount doing a pathwalk).
In which case, would it make sense to attach such a facility to the task_struct instead? I implemented a test of this using prctl, but a new syscall might be a better idea, at least for reading.
(*) int old_setting = prctl(PR_ERRMSG_ENABLE, int setting);
Enable (setting == 1) or disable (setting == 0) the facility. Disabling the facility clears the error buffer.
(*) int size = prctl(PR_ERRMSG_READ, char *buffer, int buf_size);
Read back a message and discard it.
Anyway, some questions:
(1) Is this something that would be of interest on a more global scale?
Or should I just stick with stashing it in the fs_context structure and find someway to route around the pathwalk in nfs mount?
Or is this totally a bad idea and only dmesg should ever be used?
If it is of interest globally:
(2) How big should I make each task's message buffer? My current implementation allows each task to hold a single message if enabled.
(3) Should I allow warnings in addition to errors?
(4) Should I allow wait() and co. to try and retrieve errors from zombies?
(5) Should execve() disable the facility?
(6) Could all the messages be static (not kmalloc'd) and cleared/redacted by rmmod? This would potentially prevent the use of formatted messages.
David
Em Wed, Jul 12, 2017 at 01:43:30PM +0100, David Howells escreveu:
Quoted 32 lines
Whilst undertaking a foray into container space and, related to that, looking at overhauling the mounting API, it occurred to me that I could make use of the mount context (now
fs_context) that I was creating to allow the filesystem driver to pass supplementary error information back to the userspace program that was driving it in the form of textual messages:int fd = fsopen("ext4"); write(fd, "d /dev/sda2"); write(fd, "o user_xattr"); if (fsmount(fd, "/mnt") == -1) { /* Something went wrong, read back any error info */ size = read(fd, buffer, sizeof(buffer)); /* Now print the supplementary error message */ fprintf(stderr, "%*.*s\n", size, size, buffer); }This would be particularly useful in the case of mounting a filesystem where so many things can go wrong that a small number is insufficient to represent them all. Yes, you have the dmesg log, but that's not necessarily available to you and is potentially intermixed with other things. Further, it's more user-friendly if the mount command or your GUI gives you the errors directly.
However, it occurred to me that this feature might be useful in other cases, not just mounting, and there are cases where it's not easy or not possible to get the message back to userspace because there's no user-accessible context (eg. automounting), or because the context is buried several levels down the stack (eg. NFS mount doing a pathwalk).
In which case, would it make sense to attach such a facility to the
task_structinstead? I implemented a test of this using prctl, but a new syscall might be a better idea, at least for reading.
Quoted 8 lines
(*) int
old_setting=prctl(PR_ERRMSG_ENABLE, int setting);Enable (setting == 1) or disable (setting == 0) the facility. Disabling the facility clears the error buffer.
(*) int size =
prctl(PR_ERRMSG_READ, char *buffer, int buf_size);Read back a message and discard it.
There were discussions about this in the not so distant past, perf being one of the areas where something like this would help a lot, lemme dig it, yeah, there is even a short LWN article describing it and with links to the lkml posts:
https://lwn.net/Articles/657341/
Involces prctl as yours, etc, etc.
What we do now in tools/perf/ with what we do have now is to have strerrno like messages for each class and method (well, we have for some of them), like:
int perf_evsel__open_strerror(struct perf_evsel *evsel,
struct target *target, int err, char *msg, size_t size);
where we have a switch to see, from syscall errno return and intended target (CPU, system wide, a specific thread, cgroups, etc), who is asking this (user, root, etc) and lots of other tunables, how to best translate this to the user, formatting it in a string allows us to show it in whatever GUI is in use.
- Arnaldo
Quoted 29 lines
Anyway, some questions:
(1) Is this something that would be of interest on a more global scale?
Or should I just stick with stashing it in the
fs_contextstructure and find someway to route around the pathwalk in nfs mount?Or is this totally a bad idea and only dmesg should ever be used?
If it is of interest globally:
(2) How big should I make each task's message buffer? My current implementation allows each task to hold a single message if enabled.
(3) Should I allow warnings in addition to errors?
(4) Should I allow
wait()and co. to try and retrieve errors from zombies?(5) Should
execve()disable the facility?(6) Could all the messages be static (not kmalloc'd) and cleared/redacted by rmmod? This would potentially prevent the use of formatted messages.
David
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
Em Wed, Jul 12, 2017 at 11:33:21AM -0300, Arnaldo Carvalho de Melo escreveu:
Quoted 8 lines
What we do now in
tools/perf/with what we do have now is to have strerrno like messages for each class and method (well, we have for some of them), like:int perf_evsel__open_strerror(struct
perf_evsel*evsel,
struct target *target, int err, char *msg,size_tsize);
Quoted 5 lines
where we have a switch to see, from syscall errno return and intended target (CPU, system wide, a specific thread, cgroups, etc), who is asking this (user, root, etc) and lots of other tunables, how to best translate this to the user, formatting it in a string allows us to show it in whatever GUI is in use.
To get this clearer in terms of actual usage, here is a (simplified) snippet for 'perf top':
try_again:
if (perf_evsel__open(event, cpus, threads) < 0) {
if (perf_evsel__fallback(event, errno, msg, sizeof(msg))) {
if (verbose > 0)
ui__warning("%s\n", msg);
goto try_again;
}
perf_evsel__open_strerror(event, target, errno, msg, sizeof(msg));
ui__error("%s\n", msg);
goto out_err;
}- Arnaldo
Quoted 4 lines · David Howells
In which case, would it make sense to attach such a facility to the
task_structinstead? I implemented a test of this using prctl, but a new syscall might be a better idea, at least for reading.
Quoted 8 lines
(*) int
old_setting=prctl(PR_ERRMSG_ENABLE, int setting);Enable (setting == 1) or disable (setting == 0) the facility. Disabling the facility clears the error buffer.
(*) int size =
prctl(PR_ERRMSG_READ, char *buffer, int buf_size);Read back a message and discard it.
I forgot to add that I've kept the in-kernel interface I have for this very simple for the moment:
void errorf(const char *fmt, ...); int invalf(const char *fmt, ...);
where these functions take printf-style arguments and where invalf() is the same as errorf(), but returns -EINVAL for convenience. To take an example from NFS:
- if (auth_info->flavor_len + 1 >= max_flavor_len) {
- dfprintk(MOUNT, "NFS: too many sec= flavors\n");
- return -EINVAL;
- }
+ if (auth_info->flavor_len + 1 >= max_flavor_len)
+ return invalf("NFS: too many sec= flavors");
David
Quoted 25 lines · On Wed, 12 Jul 2017 15:57:56 +0100 David Howells <[email protected]> wrote:
Quoted 4 lines · David Howells
In which case, would it make sense to attach such a facility to the
task_structinstead? I implemented a test of this using prctl, but a new syscall might be a better idea, at least for reading.Quoted 8 lines
(*) int
old_setting=prctl(PR_ERRMSG_ENABLE, int setting);Enable (setting == 1) or disable (setting == 0) the facility. Disabling the facility clears the error buffer.
(*) int size =
prctl(PR_ERRMSG_READ, char *buffer, int buf_size);Read back a message and discard it.
I forgot to add that I've kept the in-kernel interface I have for this very simple for the moment:
void errorf(const char *fmt, ...); int invalf(const char *fmt, ...);where these functions take printf-style arguments and where
invalf()is the same aserrorf(), but returns-EINVALfor convenience. To take an example from NFS:
Review
Quoted 6 lines
(quoted diff) +2
-4- if (auth_info->flavor_len + 1 >= max_flavor_len) { - dfprintk(MOUNT, "NFS: too many sec= flavors\n"); - return -EINVAL; - } + if (auth_info->flavor_len + 1 >= max_flavor_len) + return invalf("NFS: too many sec= flavors");
Netlink has recently got extended error reporting, still not used widely and library support is lacking in most places.
Quoted 3 lines · Stephen Hemminger
Netlink has recently got extended error reporting, still not used widely and library support is lacking in most places.
Yeah, and that "not widely supported and library support is lacking" is always going to be an issue with anything like that.
Along with internationalization, which is a whole nasty set of issues in itself with error messages.
It's not going to happen, in other words. The problems are basically insurmountable, and the thing it fixes will always be some special case that doesn't much matter.
Every time it comes up it is because some developer found one case that they were hunting down and it annoyed them, and the developer went "if only it had included more information and it would have been obvious".
But every time it comes up people ignore this basic issue:
[torvalds@i7 linux]$ git grep -e '-E[A-Z]\{4\}' | wc -l
182523Give it up. It's really is a horrible idea for so many reasons.
Linus
Quoted 21 lines · On Wed, 12 Jul 2017 09:19:55 -0700 Linus Torvalds…
Quoted 3 lines · Stephen Hemminger
Netlink has recently got extended error reporting, still not used widely and library support is lacking in most places.
Yeah, and that "not widely supported and library support is lacking" is always going to be an issue with anything like that.
Along with internationalization, which is a whole nasty set of issues in itself with error messages.
It's not going to happen, in other words. The problems are basically insurmountable, and the thing it fixes will always be some special case that doesn't much matter.
Every time it comes up it is because some developer found one case that they were hunting down and it annoyed them, and the developer went "if only it had included more information and it would have been obvious".
Quoted 7 lines
But every time it comes up people ignore this basic issue:
[torvalds@i7 linux]$ git grep -e '-E[A-Z]\{4\}' | wc -l 182523Give it up. It's really is a horrible idea for so many reasons.
For netlink, it isn't so bad. 80% of the usage is in iproute2 and therefore getting tool support for the usual cases isn't too hard.
I fear kernel developers think at too low a level. They think if glibc and/or 1st level command can handle an extension, their work is done. But in the modern world, there are many scripts and layers above that. For the networking case, the worst case examples are things where configuration is done in stuff like some layer on top of Openstack, in python code which is scripting ip commands, which is talking to the kernel. Good luck on trying to get any meaningful error handling out of that dog pile.
Quoted 21 lines · On Wed, 12 Jul 2017 09:19:55 -0700 Linus Torvalds…
Quoted 3 lines · Stephen Hemminger
Netlink has recently got extended error reporting, still not used widely and library support is lacking in most places.
Yeah, and that "not widely supported and library support is lacking" is always going to be an issue with anything like that.
Along with internationalization, which is a whole nasty set of issues in itself with error messages.
It's not going to happen, in other words. The problems are basically insurmountable, and the thing it fixes will always be some special case that doesn't much matter.
Every time it comes up it is because some developer found one case that they were hunting down and it annoyed them, and the developer went "if only it had included more information and it would have been obvious".
Quoted 5 lines
But every time it comes up people ignore this basic issue:
[torvalds@i7 linux]$ git grep -e '-E[A-Z]\{4\}' | wc -l 182523
Note a lot of those -E* are not going to user space. Some are in comments, and some are used internally. I use them to pass back information to other kernel only routines, as some errors are more critical than others.
Quoted 3 lines
Give it up. It's really is a horrible idea for so many reasons.
One reason that this has never taken off is that there is no good infrastructure in doing it. I wouldn't tell people to give it up, but I don't see a one size fits all. In tracing, we have ways to pass detailed errors back to user space. But that's probably one of the easier cases as we have defined methods to do so.
A more generic approach would require a lot more planning, and making it simple to use both in user space and in the kernel. If it is too complex in either place, it will be ignored.
-- Steve
Quoted 26 lines · Steven Rostedt
Quoted 23 lines · On Wed, 12 Jul 2017 09:19:55 -0700 Linus Torvalds…
Quoted 3 lines · Stephen Hemminger
Netlink has recently got extended error reporting, still not used widely and library support is lacking in most places.
Yeah, and that "not widely supported and library support is lacking" is always going to be an issue with anything like that.
Along with internationalization, which is a whole nasty set of issues in itself with error messages.
It's not going to happen, in other words. The problems are basically insurmountable, and the thing it fixes will always be some special case that doesn't much matter.
Every time it comes up it is because some developer found one case that they were hunting down and it annoyed them, and the developer went "if only it had included more information and it would have been obvious".
But every time it comes up people ignore this basic issue:
Quoted 8 lines
Quoted 3 lines
[torvalds@i7 linux]$ git grep -e '-E[A-Z]\{4\}' | wc -l 182523Note a lot of those -E* are not going to user space. Some are in comments, and some are used internally. I use them to pass back information to other kernel only routines, as some errors are more critical than others.
a) it wouldn't have to be for every error
b) kernel prints detailed error in dmesg anyway, why not allow that info to be bound to the syscall that triggered the error?
c) internationalization can be solved at the level where it matters (NOT in the kernel)
My suggestion was to keep the kernel interface really simple, e.g.:
return detailed_error(-EINVAL, "failure to do foo because of bar");
What are the insurmountable issues you are talking about?
Thanks,
Miklos
Quoted 3 lines · Miklos Szeredi
My suggestion was to keep the kernel interface really simple, e.g.:
return detailed_error(-EINVAL, "failure to do foo because of bar");
That's what I was thinking of, though I'd prefix the string with a source tag, such as "nfs", "vfs" or "dvb-core".
David
Quoted 7 lines · Linus Torvalds
But every time it comes up people ignore this basic issue:
[torvalds@i7 linux]$ git grep -e '-E[A-Z]\{4\}' | wc -l 182523Give it up. It's really is a horrible idea for so many reasons.
Are you okay with me making it possible to retrieve mount errors, warnings and informational messages through fd-arbitrated-mount I'm working on? For example (and skipping some of the parameters for brevity):
int fs_fd;
static inline void e(int x)
{
char buf[1024];
int i;
if (x == -1)
fprintf(stderr, "Mount error: %m\n");
/* Read back any messages */
while (i = read(fs_fd, buf), i != -1) {
buf[i] = 0;
fprintf(stderr, "%s\n", buf);
}
if (x == -1)
exit(1);
}
fs_fd = fsopen("ext4");
e(write(fs_fd, "d /dev/sda3"));
e(write(fs_fd, "o user_xattr"));
e(write(fs_fd, "o acl"));
e(write(fs_fd, "o data=ordered"));
e(write(fs_fd, "x create"));
e(fsmount(fs_fd, AT_FDCWD, "/mnt", MS_NODEV));
close(fs_fd);David