How to encourage driver authors to annotate integer endianness properly
Hello Ted,
As you most likely know endianness annotations like __be32 can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g. be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.
Thanks,
Bart.
Quoted 10 lines · Bart Van Assche
Hello Ted,
As you most likely know endianness annotations like
__be32can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g.be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.
Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)
thanks,
greg k-h
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche
Hello Ted,
As you most likely know endianness annotations like
__be32can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g.be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.
Quoted 4 lines
Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)
Hi Greg,
A significant challenge is that for several drivers (e.g. drivers/scsi/qla2xxx and drivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean. Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.
Bart.
Quoted 18 lines · Bart Van Assche
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche
Hello Ted,
As you most likely know endianness annotations like
__be32can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g.be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.Quoted 4 lines
Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)
Quoted 5 lines
Hi Greg,
A significant challenge is that for several drivers (e.g.
drivers/scsi/qla2xxxanddrivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean.
Then ask them to do so, and if they will not, then mark the drivers in Kconfig so they do not get built for big endian platforms.
Quoted 9 lines
Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.
These problems are not "new" at all, it's been this way for 20+ years now :)
Oh, and the maintainers summit is already half over, and the schedule is full from what I can tell...
thanks,
greg k-h
Quoted 2 lines · On Wed, Oct 25, 2017 at 12:13:55PM +0200, [email protected] wrote:
Oh, and the maintainers summit is already half over, and the schedule is full from what I can tell...
There is one TBA session at 1615, just before the TAB elections.
Quoted 20 lines · Bart Van Assche
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche
Hello Ted,
As you most likely know endianness annotations like
__be32can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g.be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.Quoted 4 lines
Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)
Hi Greg,
Quoted 12 lines
A significant challenge is that for several drivers (e.g.
drivers/scsi/qla2xxxanddrivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean. Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.
It is e very appealing to fix the driver endianness issues without involving driver authors, but it is not feasible without their cooperation and it is not always possible especially for not-active drivers.
Thanks
Quoted 6 lines
Bart.
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
Quoted 38 lines · On Wed, Oct 25, 2017 at 12:13:55PM +0200, [email protected] wrote:
Quoted 18 lines · Bart Van Assche
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche
Hello Ted,
As you most likely know endianness annotations like
__be32can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g.be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.Quoted 4 lines
Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)
Quoted 5 lines
Hi Greg,
A significant challenge is that for several drivers (e.g.
drivers/scsi/qla2xxxanddrivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean.Then ask them to do so, and if they will not, then mark the drivers in
Kconfigso they do not get built for big endian platforms.Quoted 9 lines
Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.
Quoted 1 line
These problems are not "new" at all, it's been this way for 20+ years now :)
The major difference between past and current is in the tools, they evolved.
Quoted 3 lines
Oh, and the maintainers summit is already half over, and the schedule is full from what I can tell...
There is one open TBD slot.
Quoted 8 lines
thanks,
greg k-h
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss
Quoted 39 lines · Leon Romanovsky
Quoted 38 lines · On Wed, Oct 25, 2017 at 12:13:55PM +0200, [email protected] wrote:
Quoted 18 lines · Bart Van Assche
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche
Hello Ted,
As you most likely know endianness annotations like
__be32can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g.be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.Quoted 4 lines
Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)
Quoted 5 lines
Hi Greg,
A significant challenge is that for several drivers (e.g.
drivers/scsi/qla2xxxanddrivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean.Then ask them to do so, and if they will not, then mark the drivers in
Kconfigso they do not get built for big endian platforms.Quoted 9 lines
Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.
Quoted 3 lines
Quoted 1 line
These problems are not "new" at all, it's been this way for 20+ years now :)
The major difference between past and current is in the tools, they evolved.
endian support in sparse has been there for over a decade now, right? This isn't a new issue, and if a driver author/team does not want to do the work for a specific endian, then disable the drivers from that build...
greg k-h
Quoted 4 lines · On Wed, 2017-10-25 at 12:38 +0200, [email protected] wrote:
endian support in sparse has been there for over a decade now, right? This isn't a new issue, and if a driver author/team does not want to do the work for a specific endian, then disable the drivers from that build...
Hello Greg,
Sorry if I wasn't clear enough. My concern is not about drivers that do not work for a specific endianness but about drivers that need endianness conversions because the endianness of the firmware registers differs from that of the CPU. Such drivers benefit from endianness annotations even if only one specific CPU endianness type is supported by these drivers.
Bart.
Quoted 2 lines · Bart Van Assche
I think we need a way to encourage driver authors
to pay attention to endianness annotations,
typedef struct {
uint32_t _;
} __le32;The problems of sparse are:
a) it is not mandatory part of kernel compilation, and
b) it doesn't break compilation.
Quoted 2 lines · Mark Brown
There is one TBA session at 1615, just before the TAB elections.
If people want to talk about Device Driver Endianess Problems, the room is yours at 16:15. I've added it to the agenda spread sheet.
- Ted
Quoted 7 lines · Alexey Dobriyan
Quoted 3 lines · Bart Van Assche
I think we need a way to encourage driver authors
to pay attention to endianness annotations,typedef struct {
uint32_t _;
}__le32;
I actually did that in JFFS2 years ago; before sparse could do it. It was very useful.
I'd really like to have proper compiler support for big-endian and little-endian integers too. We're half-way there now with __builtin_bswap32() et al, and compilers can optimise things properly (instead of the opaque inline asm we had before I cleaned that up).
But being able to just mark a given integer storage as big- or little- endian and have all loads and stores *automatically* do the right thing would be extremely useful in many cases.
Quoted 12 lines · David Woodhouse
Quoted 7 lines · Alexey Dobriyan
Quoted 3 lines · Bart Van Assche
I think we need a way to encourage driver authors
to pay attention to endianness annotations,typedef struct {
uint32_t _;
}__le32;I actually did that in JFFS2 years ago; before sparse could do it. It was very useful.
Quoted 8 lines
I'd really like to have proper compiler support for big-endian and little-endian integers too. We're half-way there now with
__builtin_bswap32()et al, and compilers can optimise things properly (instead of the opaque inline asm we had before I cleaned that up).But being able to just mark a given integer storage as big- or little- endian and have all loads and stores *automatically* do the right thing would be extremely useful in many cases.
See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59852 for a proposal to implement Sparse's __attribute__((bitwise)) in GCC. The GCC developers are amenable, if someone's willing to write a patch.
Quoted 3 lines · Josh Triplett
See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59852 for a proposal to implement Sparse's __attribute__((bitwise)) in GCC. The GCC developers are amenable, if someone's willing to write a patch.
Thanks Josh. I will have a look at that bugzilla ticket.
Bart.