Skip to contentSkip to conversation

How to encourage driver authors to annotate integer endianness properly

Ksummit-discuss Bart Van Assche 9 participants 14 messages Oct 25, 2017 – Oct 29, 2017

lore
Share & tools
lore
#0 lore raw

Hello Ted,

As you most likely know endianness annotations like __be32 can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g. be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.

Thanks,

Bart.

#1 lore raw
Quoted 10 lines · Bart Van Assche

Hello Ted,

As you most likely know endianness annotations like __be32 can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g. be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.

Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)

thanks,

greg k-h

#2 lore raw
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche

Hello Ted,

As you most likely know endianness annotations like __be32 can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g. be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.

Quoted 4 lines

Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)

Hi Greg,

A significant challenge is that for several drivers (e.g. drivers/scsi/qla2xxx and drivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean. Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.

Bart.

#3 lore raw
Quoted 18 lines · Bart Van Assche
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche

Hello Ted,

As you most likely know endianness annotations like __be32 can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g. be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.

Quoted 4 lines

Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)

Quoted 5 lines

Hi Greg,

A significant challenge is that for several drivers (e.g. drivers/scsi/qla2xxx and drivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean.

Then ask them to do so, and if they will not, then mark the drivers in Kconfig so they do not get built for big endian platforms.

Quoted 9 lines

Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.

These problems are not "new" at all, it's been this way for 20+ years now :)

Oh, and the maintainers summit is already half over, and the schedule is full from what I can tell...

thanks,

greg k-h

#4 lore raw
Quoted 2 lines · On Wed, Oct 25, 2017 at 12:13:55PM +0200, [email protected] wrote:

Oh, and the maintainers summit is already half over, and the schedule is full from what I can tell...

There is one TBA session at 1615, just before the TAB elections.

#5 lore raw
Quoted 2 lines · Mark Brown

There is one TBA session at 1615, just before the TAB elections.

If people want to talk about Device Driver Endianess Problems, the room is yours at 16:15. I've added it to the agenda spread sheet.

- Ted

#6 lore raw
In reply to [email protected]
Quoted 38 lines · On Wed, Oct 25, 2017 at 12:13:55PM +0200, [email protected] wrote:
Quoted 18 lines · Bart Van Assche
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche

Hello Ted,

As you most likely know endianness annotations like __be32 can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g. be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.

Quoted 4 lines

Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)

Quoted 5 lines

Hi Greg,

A significant challenge is that for several drivers (e.g. drivers/scsi/qla2xxx and drivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean.

Then ask them to do so, and if they will not, then mark the drivers in Kconfig so they do not get built for big endian platforms.

Quoted 9 lines

Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.

Quoted 1 line

These problems are not "new" at all, it's been this way for 20+ years now :)

The major difference between past and current is in the tools, they evolved.

Quoted 3 lines

Oh, and the maintainers summit is already half over, and the schedule is full from what I can tell...

There is one open TBD slot.

Quoted 8 lines

thanks,

greg k-h
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#7 lore raw
Quoted 39 lines · Leon Romanovsky
Quoted 38 lines · On Wed, Oct 25, 2017 at 12:13:55PM +0200, [email protected] wrote:
Quoted 18 lines · Bart Van Assche
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche

Hello Ted,

As you most likely know endianness annotations like __be32 can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g. be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.

Quoted 4 lines

Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)

Quoted 5 lines

Hi Greg,

A significant challenge is that for several drivers (e.g. drivers/scsi/qla2xxx and drivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean.

Then ask them to do so, and if they will not, then mark the drivers in Kconfig so they do not get built for big endian platforms.

Quoted 9 lines

Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.

Quoted 3 lines
Quoted 1 line

These problems are not "new" at all, it's been this way for 20+ years now :)

The major difference between past and current is in the tools, they evolved.

endian support in sparse has been there for over a decade now, right? This isn't a new issue, and if a driver author/team does not want to do the work for a specific endian, then disable the drivers from that build...

greg k-h

#8 lore raw
Quoted 4 lines · On Wed, 2017-10-25 at 12:38 +0200, [email protected] wrote:

endian support in sparse has been there for over a decade now, right? This isn't a new issue, and if a driver author/team does not want to do the work for a specific endian, then disable the drivers from that build...

Hello Greg,

Sorry if I wasn't clear enough. My concern is not about drivers that do not work for a specific endianness but about drivers that need endianness conversions because the endianness of the firmware registers differs from that of the CPU. Such drivers benefit from endianness annotations even if only one specific CPU endianness type is supported by these drivers.

Bart.

#9 lore raw
In reply to Bart Van Assche
Quoted 20 lines · Bart Van Assche
Quoted 12 lines · Greg KH
Quoted 10 lines · Bart Van Assche

Hello Ted,

As you most likely know endianness annotations like __be32 can be verified by the static source code analyzer called sparse. These annotations are a big help to verify whether endianness conversions in drivers are correct (e.g. be32_to_cpu()). However, many driver authors either are not familiar with sparse or do not use it to verify their work. I think we need a way to encourage driver authors to pay attention to endianness annotations, e.g. by letting the zero-day kernel test infrastructure verify endianness annotations. Please consider to add this topic to the kernel summit agenda.

Quoted 4 lines

Driver subsystem maintainers should know this, and sparse reports should be simple to run and notice these issues. If you know of a subsystem that is not paying attention to this, please let those maintainers know and send patches to resolve the issues :)

Hi Greg,

Quoted 12 lines

A significant challenge is that for several drivers (e.g. drivers/scsi/qla2xxx and drivers/infiniband/hw/nes) fixing the endianness annotations is so hard that only the driver authors can make these drivers endianness clean. Three challenges that have been encountered while trying to make drivers endianness clean are:
- Some driver authors use a single variable to store both cpu-endian and big endian data.
- Endianness bugs - using a big endian integer where a cpu endian number should have been used or vice versa.
- The definitive resource for the endianness format is the firmware documentation. For many kernel drivers the firmware documentation is either not available or only available under NDA.

It is e very appealing to fix the driver endianness issues without involving driver authors, but it is not feasible without their cooperation and it is not always possible especially for not-active drivers.

Thanks

Quoted 6 lines

Bart.
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#10 lore raw
In reply to Bart Van Assche
Quoted 2 lines · Bart Van Assche

I think we need a way to encourage driver authors
to pay attention to endianness annotations,

typedef struct {
        uint32_t _;
} __le32;

The problems of sparse are:
a) it is not mandatory part of kernel compilation, and
b) it doesn't break compilation.

#11 lore raw
Quoted 7 lines · Alexey Dobriyan
Quoted 3 lines · Bart Van Assche

I think we need a way to encourage driver authors
to pay attention to endianness annotations,

typedef struct {
        uint32_t _;
} __le32;

I actually did that in JFFS2 years ago; before sparse could do it. It was very useful.

I'd really like to have proper compiler support for big-endian and little-endian integers too. We're half-way there now with __builtin_bswap32() et al, and compilers can optimise things properly (instead of the opaque inline asm we had before I cleaned that up).

But being able to just mark a given integer storage as big- or little- endian and have all loads and stores *automatically* do the right thing would be extremely useful in many cases.

#12 lore raw
Quoted 12 lines · David Woodhouse
Quoted 7 lines · Alexey Dobriyan
Quoted 3 lines · Bart Van Assche

I think we need a way to encourage driver authors
to pay attention to endianness annotations,

typedef struct {
        uint32_t _;
} __le32;

I actually did that in JFFS2 years ago; before sparse could do it. It was very useful.

Quoted 8 lines

I'd really like to have proper compiler support for big-endian and little-endian integers too. We're half-way there now with __builtin_bswap32() et al, and compilers can optimise things properly (instead of the opaque inline asm we had before I cleaned that up).

But being able to just mark a given integer storage as big- or little- endian and have all loads and stores *automatically* do the right thing would be extremely useful in many cases.

See https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59852 for a proposal to implement Sparse's __attribute__((bitwise)) in GCC. The GCC developers are amenable, if someone's willing to write a patch.