Skip to contentSkip to conversation

More useful types in the linux kernel

Ksummit-discuss Eric W. Biederman 28 participants 82 messages Jul 19, 2016 – Aug 15, 2016 2 messages not in the archive

lore
Share & tools
lore
#0 lore raw

Historically the types in C came about because the machines fundamentally supported different data types either with different sizes or different characteristics (i.e. u8, u16, float, double). These data types and the C type system were built so programmers could tell the machine what they needed it to do.

There is another genesis of types that started with the simply typed lambda calculs that is about eliminating common errors and otherwise helping a programmer get their code right.

In the years since C was invented there has been a lot of activity and a little bit of progress in this area. Would people be receptive to improvements in this area?

I would like to talk to folks and gague what it would take to make improvements in this area acceptable, practical, and useful.

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

* This cleanly allows for doubly linked lists.

* This is useful to ensure that data is either put in another data structure where it is remembered or it is freed.

* This is useful to ensure reference counts are not leaked.

* This is useful to ensure that every lock is paired with an unlock.

My personal filter for things like this are types that can be checked in time proportional to the amount of code to be built so that it is roughly the same situation we are in now.

Given it's heritage and it's history the type system in C has serious limitations that I don't know if they are correctible, when it comes to catching programmer mistakes: silent truncation of unsigned types into smaller unsigned types, casts, etc. Would people be willing to consider a simple, link compatible alternative to C for some of the code in the kernel that had the same low level control of the machine but had a type system that made catching mistakes easier?

Deploying solutions like this will take a fair bit of grunt work, and time similar or worse than the big kernel lock removal. Given how widely Linux is used and how annoying some of these bugs can be I think it is worthwhile to dig in and see what kind of improvements can be made.

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Eric

#1 lore raw
Quoted 2 lines · Eric W. Biederman

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

There's a push from certain quarters to move away from GCC to LLVM. Not that these things are unachievable in LLVM but it's a thing.

#2 lore raw
Quoted 1 line · Mark Brown

There's a push from certain quarters to move away from GCC to LLVM.

This might actually be an interesting topic per se.

LLVM definitely has quite some nice features, but their attitude towards bugs which are rather severe for kernel programming should be taken as a warning at least. Look at the "pushf/popf being generated around local_irq_save()" trainwreck as an example.

Thanks,

Signature · 2 lines
Jiri Kosina
SUSE Labs
#3 lore raw
Quoted 10 lines · Jiri Kosina
Quoted 1 line · Mark Brown

There's a push from certain quarters to move away from GCC to LLVM.

This might actually be an interesting topic per se.

LLVM definitely has quite some nice features, but their attitude towards bugs which are rather severe for kernel programming should be taken as a warning at least. Look at the "pushf/popf being generated around local_irq_save()" trainwreck as an example.

We have the precedent of sparse that shows how we can have extended types with the existing compiler seeing the ordinary types. I can't imagine doing anything different if we continue to use C code. Anything would just be silly.

I can see value in a kernel that multiple compilers can build. But so far a switch to llvm looks scary.

Eric

#4 lore raw
In reply to Jiri Kosina
Quoted 5 lines · Jiri Kosina
Quoted 1 line · Mark Brown

There's a push from certain quarters to move away from GCC to LLVM.

This might actually be an interesting topic per se.

Yes, indeed.

Quoted 4 lines

LLVM definitely has quite some nice features, but their attitude towards bugs which are rather severe for kernel programming should be taken as a warning at least. Look at the "pushf/popf being generated around local_irq_save()" trainwreck as an example.

My understanding (which is a bit second hand here) is that LLVM upstream really values direct engagement at time of development much more than anything else - AFAICT they're more focused on driving things forward, partly due to expecting other people to do the main release management. To that end if people really want to run LLVM built kernels in production they probably need to either use a downstream that ensures that things are working well for kernel builds or work directly on testing development versions of the compilers to catch issues there.

#5 lore raw

[Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

Quoted 5 lines · Mark Brown
Quoted 3 lines
Quoted 1 line

There's a push from certain quarters to move away from GCC to LLVM.

This might actually be an interesting topic per se.

Yes, indeed.

Let's make this a real proposal then ... (subject changed). I am again a bit unsure about the core / tech division here.

People who should be invited: proponents of the push from the certain quarters mentioned by Mark above, and ideally some LLVM folks as well.

I've never actually used llvm to compile the kernel (which makes me rather poor contributor should any such discussion happen), but I've been on the "receiving side", debugging a crash that turned out to be llvm messing up with IF in a way that interfers with local_irq_save(), and also suffered from the followup frustration when I found out that this has been reported to llvm folks ages ago, and they haven't bothered to fix it (it's now at least worked around, in a very sub-optimal way (lahf/sahf)).

Thanks,

Signature · 2 lines
Jiri Kosina
SUSE Labs
#6 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

Quoted 6 lines · Jiri Kosina
Quoted 4 lines · Mark Brown
Quoted 3 lines
Quoted 1 line

There's a push from certain quarters to move away from GCC to LLVM.

This might actually be an interesting topic per se.

Quoted 7 lines
Quoted 1 line

Yes, indeed.

Let's make this a real proposal then ... (subject changed). I am again a bit unsure about the core / tech division here.

People who should be invited: proponents of the push from the certain quarters mentioned by Mark above, and ideally some LLVM folks as well.

This is a bit premature until Linux Clang patchset is empty. As an example, alternatives do not work and IIRC VLAs won't work ever:
https://llvm.org/bugs/show_bug.cgi?id=24487

Of course, compiling with Clang should be encouraged. At worst, it is yet another static checker.

#7 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

Quoted 8 lines · Alexey Dobriyan
Quoted 2 lines · Jiri Kosina

People who should be invited: proponents of the push from the certain quarters mentioned by Mark above, and ideally some LLVM folks as well.

This is a bit premature until Linux Clang patchset is empty. As an example, alternatives do not work and IIRC VLAs won't work ever:
https://llvm.org/bugs/show_bug.cgi?id=24487

The main interest I'm aware of is from people working on ARM (and it looks like there's some MIPS as well from the LLVMLinux git) so x86 specific issues might not be such a big deal. There's definitely some upstreaming still to do but the delta is getting smaller.

#8 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

In reply to Jiri Kosina
Quoted 14 lines · Jiri Kosina
Quoted 5 lines · Mark Brown
Quoted 3 lines
Quoted 1 line

There's a push from certain quarters to move away from GCC to LLVM.

This might actually be an interesting topic per se.

Yes, indeed.

Let's make this a real proposal then ... (subject changed). I am again a bit unsure about the core / tech division here.

People who should be invited: proponents of the push from the certain quarters mentioned by Mark above, and ideally some LLVM folks as well.

Quoted 7 lines

I've never actually used llvm to compile the kernel (which makes me rather poor contributor should any such discussion happen), but I've been on the "receiving side", debugging a crash that turned out to be llvm messing up with IF in a way that interfers with local_irq_save(), and also suffered from the followup frustration when I found out that this has been reported to llvm folks ages ago, and they haven't bothered to fix it (it's now at least worked around, in a very sub-optimal way (lahf/sahf)).

I got involved in building the kernel with LLVM a little while ago, after accidentally implementing .code16 support in LLVM — for other reasons, but it allowed the arch/x86/boot/ bits to be built with LLVM.

Apart from resolutely not wanting to implement variable length arrays on the stack, the LLVM folks actually seem quite keen to make things work. I'm interested in the problem you report above.. and note the absence of a bug number. Can you provide it?

You're right that it does take a while to get some things fixed, but people *are* doing a fairly good job of identifying them, filing bugs, and implementing workarounds until the bugs can be fixed.

Building with LLVM has also helped to find some real kernel bugs. I'd be keen to get this working more widely.

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#9 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

Quoted 32 lines · David Woodhouse
Quoted 14 lines · Jiri Kosina
Quoted 5 lines · Mark Brown
Quoted 3 lines
Quoted 1 line

There's a push from certain quarters to move away from GCC to LLVM.

This might actually be an interesting topic per se.

Yes, indeed.

Let's make this a real proposal then ... (subject changed). I am again a bit unsure about the core / tech division here.

People who should be invited: proponents of the push from the certain quarters mentioned by Mark above, and ideally some LLVM folks as well.

Quoted 7 lines

I've never actually used llvm to compile the kernel (which makes me rather poor contributor should any such discussion happen), but I've been on the "receiving side", debugging a crash that turned out to be llvm messing up with IF in a way that interfers with local_irq_save(), and also suffered from the followup frustration when I found out that this has been reported to llvm folks ages ago, and they haven't bothered to fix it (it's now at least worked around, in a very sub-optimal way (lahf/sahf)).

I got involved in building the kernel with LLVM a little while ago, after accidentally implementing .code16 support in LLVM — for other reasons, but it allowed the arch/x86/boot/ bits to be built with LLVM.

Apart from resolutely not wanting to implement variable length arrays on the stack, the LLVM folks actually seem quite keen to make things work. I'm interested in the problem you report above.. and note the absence of a bug number. Can you provide it?

Quoted 7 lines

You're right that it does take a while to get some things fixed, but people *are* doing a fairly good job of identifying them, filing bugs, and implementing workarounds until the bugs can be fixed.

Building with LLVM has also helped to find some real kernel bugs. I'd be keen to get this working more widely.

Would you be willing to share your experiences and the nature of bugs you were able to find using LLVM. Maybe that could be folded into this discussion as a real life experience.

thanks,
-- Shuah

#10 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

Quoted 37 lines · Shuah Khan
Quoted 36 lines · David Woodhouse
Quoted 14 lines · Jiri Kosina
Quoted 5 lines · Mark Brown
Quoted 3 lines
Quoted 1 line

There's a push from certain quarters to move away from GCC to LLVM.

This might actually be an interesting topic per se.

Yes, indeed.

Let's make this a real proposal then ... (subject changed). I am again a bit unsure about the core / tech division here.

People who should be invited: proponents of the push from the certain quarters mentioned by Mark above, and ideally some LLVM folks as well.

Quoted 7 lines

I've never actually used llvm to compile the kernel (which makes me rather poor contributor should any such discussion happen), but I've been on the "receiving side", debugging a crash that turned out to be llvm messing up with IF in a way that interfers with local_irq_save(), and also suffered from the followup frustration when I found out that this has been reported to llvm folks ages ago, and they haven't bothered to fix it (it's now at least worked around, in a very sub-optimal way (lahf/sahf)).

I got involved in building the kernel with LLVM a little while ago, after accidentally implementing .code16 support in LLVM — for other reasons, but it allowed the arch/x86/boot/ bits to be built with LLVM.

Apart from resolutely not wanting to implement variable length arrays on the stack, the LLVM folks actually seem quite keen to make things work. I'm interested in the problem you report above.. and note the absence of a bug number. Can you provide it?

You're right that it does take a while to get some things fixed, but people *are* doing a fairly good job of identifying them, filing bugs, and implementing workarounds until the bugs can be fixed.

Quoted 7 lines
Quoted 3 lines

Building with LLVM has also helped to find some real kernel bugs. I'd be keen to get this working more widely.

Would you be willing to share your experiences and the nature of bugs you were able to find using LLVM. Maybe that could be folded into this discussion as a real life experience.

http://llvm.linuxfoundation.org/index.php/Bugs

Probably horrifically out of date. Behan is the best person to ask for a current status, I suspect...

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#11 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

Quoted 10 lines · David Woodhouse
Quoted 3 lines · Shuah Khan

Would you be willing to share your experiences and the nature of bugs you were able to find using LLVM. Maybe that could be folded into this discussion as a real life experience.

http://llvm.linuxfoundation.org/index.php/Bugs

Probably horrifically out of date. Behan is the best person to ask for a current status, I suspect...

CCing in Sumit who I believe has also been looking at some issues building kernels with LLVM.

#12 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

Quoted 10 lines · Mark Brown
Quoted 7 lines · David Woodhouse
Quoted 3 lines · Shuah Khan

Would you be willing to share your experiences and the nature of bugs you were able to find using LLVM. Maybe that could be folded into this discussion as a real life experience.

http://llvm.linuxfoundation.org/index.php/Bugs
Probably horrifically out of date. Behan is the best person to ask for a current status, I suspect...

CCing in Sumit who I believe has also been looking at some issues building kernels with LLVM.

We're trying to start up the group again who was working on this a couple years ago. Part of that is figuring out an up to date list of issues. Some have been fixed, some are outstanding. And a few more things have been unearthed.

Despite other people seemingly interested in this, the traditional LLVMLinux people have mostly been moved to different projects and have had less time to concentrate on this topic, which is why things have fallen behind.

Behan

Signature · 2 lines
Behan Webster
[email protected]
#13 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

In reply to David Woodhouse
Quoted 4 lines · David Woodhouse

Apart from resolutely not wanting to implement variable length arrays on the stack, the LLVM folks actually seem quite keen to make things work. I'm interested in the problem you report above.. and note the absence of a bug number. Can you provide it?

I am currently on vacation and on super-lousy internet connection, so looking through my archives is a bit complicated ... I *think* it started in "[PATCH] usbhid: Fix lockdep unannotated irqs-off warning" thread on lkml.

In case you're not able to find it from there, I'll do my homework mid-next week when I am back properly online.

Thanks,

Signature · 2 lines
Jiri Kosina
SUSE Labs
#14 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

We're considering trying to move our builds to Clang. It's definitely something we'd be interested in talking about, and depending on how the initial attempts go we may have some anecdotal/early data.

Quoted 26 lines · Jiri Kosina
Quoted 4 lines · David Woodhouse

Apart from resolutely not wanting to implement variable length arrays on the stack, the LLVM folks actually seem quite keen to make things work. I'm interested in the problem you report above.. and note the absence of a bug number. Can you provide it?

I am currently on vacation and on super-lousy internet connection, so looking through my archives is a bit complicated ... I *think* it started in "[PATCH] usbhid: Fix lockdep unannotated irqs-off warning" thread on lkml.

In case you're not able to find it from there, I'll do my homework mid-next week when I am back properly online.

Thanks,

--
Jiri Kosina
SUSE Labs

_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#15 lore raw

Re: [Ksummit-discuss] [CORE TOPIC] [TECH TOPIC] Support (or move towards to) LLVM

In reply to Jiri Kosina
Quoted 7 lines · Jiri Kosina
Quoted 4 lines · David Woodhouse

Apart from resolutely not wanting to implement variable length arrays on the stack, the LLVM folks actually seem quite keen to make things work. I'm interested in the problem you report above.. and note the absence of a bug number. Can you provide it?

Quoted 7 lines

I am currently on vacation and on super-lousy internet connection, so looking through my archives is a bit complicated ... I *think* it started in "[PATCH] usbhid: Fix lockdep unannotated irqs-off warning" thread on lkml.

In case you're not able to find it from there, I'll do my homework mid-next week when I am back properly online.

OK, thanks. So it looks like that's acknowledged to have been a bug, there's a patch to fix it at https://reviews.llvm.org/D6629 which may even have been committed already.

The main sticking point for LLVM seems to be the variable length arrays on the stack (VLAIS), which is a GCC'ism that the LLVM/clang folks *really* don't want to support, and the fact that __builtin_constant_p() is basically always false under LLVM because it doesn't look very hard.

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#16 lore raw
In reply to Mark Brown
Quoted 2 lines · Mark Brown

There's a push from certain quarters to move away from GCC to LLVM. Not that these things are unachievable in LLVM but it's a thing.

I currently maintain a set of 26 cross-compilers (ppc64 is a box of symlinks) for building the kernel and bootloaders with gcc on Fedora:

	gcc-aarch64-linux-gnu.x86_64                 5.3.1-2.fc23   updates
	gcc-alpha-linux-gnu.x86_64                   5.3.1-2.fc23   updates
	gcc-arm-linux-gnu.x86_64                     5.3.1-2.fc23   updates
	gcc-avr32-linux-gnu.x86_64                   5.3.1-2.fc23   updates
	gcc-bfin-linux-gnu.x86_64                    5.3.1-2.fc23   updates
	gcc-c6x-linux-gnu.x86_64                     5.3.1-2.fc23   updates
	gcc-cris-linux-gnu.x86_64                    5.3.1-2.fc23   updates
	gcc-frv-linux-gnu.x86_64                     5.3.1-2.fc23   updates
	gcc-h8300-linux-gnu.x86_64                   5.3.1-2.fc23   updates
	gcc-hppa-linux-gnu.x86_64                    5.3.1-2.fc23   updates
	gcc-hppa64-linux-gnu.x86_64                  5.3.1-2.fc23   updates
	gcc-ia64-linux-gnu.x86_64                    5.3.1-2.fc23   updates
	gcc-m32r-linux-gnu.x86_64                    5.3.1-2.fc23   updates
	gcc-m68k-linux-gnu.x86_64                    5.3.1-2.fc23   updates
	gcc-microblaze-linux-gnu.x86_64              5.3.1-2.fc23   updates
	gcc-mips64-linux-gnu.x86_64                  5.3.1-2.fc23   updates
	gcc-mn10300-linux-gnu.x86_64                 5.3.1-2.fc23   updates
	gcc-nios2-linux-gnu.x86_64                   5.3.1-2.fc23   updates
	gcc-powerpc64-linux-gnu.x86_64               5.3.1-2.fc23   updates
	gcc-ppc64-linux-gnu.x86_64                   5.3.1-2.fc23   updates
	gcc-s390x-linux-gnu.x86_64                   5.3.1-2.fc23   updates
	gcc-sh-linux-gnu.x86_64                      5.3.1-2.fc23   updates
	gcc-sh64-linux-gnu.x86_64                    5.3.1-2.fc23   updates
	gcc-sparc64-linux-gnu.x86_64                 5.3.1-2.fc23   updates
	gcc-tile-linux-gnu.x86_64                    5.3.1-2.fc23   updates
	gcc-x86_64-linux-gnu.x86_64                  5.3.1-2.fc23   updates
	gcc-xtensa-linux-gnu.x86_64                  5.3.1-2.fc23   updates

These are gcc-6.1.1-based in fc24. Can LLVM cover all of these, plus the missing arches that gcc doesn't support yet?

David

#17 lore raw

On Fri, Jul 22, 2016 at 1:19 PM, David Howells <[email protected]> wrote:

I'm just gonna hijack this to ask a question...

Quoted 2 lines

I currently maintain a set of 26 cross-compilers (ppc64 is a box of symlinks) for building the kernel and bootloaders with gcc on Fedora:

(...)

Quoted 1 line
        gcc-arm-linux-gnu.x86_64                     5.3.1-2.fc23   updates

I've tested this thing and it works flawlessly with the kernel. Also U-boot recently supplies its own div64 routine and dropped the dependency on libgcc.a (IIRC).

So it works, but still of course this thing is there:

rpm -ql gcc-arm-linux-gnu-6.1.1-1.fc24.x86_64
(...)
/usr/lib/gcc/arm-linux-gnueabi/6.1.1/libgcc.a

Also this:

/usr/lib/gcc/arm-linux-gnueabi/6.1.1/crtbegin.o /usr/lib/gcc/arm-linux-gnueabi/6.1.1/crtbeginS.o /usr/lib/gcc/arm-linux-gnueabi/6.1.1/crtbeginT.o /usr/lib/gcc/arm-linux-gnueabi/6.1.1/crtend.o /usr/lib/gcc/arm-linux-gnueabi/6.1.1/crtendS.o /usr/lib/gcc/arm-linux-gnueabi/6.1.1/crtfastmath.o

I suspect these things are implicitly compiled for one and only one ISA? I honestly don't know how to even check which one. Does it use hardfloat? Or has this problem of targetting several ISAs with a .a file been fixes since I looked at it last?

I need to cross-compile ARMv4, ARMv5, ARMv6, ARMv7 and ARMv8 for my systems. (Sorry for all old crap I'm keeping it's just my job...)

That ABI problem is then manifolded if I would want to do anything userspace, so I keep a whole range of tailored cross compilers and prebuilt C libraries around that I know "just work", but I guess will grow increasingly hard to maintain and it feels pretty unelegant at times.

Is there a way for a distribution to provide a proper set of ISA-specific stuff alongside a crosscompiler, if the crosscompiler supports several different ISAs, like the ARMvN variants do?

Yours,
Linus Walleij

#18 lore raw
Quoted 5 lines · Linus Walleij

So it works, but still of course this thing is there:

rpm -ql gcc-arm-linux-gnu-6.1.1-1.fc24.x86_64
(...)
/usr/lib/gcc/arm-linux-gnueabi/6.1.1/libgcc.a

Some people really wanted libgcc libraries. Even some kernel arches require them.

Quoted 8 lines

I suspect these things are implicitly compiled for one and only one ISA? I honestly don't know how to even check which one. Does it use hardfloat? Or has this problem of targetting several ISAs with a .a file been fixes since I looked at it last?

I need to cross-compile ARMv4, ARMv5, ARMv6, ARMv7 and ARMv8 for my systems. (Sorry for all old crap I'm keeping it's just my job...)

The arm- compiler is built with:

	CONFIG_FLAGS="--with-tune=cortex-a8 --with-arch=armv7-a \
		--with-float=hard --with-fpu=vfpv3-d16 --with-abi=aapcs-linux"

I can probably add a --with-multilib-list= option to build multiple variants as I do for sh and sh64:

	sh-*)
	    CONFIG_FLAGS=--with-multilib-list=m1,m2,m2e,m2a,m2a-single,m4,m4-single,m4-single-only,m4-nofpu
	    ;;
	sh4-*)
	    CONFIG_FLAGS=--with-multilib-list=m4,m4-single,m4-single-only,m4-nofpu
	    ;;

if someone tells me what they'd like to see on there. In the sh case, this gives me all of:

	/usr/lib/gcc/sh-linux-gnu/5.3.1/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/m2/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/m2e/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/m4-nofpu/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/m4-single-only/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/m4-single/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/m4/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/mb/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/mb/m2/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/mb/m2a-single/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/mb/m2a/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/mb/m2e/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/mb/m4-nofpu/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/mb/m4-single-only/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/mb/m4-single/libgcc.a
	/usr/lib/gcc/sh-linux-gnu/5.3.1/mb/m4/libgcc.a
Quoted 5 lines

That ABI problem is then manifolded if I would want to do anything userspace, so I keep a whole range of tailored cross compilers and prebuilt C libraries around that I know "just work", but I guess will grow increasingly hard to maintain and it feels pretty unelegant at times.

Yeah. It's made more fun by the fact that if I want to do a general compiler SRPM that covers all the arches fully bootstrapped with C libraries, not every arch is supported by glibc, so I have to include uClibc as well for those arches.

Take sh as mentioned above, that has at least 16 different potential userspaces by the libgcc count.

Quoted 3 lines

Is there a way for a distribution to provide a proper set of ISA-specific stuff alongside a crosscompiler, if the crosscompiler supports several different ISAs, like the ARMvN variants do?

It *ought* to be possible to build C libraries separately, given an available cross-compiler - except that the gcc build wants stuff from the C library you're using. Having talked to some gcc people about this before, IIRC, it's something that could be managed without - if someone's willing to do the work to make gcc no longer require it.

David

#19 lore raw
In reply to Eric W. Biederman
Quoted 15 lines · Eric W. Biederman

Historically the types in C came about because the machines fundamentally supported different data types either with different sizes or different characteristics (i.e. u8, u16, float, double). These data types and the C type system were built so programmers could tell the machine what they needed it to do.

There is another genesis of types that started with the simply typed lambda calculs that is about eliminating common errors and otherwise helping a programmer get their code right.

In the years since C was invented there has been a lot of activity and a
little bit of progress in this area. Would people be receptive to improvements in this area?

Quoted 6 lines

I would like to talk to folks and gague what it would take to make improvements in this area acceptable, practical, and useful.

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

How would this be different from simply automatically running sparse in the kernel build if the binary is present (effectively making make C=1 the default)?

Quoted 3 lines

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

By this you mean a thread of execution that should be expected to free the data pointed to when it finishes? Sort of like a self garbage collecting reference?

Quoted 30 lines

* This cleanly allows for doubly linked lists.

* This is useful to ensure that data is either put in another data structure where it is remembered or it is freed.

* This is useful to ensure reference counts are not leaked.

* This is useful to ensure that every lock is paired with an
unlock.

My personal filter for things like this are types that can be checked in time proportional to the amount of code to be built so that it is roughly the same situation we are in now.

Given it's heritage and it's history the type system in C has serious limitations that I don't know if they are correctible, when it comes to catching programmer mistakes: silent truncation of unsigned types into smaller unsigned types, casts, etc. Would people be willing to consider a simple, link compatible alternative to C for some of the code in the kernel that had the same low level control of the machine but had a type system that made catching mistakes easier?

Deploying solutions like this will take a fair bit of grunt work, and time similar or worse than the big kernel lock removal. Given how widely Linux is used and how annoying some of these bugs can be I think it is worthwhile to dig in and see what kind of improvements can be made.

Quoted 4 lines

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

I've got to say that rewriting in some as yet undefined language simply to get better typing and conversions looks very daunting. Before even considering this, can you answer why an extension to sparse, which would mostly flag the problems if we use the correct annotations, wouldn't work just as well? We'd still have to add the additional annotations (and someone would have to update sparse) but it would still be a lot easier than rewriting and giving the kernel a build dependency on whatever the rewrite is done in.

James

#20 lore raw
Quoted 19 lines · James Bottomley
Quoted 18 lines · Eric W. Biederman

Historically the types in C came about because the machines fundamentally supported different data types either with different sizes or different characteristics (i.e. u8, u16, float, double). These data types and the C type system were built so programmers could tell the machine what they needed it to do.

There is another genesis of types that started with the simply typed lambda calculs that is about eliminating common errors and otherwise helping a programmer get their code right.

In the years since C was invented there has been a lot of activity and a
little bit of progress in this area. Would people be receptive to improvements in this area?

I would like to talk to folks and gague what it would take to make improvements in this area acceptable, practical, and useful.

Quoted 7 lines
Quoted 3 lines

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

How would this be different from simply automatically running sparse in the kernel build if the binary is present (effectively making make C=1 the default)?

Nothing. I am just honestly looking at ways that we can get things to
always or almost always run. Sparse isn't getting run regularly now so
I was suspect that would not be as good of a solution.

Quoted 7 lines
Quoted 3 lines

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

By this you mean a thread of execution that should be expected to free the data pointed to when it finishes? Sort of like a self garbage collecting reference?

Not really. The big idea is to make expressable in the type system the key concepts on how a programmer reasons about their data structures, instead of trying to make programmers perform some convoluted logic to express things in concepts that are trivial to implement in a type system.

Another way to talk about it would be complete alias analysis at type checking time.

Which means that the type system then knows essentially everything the programmer knows about aliases and object lifetimes and if you don't pass something on or free it, the type checker then knows you made a mistake.

Quoted 35 lines
Quoted 30 lines

* This cleanly allows for doubly linked lists.

* This is useful to ensure that data is either put in another data structure where it is remembered or it is freed.

* This is useful to ensure reference counts are not leaked.

* This is useful to ensure that every lock is paired with an
unlock.

My personal filter for things like this are types that can be checked in time proportional to the amount of code to be built so that it is roughly the same situation we are in now.

Given it's heritage and it's history the type system in C has serious limitations that I don't know if they are correctible, when it comes to catching programmer mistakes: silent truncation of unsigned types into smaller unsigned types, casts, etc. Would people be willing to consider a simple, link compatible alternative to C for some of the code in the kernel that had the same low level control of the machine but had a type system that made catching mistakes easier?

Deploying solutions like this will take a fair bit of grunt work, and time similar or worse than the big kernel lock removal. Given how widely Linux is used and how annoying some of these bugs can be I think it is worthwhile to dig in and see what kind of improvements can be made.

Quoted 4 lines

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Quoted 8 lines

I've got to say that rewriting in some as yet undefined language simply to get better typing and conversions looks very daunting. Before even considering this, can you answer why an extension to sparse, which would mostly flag the problems if we use the correct annotations, wouldn't work just as well? We'd still have to add the additional annotations (and someone would have to update sparse) but it would still be a lot easier than rewriting and giving the kernel a build dependency on whatever the rewrite is done in.

That is a very good question. The short answer is that I am not yet convinced I can figure out how to retrofit the type checks I want onto C's type system. I have implemented the type checks I am looking at in a fresh language I am toying with where I can throw out inconvinient unnecessary corner cases. After I finished boiling the ocean the basic types were similar enough to C's type system that I think I can make the connection but I haven't tried yet.

Assuming things can be sorted so they coexist nicely C's types. No advantage. If it turns out that to make the code not a pain to write we need better syntax or some basic type inference there is a large advantage in converting code.

The first place I can think of where I might get hung up in the area of discriminated unions. I am pretty certain that some cases of type safety will require them and C doesn't have anything like that today. A practical example would be a type system that ensures you call IS_ERR(ptr) before you use a pointer as a pointer.

Ordinary C unions and type casts that serve the same purpose would at the end of the day all have to be reabstracted. Because otherwise memory safety cound not be analyized.

Some of that reabstraction I will need to introduce type variables which a completely foreing concept to C.

On the flip side since the linux kernel and low level programming where precise control of the machine happens is my target audience it is definitely worth looking at what can be done in a context like sparse. That is the fastest way to connect the tools with the real world problems. If it can't be done or if it can't be made nice simple and easy to use then there will be good arguments for why we need a new syntax.

Mostly I asked the question because the type system would be so much easier in a green field setting, and I wanted to be lazy. So hearing the common sense request (Please fix C.) Is helpful to get that lazy part of myself in gear.

The end game for me is a type system that doesn't permit memory errors, reference counting errors, locking errors, or deadlocks, while at the same time allowing pretty much the code and data structures that we use in the kernel today.

Eric

#21 lore raw
Quoted 25 lines · Eric W. Biederman
Quoted 23 lines · James Bottomley
Quoted 21 lines · Eric W. Biederman

Historically the types in C came about because the machines fundamentally supported different data types either with different sizes or different characteristics (i.e. u8, u16, float, double). These data types and the C type system were built so programmers could tell the machine what they needed it to do.

There is another genesis of types that started with the simply typed lambda calculs that is about eliminating common errors and otherwise helping a programmer get their code right.

In the years since C was invented there has been a lot of activity and a
little bit of progress in this area. Would people be receptive to improvements in this area?

I would like to talk to folks and gague what it would take to make improvements in this area acceptable, practical, and useful.

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

Quoted 7 lines
Quoted 3 lines

How would this be different from simply automatically running sparse in the kernel build if the binary is present (effectively making make C=1 the default)?

Nothing. I am just honestly looking at ways that we can get things to
always or almost always run. Sparse isn't getting run regularly now so
I was suspect that would not be as good of a solution.

A problem with putting time consuming tools into the normal build process is that they work on all the code you compile, which may be a large superset of the code that is actually changed.

julia

Quoted 117 lines
Quoted 8 lines
Quoted 4 lines

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

By this you mean a thread of execution that should be expected to free the data pointed to when it finishes? Sort of like a self garbage collecting reference?

Not really. The big idea is to make expressable in the type system the key concepts on how a programmer reasons about their data structures, instead of trying to make programmers perform some convoluted logic to express things in concepts that are trivial to implement in a type system.

Another way to talk about it would be complete alias analysis at type checking time.

Which means that the type system then knows essentially everything the programmer knows about aliases and object lifetimes and if you don't pass something on or free it, the type checker then knows you made a mistake.

Quoted 35 lines
Quoted 30 lines

* This cleanly allows for doubly linked lists.

* This is useful to ensure that data is either put in another data structure where it is remembered or it is freed.

* This is useful to ensure reference counts are not leaked.

* This is useful to ensure that every lock is paired with an
unlock.

My personal filter for things like this are types that can be checked in time proportional to the amount of code to be built so that it is roughly the same situation we are in now.

Given it's heritage and it's history the type system in C has serious limitations that I don't know if they are correctible, when it comes to catching programmer mistakes: silent truncation of unsigned types into smaller unsigned types, casts, etc. Would people be willing to consider a simple, link compatible alternative to C for some of the code in the kernel that had the same low level control of the machine but had a type system that made catching mistakes easier?

Deploying solutions like this will take a fair bit of grunt work, and time similar or worse than the big kernel lock removal. Given how widely Linux is used and how annoying some of these bugs can be I think it is worthwhile to dig in and see what kind of improvements can be made.

Quoted 4 lines

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Quoted 8 lines

I've got to say that rewriting in some as yet undefined language simply to get better typing and conversions looks very daunting. Before even considering this, can you answer why an extension to sparse, which would mostly flag the problems if we use the correct annotations, wouldn't work just as well? We'd still have to add the additional annotations (and someone would have to update sparse) but it would still be a lot easier than rewriting and giving the kernel a build dependency on whatever the rewrite is done in.

That is a very good question. The short answer is that I am not yet convinced I can figure out how to retrofit the type checks I want onto C's type system. I have implemented the type checks I am looking at in a fresh language I am toying with where I can throw out inconvinient unnecessary corner cases. After I finished boiling the ocean the basic types were similar enough to C's type system that I think I can make the connection but I haven't tried yet.

Assuming things can be sorted so they coexist nicely C's types. No advantage. If it turns out that to make the code not a pain to write we need better syntax or some basic type inference there is a large advantage in converting code.

The first place I can think of where I might get hung up in the area of discriminated unions. I am pretty certain that some cases of type safety will require them and C doesn't have anything like that today. A practical example would be a type system that ensures you call IS_ERR(ptr) before you use a pointer as a pointer.

Ordinary C unions and type casts that serve the same purpose would at the end of the day all have to be reabstracted. Because otherwise memory safety cound not be analyized.

Some of that reabstraction I will need to introduce type variables which a completely foreing concept to C.

On the flip side since the linux kernel and low level programming where precise control of the machine happens is my target audience it is definitely worth looking at what can be done in a context like sparse. That is the fastest way to connect the tools with the real world problems. If it can't be done or if it can't be made nice simple and easy to use then there will be good arguments for why we need a new syntax.

Mostly I asked the question because the type system would be so much easier in a green field setting, and I wanted to be lazy. So hearing the common sense request (Please fix C.) Is helpful to get that lazy part of myself in gear.

The end game for me is a type system that doesn't permit memory errors, reference counting errors, locking errors, or deadlocks, while at the same time allowing pretty much the code and data structures that we use in the kernel today.

Eric
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#22 lore raw
In reply to Eric W. Biederman
Quoted 24 lines · Eric W. Biederman
Quoted 23 lines · James Bottomley
Quoted 21 lines · Eric W. Biederman

Historically the types in C came about because the machines fundamentally supported different data types either with different sizes or different characteristics (i.e. u8, u16, float, double). These data types and the C type system were built so programmers could tell the machine what they needed it to do.

There is another genesis of types that started with the simply typed lambda calculs that is about eliminating common errors and otherwise helping a programmer get their code right.

In the years since C was invented there has been a lot of activity and a
little bit of progress in this area. Would people be receptive to improvements in this area?

I would like to talk to folks and gague what it would take to make improvements in this area acceptable, practical, and useful.

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

Quoted 7 lines
Quoted 3 lines

How would this be different from simply automatically running sparse in the kernel build if the binary is present (effectively making make C=1 the default)?

Nothing. I am just honestly looking at ways that we can get things to
always or almost always run. Sparse isn't getting run regularly now so
I was suspect that would not be as good of a solution.

Isn't sparse run by 0-day testing? I thought it is...

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#23 lore raw
Quoted 5 lines · On Wed, 20 Jul 2016 14:11:10 +0200 Jan Kara <[email protected]> wrote:
Quoted 3 lines

Nothing. I am just honestly looking at ways that we can get things to
always or almost always run. Sparse isn't getting run regularly now so
I was suspect that would not be as good of a solution.

Isn't sparse run by 0-day testing? I thought it is...

I get sparse warnings from the 0-day testing, so I believe that it is.

-- Steve

#24 lore raw
In reply to Eric W. Biederman
Quoted 2 lines · Eric W. Biederman

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

I'd like to see those checks more widely available, ideally not just as plugins. Some exploration of that occurred upstream:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59852 (bitwise/endian types)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59856 (contexts/locking)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59851 (nocast: no implicit conversions)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59850 (address spaces)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59855 (designated_init; done)

I'd love to see someone pick those up and get them into upstream GCC.

Quoted 3 lines

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

Interesting, yes, but trying to track "ownership" gets complicated *fast* to handle real-world cases. Rust went through quite a lot of work, and multiple iterations, to get to the system it has now. I don't think you'd be able to handle many of the cases in the kernel without about that much complexity.

Quoted 4 lines

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

I think the biggest constraint is that new tools get very slow adoption, and it's incredibly difficult to introduce a new *mandatory* tool or compiler version (with the exception of tools that ship with the kernel). And optional ones have a tendency to break due to patches from people not running them. Apart from that: false positive rate.

Ideally, build something you can opt into using, such that if you explicitly use it, the false positive rate should be *zero* by design.

- Josh Triplett

#25 lore raw
Quoted 7 lines · Josh Triplett
Quoted 2 lines · Eric W. Biederman

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

I'd like to see those checks more widely available, ideally not just as plugins. Some exploration of that occurred upstream:

Quoted 7 lines

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59852 (bitwise/endian types)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59856 (contexts/locking)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59851 (nocast: no implicit conversions)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59850 (address spaces)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59855 (designated_init; done)

I'd love to see someone pick those up and get them into upstream GCC.

Interesting.

Whatever I do I intend to start and the hard problem of types that know where all of your aliases are because I have something that works and is very interesting there, so I won't volunteer for those but that does look like a reasonable part of the discussion.

Quoted 9 lines
Quoted 3 lines

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

Interesting, yes, but trying to track "ownership" gets complicated *fast* to handle real-world cases. Rust went through quite a lot of work, and multiple iterations, to get to the system it has now. I don't think you'd be able to handle many of the cases in the kernel without about that much complexity.

So "ownership" may be the wrong word. See my reply to James Bottomley. Rust never advanced past what are effectively smart pointers, and smart pointers are the wrong concept for tracking aliases.

To get past that it takes some turning of all of your trained exceptions of what a type system is tracking inside out (or at least it did for me) but when you are done there is something that is much more expressive and simpler than what Rust implemented.

Our fundamental complexity with smp synchronization may add up the complexity again but it is undoubtedly possible to do better than Rust. The way our kernel is built all of the core kernel would need to be in a Rust unsafe block. So I would not even consider doing what Rust did.

Quoted 5 lines
Quoted 4 lines

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Quoted 8 lines

I think the biggest constraint is that new tools get very slow adoption, and it's incredibly difficult to introduce a new *mandatory* tool or compiler version (with the exception of tools that ship with the kernel). And optional ones have a tendency to break due to patches from people not running them. Apart from that: false positive rate.

Ideally, build something you can opt into using, such that if you explicitly use it, the false positive rate should be *zero* by design.

With types what you do get a *zero* false positive rate by desgin. Although sometimes you get a lot of const correctness kinds of conversion requitements that are a royal PITA.

Eric

#26 lore raw
In reply to Josh Triplett
Quoted 19 lines · Josh Triplett
Quoted 2 lines · Eric W. Biederman

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

I'd like to see those checks more widely available, ideally not just as plugins. Some exploration of that occurred upstream:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59852 (bitwise/endian types)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59856 (contexts/locking)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59851 (nocast: no implicit conversions)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59850 (address spaces)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59855 (designated_init; done)

I'd love to see someone pick those up and get them into upstream GCC.

Quoted 3 lines

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

Quoted 5 lines

Interesting, yes, but trying to track "ownership" gets complicated *fast* to handle real-world cases. Rust went through quite a lot of work, and multiple iterations, to get to the system it has now. I don't think you'd be able to handle many of the cases in the kernel without about that much complexity.

Politely Rust did it the stupid way. "ownership" or perhaps better said who is allowed to modify the data is an active piece of code thing rather than a data thing, and Rust did it as a data thing.

The kernel has some very interesting data structures and all kinds of
hand rolled smp syncrhonization primitives. So I don't doubt there
will be challenges, but the situation will look nothing like what Rust did.

Quoted 5 lines
Quoted 4 lines

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Quoted 8 lines

I think the biggest constraint is that new tools get very slow adoption, and it's incredibly difficult to introduce a new *mandatory* tool or compiler version (with the exception of tools that ship with the kernel). And optional ones have a tendency to break due to patches from people not running them. Apart from that: false positive rate.

Ideally, build something you can opt into using, such that if you explicitly use it, the false positive rate should be *zero* by design.

Which is what makes types as opposed to other things nice. Types arguably don't have a false positive rate. Unfortunately types can sometimes be very pendantic and if everyone isn't using them things go splat.

Eric

#27 lore raw
Quoted 28 lines · Eric W. Biederman
Quoted 19 lines · Josh Triplett
Quoted 2 lines · Eric W. Biederman

Would a gcc plugin that checks the most interesting things that sparse checks on every build be interesting? (endianness of integer types for example)

I'd like to see those checks more widely available, ideally not just as plugins. Some exploration of that occurred upstream:

https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59852 (bitwise/endian types)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59856 (contexts/locking)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59851 (nocast: no implicit conversions)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59850 (address spaces)
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=59855 (designated_init; done)

I'd love to see someone pick those up and get them into upstream GCC.

Quoted 3 lines

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

Quoted 5 lines

Interesting, yes, but trying to track "ownership" gets complicated *fast* to handle real-world cases. Rust went through quite a lot of work, and multiple iterations, to get to the system it has now. I don't think you'd be able to handle many of the cases in the kernel without about that much complexity.

Quoted 3 lines

Politely Rust did it the stupid way. "ownership" or perhaps better said who is allowed to modify the data is an active piece of code thing rather than a data thing, and Rust did it as a data thing.

I'd be interested to hear more details on what you mean by this, because the way you've described it doesn't make sense to me. The way lifetimes are implemented in Rust seems very much like "an active piece of code thing". Can you give an example of the distinction you're making?

#28 lore raw
Quoted 15 lines · Josh Triplett
Quoted 14 lines · Eric W. Biederman
Quoted 10 lines · Josh Triplett
Quoted 3 lines · Eric W. Biederman

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

Interesting, yes, but trying to track "ownership" gets complicated *fast* to handle real-world cases. Rust went through quite a lot of work, and multiple iterations, to get to the system it has now. I don't think you'd be able to handle many of the cases in the kernel without about that much complexity.

Quoted 8 lines
Quoted 3 lines

Politely Rust did it the stupid way. "ownership" or perhaps better said who is allowed to modify the data is an active piece of code thing rather than a data thing, and Rust did it as a data thing.

I'd be interested to hear more details on what you mean by this, because the way you've described it doesn't make sense to me. The way lifetimes are implemented in Rust seems very much like "an active piece of code thing". Can you give an example of the distinction you're making?

But that is a lifetime of a piece of data, that isn't ownership of data. The data is still owned by some pointer.

Ownership by code looks roughly like:

	head = acquire_list(&task_list);

	/* At the data level head points to the first element of the list */
        /* The type of head is a recursive type that includes every
         * element on the list.
         */
         for (ptr = head; ptr; ptr = ptr->next) {
         	/* The type of ptr shares with head the type of the list.
                 * Which allows both ptr and head to be valid
                 * and the code of this function to continue owning the list.
                 */
                 ptr->scratch++;
                 /* As the owner any mutation may be performed on the list elements */
	}

	release_list(list);
        /* Accessing list past this point would be a type error */

Where acquire_list and grabs the appropriate spinlock and then returns ownership of the list to the calling function through a ponter to the first element of the list.

Similarly release_list takes a pointer to the first element of a list and the ownership of the list releases the lock.

In the area in which you have ownership of the list you can use as many different pointers as you like and they are all valid and all may be used because the code can see all of the aliases, and nothing special needs to be done because the code owns the list not any particular pointer.

Which means that since one element of the list does not contain a pointer owning the next element of the list, doubly linked lists are not a problem.

This is not to say there are not tricky bits, but things like borrows are replaced by simpler concepts.

I hope that is enough to clarify where I am coming from. If not I suspect it will have to wait until I can find the time to release my proof of concept and documentation of all of this.

Eric

#29 lore raw
Quoted 49 lines · Eric W. Biederman
Quoted 15 lines · Josh Triplett
Quoted 14 lines · Eric W. Biederman
Quoted 10 lines · Josh Triplett
Quoted 3 lines · Eric W. Biederman

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

Interesting, yes, but trying to track "ownership" gets complicated *fast* to handle real-world cases. Rust went through quite a lot of work, and multiple iterations, to get to the system it has now. I don't think you'd be able to handle many of the cases in the kernel without about that much complexity.

Quoted 8 lines
Quoted 3 lines

Politely Rust did it the stupid way. "ownership" or perhaps better said who is allowed to modify the data is an active piece of code thing rather than a data thing, and Rust did it as a data thing.

I'd be interested to hear more details on what you mean by this, because the way you've described it doesn't make sense to me. The way lifetimes are implemented in Rust seems very much like "an active piece of code thing". Can you give an example of the distinction you're making?

But that is a lifetime of a piece of data, that isn't ownership of data. The data is still owned by some pointer.

Ownership by code looks roughly like:

	head = acquire_list(&task_list);

	/* At the data level head points to the first element of the list */
        /* The type of head is a recursive type that includes every
         * element on the list.
         */
         for (ptr = head; ptr; ptr = ptr->next) {
         	/* The type of ptr shares with head the type of the list.
                 * Which allows both ptr and head to be valid
                 * and the code of this function to continue owning the list.
                 */
                 ptr->scratch++;
                 /* As the owner any mutation may be performed on the list elements */
	}

	release_list(list);
        /* Accessing list past this point would be a type error */
Quoted 3 lines

Where acquire_list and grabs the appropriate spinlock and then returns ownership of the list to the calling function through a ponter to the first element of the list.

(Nit: I think in that last line you wanted release_list(head);)

What happens if the code saves a copy of either head, ptr, or something accessed via ptr, and then some other distant code accesses that after release_list? (For instance, consider a function that takes a pointer to "scratch" and retains that pointer.) In the type system you envision, what prevents that?

What happens if the code in that region, while looking at one value of ptr, changes the list in a way that invalidates ptr? What prevents that?

#30 lore raw
Quoted 47 lines · Josh Triplett
Quoted 46 lines · Eric W. Biederman
Quoted 15 lines · Josh Triplett
Quoted 14 lines · Eric W. Biederman
Quoted 10 lines · Josh Triplett
Quoted 3 lines · Eric W. Biederman

Would a type system for pointers derived from separation logic that has the concept that a piece of data is owned by a piece of running code rather than another piece of data be interesting?

Interesting, yes, but trying to track "ownership" gets complicated *fast* to handle real-world cases. Rust went through quite a lot of work, and multiple iterations, to get to the system it has now. I don't think you'd be able to handle many of the cases in the kernel without about that much complexity.

Quoted 8 lines
Quoted 3 lines

Politely Rust did it the stupid way. "ownership" or perhaps better said who is allowed to modify the data is an active piece of code thing rather than a data thing, and Rust did it as a data thing.

I'd be interested to hear more details on what you mean by this, because the way you've described it doesn't make sense to me. The way lifetimes are implemented in Rust seems very much like "an active piece of code thing". Can you give an example of the distinction you're making?

But that is a lifetime of a piece of data, that isn't ownership of data. The data is still owned by some pointer.

Ownership by code looks roughly like:

	head = acquire_list(&task_list);

	/* At the data level head points to the first element of the list */
        /* The type of head is a recursive type that includes every
         * element on the list.
         */
         for (ptr = head; ptr; ptr = ptr->next) {
         	/* The type of ptr shares with head the type of the list.
                 * Which allows both ptr and head to be valid
                 * and the code of this function to continue owning the list.
                 */
                 ptr->scratch++;
                 /* As the owner any mutation may be performed on the list elements */
	}
Quoted 8 lines
Quoted 6 lines
	release_list(list);
        /* Accessing list past this point would be a type error */

Where acquire_list and grabs the appropriate spinlock and then returns ownership of the list to the calling function through a ponter to the first element of the list.

(Nit: I think in that last line you wanted release_list(head);)

Yes.

Quoted 5 lines

What happens if the code saves a copy of either head, ptr, or something accessed via ptr, and then some other distant code accesses that after release_list? (For instance, consider a function that takes a pointer to "scratch" and retains that pointer.) In the type system you envision, what prevents that?

It depends a bit on how the copy is saved. Without something special when you reach the line release_list(head) the return of release_list consumes the list as free does.

Which type wise means that the pointers you talk of remain valid things, but you can not dereference them because the type they point to has
become unusable. Which makes all future derefernces impossible.

Quoted 3 lines

What happens if the code in that region, while looking at one value of ptr, changes the list in a way that invalidates ptr? What prevents that?

Depends. If it invalides ptr. ptr becomes unusable (at least undereferenceable). And using dereferencing pointer past that point does not type check.

Which is a long way of saying that functions are allowed to change the destination types of pointers that are passed into them. This changing what pointers point to is the type equivalent of having preconditions and postconditions.

Plus there is the other benefit to being able to change the type of things by functions. It is possible to build types that enforce state machines by changing a the type of a variable and only making methods available when they are valid to use.

And yes all of this depends upon having enough information to know (within a small window) where all of the aliases for any value are at any given time.

Eric

#31 lore raw
In reply to Eric W. Biederman

I know it's not precisely what you're asking about, but there are a number of types I would like to see:

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

Some arches, x86_64 and ppc64 for example, can do 32-bit atomic ops, so we could make the field smaller in some cases.

We have a *lot* of flags fields in the kernel, so I wonder if we could actually save any space.

I seem to remember that the argument is (or was) that the type must be the natural word size of the machine, but how true is that in actuality?

(2) Differentiate non-BH spinlocks and BH spinlocks by type.

It seems like you can't mix BH and non-BH ops on a spinlock without lockdep barking. If that's the case, let's make this a compile-time check.

(3) Let's use bool a lot more for boolean values as the compiler might be able to make better choices with it.

And whilst we're at it, a function that I'd like to see:

(1) on_list() in addition to list_empty() (and similar for other list types).

I know this would be kind of redundant as is would be implemented exactly the same as list_empty() - but semantically you're asking a different question.

David

#32 lore raw
Quoted 7 lines · David Howells

I know it's not precisely what you're asking about, but there are a number of types I would like to see:

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

What is wrong with using DECLARE_BITMAP()? It will allocate exactly as many unsigned longs as needed (so it will be 1 on 64 and 2 on 32 arches for bitmap in range [33, 64]).

Thanks.

Signature · 1 line
Dmitry
#33 lore raw
Quoted 7 lines · Dmitry Torokhov
Quoted 4 lines

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

What is wrong with using DECLARE_BITMAP()? It will allocate exactly as many unsigned longs

You missed my point. *unsigned long* is the issue. The majority of the time that wastes 32 bits on a 64-bit machine - especially when we don't need that many flags. On some 64-bit arches we could use unsigned int instead.

David

#34 lore raw
Quoted 7 lines · David Howells
Quoted 5 lines · Dmitry Torokhov
Quoted 4 lines

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

Quoted 6 lines
Quoted 2 lines

What is wrong with using DECLARE_BITMAP()? It will allocate exactly as many unsigned longs

You missed my point. *unsigned long* is the issue. The majority of the time that wastes 32 bits on a 64-bit machine - especially when we don't need that many flags. On some 64-bit arches we could use unsigned int instead.

Indeed, but please call them flags_t and flags64_t.
test_bit et all can dispatch with __builtin_choose_expr or even _Generic.

#35 lore raw
Quoted 1 line · Alexey Dobriyan

Indeed, but please call them flags_t and flags64_t.

Fine by me, though I was trying to avoid confusion with the argument to IRQ munging functions.

Quoted 1 line

test_bit et all can dispatch with __builtin_choose_expr or even _Generic.

Or by __atomic_fetch_{and,or,xor}().

David

#36 lore raw
Quoted 6 lines · David Howells
Quoted 1 line · Alexey Dobriyan

Indeed, but please call them flags_t and flags64_t.

Fine by me, though I was trying to avoid confusion with the argument to IRQ munging functions.

I can resurrect irq_flags_t but there is no procedure for merging 1MB+ diffs without spending whole life splitting and emailing maintainers.

#37 lore raw
Quoted 10 lines · Alexey Dobriyan
Quoted 6 lines · David Howells
Quoted 1 line · Alexey Dobriyan

Indeed, but please call them flags_t and flags64_t.

Fine by me, though I was trying to avoid confusion with the argument to IRQ munging functions.

I can resurrect irq_flags_t but there is no procedure for merging 1MB+ diffs without spending whole life splitting and emailing maintainers.

I think the procedure is that you have a script (e.g. coccinelle) to make such patch and Linus applies it himself just before/after rc1.

Quoted 5 lines

_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#38 lore raw
Quoted 5 lines · Vlastimil Babka
Quoted 2 lines

I can resurrect irq_flags_t but there is no procedure for merging 1MB+ diffs without spending whole life splitting and emailing maintainers.

I think the procedure is that you have a script (e.g. coccinelle) to make such patch and Linus applies it himself just before/after rc1.

Can coccinelle manage changing the local variable declaration, especially if someone does:

	unsigned long flags, fred;

where fred is unrelated.

But, on the other hand, it should be possible to compile test very easily if irq_flags_t is a typedef'd struct.

David

#39 lore raw
Quoted 8 lines · David Howells
Quoted 5 lines · Vlastimil Babka
Quoted 2 lines

I can resurrect irq_flags_t but there is no procedure for merging 1MB+ diffs without spending whole life splitting and emailing maintainers.

I think the procedure is that you have a script (e.g. coccinelle) to make such patch and Linus applies it himself just before/after rc1.

Quoted 6 lines

Can coccinelle manage changing the local variable declaration, especially if someone does:

	unsigned long flags, fred;

where fred is unrelated.

Recent versions of Coccinelle should allow this. Older versions would crash, so you would at least know that there was a problem.

julia

Quoted 10 lines

But, on the other hand, it should be possible to compile test very easily if irq_flags_t is a typedef'd struct.

David
_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#40 lore raw
In reply to David Howells
Quoted 7 lines · David Howells
Quoted 5 lines · Dmitry Torokhov
Quoted 4 lines

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

Quoted 6 lines
Quoted 2 lines

What is wrong with using DECLARE_BITMAP()? It will allocate exactly as many unsigned longs

You missed my point. *unsigned long* is the issue. The majority of the time that wastes 32 bits on a 64-bit machine - especially when we don't need that many flags. On some 64-bit arches we could use unsigned int instead.

I was responding to your statement where you were saying you could not use bits 32 - 63 as they might not exist. If you have so many bits then DECLARE_BITMAP is useful. Otherwise we can simply use u{8|16|32} and BIT() macro.

Thanks.

Signature · 1 line
Dmitry
#41 lore raw
Quoted 11 lines · Dmitry Torokhov
Quoted 10 lines · David Howells
Quoted 7 lines · Dmitry Torokhov
Quoted 4 lines

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

What is wrong with using DECLARE_BITMAP()? It will allocate exactly as many unsigned longs

Quoted 9 lines
Quoted 3 lines

You missed my point. *unsigned long* is the issue. The majority of the time that wastes 32 bits on a 64-bit machine - especially when we don't need that many flags. On some 64-bit arches we could use unsigned int instead.

I was responding to your statement where you were saying you could not use bits 32 - 63 as they might not exist. If you have so many bits then DECLARE_BITMAP is useful. Otherwise we can simply use u{8|16|32} and BIT() macro.

... unless one needs to statically initialize the bitmap. At least for my part I have not been able to find a means to initialize a static bitmap with anything but 0. Effectively this means that such bitmaps have to use u{8|16|32|64} and thus can not use any of the bitmap operations, or one has to use unsigned long and thus can only use 32 bit.

Guenter

#42 lore raw
In reply to David Howells
Quoted 27 lines · David Howells

I know it's not precisely what you're asking about, but there are a number of types I would like to see:

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

Some arches, x86_64 and ppc64 for example, can do 32-bit atomic ops, so we could make the field smaller in some cases.

We have a *lot* of flags fields in the kernel, so I wonder if we could actually save any space.

I seem to remember that the argument is (or was) that the type must be the natural word size of the machine, but how true is that in actuality?

(2) Differentiate non-BH spinlocks and BH spinlocks by type.

It seems like you can't mix BH and non-BH ops on a spinlock without lockdep barking. If that's the case, let's make this a compile-time check.

(3) Let's use bool a lot more for boolean values as the compiler might be able to make better choices with it.

Quoted 8 lines

And whilst we're at it, a function that I'd like to see:

(1) on_list() in addition to list_empty() (and similar for other list types).

I know this would be kind of redundant as is would be implemented exactly the same as list_empty() - but semantically you're asking a different question.

Actually that's one thing on my long-term to-do list: stricter range checking for kernel functions.
enums are an obvious way to address that, but there is a broad range of functions which return an errno value without ever exhausting the entire errno range.
Which makes it really hard from the calling function to figure out which errno values are legit and which can be considered an error.

If we had a programmatic way of addressing that we could
a) validate the usage of those functions
and
b) easily do error injection with systemtap as we would know which values the systemtap function should return

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		               zSeries & Storage
[email protected]			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
#43 lore raw
Quoted 32 lines · Hannes Reinecke
Quoted 31 lines · David Howells

I know it's not precisely what you're asking about, but there are a number of types I would like to see:

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

Some arches, x86_64 and ppc64 for example, can do 32-bit atomic ops, so we could make the field smaller in some cases.

We have a *lot* of flags fields in the kernel, so I wonder if we could actually save any space.

I seem to remember that the argument is (or was) that the type must be the natural word size of the machine, but how true is that in actuality?

(2) Differentiate non-BH spinlocks and BH spinlocks by type.

It seems like you can't mix BH and non-BH ops on a spinlock without lockdep barking. If that's the case, let's make this a compile-time check.

(3) Let's use bool a lot more for boolean values as the compiler might be able to make better choices with it.

And whilst we're at it, a function that I'd like to see:

(1) on_list() in addition to list_empty() (and similar for other list types).

Quoted 7 lines
Quoted 3 lines

I know this would be kind of redundant as is would be implemented exactly the same as list_empty() - but semantically you're asking a different question.

Actually that's one thing on my long-term to-do list: stricter range checking for kernel functions.
enums are an obvious way to address that,

In C, enums are ints. Is there a Gcc option that checks them? For example, the following program compiles fine with -Wall:

enum one {ONE=1, TWO=2, THREE=3};
enum two {ONEX=7, TWOX=8, THREEX=9};

int f (int x) {
enum one o = ONE;
enum two t = THREEX;
if (x) o = t; else t = o;
return 0;
}

Quoted 3 lines

but there is a broad range of
functions which return an errno value without ever exhausting the entire errno range.

I guess that almost all functions return only a few possible error codes? But only a few call sites check the values either.

In terms of collecting this information, one would quickly run into function pointers, but that would probably not be a insurmountable obstacle.

julia

Quoted 23 lines

Which makes it really hard from the calling function to figure out which errno values are legit and which can be considered an error.

If we had a programmatic way of addressing that we could
a) validate the usage of those functions
and
b) easily do error injection with systemtap as we would know which values the systemtap function should return

Cheers,

Hannes
--
Dr. Hannes Reinecke zSeries & Storage
[email protected] +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, D. Upmanyu, G. Norton HRB 21284 (AG Nürnberg)
_______________________________________________
Ksummit-discuss mailing list [email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#44 lore raw
Quoted 50 lines · Julia Lawall
Quoted 32 lines · Hannes Reinecke
Quoted 31 lines · David Howells

I know it's not precisely what you're asking about, but there are a number of types I would like to see:

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

Some arches, x86_64 and ppc64 for example, can do 32-bit atomic ops, so we could make the field smaller in some cases.

We have a *lot* of flags fields in the kernel, so I wonder if we could actually save any space.

I seem to remember that the argument is (or was) that the type must be the natural word size of the machine, but how true is that in actuality?

(2) Differentiate non-BH spinlocks and BH spinlocks by type.

It seems like you can't mix BH and non-BH ops on a spinlock without lockdep barking. If that's the case, let's make this a compile-time check.

(3) Let's use bool a lot more for boolean values as the compiler might be able to make better choices with it.

And whilst we're at it, a function that I'd like to see:

(1) on_list() in addition to list_empty() (and similar for other list types).

Quoted 7 lines
Quoted 3 lines

I know this would be kind of redundant as is would be implemented exactly the same as list_empty() - but semantically you're asking a different question.

Actually that's one thing on my long-term to-do list: stricter range checking for kernel functions.
enums are an obvious way to address that,

In C, enums are ints. Is there a Gcc option that checks them? For example, the following program compiles fine with -Wall:

enum one {ONE=1, TWO=2, THREE=3};
enum two {ONEX=7, TWOX=8, THREEX=9};

Quoted 8 lines

int f (int x) {
enum one o = ONE;
enum two t = THREEX;
if (x) o = t; else t = o;
return 0;
}

From what I know GCC only checks enums if they are used in a switch()

statement. Other types are not checked.

Quoted 5 lines
Quoted 3 lines

but there is a broad range of
functions which return an errno value without ever exhausting the entire errno range.

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

Quoted 2 lines

But only a few call sites check the values either.

I know. But this is primarily because the caller itself isn't quite sure which values to expect.
ATM one has to rely on the function documentation if existing. And even that might be outdated.

Quoted 4 lines

In terms of collecting this information, one would quickly run into function pointers, but that would probably not be a insurmountable obstacle.

But then even the functions pointed to could / should use this syntax, so we would have a set of all possible return codes.
The important bit here is that we would be able to check programmatically which values can be returned.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		               zSeries & Storage
[email protected]			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
#45 lore raw
Quoted 78 lines · Hannes Reinecke
Quoted 50 lines · Julia Lawall
Quoted 32 lines · Hannes Reinecke
Quoted 31 lines · David Howells

I know it's not precisely what you're asking about, but there are a number of types I would like to see:

(1) A 'bits' and maybe a 'bits64' type. Currently you have to use unsigned long when you want to deploy a flags field with which you're going to use test_bit() and co. - but this typically wastes 32 bits on a 64-bit arch because you can't use bits 32-63 as they might not exist.

Some arches, x86_64 and ppc64 for example, can do 32-bit atomic ops, so we could make the field smaller in some cases.

We have a *lot* of flags fields in the kernel, so I wonder if we could actually save any space.

I seem to remember that the argument is (or was) that the type must be the natural word size of the machine, but how true is that in actuality?

(2) Differentiate non-BH spinlocks and BH spinlocks by type.

It seems like you can't mix BH and non-BH ops on a spinlock without lockdep barking. If that's the case, let's make this a compile-time check.

(3) Let's use bool a lot more for boolean values as the compiler might be able to make better choices with it.

And whilst we're at it, a function that I'd like to see:

(1) on_list() in addition to list_empty() (and similar for other list types).

Quoted 7 lines
Quoted 3 lines

I know this would be kind of redundant as is would be implemented exactly the same as list_empty() - but semantically you're asking a different question.

Actually that's one thing on my long-term to-do list: stricter range checking for kernel functions.
enums are an obvious way to address that,

In C, enums are ints. Is there a Gcc option that checks them? For example, the following program compiles fine with -Wall:

enum one {ONE=1, TWO=2, THREE=3};
enum two {ONEX=7, TWOX=8, THREEX=9};

Quoted 7 lines

int f (int x) {
enum one o = ONE;
enum two t = THREEX;
if (x) o = t; else t = o;
return 0;
}

From what I know GCC only checks enums if they are used in a switch() statement. Other types are not checked.

Quoted 5 lines
Quoted 3 lines

but there is a broad range of
functions which return an errno value without ever exhausting the entire errno range.

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

Quoted 2 lines

But only a few call sites check the values either.

I know. But this is primarily because the caller itself isn't quite sure which values to expect.
ATM one has to rely on the function documentation if existing. And even that might be outdated.

Quoted 8 lines
Quoted 4 lines

In terms of collecting this information, one would quickly run into function pointers, but that would probably not be a insurmountable obstacle.

But then even the functions pointed to could / should use this syntax, so we would have a set of all possible return codes.
The important bit here is that we would be able to check programmatically which values can be returned.

Thanks for the clarifications. Perhaps it could be possible.

julia

#46 lore raw
In reply to Hannes Reinecke
Quoted 6 lines · CEST Hannes Reinecke
Quoted 5 lines · Julia Lawall
Quoted 4 lines · Hannes Reinecke

Actually that's one thing on my long-term to-do list: stricter range checking for kernel functions.
enums are an obvious way to address that,

I think smatch has some useful checks in that area, and using a gcc plugin to have a similar implementation could help do the checks by default (if there are not too many false positives).

Quoted 13 lines
Quoted 6 lines

In C, enums are ints. Is there a Gcc option that checks them? For example, the following program compiles fine with -Wall:

enum one {ONE=1, TWO=2, THREE=3};
enum two {ONEX=7, TWOX=8, THREEX=9};

Quoted 7 lines

int f (int x) {
enum one o = ONE;
enum two t = THREEX;
if (x) o = t; else t = o;
return 0;
}

Quoted 2 lines

From what I know GCC only checks enums if they are used in a switch() statement. Other types are not checked.

I think a common bug is having a function that takes an enum argument type but that is called with integer argument, or a different enum. This might even be something that could be added to a gcc version by default.

Arnd

#47 lore raw
In reply to Hannes Reinecke
Quoted 6 lines · Hannes Reinecke
Quoted 2 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

And perpetually out of date. Because functions call through to *other* functions which might return an errno outside the 'known' set.

Any why would you *want* to know the precise set of errnos that a function might return, if not to deliberately code your error handling non-defensively?

I can understand wanting to distinguish between errors and non-errors and ensure that the ranges cannot overlap. But IS_ERR_VALUE() typically reserves the whole range to -4095 (-MAX_ERRNO) for that. And I don't think we'd ever want to do anything different.

In particular I don't want anyone ever saying "oh, -123 is a valid non- error return but no other negative numbers are. But that's OK because it'll never *actually* return an error of -ENOMEDIUM so there's no ambiguity."

Of course, that's a silly example... but it's precisely where it sounds like you're going, from the above citation :)

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#48 lore raw
Quoted 11 lines · David Woodhouse
Quoted 6 lines · Hannes Reinecke
Quoted 2 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

And perpetually out of date. Because functions call through to *other* functions which might return an errno outside the 'known' set.

What I want to catch with that are value range collisions; has the 'int' returned from that function the same meaning as the 'int' returned from the next function. Random example: drivers/net/veth.c:veth_newlink()
'rtnl_nla_parse_ifla()' returns a value which is stored in the same variable as the return value from veth_validate(). And that value is then used as the return value for the entire function. ATM we need to do code inspection to figure out if both indeed return an errno or not.

Quoted 4 lines

Any why would you *want* to know the precise set of errnos that a function might return, if not to deliberately code your error handling non-defensively?

The ultimate goal is to provide a map with the known return codes for the various functions. Then we can invert that map and _inject_ errors via systemtap and friend for those functions to test the error paths.
Using a fuzzer would work, too, but I think it's a bit too generic here
(scanning the entire range of 'int' _does_ take some time).
In general we want to trigger the 'exciting' cases (ie values where there _is_ an error path coded) to figure out if the error handling actually does behave as advertised.

Quoted 5 lines

I can understand wanting to distinguish between errors and non-errors and ensure that the ranges cannot overlap. But IS_ERR_VALUE() typically reserves the whole range to -4095 (-MAX_ERRNO) for that. And I don't think we'd ever want to do anything different.

Even that would be fine; even restricting the range from the entire 'int' to 4096 will make live easier. But ATM we don't even have a way of expressing that.

Quoted 5 lines

In particular I don't want anyone ever saying "oh, -123 is a valid non- error return but no other negative numbers are. But that's OK because it'll never *actually* return an error of -ENOMEDIUM so there's no ambiguity."

Na, of course not.

Cheers,

Hannes

Signature · 5 lines
Dr. Hannes Reinecke		               zSeries & Storage
[email protected]			               +49 911 74053 688
SUSE LINUX GmbH, Maxfeldstr. 5, 90409 Nürnberg
GF: F. Imendörffer, J. Smithard, D. Upmanyu, G. Norton
HRB 21284 (AG Nürnberg)
#49 lore raw
Quoted 40 lines · Hannes Reinecke
Quoted 8 lines · David Woodhouse
Quoted 6 lines · Hannes Reinecke
Quoted 4 lines

I guess that almost all functions return only a few possible error codes?

Quoted 8 lines
Quoted 3 lines

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

And perpetually out of date. Because functions call through to *other* functions which might return an errno outside the 'known' set.

What I want to catch with that are value range collisions; has the 'int' returned from that function the same meaning as the 'int' returned from the next function.
Random example: drivers/net/veth.c:veth_newlink() 'rtnl_nla_parse_ifla()' returns a value which is stored in the same variable as the return value from veth_validate(). And that value is then used as the return value for the entire function.
ATM we need to do code inspection to figure out if both indeed return an errno or not.

Quoted 4 lines

Any why would you *want* to know the precise set of errnos that a function might return, if not to deliberately code your error handling non-defensively?

The ultimate goal is to provide a map with the known return codes for the various functions. Then we can invert that map and _inject_ errors via systemtap and friend for those functions to test the error paths. Using a fuzzer would work, too, but I think it's a bit too generic here (scanning the entire range of 'int' _does_ take some time).
In general we want to trigger the 'exciting' cases (ie values where there _is_ an error path coded) to figure out if the error handling actually does behave as advertised.

Quoted 8 lines
Quoted 5 lines

I can understand wanting to distinguish between errors and non-errors and ensure that the ranges cannot overlap. But IS_ERR_VALUE() typically reserves the whole range to -4095 (-MAX_ERRNO) for that. And I don't think we'd ever want to do anything different.

Even that would be fine; even restricting the range from the entire 'int' to 4096 will make live easier.
But ATM we don't even have a way of expressing that.

This requires rewriting kernel in a programming language with real type system. :^)

#50 lore raw
In reply to David Woodhouse
Quoted 8 lines · David Woodhouse
Quoted 6 lines · Hannes Reinecke
Quoted 2 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

Quoted 6 lines

And perpetually out of date. Because functions call through to *other* functions which might return an errno outside the 'known' set.

Any why would you *want* to know the precise set of errnos that a function might return, if not to deliberately code your error handling non-defensively?

Java has checked exceptions.
Obviously, people "catch (e) {}" them.

#51 lore raw
In reply to David Woodhouse
Quoted 10 lines · David Woodhouse
Quoted 6 lines · Hannes Reinecke
Quoted 2 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

And perpetually out of date. Because functions call through to *other* functions which might return an errno outside the 'known' set.

If you have a script to calculate it, it doesn't have to be perpetually out of date. The problem is just the time to collect the information for the whole kernel. It could be a good intern project.

julia

Quoted 21 lines

Any why would you *want* to know the precise set of errnos that a function might return, if not to deliberately code your error handling non-defensively?

I can understand wanting to distinguish between errors and non-errors and ensure that the ranges cannot overlap. But IS_ERR_VALUE() typically reserves the whole range to -4095 (-MAX_ERRNO) for that. And I don't think we'd ever want to do anything different.

In particular I don't want anyone ever saying "oh, -123 is a valid non- error return but no other negative numbers are. But that's OK because it'll never *actually* return an error of -ENOMEDIUM so there's no ambiguity."

Of course, that's a silly example... but it's precisely where it sounds like you're going, from the above citation :)

--
David Woodhouse Open Source Technology Centre
[email protected] Intel Corporation

#52 lore raw
Quoted 12 lines · Julia Lawall
Quoted 9 lines · David Woodhouse
Quoted 7 lines · Hannes Reinecke
Quoted 3 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

Quoted 7 lines
Quoted 3 lines

And perpetually out of date. Because functions call through to *other* functions which might return an errno outside the 'known' set.

If you have a script to calculate it, it doesn't have to be perpetually out of date. The problem is just the time to collect the information for the whole kernel. It could be a good intern project.

It's a lot of pain, for what gain? What, practically would we get as a benefit if we did this? Every time I see proposals about scripting checks in the kernel, I'm reminded of our section mismatch debacle.
Life is so much easier without every kernel release generating 100s of
patches trying to correct section mismatches which didn't matter in the first place ...

James

#53 lore raw
Quoted 21 lines · James Bottomley
Quoted 12 lines · Julia Lawall
Quoted 9 lines · David Woodhouse
Quoted 7 lines · Hannes Reinecke
Quoted 3 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

Quoted 7 lines
Quoted 3 lines

And perpetually out of date. Because functions call through to *other* functions which might return an errno outside the 'known' set.

If you have a script to calculate it, it doesn't have to be perpetually out of date.  The problem is just the time to collect the information for the whole kernel.  It could be a good intern project.

Quoted 6 lines

It's a lot of pain, for what gain?  What, practically would we get as a benefit if we did this?  Every time I see proposals about scripting checks in the kernel, I'm reminded of our section mismatch debacle. Life is so much easier without every kernel release generating 100s of patches trying to correct section mismatches which didn't matter in the first place ...

To find functions where the -errno returns might be ambiguous with real valid return values, might be beneficial. If it doesn't have too many false positives.

To tell *which* specific errnos might be returned by a given function... I can't see any benefit in that ever.

Signature · 1 line
dwmw2
#54 lore raw
Quoted 41 lines

-----Original Message-----
From: [email protected] [mailto:ksummit- [email protected]] On Behalf Of David Woodhouse
Sent: Wednesday, July 27, 2016 5:34 AM
To: James Bottomley <[email protected]>; Julia Lawall <[email protected]>
Cc: [email protected]
Subject: Re: [Ksummit-discuss] [CORE TOPIC] More useful types in the linux kernel

Quoted 21 lines · James Bottomley
Quoted 12 lines · Julia Lawall
Quoted 9 lines · David Woodhouse
Quoted 7 lines · Hannes Reinecke
Quoted 3 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

Quoted 7 lines
Quoted 3 lines

And perpetually out of date. Because functions call through to *other* functions which might return an errno outside the 'known' set.

If you have a script to calculate it, it doesn't have to be perpetually out of date.  The problem is just the time to collect the information for the whole kernel.  It could be a good intern project.

Quoted 6 lines

It's a lot of pain, for what gain?  What, practically would we get as a benefit if we did this?  Every time I see proposals about scripting checks in the kernel, I'm reminded of our section mismatch debacle. Life is so much easier without every kernel release generating 100s of patches trying to correct section mismatches which didn't matter in the first place ...

Quoted 3 lines

To find functions where the -errno returns might be ambiguous with real valid return values, might be beneficial. If it doesn't have too many false positives.

It might be useful to have a list of possible errno generation points, for a particular routine, to make it easier to find the origin of a problem. Sometimes when you're unfamiliar with some bit of code, manually walking back through the call chain in the source is a hassle. I'm reminded of that trick where someone (I don't recall who) embedded the line number in the errno.

#55 lore raw
Quoted 45 lines · Timothy
Quoted 41 lines

-----Original Message-----
From: [email protected] [mailto:ksummit- [email protected]] On Behalf Of David Woodhouse
Sent: Wednesday, July 27, 2016 5:34 AM
To: James Bottomley <[email protected]>; Julia Lawall <[email protected]>
Cc: [email protected]
Subject: Re: [Ksummit-discuss] [CORE TOPIC] More useful types in the linux kernel

Quoted 21 lines · James Bottomley
Quoted 12 lines · Julia Lawall
Quoted 9 lines · David Woodhouse
Quoted 7 lines · Hannes Reinecke
Quoted 3 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

Quoted 7 lines
Quoted 3 lines

And perpetually out of date. Because functions call through to *other* functions which might return an errno outside the 'known' set.

If you have a script to calculate it, it doesn't have to be perpetually out of date. The problem is just the time to collect the information for the whole kernel. It could be a good intern project.

Quoted 6 lines

It's a lot of pain, for what gain? What, practically would we get as a benefit if we did this? Every time I see proposals about scripting checks in the kernel, I'm reminded of our section mismatch debacle.
Life is so much easier without every kernel release generating 100s of
patches trying to correct section mismatches which didn't matter in the first place ...

Quoted 3 lines

To find functions where the -errno returns might be ambiguous with real valid return values, might be beneficial. If it doesn't have too many false positives.

Quoted 5 lines

It might be useful to have a list of possible errno generation points, for a particular routine, to make it easier to find the origin of a problem. Sometimes when you're unfamiliar with some bit of code, manually walking back through the call chain in the source is a hassle. I'm reminded of that trick where someone (I don't recall who) embedded the line number in the errno.

I'd be interested if you could find what you are referring to. I would love to see something better here than my printk debugger. A list of possible errnos would help some, but I'd expect we typically have multiple called functions returning the same possible errnos, so you'd still be narrowing down the source manually. Debugging that probe of a driver failed because of a config option not being enabled is painful. Maybe I'm the only one that switches between boards/kernels a lot and does a poor job maintaining working configs.

Wouldn't something as simple as turning all the errno's into backtrace printks or tracepoints be doable. It might have to be enabled on probe() entry to avoid any "normal" errors. Any code that initializes the return value with an error would be a false positive. A major benefit to something like this is it could get rid of lots of error path printks in drivers that bloat the kernel.

Rob

#56 lore raw
Quoted 8 lines · Rob Herring
Quoted 6 lines · Timothy

It might be useful to have a list of possible errno generation points, for a particular routine, to make it easier to find the origin of a problem. Sometimes when you're unfamiliar with some bit of code, manually walking back through the call chain in the source is a hassle. I'm reminded of that trick where someone (I don't recall who) embedded the line number in the errno.

I'd be interested if you could find what you are referring to.

Hugh Dickins:

#undef EINVAL
#define EINVAL __LINE__

https://lwn.net/Articles/614446/

Although it may not work well for error pointers as Linus stated in the follow up mail.

Signature · 2 lines
Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum,
a Linux Foundation Collaborative Project
#57 lore raw
In reply to Hannes Reinecke
Quoted 6 lines · Hannes Reinecke
Quoted 2 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

I think that's a bad idea. We should be propagating errors from the functions we call. It should be able to change without breaking.

Smatch does a pretty good job of tracking return values, especially if you rebuild the database over and over once a day like I do.

In some places I hack the database manually. For legacy reasons there are a couple places that happens but the main way is through this file:
The format is function, space, old value, space, new value:

http://repo.or.cz/smatch.git/blob/HEAD:/smatch_data/db/kernel.return_fixes

One thing that causes problems for Smatch is recursion. We don't know what the function returns the first time it's called so we record that it could return anything. Then the second time we "know" that it can return anything. So the unknown propagates recursively. Another thing that causes problems is when we copy a return value from another thread or a work queue. There are a bunch of places like that where the programmer knows the return value is negative but it's hard for static analysis.

I need these manual fixes when not knowing the error code causes problems because a function does this:

	if (ret)
		return ret;
But the caller does:
	if (ret < 0)
		return ret;

There is a mismatch because Smatch thinks any non-zero is an error but the caller knows only negatives are errors.

The other reason for the file is that we want to record that the scnprintf() return value is less than the size parameter. Ideally, we could record that strnlen_user() returns "<= count + 1", but Smatch is not flexible enough to do that yet. These upper bounds are needed to prevent integer overflow and buffer overflow warnings.

One thing I get annoyed about is when functions return positive values but it's not documented what it means. For example, ocfs2_plock() returns negatives, zero or FILE_LOCK_DEFERRED. Possibly this is a bug. How should I know? Also em_sti_clock_event_next() should return zero or -ETIME but it returns zero or one so I think it's buggy.

One last thing, is that it's sometimes impossible to tell when we return zero unintentionally vs intentionally. I'm talking about code like this:

	if (frob_whatever())
		goto out;

It's a missing error code bug 75% of the time and intentional the other 25% of the time. I feel like this should _always_ have a comment next to them, just like we _always_ comment /* fall through */ in switch statements to note the missing break.

regards,
dan carpenter

#58 lore raw
Quoted 10 lines · Dan Carpenter
Quoted 6 lines · Hannes Reinecke
Quoted 2 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

I think that's a bad idea. We should be propagating errors from the functions we call. It should be able to change without breaking.

Should we? I recently faced a bug caused by a (proposed) change to btrfs which returned a different error code to the ->fh_to_dentry() function. That was being propagated up to nfsd, and out on the wire in the NFSv4 protocol.
Only the new error was invalid for the protocol and the client (correctly) reported it to user-space rather than handling it internally.

This happened because not enough thought/documentation had been given to which error codes were sensibly meaningful. I changed exportfs_decode_fh() so that any error other than ENOMEM became ESTALE, because that is all nfsd can sensibly handle.

I'm sure there are some (many!) paths where error codes should be propagated transparently, but I don't think we should assume that is always true.

Quoted 3 lines

Smatch does a pretty good job of tracking return values, especially if you rebuild the database over and over once a day like I do.

So it is OK to keep a list of valid return values in a database, but not OK to keep them in the code as documentation, and to alert the programmer when they make a change so they can declare (and maybe even document) if it was an intentional change?
Maybe I'm just misunderstanding your point of view.

Thanks,
NeilBrown

#59 lore raw
Quoted 21 lines · NeilBrown
Quoted 10 lines · Dan Carpenter
Quoted 6 lines · Hannes Reinecke
Quoted 2 lines

I guess that almost all functions return only a few possible error codes?

Precisely. If we had a way of specifying "the return value is an errno with the possible values '0', '-EIO', and '-EINVAL'" that would be _so_ cool.

I think that's a bad idea. We should be propagating errors from the functions we call. It should be able to change without breaking.

Should we? I recently faced a bug caused by a (proposed) change to btrfs which returned a different error code to the ->fh_to_dentry() function. That was being propagated up to nfsd, and out on the wire in the NFSv4 protocol.
Only the new error was invalid for the protocol and the client (correctly) reported it to user-space rather than handling it internally.

Quoted 9 lines

This happened because not enough thought/documentation had been given to which error codes were sensibly meaningful. I changed exportfs_decode_fh() so that any error other than ENOMEM became ESTALE, because that is all nfsd can sensibly handle.

I'm sure there are some (many!) paths where error codes should be propagated transparently, but I don't think we should assume that is always true.

My guess is that 97% of the time it should be propagated.

Quoted 8 lines
Quoted 3 lines

Smatch does a pretty good job of tracking return values, especially if you rebuild the database over and over once a day like I do.

So it is OK to keep a list of valid return values in a database, but not OK to keep them in the code as documentation, and to alert the programmer when they make a change so they can declare (and maybe even document) if it was an intentional change?

What I'm saying is that no one wants a thing where if we change a return code, we have to update the documentation for all the call trees.

regards,
dan carpenter

#60 lore raw
In reply to Dan Carpenter
Quoted 9 lines · Dan Carpenter

I need these manual fixes when not knowing the error code causes problems because a function does this:

        if (ret)
                return ret;
But the caller does:
        if (ret < 0)
                return ret;

There is a mismatch because Smatch thinks any non-zero is an error but the caller knows only negatives are errors.

Can we introduce types for this? We have a number of different return type conventions in the kernel:

bool
errno_t (-4095 to 0 are valid)
count_t (-4095 to INT_MAX)
long_count_t (-4095 to LONG_MAX)
ulong_count_t (-4095 to -4096)
struct foo _err*

I think this is good programmer documentation in addition to being potentially useful to smatch.

#61 lore raw
Quoted 21 lines · Matthew Wilcox
Quoted 9 lines · Dan Carpenter

I need these manual fixes when not knowing the error code causes problems because a function does this:

        if (ret)
                return ret;
But the caller does:
        if (ret < 0)
                return ret;

There is a mismatch because Smatch thinks any non-zero is an error but the caller knows only negatives are errors.

Can we introduce types for this? We have a number of different return type conventions in the kernel:

bool
errno_t (-4095 to 0 are valid)
count_t (-4095 to INT_MAX)
long_count_t (-4095 to LONG_MAX)
ulong_count_t (-4095 to -4096)
struct foo _err*

Quoted 2 lines

I think this is good programmer documentation in addition to being potentially useful to smatch.

I'd love to see an explicit type distinct from "int" for "potentially an errno". And if any code uses "potentially an errno *or* a non-errno non-zero return value", that should ideally use a distinct type as well.

#62 lore raw
Quoted 2 lines · Josh Triplett
Quoted 1 line · Matthew Wilcox

Can we introduce types for this? We have a number of different return

type

Quoted 9 lines
Quoted 9 lines

conventions in the kernel:

bool
errno_t (-4095 to 0 are valid)
count_t (-4095 to INT_MAX)
long_count_t (-4095 to LONG_MAX)
ulong_count_t (-4095 to -4096)
struct foo _err*

Quoted 6 lines
Quoted 2 lines

I think this is good programmer documentation in addition to being potentially useful to smatch.

I'd love to see an explicit type distinct from "int" for "potentially an errno". And if any code uses "potentially an errno *or* a non-errno non-zero return value", that should ideally use a distinct type as well.

I think the biggest problem is coming up with good names for the types. And the churn of introducing them, particularly converting function pointers and all occurrences.

#63 lore raw
Quoted 16 lines · Matthew Wilcox
Quoted 2 lines · Josh Triplett
Quoted 1 line · Matthew Wilcox

Can we introduce types for this? We have a number of different return

type

Quoted 12 lines
Quoted 11 lines

conventions in the kernel:

bool
errno_t (-4095 to 0 are valid)
count_t (-4095 to INT_MAX)
long_count_t (-4095 to LONG_MAX)
ulong_count_t (-4095 to -4096)
struct foo _err*

I think this is good programmer documentation in addition to being potentially useful to smatch.

Quoted 7 lines
Quoted 3 lines

I'd love to see an explicit type distinct from "int" for "potentially an errno". And if any code uses "potentially an errno *or* a non-errno non-zero return value", that should ideally use a distinct type as well.

I think the biggest problem is coming up with good names for the types. And the churn of introducing them, particularly converting function pointers and all occurrences.

Names are easy part (errno_t is perfect actually). The problem is that once error is cleared, variable doesn't change to regular type anymore:

	errno_t rv;

	rv = f();
	if (rv < 0)
		return rv;
	int rv = rv;

which agains boils down to a language with real type system.

#64 lore raw
Quoted 28 lines · Alexey Dobriyan
Quoted 16 lines · Matthew Wilcox
Quoted 2 lines · Josh Triplett
Quoted 1 line · Matthew Wilcox

Can we introduce types for this? We have a number of different return

type

Quoted 12 lines
Quoted 11 lines

conventions in the kernel:

bool
errno_t (-4095 to 0 are valid)
count_t (-4095 to INT_MAX)
long_count_t (-4095 to LONG_MAX)
ulong_count_t (-4095 to -4096)
struct foo _err*

I think this is good programmer documentation in addition to being potentially useful to smatch.

Quoted 7 lines
Quoted 3 lines

I'd love to see an explicit type distinct from "int" for "potentially an errno". And if any code uses "potentially an errno *or* a non-errno non-zero return value", that should ideally use a distinct type as well.

I think the biggest problem is coming up with good names for the types. And the churn of introducing them, particularly converting function pointers and all occurrences.

Names are easy part (errno_t is perfect actually). The problem is that once error is cleared, variable doesn't change to regular type anymore:

Quoted 8 lines
	errno_t rv;

	rv = f();
	if (rv < 0)
		return rv;
	int rv = rv;

which agains boils down to a language with real type system.

We could maybe do

errno_t rv;

rv = f();
if (IS_ERR(rv))
return rv;
int r = CHECKED(rv);

Tools could maybe verify that all paths to CHECKED are actually going through an IS_ERR test as well.

Quoted 4 lines

_______________________________________________
Ksummit-discuss mailing list
[email protected]
https://lists.linuxfoundation.org/mailman/listinfo/ksummit-discuss

#65 lore raw
Quoted 48 lines · Michael S. Tsirkin
Quoted 28 lines · Alexey Dobriyan
Quoted 16 lines · Matthew Wilcox
Quoted 2 lines · Josh Triplett
Quoted 1 line · Matthew Wilcox

Can we introduce types for this? We have a number of different return

type

Quoted 12 lines
Quoted 11 lines

conventions in the kernel:

bool
errno_t (-4095 to 0 are valid)
count_t (-4095 to INT_MAX)
long_count_t (-4095 to LONG_MAX)
ulong_count_t (-4095 to -4096)
struct foo _err*

I think this is good programmer documentation in addition to being potentially useful to smatch.

Quoted 7 lines
Quoted 3 lines

I'd love to see an explicit type distinct from "int" for "potentially an errno". And if any code uses "potentially an errno *or* a non-errno non-zero return value", that should ideally use a distinct type as well.

I think the biggest problem is coming up with good names for the types. And the churn of introducing them, particularly converting function pointers and all occurrences.

Names are easy part (errno_t is perfect actually). The problem is that once error is cleared, variable doesn't change to regular type anymore:

Quoted 8 lines
	errno_t rv;

	rv = f();
	if (rv < 0)
		return rv;
	int rv = rv;

which agains boils down to a language with real type system.

We could maybe do

(quoted diff) +0 -0
 	errno_t rv;
 
 	rv = f();
 	if (IS_ERR(rv))
 		return rv;
 	int r = CHECKED(rv);
Quoted 2 lines

Tools could maybe verify that all paths to CHECKED are actually going through an IS_ERR test as well.

What does the return value of f look like?
Was it intentional to use IS_ERR, which checks for a pointer? At least in that case gcc would ensure that CHECKED is used.

julia

#66 lore raw
Quoted 52 lines · Julia Lawall
Quoted 48 lines · Michael S. Tsirkin
Quoted 28 lines · Alexey Dobriyan
Quoted 16 lines · Matthew Wilcox
Quoted 2 lines · Josh Triplett
Quoted 1 line · Matthew Wilcox

Can we introduce types for this? We have a number of different return

type

Quoted 12 lines
Quoted 11 lines

conventions in the kernel:

bool
errno_t (-4095 to 0 are valid)
count_t (-4095 to INT_MAX)
long_count_t (-4095 to LONG_MAX)
ulong_count_t (-4095 to -4096)
struct foo _err*

I think this is good programmer documentation in addition to being potentially useful to smatch.

Quoted 7 lines
Quoted 3 lines

I'd love to see an explicit type distinct from "int" for "potentially an errno". And if any code uses "potentially an errno *or* a non-errno non-zero return value", that should ideally use a distinct type as well.

I think the biggest problem is coming up with good names for the types. And the churn of introducing them, particularly converting function pointers and all occurrences.

Names are easy part (errno_t is perfect actually). The problem is that once error is cleared, variable doesn't change to regular type anymore:

Quoted 8 lines
	errno_t rv;

	rv = f();
	if (rv < 0)
		return rv;
	int rv = rv;

which agains boils down to a language with real type system.

We could maybe do

(quoted diff) +0 -0
 	errno_t rv;
 
 	rv = f();
 	if (IS_ERR(rv))
 		return rv;
 	int r = CHECKED(rv);
Quoted 4 lines
Quoted 2 lines

Tools could maybe verify that all paths to CHECKED are actually going through an IS_ERR test as well.

What does the return value of f look like?

errno_t f(void);

Quoted 1 line

Was it intentional to use IS_ERR, which checks for a pointer?

It seems like a nice name but yes, it's already used to convert pointers to integers.

Either we find a macro trick to teach IS_ERR to handle integers as well, or we use a different macro. TEST_ERR?

Quoted 4 lines

At least in
that case gcc would ensure that CHECKED is used.

julia

I guess
typedef void * errno_t;
is one way, maybe marking it noderef for good measure.

This means the values are 64 bit unfortunately, which is often overkill.

Signature · 1 line
MST
#67 lore raw
Quoted 7 lines · Michael S. Tsirkin

I guess
typedef void * errno_t;
is one way, maybe marking it noderef for good measure.

This means the values are 64 bit unfortunately, which is often overkill.

s/64-bit/native word size/

Does anyone know of an ABI where that's *inefficient*?! Or at least, less efficient than returning a 32-bit int? Pretty much every ABI I ever saw says something along the lines of 'register FOO contains the return value if the type is small enough, otherwise for returning aggregates, register FOO contains ...'

I can't believe anyone would deliberately design an ABI where calling void *foo(void) is less efficient than calling int bar(void);

(and if you have some weird DSP or the CDC6600 in mind where Linux won't run anyway, then comp.arch.necrophilia is --> that way)

#68 lore raw
In reply to Michael S. Tsirkin
Quoted 61 lines · Michael S. Tsirkin
Quoted 52 lines · Julia Lawall
Quoted 48 lines · Michael S. Tsirkin
Quoted 28 lines · Alexey Dobriyan
Quoted 16 lines · Matthew Wilcox
Quoted 2 lines · Josh Triplett
Quoted 1 line · Matthew Wilcox

Can we introduce types for this? We have a number of different return

type

Quoted 12 lines
Quoted 11 lines

conventions in the kernel:

bool
errno_t (-4095 to 0 are valid)
count_t (-4095 to INT_MAX)
long_count_t (-4095 to LONG_MAX)
ulong_count_t (-4095 to -4096)
struct foo _err*

I think this is good programmer documentation in addition to being potentially useful to smatch.

Quoted 7 lines
Quoted 3 lines

I'd love to see an explicit type distinct from "int" for "potentially an errno". And if any code uses "potentially an errno *or* a non-errno non-zero return value", that should ideally use a distinct type as well.

I think the biggest problem is coming up with good names for the types. And the churn of introducing them, particularly converting function pointers and all occurrences.

Names are easy part (errno_t is perfect actually). The problem is that once error is cleared, variable doesn't change to regular type anymore:

Quoted 8 lines
	errno_t rv;

	rv = f();
	if (rv < 0)
		return rv;
	int rv = rv;

which agains boils down to a language with real type system.

We could maybe do

(quoted diff) +0 -0
 	errno_t rv;
 
 	rv = f();
 	if (IS_ERR(rv))
 		return rv;
 	int r = CHECKED(rv);
Quoted 4 lines
Quoted 2 lines

Tools could maybe verify that all paths to CHECKED are actually going through an IS_ERR test as well.

What does the return value of f look like?

errno_t f(void);

Quoted 8 lines
Quoted 1 line

Was it intentional to use IS_ERR, which checks for a pointer?

It seems like a nice name but yes, it's already used to convert pointers to integers.

Either we find a macro trick to teach IS_ERR to handle integers as well, or we use a different macro. TEST_ERR?

IS_ERRNO?

julia

Quoted 17 lines
Quoted 5 lines

At least in
that case gcc would ensure that CHECKED is used.

julia

I guess
typedef void * errno_t;
is one way, maybe marking it noderef for good measure.

This means the values are 64 bit unfortunately, which is often overkill.

--
MST

#69 lore raw
In reply to Alexey Dobriyan
Quoted 13 lines · Alexey Dobriyan
Quoted 12 lines · Matthew Wilcox
Quoted 2 lines · Matthew Wilcox
Quoted 1 line

Can we introduce types for this? We have a number of different return

type

Quoted 8 lines
Quoted 8 lines

conventions in the kernel:

bool
errno_t (-4095 to 0 are valid) count_t (-4095 to INT_MAX) long_count_t (-4095 to LONG_MAX) ulong_count_t (-4095 to -4096) struct foo _err*

Quoted 6 lines
Quoted 1 line

I think the biggest problem is coming up with good names for the types.

And

Quoted 2 lines

the churn of introducing them, particularly converting function pointers and all occurrences.

Names are easy part (errno_t is perfect actually).

I agree that errno_t is perfect, but it's not really part of a nice family -- count_t, long_count_t and ulong_count_t are all pretty crappy. They don't scream out I MIGHT CONTAIN AN ERRNO, BETTER CHECK ME! the way I would like. count_might_be_errno_t is not exactly euphonious. err_count_t, perhaps?

Of course, it might not be a count ... at least at one point the NVMe driver had variables which either contained [-4095 to -1] (Linux errno, usually -ENOMEM), 0 (success) or 1-65535 (NVMe status code indicating some device failure). I think those variables are all gone now, but that's not an unreasonable thing to want and calling that beast a ushort_count_t would be untrue. I suspect for that kind of thing, the driver should create its own type (or if we did something similar in SCSI, the subsystem would create a scsi_status_t that was pretty much private to the SCSI subsystem).

#70 lore raw
In reply to Julia Lawall
Quoted 7 lines · Julia Lawall

In C, enums are ints. Is there a Gcc option that checks them? For example, the following program compiles fine with -Wall:

enum one {ONE=1, TWO=2, THREE=3};
enum two {ONEX=7, TWOX=8, THREEX=9};

Quoted 6 lines

int f (int x) {
enum one o = ONE;
enum two t = THREEX;
if (x) o = t; else t = o;
return 0;
}

However with this slight change (and line-numbers added for clarity)

     1	
     2	enum one {ONE=1, TWO=2, THREE=3} __attribute((bitwise));
     3	enum two {ONEX=7, TWOX=8, THREEX=9} __attribute((bitwise));
     4	
     5	int f (int x) {
     6	  enum one o = ONE;
     7	  enum two t = THREEX;
     8	  if (x) o = t; else t = o;
     9	  return 0;
    10	}

sparse complains:

/tmp/test.c:8:14: warning: mixing different enum types
/tmp/test.c:8:14: int enum two versus
/tmp/test.c:8:14: int enum one
/tmp/test.c:8:26: warning: mixing different enum types
/tmp/test.c:8:26: int enum one versus
/tmp/test.c:8:26: int enum two

Is that what you were hoping for?

NeilBrown

#71 lore raw
Quoted 31 lines · NeilBrown
Quoted 7 lines · Julia Lawall

In C, enums are ints. Is there a Gcc option that checks them? For example, the following program compiles fine with -Wall:

enum one {ONE=1, TWO=2, THREE=3};
enum two {ONEX=7, TWOX=8, THREEX=9};

Quoted 6 lines

int f (int x) {
enum one o = ONE;
enum two t = THREEX;
if (x) o = t; else t = o;
return 0;
}

However with this slight change (and line-numbers added for clarity)

     1
     2	enum one {ONE=1, TWO=2, THREE=3} __attribute((bitwise));
     3	enum two {ONEX=7, TWOX=8, THREEX=9} __attribute((bitwise));
     4
     5	int f (int x) {
     6	  enum one o = ONE;
     7	  enum two t = THREEX;
     8	  if (x) o = t; else t = o;
     9	  return 0;
    10	}

sparse complains:

Quoted 8 lines

/tmp/test.c:8:14: warning: mixing different enum types
/tmp/test.c:8:14: int enum two versus
/tmp/test.c:8:14: int enum one
/tmp/test.c:8:26: warning: mixing different enum types
/tmp/test.c:8:26: int enum one versus
/tmp/test.c:8:26: int enum two

Is that what you were hoping for?

Yes, perfect, thanks.

julia

#72 lore raw
In reply to David Howells
Quoted 6 lines · On Thu, 21 Jul 2016 16:05:25 +0100 David Howells <[email protected]> wrote:

(2) Differentiate non-BH spinlocks and BH spinlocks by type.

It seems like you can't mix BH and non-BH ops on a spinlock without lockdep barking. If that's the case, let's make this a compile-time check.

Not sure what you mean here. You can use BH spinlocks as non BH spinlocks while in a BH, or if BH is already disabled, and lockdep will not complain.

-- Steve

#73 lore raw
Quoted 10 lines · Steven Rostedt
Quoted 6 lines

(2) Differentiate non-BH spinlocks and BH spinlocks by type.

It seems like you can't mix BH and non-BH ops on a spinlock without lockdep barking. If that's the case, let's make this a compile-time check.

Not sure what you mean here. You can use BH spinlocks as non BH spinlocks while in a BH, or if BH is already disabled, and lockdep will not complain.

Okay, fair enough.

David

#74 lore raw
In reply to David Howells
Quoted 2 lines · David Howells

(3) Let's use bool a lot more for boolean values as the compiler might be able to make better choices with it.

This would be particularly useful for boolean one-bit struct bitfield flags (not least because assigning any positive even number to unsigned int foo:1 will result in 0) *but* we've found gcc produces worse code for bool:1 in our case. Details at [1].

BR,
Jani.

[1] http://patchwork.freedesktop.org/patch/msgid/[email protected]

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
#75 lore raw
Quoted 9 lines · Jani Nikula
Quoted 2 lines · David Howells

  (3) Let's use bool a lot more for boolean values as the compiler might be able to make better choices with it.

This would be particularly useful for boolean one-bit struct bitfield flags (not least because assigning any positive even number to unsigned int foo:1 will result in 0) *but* we've found gcc produces worse code for bool:1 in our case. Details at [1].

Quoted 5 lines

BR,
Jani.

[1] http://patchwork.freedesktop.org/patch/msgid/[email protected]

In that entire discussion I don't see any mention of a GCC PR being filed.

Why?

Signature · 2 lines
David Woodhouse                            Open Source Technology Centre
[email protected]                              Intel Corporation
#76 lore raw
Quoted 14 lines · David Woodhouse
Quoted 13 lines · Jani Nikula
Quoted 2 lines · David Howells

  (3) Let's use bool a lot more for boolean values as the compiler might be able to make better choices with it.

This would be particularly useful for boolean one-bit struct bitfield flags (not least because assigning any positive even number to unsigned int foo:1 will result in 0) *but* we've found gcc produces worse code for bool:1 in our case. Details at [1].

BR,
Jani.

Quoted 6 lines
Quoted 1 line

[1] http://patchwork.freedesktop.org/patch/msgid/[email protected]

In that entire discussion I don't see any mention of a GCC PR being filed.

Why?

I guess something along these lines: Everybody was sure that Somebody would do it. Anybody could have done it, but Nobody did it. Somebody got angry about that, because it was Everybody's job. Everybody thought Anybody could do it, but Nobody realized that Everybody wouldn't do it. It ended up that Everybody blamed Somebody when Nobody did what Anybody could have.

Dave (Gordon, Cc'd), you seemed to have the best grasp of what was going on, would you mind filing that GCC bug please? https://gcc.gnu.org/bugs/

BR,
Jani.

Signature · 1 line
Jani Nikula, Intel Open Source Technology Center
#77 lore raw
In reply to Eric W. Biederman
Quoted 6 lines · Eric W. Biederman

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Eric

Surprised that no one mentioned this yet - I think tagging integers/structs as coming from userspace could be useful, if we can teach e.g. smatch that access to a kernel pointer through this offset might fault.

Signature · 1 line
MST

A message that is not in the archive (sent off-list or not yet received): [email protected]

#79 lore raw
Quoted 23 lines · NeilBrown
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Eric W. Biederman

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Quoted 6 lines
Quoted 1 line

Eric

Surprised that no one mentioned this yet - I think tagging integers/structs as coming from userspace could be useful, if we can teach e.g. smatch that access to a kernel pointer through this offset might fault.

We already have that.
Sparse recognizes

    __attribute__((noderef, address_space(1)))

to mean "this is a pointer to a different address space which cannot be dereferened" and linux has

# define __user                __attribute__((noderef, address_space(1)))
Quoted 8 lines

so if you mark a pointer as "__user", then sparse will complain if you dereference it.

We've had this for over a decade :-)

https://lwn.net/Articles/87538/

NeilBrown

Of course, everyone uses these. But what I mean is tagging index types:

	int data[256];

	int foo(u32 __user *ptr)
	{
		u32 i;
		if (get_user(i, ptr))
			return -EFAULT;

		data[i] = 0;
			^^^ security vulnerability

	}

Above, i is coming from userspace and so must always be range-checked before it's used as an index.

Maybe we could change get_user return a tagged result: __from_user int. And have above warn because __from_user can not be assigned to plain int.

Then rework the code along the following lines:

	int data[256];

	int force_range(__unsafe u32 value, unsigned idx)
	{
		return ((__force int)value) % idx;
	}

	int foo(u32 __user *ptr)
	{
		__unsafe u32 i;
		int ichecked;
		if (get_user(i, ptr))
			return -EFAULT;

		ichecked = force_range(i, sizeof data);
		data[ichecked] = 0;
			^^^ ok now

	}
Signature · 1 line
MST
#80 lore raw
Quoted 49 lines · Michael S. Tsirkin
Quoted 23 lines · NeilBrown
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Eric W. Biederman

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Quoted 6 lines
Quoted 1 line

Eric

Surprised that no one mentioned this yet - I think tagging integers/structs as coming from userspace could be useful, if we can teach e.g. smatch that access to a kernel pointer through this offset might fault.

We already have that.
Sparse recognizes

    __attribute__((noderef, address_space(1)))

to mean "this is a pointer to a different address space which cannot be dereferened" and linux has

# define __user                __attribute__((noderef, address_space(1)))
Quoted 8 lines

so if you mark a pointer as "__user", then sparse will complain if you dereference it.

We've had this for over a decade :-)

https://lwn.net/Articles/87538/

NeilBrown

Of course, everyone uses these. But what I mean is tagging index types:

	int data[256];

	int foo(u32 __user *ptr)
	{
		u32 i;
		if (get_user(i, ptr))
			return -EFAULT;

		data[i] = 0;
			^^^ security vulnerability

	}
Quoted 2 lines

Above, i is coming from userspace and so must always be range-checked before it's used as an index.

Ahhh, I see. Thanks spelling it out for me.

Quoted 19 lines

Maybe we could change get_user return a tagged result: __from_user int. And have above warn because __from_user can not be assigned to plain int.

Then rework the code along the following lines:

	int data[256];

	int force_range(__unsafe u32 value, unsigned idx)
	{
		return ((__force int)value) % idx;
	}

	int foo(u32 __user *ptr)
	{
		__unsafe u32 i;
		int ichecked;
Quoted 8 lines
		if (get_user(i, ptr))
			return -EFAULT;

		ichecked = force_range(i, sizeof data);
		data[ichecked] = 0;
			^^^ ok now

	}

You could probably do this today using __attribute__((bitwise))

typedef int __attribute__((bitwise)) unsafe32;

Then use "unsafe32" wherever you have "__unsafe u32".

When you try

  data[i] = 0;

sparse says "warning: restricted int degrades to integer"

Of course, changing all the code would be a pain.
I guess you introduce "get_safe_user" then gradually transition code over.

NeilBrown

A message that is not in the archive (sent off-list or not yet received): [email protected]

#82 lore raw
Quoted 89 lines · NeilBrown
Quoted 49 lines · Michael S. Tsirkin
Quoted 23 lines · NeilBrown
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Eric W. Biederman

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Quoted 6 lines
Quoted 1 line

Eric

Surprised that no one mentioned this yet - I think tagging integers/structs as coming from userspace could be useful, if we can teach e.g. smatch that access to a kernel pointer through this offset might fault.

We already have that.
Sparse recognizes

    __attribute__((noderef, address_space(1)))

to mean "this is a pointer to a different address space which cannot be dereferened" and linux has

# define __user                __attribute__((noderef, address_space(1)))
Quoted 8 lines

so if you mark a pointer as "__user", then sparse will complain if you dereference it.

We've had this for over a decade :-)

https://lwn.net/Articles/87538/

NeilBrown

Of course, everyone uses these. But what I mean is tagging index types:

	int data[256];

	int foo(u32 __user *ptr)
	{
		u32 i;
		if (get_user(i, ptr))
			return -EFAULT;

		data[i] = 0;
			^^^ security vulnerability

	}
Quoted 2 lines

Above, i is coming from userspace and so must always be range-checked before it's used as an index.

Ahhh, I see. Thanks spelling it out for me.

Quoted 19 lines

Maybe we could change get_user return a tagged result: __from_user int. And have above warn because __from_user can not be assigned to plain int.

Then rework the code along the following lines:

	int data[256];

	int force_range(__unsafe u32 value, unsigned idx)
	{
		return ((__force int)value) % idx;
	}

	int foo(u32 __user *ptr)
	{
		__unsafe u32 i;
		int ichecked;
Quoted 8 lines
		if (get_user(i, ptr))
			return -EFAULT;

		ichecked = force_range(i, sizeof data);
		data[ichecked] = 0;
			^^^ ok now

	}

You could probably do this today using __attribute__((bitwise))

typedef int __attribute__((bitwise)) unsafe32;

Quoted 7 lines

Then use "unsafe32" wherever you have "__unsafe u32".

When you try

  data[i] = 0;

sparse says "warning: restricted int degrades to integer"

Yes, but inability to do math on bitwise integers is rather annoying.

E.g. if index is in bytes:

	int j = i / sizeof(*data);

will warn even though nothing bad is going on.

Maybe we could benefit from a separate integer type, such that unsafe integers can interact with safe (regular) integers, resulting in unsafe integers.

Quoted 6 lines

Of course, changing all the code would be a pain.
I guess you introduce "get_safe_user" then gradually transition code over.

NeilBrown

#83 lore raw
In reply to Michael S. Tsirkin
Quoted 6 lines · Michael S. Tsirkin
Quoted 5 lines · Eric W. Biederman

I would really like to get a feel among kernel maintainers and developers if this is something that is interesting, and what kind of constraints they think something like this would need to be usable for the kernel?

Quoted 6 lines
Quoted 1 line

Eric

Surprised that no one mentioned this yet - I think tagging integers/structs as coming from userspace could be useful, if we can teach e.g. smatch that access to a kernel pointer through this offset might fault.

We already have that.
Sparse recognizes

    __attribute__((noderef, address_space(1)))

to mean "this is a pointer to a different address space which cannot be dereferened" and linux has

# define __user                __attribute__((noderef, address_space(1)))

so if you mark a pointer as "__user", then sparse will complain if you dereference it.

We've had this for over a decade :-)

https://lwn.net/Articles/87538/

NeilBrown