Skip to contentSkip to conversation

scripts/update-linux-headers.sh: pull virtio hdrs

RFC PATCH Superseded Michael S. Tsirkin 3 participants 11 messages Feb 9, 2015 – Feb 11, 2015

v1 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 scripts/update-linux-headers.sh: pull virtio hdrs

It doesn't make sense to copy values manually:
the only issue with getting headers from linux seems to be dealing with linux/types, we can easily fix that automatically while importing.

---

FYI this is what I propose instead of the recently suggested
virtio: uniform virtio device IDs
we can then rework existing code to include these headers.

Will automatically bring in goodies as they arrive in linux.

This doesn't yet import virtio ccw header, that won't be hard to add later.

 scripts/update-linux-headers.sh | 10 ++++++++++
 1 file changed, 10 insertions(+)
scripts/update-linux-headers.sh +10 -0
index c8e026d..0bd8437 100755
@@ -76,4 +76,14 @@ else
     cp "$linux/COPYING" "$output/linux-headers"
 fi
 
+rm -rf "$output/standard-headers/linux"
+mkdir -p "$output/standard-headers/linux"
+for f in $tmpdir/include/linux/virtio*h; do
+    header=$(expr "$f" : '.*/\(.*\)');
+    sed -e 's/__u\([0-9][0-9]*\)/uint\1_t/g' \
+        -e 's/linux\/types/inttypes/' \
+        -e 's/__bitwise__//' \
+        "$tmpdir/include/linux/$header" > \
+        "$output/standard-headers/linux/$header";
+done
 rm -rf "$tmpdir"
Signature · 1 line
MST
#1 lore raw
Quoted 26 lines · Michael S. Tsirkin

It doesn't make sense to copy values manually:
the only issue with getting headers from linux seems to be dealing with linux/types, we can easily fix that automatically while importing.

---

FYI this is what I propose instead of the recently suggested
virtio: uniform virtio device IDs
we can then rework existing code to include these headers.

Will automatically bring in goodies as they arrive in linux.

This doesn't yet import virtio ccw header, that won't be hard to add later.

  scripts/update-linux-headers.sh | 10 ++++++++++
  1 file changed, 10 insertions(+)
scripts/update-linux-headers.sh +0 -0
index c8e026d..0bd8437 100755

scripts/update-linux-headers.sh else · line 80

Quoted 5 lines
scripts/update-linux-headers.sh +2 -0
@@ -76,4 +76,14 @@ else
      cp "$linux/COPYING" "$output/linux-headers"
  fi
 
+rm -rf "$output/standard-headers/linux"
+mkdir -p "$output/standard-headers/linux"

Shouldn't we add something in configure file to execute this automatically?

Or instead of creating 'standard-headers/, why can't we go that existing linux-headers/?

Thanks
Tiejun

Quoted 10 lines
scripts/update-linux-headers.sh +8 -0
+for f in $tmpdir/include/linux/virtio*h; do
+    header=$(expr "$f" : '.*/\(.*\)');
+    sed -e 's/__u\([0-9][0-9]*\)/uint\1_t/g' \
+        -e 's/linux\/types/inttypes/' \
+        -e 's/__bitwise__//' \
+        "$tmpdir/include/linux/$header" > \
+        "$output/standard-headers/linux/$header";
+done
  rm -rf "$tmpdir"
#2 lore raw
Quoted 32 lines · Tiejun
Quoted 31 lines · Michael S. Tsirkin

It doesn't make sense to copy values manually:
the only issue with getting headers from linux seems to be dealing with linux/types, we can easily fix that automatically while importing.

---

FYI this is what I propose instead of the recently suggested
virtio: uniform virtio device IDs
we can then rework existing code to include these headers.

Will automatically bring in goodies as they arrive in linux.

This doesn't yet import virtio ccw header, that won't be hard to add later.

  scripts/update-linux-headers.sh | 10 ++++++++++
  1 file changed, 10 insertions(+)
scripts/update-linux-headers.sh +0 -0
index c8e026d..0bd8437 100755
@@ -76,4 +76,14 @@ else
      cp "$linux/COPYING" "$output/linux-headers"
  fi
 
Quoted 5 lines

scripts/update-linux-headers.sh else · line 80

Quoted 2 lines
scripts/update-linux-headers.sh +2 -0
+rm -rf "$output/standard-headers/linux"
+mkdir -p "$output/standard-headers/linux"

Shouldn't we add something in configure file to execute this automatically?

No. We want to run this script only when we're updating the header files, which only happens when we have a valid kernel source tree available and you're a developer doing it as a specific action. configure is run by everybody and should definitely not be doing header updates.

Quoted 2 lines

Or instead of creating 'standard-headers/, why can't we go that existing linux-headers/?

The linux-headers/ directory contains header files which can only validly be included if the host we're compiling on is Linux. Some of them will cause compile failures on OSX or Windows if they are in the include path. The idea of this patch is that the standard-headers/ directory has "sanitized" header files which have had the linux-specific types and includes stripped out. So if we take the route this patch proposes we do need two directories.

-- PMM

#3 lore raw
Quoted 48 lines · Peter Maydell
Quoted 32 lines · Tiejun
Quoted 31 lines · Michael S. Tsirkin

It doesn't make sense to copy values manually:
the only issue with getting headers from linux seems to be dealing with linux/types, we can easily fix that automatically while importing.

---

FYI this is what I propose instead of the recently suggested
virtio: uniform virtio device IDs
we can then rework existing code to include these headers.

Will automatically bring in goodies as they arrive in linux.

This doesn't yet import virtio ccw header, that won't be hard to add later.

   scripts/update-linux-headers.sh | 10 ++++++++++
   1 file changed, 10 insertions(+)
scripts/update-linux-headers.sh +0 -0
index c8e026d..0bd8437 100755
@@ -76,4 +76,14 @@ else
       cp "$linux/COPYING" "$output/linux-headers"
   fi
 
Quoted 5 lines

scripts/update-linux-headers.sh else · line 80

Quoted 2 lines
scripts/update-linux-headers.sh +2 -0
+rm -rf "$output/standard-headers/linux"
+mkdir -p "$output/standard-headers/linux"

Shouldn't we add something in configure file to execute this automatically?

No. We want to run this script only when we're updating the header files, which only happens when we have a valid kernel source tree available and you're a developer doing it as a specific action. configure is run by everybody and should definitely not be doing header updates.

Quoted 2 lines

Or instead of creating 'standard-headers/, why can't we go that existing linux-headers/?

Quoted 9 lines

The linux-headers/ directory contains header files which can only validly be included if the host we're compiling on is Linux. Some of them will cause compile failures on OSX or Windows if they are in the include path. The idea of this patch is that the standard-headers/ directory has "sanitized" header files which have had the linux-specific types and includes stripped out. So if we take the route this patch proposes we do need two directories.

This confounds me since for instance, one of goals based on this patch is, it exposes those Virtio devices ID definition to hw/virtio, instead of my original patch, right? So without this sort of standard-hearders, how can we compile virtio? Or you mean we still keep those original stuff in include/hw/virtio*, but somehow update them once we execute that script manually.

Thanks
Tiejun

#4 lore raw
Quoted 11 lines · Tiejun
Quoted 9 lines · Peter Maydell

The linux-headers/ directory contains header files which can only validly be included if the host we're compiling on is Linux. Some of them will cause compile failures on OSX or Windows if they are in the include path. The idea of this patch is that the standard-headers/ directory has "sanitized" header files which have had the linux-specific types and includes stripped out. So if we take the route this patch proposes we do need two directories.

Quoted 6 lines

This confounds me since for instance, one of goals based on this patch is, it exposes those Virtio devices ID definition to hw/virtio, instead of my original patch, right? So without this sort of standard-hearders, how can we compile virtio? Or you mean we still keep those original stuff in include/hw/virtio*, but somehow update them once we execute that script manually.

I'm confused about why you're confused. We have two basic approaches we can take:

(1) What we do at the moment. There are headers defining the virtio interface in include/hw/virtio, and these are basically manually created and updated as necessary.

(2) What this patch is proposing. The headers defining virtio are automatically copied into standard-headers/ and fixed up to make them work with QEMU on all the hosts we support. This happens when this script is run by a developer to update QEMU's headers based on some new upstream kernel.

Personally I think that option 1 is more reliable and overall less effort, since automatiing the fixups is hard and virtio doesn't change very much.

-- PMM

#5 lore raw
Quoted 26 lines · Peter Maydell
Quoted 11 lines · Tiejun
Quoted 9 lines · Peter Maydell

The linux-headers/ directory contains header files which can only validly be included if the host we're compiling on is Linux. Some of them will cause compile failures on OSX or Windows if they are in the include path. The idea of this patch is that the standard-headers/ directory has "sanitized" header files which have had the linux-specific types and includes stripped out. So if we take the route this patch proposes we do need two directories.

Quoted 6 lines

This confounds me since for instance, one of goals based on this patch is, it exposes those Virtio devices ID definition to hw/virtio, instead of my original patch, right? So without this sort of standard-hearders, how can we compile virtio? Or you mean we still keep those original stuff in include/hw/virtio*, but somehow update them once we execute that script manually.

I'm confused about why you're confused. We have two basic approaches we can take:

(1) What we do at the moment. There are headers defining the virtio interface in include/hw/virtio, and these are basically manually created and updated as necessary.

Quoted 5 lines

(2) What this patch is proposing. The headers defining virtio are automatically copied into standard-headers/ and fixed up to make them work with QEMU on all the hosts we support. This happens when this script is run by a developer to update QEMU's headers based on some new upstream kernel.

I guess this mean this patch should be extended to smooth something in include/hw/virtio* in some ways.

Quoted 2 lines

Personally I think that option 1 is more reliable and overall

Agreed.

Quoted 3 lines

less effort, since automatiing the fixups is hard and virtio doesn't change very much.

So sounds my original patch is fine to you.

Thanks
Tiejun

#6 lore raw
In reply to Peter Maydell
Quoted 32 lines · Peter Maydell
Quoted 11 lines · Tiejun
Quoted 9 lines · Peter Maydell

The linux-headers/ directory contains header files which can only validly be included if the host we're compiling on is Linux. Some of them will cause compile failures on OSX or Windows if they are in the include path. The idea of this patch is that the standard-headers/ directory has "sanitized" header files which have had the linux-specific types and includes stripped out. So if we take the route this patch proposes we do need two directories.

Quoted 6 lines

This confounds me since for instance, one of goals based on this patch is, it exposes those Virtio devices ID definition to hw/virtio, instead of my original patch, right? So without this sort of standard-hearders, how can we compile virtio? Or you mean we still keep those original stuff in include/hw/virtio*, but somehow update them once we execute that script manually.

I'm confused about why you're confused. We have two basic approaches we can take:

(1) What we do at the moment. There are headers defining the virtio interface in include/hw/virtio, and these are basically manually created and updated as necessary.

(2) What this patch is proposing. The headers defining virtio are automatically copied into standard-headers/ and fixed up to make them work with QEMU on all the hosts we support. This happens when this script is run by a developer to update QEMU's headers based on some new upstream kernel.

Quoted 5 lines

Personally I think that option 1 is more reliable and overall less effort, since automatiing the fixups is hard and virtio doesn't change very much.

-- PMM

It picked up speed recently, so it's too much effort I think. My script seems to have automated fixups - didn't seem hard.

Signature · 1 line
MST
#7 lore raw
In reply to Michael S. Tsirkin

scripts/update-linux-headers.sh else · line 88

Quoted 10 lines · Michael S. Tsirkin
scripts/update-linux-headers.sh +10 -0
+rm -rf "$output/standard-headers/linux"
+mkdir -p "$output/standard-headers/linux"
+for f in $tmpdir/include/linux/virtio*h; do
+    header=$(expr "$f" : '.*/\(.*\)');
+    sed -e 's/__u\([0-9][0-9]*\)/uint\1_t/g' \
+        -e 's/linux\/types/inttypes/' \
+        -e 's/__bitwise__//' \
+        "$tmpdir/include/linux/$header" > \
+        "$output/standard-headers/linux/$header";
+done

This doesn't seem to be doing anything to fix up the '__attribute__((packed))' annotations. Presumably you're intending to put standard-headers/ on the include path? It would probably be better to not make the headers be in "linux/" in that case, since it would mean confusion/clashes for what "linux/virtio_net.h" etc mean -- are they the QEMU sanitized versions or the host OS's? (Having them in linux/ also makes code review harder since it breaks the current rule of thumb that is "no include of linux/anything in code that's not Linux-host-specific".)

You need to strip out all the #include <linux/something.h> from these headers, otherwise this won't build on non Linux hosts. Then you need to add in whatever the equivalent is to get the defines/types those includes were providing (for instance our virtio-net.h does a simple #define of ETH_ALEN). You probably want to make the update script fail if there's an include it's not expecting to deal with, otherwise you're likely to end up with a set of headers that seem OK on Linux but fail when tested on other OSes -- better to fail early and for the person trying to do the header update than to end up with a change that won't pass my build tests and gets bounced.

All that makes it seem to me like it's more trouble than it's worth compared to doing a one-time manual import.

-- PMM

#8 lore raw
Quoted 14 lines · Peter Maydell

scripts/update-linux-headers.sh else · line 88

Quoted 10 lines · Michael S. Tsirkin
scripts/update-linux-headers.sh +10 -0
+rm -rf "$output/standard-headers/linux"
+mkdir -p "$output/standard-headers/linux"
+for f in $tmpdir/include/linux/virtio*h; do
+    header=$(expr "$f" : '.*/\(.*\)');
+    sed -e 's/__u\([0-9][0-9]*\)/uint\1_t/g' \
+        -e 's/linux\/types/inttypes/' \
+        -e 's/__bitwise__//' \
+        "$tmpdir/include/linux/$header" > \
+        "$output/standard-headers/linux/$header";
+done

This doesn't seem to be doing anything to fix up the '__attribute__((packed))' annotations.

I don't know - what needs to be fixed up?

Quoted 9 lines

Presumably you're intending to put standard-headers/ on the include path? It would probably be better to not make the headers be in "linux/" in that case, since it would mean confusion/clashes for what "linux/virtio_net.h" etc mean -- are they the QEMU sanitized versions or the host OS's? (Having them in linux/ also makes code review harder since it breaks the current rule of thumb that is "no include of linux/anything in code that's not Linux-host-specific".)

Agreed, I'll change this.

Quoted 6 lines

You need to strip out all the #include <linux/something.h> from these headers, otherwise this won't build on non Linux hosts.
Then you need to add in whatever the
equivalent is to get the defines/types those includes were providing (for instance our virtio-net.h does a

Quoted 8 lines

simple #define of ETH_ALEN). You probably want to make the update script fail if there's an include it's not expecting to deal with, otherwise you're likely to end up with a set of headers that seem OK on Linux but fail when tested on other OSes -- better to fail early and for the person trying to do the header update than to end up with a change that won't pass my build tests and gets bounced.

OK, seems easy.
Will do.

Quoted 5 lines

All that makes it seem to me like it's more trouble than it's worth compared to doing a one-time manual import.

-- PMM

Only true if we are perfect and don't make mistakes. If we do make mistakes, I want to fix them in one place and propagate the fix automatically.
And since we are working on virtio 1.0 which is a huge change, introducing mistakes seems more likely.

Signature · 1 line
MST
#9 lore raw
Quoted 16 lines · Michael S. Tsirkin
Quoted 14 lines · Peter Maydell

scripts/update-linux-headers.sh else · line 88

Quoted 10 lines · Michael S. Tsirkin
scripts/update-linux-headers.sh +10 -0
+rm -rf "$output/standard-headers/linux"
+mkdir -p "$output/standard-headers/linux"
+for f in $tmpdir/include/linux/virtio*h; do
+    header=$(expr "$f" : '.*/\(.*\)');
+    sed -e 's/__u\([0-9][0-9]*\)/uint\1_t/g' \
+        -e 's/linux\/types/inttypes/' \
+        -e 's/__bitwise__//' \
+        "$tmpdir/include/linux/$header" > \
+        "$output/standard-headers/linux/$header";
+done

This doesn't seem to be doing anything to fix up
the '__attribute__((packed))' annotations.

Quoted 1 line

I don't know - what needs to be fixed up?

Needs to use the QEMU_PACKED macro.

-- PMM

#10 lore raw
Quoted 17 lines · Peter Maydell
Quoted 16 lines · Michael S. Tsirkin
Quoted 14 lines · Peter Maydell

scripts/update-linux-headers.sh else · line 88

Quoted 10 lines · Michael S. Tsirkin
scripts/update-linux-headers.sh +10 -0
+rm -rf "$output/standard-headers/linux"
+mkdir -p "$output/standard-headers/linux"
+for f in $tmpdir/include/linux/virtio*h; do
+    header=$(expr "$f" : '.*/\(.*\)');
+    sed -e 's/__u\([0-9][0-9]*\)/uint\1_t/g' \
+        -e 's/linux\/types/inttypes/' \
+        -e 's/__bitwise__//' \
+        "$tmpdir/include/linux/$header" > \
+        "$output/standard-headers/linux/$header";
+done

This doesn't seem to be doing anything to fix up
the '__attribute__((packed))' annotations.

Quoted 5 lines
Quoted 1 line

I don't know - what needs to be fixed up?

Needs to use the QEMU_PACKED macro.

-- PMM

Thanks.