Skip to contentSkip to conversation

Linux-4.X-rcY patches can't be applied with git?

DISCUSSION Jarod Wilson 5 participants 14 messages Oct 24, 2016 – Nov 8, 2016

lore
Share & tools
lore
#0 lore raw

It's entirely possible that we (Red Hat and the Fedora kernel team) are doing something wrong here, but to the best of our knowledge, it seems that the canonical upstream RC snap patches can't be applied to a tree using either git or old-fashioned patch.

For example:

$ cd /tmp
$ wget http://cdn.kernel.org/pub/linux/kernel/v4.x/linux-4.8.tar.xz
$ tar xf linux-4.8.xz
$ cd linux-4.8
$ wget http://cdn.kernel.org/pub/linux/kernel/v4.x/testing/patch-4.9-rc2.xz
$ unxz patch-4.9-rc2.xz
$ git --version
git version 2.10.0
$ git apply patch-4.9-rc2
patch-4.9-rc2:15822: space before tab in indent.
                                IRQ_AUDIO pin
patch-4.9-rc2:15832: space before tab in indent.
                                IRQ_RTC pin
patch-4.9-rc2:19120: space before tab in indent.
                        };
patch-4.9-rc2:20847: space before tab in indent.
        - "moxa,moxart-timer"
patch-4.9-rc2:20848: space before tab in indent.
        - "aspeed,ast2400-timer"
error: cannot apply binary patch to 'Documentation/media/media_api_files/typical_media_device.pdf' without full index line
error: Documentation/media/media_api_files/typical_media_device.pdf: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/dev-raw-vbi_files/vbi_525.pdf' without full index line
error: Documentation/media/uapi/v4l/dev-raw-vbi_files/vbi_525.pdf: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/dev-raw-vbi_files/vbi_625.pdf' without full index line
error: Documentation/media/uapi/v4l/dev-raw-vbi_files/vbi_625.pdf: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt.gif' without full index line
error: Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt.gif: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt.png' without full index line
error: Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt.png: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt_example.gif' without full index line
error: Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt_example.gif: patch does not apply
error: cannot apply binary patch to 'Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt_example.png' without full index line
error: Documentation/media/uapi/v4l/pixfmt-nv12mt_files/nv12mt_example.png: patch does not apply
$ echo $?
1

Most of the patch applies with 'patch -p1 < patch-4.9-rc2', but those same binary files fail. No big deal, except that there's an error return from patch, which means an rpm build using standard rpm macro stuff fails. That's actually one of the reasons we switched to using git to apply any patches in the Fedora kernel rpm spec files. But as per above, that doesn't work all the tme either, which has resulted in the addition of a perl script that gets called to strip binary patch data out entirely, which of course, makes the resulting rpm omit the binary changes entirely, and thus not *quite* 4.9-rc2 that gets spit out here.

Perl script:
http://pkgs.fedoraproject.org/cgit/rpms/kernel.git/tree/remove-binary-diff.pl

However, if the RC patch is generated like so:

$ cd ~/src/linux
$ git diff --binary v4.8..v4.9-rc2 > /tmp/linux-4.8/patch-4.9-rc2

Then:

$ cd /tmp/linux-4.8
$ git apply patch-4.9-rc2
patch-4.9-rc2:8359: space before tab in indent.
        ./configure
patch-4.9-rc2:8719: space before tab in indent.
        alg = crypto_attr_alg(tb[1], CRYPTO_ALG_TYPE_CIPHER,
patch-4.9-rc2:8720: space before tab in indent.
                                  CRYPTO_ALG_TYPE_MASK);
patch-4.9-rc2:8721: space before tab in indent.
        if (IS_ERR(alg))
patch-4.9-rc2:8725: space before tab in indent.
        /* Block size must be >= 4 bytes. */
warning: squelched 104 whitespace errors
warning: 109 lines add whitespace errors.
$ echo $?
0

I just noticed that the 'space before tab in indent' noise is different between the two, but otherwise, things seem to work much better. I know it results in a slightly larger patch file to download (unxz size is 2MB larger), and it still will not apply with plain old patch, but at least it applies with git.

Or am I missing something blindingly obvious in the git apply options that would work with the existing patches?

Signature · 2 lines
Jarod Wilson
[email protected]
#1 lore raw
Quoted 4 lines · Jarod Wilson

It's entirely possible that we (Red Hat and the Fedora kernel team) are doing something wrong here, but to the best of our knowledge, it seems that the canonical upstream RC snap patches can't be applied to a tree using either git or old-fashioned patch.

No, you're not imagining it, it's definitely happening.

What is going on is that I generate patches without the "--binary" flag, which means that git skips the binary diffs entirely. So the diff just contains

Binary files
a/Documentation/media/media_api_files/typical_media_device.pdf and b/Documentation/media/media_api_files/typical_media_device.pdf differ

Then, when you do "git apply", "git apply" will see that, and try to use the index lines to regenerate the thing, which obviously only works in a repository that already _has_ those objects.

This is actually not new. I've skipped binary files for the last ten years or so in the diffs, because the diffs are completely illegible, and nobody has ever cared - and non-git users haven't been able to use them anyway.

Obviously, part of it is too that we simply don't have very many binary files, so it almost never ends up being a problem. The documentation changes made them happen now.

But quite frankly, I see the tar-balls and diffs as a way to have non-git people have a minimally working system, and their *only* point is for people who don't have git.

And since plain "patch" cannot handle git binary diffs anyway, there was never any valid reason to include the binary diffs in the diff.

Btw, this is why the diffs don't have renames either.

The diffs would often be much smaller, and certainly much more legible, if I used the "-M" or "-C" flags, but since the primary reason for the tarballs and diffs existing is for non-git users, and traditional "patch" doesn't understand rename diffs, I don't do it.

(Yes, modern GNU patch has actually grown support for rename diffs, although last I looked it gets it wrong for some of the more complex cases - notably cross renames).

Summary:

- if you have git, you shouldn't use the tar-balls and patches

- if you don't have git, binary patches and renames wouldn't work for
you anyway, so generating them is pointless and would potentially keep you from getting a working tree.

I could easily add "--binary" to my script, because I _think_ traditional "patch" will just ignore it as noise, but I'd honestly rather discourage people from downloading full tar-balls in the first place.

Hmm?

Linus

#2 lore raw
Quoted 55 lines · Linus Torvalds
Quoted 4 lines · Jarod Wilson

It's entirely possible that we (Red Hat and the Fedora kernel team) are doing something wrong here, but to the best of our knowledge, it seems that the canonical upstream RC snap patches can't be applied to a tree using either git or old-fashioned patch.

No, you're not imagining it, it's definitely happening.

What is going on is that I generate patches without the "--binary" flag, which means that git skips the binary diffs entirely. So the diff just contains

Binary files
a/Documentation/media/media_api_files/typical_media_device.pdf and b/Documentation/media/media_api_files/typical_media_device.pdf differ

Then, when you do "git apply", "git apply" will see that, and try to use the index lines to regenerate the thing, which obviously only works in a repository that already _has_ those objects.

This is actually not new. I've skipped binary files for the last ten years or so in the diffs, because the diffs are completely illegible, and nobody has ever cared - and non-git users haven't been able to use them anyway.

Obviously, part of it is too that we simply don't have very many binary files, so it almost never ends up being a problem. The documentation changes made them happen now.

But quite frankly, I see the tar-balls and diffs as a way to have non-git people have a minimally working system, and their *only* point is for people who don't have git.

And since plain "patch" cannot handle git binary diffs anyway, there was never any valid reason to include the binary diffs in the diff.

Btw, this is why the diffs don't have renames either.

The diffs would often be much smaller, and certainly much more legible, if I used the "-M" or "-C" flags, but since the primary reason for the tarballs and diffs existing is for non-git users, and traditional "patch" doesn't understand rename diffs, I don't do it.

(Yes, modern GNU patch has actually grown support for rename diffs, although last I looked it gets it wrong for some of the more complex cases - notably cross renames).

Summary:

- if you have git, you shouldn't use the tar-balls and patches

- if you don't have git, binary patches and renames wouldn't work for
you anyway, so generating them is pointless and would potentially keep you from getting a working tree.

Quoted 6 lines

I could easily add "--binary" to my script, because I _think_ traditional "patch" will just ignore it as noise, but I'd honestly rather discourage people from downloading full tar-balls in the first place.

Hmm?

The benefit of tarballs and patches from a distribution standpoint is purely size. And yes, disk is very cheap but the size implications are magnified by the fact that e.g. SRPMs are required and stored for every binary build. Today, the Fedora kernel SRPMs are ~94.5MB each. If we just used git-archive or something to produce a tarball of a git tree, it would explode to 664MB. That's from a fresh clone of your tree, run through 'git archive -o ../linux.tar.xz master'. If we just tarred it up without xz compression, a fresh tree is 1.8GB. That's a lot of storage, particularly when you're talking about daily builds.

You could just tar up the source files themselves from a git tree and skip .git, but that means you lose the history and more importantly you lose any semblance of verified sources because all the signed commits are then gone. Essentially, it's recreating what is uploaded on kernel.org *minus* the fact that it is signed. That just seems dumb.

Perhaps I'm not thinking of something obvious, but it seems tarballs+patches still have legitimate use cases from a distribution point of view. Making the patches provided on kernel.org usable by the tooling we have available seems like a worthwhile goal to me...

josh

#3 lore raw
Quoted 3 lines · Josh Boyer

The benefit of tarballs and patches from a distribution standpoint is purely size.

I wonder if you wouldn't be better off just generating your own diffs.

If you know the other end uses "git apply", then not only can you use "--binary", but you can also enable rename detection.

That will often shrink the diffs a *lot*.

So for _me_, the primary use of patches is for non-git people.

They are hopefully rare by now, but quite frankly, if they go away as an issue, then the solution for me isn't "start using fancier patches that rely on git", but "don't even bother with tar-balls and patches at all".

See?

But if you want to see patches as a distribution format for a distro, then your issues are different, and you may well want to use "--binary" and "-M" to generate patches.

It's just that your reasons to use patches sound very different from _my_ reasons to use patches..

Linus

#4 lore raw
Quoted 12 lines · Linus Torvalds
Quoted 3 lines · Josh Boyer

The benefit of tarballs and patches from a distribution standpoint is purely size.

I wonder if you wouldn't be better off just generating your own diffs.

If you know the other end uses "git apply", then not only can you use "--binary", but you can also enable rename detection.

That will often shrink the diffs a *lot*.

Quoted 8 lines

So for _me_, the primary use of patches is for non-git people.

They are hopefully rare by now, but quite frankly, if they go away as an issue, then the solution for me isn't "start using fancier patches that rely on git", but "don't even bother with tar-balls and patches at all".

See?

Yes and no. What are these non-git people doing with the patches? Since patch won't apply them, are they just reading them for personal enlightenment? i.e., who really actually uses tarballs and the patches anymore outside of distros for packaging, and for what? From a distro standpoint, I think we'd like to at least continue to see release tarballs, but we could quite easily make do with the "just don't even bother" approach for RC patches.

Quoted 6 lines

But if you want to see patches as a distribution format for a distro, then your issues are different, and you may well want to use "--binary" and "-M" to generate patches.

It's just that your reasons to use patches sound very different from _my_ reasons to use patches..

If "just generate your own" is the best answer, I'm okay with that for my own use case, and I'll let Josh speak for the Fedora case. Just wanted to make sure we weren't missing something obvious, and figured it couldn't hurt to ask if it were simply a case of nobody having said there were issues with them for what I imagine is one of their few remaining uses. :)

Signature · 2 lines
Jarod Wilson
[email protected]
#5 lore raw
Quoted 4 lines · Jarod Wilson

Yes and no. What are these non-git people doing with the patches? Since patch won't apply them, are they just reading them for personal enlightenment?

Oh, patch applies them fine.

They might end up not seeing some of the updates to binary files, that's all. So things like the logo updates or some pdf file might not work out for them, but that doesn't mean that they can't use the patch and the resulting kernel.

Linus

#6 lore raw
Quoted 6 lines · Linus Torvalds
Quoted 4 lines · Jarod Wilson

Yes and no. What are these non-git people doing with the patches? Since patch won't apply them, are they just reading them for personal enlightenment?

Quoted 6 lines

Oh, patch applies them fine.

They might end up not seeing some of the updates to binary files, that's all. So things like the logo updates or some pdf file might not work out for them, but that doesn't mean that they can't use the patch and the resulting kernel.

Ah, you're right, of course. I was conflating "patch won't apply them" with "rpmbuild fails, because patch returns an error code due to the binary bits failing to apply".

But in that case, what if your patch generation script used a filter to exclude those binary files? No harm to that target audience, and it would actually make them behave better for distro builds. Though that might be counter to the goal of making them disappear entirely. :)

Signature · 2 lines
Jarod Wilson
[email protected]
#7 lore raw
Quoted 5 lines · Jarod Wilson

But in that case, what if your patch generation script used a filter to exclude those binary files? No harm to that target audience, and it would actually make them behave better for distro builds. Though that might be counter to the goal of making them disappear entirely. :)

Heh, I'd rather people get the warning that "oops, something is incomplete". They can still work with the end result, but at least they got some indication that hey, that patch didn't work wonderfully well...

To be honest, I really would like to not do the tar-balls and patches at all.

But maybe rather than saying "it's only for legacy 'patch' users", I could just say that it's getting phased out, and say "you have to use 'git apply' to apply them".

Then I could just enable "--binary" and "-M", and see what happens.

I suspect that these days, git is so ubiquitous that it's ok.

And then in a few years, maybe I can just stop doing patches entirely, having proved the point that everybody already has git ;)

Linus

#8 lore raw
Quoted 12 lines · Linus Torvalds
Quoted 5 lines · Jarod Wilson

But in that case, what if your patch generation script used a filter to exclude those binary files? No harm to that target audience, and it would actually make them behave better for distro builds. Though that might be counter to the goal of making them disappear entirely. :)

Heh, I'd rather people get the warning that "oops, something is incomplete". They can still work with the end result, but at least they got some indication that hey, that patch didn't work wonderfully well...

Quoted 7 lines

To be honest, I really would like to not do the tar-balls and patches at all.

But maybe rather than saying "it's only for legacy 'patch' users", I could just say that it's getting phased out, and say "you have to use 'git apply' to apply them".

Then I could just enable "--binary" and "-M", and see what happens.

I like this idea!

Quoted 4 lines

I suspect that these days, git is so ubiquitous that it's ok.

And then in a few years, maybe I can just stop doing patches entirely, having proved the point that everybody already has git ;)

Honestly, the only people that don't have access to git to pull down kernel sources? People who haven't yet got a kernel up and running, who will probably get there via a distro kernel. ;)

Side note in favor of tarballs: I know Fedora likes upstream to have tarballs, with checksums provided, so that packages can be verified to contain a legitimate upstream source tarball, rather than a random tarball created by the packager that could have some extraneous bits (possibly malicious) added to them. One can certainly examine and validate a generated tarball too, but it's a fair bit more work than just "does the checksum match?" and not as easily automated.

Signature · 2 lines
Jarod Wilson
[email protected]
#9 lore raw
Quoted 23 lines · Jarod Wilson
Quoted 12 lines · Linus Torvalds
Quoted 5 lines · Jarod Wilson

But in that case, what if your patch generation script used a filter to exclude those binary files? No harm to that target audience, and it would actually make them behave better for distro builds. Though that might be counter to the goal of making them disappear entirely. :)

Heh, I'd rather people get the warning that "oops, something is incomplete". They can still work with the end result, but at least they got some indication that hey, that patch didn't work wonderfully well...

Quoted 7 lines

To be honest, I really would like to not do the tar-balls and patches at all.

But maybe rather than saying "it's only for legacy 'patch' users", I could just say that it's getting phased out, and say "you have to use 'git apply' to apply them".

Then I could just enable "--binary" and "-M", and see what happens.

I like this idea!

Quoted 8 lines
Quoted 4 lines

I suspect that these days, git is so ubiquitous that it's ok.

And then in a few years, maybe I can just stop doing patches entirely, having proved the point that everybody already has git ;)

Honestly, the only people that don't have access to git to pull down kernel sources? People who haven't yet got a kernel up and running, who will probably get there via a distro kernel. ;)

Possibly. And to your earlier idea of generating our own diffs, we do that for the subset of git snapshot builds we do in Rawhide between the -rcX releases. The problem with doing that all the time for everything is that today the -rcX patches are signed by Linus. Doing it ourselves means they aren't. Fedora/other distros could have the maintainers sign their generated diffs but it loses some of the verification chain.

Quoted 7 lines

Side note in favor of tarballs: I know Fedora likes upstream to have tarballs, with checksums provided, so that packages can be verified to contain a legitimate upstream source tarball, rather than a random tarball created by the packager that could have some extraneous bits (possibly malicious) added to them. One can certainly examine and validate a generated tarball too, but it's a fair bit more work than just "does the checksum match?" and not as easily automated.

Right, this and the signing issue I pointed to above.

josh

#10 lore raw
In reply to Linus Torvalds
Quoted 2 lines · Linus Torvalds

Then I could just enable "--binary" and "-M", and see what happens.

Let's see what happens with 4.9-rc3. Will anybody notice? Did I do it right?

Linus

#11 lore raw
Quoted 9 lines · Linus Torvalds
Quoted 2 lines · Linus Torvalds

Then I could just enable "--binary" and "-M", and see what happens.

Let's see what happens with 4.9-rc3. Will anybody notice? Did I do it right?

Linus

'git apply' works at least. I'll have to spend some time fixing up the Fedora spec file to use git am instead of just patch.

Thanks,
Laura

#12 lore raw
In reply to Linus Torvalds
Quoted 6 lines · Linus Torvalds
Quoted 2 lines · Linus Torvalds

Then I could just enable "--binary" and "-M", and see what happens.

Let's see what happens with 4.9-rc3. Will anybody notice? Did I do it right?

Looks good to me, works perfectly for my use case. Thanks much!

Signature · 2 lines
Jarod Wilson
[email protected]
#13 lore raw
In reply to Linus Torvalds

Hi!

Quoted 16 lines
Quoted 4 lines

But in that case, what if your patch generation script used a filter to exclude those binary files? No harm to that target audience, and it would actually make them behave better for distro builds. Though that might be counter to the goal of making them disappear entirely. :)

Heh, I'd rather people get the warning that "oops, something is incomplete". They can still work with the end result, but at least they got some indication that hey, that patch didn't work wonderfully well...

To be honest, I really would like to not do the tar-balls and patches at all.

But maybe rather than saying "it's only for legacy 'patch' users", I could just say that it's getting phased out, and say "you have to use 'git apply' to apply them".

Quoted 6 lines

Then I could just enable "--binary" and "-M", and see what happens.

I suspect that these days, git is so ubiquitous that it's ok.

And then in a few years, maybe I can just stop doing patches entirely, having proved the point that everybody already has git ;)

Well, having git is not quite the same thing as "having big enough machine to run git on tree as big as kernel". Handheld zaurus was powerful enough to compile kernel (in few hours) but git was pretty painful to use there.

I'm now cross-compiling for n900, so problem is gone there, but there is difference between downloading single release, and downloading full history when you are on a slow line...

Best regards,

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html