mtd, ubi, ubifs: resync with Linux-3.14

- Following Code in drivers/mtd/nand/nand_base.c nand_do_write_ops() adapted for U-Boot:

+#ifndef __UBOOT__

        /* Reject writes, which are not page aligned */
        if (NOTALIGNED(to) || NOTALIGNED(ops->len)) {

+else
+ /* Reject writes, which are not page aligned */
+ if (NOTALIGNED(to)) {
+endif

as the original linux code leads in not working use of the env var "filesize". For example a "nand write 80000000 80000 ${filesize}" would not work with it ...

- add CONFIG_MTD_NAND_VERIFY_WRITE from U-Boot code (only compile tested)

- Documented the config defines in README

- kmalloc now uses memalign for allocating memory. This prevented crashes when tested ubi/ubifs on imx6 board.

- To produce this patch I did three steps:
- copied the linux source files to U-Boot tree -> commit this
- adapt license text in each file -> commit this
- make the code again compile clean and working -> commit this

Quoted 10 lines

- Following Code in drivers/mtd/nand/nand_base.c nand_do_write_ops() adapted for U-Boot:

   +#ifndef __UBOOT__
         /* Reject writes, which are not page aligned */
         if (NOTALIGNED(to) || NOTALIGNED(ops->len)) {
   +else
   +     /* Reject writes, which are not page aligned */
   +     if (NOTALIGNED(to)) {
   +endif

This sort of ifdeffing is a bad idea -- if a larger number of lines are ifdeffed, to avoid context conflicts, subsequent merges can silently put updates into the non-U-Boot section that are needed in the U-Boot section.

Plus, it impairs readability, and in this case breaks text-editor brace-matching.

Plus plus, that line has been that way in Linux since 2006 -- why is it being touched at all? Did you merge in changes from the previous mtd sync, or did you just copy the new files over and then try to fix resulting problems? (answered below)

Quoted 12 lines

as the original linux code leads in not working use of the env var "filesize". For example a "nand write 80000000 80000 ${filesize}" would not work with it ...

- add CONFIG_MTD_NAND_VERIFY_WRITE from U-Boot code (only compile tested)

- Documented the config defines in README

- kmalloc now uses memalign for allocating memory. This prevented crashes when tested ubi/ubifs on imx6 board.

Quoted 4 lines

- To produce this patch I did three steps:
- copied the linux source files to U-Boot tree -> commit this
- adapt license text in each file -> commit this
- make the code again compile clean and working -> commit this

OK, so you did just copy stuff over. This method will lead to U-Boot

changes being reverted every time. You should look in the U-Boot

history to see what Linux version corresponded to the last mtd sync, and generate a diff relative to that.

Hello Scott,

Am 17.06.2014 17:54, schrieb Scott Wood:

Quoted 12 lines · Heiko Schocher

Quoted 10 lines

- Following Code in drivers/mtd/nand/nand_base.c nand_do_write_ops() adapted for U-Boot:

   +#ifndef __UBOOT__
         /* Reject writes, which are not page aligned */
         if (NOTALIGNED(to) || NOTALIGNED(ops->len)) {
   +else
   +     /* Reject writes, which are not page aligned */
   +     if (NOTALIGNED(to)) {
   +endif

Quoted 4 lines

This sort of ifdeffing is a bad idea -- if a larger number of lines are ifdeffed, to avoid context conflicts, subsequent merges can silently put updates into the non-U-Boot section that are needed in the U-Boot section.

Yes, in general I agree, this is a bad idea. But I introduced it here to get all differences between U-Boot and Linux marked with the "_UBOOT_" define, so a future sync with linux should be easier ...

Quoted 2 lines

Plus, it impairs readability, and in this case breaks text-editor brace-matching.

Yes, could be fixed, see below.

Quoted 4 lines

Plus plus, that line has been that way in Linux since 2006 -- why is it being touched at all? Did you merge in changes from the previous mtd sync, or did you just copy the new files over and then try to fix resulting problems? (answered below)

Yes. Above Code is a good example. In Linux this piece of code looks like:
-------------
static int nand_do_write_ops(struct mtd_info *mtd, loff_t to,
struct mtd_oob_ops *ops)
[...]

         ops->retlen = 0;
         if (!writelen)
                 return 0;

         /* Reject writes, which are not page aligned */
         if (NOTALIGNED(to) || NOTALIGNED(ops->len)) {
                 pr_notice("%s: attempt to write non page aligned data\n",
                            __func__);
                 return -EINVAL;
         }

         column = to & (mtd->writesize - 1);

-------------

in current U-Boot:

-------------
static int nand_do_write_ops(struct mtd_info *mtd, loff_t to,
struct mtd_oob_ops *ops)
[...]

         ops->retlen = 0;
         if (!writelen)
                 return 0;

         column = to & (mtd->writesize - 1);

-------------

Why is this "if (NOTALIGNED(to) || NOTALIGNED(ops->len)) {" missing in U-Boot? Do we allow "not aligned writes" in U-Boot?

If so, we should have documented this, and best place for it, I think, would be with a comment in the source code...

Above Code is a good example for that. Linux has this code since 2006, why has U-Boot not this code?

Quoted 2 lines

history to see what Linux version corresponded to the last mtd sync, and generate a diff relative to that.

I think, we should have the linux code as the reference, and on top of this, we should add our U-Boot changes ... Thats the reason for the git tree [1].

There we have the "simple copy from linux patch" [2]. It overwrites the U-Boot specific changes, yes, but only in an intermediate step in this git tree ... (Maybe I should add in the commit message, that this is a simple copy from linux sources)

Then a U-Boot specifc patch [3], which introduces again hopefully all U-Boot specific changes... so *all* U-Boot specific changs are documented. At last a seperate licencse file patch [4] added which changes the license text.

My hope was, that future linux mtd syncs could look like:

a) copy new linux source files to u-boot tree [1] (make a new branch go back to commit [2] ... and copy new files from linux) (commit this, to have all linux changes since last sync with linux documented)
b) apply [3] and fix problems
All U-Boot changes documented
c) apply [4] and fix problems
d) add newly added U-Boot specific patches
(mark them with _UBOOT_ define if they are not yet marked with it) (maybe squash them to step b ?)

-> new git tree ready

e) squash all patches into one patch and send it to mainline ... so the mainline patch has only new changes with all U-Boot specific changes.

Thats sounds easy ... and you have all linux and all U-Boot changes visible and documented ... I must admit, I did not tried your proposal to look into the linux changes since the last mtd sync ... but that sounds the more tricky way to get this sync, as the history can be long ... and shouldn;t be the result the same?

Quoted 1 line · Heiko Schocher

Hello Scott,

Quoted 6 lines

Quoted 2 lines

history to see what Linux version corresponded to the last mtd sync, and generate a diff relative to that.

I think, we should have the linux code as the reference, and on top of this, we should add our U-Boot changes ... Thats the reason for the git tree [1].

One of the strengths of version control is the ability to do three-way merges, rather than throw away the downstream code and reassemble it from scratch.

What does #ifndef __UBOOT__ tell you that a diff between the U-Boot and Linux files, as of the SHA1 that was last merged into U-Boot, doesn't tell you?

Quoted 23 lines

There we have the "simple copy from linux patch" [2]. It overwrites the U-Boot specific changes, yes, but only in an intermediate step in this git tree ... (Maybe I should add in the commit message, that this is a simple copy from linux sources)

Then a U-Boot specifc patch [3], which introduces again hopefully all U-Boot specific changes... so *all* U-Boot specific changs are documented. At last a seperate licencse file patch [4] added which changes the license text.

My hope was, that future linux mtd syncs could look like:

a) copy new linux source files to u-boot tree [1] (make a new branch go back to commit [2] ... and copy new files from linux) (commit this, to have all linux changes since last sync with linux documented)
b) apply [3] and fix problems
All U-Boot changes documented
c) apply [4] and fix problems
d) add newly added U-Boot specific patches
(mark them with _UBOOT_ define if they are not yet marked with it) (maybe squash them to step b ?)

Quoted 5 lines

-> new git tree ready

e) squash all patches into one patch and send it to mainline ... so the mainline patch has only new changes with all U-Boot specific changes.

If you take over as NAND custodian, you can of course do the updates as you please (unless Tom or Wolfgang disagree), but until then I insist on doing a proper merge, and not littering the code with #ifndef __UBOOT__.

Quoted 6 lines

Thats sounds easy ... and you have all linux and all U-Boot changes visible and documented ... I must admit, I did not tried your proposal to look into the linux changes since the last mtd sync ... but that sounds the more tricky way to get this sync, as the history can be long ... and shouldn;t be the result the same?

Why does the length of the history matter? I'm not asking you to go patch by patch. Just take a diff between the SHA1 of last merge and the SHA1 you want to merge now, and apply that patch.

Quoted 2 lines · Heiko Schocher

Quoted 1 line

Hello Scott,

[snip]

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

history to see what Linux version corresponded to the last mtd sync, and generate a diff relative to that.

I think, we should have the linux code as the reference, and on top of this, we should add our U-Boot changes ... Thats the reason for the git tree [1].

One of the strengths of version control is the ability to do three-way merges, rather than throw away the downstream code and reassemble it from scratch.

Indeed.

[snip]

Quoted 45 lines

What does #ifndef __UBOOT__ tell you that a diff between the U-Boot and Linux files, as of the SHA1 that was last merged into U-Boot, doesn't tell you?

Quoted 23 lines

There we have the "simple copy from linux patch" [2]. It overwrites the U-Boot specific changes, yes, but only in an intermediate step in this git tree ... (Maybe I should add in the commit message, that this is a simple copy from linux sources)

Then a U-Boot specifc patch [3], which introduces again hopefully all U-Boot specific changes... so *all* U-Boot specific changs are documented. At last a seperate licencse file patch [4] added which changes the license text.

My hope was, that future linux mtd syncs could look like:

a) copy new linux source files to u-boot tree [1] (make a new branch go back to commit [2] ... and copy new files from linux) (commit this, to have all linux changes since last sync with linux documented)
b) apply [3] and fix problems
All U-Boot changes documented
c) apply [4] and fix problems
d) add newly added U-Boot specific patches
(mark them with _UBOOT_ define if they are not yet marked with it) (maybe squash them to step b ?)

Quoted 5 lines

-> new git tree ready

e) squash all patches into one patch and send it to mainline ... so the mainline patch has only new changes with all U-Boot specific changes.

If you take over as NAND custodian, you can of course do the updates as you please (unless Tom or Wolfgang disagree), but until then I insist on doing a proper merge, and not littering the code with #ifndef __UBOOT__.

Quoted 6 lines

Thats sounds easy ... and you have all linux and all U-Boot changes visible and documented ... I must admit, I did not tried your proposal to look into the linux changes since the last mtd sync ... but that sounds the more tricky way to get this sync, as the history can be long ... and shouldn;t be the result the same?

Quoted 3 lines

Why does the length of the history matter? I'm not asking you to go patch by patch. Just take a diff between the SHA1 of last merge and the SHA1 you want to merge now, and apply that patch.

But it's not even that hard to do so! git format-patch -o mtd-resync v3.7..v3.14 drivers/mtd/nand will give you only the relevant patches,

and they should mostly apply, and then you can squash them after the fact.

Hello Tom, Scott,

Am 18.06.2014 23:42, schrieb Tom Rini:

Quoted 67 lines · Scott Wood

Quoted 2 lines · Heiko Schocher

Quoted 1 line

Hello Scott,

[snip]

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

history to see what Linux version corresponded to the last mtd sync, and generate a diff relative to that.

I think, we should have the linux code as the reference, and on top of this, we should add our U-Boot changes ... Thats the reason for the git tree [1].

One of the strengths of version control is the ability to do three-way merges, rather than throw away the downstream code and reassemble it from scratch.

Indeed.

[snip]

Quoted 48 lines

What does #ifndef __UBOOT__ tell you that a diff between the U-Boot and Linux files, as of the SHA1 that was last merged into U-Boot, doesn't tell you?

Quoted 23 lines

There we have the "simple copy from linux patch" [2]. It overwrites the U-Boot specific changes, yes, but only in an intermediate step in this git tree ... (Maybe I should add in the commit message, that this is a simple copy from linux sources)

Then a U-Boot specifc patch [3], which introduces again hopefully all U-Boot specific changes... so *all* U-Boot specific changs are documented. At last a seperate licencse file patch [4] added which changes the license text.

My hope was, that future linux mtd syncs could look like:

a) copy new linux source files to u-boot tree [1] (make a new branch go back to commit [2] ... and copy new files from linux) (commit this, to have all linux changes since last sync with linux documented)
b) apply [3] and fix problems
All U-Boot changes documented
c) apply [4] and fix problems
d) add newly added U-Boot specific patches
(mark them with _UBOOT_ define if they are not yet marked with it) (maybe squash them to step b ?)

Quoted 5 lines

-> new git tree ready

e) squash all patches into one patch and send it to mainline ... so the mainline patch has only new changes with all U-Boot specific changes.

If you take over as NAND custodian, you can of course do the updates as you please (unless Tom or Wolfgang disagree), but until then I insist on doing a proper merge, and not littering the code with #ifndef __UBOOT__.

Quoted 6 lines

Thats sounds easy ... and you have all linux and all U-Boot changes visible and documented ... I must admit, I did not tried your proposal to look into the linux changes since the last mtd sync ... but that sounds the more tricky way to get this sync, as the history can be long ... and shouldn;t be the result the same?

Why does the length of the history matter? I'm not asking you to go patch by patch. Just take a diff between the SHA1 of last merge and the SHA1 you want to merge now, and apply that patch.

Quoted 2 lines

But it's not even that hard to do so! git format-patch -o mtd-resync v3.7..v3.14 drivers/mtd/nand will give you only the relevant patches,

"git format-patch -o mtd-resync v3.7..v3.14 drivers/mtd/nand" results in 440 patches ... :-(

"git am" first patch immediately fails ...

Ok, one step back ... I try only "drivers/mtd/mtdcore.c":

$ git format-patch -o mtdcore v3.7..v3.14 drivers/mtd/mtdcore.c
mtdcore/0001-mtd-convert-to-idr_alloc.patch
mtdcore/0002-mtd-add-const-qualifier-to-a-couple-of-register-func.patch
mtdcore/0003-mtd-mtdcore-remove-few-useless-ifdef-s.patch
mtdcore/0004-mtd-merge-mtdchar-module-with-mtdcore.patch
mtdcore/0005-Include-missing-linux-slab.h-inclusions.patch
mtdcore/0006-drivers-avoid-format-string-in-dev_set_name.patch
mtdcore/0007-mtd-add-a-new-sys-node-to-show-the-ecc-step-size.patch
mtdcore/0008-mtd-add-MTD_MLCNANDFLASH-case-for-mtd_type_show.patch
mtdcore/0009-mtd-Move-major-number-definitions-to-major.h.patch
mtdcore/0010-mtd-remove-duplicated-include-from-mtdcore.c.patch
mtdcore/0011-mtd-convert-to-use-ATTRIBUTE_GROUPS.patch
$

Only 11 patches, great ... but:

$ git am -3 mtdcore/0001-mtd-convert-to-idr_alloc.patch
Wende an: mtd: convert to idr_alloc()
fatal: sha1 information is lacking or useless (drivers/mtd/mtdcore.c).

Quoted 2 lines

and they should mostly apply, and then you can squash them after the fact.

I try to try this ASAP and report (first steps see above ...). But as a lot of linux code is missing in U-Boot code, I feel, that a lot of chunks will not apply ... but maybe (hopefully) I am wrong ...

Thats the question we must solve, do we want the complete Linux MTD code in U-Boot and mark not used code?

If no, I delete this pieces of code...

Hmm... looking in the history of drivers/mtd/ubi it seems the code is basically from 2008!
http://git.denx.de/?p=u-boot.git;a=history;f=drivers/mtd/ubi;hb=7d2357c1999ff1f93f795282526230a8bd176106

Hmm.. what was the Linux base of this ?

and ubifs from 2009?
http://git.denx.de/?p=u-boot.git;a=history;f=fs/ubifs;hb=7d2357c1999ff1f93f795282526230a8bd176106
Base: "2.6.29-rc6" ... uh, very, very old!

I see there no real syncs with newer linux code ... is this true? Just some fixups ...

I really fear a patch "git format-patch -o ubifs-resync 2.6.29-rc6..v3.14 fs/ubifs" (The result from this command are 355 patches ...)

So, in sum, this are 795 patches + missing ubi patches which must be applied/reworked/reviewed... Sounds painful...

So another question is:
How to generate an update-sync patch:

version a):
- copy the new linux files from Linux v3.14 to U-Boot
- work in the U-Boot specific changes
- make a patch from this

Is this really different to:

version b):
- git format-patch -o mtd-resync v3.7..v3.14 drivers/mtd/nand
- apply them
- make a patch from this

Currently it is differrent, as I introduced the complete linux code and marked unused/different code in U-Boot with __UBOOT__. If I delete this pieces ... version a) and b) should be the same.

I try it ... if I not give up due to count of patches ...

Quoted 70 lines

Hello Tom, Scott,

Am 18.06.2014 23:42, schrieb Tom Rini:

Quoted 67 lines · Scott Wood

Quoted 2 lines · Heiko Schocher

Quoted 1 line

Hello Scott,

[snip]

Quoted 10 lines

Quoted 6 lines

Quoted 2 lines

history to see what Linux version corresponded to the last mtd sync, and generate a diff relative to that.

I think, we should have the linux code as the reference, and on top of this, we should add our U-Boot changes ... Thats the reason for the git tree [1].

One of the strengths of version control is the ability to do three-way merges, rather than throw away the downstream code and reassemble it from scratch.

Indeed.

[snip]

Quoted 48 lines

What does #ifndef __UBOOT__ tell you that a diff between the U-Boot and Linux files, as of the SHA1 that was last merged into U-Boot, doesn't tell you?

Quoted 23 lines

There we have the "simple copy from linux patch" [2]. It overwrites the U-Boot specific changes, yes, but only in an intermediate step in this git tree ... (Maybe I should add in the commit message, that this is a simple copy from linux sources)

Then a U-Boot specifc patch [3], which introduces again hopefully all U-Boot specific changes... so *all* U-Boot specific changs are documented. At last a seperate licencse file patch [4] added which changes the license text.

My hope was, that future linux mtd syncs could look like:

a) copy new linux source files to u-boot tree [1] (make a new branch go back to commit [2] ... and copy new files from linux) (commit this, to have all linux changes since last sync with linux documented)
b) apply [3] and fix problems
All U-Boot changes documented
c) apply [4] and fix problems
d) add newly added U-Boot specific patches
(mark them with _UBOOT_ define if they are not yet marked with it) (maybe squash them to step b ?)

Quoted 5 lines

-> new git tree ready

e) squash all patches into one patch and send it to mainline ... so the mainline patch has only new changes with all U-Boot specific changes.

If you take over as NAND custodian, you can of course do the updates as you please (unless Tom or Wolfgang disagree), but until then I insist on doing a proper merge, and not littering the code with #ifndef __UBOOT__.

Quoted 6 lines

Thats sounds easy ... and you have all linux and all U-Boot changes visible and documented ... I must admit, I did not tried your proposal to look into the linux changes since the last mtd sync ... but that sounds the more tricky way to get this sync, as the history can be long ... and shouldn;t be the result the same?

Why does the length of the history matter? I'm not asking you to go patch by patch. Just take a diff between the SHA1 of last merge and the SHA1 you want to merge now, and apply that patch.

Quoted 5 lines

Quoted 2 lines

But it's not even that hard to do so! git format-patch -o mtd-resync v3.7..v3.14 drivers/mtd/nand will give you only the relevant patches,

"git format-patch -o mtd-resync v3.7..v3.14 drivers/mtd/nand" results in 440 patches ... :-(

This is why I suggested taking one big diff rather than going patch-to-patch. Then you don't have to resolve the same conflicts over and over.

You should also exclude files that U-Boot doesn't have, and include header files that are relevant.

Quoted 18 lines

"git am" first patch immediately fails ...

Ok, one step back ... I try only "drivers/mtd/mtdcore.c":

$ git format-patch -o mtdcore v3.7..v3.14 drivers/mtd/mtdcore.c
mtdcore/0001-mtd-convert-to-idr_alloc.patch
mtdcore/0002-mtd-add-const-qualifier-to-a-couple-of-register-func.patch
mtdcore/0003-mtd-mtdcore-remove-few-useless-ifdef-s.patch
mtdcore/0004-mtd-merge-mtdchar-module-with-mtdcore.patch
mtdcore/0005-Include-missing-linux-slab.h-inclusions.patch
mtdcore/0006-drivers-avoid-format-string-in-dev_set_name.patch
mtdcore/0007-mtd-add-a-new-sys-node-to-show-the-ecc-step-size.patch
mtdcore/0008-mtd-add-MTD_MLCNANDFLASH-case-for-mtd_type_show.patch
mtdcore/0009-mtd-Move-major-number-definitions-to-major.h.patch
mtdcore/0010-mtd-remove-duplicated-include-from-mtdcore.c.patch
mtdcore/0011-mtd-convert-to-use-ATTRIBUTE_GROUPS.patch
$

Quoted 5 lines

Only 11 patches, great ... but:

$ git am -3 mtdcore/0001-mtd-convert-to-idr_alloc.patch
Wende an: mtd: convert to idr_alloc()
fatal: sha1 information is lacking or useless (drivers/mtd/mtdcore.c).

If you fetch the Linux tree into your U-boot repository, then -3 will work.

Quoted 6 lines

Quoted 2 lines

and they should mostly apply, and then you can squash them after the fact.

I try to try this ASAP and report (first steps see above ...). But as a lot of linux code is missing in U-Boot code, I feel, that a lot of chunks will not apply ... but maybe (hopefully) I am wrong ...

If the conflict is due to a change that applies to code that is missing because it doesn't make sense in U-Boot, then just delete that conflict and move on. Unlike having the change silently apply to an ifndef region, you can first evaluate the change to see if it should apply to some alternative U-Boot version of the code.

Quoted 2 lines

Thats the question we must solve, do we want the complete Linux MTD code in U-Boot and mark not used code?

No.

Quoted 7 lines

If no, I delete this pieces of code...

Hmm... looking in the history of drivers/mtd/ubi it seems the code is basically from 2008!
http://git.denx.de/?p=u-boot.git;a=history;f=drivers/mtd/ubi;hb=7d2357c1999ff1f93f795282526230a8bd176106

Hmm.. what was the Linux base of this ?

See commit dfe64e2c89731a3f9950d7acd8681b68df2bae03 "mtd: resync with Linux-3.7.1"

Quoted 6 lines

and ubifs from 2009?
http://git.denx.de/?p=u-boot.git;a=history;f=fs/ubifs;hb=7d2357c1999ff1f93f795282526230a8bd176106
Base: "2.6.29-rc6" ... uh, very, very old!

I see there no real syncs with newer linux code ... is this true? Just some fixups ...

CCing Kyungmin who is the UBI maintainer.

Quoted 5 lines

I really fear a patch "git format-patch -o ubifs-resync 2.6.29-rc6..v3.14 fs/ubifs" (The result from this command are 355 patches ...)

So, in sum, this are 795 patches + missing ubi patches which must be applied/reworked/reviewed... Sounds painful...

I don't know about ubifs, but the core NAND code (and apparently the core UBI code) is currently synced with Linux 3.7.1.

Quoted 8 lines

So another question is:
How to generate an update-sync patch:

version a):
- copy the new linux files from Linux v3.14 to U-Boot
- work in the U-Boot specific changes
- make a patch from this

Quoted 6 lines

Is this really different to:

version b):
- git format-patch -o mtd-resync v3.7..v3.14 drivers/mtd/nand
- apply them
- make a patch from this

It is different because "work in the U-Boot specific changes" is a manual, error-prone step compared to letting git do most of the work.

Quoted 5 lines

Currently it is differrent, as I introduced the complete linux code and marked unused/different code in U-Boot with __UBOOT__. If I delete this pieces ... version a) and b) should be the same.

I try it ... if I not give up due to count of patches ...

Again, there's no need to go patch-by-patch. We haven't done so on prior updates.

Hello Scott,

Am 20.06.2014 01:45, schrieb Scott Wood:

Quoted 4 lines · Heiko Schocher

Quoted 3 lines

Hello Tom, Scott,

Am 18.06.2014 23:42, schrieb Tom Rini:

Quoted 31 lines

Quoted 5 lines

I really fear a patch "git format-patch -o ubifs-resync 2.6.29-rc6..v3.14 fs/ubifs" (The result from this command are 355 patches ...)

So, in sum, this are 795 patches + missing ubi patches which must be applied/reworked/reviewed... Sounds painful...

I don't know about ubifs, but the core NAND code (and apparently the core UBI code) is currently synced with Linux 3.7.1.

Quoted 8 lines

So another question is:
How to generate an update-sync patch:

version a):
- copy the new linux files from Linux v3.14 to U-Boot
- work in the U-Boot specific changes
- make a patch from this

Quoted 6 lines

Is this really different to:

version b):
- git format-patch -o mtd-resync v3.7..v3.14 drivers/mtd/nand
- apply them
- make a patch from this

It is different because "work in the U-Boot specific changes" is a manual, error-prone step compared to letting git do most of the work.

Quoted 4 lines

Currently it is differrent, as I introduced the complete linux code and marked unused/different code in U-Boot with __UBOOT__. If I delete this pieces ... version a) and b) should be the same.

Quoted 4 lines

Quoted 1 line

I try it ... if I not give up due to count of patches ...

Again, there's no need to go patch-by-patch. We haven't done so on prior updates.

I try it this way.

And I have no chance to detect this difference, when using "git am -3 ..." ... it just remains in the code ...

I vote for copying the linux files, marking U-Boot specific code with __UBOOT__ ...

Dear Heiko,

Quoted 6 lines · In message <[email protected]> you wrote:

And I have no chance to detect this difference, when using "git am -3 ..." ... it just remains in the code ...

I vote for copying the linux files, marking U-Boot specific code with __UBOOT__ ...

Given the complexity of the code - and of the changes added in U-Boot to the original (old) Linux code (which were done over a period of time) - I think indeed that your original approach is the better one; better both in the sense of requiring less efforts (for creating and verifying that all chanes were covered), and less risk to miss individual modifications either from the Linux or from the U-Boot side.

Scott, I agree that your suggestion is what usually should work fine, but here it apparently fails in a number of places that would be time consuming to sort out - and I would expect that the same would happen again whenever we update to another new version of the Linux code base. So I think we should stick with Heiko's approach here; it documents clearly what he did, it looks complete, and it is working. I think it would be a waste of time to redo all the work, just differently.

Quoted 18 lines

Dear Heiko,

Quoted 6 lines · In message<[email protected]> you wrote:

And I have no chance to detect this difference, when using "git am -3 ..." ... it just remains in the code ...

I vote for copying the linux files, marking U-Boot specific code with __UBOOT__ ...

Given the complexity of the code - and of the changes added in U-Boot to the original (old) Linux code (which were done over a period of time) - I think indeed that your original approach is the better one; better both in the sense of requiring less efforts (for creating and verifying that all chanes were covered), and less risk to miss individual modifications either from the Linux or from the U-Boot side.

Quoted 8 lines

Scott, I agree that your suggestion is what usually should work fine, but here it apparently fails in a number of places that would be time consuming to sort out - and I would expect that the same would happen again whenever we update to another new version of the Linux code base. So I think we should stick with Heiko's approach here; it documents clearly what he did, it looks complete, and it is working. I think it would be a waste of time to redo all the work, just differently.

OK, lets try this. One worry at the back of my mind is the fallout we had when we re-synced to v3.7.1 and tested things as best we were able to prior to merge. I think it's too late in the cycle to pull this in

for v2014.07, but lets get something in for v2014.10. And since v3.15

is already out, lets do (as part of testing our theories out), a follow up patch to sync up with v3.15. And finally, I think we need, in order to keep this pain point down, sync per kernel release.

Hello Tom,

Am 23.06.2014 17:05, schrieb Tom Rini:

Quoted 27 lines · Wolfgang Denk

Quoted 25 lines

Dear Heiko,

Quoted 5 lines · In message<[email protected]> you wrote:

And I have no chance to detect this difference, when using "git am -3 ..." ... it just remains in the code ...

I vote for copying the linux files, marking U-Boot specific code with __UBOOT__ ...

Given the complexity of the code - and of the changes added in U-Boot to the original (old) Linux code (which were done over a period of time) - I think indeed that your original approach is the better one; better both in the sense of requiring less efforts (for creating and verifying that all chanes were covered), and less risk to miss individual modifications either from the Linux or from the U-Boot side.

Scott, I agree that your suggestion is what usually should work fine, but here it apparently fails in a number of places that would be time consuming to sort out - and I would expect that the same would happen again whenever we update to another new version of the Linux code base. So I think we should stick with Heiko's approach here; it documents clearly what he did, it looks complete, and it is working. I think it would be a waste of time to redo all the work, just differently.

Quoted 3 lines

OK, lets try this. One worry at the back of my mind is the fallout we had when we re-synced to v3.7.1 and tested things as best we were able to prior to merge. I think it's too late in the cycle to pull this in

Yes I fear such fallout too ... I could also test on some boards only, the rest is compile clean only ... thats the reason why I had the sync serie as RFC ... maybe we create a mtd-test branch? But on the other hand, things maybe get tested only, if they are in mainline ...

Quoted 34 lines · Heiko Schocher

Hello Tom,

Am 23.06.2014 17:05, schrieb Tom Rini:

Quoted 30 lines · Wolfgang Denk

Quoted 17 lines

Dear Heiko,

Quoted 5 lines · In message<[email protected]> you wrote:

And I have no chance to detect this difference, when using "git am -3 ..." ... it just remains in the code ...

I vote for copying the linux files, marking U-Boot specific code with __UBOOT__ ...

Given the complexity of the code - and of the changes added in U-Boot to the original (old) Linux code (which were done over a period of time) - I think indeed that your original approach is the better one; better both in the sense of requiring less efforts (for creating and verifying that all chanes were covered), and less risk to miss individual modifications either from the Linux or from the U-Boot side.

Quoted 8 lines

Scott, I agree that your suggestion is what usually should work fine, but here it apparently fails in a number of places that would be time consuming to sort out - and I would expect that the same would happen again whenever we update to another new version of the Linux code base. So I think we should stick with Heiko's approach here; it documents clearly what he did, it looks complete, and it is working. I think it would be a waste of time to redo all the work, just differently.

OK, lets try this. One worry at the back of my mind is the fallout we had when we re-synced to v3.7.1 and tested things as best we were able to prior to merge. I think it's too late in the cycle to pull this in

Quoted 4 lines

Yes I fear such fallout too ... I could also test on some boards only, the rest is compile clean only ... thats the reason why I had the sync serie as RFC ... maybe we create a mtd-test branch? But on the other hand, things maybe get tested only, if they are in mainline ...

I'd vouch to apply this first thing after .07 is out and fix the problems which arise as we see them.

It'd also be really nice to see MTD synced more often, any volunteers ?

Hello Tom,

Am 23.06.2014 17:05, schrieb Tom Rini:

Quoted 28 lines · Wolfgang Denk

Quoted 18 lines

Dear Heiko,

Quoted 6 lines · In message<[email protected]> you wrote:

And I have no chance to detect this difference, when using "git am -3 ..." ... it just remains in the code ...

I vote for copying the linux files, marking U-Boot specific code with __UBOOT__ ...

Given the complexity of the code - and of the changes added in U-Boot to the original (old) Linux code (which were done over a period of time) - I think indeed that your original approach is the better one; better both in the sense of requiring less efforts (for creating and verifying that all chanes were covered), and less risk to miss individual modifications either from the Linux or from the U-Boot side.

Quoted 8 lines

Scott, I agree that your suggestion is what usually should work fine, but here it apparently fails in a number of places that would be time consuming to sort out - and I would expect that the same would happen again whenever we update to another new version of the Linux code base. So I think we should stick with Heiko's approach here; it documents clearly what he did, it looks complete, and it is working. I think it would be a waste of time to redo all the work, just differently.

Quoted 3 lines

OK, lets try this. One worry at the back of my mind is the fallout we had when we re-synced to v3.7.1 and tested things as best we were able to prior to merge. I think it's too late in the cycle to pull this in

Yes I fear such fallout too ... I could also test on some boards only, the rest is compile clean only ... thats the reason why I had the sync serie as RFC ... maybe we create a mtd-test branch? But on the other hand, things maybe get tested only, if they are in mainline ...

Quoted 1 line

for v2014.07, but lets get something in for v2014.10. And since v3.15

Yep, that was my plan too.

Quoted 3 lines

is already out, lets do (as part of testing our theories out), a follow up patch to sync up with v3.15. And finally, I think we need, in order to keep this pain point down, sync per kernel release.

Ok, I had prepared a v5, posting it soon.

I try to do also to prepare a seperate sync with v3.15 patch, but give me some time ... also I found a bug in current v3.16-rc1 kernel, using ubi fastmap, see:

"UBI: fix rb_tree node comparison in add_map commit buggy?"
http://lists.infradead.org/pipermail/linux-mtd/2014-June/054348.html

So current v3.16-rc1 kernel could not attach a ubi volume using ubi fastmap and created with a kernel not having commit 604b592e6fd3c98f21435e1181ba7723ffc24715 applied ... which is the case for v3.14 ... with my fixup, a v3.16-rc1 kernel could again attach a v3.14 kernel or U-Boot (with my v3.14 sync patches applied) created ubi volume ...

Maybe we wait with the sync, until this is sorted out?

To sync us with each kernel release would be a good idea, but the problem would be, to find time for it ...

Quoted 7 lines

Hello Tom,

Am 23.06.2014 17:05, schrieb Tom Rini:

Quoted 4 lines · Wolfgang Denk

Quoted 3 lines

Dear Heiko,

In message<[email protected]> you wrote:

Quoted 20 lines

Quoted 3 lines

is already out, lets do (as part of testing our theories out), a follow up patch to sync up with v3.15. And finally, I think we need, in order to keep this pain point down, sync per kernel release.

Ok, I had prepared a v5, posting it soon.

I try to do also to prepare a seperate sync with v3.15 patch, but give me some time ... also I found a bug in current v3.16-rc1 kernel, using ubi fastmap, see:

"UBI: fix rb_tree node comparison in add_map commit buggy?"
http://lists.infradead.org/pipermail/linux-mtd/2014-June/054348.html

So current v3.16-rc1 kernel could not attach a ubi volume using ubi fastmap and created with a kernel not having commit 604b592e6fd3c98f21435e1181ba7723ffc24715 applied ... which is the case for v3.14 ... with my fixup, a v3.16-rc1 kernel could again attach a v3.14 kernel or U-Boot (with my v3.14 sync patches applied) created ubi volume ...

Quoted 1 line

Maybe we wait with the sync, until this is sorted out?

Yes, that's fine. Perhaps we'll end up with a sync to v3.14 series, a v3.15 on top of it, and then backporting your bugfix patch(es) on top of that.

Quoted 2 lines

To sync us with each kernel release would be a good idea, but the problem would be, to find time for it ...

Yes, but I think if we do it more regularly it will be better for us and take less time.

Ok, did a "git diff v3.7.1..v3.14 ..." for drivers/mtd/nand and drivers/mtd/mtd*.{c,h} and applied it with "git am -3 ..." after fetching the Linux tree into my U-boot repository ... this command worked now nice ... but there are subtle diffs between the sources in the "copied from linux" tree and the "git am -3" tree ... which did not pop up when doing a "git am -3 ...", just detected them because I could do a diff between "my copied from linux" version and the "git am -3 ..." created version!

And I have no chance to detect this difference, when using "git am -3 ..." ... it just remains in the code ...

I vote for copying the linux files, marking U-Boot specific code with __UBOOT__ ...

Quoted 8 lines

Ok, did a "git diff v3.7.1..v3.14 ..." for drivers/mtd/nand and drivers/mtd/mtd*.{c,h} and applied it with "git am -3 ..." after fetching the Linux tree into my U-boot repository ... this command worked now nice ... but there are subtle diffs between the sources in the "copied from linux" tree and the "git am -3" tree ... which did not pop up when doing a "git am -3 ...", just detected them because I could do a diff between "my copied from linux" version and the "git am -3 ..." created version!

It looks like the 3.7.1 sync was mainly just the NAND code, not the rest of mtd, other than some API changes needed by the NAND code.

Did you find any differences in the NAND code?

Quoted 5 lines

And I have no chance to detect this difference, when using "git am -3 ..." ... it just remains in the code ...

I vote for copying the linux files, marking U-Boot specific code with __UBOOT__ ...

Yes, the 3-way merge approach requires that we know where and when the old code came from. Unfortunately that information is not present outside of drivers/mtd/nand, so for those directories starting from scratch is reasonable (unless we can figure it out based on when the code was added to U-Boot...). Once that's been done, future updates can be done with 3-way merges.

I still disagree with #ifndef __UBOOT__ as it will hurt more than it helps with future merges.

I still disagree with #ifndef __UBOOT__ as it will hurt more than it helps with future merges.

Dear Scott,

Quoted 3 lines · In message <[email protected]> you wrote:

I still disagree with #ifndef __UBOOT__ as it will hurt more than it helps with future merges.

I agree that #ifdef's should be avoided, but then here they also serve a documentation purpose as they clearly mark areas of code that are specific to U-Boot, or that are not used in U-Boot.

Do you think just of a removal, so there will be no trace left of which code was added for U-Boot, and which was removed? Or should these be turned into comments, something like

	/* BEGIN U-Boot specific code */
	...
	/* END U-Boot specific code */

[but that would not handle code removal well].

So what exactly is your proposal how the #fdef's should be removed?

Quoted 4 lines

I agree that #ifdef's should be avoided, but then here they also serve a documentation purpose as they clearly mark areas of code that are specific to U-Boot, or that are not used in U-Boot.

git diff can do that too, without reducing readability or increasing the likelihood of mismerges.

Dear Scott,

Quoted 5 lines · In message <[email protected]> you wrote:

Quoted 4 lines

I agree that #ifdef's should be avoided, but then here they also serve a documentation purpose as they clearly mark areas of code that are specific to U-Boot, or that are not used in U-Boot.

Quoted 2 lines

git diff can do that too, without reducing readability or increasing the likelihood of mismerges.

(git) diff needs a reference to diff against. Maybe I missed some earlier comments about that, but how exactly should this be done here?

If we want to have strictly bisectable code in mainline, I don't really see how to provide an unmodified source reference to show the U-Boot specific diffs.

Can you please explain (maybe again, sorry) how you think the code update should be done do both allow such a git diff and remain bisectable? The ways I can see all require a non-working / non-com- piling intermediate step.

Quoted 4 lines

(git) diff needs a reference to diff against. Maybe I missed some earlier comments about that, but how exactly should this be done here?

When the code is synced, the corresponding Linux SHA1 or tag should be placed in the commit message. This is the case for the NAND code. Unfortunately it has never been true of the mtd/ubi code, but this do-over in those trees is an opportunity to fix that.

There is no intermediate step.

The unmodified source is in the Linux tree -- most likely in a more accurate/complete form than the ifdefs convey, since minor differences and subsequent local changes are unlikely to be marked.

You can either fetch the Linux tree into your local U-Boot repo (or vice versa) so that git diff can compare them, or you can check out separate repositories to the proper tags/SHAs and use ordinary diff. It's not something that I'd expect one to want to do very often, though. Usually

you want to know how things work in the codebase you're working on, or you want to compare some specific aspect of the code which can more easily be done manually.

Quoted 4 lines

When the code is synced, the corresponding Linux SHA1 or tag should be placed in the commit message. This is the case for the NAND code. Unfortunately it has never been true of the mtd/ubi code, but this do-over in those trees is an opportunity to fix that.

Heiko already did that.

Quoted 10 lines

There is no intermediate step.

The unmodified source is in the Linux tree -- most likely in a more accurate/complete form than the ifdefs convey, since minor differences and subsequent local changes are unlikely to be marked.

You can either fetch the Linux tree into your local U-Boot repo (or vice versa) so that git diff can compare them, or you can check out separate repositories to the proper tags/SHAs and use ordinary diff. It's not something that I'd expect one to want to do very often, though. Usually

Are you talking here about the theoretitical possibility to do that, or have you actually done that - and found the result useful? Maybe I'm doing something wrong, but I get such a huge volume of new resp. deleted files that I find it really, really hard to see any diffs at all.

Quoted 8 lines · Wolfgang Denk

Dear Scott,

Quoted 5 lines · In message <[email protected]> you wrote:

Quoted 4 lines

I agree that #ifdef's should be avoided, but then here they also serve a documentation purpose as they clearly mark areas of code that are specific to U-Boot, or that are not used in U-Boot.

Quoted 6 lines

Quoted 2 lines

git diff can do that too, without reducing readability or increasing the likelihood of mismerges.

(git) diff needs a reference to diff against. Maybe I missed some earlier comments about that, but how exactly should this be done here?

When the code is synced, the corresponding Linux SHA1 or tag should be placed in the commit message. This is the case for the NAND code. Unfortunately it has never been true of the mtd/ubi code, but this do-over in those trees is an opportunity to fix that.

Quoted 8 lines

If we want to have strictly bisectable code in mainline, I don't really see how to provide an unmodified source reference to show the U-Boot specific diffs.

Can you please explain (maybe again, sorry) how you think the code update should be done do both allow such a git diff and remain bisectable? The ways I can see all require a non-working / non-com- piling intermediate step.

There is no intermediate step.

The unmodified source is in the Linux tree -- most likely in a more accurate/complete form than the ifdefs convey, since minor differences and subsequent local changes are unlikely to be marked.

You can either fetch the Linux tree into your local U-Boot repo (or vice versa) so that git diff can compare them, or you can check out separate repositories to the proper tags/SHAs and use ordinary diff. It's not something that I'd expect one to want to do very often, though. Usually you want to know how things work in the codebase you're working on, or you want to compare some specific aspect of the code which can more easily be done manually.

Quoted 8 lines

You can either fetch the Linux tree into your local U-Boot repo (or vice versa) so that git diff can compare them, or you can check out separate repositories to the proper tags/SHAs and use ordinary diff. It's not something that I'd expect one to want to do very often, though. Usually you want to know how things work in the codebase you're working on, or you want to compare some specific aspect of the code which can more easily be done manually.

So an example of this would be doing say:

$ git diff 3dad234 drivers/mtd/nand/nand_base.c

Where 3dad234 is the last kernel commit to U-Boot drivers/mtd/nand/nand_base.c where we synced right (and we've fetched the kernel sources into this repo as well) ? (This commit is also valid for atmel_nand.c and I looked over the results there too..).

We must, really, follow the general rule where when we sync stuff from the kernel we say what the commit we synced with is, for UBI/etc. We may not have been for forever but the last time we also said that tag it was against.

In sum, if Heiko is going to be owning UBI/related code and syncs, whatever makes life easiest on _him_ to make sure we don't have regressions is fine with me (within reason, and this seems to be within reason). Do not get me wrong please, I appreciate all the time you've spent thus far in U-Boot and wish and want you to comment / review / advise as you're still able to find time. But if someone else is taking up doing the re-syncs it needs to also fit their workflow.

Quoted 11 lines · Scott Wood

The unmodified source is in the Linux tree -- most likely in a more accurate/complete form than the ifdefs convey, since minor differences and subsequent local changes are unlikely to be marked.

You can either fetch the Linux tree into your local U-Boot repo (or vice versa) so that git diff can compare them, or you can check out separate repositories to the proper tags/SHAs and use ordinary diff. It's not something that I'd expect one to want to do very often, though. Usually you want to know how things work in the codebase you're working on, or you want to compare some specific aspect of the code which can more easily be done manually.

So an example of this would be doing say:

$ git diff 3dad234 drivers/mtd/nand/nand_base.c

Where 3dad234 is the last kernel commit to U-Boot drivers/mtd/nand/nand_base.c where we synced right (and we've fetched the kernel sources into this repo as well) ?

(This commit is
also valid for atmel_nand.c and I looked over the results there too..).

We must, really, follow the general rule where when we sync stuff from the kernel we say what the commit we synced with is, for UBI/etc. We may not have been for forever but the last time we also said that tag it was against.

But how we denote U-Boot vs Linux Kernel differences is somewhat up to the subsystem maintainer. We must be careful when syncing back. But looking over the above diff, as a more casual nand developer, I wish there was more in-code context about why we are different. But that's just me and how I develop. It would be pretty easy (but not quite easily scriptable) to whack out the #ifndef/else/endif __UBOOT__'s, commit that, and be back to having the output one wants, if that's how one works.

In sum, if Heiko is going to be owning UBI/related code and syncs, whatever makes life easiest on _him_ to make sure we don't have regressions is fine with me (within reason, and this seems to be within reason). Do not get me wrong please, I appreciate all the time you've spent thus far in U-Boot and wish and want you to comment / review / advise as you're still able to find time. But if someone else is taking up doing the re-syncs it needs to also fit their workflow.