wic support different sector size
v2 cover wic support different sector size
v1:
- wic support different sector size
- util-linux: sfdisk sector size improvements (merged)
v2:
- For "wic ls" modify get_partitions() to support WIC_SECTOR_SIZE.
- Add oe-selftest for WIC_SECTOR_SIZE.
Vince Chang (2):
wic: add WIC_SECTOR_SIZE variable
selftest: wic: add test for WIC_SECTOR_SIZE
meta/lib/oeqa/selftest/cases/wic.py | 50 +++++++++++++++++++++ scripts/lib/wic/engine.py | 18 +++++++- scripts/lib/wic/plugins/imager/direct.py | 56 ++++++++++++++---------- 3 files changed, 100 insertions(+), 24 deletions(-)
Signature · 1 line
2.25.1
PATCH v2 1/2 wic: add WIC_SECTOR_SIZE variable
Currently WIC is unable to generate images that requires a sector size different of 512. Add WIC_SECTOR_SIZE variable to handle the sector size of 4096 for UFS.
For "wic ls" command modify get_partitions() to support WIC_SECTOR_SIZE.
--- scripts/lib/wic/engine.py | 18 +++++++- scripts/lib/wic/plugins/imager/direct.py | 56 ++++++++++++++---------- 2 files changed, 50 insertions(+), 24 deletions(-)
scripts/lib/wic/engine.py +17 -1
@@ -232,6 +232,16 @@ class Disk: self._psector_size = None self._ptable_format = None + # define sector size + sector_size_str = get_bitbake_var('WIC_SECTOR_SIZE') + if sector_size_str is not None: + try: + self.sector_size = int(sector_size_str) + except ValueError: + self.sector_size = None + else: + self.sector_size = None + # find parted # read paths from $PATH environment variable # if it fails, use hardcoded paths
@@ -258,7 +268,13 @@ class Disk: def get_partitions(self): if self._partitions is None: self._partitions = OrderedDict() - out = exec_cmd("%s -sm %s unit B print" % (self.parted, self.imagepath)) + + if self.sector_size is not None: + out = exec_cmd("export PARTED_SECTOR_SIZE=%d; %s -sm %s unit B print" % \ + (self.sector_size, self.parted, self.imagepath), True) + else: + out = exec_cmd("%s -sm %s unit B print" % (self.parted, self.imagepath)) + parttype = namedtuple("Part", "pnum start end size fstype") splitted = out.splitlines() # skip over possible errors in exec_cmd output
scripts/lib/wic/plugins/imager/direct.py +33 -23
114 lines not on this page · raw message
Signature · 1 line
2.25.1
PATCH v2 2/2 selftest: wic: add test for WIC_SECTOR_SIZE
In reply to Vince ChangAdd test for WIC_SECTOR_SIZE=4096. Verified it on ext4 file system.
--- meta/lib/oeqa/selftest/cases/wic.py | 50 +++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+)
meta/lib/oeqa/selftest/cases/wic.py +50 -0
@@ -839,6 +839,56 @@ bootloader --ptable gpt""") finally: os.remove(wks_file) + def test_wic_sector_size(self): + """Test generation image sector size""" + # Add WIC_SECTOR_SIZE into config + config = 'WIC_SECTOR_SIZE = "4096"\n'\ + 'WIC_BLOCK_SIZE = "4096"\n'\ + 'WICVARS:append = " WIC_SECTOR_SIZE WIC_BLOCK_SIZE"\n' + self.append_config(config) + bitbake('core-image-minimal') + + # Check WIC_SECTOR_SIZE apply to bitbake variable + wic_sector_size_str = get_bb_var('WIC_SECTOR_SIZE', 'core-image-minimal') + wic_sector_size = int(wic_sector_size_str) + self.assertEqual(4096, wic_sector_size) + + self.logger.info("Test wic_sector_size: %d \n" % wic_sector_size) + + with NamedTemporaryFile("w", suffix=".wks") as wks: + wks.writelines( + ['bootloader --ptable gpt\n', + 'part --fstype ext4 --source rootfs --label rofs-a --mkfs-extraopts "-b 4096"\n', + 'part --fstype ext4 --source rootfs --use-uuid --mkfs-extraopts "-b 4096"\n']) + wks.flush() + cmd = "wic create %s -e core-image-minimal -o %s" % (wks.name, self.resultdir) + runCmd(cmd) + wksname = os.path.splitext(os.path.basename(wks.name))[0] + images = glob(os.path.join(self.resultdir, "%s-*direct" % wksname)) + self.assertEqual(1, len(images)) + + sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools') + # list partitions + result = runCmd("wic ls %s -n %s" % (images[0], sysroot)) + self.assertEqual(3, len(result.output.split('\n'))) + self.logger.info("result: %s \n" % result.output) + + # verify partition size with wic + res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0])) + + self.logger.info("res: %s \n" % res.output) + # parse parted output which looks like this: + # BYT;\n + # /var/tmp/wic/build/tmpgjzzefdd-202410281021-sda.direct:78569472B:file:4096:4096:gpt::;\n + # 1:139264B:39284735B:39145472B:ext4:rofs-a:;\n + # 2:39284736B:78430207B:39145472B:ext4:primary:;\n + disk_info = res.output.splitlines()[1] + # Check sector sizes + sector_size_logical = int(disk_info.split(":")[3]) + sector_size_physical = int(disk_info.split(":")[4]) + self.assertEqual(wic_sector_size, sector_size_logical, "Logical sector size is not %d." % wic_sector_size) + self.assertEqual(wic_sector_size, sector_size_physical, "Physical sector size is not %d." % wic_sector_size) + class Wic2(WicTestCase):
Signature · 1 line
2.25.1
These two patches have merged, and unfortunately there are now failures on the autobuilder:
https://valkyrie.yoctoproject.org/#/builders/35/builds/346/steps/14/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/54/builds/325/steps/14/logs/stdio
Alex
Quoted 30 lines · Vince Chang via lists.openembedded.org
v1:
- wic support different sector size
- util-linux: sfdisk sector size improvements (merged)v2:
- For "wic ls" modifyget_partitions()to supportWIC_SECTOR_SIZE.
- Add oe-selftest forWIC_SECTOR_SIZE.Vince Chang (2):
wic: addWIC_SECTOR_SIZEvariable
selftest: wic: add test forWIC_SECTOR_SIZEmeta/lib/oeqa/selftest/cases/wic.py | 50 +++++++++++++++++++++ scripts/lib/wic/engine.py | 18 +++++++- scripts/lib/wic/plugins/imager/direct.py | 56 ++++++++++++++---------- 3 files changed, 100 insertions(+), 24 deletions(-)--
2.25.1-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206417): https://lists.openembedded.org/g/openembedded-core/message/206417
Mute This Topic: https://lists.openembedded.org/mt/109251303/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Hi Alex,
On Wed, 30 Oct 2024 at 21:01, Alexander Kanavin via lists.openembedded.org
Quoted 4 lines
These two patches have merged, and unfortunately there are now failures on the autobuilder:
https://valkyrie.yoctoproject.org/#/builders/35/builds/346/steps/14/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/54/builds/325/steps/14/logs/stdio
It's strange that this line of code fails to execute. It works fine on my build machine. Is there a way to find out which version of parted is being used on the autobuilder?
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 877, in test_wic_sector_size
res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0]))
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3537958/wic-tmp/tmpr6vrw11i-202410301331-sda.direct unit b p 2>/dev/null' returned non-zero exit status 127:
Thanks,
Vince
Quoted 9 lines · Vince Chang via lists.openembedded.org
Hi Alex,
On Wed, 30 Oct 2024 at 21:01, Alexander Kanavin via lists.openembedded.org
Quoted 4 lines
These two patches have merged, and unfortunately there are now failures on the autobuilder:
https://valkyrie.yoctoproject.org/#/builders/35/builds/346/steps/14/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/54/builds/325/steps/14/logs/stdio
Quoted 8 lines
It's strange that this line of code fails to execute. It works fine on my build machine. Is there a way to find out which version of parted is being used on the autobuilder?
File "
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 877, intest_wic_sector_size
res =runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0]))
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raiseAssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3537958/wic-tmp/tmpr6vrw11i-202410301331-sda.directunit b p 2>/dev/null' returned non-zero exit status 127:
perhaps make it
PARTED_SECTOR_SIZE=%d parted -s %s mklabel %s" % (self.sector_size, device, ptable_format), self.native_sysroot)
Quoted 11 lines
Thanks,
Vince-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206566): https://lists.openembedded.org/g/openembedded-core/message/206566
Mute This Topic: https://lists.openembedded.org/mt/109308341/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
Quoted 10 lines · Khem Raj
Quoted 6 lines
File "
/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 877, intest_wic_sector_size
res =runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0]))
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raiseAssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m/srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3537958/wic-tmp/tmpr6vrw11i-202410301331-sda.directunit b p 2>/dev/null' returned non-zero exit status 127:perhaps make it
PARTED_SECTOR_SIZE=%d parted -s %s mklabel %s" % (self.sector_size, device,ptable_format), self.native_sysroot)
I tried this (dropping 2>/dev/null), and indeed the error is no longer obscured:
AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /home/akanavin/poky/build-st/wic-tmp/tmprac0npku-202410311007-sda.direct unit b p' returned non-zero exit status 127:/bin/sh: 1: parted: not found
So basically the test forgets to adjust the PATH to include wic-tools, like others wic tests (that need to run parted) do. Some build hosts have it, others don't, so the error is intermittent.
Also if you can (in a separate commit) drop all usage of this "2>/dev/null" pattern in wic tests that only harms fail diagnostics, that would be helpful.
Alex
Hi Alex,
Quoted 3 lines · Alexander Kanavin
Also if you can (in a separate commit) drop all usage of this "2>/dev/null" pattern in wic tests that only harms fail diagnostics, that would be helpful.
I will send a separate commit to remove "2>/dev/null" in the wic test cases.
Thanks,
Vince
Hi Alex,
Quoted 8 lines · Alexander Kanavin
Quoted 4 lines
Also if you can (in a separate commit) drop all usage of this "2>/dev/null" pattern in wic tests that only harms fail diagnostics, that would be helpful.
I will send a separate commit to remove "2>/dev/null" in the wic test cases.
The wic test cases use out.splitlines to get the command result. If we remove "2>/dev/null", it may capture other warning messages, making it unable to get the current result.
To avoid needing additional code to use splitted.index("BYT;") and considering that ‘2>/dev/null’ is also used in ‘debugfs’, I would suggest not removing "2>/dev/null". If the same issue arises again, we can manually remove it to see where the error is. What do you think?
Thanks,
Vince
Quoted 10 lines · Vince Chang
Quoted 6 lines
Quoted 3 lines
Also if you can (in a separate commit) drop all usage of this "2>/dev/null" pattern in wic tests that only harms fail diagnostics, that would be helpful.
I will send a separate commit to remove "2>/dev/null" in the wic test cases.
The wic test cases use out.splitlines to get the command result. If we remove "2>/dev/null", it may capture other warning messages, making it unable to get the current result.
To avoid needing additional code to usesplitted.index("BYT;")and considering that ‘2>/dev/null’ is also used in ‘debugfs’, I would suggest not removing "2>/dev/null". If the same issue arises again, we can manually remove it to see where the error is. What do you think?
If the issue is intermittent or difficult to reproduce (and in this case it was - I had to spend a couple hours reproducing it for you on a particular build host in yocto CI cluster), it is far better to write the code such that the needed diagnostics are printed right when the error occurs in the first place in CI.
Anyway, implementation runCmd() indeed defaults to stderr being redirected and appended to stdout, so the standard stream and the error stream are mashed together.
To avoid that, you should also set stderr=subprocess.PIPE when calling runCmd(). Check meta/lib/oeqa/selftest/cases/runcmd.py for examples:
def test_output(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True, sync=False) self.assertEqual("stdout\nstderr", result.output) self.assertEqual("", result.error)
def test_output_split(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
stderr=subprocess.PIPE, sync=False)
self.assertEqual("stdout", result.output) self.assertEqual("stderr", result.error)
Alex
Hi Alex,
Quoted 13 lines · Alexander Kanavin
Anyway, implementation
runCmd()indeed defaults to stderr being redirected and appended to stdout, so the standard stream and the error stream are mashed together.To avoid that, you should also set stderr=subprocess.PIPE when calling
runCmd(). Checkmeta/lib/oeqa/selftest/cases/runcmd.pyfor examples:def
test_output(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
sync=False)
self.assertEqual("stdout\nstderr", result.output)self.assertEqual("", result.error)
Quoted 5 lines
def
test_output_split(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
stderr=subprocess.PIPE, sync=False)
self.assertEqual("stdout", result.output)self.assertEqual("stderr", result.error)
I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit.
https://lists.openembedded.org/g/openembedded-core/message/206602
Thanks,
Vince
Thanks! But you should still fix the original fail I think?
Alex
Quoted 29 lines · Vince Chang
Hi Alex,
Quoted 13 lines · Alexander Kanavin
Anyway, implementation
runCmd()indeed defaults to stderr being redirected and appended to stdout, so the standard stream and the error stream are mashed together.To avoid that, you should also set stderr=subprocess.PIPE when calling
runCmd(). Checkmeta/lib/oeqa/selftest/cases/runcmd.pyfor examples:def
test_output(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
sync=False)
self.assertEqual("stdout\nstderr", result.output)self.assertEqual("", result.error)Quoted 5 lines
def
test_output_split(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
stderr=subprocess.PIPE, sync=False)
self.assertEqual("stdout", result.output)self.assertEqual("stderr", result.error)I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit.
https://lists.openembedded.org/g/openembedded-core/message/206602Thanks,
Vince
Hi Alex,
Yes, I have submitted a commit to add wic-tools into the PATH.
https://lists.openembedded.org/g/openembedded-core/message/206607
Thanks,
Vince
From: Alexander Kanavin <[email protected]>
Sent: Friday, November 1, 2024 3:39 PM
To: Vince Chang <[email protected]>
Cc: Richard Purdie <[email protected]>; [email protected]; Jamin Lin <[email protected]>; Troy Lee <[email protected]>; Khem Raj <[email protected]>
Subject: Re: [OE-core] [PATCH v2 0/2] wic support different sector size
Thanks! But you should still fix the original fail I think?
Alex
On Fri 1. Nov 2024 at 8.37, Vince Chang <[email protected]<mailto:[email protected]>> wrote:
Hi Alex,
Quoted 13 lines · Alexander Kanavin
Anyway, implementation
runCmd()indeed defaults to stderr being redirected and appended to stdout, so the standard stream and the error stream are mashed together.To avoid that, you should also set stderr=subprocess.PIPE when calling
runCmd(). Checkmeta/lib/oeqa/selftest/cases/runcmd.pyfor examples:def
test_output(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
sync=False)
self.assertEqual("stdout\nstderr", result.output)self.assertEqual("", result.error)
Quoted 5 lines
def
test_output_split(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
stderr=subprocess.PIPE, sync=False)
self.assertEqual("stdout", result.output)self.assertEqual("stderr", result.error)
I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit.
https://lists.openembedded.org/g/openembedded-core/message/206602
Thanks,
Vince
Quoted 2 lines · Vince Chang
I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit.
https://lists.openembedded.org/g/openembedded-core/message/206602
I just added one more fix that makes runCmd actually print stderr when it's split from stdout and an error occurs:
https://lists.openembedded.org/g/openembedded-core/message/206610
and with that and your two fixes everything should be working correctly.
Alex