Skip to contentSkip to conversation

wic support different sector size

PATCH Superseded Vince Chang 3 participants 14 messages Oct 28, 2024 – Nov 1, 2024

v2 of 3 Superseded Latest is v3 →

lore
Share & tools
lore
#0 lore raw

v2 cover wic support different sector size

v1:
- wic support different sector size
- util-linux: sfdisk sector size improvements (merged)

v2:
- For "wic ls" modify get_partitions() to support WIC_SECTOR_SIZE.
- Add oe-selftest for WIC_SECTOR_SIZE.

Vince Chang (2):
wic: add WIC_SECTOR_SIZE variable
selftest: wic: add test for WIC_SECTOR_SIZE

 meta/lib/oeqa/selftest/cases/wic.py      | 50 +++++++++++++++++++++
 scripts/lib/wic/engine.py                | 18 +++++++-
 scripts/lib/wic/plugins/imager/direct.py | 56 ++++++++++++++----------
 3 files changed, 100 insertions(+), 24 deletions(-)
Signature · 1 line
2.25.1
#1 lore raw

PATCH v2 1/2 wic: add WIC_SECTOR_SIZE variable

Currently WIC is unable to generate images that requires a sector size different of 512. Add WIC_SECTOR_SIZE variable to handle the sector size of 4096 for UFS.

For "wic ls" command modify get_partitions() to support WIC_SECTOR_SIZE.

---
 scripts/lib/wic/engine.py                | 18 +++++++-
 scripts/lib/wic/plugins/imager/direct.py | 56 ++++++++++++++----------
 2 files changed, 50 insertions(+), 24 deletions(-)
scripts/lib/wic/engine.py +17 -1
index ce7e6c5d75..64b1d52882 100644
@@ -232,6 +232,16 @@ class Disk:
         self._psector_size = None
         self._ptable_format = None
 
+        # define sector size
+        sector_size_str = get_bitbake_var('WIC_SECTOR_SIZE')
+        if sector_size_str is not None:
+            try:
+                self.sector_size = int(sector_size_str)
+            except ValueError:
+                self.sector_size = None
+        else:
+            self.sector_size = None
+
         # find parted
         # read paths from $PATH environment variable
         # if it fails, use hardcoded paths
@@ -258,7 +268,13 @@ class Disk:
     def get_partitions(self):
         if self._partitions is None:
             self._partitions = OrderedDict()
-            out = exec_cmd("%s -sm %s unit B print" % (self.parted, self.imagepath))
+
+            if self.sector_size is not None:
+                out = exec_cmd("export PARTED_SECTOR_SIZE=%d; %s -sm %s unit B print" % \
+                           (self.sector_size, self.parted, self.imagepath), True)
+            else:
+                out = exec_cmd("%s -sm %s unit B print" % (self.parted, self.imagepath))
+
             parttype = namedtuple("Part", "pnum start end size fstype")
             splitted = out.splitlines()
             # skip over possible errors in exec_cmd output
scripts/lib/wic/plugins/imager/direct.py +33 -23

114 lines not on this page · raw message

Signature · 1 line
2.25.1
#2 lore raw

PATCH v2 2/2 selftest: wic: add test for WIC_SECTOR_SIZE

In reply to Vince Chang

Add test for WIC_SECTOR_SIZE=4096. Verified it on ext4 file system.

---
 meta/lib/oeqa/selftest/cases/wic.py | 50 +++++++++++++++++++++++++++++
 1 file changed, 50 insertions(+)
meta/lib/oeqa/selftest/cases/wic.py +50 -0
index 5e475f9e3f..c96018abbb 100644
@@ -839,6 +839,56 @@ bootloader --ptable gpt""")
         finally:
             os.remove(wks_file)
 
+    def test_wic_sector_size(self):
+        """Test generation image sector size"""
+        # Add WIC_SECTOR_SIZE into config
+        config = 'WIC_SECTOR_SIZE = "4096"\n'\
+                 'WIC_BLOCK_SIZE = "4096"\n'\
+                 'WICVARS:append = " WIC_SECTOR_SIZE WIC_BLOCK_SIZE"\n'
+        self.append_config(config)
+        bitbake('core-image-minimal')
+
+        # Check WIC_SECTOR_SIZE apply to bitbake variable
+        wic_sector_size_str = get_bb_var('WIC_SECTOR_SIZE', 'core-image-minimal')
+        wic_sector_size = int(wic_sector_size_str)
+        self.assertEqual(4096, wic_sector_size)
+
+        self.logger.info("Test wic_sector_size: %d \n" % wic_sector_size)
+
+        with NamedTemporaryFile("w", suffix=".wks") as wks:
+            wks.writelines(
+                ['bootloader --ptable gpt\n',
+                 'part --fstype ext4 --source rootfs --label rofs-a --mkfs-extraopts "-b 4096"\n',
+                 'part --fstype ext4 --source rootfs --use-uuid --mkfs-extraopts "-b 4096"\n'])
+            wks.flush()
+            cmd = "wic create %s -e core-image-minimal -o %s" % (wks.name, self.resultdir)
+            runCmd(cmd)
+            wksname = os.path.splitext(os.path.basename(wks.name))[0]
+            images = glob(os.path.join(self.resultdir, "%s-*direct" % wksname))
+            self.assertEqual(1, len(images))
+
+        sysroot = get_bb_var('RECIPE_SYSROOT_NATIVE', 'wic-tools')
+        # list partitions
+        result = runCmd("wic ls %s -n %s" % (images[0], sysroot))
+        self.assertEqual(3, len(result.output.split('\n')))
+        self.logger.info("result: %s \n" % result.output)
+
+        # verify partition size with wic
+        res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0]))
+
+        self.logger.info("res: %s \n" % res.output)
+        # parse parted output which looks like this:
+        # BYT;\n
+        # /var/tmp/wic/build/tmpgjzzefdd-202410281021-sda.direct:78569472B:file:4096:4096:gpt::;\n
+        # 1:139264B:39284735B:39145472B:ext4:rofs-a:;\n
+        # 2:39284736B:78430207B:39145472B:ext4:primary:;\n
+        disk_info = res.output.splitlines()[1]
+        # Check sector sizes
+        sector_size_logical = int(disk_info.split(":")[3])
+        sector_size_physical = int(disk_info.split(":")[4])
+        self.assertEqual(wic_sector_size, sector_size_logical, "Logical sector size is not %d." % wic_sector_size)
+        self.assertEqual(wic_sector_size, sector_size_physical, "Physical sector size is not %d." % wic_sector_size)
+
 
 class Wic2(WicTestCase):
 
Signature · 1 line
2.25.1
#3 lore raw
In reply to Vince Chang

These two patches have merged, and unfortunately there are now failures on the autobuilder:
https://valkyrie.yoctoproject.org/#/builders/35/builds/346/steps/14/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/54/builds/325/steps/14/logs/stdio

Alex

Quoted 30 lines · Vince Chang via lists.openembedded.org

v1:
- wic support different sector size
- util-linux: sfdisk sector size improvements (merged)

v2:
- For "wic ls" modify get_partitions() to support WIC_SECTOR_SIZE.
- Add oe-selftest for WIC_SECTOR_SIZE.

Vince Chang (2):
wic: add WIC_SECTOR_SIZE variable
selftest: wic: add test for WIC_SECTOR_SIZE

 meta/lib/oeqa/selftest/cases/wic.py      | 50 +++++++++++++++++++++
 scripts/lib/wic/engine.py                | 18 +++++++-
 scripts/lib/wic/plugins/imager/direct.py | 56 ++++++++++++++----------
 3 files changed, 100 insertions(+), 24 deletions(-)

--
2.25.1

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206417): https://lists.openembedded.org/g/openembedded-core/message/206417
Mute This Topic: https://lists.openembedded.org/mt/109251303/1686489
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#4 lore raw

Hi Alex,

On Wed, 30 Oct 2024 at 21:01, Alexander Kanavin via lists.openembedded.org

Quoted 4 lines

These two patches have merged, and unfortunately there are now failures on the autobuilder:
https://valkyrie.yoctoproject.org/#/builders/35/builds/346/steps/14/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/54/builds/325/steps/14/logs/stdio

It's strange that this line of code fails to execute. It works fine on my build machine. Is there a way to find out which version of parted is being used on the autobuilder?

File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 877, in test_wic_sector_size
res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0]))
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3537958/wic-tmp/tmpr6vrw11i-202410301331-sda.direct unit b p 2>/dev/null' returned non-zero exit status 127:

Thanks,
Vince

#5 lore raw
Quoted 9 lines · Vince Chang via lists.openembedded.org

Hi Alex,

On Wed, 30 Oct 2024 at 21:01, Alexander Kanavin via lists.openembedded.org

Quoted 4 lines

These two patches have merged, and unfortunately there are now failures on the autobuilder:
https://valkyrie.yoctoproject.org/#/builders/35/builds/346/steps/14/logs/stdio
https://valkyrie.yoctoproject.org/#/builders/54/builds/325/steps/14/logs/stdio

Quoted 8 lines

It's strange that this line of code fails to execute. It works fine on my build machine. Is there a way to find out which version of parted is being used on the autobuilder?

File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 877, in test_wic_sector_size
res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0]))
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3537958/wic-tmp/tmpr6vrw11i-202410301331-sda.direct unit b p 2>/dev/null' returned non-zero exit status 127:

perhaps make it
PARTED_SECTOR_SIZE=%d parted -s %s mklabel %s" % (self.sector_size, device, ptable_format), self.native_sysroot)

Quoted 11 lines

Thanks,
Vince

-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#206566): https://lists.openembedded.org/g/openembedded-core/message/206566
Mute This Topic: https://lists.openembedded.org/mt/109308341/1997914
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-

#6 lore raw
Quoted 10 lines · Khem Raj
Quoted 6 lines

File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/selftest/cases/wic.py", line 877, in test_wic_sector_size
res = runCmd("export PARTED_SECTOR_SIZE=%d; parted -m %s unit b p 2>/dev/null" % (wic_sector_size, images[0]))
File "/srv/pokybuild/yocto-worker/oe-selftest-debian/build/meta/lib/oeqa/utils/commands.py", line 212, in runCmd
raise AssertionError("Command '%s' returned non-zero exit status %d:\n%s" % (command, result.status, exc_output))
AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /srv/pokybuild/yocto-worker/oe-selftest-debian/build/build-st-3537958/wic-tmp/tmpr6vrw11i-202410301331-sda.direct unit b p 2>/dev/null' returned non-zero exit status 127:

perhaps make it
PARTED_SECTOR_SIZE=%d parted -s %s mklabel %s" % (self.sector_size, device, ptable_format), self.native_sysroot)

I tried this (dropping 2>/dev/null), and indeed the error is no longer obscured:

AssertionError: Command 'export PARTED_SECTOR_SIZE=4096; parted -m /home/akanavin/poky/build-st/wic-tmp/tmprac0npku-202410311007-sda.direct unit b p' returned non-zero exit status 127:
/bin/sh: 1: parted: not found

So basically the test forgets to adjust the PATH to include wic-tools, like others wic tests (that need to run parted) do. Some build hosts have it, others don't, so the error is intermittent.

Also if you can (in a separate commit) drop all usage of this "2>/dev/null" pattern in wic tests that only harms fail diagnostics, that would be helpful.

Alex

#7 lore raw

Hi Alex,

Quoted 3 lines · Alexander Kanavin

Also if you can (in a separate commit) drop all usage of this "2>/dev/null" pattern in wic tests that only harms fail diagnostics, that would be helpful.

I will send a separate commit to remove "2>/dev/null" in the wic test cases.

Thanks,
Vince

#8 lore raw

Hi Alex,

Quoted 8 lines · Alexander Kanavin
Quoted 4 lines

Also if you can (in a separate commit) drop all usage of this "2>/dev/null" pattern in wic tests that only harms fail diagnostics, that would be helpful.

I will send a separate commit to remove "2>/dev/null" in the wic test cases.

The wic test cases use out.splitlines to get the command result. If we remove "2>/dev/null", it may capture other warning messages, making it unable to get the current result.
To avoid needing additional code to use splitted.index("BYT;") and considering that ‘2>/dev/null’ is also used in ‘debugfs’, I would suggest not removing "2>/dev/null". If the same issue arises again, we can manually remove it to see where the error is. What do you think?

Thanks,
Vince

#9 lore raw
Quoted 10 lines · Vince Chang
Quoted 6 lines
Quoted 3 lines

Also if you can (in a separate commit) drop all usage of this "2>/dev/null" pattern in wic tests that only harms fail diagnostics, that would be helpful.

I will send a separate commit to remove "2>/dev/null" in the wic test cases.

The wic test cases use out.splitlines to get the command result. If we remove "2>/dev/null", it may capture other warning messages, making it unable to get the current result.
To avoid needing additional code to use splitted.index("BYT;") and considering that ‘2>/dev/null’ is also used in ‘debugfs’, I would suggest not removing "2>/dev/null". If the same issue arises again, we can manually remove it to see where the error is. What do you think?

If the issue is intermittent or difficult to reproduce (and in this case it was - I had to spend a couple hours reproducing it for you on a particular build host in yocto CI cluster), it is far better to write the code such that the needed diagnostics are printed right when the error occurs in the first place in CI.

Anyway, implementation runCmd() indeed defaults to stderr being redirected and appended to stdout, so the standard stream and the error stream are mashed together.

To avoid that, you should also set stderr=subprocess.PIPE when calling runCmd(). Check meta/lib/oeqa/selftest/cases/runcmd.py for examples:

def test_output(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True, sync=False) self.assertEqual("stdout\nstderr", result.output) self.assertEqual("", result.error)

def test_output_split(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
stderr=subprocess.PIPE, sync=False)
self.assertEqual("stdout", result.output) self.assertEqual("stderr", result.error)

Alex

#10 lore raw

Hi Alex,

Quoted 13 lines · Alexander Kanavin

Anyway, implementation runCmd() indeed defaults to stderr being redirected and appended to stdout, so the standard stream and the error stream are mashed together.

To avoid that, you should also set stderr=subprocess.PIPE when calling runCmd(). Check meta/lib/oeqa/selftest/cases/runcmd.py for examples:

def test_output(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
sync=False)
self.assertEqual("stdout\nstderr", result.output) self.assertEqual("", result.error)

Quoted 5 lines

def test_output_split(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
stderr=subprocess.PIPE, sync=False)
self.assertEqual("stdout", result.output) self.assertEqual("stderr", result.error)

I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit.
https://lists.openembedded.org/g/openembedded-core/message/206602

Thanks,
Vince

#11 lore raw

Thanks! But you should still fix the original fail I think?

Alex

Quoted 29 lines · Vince Chang

Hi Alex,

Quoted 13 lines · Alexander Kanavin

Anyway, implementation runCmd() indeed defaults to stderr being redirected and appended to stdout, so the standard stream and the error stream are mashed together.

To avoid that, you should also set stderr=subprocess.PIPE when calling runCmd(). Check meta/lib/oeqa/selftest/cases/runcmd.py for examples:

def test_output(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
sync=False)
self.assertEqual("stdout\nstderr", result.output) self.assertEqual("", result.error)

Quoted 5 lines

def test_output_split(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
stderr=subprocess.PIPE, sync=False)
self.assertEqual("stdout", result.output) self.assertEqual("stderr", result.error)

I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit.
https://lists.openembedded.org/g/openembedded-core/message/206602

Thanks,
Vince

#12 lore raw

Hi Alex,

Yes, I have submitted a commit to add wic-tools into the PATH.
https://lists.openembedded.org/g/openembedded-core/message/206607

Thanks,
Vince

From: Alexander Kanavin <[email protected]>
Sent: Friday, November 1, 2024 3:39 PM
To: Vince Chang <[email protected]>
Cc: Richard Purdie <[email protected]>; [email protected]; Jamin Lin <[email protected]>; Troy Lee <[email protected]>; Khem Raj <[email protected]>
Subject: Re: [OE-core] [PATCH v2 0/2] wic support different sector size

Thanks! But you should still fix the original fail I think?

Alex

On Fri 1. Nov 2024 at 8.37, Vince Chang <[email protected]<mailto:[email protected]>> wrote:
Hi Alex,

Quoted 13 lines · Alexander Kanavin

Anyway, implementation runCmd() indeed defaults to stderr being redirected and appended to stdout, so the standard stream and the error stream are mashed together.

To avoid that, you should also set stderr=subprocess.PIPE when calling runCmd(). Check meta/lib/oeqa/selftest/cases/runcmd.py for examples:

def test_output(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
sync=False)
self.assertEqual("stdout\nstderr", result.output) self.assertEqual("", result.error)

Quoted 5 lines

def test_output_split(self):
result = runCmd("echo stdout; echo stderr >&2", shell=True,
stderr=subprocess.PIPE, sync=False)
self.assertEqual("stdout", result.output) self.assertEqual("stderr", result.error)

I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit.
https://lists.openembedded.org/g/openembedded-core/message/206602

Thanks,
Vince

#13 lore raw
In reply to Vince Chang
Quoted 2 lines · Vince Chang

I didn't realize that stderr=subprocess.PIPE could be used. I've submitted a separate commit.
https://lists.openembedded.org/g/openembedded-core/message/206602

I just added one more fix that makes runCmd actually print stderr when it's split from stdout and an error occurs:
https://lists.openembedded.org/g/openembedded-core/message/206610
and with that and your two fixes everything should be working correctly.

Alex