Skip to contentSkip to conversation

Where are we at?

DISCUSSION Richard Purdie 5 participants 9 messages May 9, 2024 – May 15, 2024 4 messages not in the archive

lore
Share & tools
lore
#0 lore raw

I think I need to write down a bit of a summary status on where we (and
I) am at with things.

The good news is that scarthgap is released. On the downside, we messed up the public hash server url in local.conf.sample so we have a point release in progress to fix that.

The Sovereign Tech Fund (STF) work is drawing to a close now, some tweaks are needed on some patch series to get them over the line and able to merge but the basic pieces are there. We have the PoC for bitbake-setup too. I have not had the time to look at it in detail yet (see below for an idea of why). That work brings us many steps forward in many key areas of the project and takes a number of weights off my mind. It adds some new ones too but change is at least refreshing.

It isn't all rosy, in particular the autobuilder is struggling. The quality of builds for scarthgap isn't what I'd have liked and Steve is already abandoning rc builds due to intermittent failures. We're seeing the same failures with master.

In particular:

A rust reproducibility issue lurking somewhere:
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/8318
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/8906/steps/17/logs/stdio
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15185
[both master and scarthgap]

Intermittent do_compile failure in libportal:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4760/steps/12/logs/stdio

debuginfod keeps breaking:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3299/steps/14/logs/stdio
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14937

qemuppc with intermittent issues:
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/8906/steps/17/logs/stdio

qemuarm eSDK intermittent failure:
https://autobuilder.yoctoproject.org/typhoon/#/builders/53/builds/8946/steps/15/logs/stdio
(network glitch or a problem with git/curl/certs in the tools?)

We did have an intermittent python3 test hang. We've taken the decision just to disable that test for now as we don't have the time to debug it.

The weird hang in the 6.6 kernel in early boot also remains. Paul Gortmaker made some progress in debugging it but I haven't found any time to help/support him.

There is some kind of issue where the longer the arm servers have uptime, the more broken the builds get too. It is hard to be more specific, someone needs to do some analysis of the failure and try and find the pattern.

We then get to the non oe-core failures that seem to be piling up:

meta-oe source mirroring giving warnings:
https://autobuilder.yoctoproject.org/typhoon/#/builders/156/builds/353/steps/12/logs/warnings

meta-oe source mirroring fetch failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/156/builds/353/steps/16/logs/stdio

kirkstone meta-oe metrics failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/138/builds/1612/steps/16/logs/stdio

meta-ti nightly layer check failures:
https://autobuilder.yoctoproject.org/typhoon/#/builders/121/builds/2064/steps/19/logs/stdio

Yes, some of these need reporting to the right people, e.g. meta-ti. *Someone* actually needs to do it. If someone doesn't do it, someone needs to notice and find/remind someone to do it. We expanded testing to meta-oe on the basiss there would be help in addressing the issues but few people seem to be paying attention :(.

Meanwhile, we have a pile of gcc 14 fixes piling up and pressure to get gcc 14 in so we can build a new uninative to fix things for people on Fedora 40 and arch. There are simple things others could do to help like replying to patches about missing patch information.

The current autobuilder hardware is failing. We have a plan for a new setup however that new setup has issues that need debugging. The one resource I can direct to help with that is me, everyone else just runs away from intermittent issues or says it is too hard or they don't have access/experience/whatever.

We have challenges in the CVE space, I'm hoping Marta has some patches coming which can help there soon. The open letter idea is all well and good but it needs socialising with people outside the project.

There are also some bigger changes I think the project needs to make, WORKDIR changes are one example, we also need to revisit the layer dependency situation and "Yocto Project Compatible" needs work too, amongst many other things.

Personally, I'm now really struggling. The intensity of the last few months has taken a toll on me and I simply can't switch into handling all of the above, I'm running on empty. This means I need a break, everyone knows that and agrees. The twist is that yes, everyone encourages me to do that, as long as I fix the issues on the current autobuilder, the issues on the new one, review and merge patches, make architecture/development changes at the right point in the release cycle, ensure proposals made for key changes are well thought out and don't upset anyone and so on.

I'm sharing this not as a complaint, but to try and show people that if their patches aren't merging, it isn't that I/we don't care, it is just that there simply aren't enough hours in the day and I'm at breaking point if I don't let some things slow down.

One of the key issues is that I act like a traffic director in the middle of all this. Nobody else is willing to step into all those different streams and help try and tame/direct them. The downside to that is that if I stop or slow down, things do break.

Obviously, help in any of the most regular autobuilder failures would be much appreciated.

Cheers,

Richard

#1 lore raw
Quoted 1 line

Personally, I'm now really struggling. The intensity of the last few months has taken a toll on me and I simply can't switch into handling all of the above, I'm running on empty. This means I need a break, everyone knows that and agrees.

...

Quoted 1 line

One of the key issues is that I act like a traffic director in the middle of all this. Nobody else is willing to step into all those different streams and help try and tame/direct them. The downside to that is that if I stop or slow down, things do break.

You are personally responsible for your personal wellbeing. There's only one you. If you burn out, it can mean even bigger things - both to you personally, but also to the overall community as recovering from a burnout will take much longer than a short break. So, you should take the break.

Also, as is with things & the pain being in the right place - the people in power (i.e. people who can actually fund things) - are not going to ramp up the funding, unless they start feeling the pain. This might be one way to getting that done. Or the more optimistic outlook being - some people will step up & start helping more.

Take the break.

Best Regards,

Janne Kiiskilä

#2 lore raw
Quoted 1 line · Janne Kiiskila via lists.yoctoproject.org

Also, as is with things & the pain being in the right place - the people in power (i.e. people who can actually fund things) - are not going to ramp up the funding, unless they start feeling the pain. This might be one way to getting that done. Or the more optimistic outlook being - some people will step up & start helping more.

It would take something particularly drastic for the pain to even register slightly. Such as:
- RP takes a leave of absence for an entire release cycle
- there's no release in the autumn
- all patches are rejected with that message

I strongly suspect this is both not feasible and at the same time not enough, as the people in power will be able to ship their products based on existing LTS versions, pile up patches in private trees and derive an income from that for quite some time before their own technical debt starts to sting.

Alex

#3 lore raw
In reply to Richard Purdie
Quoted 5 lines · Richard Purdie via lists.openembedded.org

Intermittent do_compile failure in libportal:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4760/steps/12/logs/stdio

Root caused this, fix incoming.

Quoted 3 lines

debuginfod keeps breaking:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3299/steps/14/logs/stdio
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14937

I realised that I didn’t do what I thought I had previously so understand this now, hopefully fixable tomorrow.

Ross

#4 lore raw

We all here are trying to be somewhere... With Yocto!

Not an easy task. Not at all.

We believe in Fate. In Hope. In Yocto.

Hope dies last. Doesn't it?

Zoran Stojsavljevic
_______

Quoted 18 lines · Ross Burton via lists.yoctoproject.org
Quoted 6 lines · Richard Purdie via lists.openembedded.org

Intermittent do_compile failure in libportal:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4760/steps/12/logs/stdio

Root caused this, fix incoming.

Quoted 3 lines

debuginfod keeps breaking:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3299/steps/14/logs/stdio
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14937

I realised that I didn’t do what I thought I had previously so understand this now, hopefully fixable tomorrow.

Ross

#5 lore raw
In reply to Ross Burton
Quoted 8 lines · Ross Burton via lists.openembedded.org
Quoted 6 lines · Richard Purdie via lists.openembedded.org

Intermittent do_compile failure in libportal:
https://autobuilder.yoctoproject.org/typhoon/#/builders/117/builds/4760/steps/12/logs/stdio

Root caused this, fix incoming.

Patch on the list.

Quoted 5 lines
Quoted 3 lines

debuginfod keeps breaking:
https://autobuilder.yoctoproject.org/typhoon/#/builders/127/builds/3299/steps/14/logs/stdio
https://bugzilla.yoctoproject.org/show_bug.cgi?id=14937

I realised that I didn’t do what I thought I had previously so understand this now, hopefully fixable tomorrow.

I have a fix, it needs cleaning up… I shall discuss this with RP next week.

Ross

#6 lore raw
In reply to Richard Purdie
Quoted 5 lines · Richard Purdie via lists.yoctoproject.org

A rust reproducibility issue lurking somewhere:
https://autobuilder.yoctoproject.org/typhoon/#/builders/97/builds/8318
https://autobuilder.yoctoproject.org/typhoon/#/builders/76/builds/8906/steps/17/logs/stdio
https://bugzilla.yoctoproject.org/show_bug.cgi?id=15185
[both master and scarthgap]

I have reproduced this. The filenames differ depending on whether the build host is arm or x86. If packagedata sstate comes from arm and package sstate from x86, the reported error will occur.

I'm going to look into how the hash part is formed and how the build host architecture leaks into it (on monday).

Alex

#10 lore raw

build host architecture leaking into target artefacts - new twist on reproducibility

In reply to a message not in the archive
Quoted 6 lines · Richard Purdie

Run a build of "MACHINE=qemux86-64 bitbake rust" on both an x86_64 build server and an aarch64 build server, then compare the output.

It shouldn't matter which server architecture they're built on, the files should have the same name. Apparently they do not and this is the issue we're hitting.

The incomplete patch for this issue is here on top:
https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/package-version-updates

Unfortunately, that is not all, as file names and file sizes are now the same in libstd-rs, but content still isn't for some of them (different hashes reported by sha256sum). And in target rust both names and content differs.

I'm out of time today to look further into it, but you are welcome to pick it up.

To catch such issues closer to where they first occur, we need to run reproducible-armhost (that would compare sstate coming primarily from x86 hosts against a pristine build on an arm host). I already found out that plain llvm also leaks host architecture into its installation.

Alex

#12 lore raw

Re: [OE-core] build host architecture leaking into target artefacts - new twist on reproducibility

In reply to a message not in the archive
Quoted 10 lines · Alexander Kanavin via lists.openembedded.org

The incomplete patch for this issue is here on top:
https://git.yoctoproject.org/poky-contrib/log/?h=akanavin/package-version-updates

Unfortunately, that is not all, as file names and file sizes are now the same in libstd-rs, but content still isn't for some of them (different hashes reported by sha256sum). And in target rust both names and content differs.

I'm out of time today to look further into it, but you are welcome to pick it up.

This is the worst rabbit hole of this year so far, made worse by having to confirm every point with a 30 minute build or test execution.

The good news is I have a fix. The bad news is that the fix breaks rust selftest, and fixing the rust selftest breaks rust reproducibility.

Let's review:

- pre-built cargo from upstream snapshots writes output of 'rust -vV' into hashes used to form filenames of build artefacts, and that output includes the build host architecture.

- we can patch this out in cargo-native and I have the patch.

- to use cargo-native we need to tweak rust's build config so that it uses that and not cargo from upstream binary snapshot for bootstrapping. At the same time we need to tweak location of rustc in the same config, as they would otherwise mismatch and build will break. I have the patch for that too, and I confirmed the target filenames start to match with it.

- this breaks the rust selftest, as it is now looking into native sysroot instead of rust-snapshot/ and can no longer find rustdoc there that it needs for the tests

- rustdoc is present in binary rust snapshots but not in rust-native installations. Why? Because it used to cause reproducibility problems (and still does), and the purported reproducibility fix simply prevented rustdoc from being installed:
https://git.yoctoproject.org/poky/commit/?id=321aebfa281bd28e368c684ece57867f6bd0cbe7
(the culprit is the rust-demangler line which replaces rust's default installation set (that includes rustdoc) with just that utility:
https://github.com/rust-lang/rust/blob/master/config.example.toml#L320
)

- I don't know if pgo tweak in the above indeed makes rustdoc reproducible, as that tweak was later reverted due to causing issues with chromium:
https://git.yoctoproject.org/poky/commit/?id=694b85a217209787fe8c503a4b697dc6c2bfbe84

I would strongly suggest that fixing random autobuilder breakage is far more important than having a passing test suite for a specific component or having that component pass reproducibility, so the proposal is:

- merge my fixes
- disable rust selftest
- re-enable rustdoc installation, as that is the default and is expected in typical rust setups
- put rust back into reproducibility exclusion due to rustdoc
- update rust to latest upstream release (we're now three releases behind)
- and only then look again, properly, into re-enabling both reproducibility and selftests, keeping all of the above in mind

Cheers,
Alex