breakpseudo: Add demo of breaking pseudo with symlink creation
I think I can conceptually see why pseudo is struggling with link/renames.
The reproducer for the problems involves overwriting a given target multiple times in parallel, i.e. ln or mv commands in parallel on the same target file. At the end there will be one "winner" but while in progress, you can't predict which file will eventually win.
Thankfully such scenarios are rare in our builds. They imply build races and therefore tend to get fixed. That said, we do really want pseudo to be robust against such scenarios.
To understand what is going on you have to understand a bit about pseudo's internal state. When the system calls "mv A B", we have to contact pseudo and:
a) mark both A and B as "might be deleted" in the database.
b) call the real system mv/rename function call
c) see what happened, did it succeed?
d) if so, tell pseudo B was deleted and A is no longer pending deletion and rename it in the database
e) if not, mark A and B as no longer pending deletion
We need to know about both pending states since something might try and access A or B during these steps and whilst that will give us a hint of what is happening, we don't know until the process performing the op reports back.
We need to have these pending flags as we have to preserve the extra info against a given path/inode and ultimately we need to know which permissions/ownership to give the final file.
The challenge is this "pending deletion" flag in pseudo. If the operation above completes, it clears the flags against B, even if multiple processes have a pending deletion against a path. I think this is where the current code breaks down, only one client has a pending flag and in reality, multiple processes can.
The second issue is that new files clear any pending deletions against a path. We have no guarantee that the new file hasn't actually already been overwritten.
I can see two ways to resolve this. The first would be to separate deletion out to a separate table, so multiple clients can have "deletion" pending against a given path. Deletion is keyed to the clients already (by client ID).
The second might be to make some ops block on the server side waiting for a second operation to follow up from the same client which would allow the situation to resolve before anything else continues.
I'm writing this down so we at least have some record of what I think is happening, while I sort of understand it, or at least think I do.
Cheers,
Richard
Thank-you so much for taking the time to explain that.
I had no concept of how complicated it was, truly I am a beginner.
It reminds me of this aphorism: oh what a tangled web we weave, when first we set out to deceive.
My mind is blown! I will ponder on it.
Maybe unprivileged namespaces with bind mounts and user ID mapping is the future, but I think it won't be ready for a few years.
Sam
Quoted 70 lines · On Fri, 4 Apr 2025, 18:04 Richard Purdie via lists.openembedded.org…
I think I can conceptually see why pseudo is struggling with link/renames.
The reproducer for the problems involves overwriting a given target multiple times in parallel, i.e. ln or mv commands in parallel on the same target file. At the end there will be one "winner" but while in progress, you can't predict which file will eventually win.
Thankfully such scenarios are rare in our builds. They imply build races and therefore tend to get fixed. That said, we do really want pseudo to be robust against such scenarios.
To understand what is going on you have to understand a bit about pseudo's internal state. When the system calls "mv A B", we have to contact pseudo and:
a) mark both A and B as "might be deleted" in the database.
b) call the real system mv/rename function call
c) see what happened, did it succeed?
d) if so, tell pseudo B was deleted and A is no longer pending deletion and rename it in the database
e) if not, mark A and B as no longer pending deletionWe need to know about both pending states since something might try and access A or B during these steps and whilst that will give us a hint of what is happening, we don't know until the process performing the op reports back.
We need to have these pending flags as we have to preserve the extra info against a given path/inode and ultimately we need to know which permissions/ownership to give the final file.
The challenge is this "pending deletion" flag in pseudo. If the operation above completes, it clears the flags against B, even if multiple processes have a pending deletion against a path. I think this is where the current code breaks down, only one client has a pending flag and in reality, multiple processes can.
The second issue is that new files clear any pending deletions against a path. We have no guarantee that the new file hasn't actually already been overwritten.
I can see two ways to resolve this. The first would be to separate deletion out to a separate table, so multiple clients can have "deletion" pending against a given path. Deletion is keyed to the clients already (by client ID).
The second might be to make some ops block on the server side waiting for a second operation to follow up from the same client which would allow the situation to resolve before anything else continues.
I'm writing this down so we at least have some record of what I think is happening, while I sort of understand it, or at least think I do.
Cheers,
Richard
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group. View/Reply Online (#214364):
https://lists.openembedded.org/g/openembedded-core/message/214364
Mute This Topic: https://lists.openembedded.org/mt/111788756/7497305
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [ [email protected]]
-=-=-=-=-=-=-=-=-=-=-=-