TMPDIR, bitbake v posix
A message that is not in the archive (sent off-list or not yet received): [email protected]
Quoted 29 lines · Rasmus Villemoes via lists.openembedded.org
Hi
When some task uses mktemp(1) or similar functionality (e.g. the python tempfile module) to create a temporary file or dir, those end up being created in the global /tmp directory. That's obviously not ideal, as those might be left behind in case of an unclean shutdown or if the script is not properly written, and won't be cleaned by
do_cleanorRM_WORK. Also, /tmp is often a tmpfs instance, while some tasks might want to create rather large temporary files.Most such utilities/interfaces respect the TMPDIR environment variable, if set. However, since that name is used by bitbake for a rather central concept, one cannot simply 'export TMPDIR', as one doesn't want that top-level build dir polluted by random build files, and it still wouldn't be removed by
do_cleanorRM_WORK. Ideally, all tasks would have a TMPDIR _environment_ variable set to something below ${WORKDIR}, maybe even something unique per task (say ${T}/${taskname}.d). But there's no simple way to do that that doesn't conflict with bitbake's use of the TMPDIR name.Can we do anything about this?
(1) Do nothing, live with this limitation.
(2) Special-case TMPDIR, so that all tasks are created with a proper value of TMPDIR in the environment, not derived from the bitbake variable of the same name. Could be settable with some new
TASK_TMPDIRor do_foo[tmpdir] flag. Somewhat hacky, though.
Quoted 5 lines
(3) Bite the bullet and plan for renaming TMPDIR -> BBTMPDIR, so that the TMPDIR name becomes available for use with its standard POSIX meaning. Obviously a huge thing to do, but hopefully most direct TMPDIR references are in core or bitbake and used for deriving other standard variables.
I tried doing TMPDIR -> BB_TMPDIR in bitbake and most places in core (there are some places which use TMPDIR in the posix sense), and it seems to build a core-image-minimal without problems.
So it's not impossible, and one could prepare for the transition by eliminating quite a few direct TMPDIR references using already existing or new derived variables (e.g. BASE_SHAREDWORKDIR = ${TMPDIR}/work-shared), making the number of places that would need the rename rather small.
I'd be happy to send such preparatory patches, but it would be pointless if the community doesn't deem this to be a problem worth fixing.
Rasmus
Quoted 41 lines · Rasmus Villemoes via lists.openembedded.org
Quoted 29 lines · Rasmus Villemoes via lists.openembedded.org
Hi
When some task uses mktemp(1) or similar functionality (e.g. the python tempfile module) to create a temporary file or dir, those end up being created in the global /tmp directory. That's obviously not ideal, as those might be left behind in case of an unclean shutdown or if the script is not properly written, and won't be cleaned by
do_cleanorRM_WORK. Also, /tmp is often a tmpfs instance, while some tasks might want to create rather large temporary files.Most such utilities/interfaces respect the TMPDIR environment variable, if set. However, since that name is used by bitbake for a rather central concept, one cannot simply 'export TMPDIR', as one doesn't want that top-level build dir polluted by random build files, and it still wouldn't be removed by
do_cleanorRM_WORK. Ideally, all tasks would have a TMPDIR _environment_ variable set to something below ${WORKDIR}, maybe even something unique per task (say ${T}/${taskname}.d). But there's no simple way to do that that doesn't conflict with bitbake's use of the TMPDIR name.Can we do anything about this?
(1) Do nothing, live with this limitation.
(2) Special-case TMPDIR, so that all tasks are created with a proper value of TMPDIR in the environment, not derived from the bitbake variable of the same name. Could be settable with some new
TASK_TMPDIRor do_foo[tmpdir] flag. Somewhat hacky, though.Quoted 5 lines
(3) Bite the bullet and plan for renaming TMPDIR -> BBTMPDIR, so that the TMPDIR name becomes available for use with its standard POSIX meaning. Obviously a huge thing to do, but hopefully most direct TMPDIR references are in core or bitbake and used for deriving other standard variables.
I tried doing TMPDIR ->
BB_TMPDIRin bitbake and most places in core (there are some places which use TMPDIR in the posix sense), and it seems to build a core-image-minimal without problems.
Quoted 8 lines
So it's not impossible, and one could prepare for the transition by eliminating quite a few direct TMPDIR references using already existing or new derived variables (e.g.
BASE_SHAREDWORKDIR= ${TMPDIR}/work-shared), making the number of places that would need the rename rather small.I'd be happy to send such preparatory patches, but it would be pointless if the community doesn't deem this to be a problem worth fixing.
Hi Rasmus,
I think it's worth trying to disambiguate between the two uses of TMPDIR if we can, but it's not a change we can take this close to the next LTS release. I recommend sending an RFC patch for people to ealuate, after the 6.0 "wrynose" release.
Thanks,
Signature · 1 line
Paul Barker
Quoted 41 lines · Rasmus Villemoes
Quoted 29 lines · Rasmus Villemoes via lists.openembedded.org
Hi
When some task uses mktemp(1) or similar functionality (e.g. the python tempfile module) to create a temporary file or dir, those end up being created in the global /tmp directory. That's obviously not ideal, as those might be left behind in case of an unclean shutdown or if the script is not properly written, and won't be cleaned by
do_cleanorRM_WORK. Also, /tmp is often a tmpfs instance, while some tasks might want to create rather large temporary files.Most such utilities/interfaces respect the TMPDIR environment variable, if set. However, since that name is used by bitbake for a rather central concept, one cannot simply 'export TMPDIR', as one doesn't want that top-level build dir polluted by random build files, and it still wouldn't be removed by
do_cleanorRM_WORK. Ideally, all tasks would have a TMPDIR _environment_ variable set to something below ${WORKDIR}, maybe even something unique per task (say ${T}/${taskname}.d). But there's no simple way to do that that doesn't conflict with bitbake's use of the TMPDIR name.Can we do anything about this?
(1) Do nothing, live with this limitation.
(2) Special-case TMPDIR, so that all tasks are created with a proper value of TMPDIR in the environment, not derived from the bitbake variable of the same name. Could be settable with some new
TASK_TMPDIRor do_foo[tmpdir] flag. Somewhat hacky, though.Quoted 5 lines
(3) Bite the bullet and plan for renaming TMPDIR -> BBTMPDIR, so that the TMPDIR name becomes available for use with its standard POSIX meaning. Obviously a huge thing to do, but hopefully most direct TMPDIR references are in core or bitbake and used for deriving other standard variables.
I tried doing TMPDIR ->
BB_TMPDIRin bitbake and most places in core (there are some places which use TMPDIR in the posix sense), and it seems to build a core-image-minimal without problems.
Quoted 8 lines
So it's not impossible, and one could prepare for the transition by eliminating quite a few direct TMPDIR references using already existing or new derived variables (e.g.
BASE_SHAREDWORKDIR= ${TMPDIR}/work-shared), making the number of places that would need the rename rather small.I'd be happy to send such preparatory patches, but it would be pointless if the community doesn't deem this to be a problem worth fixing.
I'm not against the idea however right now, we have several other ongoing issues and the feature freeze for the LTS looming one week away. I'm struggling to get the intermittent failures in our automated testing under control and this particular issue doesn't seem as pressing as others. I feel like I'm drowning with different problems and adding the fall out from this change isn't particularly appealing to me.
The change will potentially have huge impact as you mention and it needs to be handled correctly.
I've thought "tmp" was a bad choice of name so if we do this, we should take the opportunity to find a better one. The first step would therefore probably be to get agreement on what the new name would be.
As well as getting the general community buy in on the idea, we'd also need to agree when we'd try and make the change.
Cheers,
Richard