Uprev to pseudo 1.2
cover Uprev to pseudo 1.2
Uprev to pseudo 1.2.
This was heavily tested by building oe-core with numerous image configurations. Each configuration generated the same image before and after the change from the current to new version of pseudo.
The primary purpose of this change is to enable the PSEUDO_UNLOAD functionality that may be used for performance enhancements in future versions of oe-core.
The following changes since commit 37579d7d74d127c90c1e078d05c5bf4ba0b3f755:
meta: glib-2.0: don't apply qsort_r test removable patch for native version (2011-11-02 09:08:21 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib mhatle/pseudo
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/pseudo
Mark Hatle (1):
pseudo: Uprev pseudo to version 1.2
.../conf/distro/include/distro_tracking_fields.inc | 6 +-
.../pseudo/pseudo/realpath_fix.patch | 165 --------------------
.../pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} | 7 +-
meta/recipes-devtools/pseudo/pseudo_git.bb | 6 +-
4 files changed, 9 insertions(+), 175 deletions(-)
delete mode 100644 meta/recipes-devtools/pseudo/pseudo/realpath_fix.patch
rename meta/recipes-devtools/pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} (48%)Signature · 1 line
1.7.3.4
PATCH 1/1 pseudo: Uprev pseudo to version 1.2
This adds a new feature, PSEUDO_UNLOAD, which can be used to eliminate overhead of LD_PRELOAD when no longer necessary.
Also the, clone(2), support on Linux has been updated to resolve some potential defects in the previous implementation.
---
.../conf/distro/include/distro_tracking_fields.inc | 6 +-
.../pseudo/pseudo/realpath_fix.patch | 165 --------------------
.../pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} | 7 +-
meta/recipes-devtools/pseudo/pseudo_git.bb | 6 +-
4 files changed, 9 insertions(+), 175 deletions(-)
delete mode 100644 meta/recipes-devtools/pseudo/pseudo/realpath_fix.patch
rename meta/recipes-devtools/pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} (48%)meta/conf/distro/include/distro_tracking_fields.inc +3 -3
@@ -5944,11 +5944,11 @@ RECIPE_LAST_UPDATE_pn-gettext = "May 24, 2011" RECIPE_MANUAL_CHECK_DATE_pn-gettext = "Sep 29, 2011" RECIPE_STATUS_pn-pseudo = "green" -RECIPE_LATEST_VERSION_pn-pseudo = "1.1.1" -RECIPE_LAST_UPDATE_pn-pseudo = "Jun 06, 2011" +RECIPE_LATEST_VERSION_pn-pseudo = "1.2" +RECIPE_LAST_UPDATE_pn-pseudo = "Nov 02, 2011" RECIPE_MAINTAINER_pn-pseudo = "Mark Hatle <[email protected]>" RECIPE_COMMENTS_pn-pseudo = "Yocto Project maintained" -RECIPE_MANUAL_CHECK_DATE_pn-pseudo = "Jun 06, 2011" +RECIPE_MANUAL_CHECK_DATE_pn-pseudo = "Nov 02, 2011" DISTRO_PN_ALIAS_pn-pseudo = "Windriver" DISTRO_PN_ALIAS_pn-rt-tests = "Debian=rt-tests Ubuntu=rt-tests"
meta/recipes-devtools/pseudo/pseudo/realpath_fix.patch deleted +0 -165
165 lines not on this page · raw message
meta/recipes-devtools/pseudo/pseudo_1.2.bb renamed from meta/recipes-devtools/pseudo/pseudo_1.1.1.bb +3 -4
@@ -1,11 +1,10 @@ require pseudo.inc -PR = "r2" +PR = "r3" SRC_URI = "http://www.yoctoproject.org/downloads/${BPN}/${BPN}-${PV}.tar.bz2 \ file://oe-config.patch \ - file://realpath_fix.patch \ file://static_sqlite.patch" -SRC_URI[md5sum] = "dd59766c17e199fe6144fce8a2c67802" -SRC_URI[sha256sum] = "c697f643577d661c3ce826504b9dcd11fa98e78a5d10e3c83931da8942f6bfad" +SRC_URI[md5sum] = "a2819084bab7e991f06626d02cf55048" +SRC_URI[sha256sum] = "4749a22df687f44d24c26e97170d4781a1bd52d5ee092364a40877e4d96ff058"
meta/recipes-devtools/pseudo/pseudo_git.bb +3 -3
@@ -1,8 +1,8 @@ require pseudo.inc -SRCREV = "c2f7c5ad8ef0f9c94a2a8382c109c8c6e16c8b18" -PV = "1.1.1+git${SRCPV}" -PR = "r19" +SRCREV = "17c2233f93692f79684792750001ee6d13e03925" +PV = "1.2+git${SRCPV}" +PR = "r20" DEFAULT_PREFERENCE = "-1"
Signature · 1 line
1.7.3.4
Quoted 10 lines · Mark Hatle
Uprev to pseudo 1.2.
This was heavily tested by building oe-core with numerous image configurations. Each configuration generated the same image before and after the change from the current to new version of pseudo.
The primary purpose of this change is to enable the
PSEUDO_UNLOADfunctionality that may be used for performance enhancements in future versions of oe-core.
Quoted 7 lines
The following changes since commit
37579d7d74d127c90c1e078d05c5bf4ba0b3f755:meta: glib-2.0: don't apply
qsort_rtest removable patch for native version (2011-11-02 09:08:21 +0000)are available in the git repository at:
git://git.pokylinux.org/poky-contribmhatle/pseudo
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/pseudo
Is there branch with pseudo-1.2 for oe-core?
I would like to try it because we have strange errors when libstc++ isn't found when building with pseudo, but same binaries are working fine without pseudo (seems related to /etc/ld.so.cache being ignored sometimes from pseudo env).
And while I was trying to debug it I've noticed that files.db and pseudo.log is quite big.
-rw-r--r-- 1 bitbake bitbake 240M Aug 15 13:49 files.db -rw-r--r-- 1 bitbake bitbake 91K Aug 15 13:50 logs.db -rw-r--r-- 1 bitbake bitbake 305M Aug 15 13:49 pseudo.log
with pseudo.log full of errors like this
pseudo: path mismatch [12 links]: ino 39721500 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor' req
'/OE/shr-core/tmp/work/om_gta02-oe-linux-gnueabi/task-shr-feed-1.0-r96'.
Cheers,
Quoted 20 lines
Mark Hatle (1):
pseudo: Uprev pseudo to version 1.2.../conf/distro/include/distro_tracking_fields.inc | 6 +- .../pseudo/pseudo/realpath_fix.patch | 165 -------------------- .../pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} | 7 +- meta/recipes-devtools/pseudo/pseudo_git.bb | 6 +- 4 files changed, 9 insertions(+), 175 deletions(-) delete mode 100644 meta/recipes-devtools/pseudo/pseudo/realpath_fix.patch rename meta/recipes-devtools/pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} (48%)Signature · 7 lines
1.7.3.4 _______________________________________________ Openembedded-core mailing list [email protected] http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 13 lines · Martin Jansa
Quoted 12 lines · Mark Hatle
Uprev to pseudo 1.2.
This was heavily tested by building oe-core with numerous image configurations. Each configuration generated the same image before and after the change from the current to new version of pseudo.
The primary purpose of this change is to enable the
PSEUDO_UNLOADfunctionality that may be used for performance enhancements in future versions of oe-core.The following changes since commit
37579d7d74d127c90c1e078d05c5bf4ba0b3f755:
Quoted 7 lines
Quoted 5 lines
meta: glib-2.0: don't apply
qsort_rtest removable patch for native version (2011-11-02 09:08:21 +0000)are available in the git repository at:
git://git.pokylinux.org/poky-contribmhatle/pseudo
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/pseudoIs there branch with pseudo-1.2 for oe-core?
^^^^^^^^^^^^^^^^^^^^^^^^
git://git.pokylinux.org/poky-contrib mhatle/pseudo
Grab that -- or simply grab the top commit and add it to your oe-core checkout.
Review
Quoted 10 lines
I would like to try it because we have strange errors when libstc++ isn't found when building with pseudo, but same binaries are working fine without pseudo (seems related to
/etc/ld.so.cachebeing ignored sometimes from pseudo env).And while I was trying to debug it I've noticed that files.db and pseudo.log is quite big.
(quoted diff) +0
-3-rw-r--r-- 1 bitbake bitbake 240M Aug 15 13:49 files.db -rw-r--r-- 1 bitbake bitbake 91K Aug 15 13:50 logs.db -rw-r--r-- 1 bitbake bitbake 305M Aug 15 13:49 pseudo.log
The files.db is based on the number of files accessed during the time pseudo is running.
logs.db and pseudo.log are based on the debug settings of pseudo. If you don't have debugging enabled (you would have to do this manually) and the logs are that big, something is going seriously wrong with the path/inode translation...
It's fairly normal to have a set of logs that are in the 10-20k range for
complex packages.. (These are documenting cases where pseudo has to guess about inode to path mapping..)
Quoted 5 lines
with pseudo.log full of errors like this
pseudo: path mismatch [12 links]: ino 39721500 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor' req
'/OE/shr-core/tmp/work/om_gta02-oe-linux-gnueabi/task-shr-feed-1.0-r96'.
Are you building on a NFS filesystem? NFS (and a few other filesystems) have a habit of remapping inodes out from under the filesystem. pseudo uses the inodes as a validation that the file hasn't changed. If these inodes are changing, pseudo will do it's best to keep things in sync, but there is a chance it won't always succeed.
(This is done because someone could access a file in pseudo control that was created outside.. so pseudo could only capture the inode.. then if in a future action the file is accessed the inode/path can be checked. If the inode matches it assumes it's the same file and now a path name exists.. if someone makes changes outside of pseudo control a similar check happens.. this time it'll see the path is the same but inode has changed.. so it guesses the file was modified
and I believe resets everything based on the new inode. Each of these will
generate a warning in the logs.)
--Mark
Quoted 30 lines
Cheers,
Quoted 4 lines
Mark Hatle (1):
pseudo: Uprev pseudo to version 1.2Quoted 16 lines
.../conf/distro/include/distro_tracking_fields.inc | 6 +- .../pseudo/pseudo/realpath_fix.patch | 165 -------------------- .../pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} | 7 +- meta/recipes-devtools/pseudo/pseudo_git.bb | 6 +- 4 files changed, 9 insertions(+), 175 deletions(-) delete mode 100644 meta/recipes-devtools/pseudo/pseudo/realpath_fix.patch rename meta/recipes-devtools/pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} (48%)Signature · 7 lines
1.7.3.4 _______________________________________________ Openembedded-core mailing list [email protected] http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Martin Jansa
Quoted 17 lines · Mark Hatle
Uprev to pseudo 1.2.
This was heavily tested by building oe-core with numerous image configurations. Each configuration generated the same image before and after the change from the current to new version of pseudo.
The primary purpose of this change is to enable the
PSEUDO_UNLOADfunctionality that may be used for performance enhancements in future versions of oe-core.The following changes since commit
37579d7d74d127c90c1e078d05c5bf4ba0b3f755:meta: glib-2.0: don't apply
qsort_rtest removable patch for native version (2011-11-02 09:08:21 +0000)are available in the git repository at:
git://git.pokylinux.org/poky-contribmhatle/pseudo
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/pseudo
Quoted 7 lines
Quoted 1 line
Is there branch with pseudo-1.2 for oe-core?
^^^^^^^^^^^^^^^^^^^^^^^^
git://git.pokylinux.org/poky-contribmhatle/pseudoGrab that -- or simply grab the top commit and add it to your oe-core checkout.
Yes, but git cherry-pick would be faster and adding poky-contrib as another git remote is not very efficient as it doesn't share any git objects with oe-core repo so it checkouts "whole poky" again :/.
So I'll use 2nd easiest option "pw-am.sh 14227".
Quoted 5 lines
Quoted 5 lines
I would like to try it because we have strange errors when libstc++ isn't found when building with pseudo, but same binaries are working fine without pseudo (seems related to
/etc/ld.so.cachebeing ignored sometimes from pseudo env).
Quoted 8 lines
Review
Quoted 5 lines
And while I was trying to debug it I've noticed that files.db and pseudo.log is quite big.
(quoted diff) +0
-3-rw-r--r-- 1 bitbake bitbake 240M Aug 15 13:49 files.db -rw-r--r-- 1 bitbake bitbake 91K Aug 15 13:50 logs.db -rw-r--r-- 1 bitbake bitbake 305M Aug 15 13:49 pseudo.logThe files.db is based on the number of files accessed during the time pseudo is running.
Ah ok, this was in sysroot which was created "Aug 9" so it gets pretty big fast, lets hope that sqlite3 can handle it efficiently, or is there some way to "cleanup" stale entries in files table?
ie I'm using eglibc-2.14, but files table is full of eglibc-2.13 entries and _none_ from eglibc-2.14, is it supposed to be like this or is it sign of some pseudo problems?
sqlite> select count(*) from files where path like '%eglibc-2.13%'; 20221
sqlite> select count(*) from files where path like '%eglibc-2.14%'; 0
sqlite> select count(*) from files;
1085380
Quoted 6 lines
logs.db and pseudo.log are based on the debug settings of pseudo. If you don't have debugging enabled (you would have to do this manually) and the logs are that big, something is going seriously wrong with the path/inode translation...
It's fairly normal to have a set of logs that are in the 10-20k range for
complex packages.. (These are documenting cases where pseudo has to guess about inode to path mapping..)
This was generated without debug enabled.. but maybe because of rm_work, see below...
Quoted 11 lines
Quoted 5 lines
with pseudo.log full of errors like this
pseudo: path mismatch [12 links]: ino 39721500 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor' req
'/OE/shr-core/tmp/work/om_gta02-oe-linux-gnueabi/task-shr-feed-1.0-r96'.Are you building on a NFS filesystem? NFS (and a few other filesystems) have a habit of remapping inodes out from under the filesystem. pseudo uses the inodes as a validation that the file hasn't changed. If these inodes are changing, pseudo will do it's best to keep things in sync, but there is a chance it won't always succeed.
No I'm building on local ext3 filesystem, but I'm also using rm_work, so /OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor is probably long gone and inode 39721500 wasn't marked as unused in
pseudo world or do I read it wrong?
Quoted 8 lines
(This is done because someone could access a file in pseudo control that was created outside.. so pseudo could only capture the inode.. then if in a future action the file is accessed the inode/path can be checked. If the inode matches it assumes it's the same file and now a path name exists.. if someone makes changes outside of pseudo control a similar check happens.. this time it'll see the path is the same but inode has changed.. so it guesses the file was modified
and I believe resets everything based on the new inode. Each of these will
generate a warning in the logs.)
Thanks for clarifying that.
Cheers,
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 49 lines · Martin Jansa
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Martin Jansa
Quoted 17 lines · Mark Hatle
Uprev to pseudo 1.2.
This was heavily tested by building oe-core with numerous image configurations. Each configuration generated the same image before and after the change from the current to new version of pseudo.
The primary purpose of this change is to enable the
PSEUDO_UNLOADfunctionality that may be used for performance enhancements in future versions of oe-core.The following changes since commit
37579d7d74d127c90c1e078d05c5bf4ba0b3f755:meta: glib-2.0: don't apply
qsort_rtest removable patch for native version (2011-11-02 09:08:21 +0000)are available in the git repository at:
git://git.pokylinux.org/poky-contribmhatle/pseudo
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/pseudoQuoted 7 lines
Quoted 1 line
Is there branch with pseudo-1.2 for oe-core?
^^^^^^^^^^^^^^^^^^^^^^^^
git://git.pokylinux.org/poky-contribmhatle/pseudoGrab that -- or simply grab the top commit and add it to your oe-core checkout.
Yes, but git cherry-pick would be faster and adding poky-contrib as another git remote is not very efficient as it doesn't share any git objects with oe-core repo so it checkouts "whole poky" again :/.
So I'll use 2nd easiest option "
pw-am.sh14227".Quoted 5 lines
Quoted 5 lines
I would like to try it because we have strange errors when libstc++ isn't found when building with pseudo, but same binaries are working fine without pseudo (seems related to
/etc/ld.so.cachebeing ignored sometimes from pseudo env).Quoted 8 lines
Review
Quoted 5 lines
And while I was trying to debug it I've noticed that files.db and pseudo.log is quite big.
(quoted diff) +0
-3-rw-r--r-- 1 bitbake bitbake 240M Aug 15 13:49 files.db -rw-r--r-- 1 bitbake bitbake 91K Aug 15 13:50 logs.db -rw-r--r-- 1 bitbake bitbake 305M Aug 15 13:49 pseudo.logThe files.db is based on the number of files accessed during the time pseudo is running.
Quoted 3 lines
Ah ok, this was in sysroot which was created "Aug 9" so it gets pretty big fast, lets hope that sqlite3 can handle it efficiently, or is there some way to "cleanup" stale entries in files table?
When you clean the build it will restart the pseudo DB.
Quoted 10 lines
ie I'm using eglibc-2.14, but files table is full of eglibc-2.13 entries and _none_ from eglibc-2.14, is it supposed to be like this or is it sign of some pseudo problems?
sqlite> select
count(*)from files where path like '%eglibc-2.13%'; 20221
sqlite> selectcount(*)from files where path like '%eglibc-2.14%'; 0
sqlite> selectcount(*)from files;
1085380
The pseudo DB is specific to the tmp work directory. If the work directory hasn't changed (i.e. you aren't updating versions and pr) then the pseudo DB will continue to grow to accommodate the new files.
There are flags within the DB if an erase operation occurs, but entries are only tagged as erased -- but not removed. This is due to speed issues. pseudo is significantly slower on remove and replace operations if the DB entries are removed.
Quoted 22 lines
Quoted 6 lines
logs.db and pseudo.log are based on the debug settings of pseudo. If you don't have debugging enabled (you would have to do this manually) and the logs are that big, something is going seriously wrong with the path/inode translation...
It's fairly normal to have a set of logs that are in the 10-20k range for
complex packages.. (These are documenting cases where pseudo has to guess about inode to path mapping..)This was generated without debug enabled.. but maybe because of
rm_work, see below...Quoted 11 lines
Quoted 5 lines
with pseudo.log full of errors like this
pseudo: path mismatch [12 links]: ino 39721500 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor' req
'/OE/shr-core/tmp/work/om_gta02-oe-linux-gnueabi/task-shr-feed-1.0-r96'.Are you building on a NFS filesystem? NFS (and a few other filesystems) have a habit of remapping inodes out from under the filesystem. pseudo uses the inodes as a validation that the file hasn't changed. If these inodes are changing, pseudo will do it's best to keep things in sync, but there is a chance it won't always succeed.
Quoted 4 lines
No I'm building on local ext3 filesystem, but I'm also using
rm_work, so /OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor is probably long gone and inode 39721500 wasn't marked as unused in
pseudo world or do I read it wrong?
If you use rm_work, then the pseudo DB located within the work directory will also be removed. So I'm not sure why you would be having such a large DB.
Quoted 20 lines
Quoted 8 lines
(This is done because someone could access a file in pseudo control that was created outside.. so pseudo could only capture the inode.. then if in a future action the file is accessed the inode/path can be checked. If the inode matches it assumes it's the same file and now a path name exists.. if someone makes changes outside of pseudo control a similar check happens.. this time it'll see the path is the same but inode has changed.. so it guesses the file was modified
and I believe resets everything based on the new inode. Each of these will
generate a warning in the logs.)Thanks for clarifying that.
Cheers,
_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Quoted 97 lines · Mark Hatle
Quoted 49 lines · Martin Jansa
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Martin Jansa
Quoted 17 lines · Mark Hatle
Uprev to pseudo 1.2.
This was heavily tested by building oe-core with numerous image configurations. Each configuration generated the same image before and after the change from the current to new version of pseudo.
The primary purpose of this change is to enable the
PSEUDO_UNLOADfunctionality that may be used for performance enhancements in future versions of oe-core.The following changes since commit
37579d7d74d127c90c1e078d05c5bf4ba0b3f755:meta: glib-2.0: don't apply
qsort_rtest removable patch for native version (2011-11-02 09:08:21 +0000)are available in the git repository at:
git://git.pokylinux.org/poky-contribmhatle/pseudo
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/pseudoQuoted 7 lines
Quoted 1 line
Is there branch with pseudo-1.2 for oe-core?
^^^^^^^^^^^^^^^^^^^^^^^^
git://git.pokylinux.org/poky-contribmhatle/pseudoGrab that -- or simply grab the top commit and add it to your oe-core checkout.
Yes, but git cherry-pick would be faster and adding poky-contrib as another git remote is not very efficient as it doesn't share any git objects with oe-core repo so it checkouts "whole poky" again :/.
So I'll use 2nd easiest option "
pw-am.sh14227".Quoted 5 lines
Quoted 5 lines
I would like to try it because we have strange errors when libstc++ isn't found when building with pseudo, but same binaries are working fine without pseudo (seems related to
/etc/ld.so.cachebeing ignored sometimes from pseudo env).Quoted 8 lines
Review
Quoted 5 lines
And while I was trying to debug it I've noticed that files.db and pseudo.log is quite big.
(quoted diff) +0
-3-rw-r--r-- 1 bitbake bitbake 240M Aug 15 13:49 files.db -rw-r--r-- 1 bitbake bitbake 91K Aug 15 13:50 logs.db -rw-r--r-- 1 bitbake bitbake 305M Aug 15 13:49 pseudo.logThe files.db is based on the number of files accessed during the time pseudo is running.
Quoted 3 lines
Ah ok, this was in sysroot which was created "Aug 9" so it gets pretty big fast, lets hope that sqlite3 can handle it efficiently, or is there some way to "cleanup" stale entries in files table?
When you clean the build it will restart the pseudo DB.
Quoted 10 lines
ie I'm using eglibc-2.14, but files table is full of eglibc-2.13 entries and _none_ from eglibc-2.14, is it supposed to be like this or is it sign of some pseudo problems?
sqlite> select
count(*)from files where path like '%eglibc-2.13%'; 20221
sqlite> selectcount(*)from files where path like '%eglibc-2.14%'; 0
sqlite> selectcount(*)from files;
1085380The pseudo DB is specific to the tmp work directory. If the work directory hasn't changed (i.e. you aren't updating versions and pr) then the pseudo DB will continue to grow to accommodate the new files.
There are flags within the DB if an erase operation occurs, but entries are only tagged as erased -- but not removed. This is due to speed issues. pseudo is significantly slower on remove and replace operations if the DB entries are removed.
Quoted 22 lines
Quoted 6 lines
logs.db and pseudo.log are based on the debug settings of pseudo. If you don't have debugging enabled (you would have to do this manually) and the logs are that big, something is going seriously wrong with the path/inode translation...
It's fairly normal to have a set of logs that are in the 10-20k range for
complex packages.. (These are documenting cases where pseudo has to guess about inode to path mapping..)This was generated without debug enabled.. but maybe because of
rm_work, see below...Quoted 11 lines
Quoted 5 lines
with pseudo.log full of errors like this
pseudo: path mismatch [12 links]: ino 39721500 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor' req
'/OE/shr-core/tmp/work/om_gta02-oe-linux-gnueabi/task-shr-feed-1.0-r96'.Are you building on a NFS filesystem? NFS (and a few other filesystems) have a habit of remapping inodes out from under the filesystem. pseudo uses the inodes as a validation that the file hasn't changed. If these inodes are changing, pseudo will do it's best to keep things in sync, but there is a chance it won't always succeed.
Quoted 7 lines
Quoted 4 lines
No I'm building on local ext3 filesystem, but I'm also using
rm_work, so /OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor is probably long gone and inode 39721500 wasn't marked as unused in
pseudo world or do I read it wrong?If you use
rm_work, then the pseudo DB located within the work directory will also be removed. So I'm not sure why you would be having such a large DB.
But I was talking about pseudo db here:
tmp/sysroots/x86_64-linux/var/pseudo/
now with pseudo-1.2 I still have quite a few warnings about inodes in pseudo.log now in work directory, ie python build:
OE @ ~/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1 $
-rw-r--r-- 1 bitbake bitbake 5.2M Nov 2 23:44 files.db -rw-r--r-- 1 bitbake bitbake 4.0K Nov 2 23:44 logs.db -rw-r--r-- 1 bitbake bitbake 46K Nov 2 23:44 pseudo.log
pseudo: dir err : 1240102 ['/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/deploy-ipks/armv4t/IPKG_BUILD.25064'] (db 'no path') db mode 0100644, header mode 040755 (unlinking db)
pseudo: creat for '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/deploy-ipks/armv4t/stvITxha' replaces existing 1240107 ['no path'].
wxr-xr-x 1 bitbake bitbake 0 Nov 2 23:44 pseudo.socket
pseudo: path mismatch [2 links]: ino 1240673 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/sstate-build-package/package/usr/bin/python2.7' req '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/sstate-build-package/package/usr/bin/python'.
Signature · 1 line
Martin 'JaMa' Jansa jabber: [email protected]
Quoted 103 lines · Martin Jansa
Quoted 102 lines · Mark Hatle
Quoted 49 lines · Martin Jansa
Quoted 20 lines · Mark Hatle
Quoted 19 lines · Martin Jansa
Quoted 17 lines · Mark Hatle
Uprev to pseudo 1.2.
This was heavily tested by building oe-core with numerous image configurations. Each configuration generated the same image before and after the change from the current to new version of pseudo.
The primary purpose of this change is to enable the
PSEUDO_UNLOADfunctionality that may be used for performance enhancements in future versions of oe-core.The following changes since commit
37579d7d74d127c90c1e078d05c5bf4ba0b3f755:meta: glib-2.0: don't apply
qsort_rtest removable patch for native version (2011-11-02 09:08:21 +0000)are available in the git repository at:
git://git.pokylinux.org/poky-contribmhatle/pseudo
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/pseudoQuoted 7 lines
Quoted 1 line
Is there branch with pseudo-1.2 for oe-core?
^^^^^^^^^^^^^^^^^^^^^^^^
git://git.pokylinux.org/poky-contribmhatle/pseudoGrab that -- or simply grab the top commit and add it to your oe-core checkout.
Yes, but git cherry-pick would be faster and adding poky-contrib as another git remote is not very efficient as it doesn't share any git objects with oe-core repo so it checkouts "whole poky" again :/.
So I'll use 2nd easiest option "
pw-am.sh14227".Quoted 5 lines
Quoted 5 lines
I would like to try it because we have strange errors when libstc++ isn't found when building with pseudo, but same binaries are working fine without pseudo (seems related to
/etc/ld.so.cachebeing ignored sometimes from pseudo env).Quoted 8 lines
Review
Quoted 5 lines
And while I was trying to debug it I've noticed that files.db and pseudo.log is quite big.
(quoted diff) +0
-3-rw-r--r-- 1 bitbake bitbake 240M Aug 15 13:49 files.db -rw-r--r-- 1 bitbake bitbake 91K Aug 15 13:50 logs.db -rw-r--r-- 1 bitbake bitbake 305M Aug 15 13:49 pseudo.logThe files.db is based on the number of files accessed during the time pseudo is running.
Quoted 3 lines
Ah ok, this was in sysroot which was created "Aug 9" so it gets pretty big fast, lets hope that sqlite3 can handle it efficiently, or is there some way to "cleanup" stale entries in files table?
When you clean the build it will restart the pseudo DB.
Quoted 10 lines
ie I'm using eglibc-2.14, but files table is full of eglibc-2.13 entries and _none_ from eglibc-2.14, is it supposed to be like this or is it sign of some pseudo problems?
sqlite> select
count(*)from files where path like '%eglibc-2.13%'; 20221
sqlite> selectcount(*)from files where path like '%eglibc-2.14%'; 0
sqlite> selectcount(*)from files;
1085380The pseudo DB is specific to the tmp work directory. If the work directory hasn't changed (i.e. you aren't updating versions and pr) then the pseudo DB will continue to grow to accommodate the new files.
There are flags within the DB if an erase operation occurs, but entries are only tagged as erased -- but not removed. This is due to speed issues. pseudo is significantly slower on remove and replace operations if the DB entries are removed.
Quoted 26 lines
Quoted 6 lines
logs.db and pseudo.log are based on the debug settings of pseudo. If you don't have debugging enabled (you would have to do this manually) and the logs are that big, something is going seriously wrong with the path/inode translation...
It's fairly normal to have a set of logs that are in the 10-20k range for
complex packages.. (These are documenting cases where pseudo has to guess about inode to path mapping..)This was generated without debug enabled.. but maybe because of
rm_work, see below...Quoted 11 lines
Quoted 5 lines
with pseudo.log full of errors like this
pseudo: path mismatch [12 links]: ino 39721500 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor' req
'/OE/shr-core/tmp/work/om_gta02-oe-linux-gnueabi/task-shr-feed-1.0-r96'.Are you building on a NFS filesystem? NFS (and a few other filesystems) have a habit of remapping inodes out from under the filesystem. pseudo uses the inodes as a validation that the file hasn't changed. If these inodes are changing, pseudo will do it's best to keep things in sync, but there is a chance it won't always succeed.
No I'm building on local ext3 filesystem, but I'm also using
rm_work, so /OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor is probably long gone and inode 39721500 wasn't marked as unused in
pseudo world or do I read it wrong?
Quoted 5 lines
Quoted 2 lines
If you use
rm_work, then the pseudo DB located within the work directory will also be removed. So I'm not sure why you would be having such a large DB.But I was talking about pseudo db here:
tmp/sysroots/x86_64-linux/var/pseudo/
I just checked my system and the logs in the sysroot isn't very large. I'm curious as to why things are as big on your system.
Quoted 11 lines
now with pseudo-1.2 I still have quite a few warnings about inodes in pseudo.log now in work directory, ie python build:
OE @ ~/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1 $(quoted diff) +0
-3-rw-r--r-- 1 bitbake bitbake 5.2M Nov 2 23:44 files.db -rw-r--r-- 1 bitbake bitbake 4.0K Nov 2 23:44 logs.db -rw-r--r-- 1 bitbake bitbake 46K Nov 2 23:44 pseudo.logpseudo: dir err : 1240102 ['/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/deploy-ipks/armv4t/IPKG_BUILD.25064'] (db 'no path') db mode 0100644, header mode 040755 (unlinking db)
pseudo: creat for '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/deploy-ipks/armv4t/stvITxha' replaces existing 1240107 ['no path'].
wxr-xr-x 1 bitbake bitbake 0 Nov 2 23:44 pseudo.socket
pseudo: path mismatch [2 links]: ino 1240673 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/sstate-build-package/package/usr/bin/python2.7' req '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/python-2.7.2-r0.1/sstate-build-package/package/usr/bin/python'.
None of the inode processing was changed in 1.2, only code that was specific to pseudo preloading and the clone(2) function wrapper.
The 'no path' messages.. (see the first and second ones above) are the ones where a file and/or directory was created outside of pseudo, and then referenced by the inode directly. Once referenced by name and not inode it was able to correlate them. This is somewhat expected and a model that will be looked at in the near? future by the pseudo maintainer.. (he and I talked about it.. and the fact we're getting so many of those messages is surprising.)
The later one indicates that the inode 1240673 was defined as belonging to one item, but a second item showed up instead. This usually means that something was renamed outside of the control of pseudo. This is more likely to be a problem then the first two.
--Mark
Quoted 21 lines · Mark Hatle
Quoted 20 lines · Martin Jansa
Quoted 19 lines · Mark Hatle
Quoted 17 lines · Martin Jansa
Quoted 11 lines · Mark Hatle
Quoted 5 lines
with pseudo.log full of errors like this
pseudo: path mismatch [12 links]: ino 39721500 db '/OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor' req
'/OE/shr-core/tmp/work/om_gta02-oe-linux-gnueabi/task-shr-feed-1.0-r96'.Are you building on a NFS filesystem? NFS (and a few other filesystems) have a habit of remapping inodes out from under the filesystem. pseudo uses the inodes as a validation that the file hasn't changed. If these inodes are changing, pseudo will do it's best to keep things in sync, but there is a chance it won't always succeed.
No I'm building on local ext3 filesystem, but I'm also using
rm_work, so /OE/shr-core/tmp/work/armv4t-oe-linux-gnueabi/vim-7.2.446-r10.3/image/usr/share/vim/vim72/tutor is probably long gone and inode 39721500 wasn't marked as unused in
pseudo world or do I read it wrong?
Quoted 8 lines
Quoted 5 lines
Quoted 2 lines
If you use
rm_work, then the pseudo DB located within the work directory will also be removed. So I'm not sure why you would be having such a large DB.But I was talking about pseudo db here:
tmp/sysroots/x86_64-linux/var/pseudo/I just checked my system and the logs in the sysroot isn't very large. I'm curious as to why things are as big on your system.
We did have a problem a while back where pseudo was active when it shouldn't have been and I noticed huge db/logs on my system too. The environment changes that were pushed shortly before 1.1 fixed that problem as far as I know but if this is an old build directory, it could have those left behind.
This actually accounted for the significant load time bitbake had on one of my systems too - it was spending the time loading the pseudo db.
Cheers,
Richard
Saul Wold
AppliedQuoted 21 lines · Mark Hatle
Uprev to pseudo 1.2.
This was heavily tested by building oe-core with numerous image configurations. Each configuration generated the same image before and after the change from the current to new version of pseudo.
The primary purpose of this change is to enable the
PSEUDO_UNLOADfunctionality that may be used for performance enhancements in future versions of oe-core.The following changes since commit
37579d7d74d127c90c1e078d05c5bf4ba0b3f755:meta: glib-2.0: don't apply qsort_r test removable patch for native version (2011-11-02 09:08:21 +0000)are available in the git repository at:
git://git.pokylinux.org/poky-contribmhatle/pseudo
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=mhatle/pseudoMark Hatle (1):
pseudo: Uprev pseudo to version 1.2
Quoted 8 lines
.../conf/distro/include/distro_tracking_fields.inc | 6 +- .../pseudo/pseudo/realpath_fix.patch | 165 -------------------- .../pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} | 7 +- meta/recipes-devtools/pseudo/pseudo_git.bb | 6 +- 4 files changed, 9 insertions(+), 175 deletions(-)delete mode 100644
meta/recipes-devtools/pseudo/pseudo/realpath_fix.patchrename meta/recipes-devtools/pseudo/{pseudo_1.1.1.bb => pseudo_1.2.bb} (48%)
Merged to OE-Core
Thanks
Sau!