Fix for buildstats diskio on non physical disks
cover Fix for buildstats diskio on non physical disks
From: Elizabeth Flanagan <[email protected]>
tmpfs/encryptfs/ramfs have no entry in /proc/diskstats. This modifies buildstats to not collect diskio statistics when we encounter a case where the os.major/os.minor is not represented with an entry in /proc/diskstats.
Longer term solution to this is to:
- Identify all fs types that don't have representation in /proc/diskstats
- Gather meaningful iostats of these fs types if possible.
The following changes since commit 4a951e0433a99cd985515843f0a48fadc7050aca:
Fix HOMEPAGE values in libzypp and sat-solver .bb files (2011-11-01 18:28:19 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib eflanagan/diskio_bug1700
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/diskio_bug1700
Elizabeth Flanagan (1):
[Yocto Bug 1700] Fix for buildstats on tmpfs
meta/classes/buildstats.bbclass | 37 ++++++++++++++++++++++++++----------- 1 files changed, 26 insertions(+), 11 deletions(-)
PATCH 1/1 Fix for buildstats on tmpfs
From: Elizabeth Flanagan <[email protected]>
tmpfs/encryptfs/(and most likely, but not confirmed)ramfs TMPDIRs cause diskstats to choke. No device entry ends up in /proc/diskstats for these fs types, which ends up causing the failure.
The short term solution is to exclude these fs types from diskstat collection. Longer term we will want to see if we can collect meaningful diskio for each of these, and other, use cases, but for this cleans up Bug 1700.
--- meta/classes/buildstats.bbclass | 37 ++++++++++++++++++++++++++----------- 1 files changed, 26 insertions(+), 11 deletions(-)
meta/classes/buildstats.bbclass +26 -11
@@ -48,13 +48,24 @@ def set_device(e): # something like stick DL_DIR on a different partition and this would # throw stats gathering off. The same goes with SSTATE_DIR. However, let's # get the basics in here and work on the cornercases later. + # A note. /proc/diskstats does not contain info on encryptfs, tmpfs, etc. + # If we end up hitting one of these fs, we'll just skip diskstats collection. ############################################################################ device=os.stat(tmpdir) majordev=os.major(device.st_dev) minordev=os.minor(device.st_dev) + ############################################################################ + # Bug 1700: + # Because tmpfs/encryptfs/ramfs etc inserts no entry in /proc/diskstats + # we set rdev to NoLogicalDevice and search for it later. If we find NLD + # we do not collect diskstats as the method to collect meaningful statistics + # for these fs types requires a bit more research. + ############################################################################ for line in open("/proc/diskstats", "r"): if majordev == int(line.split()[0]) and minordev == int(line.split()[1]): rdev=line.split()[2] + else: + rdev="NoLogicalDevice" file = open(bb.data.getVar('DEVFILE', e.data, True), "w") file.write(rdev) file.close()
@@ -133,10 +144,11 @@ def write_task_data(status, logfile, dev, e): # For the best information, running things with BB_TOTAL_THREADS = "1" # would return accurate per task results. ############################################################################ - diskdata = get_diskdata("__diskdata_task", dev, e.data) - if diskdata: - for key in sorted(diskdata.iterkeys()): - file.write(key + ": " + diskdata[key] + "\n") + if dev != "NoLogicalDevice": + diskdata = get_diskdata("__diskdata_task", dev, e.data) + if diskdata: + for key in sorted(diskdata.iterkeys()): + file.write(key + ": " + diskdata[key] + "\n") if status is "passed": file.write("Status: PASSED \n") else:
@@ -169,7 +181,8 @@ python run_buildstats () { bb.mkdirhier(bsdir) except: pass - set_diskdata("__diskdata_build", device, e.data) + if device != "NoLogicalDevice": + set_diskdata("__diskdata_build", device, e.data) set_timedata("__timedata_build", e.data) build_time = os.path.join(bsdir, "build_stats") # write start of build into build_time
@@ -185,7 +198,7 @@ python run_buildstats () { elif isinstance(e, bb.event.BuildCompleted): bn = get_bn(e) - dev = get_device(e) + device = get_device(e) bsdir = os.path.join(bb.data.getVar('BUILDSTATS_BASE', e.data, True), bn) taskdir = os.path.join(bsdir, bb.data.expand("${PF}", e.data)) build_time = os.path.join(bsdir, "build_stats")
@@ -201,10 +214,11 @@ python run_buildstats () { file.write("Elapsed time: %0.2f seconds \n" % (time)) if cpu: file.write("CPU usage: %0.1f%% \n" % cpu) - diskio = get_diskdata("__diskdata_build", dev, e.data) - if diskio: - for key in sorted(diskio.iterkeys()): - file.write(key + ": " + diskio[key] + "\n") + if device != "NoLogicalDevice": + diskio = get_diskdata("__diskdata_build", device, e.data) + if diskio: + for key in sorted(diskio.iterkeys()): + file.write(key + ": " + diskio[key] + "\n") file.close() if isinstance(e, bb.build.TaskStarted):
@@ -212,7 +226,8 @@ python run_buildstats () { device = get_device(e) bsdir = os.path.join(bb.data.getVar('BUILDSTATS_BASE', e.data, True), bn) taskdir = os.path.join(bsdir, bb.data.expand("${PF}", e.data)) - set_diskdata("__diskdata_task", device, e.data) + if device != "NoLogicalDevice": + set_diskdata("__diskdata_task", device, e.data) set_timedata("__timedata_task", e.data) try: bb.mkdirhier(taskdir)
Signature · 1 line
1.7.1
Saul Wold
AppliedRe: [PATCH 1/1] [Yocto Bug 1700] Fix for buildstats on tmpfs
Quoted 97 lines · Beth Flanagan
From: Elizabeth Flanagan<[email protected]>
tmpfs/encryptfs/(and most likely, but not confirmed)ramfs TMPDIRs cause diskstats to choke. No device entry ends up in
/proc/diskstatsfor these fs types, which ends up causing the failure.The short term solution is to exclude these fs types from diskstat collection. Longer term we will want to see if we can collect meaningful diskio for each of these, and other, use cases, but for this cleans up Bug 1700.
Signed-off-by Elizabeth Flanagan<[email protected]>--- meta/classes/buildstats.bbclass | 37 ++++++++++++++++++++++++++----------- 1 files changed, 26 insertions(+), 11 deletions(-)meta/classes/buildstats.bbclass +24
-10@@ -48,13 +48,24 @@ def set_device(e): # something like stick DL_DIR on a different partition and this would # throw stats gathering off. The same goes with SSTATE_DIR. However, let's # get the basics in here and work on the cornercases later. + # A note. /proc/diskstats does not contain info on encryptfs, tmpfs, etc. + # If we end up hitting one of these fs, we'll just skip diskstats collection. ############################################################################ device=os.stat(tmpdir) majordev=os.major(device.st_dev) minordev=os.minor(device.st_dev) + ############################################################################ + # Bug 1700: + # Because tmpfs/encryptfs/ramfs etc inserts no entry in /proc/diskstats + # we set rdev to NoLogicalDevice and search for it later. If we find NLD + # we do not collect diskstats as the method to collect meaningful statistics + # for these fs types requires a bit more research. + ############################################################################ for line in open("/proc/diskstats", "r"): if majordev == int(line.split()[0]) and minordev == int(line.split()[1]): rdev=line.split()[2] + else: + rdev="NoLogicalDevice" file = open(bb.data.getVar('DEVFILE', e.data, True), "w") file.write(rdev) file.close()@@ -133,10 +144,11 @@ def write_task_data(status, logfile, dev, e): # For the best information, running things with BB_TOTAL_THREADS = "1" # would return accurate per task results. ############################################################################ - diskdata = get_diskdata("__diskdata_task", dev, e.data) - if diskdata: - for key in sorted(diskdata.iterkeys()): - file.write(key + ": " + diskdata[key] + "\n") + if dev != "NoLogicalDevice": + diskdata = get_diskdata("__diskdata_task", dev, e.data) + if diskdata: + for key in sorted(diskdata.iterkeys()): + file.write(key + ": " + diskdata[key] + "\n") if status is "passed": file.write("Status: PASSED \n") else:@@ -169,7 +181,8 @@ python run_buildstats () { bb.mkdirhier(bsdir) except: pass - set_diskdata("__diskdata_build", device, e.data) + if device != "NoLogicalDevice": + set_diskdata("__diskdata_build", device, e.data) set_timedata("__timedata_build", e.data) build_time = os.path.join(bsdir, "build_stats") # write start of build into build_time@@ -185,7 +198,7 @@ python run_buildstats () { elif isinstance(e, bb.event.BuildCompleted): bn = get_bn(e) - dev = get_device(e) + device = get_device(e) bsdir = os.path.join(bb.data.getVar('BUILDSTATS_BASE', e.data, True), bn) taskdir = os.path.join(bsdir, bb.data.expand("${PF}", e.data)) build_time = os.path.join(bsdir, "build_stats")@@ -201,10 +214,11 @@ python run_buildstats () { file.write("Elapsed time: %0.2f seconds \n" % (time)) if cpu: file.write("CPU usage: %0.1f%% \n" % cpu) - diskio = get_diskdata("__diskdata_build", dev, e.data) - if diskio: - for key in sorted(diskio.iterkeys()): - file.write(key + ": " + diskio[key] + "\n") + if device != "NoLogicalDevice": + diskio = get_diskdata("__diskdata_build", device, e.data) + if diskio: + for key in sorted(diskio.iterkeys()): + file.write(key + ": " + diskio[key] + "\n") file.close() if isinstance(e, bb.build.TaskStarted):
meta/classes/buildstats.bbclass python run_buildstats () { · line 233
Quoted 9 lines
meta/classes/buildstats.bbclass +2
-1@@ -212,7 +226,8 @@ python run_buildstats () { device = get_device(e) bsdir = os.path.join(bb.data.getVar('BUILDSTATS_BASE', e.data, True), bn) taskdir = os.path.join(bsdir, bb.data.expand("${PF}", e.data)) - set_diskdata("__diskdata_task", device, e.data) + if device != "NoLogicalDevice": + set_diskdata("__diskdata_task", device, e.data) set_timedata("__timedata_task", e.data) try: bb.mkdirhier(taskdir)
Merged into OE-Core with minor tweak to fix commit message formating
Thanks
Sau!
Quoted 5 lines · Beth Flanagan
From: Elizabeth Flanagan <[email protected]>
tmpfs/encryptfs/ramfs have no entry in
/proc/diskstats. This modifies buildstats to not collect diskio statistics when we encounter a case where the os.major/os.minor is not represented with an entry in/proc/diskstats.
A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes. The problem there is, that btrfs's stat() reports fake device ids that cannot be found in /proc/diskstats.
Quoted 23 lines
Longer term solution to this is to:
- Identify all fs types that don't have representation in/proc/diskstats
- Gather meaningful iostats of these fs types if possible.The following changes since commit
4a951e0433a99cd985515843f0a48fadc7050aca:Fix HOMEPAGE values in libzypp and sat-solver .bb files (2011-11-01 18:28:19 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contribeflanagan/diskio_bug1700
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=eflanagan/diskio_bug1700Elizabeth Flanagan (1):
[Yocto Bug 1700] Fix for buildstats on tmpfsmeta/classes/buildstats.bbclass | 37 ++++++++++++++++++++++++++----------- 1 files changed, 26 insertions(+), 11 deletions(-)_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 13 lines
DI Wolfram Stering (Entwicklung) HALE electronic GmbH Eugen-Müller-Straße 18, 5020 Salzburg, Austria Tel: +43 (662) 439011 550 Fax: +43 (662) 439011 9 http://www.hale.at/ Firmenbuchnummer: FN 66801m HG Salzburg -- Scanned by MailScanner.
Quoted 11 lines · Wolfram Stering
Quoted 5 lines · Beth Flanagan
From: Elizabeth Flanagan <[email protected]>
tmpfs/encryptfs/ramfs have no entry in
/proc/diskstats. This modifies buildstats to not collect diskio statistics when we encounter a case where the os.major/os.minor is not represented with an entry in/proc/diskstats.A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes. The problem there is, that btrfs'sstat()reports fake device ids that cannot be found in/proc/diskstats.
Did this patch help address that problem for you too?
Cheers,
Richard
Quoted 4 lines · Richard Purdie
Quoted 3 lines · Wolfram Stering
Quoted 2 lines · Beth Flanagan
From: Elizabeth Flanagan <[email protected]>
Quoted 8 lines
Quoted 7 lines
Quoted 3 lines
tmpfs/encryptfs/ramfs have no entry in
/proc/diskstats. This modifies buildstats to not collect diskio statistics when we encounter a case where the os.major/os.minor is not represented with an entry in/proc/diskstats.A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes. The problem there is, that btrfs'sstat()reports fake device ids that cannot be found in/proc/diskstats.Did this patch help address that problem for you too?
I'll be able to check that tomorrow and report back.
regards,
- wolfi
Quoted 3 lines
Cheers,
Richard
Signature · 13 lines
Wolfram Stering (Entwicklung) HALE electronic GmbH Eugen-Müller-Straße 18, 5020 Salzburg, Austria Tel: +43 (662) 439011 550 Fax: +43 (662) 439011 9 http://www.hale.at/ Firmenbuchnummer: FN 66801m HG Salzburg -- Scanned by MailScanner.
Quoted 8 lines · Wolfram Stering
Quoted 7 lines · Richard Purdie
Quoted 6 lines · Wolfram Stering
Quoted 5 lines · Beth Flanagan
From: Elizabeth Flanagan <[email protected]>
tmpfs/encryptfs/ramfs have no entry in
/proc/diskstats. This modifies buildstats to not collect diskio statistics when we encounter a case where the os.major/os.minor is not represented with an entry in/proc/diskstats.
Quoted 7 lines
Quoted 5 lines
Quoted 4 lines
A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes. The problem there is, that btrfs'sstat()reports fake device ids that cannot be found in/proc/diskstats.Did this patch help address that problem for you too?
I'll be able to check that tomorrow and report back.
It should as the patch will just disable diskio collection if it cannot find a valid device id in /proc/diskstats. Let me know if it doesn't and I'll rework it and resubmit.
-b
Quoted 32 lines
regards,
- wolfi
Quoted 3 lines
Cheers,
Richard
--
Wolfram Stering
(Entwicklung)
HALE electronic GmbH
Eugen-Müller-Straße 18, 5020 Salzburg, Austria
Tel: +43 (662) 439011 550
Fax: +43 (662) 439011 9
http://www.hale.at/
Firmenbuchnummer: FN 66801m HG Salzburg--
Scanned by MailScanner._______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release
Quoted 13 lines · Elizabeth
Quoted 12 lines · Wolfram Stering
Quoted 11 lines · Richard Purdie
Quoted 10 lines · Wolfram Stering
Quoted 5 lines · Beth Flanagan
From: Elizabeth Flanagan <[email protected]>
tmpfs/encryptfs/ramfs have no entry in
/proc/diskstats. This modifies buildstats to not collect diskio statistics when we encounter a case where the os.major/os.minor is not represented with an entry in/proc/diskstats.A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes. The problem there is, that btrfs'sstat()reports fake device ids that cannot be found in/proc/diskstats.
Quoted 7 lines
Quoted 2 lines
Quoted 1 line
Did this patch help address that problem for you too?
I'll be able to check that tomorrow and report back.
It should as the patch will just disable diskio collection if it cannot find a valid device id in
/proc/diskstats. Let me know if it doesn't and I'll rework it and resubmit.-b
Your patch fixes the buildstats issue for building on a btrfs volume as well.
bitbake no longer hits the exception and diskstats are omitted.
Theoretically, this information would be available for btrfs, but it is not discoverable
in the way buildstats currently collects the disk statistics. However, this is a seaprate
issue, I think.
Thanks a lot for fixing this,
-wolfi
Signature · 13 lines
Wolfram Stering (Entwicklung) HALE electronic GmbH Eugen-Müller-Straße 18, 5020 Salzburg, Austria Tel: +43 (662) 439011 550 Fax: +43 (662) 439011 9 http://www.hale.at/ Firmenbuchnummer: FN 66801m HG Salzburg -- Scanned by MailScanner.
Quoted 26 lines · Wolfram Stering
Quoted 13 lines · Elizabeth
Quoted 12 lines · Wolfram Stering
Quoted 11 lines · Richard Purdie
Quoted 10 lines · Wolfram Stering
Quoted 5 lines · Beth Flanagan
From: Elizabeth Flanagan <[email protected]>
tmpfs/encryptfs/ramfs have no entry in
/proc/diskstats. This modifies buildstats to not collect diskio statistics when we encounter a case where the os.major/os.minor is not represented with an entry in/proc/diskstats.A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes. The problem there is, that btrfs'sstat()reports fake device ids that cannot be found in/proc/diskstats.Quoted 7 lines
Quoted 2 lines
Quoted 1 line
Did this patch help address that problem for you too?
I'll be able to check that tomorrow and report back.
It should as the patch will just disable diskio collection if it cannot find a valid device id in
/proc/diskstats. Let me know if it doesn't and I'll rework it and resubmit.-b
Your patch fixes the buildstats issue for building on a btrfs volume as well.
bitbake no longer hits the exception and diskstats are omitted.
Quoted 5 lines
Theoretically, this information would be available for btrfs, but it is not discoverable
in the way buildstats currently collects the disk statistics. However, this is a seaprate
issue, I think.
Yes, and I'm open for suggestions on the best way to implement finding disk io data for the myriad of filesystem types out there. :/ Or at the very list a way to pull diskstats from a trimmed down list that includes:
ext2/3/4
btrfs
tmpfs
encryptfs....
I'm not sure performance data is even collected for for non-block devices. dstat and iostat as far as I can tell can't collect data for tmpfs.
Ideas?
-b
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release
Op 8 nov. 2011, om 16:57 heeft Flanagan, Elizabeth het volgende geschreven:
Quoted 13 lines · Wolfram Stering
Quoted 12 lines · Richard Purdie
Quoted 11 lines · Wolfram Stering
Quoted 10 lines · Beth Flanagan
Quoted 5 lines
From: Elizabeth Flanagan <[email protected]>
tmpfs/encryptfs/ramfs have no entry in
/proc/diskstats. This modifies buildstats to not collect diskio statistics when we encounter a case where the os.major/os.minor is not represented with an entry in/proc/diskstats.A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes. The problem there is, that btrfs'sstat()reports fake device ids that cannot be found in/proc/diskstats.
Quoted 7 lines
Quoted 3 lines
Quoted 1 line
Did this patch help address that problem for you too?
I'll be able to check that tomorrow and report back.
It should as the patch will just disable diskio collection if it cannot find a valid device id in
/proc/diskstats. Let me know if it doesn't and I'll rework it and resubmit.
Related question: what do you use to visualize the buildstats? I heard rumours about getting bootchart to support our format natively, is that the case? I will be needing pretty graphics for a report on the oe-core buildflow in a few weeks :)
regards,
Koen
Quoted 20 lines · Koen Kooi
Op 8 nov. 2011, om 16:57 heeft Flanagan, Elizabeth het volgende geschreven:
Quoted 17 lines · Wolfram Stering
Quoted 15 lines · Richard Purdie
Quoted 12 lines · Wolfram Stering
Quoted 10 lines · Beth Flanagan
Quoted 5 lines
From: Elizabeth Flanagan <[email protected]>
tmpfs/encryptfs/ramfs have no entry in
/proc/diskstats. This modifies buildstats to not collect diskio statistics when we encounter a case where the os.major/os.minor is not represented with an entry in/proc/diskstats.A similar issue exists for building on a btrfs partition.
I posted a message on Oct 28 concerning buildstats on btrfs volumes. The problem there is, that btrfs'sstat()reports fake device ids that cannot be found in/proc/diskstats.Did this patch help address that problem for you too?
I'll be able to check that tomorrow and report back.
Quoted 7 lines
Quoted 3 lines
It should as the patch will just disable diskio collection if it cannot find a valid device id in
/proc/diskstats. Let me know if it doesn't and I'll rework it and resubmit.Related question: what do you use to visualize the buildstats? I heard rumours about getting bootchart to support our format natively, is that the case? I will be needing pretty graphics for a report on the oe-core buildflow in a few weeks :)
regards,
Koen,
There is a patch to pybootchartgui that allows visualization. RP has the patchset IIRC. I'm going to be working on buildstats visualization soon and I'm leaning towards using jquery/flot to create a way to allow end users to generate via browser some comparative graphing of buildstats against different builds/statistics. This way I can set it up for the AB to compare all the builds we're doing there and end users can just use it through a browser either locally or via a server.
-b
Quoted 15 lines
Koen
-----BEGIN PGP SIGNATURE-----iEYEARECAAYFAk66bmcACgkQMkyGM64RGpHMCQCguHdgtS3SrIVV5+T1dOktTpAT FkQAnRZzZqwPJlGa3QlGGmxRrZnNtv/8
=fEA3
-----END PGP SIGNATURE-----_______________________________________________
Openembedded-core mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/openembedded-core
Signature · 3 lines
Elizabeth Flanagan Yocto Project Build and Release