Disk space monitoring

PATCH In review Robert Yang 2 participants 6 messages Dec 15, 2011 – Dec 16, 2011

Share & tools

Robert Yang

author

2011-12-15 10:52 UTC #0 lore raw

cover Disk space monitoring

Here are the test info:
1) set the following variables in conf/local.conf for testing:
BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1" BB_DISKMON_MINSPACE = "200G"
BB_DISKMON_MININODES = "20M"
BB_DISKMON_INTERVAL = "30s"

The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in my build, so I added another direcotory /home/lyang1 to test the ability of monitoring multi mount points.

2) bitbake core-image-sato
WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)

3) Aalso have tested about the following items:
a) Use ctrl-c to stop the monitor
b) The monitor should stop when the build stops
c) Test the space unit G, B, K(case in-sensitive) and the time unit h, m, s(case-sensitive)

// Robert

The following changes since commit 9d136b2db8f906c562cbdb23a9b238f0e237074b:

gdk-pixbuf: Ensure the binaries can be relocated (2011-12-13 18:00:25 +0000)

are available in the git repository at:
git://git.pokylinux.org/poky-contrib robert/disk-space-monitor
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/disk-space-monitor

Robert Yang (2):
Disk space monitoring
Add config sample for disk space monitoring

 bitbake/bin/bitbake               |   12 +++
 bitbake/lib/bb/monitordisk.py     |  153 +++++++++++++++++++++++++++++++++++++
 meta-yocto/conf/local.conf.sample |   28 +++++++
 3 files changed, 193 insertions(+), 0 deletions(-)
 create mode 100644 bitbake/lib/bb/monitordisk.py

Signature · 1 line

1.7.4.1

Robert Yang

author

2011-12-15 10:52 UTC #1 lore raw

PATCH 1/2 Disk space monitoring

Monitor disk availability and warn the user when the free disk space or amount of free inode is running low, it is enabled when BB_DISKMON_DIRS is set.

* Variable meanings(from meta-yocto/conf/local.conf.sample):

  # Set the directories to monitor for disk usage, if more than one
  # directories are in the same device, then only one directory would be
  # warned since the monitor is based on the device.
  #
  BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR}"
  #
  # Set the minimum amount of disk space to warn, the unit can be G,
  # M or K, but do not use GB, MB or KB (B is not needed).
  # The default vaule is 1G if not set.
  #
  #BB_DISKMON_MINSPACE = "1G"
  #
  # Set the minimum amount of inodes to warn, the unit can be G, M or K.
  # The default value is 1K if not set.
  #
  #BB_DISKMON_MININODES = "1K"
  #
  # Set the monitor frequency, the unit can be h(hour), m(minute) or
  # s(second), when no unit is specified, the default unit is s(second),
  # for example, "60" means 60 seconds.
  # The default value is 60s if not set.
  #
  #BB_DISKMON_INTERVAL = "60s"

* Problems:

1) The disk space monitor is running in a sub process of bin/bitbake, so it can't use the same logger as bitbake(or I don't know how to use it, I've tried to use the logger.warn('some information'), but it would print nothing in the subprocess), another solution maybe use an independent logger in the subprocess, but it seems that it doesn't worth because of the two reasons:
a) The logger of bin/bitbake can't control the monitor's independent logger
b) The monitor doesn't need print many things

It would be great if anyone could give me more instructions.

2) The code is placed in the new file lib/bb/monitordisk.py, since it seems that it is more independent.

[YOCTO #1589]

Signed-off-by Robert Yang <[email protected]>

---
 bitbake/bin/bitbake           |   12 +++
 bitbake/lib/bb/monitordisk.py |  153 +++++++++++++++++++++++++++++++++++++++++
 2 files changed, 165 insertions(+), 0 deletions(-)
 create mode 100644 bitbake/lib/bb/monitordisk.py

bitbake/bin/bitbake +12 -0

index 9eb558b..5761376 100755
@@ -39,6 +39,8 @@ import bb.msg
 from bb import cooker
 from bb import ui
 from bb import server
+from bb import monitordisk
+from multiprocessing import Process
 
 __version__ = "1.15.0"
 logger = logging.getLogger("BitBake")

@@ -217,6 +219,13 @@ Default BBFILES are the .bb files in the current directory.""")
     server.saveConnectionDetails()
     server.detach(cooker_logfile)
 
+    # For disk space monitor
+    diskMonDirs = cooker.configuration.data.getVar("BB_DISKMON_DIRS", 1)
+    if diskMonDirs:
+        monitorProcess = Process(target=monitordisk.monitorDiskSpace, \
+            args=(diskMonDirs, cooker.configuration.data))
+        monitorProcess.start()
+
     # Should no longer need to ever reference cooker
     del cooker

@@ -230,6 +239,9 @@ Default BBFILES are the .bb files in the current directory.""")
     finally:
         bb.event.ui_queue = []
         server_connection.terminate()
+        # Stop the disk space monitor
+        if diskMonDirs:
+            monitorProcess.terminate()
 
     return 1

bitbake/lib/bb/monitordisk.py new file +153 -0

153 lines not on this page · raw message

Signature · 1 line

1.7.4.1

Robert Yang

author

2011-12-15 10:52 UTC #2 lore raw

PATCH 2/2 Add config sample for disk space monitoring

In reply to Robert Yang

Add config sample for disk space monitoring to
meta-yocto/conf/local.conf.sample

[YOCTO #1589]

Signed-off-by Robert Yang <[email protected]>

---
 meta-yocto/conf/local.conf.sample |   28 ++++++++++++++++++++++++++++
 1 files changed, 28 insertions(+), 0 deletions(-)

meta-yocto/conf/local.conf.sample +28 -0

index ca8636c..d863774 100644
@@ -222,3 +222,31 @@ PATCHRESOLVE = "noop"
 # track the version of this file when it was generated. This can safely be ignored if
 # this doesn't mean anything to you.
 CONF_VERSION = "1"
+
+#
+# Disk space monitor, warning when the disk space or the amount of inode
+# is running low, it is enabled when BB_DISKMON_DIRS is set.
+#
+# Set the directories to monitor for disk usage, if more than one
+# directories are in the same device, then only one directory would be
+# warned since the monitor is based on the device.
+#
+BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR}"
+#
+# Set the minimum amount of disk space to warn, the unit can be G,
+# M or K, but do not use GB, MB or KB (B is not needed).
+# The default vaule is 1G if not set.
+#
+#BB_DISKMON_MINSPACE = "1G"
+#
+# Set the minimum amount of inodes to warn, the unit can be G, M or K.
+# The default value is 1K if not set.
+#
+#BB_DISKMON_MININODES = "1K"
+#
+# Set the monitor frequency, the unit can be h(hour), m(minute) or
+# s(second), when no unit is specified, the default unit is s(second),
+# for example, "60" means 60 seconds.
+# The default value is 60s if not set.
+#
+#BB_DISKMON_INTERVAL = "60s"

Signature · 1 line

1.7.4.1

Robert Yang

author

2011-12-15 11:05 UTC #3 lore raw

In reply to Robert Yang

Quoted 11 lines · Robert Yang

Here are the test info:
1) set the following variables in conf/local.conf for testing:
BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1" BB_DISKMON_MINSPACE = "200G"
BB_DISKMON_MININODES = "20M"
BB_DISKMON_INTERVAL = "30s"
The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in my build, so I added another direcotory /home/lyang1 to test the ability of monitoring multi mount points.

Quoted 6 lines

2) bitbake core-image-sato
WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)

And the messages will print every 30 seconds(the BB_DISKMON_INTERVAL), until the build stops.

// Robert

Quoted 27 lines

3) Aalso have tested about the following items:
a) Use ctrl-c to stop the monitor
b) The monitor should stop when the build stops
c) Test the space unit G, B, K(case in-sensitive) and the time unit h, m, s(case-sensitive)
// Robert
The following changes since commit 9d136b2db8f906c562cbdb23a9b238f0e237074b:
gdk-pixbuf: Ensure the binaries can be relocated (2011-12-13 18:00:25 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib robert/disk-space-monitor
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/disk-space-monitor
Robert Yang (2):
Disk space monitoring
Add config sample for disk space monitoring
  bitbake/bin/bitbake               |   12 +++
  bitbake/lib/bb/monitordisk.py     |  153 +++++++++++++++++++++++++++++++++++++
  meta-yocto/conf/local.conf.sample |   28 +++++++
  3 files changed, 193 insertions(+), 0 deletions(-)
create mode 100644 bitbake/lib/bb/monitordisk.py

Mark Hatle

2011-12-15 16:23 UTC #4 lore raw

In reply to Robert Yang

Is the disk space monitoring implemented in the correct place in the system?

I would have expected this to have been implemented in the meta/class files, both in the initial sanity checking and also in something that can watch the events and doing the repetition/rechecking as necessary.

(BTW the code looks fine to me as implemented, I'm just not sure if it belongs in bitbake itself.)

--Mark

Quoted 44 lines · Robert Yang

Here are the test info:
1) set the following variables in conf/local.conf for testing:
BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1" BB_DISKMON_MINSPACE = "200G"
BB_DISKMON_MININODES = "20M"
BB_DISKMON_INTERVAL = "30s"
The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in my build, so I added another direcotory /home/lyang1 to test the ability of monitoring multi mount points.
2) bitbake core-image-sato
WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)
3) Aalso have tested about the following items:
a) Use ctrl-c to stop the monitor
b) The monitor should stop when the build stops
c) Test the space unit G, B, K(case in-sensitive) and the time unit h, m, s(case-sensitive)
// Robert
The following changes since commit 9d136b2db8f906c562cbdb23a9b238f0e237074b:
gdk-pixbuf: Ensure the binaries can be relocated (2011-12-13 18:00:25 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib robert/disk-space-monitor
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/disk-space-monitor
Robert Yang (2):
Disk space monitoring
Add config sample for disk space monitoring
  bitbake/bin/bitbake               |   12 +++
  bitbake/lib/bb/monitordisk.py     |  153 +++++++++++++++++++++++++++++++++++++
  meta-yocto/conf/local.conf.sample |   28 +++++++
  3 files changed, 193 insertions(+), 0 deletions(-)
create mode 100644 bitbake/lib/bb/monitordisk.py

Robert Yang

author

2011-12-16 03:32 UTC #5 lore raw

Hi Mark,

Thanks for your replying, Richard suggested we add this to bitbake itself, here is his reply in 8 months ago:

As for disk monitoring, I'd suggest we just add this to bitbake itself as it doesn't really fit the task model. It can be controlled via some variables like:

# Set the directories to monitor for disk usage
BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR}"
# Set the minimum amount of disk space to exit on
BB_DISKMON_MINSPACE = "1GB"
# Set the monitor frequency
BB_DISKMON_INTERVAL = "60s"

and if the variables are not set, it doesn't get activated.

// Robert

Quoted 63 lines · Mark Hatle

Is the disk space monitoring implemented in the correct place in the system?
I would have expected this to have been implemented in the meta/class files, both in the initial sanity checking and also in something that can watch the events and doing the repetition/rechecking as necessary.
(BTW the code looks fine to me as implemented, I'm just not sure if it belongs in bitbake itself.)
--Mark
Quoted 38 lines · Robert Yang
Here are the test info:
1) set the following variables in conf/local.conf for testing:
BB_DISKMON_DIRS = "${TMPDIR} ${DL_DIR} ${SSTATE_DIR} /home/lyang1" BB_DISKMON_MINSPACE = "200G"
BB_DISKMON_MININODES = "20M"
BB_DISKMON_INTERVAL = "30s"
The TMPDIR, DL_DIR and SSTATE_DIR are in the same disk(/dev/sdb4) in my build, so I added another direcotory /home/lyang1 to test the ability of monitoring multi mount points.
2) bitbake core-image-sato
WARNING: The free space of /dev/sdb5 is running low (25.915GB left)
WARNING: The free inode of /dev/sdb5 is running low (1788.202K left)
WARNING: The free space of /dev/sdb4 is running low (189.309GB left)
WARNING: The free inode of /dev/sdb4 is running low (18005.443K left)
3) Aalso have tested about the following items:
a) Use ctrl-c to stop the monitor
b) The monitor should stop when the build stops
c) Test the space unit G, B, K(case in-sensitive) and the time unit h, m, s(case-sensitive)
// Robert
The following changes since commit 9d136b2db8f906c562cbdb23a9b238f0e237074b:
gdk-pixbuf: Ensure the binaries can be relocated (2011-12-13 18:00:25 +0000)
are available in the git repository at:
git://git.pokylinux.org/poky-contrib robert/disk-space-monitor
http://git.pokylinux.org/cgit.cgi/poky-contrib/log/?h=robert/disk-space-monitor
Robert Yang (2):
Disk space monitoring
Add config sample for disk space monitoring
Quoted 6 lines
bitbake/bin/bitbake | 12 +++
bitbake/lib/bb/monitordisk.py | 153 +++++++++++++++++++++++++++++++++++++ meta-yocto/conf/local.conf.sample | 28 +++++++
3 files changed, 193 insertions(+), 0 deletions(-)
create mode 100644 bitbake/lib/bb/monitordisk.py
_______________________________________________
bitbake-devel mailing list
[email protected]
http://lists.linuxtogo.org/cgi-bin/mailman/listinfo/bitbake-devel