Taking hera down to deal with the load issues
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Hey Everyone,
Ok so since I'm back from vacation, and things don't seem to be getting better, I'm going to take hera/master offline in about 30 minutes or so (5pm PDT or Tue Aug 17 0:0:0 UTC 2011). This is going to involve a base OS upgrade, and a full on new kernel.
This should get the loads back into what we are expecting, and give me a chance to more thoroughly investigate the panics from last week. If you've got questions or comments, please give me a shout as soon as possible.
- - John 'Warthog9' Hawley
Chief Kernel.org Administrator
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk5K/PkACgkQ/E3kyWU9dieV8wCcCUmHmIYqeqK0eBUt7f3ne07J 994An3MrErlAjOLx/zkpB6T1dNsHVLtC
=CSPj
-----END PGP SIGNATURE-----
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
Sorry about the delay, the upgrade took longer than expected and the initial kernel didn't actually solve the load issue completely. I've since installed a very slightly modified build of the initial kernel and had to reboot hera twice more (both to the same kernel, but I again ran lsof -p <pid> and had Linux panic on me. A feature that has very consistently bitten me on a number of kernels and a number of systems lately).
This second kernel is at least, so far, acting a lot closer to what I'd expect from hera, but the disk cache is still filling. I'm going to keep an eye on it for a bit longer tonight. As a note, the only difference between the first 3.1-rc1 kernel and the second, that is currently running, was the explicit disabling of IMA and recompiling the kernel.
I'm going to give the all clear at this point, I won't be doing any more reboots without an additional e-mail and notice. If you see any problems or issue, or you catch that the loads are spiking up into the stratosphere, please let me know.
- - John 'Warthog9' Hawley
Quoted 7 lines · J.H.
Hey Everyone,
Ok so since I'm back from vacation, and things don't seem to be getting better, I'm going to take hera/master offline in about 30 minutes or so (5pm PDT or Tue Aug 17 0:0:0 UTC 2011). This is going to involve a base OS upgrade, and a full on new kernel.
Quoted 7 lines
This should get the loads back into what we are expecting, and give me a chance to more thoroughly investigate the panics from last week. If you've got questions or comments, please give me a shout as soon as possible.
- John 'Warthog9' Hawley
Chief Kernel.org Administrator
_______________________________________________
Users mailing list
[email protected]
http://linux.kernel.org/mailman/listinfo/users
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.4.11 (GNU/Linux)
Comment: Using GnuPG with Fedora - http://enigmail.mozdev.org/
iEYEARECAAYFAk5LdmQACgkQ/E3kyWU9dic7rgCcDuBBmmfmNeyqs/zC1t9cByzy 5J0AnRcfO4701ix0HAr+KVmNKVedl38O
=FQPl
-----END PGP SIGNATURE-----
Hi,
Quoted 8 lines · J.H.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
This second kernel is at least, so far, acting a lot closer to what I'd expect from hera, but the disk cache is still filling. I'm going to keep an eye on it for a bit longer tonight. As a note, the only difference between the first 3.1-rc1 kernel and the second, that is currently running, was the explicit disabling of IMA and recompiling the kernel.
What do you exactly mean by "disk cache is still filling"? 3.1-rc1 has
quite some changes in writeback code so it's possible something went wrong...
Honza
Signature · 2 lines
Jan Kara <[email protected]> SUSE Labs, CR
Quoted 11 lines · Jan Kara
Hi,
Quoted 8 lines · J.H.
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1
This second kernel is at least, so far, acting a lot closer to what I'd expect from hera, but the disk cache is still filling. I'm going to keep an eye on it for a bit longer tonight. As a note, the only difference between the first 3.1-rc1 kernel and the second, that is currently running, was the explicit disabling of IMA and recompiling the kernel.
Quoted 3 lines
What do you exactly mean by "disk cache is still filling"? 3.1-rc1 has
quite some changes in writeback code so it's possible something went wrong...
Mostly that it takes time to read the bits off of the spinning rust and have it cached into memory. Not specifically that there's something wrong with the writeback, I'm at least not seeing one I believe.
- John 'Warthog9' Hawley
From: "J.H." <[email protected]>
Date: Wed, 17 Aug 2011 01:05:56 -0700
Quoted 6 lines
This second kernel is at least, so far, acting a lot closer to what I'd expect from hera, but the disk cache is still filling. I'm going to keep an eye on it for a bit longer tonight. As a note, the only difference between the first 3.1-rc1 kernel and the second, that is currently running, was the explicit disabling of IMA and recompiling the kernel.
I wonder if the IMA bits are generating crypto hashes for every file modification done on the machine.
Quoted 10 lines · David Miller
From: "J.H." <[email protected]>
Date: Wed, 17 Aug 2011 01:05:56 -0700Quoted 6 lines
This second kernel is at least, so far, acting a lot closer to what I'd expect from hera, but the disk cache is still filling. I'm going to keep an eye on it for a bit longer tonight. As a note, the only difference between the first 3.1-rc1 kernel and the second, that is currently running, was the explicit disabling of IMA and recompiling the kernel.
Quoted 2 lines
I wonder if the IMA bits are generating crypto hashes for every file modification done on the machine.
Possibly, but IMA doesn't recalculate the hash for every modification. Without the boot command line parameter 'ima_tcb', nothing should be measured. With 'ima_tcb', only files accessed by root are measured and then only re-measured, if flushed from the inode cache or has been marked changed, which occurs on __fput(), not every write.
thanks,
Mimi