Skip to contentSkip to conversation

Reduce noise about missing devices.

RFC PATCH New Petr Rockai 3 participants 5 messages Apr 27, 2010 – May 7, 2010

lore
Share & tools
lore
#0 lore raw

RFC PATCH 1/1 Reduce noise about missing devices.

Hi,

this is not intended to be directly applied, but more to start a discussion. One problem with our handling of missing devices is the huge amount of log noise each missing device produces. In a single run of lvconvert --repair, I get multiple screens of "read error" and "Couldn't find device with uuid"...

The attached patch reduces that noise by only reporting these things once per run. I have used two different approaches for read errors and for uuid's. Neither is perfect...

Another possible approach (one that I would probably prefer) is to add a global log_error_once to log.c that would maintain a dm_hash_table of error messages and downgrade already-reported errors to log_debug or log_very_verbose automatically. A few error paths could use this.

The downside of all these approaches is that this is sort of global knowledge, but I guess that's hard to avoid. Presumably, adding a call to erase the hashtable to _init_logging would solve most of the problems with this. I suspect that threading a toolcontext pointer through everything so it could be passed to logging functions is not a viable option.

About the device read errors, I am ambivalent what exactly to report as log_error. It makes sense to only report first error per device, but it also makes sense to report all unique errors per device. The global log_error_once approach would give the latter, which I think may be actually preferable.

Opinions?

Yours,
Petr.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: reduce-dup-errors.diff
Type: text/x-diff
Size: 3559 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20100427/824c0fca/attachment.bin>

#1 lore raw

RFC PATCH 1/1 Reduce noise about missing devices.

Hi,

Quoted 4 lines · Petr Rockai

Another possible approach (one that I would probably prefer) is to add a global log_error_once to log.c that would maintain a dm_hash_table of error messages and downgrade already-reported errors to log_debug or log_very_verbose automatically. A few error paths could use this.

here's a patch that implements this. I think this is fairly reasonable and could be checked in. Some minor function renaming might be necessary.

Yours,
Petr.

-------------- next part --------------
A non-text attachment was scrubbed...
Name: log_error_once.diff
Type: text/x-diff
Size: 5980 bytes
Desc: not available
URL: <http://listman.redhat.com/archives/lvm-devel/attachments/20100428/a4a1610f/attachment.bin>

#2 lore raw

RFC PATCH 1/1 Reduce noise about missing devices.

In reply to Petr Rockai
Quoted 5 lines · Peter Rockai

this is not intended to be directly applied, but more to start a discussion. One problem with our handling of missing devices is the huge amount of log noise each missing device produces. In a single run of lvconvert --repair, I get multiple screens of "read error" and "Couldn't find device with uuid"...

Well my first question before suppressing messages would be: Are all the repeated accesses that generate the messages actually necessary? Or are the messages just a symptom of other problems which should be addressed instead?

Alasdair

#3 lore raw

RFC PATCH 1/1 Reduce noise about missing devices.

Quoted 11 lines · Alasdair G Kergon
Quoted 5 lines · Peter Rockai

this is not intended to be directly applied, but more to start a discussion. One problem with our handling of missing devices is the huge amount of log noise each missing device produces. In a single run of lvconvert --repair, I get multiple screens of "read error" and "Couldn't find device with uuid"...

Well my first question before suppressing messages would be: Are all the repeated accesses that generate the messages actually necessary? Or are the messages just a symptom of other problems which should be addressed instead?

I checked how many times a broken device was accessed by the lvconvert command when 'lvconvert --repair' is executed using the following lvm environment:

VG: 3PVs
LV: mirror composed of 2 legs and 1 disk log

vg00-lv00 (253:3)

   |-vg00-lv00_mimage_1 (253:2)
   |  `- (8:48)
   |-vg00-lv00_mimage_0 (253:1)
   |  `- (8:32)
   `-vg00-lv00_mlog (253:0)
      `- (8:64)

When a primary leg is broken, 'lvconvert --repair' accesses the primary leg by '47 times.' I think what we should do is to suppress 'access to a broken device' instead of suppressing 'displaying error messages.'

Suppress error messages might be friendly to some users, but other users might not notice how often a broken device is accessed by the lvconvert command.

Do we need to access a broken device so many times? If lvm commands detects errors on a device a few times, my opinion is that lvm commands should quit accessing the device.

Thanks,
Taka

#4 lore raw

RFC PATCH 1/1 Reduce noise about missing devices.

Quoted 3 lines · Takahiro Yasui

Do we need to access a broken device so many times? If lvm commands detects errors on a device a few times, my opinion is that lvm commands should quit accessing the device.

Agree.

Alasdair