Skip to contentSkip to conversation

storage-logger: Recording changes to the udev database

DISCUSSION Alasdair G Kergon 2 participants 5 messages Mar 26, 2020 – Mar 27, 2020 Device mapper

lore
Share & tools
lore

Alasdair G Kergon

author maintainer
#0 lore raw

I'm experimenting with ways of recording changes to the udev database so you can look back at the history of the storage stack on a particular machine. This is still a work-in-progress, but it's reached a point where I'd like more people to try it out.

I've written a shell script that records data related to storage uevents in the system journal and a perl script that helps you to interrogate this data later to create a representation of the storage components.

If you're interested, please try this out and let me know if you think pursing this approach further would lead to something that you would use and distributions should ship.

Source code:
https://github.com/lvmteam/storage-logger
Fedora builds:
https://copr.fedorainfracloud.org/coprs/agk/storage-logger/build/1320735/
Presentation:
https://fosdem.org/2020/schedule/event/storage_logger/

Storage-logger
==============
The storage-logger project maintains a record of the storage configuration of a linux system as it changes over time. The idea is to provide a quick way to check the state of a system at times in the past.

Logging
=======
The initial logging implementation is triggered by storage uevents and consists of two components:

1. A new udev rule file, 99-zzz-storage-logger.rules, which runs after all the other rules have run and invokes:

2. A script, udev_storage_logger.sh, that captures relevant information about devices that changed and stores it in the system journal.

The effect is to log relevant uevents plus some supplementary information. It does not yet handle filesystem-related events.

Reporting
=========
Two methods to query the data are offered:

1. journalctl
Reports the raw data using simple filtering.
Data is tagged with the identifier UDEVLOG and retrievable as key-value pairs.
All the captured data:
journalctl -t UDEVLOG --output verbose
or as JSON:
journalctl -t UDEVLOG --output json
Between a time range:
--since 'YYYY-MM-DD HH:MM:SS'
--until 'YYYY-MM-DD HH:MM:SS'
Other filtering features are described in the man page.

2. lsblkj
This wrapper creates a dummy system environment that "looks like" the system did at a specified earlier time and then runs lsblk against it. It accepts --since and --until arguments to pass to journalctl to select the desired data, and passes other arguments controlling the output format to the real lsblk.
Use --verbose to watch it setting up the temporary environment . Use --dry-run to see what it would do without actually doing it. Use --git to create a git repository recording the changes over time.

Alasdair

Signature · 1 line
[email protected]
#1 lore raw
Quoted 5 lines · Alasdair G Kergon

I'm experimenting with ways of recording changes to the udev database so you can look back at the history of the storage stack on a particular machine. This is still a work-in-progress, but it's reached a point where I'd like more people to try it out.

Quoted 7 lines

I've written a shell script that records data related to storage uevents in the system journal and a perl script that helps you to interrogate this data later to create a representation of the storage components.

If you're interested, please try this out and let me know if you think pursing this approach further would lead to something that you would use and distributions should ship.

Alasdair,

I have downloaded and installed in a Debian 9 environment.

There are a couple of differences that I have seen so far.

First, most importantly, the udev rules are in /etc/udev/rules.d.

Secondly, it appears that the journalctl command does not like the output fields option.

I will install this on another machine that has a bit more device activity, and comment later.

Brian

Quoted 10 lines

Alasdair

Signature · 7 lines
[email protected]

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

#2 lore raw
In reply to Alasdair G Kergon
Quoted 5 lines · Alasdair G Kergon

I'm experimenting with ways of recording changes to the udev database so you can look back at the history of the storage stack on a particular machine. This is still a work-in-progress, but it's reached a point where I'd like more people to try it out.

Quoted 7 lines

I've written a shell script that records data related to storage uevents in the system journal and a perl script that helps you to interrogate this data later to create a representation of the storage components.

If you're interested, please try this out and let me know if you think pursing this approach further would lead to something that you would use and distributions should ship.

Quick note, Alasdair.

In your instructions, you say to put the "working" script in /usr/sbin, while in the udev rule, it is in /sbin/

I tried adding and removing a USB-connected drive, but did not see any response in the journal. Should I have seen something, or are those not seen?

Thanks,
Brian

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/

author maintainer
#3 lore raw
Quoted 2 lines · Brian McCullough

In your instructions, you say to put the "working" script in /usr/sbin, while in the udev rule, it is in /sbin/

In Fedora they are the same.

Quoted 3 lines

I tried adding and removing a USB-connected drive, but did not see any response in the journal. Should I have seen something, or are those not seen?

You should test the script in isolation in case there are any other differences on Debian. Change the top line to '...bash -x' so you can watch it, set some environment variables (like ACTION=add) and run it directly.

Alasdair

#4 lore raw
Quoted 5 lines · Alasdair G Kergon
Quoted 2 lines · Brian McCullough

In your instructions, you say to put the "working" script in /usr/sbin, while in the udev rule, it is in /sbin/

In Fedora they are the same.

Ah. So they are aliases.

Quoted 8 lines
Quoted 3 lines

I tried adding and removing a USB-connected drive, but did not see any response in the journal. Should I have seen something, or are those not seen?

You should test the script in isolation in case there are any other differences on Debian. Change the top line to '...bash -x' so you can watch it, set some environment variables (like ACTION=add) and run it directly.

Thank you. I will do so over the weekend, and report.

Quoted 1 line

Alasdair

Brian

_______________________________________________
linux-lvm mailing list
[email protected]
https://www.redhat.com/mailman/listinfo/linux-lvm
read the LVM HOW-TO at http://tldp.org/HOWTO/LVM-HOWTO/