Skip to contentSkip to conversation

Query: Best way to know if a watchdog is active (kicked)

DISCUSSION Pratyush Anand 3 participants 13 messages Aug 18, 2015 – Aug 21, 2015 Watchdog device drivers

lore
Share & tools
lore
#0 lore raw

Hi,

I am looking for the best way to know if a watchdog has been kicked and active.

I can see a way is to read timeout(WDIOC_GETTIMEOUT) and timeleft( WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.

But what if we tried to read timeleft just in time when watchdog daemon/or some other application had kicked it. May be we read timeleft twice at the interval of 1 sec.

Please let me know if there is any other alternative which could be a better way to know if watchdog is active? Or may be it would be good to implement an ioctl WDIOC_ACTIVE?

Thanks

~Pratyush

maintainer
#1 lore raw
Quoted 7 lines · Pratyush Anand

Hi,

I am looking for the best way to know if a watchdog has been kicked and active.

I can see a way is to read timeout(WDIOC_GETTIMEOUT) and timeleft( WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.

Quoted 8 lines

But what if we tried to read timeleft just in time when watchdog daemon/or some other application had kicked it. May be we read timeleft twice at the interval of 1 sec.

Please let me know if there is any other alternative which could be a better way to know if watchdog is active? Or may be it would be good to implement an ioctl WDIOC_ACTIVE?

Normally the watchdog is active if the watchdog device is open, unless the application controlling it explicitly disabled it with WDIOC_SETOPTIONS. Therefore, the controlling application should always know the status. A different application can not open the watchdog device, so it won't be able to get its status using an ioctl anyway.

Why is that insufficient ?

Guenter

#2 lore raw

Hi Guenter,

Thanks a lot for your quick reply.

Quoted 17 lines · Guenter Roeck
Quoted 7 lines · Pratyush Anand

Hi,

I am looking for the best way to know if a watchdog has been kicked and active.

I can see a way is to read timeout(WDIOC_GETTIMEOUT) and timeleft( WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.

Quoted 8 lines

But what if we tried to read timeleft just in time when watchdog daemon/or some other application had kicked it. May be we read timeleft twice at the interval of 1 sec.

Please let me know if there is any other alternative which could be a better way to know if watchdog is active? Or may be it would be good to implement an ioctl WDIOC_ACTIVE?

Quoted 5 lines

Normally the watchdog is active if the watchdog device is open, unless the application controlling it explicitly disabled it with WDIOC_SETOPTIONS. Therefore, the controlling application should always know the status. A different application can not open the watchdog device, so it won't be able to get its status using an ioctl anyway.

Yes, A different application can not open in parallel, but can open once the previous application has closed it. For example this is what I see:

--------------------------------------------------------------

# cat /dev/watchdog1 ; sleep 5; wdctl /dev/watchdog1
cat: /dev/watchdog1: Invalid argument
wdctl: write failed: Invalid argument
Device:        /dev/watchdog1
Identity:      iTCO_wdt [version 0]
Timeout:       30 seconds
Timeleft:      24 seconds
FLAG           DESCRIPTION               STATUS BOOT-STATUS
KEEPALIVEPING  Keep alive ping reply          0           0
MAGICCLOSE     Supports magic close char      0           0
SETTIMEOUT     Set timeout (in seconds)       0           0
--------------------------------------------------------------
So, cat opened it and kicked it as well. But, it could not stop it as magic
character "V" had not not received. Therefore, when wdctl opened and read
Timeleft, it was different than Timeout.
Quoted 2 lines

Why is that insufficient ?

Well, let me explain the use case. Consider the situation when:
-- A system has activated its watchdog to take care of software hang. So, when software has hanged, wdt causes to reboot, else it is kicked again before timeout.
-- The same system has also activated kdump(kdump is a method to reboot to a minimal stable secondary kernel in case of kernel crash). Now when wdt was still active, there was a kernel crash and system booted to a secondary stable kernel which copies crash related data to a safe location. Since, wdt was active so before the desired process could complete in secondary kernel, hardware rebooted. -- So, the watchdog device need to be stoped in secondary kernel as early as possible. Loading of driver/module itself stops a kicked device. So, if there could be a way to know active wdt from kernel, then the two daemon (one which manages watchdog and other which manages kdump) can play independently, and kdump daemon can correctly program a kdump file system to load relevant watchdog module as early as possible.
-- Current distro implementations loads all the watchdog devices driver module in secondary kernel, which is not nice (secondary kdump kernel should be as minimal as possible).

~Pratyush

#3 lore raw

Hi, Pratyush

A comment from my understanding about the background..

Quoted 51 lines · Pratyush Anand

Hi Guenter,

Thanks a lot for your quick reply.

Quoted 17 lines · Guenter Roeck
Quoted 7 lines · Pratyush Anand

Hi,

I am looking for the best way to know if a watchdog has been kicked and active.

I can see a way is to read timeout(WDIOC_GETTIMEOUT) and timeleft( WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.

Quoted 8 lines

But what if we tried to read timeleft just in time when watchdog daemon/or some other application had kicked it. May be we read timeleft twice at the interval of 1 sec.

Please let me know if there is any other alternative which could be a better way to know if watchdog is active? Or may be it would be good to implement an ioctl WDIOC_ACTIVE?

Quoted 5 lines

Normally the watchdog is active if the watchdog device is open, unless the application controlling it explicitly disabled it with WDIOC_SETOPTIONS. Therefore, the controlling application should always know the status. A different application can not open the watchdog device, so it won't be able to get its status using an ioctl anyway.

Yes, A different application can not open in parallel, but can open once the previous application has closed it. For example this is what I see:

--------------------------------------------------------------

# cat /dev/watchdog1 ; sleep 5; wdctl /dev/watchdog1
cat: /dev/watchdog1: Invalid argument
wdctl: write failed: Invalid argument
Device:        /dev/watchdog1
Identity:      iTCO_wdt [version 0]
Timeout:       30 seconds
Timeleft:      24 seconds
FLAG           DESCRIPTION               STATUS BOOT-STATUS
KEEPALIVEPING  Keep alive ping reply          0           0
MAGICCLOSE     Supports magic close char      0           0
SETTIMEOUT     Set timeout (in seconds)       0           0
--------------------------------------------------------------
So, cat opened it and kicked it as well. But, it could not stop it as magic
character "V" had not not received. Therefore, when wdctl opened and read
Timeleft, it was different than Timeout.
Quoted 2 lines

Why is that insufficient ?

Quoted 10 lines

Well, let me explain the use case. Consider the situation when:
-- A system has activated its watchdog to take care of software hang. So, when software has hanged, wdt causes to reboot, else it is kicked again before timeout.
-- The same system has also activated kdump(kdump is a method to reboot to a minimal stable secondary kernel in case of kernel crash). Now when wdt was still active, there was a kernel crash and system booted to a secondary stable kernel which copies crash related data to a safe location. Since, wdt was active so before the desired process could complete in secondary kernel, hardware rebooted. -- So, the watchdog device need to be stoped in secondary kernel as early as

Either stop it or continue kicking before timeout are fine.

Quoted 5 lines

possible. Loading of driver/module itself stops a kicked device. So, if there could be a way to know active wdt from kernel, then the two daemon (one which manages watchdog and other which manages kdump) can play independently, and kdump daemon can correctly program a kdump file system to load relevant watchdog module as early as possible.

Some drivers like iTCO_wdt can stop it during module loading. But I'm not sure all drivers work. At least under 'nowayout' mode.

So the better way (still is a best effort solution though) should be kicking it again before timeout.

Quoted 5 lines

-- Current distro implementations loads all the watchdog devices driver module in secondary kernel, which is not nice (secondary kdump kernel should be as minimal as possible).

~Pratyush

Thanks
Dave

#4 lore raw

Hi Dave,

Quoted 3 lines · Dave Young

Hi, Pratyush

A comment from my understanding about the background..

I agree with your understanding.
Thanks a lot for clarifying it more.

~Pratyush

maintainer
#5 lore raw
In reply to Pratyush Anand
Quoted 22 lines · Pratyush Anand

Hi Guenter,

Thanks a lot for your quick reply.

Quoted 17 lines · Guenter Roeck
Quoted 15 lines · Pratyush Anand

Hi,

I am looking for the best way to know if a watchdog has been kicked and active.

I can see a way is to read timeout(WDIOC_GETTIMEOUT) and timeleft( WDIOC_GETTIMELEFT). If they do not match, it means that wdt is active.

But what if we tried to read timeleft just in time when watchdog daemon/or some other application had kicked it. May be we read timeleft twice at the interval of 1 sec.

Please let me know if there is any other alternative which could be a better way to know if watchdog is active? Or may be it would be good to implement an ioctl WDIOC_ACTIVE?

Quoted 9 lines
Quoted 5 lines

Normally the watchdog is active if the watchdog device is open, unless the application controlling it explicitly disabled it with WDIOC_SETOPTIONS. Therefore, the controlling application should always know the status. A different application can not open the watchdog device, so it won't be able to get its status using an ioctl anyway.

Yes, A different application can not open in parallel, but can open once the previous application has closed it. For example this is what I see:

Just by opening the watchdog it is going to be activated. So all you end up knowing is if the watchdog was active before, but in either case it will now be active. That doesn't seem to be very helpful.

Quoted 30 lines

--------------------------------------------------------------

# cat /dev/watchdog1 ; sleep 5; wdctl /dev/watchdog1
cat: /dev/watchdog1: Invalid argument
wdctl: write failed: Invalid argument
Device:        /dev/watchdog1
Identity:      iTCO_wdt [version 0]
Timeout:       30 seconds
Timeleft:      24 seconds
FLAG           DESCRIPTION               STATUS BOOT-STATUS
KEEPALIVEPING  Keep alive ping reply          0           0
MAGICCLOSE     Supports magic close char      0           0
SETTIMEOUT     Set timeout (in seconds)       0           0
--------------------------------------------------------------
So, cat opened it and kicked it as well. But, it could not stop it as magic
character "V" had not not received. Therefore, when wdctl opened and read
Timeleft, it was different than Timeout.
Quoted 2 lines

Why is that insufficient ?

Well, let me explain the use case. Consider the situation when:
-- A system has activated its watchdog to take care of software hang. So, when software has hanged, wdt causes to reboot, else it is kicked again before timeout.
-- The same system has also activated kdump(kdump is a method to reboot to a minimal stable secondary kernel in case of kernel crash). Now when wdt was still active, there was a kernel crash and system booted to a secondary stable kernel which copies crash related data to a safe location. Since, wdt was active so before the desired process could complete in secondary kernel, hardware rebooted. -- So, the watchdog device need to be stoped in secondary kernel as early as

Quoted 8 lines

possible. Loading of driver/module itself stops a kicked device. So, if there could be a way to know active wdt from kernel, then the two daemon (one which manages watchdog and other which manages kdump) can play independently, and kdump daemon can correctly program a kdump file system to load relevant watchdog module as early as possible.
-- Current distro implementations loads all the watchdog devices driver module in secondary kernel, which is not nice (secondary kdump kernel should be as minimal as possible).

If the watchdog was active in the original kernel, it needs to be activated in the crashdump kernel. I don't see a way around that. I can not comment on "loads all the watchdog device driver module in the secondary kernel". I would think it should be known which module to load.

One possible solution to your problem might be to have some sysfs attributes associated with watchdog devices, one of which would be its state. That has been on my mental task list for a while, but unfortunately I never found the time to implement it.

Guenter

#6 lore raw
Quoted 1 line · Guenter Roeck

On 08/17/2015 11:57 PM, Pratyush Anand wrote:

[...]

Quoted 5 lines

One possible solution to your problem might be to have some sysfs attributes associated with watchdog devices, one of which would be its state. That has been on my mental task list for a while, but unfortunately I never found the time to implement it.

Well, I will give an attempt to implement state as a sysfs attributes associated with watchdog devices, having two values ACTIVE and INACTIVE. Will send the modification for review.
We can always add latter on more attributes and more state values.

Thanks a lot for your suggestion.

~Pratyush

maintainer
#7 lore raw
Quoted 11 lines · Pratyush Anand
Quoted 1 line · Guenter Roeck

On 08/17/2015 11:57 PM, Pratyush Anand wrote:

[...]

Quoted 5 lines

One possible solution to your problem might be to have some sysfs attributes associated with watchdog devices, one of which would be its state. That has been on my mental task list for a while, but unfortunately I never found the time to implement it.

Quoted 5 lines

Well, I will give an attempt to implement state as a sysfs attributes associated with watchdog devices, having two values ACTIVE and INACTIVE. Will send the modification for review.
We can always add latter on more attributes and more state values.

Sounds good. We should have a 'name' attribute as well. We'll also need 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'. Not sure what else.

The attributes should be implemented as class attributes, to ensure that they are created and removed automatically. That may require the watchdog class to be static instead of a pointer.

Thanks,
Guenter

#8 lore raw
Quoted 7 lines · Guenter Roeck

Sounds good. We should have a 'name' attribute as well. We'll also need 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'. Not sure what else.

The attributes should be implemented as class attributes, to ensure that they are created and removed automatically. That may require the watchdog class to be static instead of a pointer.

Should n't attributes be implemented as device attribute as they are properties of each device. Under the class watchdog_class, we can have multiple device on a system, and these attributes would be different for different devices. So, what I am thinking is to create device in __watchdog_register_device using device_create_with_groups instead of device_create. Here we can link a attribute_group with all attributes you suggested.

~Pratyush

maintainer
#9 lore raw
Quoted 9 lines · Pratyush Anand
Quoted 7 lines · Guenter Roeck

Sounds good. We should have a 'name' attribute as well. We'll also need 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'. Not sure what else.

The attributes should be implemented as class attributes, to ensure that they are created and removed automatically. That may require the watchdog class to be static instead of a pointer.

Quoted 7 lines

Should n't attributes be implemented as device attribute as they are properties of each device. Under the class watchdog_class, we can have multiple device on a system, and these attributes would be different for different devices. So, what I am thinking is to create device in __watchdog_register_device using device_create_with_groups instead of device_create. Here we can link a attribute_group with all attributes you suggested.

Class attibutes are created automatically for each device registered which is a member of the class. They are attached to the device, not to the class. In other words, the infrastructure creates the attributes, the driver doesn't have to do it, and there is no race condition where the device exists but the attributes are not yet there.

Not sure what you mean with "different". Can you elaborate ?

Guenter

#10 lore raw
Quoted 17 lines · Guenter Roeck
Quoted 9 lines · Pratyush Anand
Quoted 7 lines · Guenter Roeck

Sounds good. We should have a 'name' attribute as well. We'll also need 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'. Not sure what else.

The attributes should be implemented as class attributes, to ensure that they are created and removed automatically. That may require the watchdog class to be static instead of a pointer.

Quoted 7 lines

Should n't attributes be implemented as device attribute as they are properties of each device. Under the class watchdog_class, we can have multiple device on a system, and these attributes would be different for different devices. So, what I am thinking is to create device in __watchdog_register_device using device_create_with_groups instead of device_create. Here we can link a attribute_group with all attributes you suggested.

Quoted 2 lines

Class attibutes are created automatically for each device registered which is a member of the class. They are attached to the device, not to the class.

Not sure, if we both are referring to same thing..
So, I did a basic testing. I changed "static struct class *watchdog_class;" to "tatic struct class watchdog_class" and other than .name and .owner I added .class_attrs and then further added one node "state" in this class_attrs.

I have two watchdog in my system, so I was expecting /sys/class/watchdog/watchdog0/state and /sys/class/watchdog/watchdog1/state. But I see that there is only /sys/class/watchdog/state with the above approach.

Quoted 3 lines

In other words, the infrastructure creates the attributes, the driver doesn't have to do it, and there is no race condition where the device exists but the attributes are not yet there.

Yes, I agree that individual driver does not need to do anything. watchdog_core.c and watchdog_dev.c will take care for each of the device registered with it. I am able to do that, but not using .class_attrs, rather using device_create_with_groups.

Quoted 2 lines

Not sure what you mean with "different". Can you elaborate ?

As I said above, with the "different" I meant that we should have state in each /sys/class/watchdog/watchdogn.

~Pratyush

maintainer
#11 lore raw
Quoted 18 lines · Pratyush Anand
Quoted 17 lines · Guenter Roeck
Quoted 16 lines · Pratyush Anand
Quoted 7 lines · Guenter Roeck

Sounds good. We should have a 'name' attribute as well. We'll also need 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'. Not sure what else.

The attributes should be implemented as class attributes, to ensure that they are created and removed automatically. That may require the watchdog class to be static instead of a pointer.

Should n't attributes be implemented as device attribute as they are properties of each device. Under the class watchdog_class, we can have multiple device on a system, and these attributes would be different for different devices. So, what I am thinking is to create device in __watchdog_register_device using device_create_with_groups instead of device_create. Here we can link a attribute_group with all attributes you suggested.

Quoted 8 lines
Quoted 2 lines

Class attibutes are created automatically for each device registered which is a member of the class. They are attached to the device, not to the class.

Not sure, if we both are referring to same thing..
So, I did a basic testing. I changed "static struct class *watchdog_class;" to "tatic struct class watchdog_class" and other than .name and .owner I added .class_attrs and then further added one node "state" in this class_attrs.

If you use .dev_groups instead of .class_attrs it should work.

Guenter

#12 lore raw
Quoted 22 lines · Guenter Roeck
Quoted 21 lines · Pratyush Anand
Quoted 19 lines · Guenter Roeck
Quoted 9 lines · Pratyush Anand
Quoted 7 lines · Guenter Roeck

Sounds good. We should have a 'name' attribute as well. We'll also need 'timeout', 'keepalive' to ping the watchdog, 'timeleft', and 'bootstatus'. Not sure what else.

The attributes should be implemented as class attributes, to ensure that they are created and removed automatically. That may require the watchdog class to be static instead of a pointer.

Quoted 7 lines

Should n't attributes be implemented as device attribute as they are properties of each device. Under the class watchdog_class, we can have multiple device on a system, and these attributes would be different for different devices. So, what I am thinking is to create device in __watchdog_register_device using device_create_with_groups instead of device_create. Here we can link a attribute_group with all attributes you suggested.

Class attibutes are created automatically for each device registered which is a member of the class. They are attached to the device, not to the class.

Quoted 6 lines
Quoted 5 lines

Not sure, if we both are referring to same thing..
So, I did a basic testing. I changed "static struct class *watchdog_class;" to "tatic struct class watchdog_class" and other than .name and .owner I added .class_attrs and then further added one node "state" in this class_attrs.

If you use .dev_groups instead of .class_attrs it should work.

OK.. Got it..Thanks for this info. I see another issue with that. I will have to make watchdog_class as global. Let me send my RFC patch, which will be very soon and then may be we can see if some improvement can be done further.

~Pratyush