lvm: add lvm cache logical volume handling
PATCH 1/1 lvm: add lvm cache logical volume handling
The lvm cache logical volume is the logical volume consisting of the original and the cache pool logical volume. The original is usually on a larger and slower storage device while the cache pool is on a smaller and faster one. The performance of the original volume can be improved by storing the frequently used data on the cache pool to utilize the greater performance of faster device.
The default cache mode "writethrough" ensures that any data written will be stored both in the cache and on the origin LV, therefore grub can go straight to read the original lv as no data loss is guarenteed.
The second cache mode is "writeback", which delays writing from the cache pool back to the origin LV to have increased performance. The drawback is potential data loss if losing the associated cache device.
During the boot time grub reads the LVM offline i.e. LVM volumes are not activated and mounted, IMHO it should be fine to read directly from original lv since all cached data should have been flushed back in the process of taking it offline.
--- grub-core/disk/lvm.c | 125 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+)
grub-core/disk/lvm.c +125 -0
@@ -272,6 +272,13 @@ grub_lvm_detect (grub_disk_t disk,if (! vg) { +struct cache_lv { +struct grub_diskfilter_lv *lv; +char *cache_pool; +char *origin; +struct cache_lv *next; + } *cache_lvs =NULL ; +/* First time we see this volume group. We've to create the whole volume group structure. */ vg =grub_malloc (sizeof (*vg));
@@ -701,6 +708,72 @@ grub_lvm_detect (grub_disk_t disk, seg->nodes[seg->node_count -1 ].name = tmp; } } +else if (grub_memcmp (p,"cache\"" , +sizeof ("cache\"" ) -1 ) ==0 ) + { +struct cache_lv *cache; + +char *p2, *p3; +grub_ssize_t sz; + + cache =grub_zalloc (sizeof (*cache)); + cache->lv =grub_zalloc (sizeof (*cache->lv)); +grub_memcpy (cache->lv, lv,sizeof (*cache->lv)); + +if (lv->fullname) + cache->lv->fullname =grub_strdup (lv->fullname); +if (lv->idname) + cache->lv->idname =grub_strdup (lv->idname); +if (lv->name) + cache->lv->name =grub_strdup (lv->name); + + skip_lv =1 ; + +if (!(p2 =grub_strstr (p,"cache_pool = \"" ))) + goto cache_lv_fail; + +if (!(p2 =grub_strchr (p2,'"' ))) + goto cache_lv_fail; + + p3 = ++p2; +if (!(p3 =grub_strchr (p3,'"' ))) + goto cache_lv_fail; + + sz = p3 - p2; + + cache->cache_pool =grub_malloc (sz +1 ); +grub_memcpy (cache->cache_pool, p2, sz); + cache->cache_pool[sz] ='\0' ; + +if (!(p2 =grub_strstr (p,"origin = \"" ))) + goto cache_lv_fail; + +if (!(p2 =grub_strchr (p2,'"' ))) + goto cache_lv_fail; + + p3 = ++p2; +if (!(p3 =grub_strchr (p3,'"' ))) + goto cache_lv_fail; + + sz = p3 - p2; + + cache->origin =grub_malloc (sz +1 ); +grub_memcpy (cache->origin, p2, sz); + cache->origin[sz] ='\0' ; + + cache->next = cache_lvs; + cache_lvs = cache; +break ; + + cache_lv_fail: +grub_free (cache->origin); +grub_free (cache->cache_pool); +grub_free (cache->lv->fullname); +grub_free (cache->lv->idname); +grub_free (cache->lv->name); +grub_free (cache); +break ; + }else {#ifdef GRUB_UTIL
@@ -777,6 +850,58 @@ grub_lvm_detect (grub_disk_t disk, } } + + { +struct cache_lv *cache; + +for (cache = cache_lvs; cache; cache = cache->next) + { +struct grub_diskfilter_lv *lv; + +for (lv = vg->lvs; lv; lv = lv->next) + { +if (grub_strcmp (lv->name, cache->origin) ==0 ) +break ; + } +if (lv) + { + cache->lv->segment_count = lv->segment_count; + cache->lv->segments =grub_malloc (lv->segment_count *sizeof (*lv->segments)); +grub_memcpy (cache->lv->segments, lv->segments, lv->segment_count *sizeof (*lv->segments)); + +for (i =0 ; i < lv->segment_count; ++i) + { +struct grub_diskfilter_node *nodes = lv->segments[i].nodes; +grub_size_t node_count = lv->segments[i].node_count; + cache->lv->segments[i].nodes =grub_malloc (node_count *sizeof (*nodes)); +grub_memcpy (cache->lv->segments[i].nodes, nodes, node_count *sizeof (*nodes)); + } + cache->lv->vg = vg; + cache->lv->next = vg->lvs; + vg->lvs = cache->lv; + cache->lv =NULL ; + } + } + +while ((cache = cache_lvs)) + { + cache_lvs = cache_lvs->next; + +if (cache->lv) + { +for (i =0 ; i < cache->lv->segment_count; ++i) +grub_free (cache->lv->segments[i].nodes); +grub_free (cache->lv->fullname); +grub_free (cache->lv->idname); +grub_free (cache->lv->name); + } +grub_free (cache->lv); +grub_free (cache->origin); +grub_free (cache->cache_pool); +grub_free (cache); + } + } +if (grub_diskfilter_vg_register (vg)) goto fail4; }
Signature · 1 line
2.16.4
Quoted 11 lines · Michael Chang
The lvm cache logical volume is the logical volume consisting of the original and the cache pool logical volume. The original is usually on a larger and slower storage device while the cache pool is on a smaller and faster one. The performance of the original volume can be improved by storing the frequently used data on the cache pool to utilize the greater performance of faster device.
The default cache mode "writethrough" ensures that any data written will be stored both in the cache and on the origin LV, therefore grub can go straight to read the original lv as no data loss is guarenteed.
Quoted 8 lines
The second cache mode is "writeback", which delays writing from the cache pool back to the origin LV to have increased performance. The drawback is potential data loss if losing the associated cache device.
During the boot time grub reads the LVM offline i.e. LVM volumes are not activated and mounted, IMHO it should be fine to read directly from original lv since all cached data should have been flushed back in the process of taking it offline.
Is it possible to enforce all GRUB writes to the original device instead of cache during installation process?
Quoted 10 lines
Signed-off-by Michael Chang <[email protected]>--- grub-core/disk/lvm.c | 125 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+)grub-core/disk/lvm.c +0
-0
grub-core/disk/lvm.c grub_lvm_detect (grub_disk_t disk, · line 280
Quoted 9 lines
grub-core/disk/lvm.c +6
-0@@ -272,6 +272,13 @@ grub_lvm_detect (grub_disk_t disk,if (! vg) { +struct cache_lv { +struct grub_diskfilter_lv *lv; +char *cache_pool; +char *origin; +struct cache_lv *next; + } *cache_lvs =NULL ;
I would move struct cache_lv definition to the beginning of the file. Though you can leave variable declaration here.
Quoted 8 lines
grub-core/disk/lvm.c +1
-0+/* First time we see this volume group. We've to create the whole volume group structure. */ vg =grub_malloc (sizeof (*vg));@@ -701,6 +708,72 @@ grub_lvm_detect (grub_disk_t disk, seg->nodes[seg->node_count -1 ].name = tmp; } }
grub-core/disk/lvm.c grub_lvm_detect (grub_disk_t disk, · line 720
Quoted 10 lines
grub-core/disk/lvm.c +10
-0+else if (grub_memcmp (p,"cache\"" , +sizeof ("cache\"" ) -1 ) ==0 ) + { +struct cache_lv *cache; + +char *p2, *p3; +grub_ssize_t sz; + + cache =grub_zalloc (sizeof (*cache)); + cache->lv =grub_zalloc (sizeof (*cache->lv));
You blindly assume that grub_zalloc() does not fail. This is dangerous. Please fix it.
grub-core/disk/lvm.c grub_lvm_detect (grub_disk_t disk, · line 728
Quoted 8 lines
grub-core/disk/lvm.c +8
-0+grub_memcpy (cache->lv, lv,sizeof (*cache->lv)); + +if (lv->fullname) + cache->lv->fullname =grub_strdup (lv->fullname); +if (lv->idname) + cache->lv->idname =grub_strdup (lv->idname); +if (lv->name) + cache->lv->name =grub_strdup (lv->name);
Same for grub_strdup()...
Quoted 6 lines
grub-core/disk/lvm.c +6
-0+ + skip_lv =1 ; + +if (!(p2 =grub_strstr (p,"cache_pool = \"" ))) + goto cache_lv_fail; +
grub-core/disk/lvm.c grub_lvm_detect (grub_disk_t disk, · line 744
Quoted 10 lines
grub-core/disk/lvm.c +10
-0+if (!(p2 =grub_strchr (p2,'"' ))) + goto cache_lv_fail; + + p3 = ++p2; +if (!(p3 =grub_strchr (p3,'"' ))) + goto cache_lv_fail; + + sz = p3 - p2; + + cache->cache_pool =grub_malloc (sz +1 );
Same for grub_malloc() and below. Please fix all of them.
Daniel
Quoted 16 lines · Daniel Kiper
Quoted 15 lines · Michael Chang
The lvm cache logical volume is the logical volume consisting of the original and the cache pool logical volume. The original is usually on a larger and slower storage device while the cache pool is on a smaller and faster one. The performance of the original volume can be improved by storing the frequently used data on the cache pool to utilize the greater performance of faster device.
The default cache mode "writethrough" ensures that any data written will be stored both in the cache and on the origin LV, therefore grub can go straight to read the original lv as no data loss is guarenteed.
The second cache mode is "writeback", which delays writing from the cache pool back to the origin LV to have increased performance. The drawback is potential data loss if losing the associated cache device.
Quoted 7 lines
Quoted 4 lines
During the boot time grub reads the LVM offline i.e. LVM volumes are not activated and mounted, IMHO it should be fine to read directly from original lv since all cached data should have been flushed back in the process of taking it offline.
Is it possible to enforce all GRUB writes to the original device instead of cache during installation process?
I couldn't give concrete answer whether it is possible, but it sounds to me not good idea because in general bypassing the cache could potentially break the consistency of data and the data being cached.
Perhaps some system calls could help to sync the data out of the lvm cache to the original LV during installation process. It seems fsync does it for us, but I didn't have good idea either if it is enough.
Quoted 12 lines
Quoted 12 lines
Signed-off-by Michael Chang <[email protected]>--- grub-core/disk/lvm.c | 125 +++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 125 insertions(+)grub-core/disk/lvm.c +0
-0@@ -272,6 +272,13 @@ grub_lvm_detect (grub_disk_t disk,
Quoted 11 lines
grub-core/disk/lvm.c grub_lvm_detect (grub_disk_t disk, · line 280
Quoted 8 lines
grub-core/disk/lvm.c +6
-0if (! vg) { +struct cache_lv { +struct grub_diskfilter_lv *lv; +char *cache_pool; +char *origin; +struct cache_lv *next; + } *cache_lvs =NULL ;I would move struct
cache_lvdefinition to the beginning of the file. Though you can leave variable declaration here.
OK. I will do in next patch.
Quoted 14 lines
Quoted 14 lines
grub-core/disk/lvm.c +6
-0+/* First time we see this volume group. We've to create the whole volume group structure. */ vg =grub_malloc (sizeof (*vg));@@ -701,6 +708,72 @@ grub_lvm_detect (grub_disk_t disk, seg->nodes[seg->node_count -1 ].name = tmp; } } +else if (grub_memcmp (p,"cache\"" , +sizeof ("cache\"" ) -1 ) ==0 ) + { +struct cache_lv *cache; +
Quoted 8 lines
grub-core/disk/lvm.c grub_lvm_detect (grub_disk_t disk, · line 720
Quoted 5 lines
grub-core/disk/lvm.c +5
-0+char *p2, *p3; +grub_ssize_t sz; + + cache =grub_zalloc (sizeof (*cache)); + cache->lv =grub_zalloc (sizeof (*cache->lv));You blindly assume that
grub_zalloc()does not fail. This is dangerous. Please fix it.
OK. I will do in next patch.
Quoted 11 lines
grub-core/disk/lvm.c grub_lvm_detect (grub_disk_t disk, · line 728
Quoted 9 lines
grub-core/disk/lvm.c +8
-0+grub_memcpy (cache->lv, lv,sizeof (*cache->lv)); + +if (lv->fullname) + cache->lv->fullname =grub_strdup (lv->fullname); +if (lv->idname) + cache->lv->idname =grub_strdup (lv->idname); +if (lv->name) + cache->lv->name =grub_strdup (lv->name);Same for
grub_strdup()...
OK. I will do in next patch.
Quoted 10 lines
Quoted 10 lines
grub-core/disk/lvm.c +10
-0+ + skip_lv =1 ; + +if (!(p2 =grub_strstr (p,"cache_pool = \"" ))) + goto cache_lv_fail; + +if (!(p2 =grub_strchr (p2,'"' ))) + goto cache_lv_fail; + + p3 = ++p2;
Quoted 8 lines
grub-core/disk/lvm.c grub_lvm_detect (grub_disk_t disk, · line 744
Quoted 6 lines
grub-core/disk/lvm.c +6
-0+if (!(p3 =grub_strchr (p3,'"' ))) + goto cache_lv_fail; + + sz = p3 - p2; + + cache->cache_pool =grub_malloc (sz +1 );Same for
grub_malloc()and below. Please fix all of them.
OK. I will do in next patch.
Thanks a lot for your review and comments. :)
Regards,
Michael
Quoted 7 lines
Daniel
_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel
Quoted 25 lines · Michael Chang
Quoted 16 lines · Daniel Kiper
Quoted 15 lines · Michael Chang
The lvm cache logical volume is the logical volume consisting of the original and the cache pool logical volume. The original is usually on a larger and slower storage device while the cache pool is on a smaller and faster one. The performance of the original volume can be improved by storing the frequently used data on the cache pool to utilize the greater performance of faster device.
The default cache mode "writethrough" ensures that any data written will be stored both in the cache and on the origin LV, therefore grub can go straight to read the original lv as no data loss is guarenteed.
The second cache mode is "writeback", which delays writing from the cache pool back to the origin LV to have increased performance. The drawback is potential data loss if losing the associated cache device.
Quoted 7 lines
Quoted 4 lines
During the boot time grub reads the LVM offline i.e. LVM volumes are not activated and mounted, IMHO it should be fine to read directly from original lv since all cached data should have been flushed back in the process of taking it offline.
Is it possible to enforce all GRUB writes to the original device instead of cache during installation process?
Quoted 7 lines
I couldn't give concrete answer whether it is possible, but it sounds to me not good idea because in general bypassing the cache could potentially break the consistency of data and the data being cached.
Perhaps some system calls could help to sync the data out of the lvm cache to the original LV during installation process. It seems fsync does it for us, but I didn't have good idea either if it is enough.
May I ask you to investigate that and if it is needed add required install code. I mean fsync(), etc.
Daniel
Quoted 30 lines · Daniel Kiper
Quoted 29 lines · Michael Chang
Quoted 16 lines · Daniel Kiper
Quoted 15 lines · Michael Chang
The lvm cache logical volume is the logical volume consisting of the original and the cache pool logical volume. The original is usually on a larger and slower storage device while the cache pool is on a smaller and faster one. The performance of the original volume can be improved by storing the frequently used data on the cache pool to utilize the greater performance of faster device.
The default cache mode "writethrough" ensures that any data written will be stored both in the cache and on the origin LV, therefore grub can go straight to read the original lv as no data loss is guarenteed.
The second cache mode is "writeback", which delays writing from the cache pool back to the origin LV to have increased performance. The drawback is potential data loss if losing the associated cache device.
Quoted 7 lines
Quoted 4 lines
During the boot time grub reads the LVM offline i.e. LVM volumes are not activated and mounted, IMHO it should be fine to read directly from original lv since all cached data should have been flushed back in the process of taking it offline.
Is it possible to enforce all GRUB writes to the original device instead of cache during installation process?
I couldn't give concrete answer whether it is possible, but it sounds to me not good idea because in general bypassing the cache could potentially break the consistency of data and the data being cached.
Quoted 6 lines
Quoted 3 lines
Perhaps some system calls could help to sync the data out of the lvm cache to the original LV during installation process. It seems fsync does it for us, but I didn't have good idea either if it is enough.
May I ask you to investigate that and if it is needed add required install code. I mean
fsync(), etc.
Sure. And please bear with me if it takes long to answer as other task was sidetracking.
Thanks,
Michael
Quoted 7 lines
Daniel
_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel
Quoted 30 lines · Daniel Kiper
Quoted 29 lines · Michael Chang
Quoted 16 lines · Daniel Kiper
Quoted 15 lines · Michael Chang
The lvm cache logical volume is the logical volume consisting of the original and the cache pool logical volume. The original is usually on a larger and slower storage device while the cache pool is on a smaller and faster one. The performance of the original volume can be improved by storing the frequently used data on the cache pool to utilize the greater performance of faster device.
The default cache mode "writethrough" ensures that any data written will be stored both in the cache and on the origin LV, therefore grub can go straight to read the original lv as no data loss is guarenteed.
The second cache mode is "writeback", which delays writing from the cache pool back to the origin LV to have increased performance. The drawback is potential data loss if losing the associated cache device.
Quoted 7 lines
Quoted 4 lines
During the boot time grub reads the LVM offline i.e. LVM volumes are not activated and mounted, IMHO it should be fine to read directly from original lv since all cached data should have been flushed back in the process of taking it offline.
Is it possible to enforce all GRUB writes to the original device instead of cache during installation process?
I couldn't give concrete answer whether it is possible, but it sounds to me not good idea because in general bypassing the cache could potentially break the consistency of data and the data being cached.
Quoted 6 lines
Quoted 3 lines
Perhaps some system calls could help to sync the data out of the lvm cache to the original LV during installation process. It seems fsync does it for us, but I didn't have good idea either if it is enough.
May I ask you to investigate that and if it is needed add required install code. I mean
fsync(), etc.
As far as what I learned, the fsync did not write back dirty cache to the orginal device, instead it would update associated cache metadata to complete the write transaction with the cache device and then having the write cache up to date. Without the operation, metadata is commited every second and data would be completely lost/evaporated if power failure happened in between.
To write back dirty cache, as lvm cache did not support dirty cache flush per block range, there'no way to do it for file. Instead the "cleaner" policy is implemented and can be used to write back "all" dirty blocks in a cache, which effectively drain all dirty cache gradually to attain and last in the "clean" state, which can be useful for shrinking or decommissioning a cache. The result and effect is not what we are looking for here.
In conclusion, it seems no way to enforce all grub writes to the original device. That means grub may suffer from power failure that leads to dirty cache not being flushed and is inept reading data flagged as dirty in cache. But even though, as long as such case is only subjected to writeback mode that is generally difficult to deal with data lost whenever accident comes up, I'd still like to propose my (relatively simple) patch and treat reading dirty data in cache as an enhancement to be more resilience to potential data loss.
Thanks,
Michael
Quoted 7 lines
Daniel
_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel
Quoted 53 lines · Michael Chang
Quoted 30 lines · Daniel Kiper
Quoted 29 lines · Michael Chang
Quoted 16 lines · Daniel Kiper
Quoted 15 lines · Michael Chang
The lvm cache logical volume is the logical volume consisting of the original and the cache pool logical volume. The original is usually on a larger and slower storage device while the cache pool is on a smaller and faster one. The performance of the original volume can be improved by storing the frequently used data on the cache pool to utilize the greater performance of faster device.
The default cache mode "writethrough" ensures that any data written will be stored both in the cache and on the origin LV, therefore grub can go straight to read the original lv as no data loss is guarenteed.
The second cache mode is "writeback", which delays writing from the cache pool back to the origin LV to have increased performance. The drawback is potential data loss if losing the associated cache device.
Quoted 7 lines
Quoted 4 lines
During the boot time grub reads the LVM offline i.e. LVM volumes are not activated and mounted, IMHO it should be fine to read directly from original lv since all cached data should have been flushed back in the process of taking it offline.
Is it possible to enforce all GRUB writes to the original device instead of cache during installation process?
I couldn't give concrete answer whether it is possible, but it sounds to me not good idea because in general bypassing the cache could potentially break the consistency of data and the data being cached.
Quoted 6 lines
Quoted 3 lines
Perhaps some system calls could help to sync the data out of the lvm cache to the original LV during installation process. It seems fsync does it for us, but I didn't have good idea either if it is enough.
May I ask you to investigate that and if it is needed add required install code. I mean
fsync(), etc.As far as what I learned, the fsync did not write back dirty cache to the orginal device, instead it would update associated cache metadata to complete the write transaction with the cache device and then having the write cache up to date. Without the operation, metadata is commited every second and data would be completely lost/evaporated if power failure happened in between.
To write back dirty cache, as lvm cache did not support dirty cache flush per block range, there'no way to do it for file. Instead the "cleaner" policy is implemented and can be used to write back "all" dirty blocks in a cache, which effectively drain all dirty cache gradually to attain and last in the "clean" state, which can be useful for shrinking or decommissioning a cache. The result and effect is not what we are looking for here.
Quoted 8 lines
In conclusion, it seems no way to enforce all grub writes to the original device. That means grub may suffer from power failure that leads to dirty cache not being flushed and is inept reading data flagged as dirty in cache. But even though, as long as such case is only subjected to writeback mode that is generally difficult to deal with data lost whenever accident comes up, I'd still like to propose my (relatively simple) patch and treat reading dirty data in cache as an enhancement to be more resilience to potential data loss.
I would like to see more or less this description in the commit message. And I think that you should add similar paragraph to the GRUB manual too.
Daniel
Quoted 63 lines · Daniel Kiper
Quoted 53 lines · Michael Chang
Quoted 30 lines · Daniel Kiper
Quoted 29 lines · Michael Chang
Quoted 16 lines · Daniel Kiper
Quoted 15 lines · Michael Chang
The lvm cache logical volume is the logical volume consisting of the original and the cache pool logical volume. The original is usually on a larger and slower storage device while the cache pool is on a smaller and faster one. The performance of the original volume can be improved by storing the frequently used data on the cache pool to utilize the greater performance of faster device.
The default cache mode "writethrough" ensures that any data written will be stored both in the cache and on the origin LV, therefore grub can go straight to read the original lv as no data loss is guarenteed.
The second cache mode is "writeback", which delays writing from the cache pool back to the origin LV to have increased performance. The drawback is potential data loss if losing the associated cache device.
Quoted 7 lines
Quoted 4 lines
During the boot time grub reads the LVM offline i.e. LVM volumes are not activated and mounted, IMHO it should be fine to read directly from original lv since all cached data should have been flushed back in the process of taking it offline.
Is it possible to enforce all GRUB writes to the original device instead of cache during installation process?
I couldn't give concrete answer whether it is possible, but it sounds to me not good idea because in general bypassing the cache could potentially break the consistency of data and the data being cached.
Quoted 6 lines
Quoted 3 lines
Perhaps some system calls could help to sync the data out of the lvm cache to the original LV during installation process. It seems fsync does it for us, but I didn't have good idea either if it is enough.
May I ask you to investigate that and if it is needed add required install code. I mean
fsync(), etc.As far as what I learned, the fsync did not write back dirty cache to the orginal device, instead it would update associated cache metadata to complete the write transaction with the cache device and then having the write cache up to date. Without the operation, metadata is commited every second and data would be completely lost/evaporated if power failure happened in between.
To write back dirty cache, as lvm cache did not support dirty cache flush per block range, there'no way to do it for file. Instead the "cleaner" policy is implemented and can be used to write back "all" dirty blocks in a cache, which effectively drain all dirty cache gradually to attain and last in the "clean" state, which can be useful for shrinking or decommissioning a cache. The result and effect is not what we are looking for here.
Quoted 8 lines
In conclusion, it seems no way to enforce all grub writes to the original device. That means grub may suffer from power failure that leads to dirty cache not being flushed and is inept reading data flagged as dirty in cache. But even though, as long as such case is only subjected to writeback mode that is generally difficult to deal with data lost whenever accident comes up, I'd still like to propose my (relatively simple) patch and treat reading dirty data in cache as an enhancement to be more resilience to potential data loss.
Quoted 2 lines
I would like to see more or less this description in the commit message. And I think that you should add similar paragraph to the GRUB manual too.
OK. I will add description to the commit message and manual in next version sent here for review.
Regards,
Michael
Quoted 7 lines
Daniel
_______________________________________________
Grub-devel mailing list
[email protected]
https://lists.gnu.org/mailman/listinfo/grub-devel