Skip to contentSkip to conversation

makedumpfile header to show LZO/snappy/zlib

RFC PATCH In review Dave Anderson 3 participants 5 messages Sep 12, 2013 – Sep 26, 2013 1 message not in the archive

Subsystems (2)
lore
Share & tools
lore
#1 lore raw

RFC PATCH 1/1 makedumpfile header to show LZO/snappy/zlib

In reply to a message not in the archive

With the advent of LZO and snappy compression support, it would be helpful if it could be readily determined which type of compression was used to create the dumpfile.

The zlib/LZO/snappy indicator is only found down in the bowels of each page_desc structure:

  /* page flags */
  #define DUMP_DH_COMPRESSED_ZLIB 0x1     /* page is compressed with zlib */
  #define DUMP_DH_COMPRESSED_LZO  0x2     /* paged is compressed with lzo */
  #define DUMP_DH_COMPRESSED_SNAPPY       0x4
                                          /* paged is compressed with snappy */

/* descriptor of each page for vmcore */
typedef struct page_desc {

          off_t                   offset;         /* the offset of the page data*/
          unsigned int            size;           /* the size of this dump page */
          unsigned int            flags;          /* flags */
          unsigned long long      page_flags;     /* page flags */

} page_desc_t;

The compressed kdump's header does contain a "status" flag:

struct disk_dump_header {

          char                    signature[SIG_LEN];     /* = "KDUMP   " */
          int                     header_version; /* Dump header version */
          struct new_utsname      utsname;        /* copy of system_utsname */
          struct timeval          timestamp;      /* Time stamp */
          unsigned int            status;         /* Above flags */

...

which has these 3 flags:

  #define DUMP_HEADER_COMPLETED   0
  #define DUMP_HEADER_INCOMPLETED 1
  #define DUMP_HEADER_COMPRESSED  8

But they are all remnants of the old "diskdump" facility, and none of them are ever used by makedumpfile.

Any chance that the LZO/snappy/zlib method could also be put up in the disk_dump_header?

Dave

makedumpfile.c renamed from makedumpfile.c.orig +10 -0
@@ -5162,6 +5162,16 @@ write_kdump_header(void)
 	dh->bitmap_blocks  = divideup(info->len_bitmap, dh->block_size);
 	memcpy(&dh->timestamp, &info->timestamp, sizeof(dh->timestamp));
 	memcpy(&dh->utsname, &info->system_utsname, sizeof(dh->utsname));
+	if (info->flag_compress & DUMP_DH_COMPRESSED_ZLIB)
+		dh->status |= DUMP_DH_COMPRESSED_ZLIB; 
+#ifdef USELZO
+	else if (info->flag_compress & DUMP_DH_COMPRESSED_LZO)
+		dh->status |= DUMP_DH_COMPRESSED_LZO)
+#endif
+#ifdef USESNAPPY
+	else if (info->flag_compress & DUMP_DH_COMPRESSED_SNAPPY)
+		dh->status |= DUMP_DH_COMPRESSED_SNAPPY)
+#endif
 
 	size = sizeof(struct disk_dump_header);
 	if (!write_buffer(info->fd_dumpfile, 0, dh, size, info->name_dumpfile))

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#2 lore raw

Hello Dave,

Quoted 52 lines · Dave Anderson

With the advent of LZO and snappy compression support, it would be helpful if it could be readily determined which type of compression was used to create the dumpfile.

The zlib/LZO/snappy indicator is only found down in the bowels of each page_desc structure:

   /* page flags */
   #define DUMP_DH_COMPRESSED_ZLIB 0x1     /* page is compressed with zlib */
   #define DUMP_DH_COMPRESSED_LZO  0x2     /* paged is compressed with lzo */
   #define DUMP_DH_COMPRESSED_SNAPPY       0x4
                                           /* paged is compressed with snappy */

/* descriptor of each page for vmcore */
typedef struct page_desc {

           off_t                   offset;         /* the offset of the page data*/
           unsigned int            size;           /* the size of this dump page */
           unsigned int            flags;          /* flags */
           unsigned long long      page_flags;     /* page flags */

} page_desc_t;

The compressed kdump's header does contain a "status" flag:

struct disk_dump_header {

           char                    signature[SIG_LEN];     /* = "KDUMP   " */
           int                     header_version; /* Dump header version */
           struct new_utsname      utsname;        /* copy of system_utsname */
           struct timeval          timestamp;      /* Time stamp */
           unsigned int            status;         /* Above flags */

...

which has these 3 flags:

   #define DUMP_HEADER_COMPLETED   0
   #define DUMP_HEADER_INCOMPLETED 1
   #define DUMP_HEADER_COMPRESSED  8

But they are all remnants of the old "diskdump" facility, and none of them are ever used by makedumpfile.

Any chance that the LZO/snappy/zlib method could also be put up in the disk_dump_header?

Dave

makedumpfile.c renamed from makedumpfile.c.orig +0 -0
@@ -5162,6 +5162,16 @@ write_kdump_header(void)
     dh->bitmap_blocks  = divideup(info->len_bitmap, dh->block_size);
     memcpy(&dh->timestamp, &info->timestamp, sizeof(dh->timestamp));
     memcpy(&dh->utsname, &info->system_utsname, sizeof(dh->utsname));

makedumpfile.c write_kdump_header(void) · line 5174

Quoted 10 lines
makedumpfile.c renamed from makedumpfile.c.orig +10 -0
+    if (info->flag_compress & DUMP_DH_COMPRESSED_ZLIB)
+        dh->status |= DUMP_DH_COMPRESSED_ZLIB;
+#ifdef USELZO
+    else if (info->flag_compress & DUMP_DH_COMPRESSED_LZO)
+        dh->status |= DUMP_DH_COMPRESSED_LZO)
+#endif
+#ifdef USESNAPPY
+    else if (info->flag_compress & DUMP_DH_COMPRESSED_SNAPPY)
+        dh->status |= DUMP_DH_COMPRESSED_SNAPPY)
+#endif

dh->status will not guarantee that *all pages* are compressed by the check below:

Quoted 7 lines
makedumpfile.c renamed from makedumpfile.c.orig +0 -0
 #ifdef USELZO
                 } else if (info->flag_lzo_support
                            && (info->flag_compress & DUMP_DH_COMPRESSED_LZO)
                            && ((size_out = info->page_size),
                                lzo1x_1_compress(buf, info->page_size, buf_out,
                                                 &size_out, wrkmem) == LZO_E_OK)
                            && (size_out < info->page_size)) {
                                  ^^^^^^^^^^^^^^^^^^^^^^^^^^^

When there is no effect of compression, the page will not be compressed.

Quoted 4 lines
                         pd.flags = DUMP_DH_COMPRESSED_LZO;
                         pd.size  = size_out;
                         memcpy(buf, buf_out, pd.size);

#endif

If we are on the same page, I'll accept your patch.

Thanks
Atsushi Kumagai

Quoted 3 lines
     size = sizeof(struct disk_dump_header);
     if (!write_buffer(info->fd_dumpfile, 0, dh, size, info->name_dumpfile))

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#3 lore raw

----- Original Message -----

Quoted 88 lines

Hello Dave,

Quoted 60 lines · Dave Anderson

With the advent of LZO and snappy compression support, it would be helpful if it could be readily determined which type of compression was used to create the dumpfile.

The zlib/LZO/snappy indicator is only found down in the bowels of each page_desc structure:

   /* page flags */
   #define DUMP_DH_COMPRESSED_ZLIB 0x1     /* page is compressed with zlib
   */
   #define DUMP_DH_COMPRESSED_LZO  0x2     /* paged is compressed with lzo
   */
   #define DUMP_DH_COMPRESSED_SNAPPY       0x4
                                           /* paged is compressed with
                                           snappy */

/* descriptor of each page for vmcore */
typedef struct page_desc {
off_t offset; /* the offset of the
page data*/
unsigned int size; /* the size of this dump
page */
unsigned int flags; /* flags */
unsigned long long page_flags; /* page flags */
} page_desc_t;

The compressed kdump's header does contain a "status" flag:

struct disk_dump_header {
char signature[SIG_LEN]; /* = "KDUMP "
*/
int header_version; /* Dump header version
*/
struct new_utsname utsname; /* copy of
system_utsname */
struct timeval timestamp; /* Time stamp */
unsigned int status; /* Above flags */
...

which has these 3 flags:

   #define DUMP_HEADER_COMPLETED   0
   #define DUMP_HEADER_INCOMPLETED 1
   #define DUMP_HEADER_COMPRESSED  8

But they are all remnants of the old "diskdump" facility, and none of them are ever used by makedumpfile.

Any chance that the LZO/snappy/zlib method could also be put up in the disk_dump_header?

Dave

makedumpfile.c renamed from makedumpfile.c.orig +0 -0
@@ -5162,6 +5162,16 @@ write_kdump_header(void)
     dh->bitmap_blocks  = divideup(info->len_bitmap, dh->block_size);
     memcpy(&dh->timestamp, &info->timestamp, sizeof(dh->timestamp));
     memcpy(&dh->utsname, &info->system_utsname, sizeof(dh->utsname));

makedumpfile.c write_kdump_header(void) · line 5174

Quoted 10 lines
makedumpfile.c renamed from makedumpfile.c.orig +10 -0
+    if (info->flag_compress & DUMP_DH_COMPRESSED_ZLIB)
+        dh->status |= DUMP_DH_COMPRESSED_ZLIB;
+#ifdef USELZO
+    else if (info->flag_compress & DUMP_DH_COMPRESSED_LZO)
+        dh->status |= DUMP_DH_COMPRESSED_LZO)
+#endif
+#ifdef USESNAPPY
+    else if (info->flag_compress & DUMP_DH_COMPRESSED_SNAPPY)
+        dh->status |= DUMP_DH_COMPRESSED_SNAPPY)
+#endif

dh->status will not guarantee that *all pages* are compressed by the check below:

Quoted 10 lines
makedumpfile.c renamed from makedumpfile.c.orig +0 -0
 #ifdef USELZO
                 } else if (info->flag_lzo_support
                            && (info->flag_compress &
                            DUMP_DH_COMPRESSED_LZO)
                            && ((size_out = info->page_size),
                                lzo1x_1_compress(buf, info->page_size,
                                buf_out,
                                                 &size_out, wrkmem) ==
                                                 LZO_E_OK)
                            && (size_out < info->page_size)) {
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^
Quoted 8 lines

When there is no effect of compression, the page will not be compressed.

Quoted 4 lines
                         pd.flags = DUMP_DH_COMPRESSED_LZO;
                         pd.size  = size_out;
                         memcpy(buf, buf_out, pd.size);

#endif

If we are on the same page, I'll accept your patch.

Right, I understand.

Thanks,
Dave

Quoted 10 lines

Thanks
Atsushi Kumagai

Quoted 4 lines
     size = sizeof(struct disk_dump_header);
     if (!write_buffer(info->fd_dumpfile, 0, dh, size,
     info->name_dumpfile))

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#4 lore raw
Quoted 93 lines · On Fri, 20 Sep 2013 09:04:40 -0400 (EDT) Dave Anderson <[email protected]>…

----- Original Message -----

Quoted 90 lines

Hello Dave,

Quoted 60 lines · Dave Anderson

With the advent of LZO and snappy compression support, it would be helpful if it could be readily determined which type of compression was used to create the dumpfile.

The zlib/LZO/snappy indicator is only found down in the bowels of each page_desc structure:

   /* page flags */
   #define DUMP_DH_COMPRESSED_ZLIB 0x1     /* page is compressed with zlib
   */
   #define DUMP_DH_COMPRESSED_LZO  0x2     /* paged is compressed with lzo
   */
   #define DUMP_DH_COMPRESSED_SNAPPY       0x4
                                           /* paged is compressed with
                                           snappy */

/* descriptor of each page for vmcore */
typedef struct page_desc {
off_t offset; /* the offset of the
page data*/
unsigned int size; /* the size of this dump
page */
unsigned int flags; /* flags */
unsigned long long page_flags; /* page flags */
} page_desc_t;

The compressed kdump's header does contain a "status" flag:

struct disk_dump_header {
char signature[SIG_LEN]; /* = "KDUMP "
*/
int header_version; /* Dump header version
*/
struct new_utsname utsname; /* copy of
system_utsname */
struct timeval timestamp; /* Time stamp */
unsigned int status; /* Above flags */
...

which has these 3 flags:

   #define DUMP_HEADER_COMPLETED   0
   #define DUMP_HEADER_INCOMPLETED 1
   #define DUMP_HEADER_COMPRESSED  8

But they are all remnants of the old "diskdump" facility, and none of them are ever used by makedumpfile.

Any chance that the LZO/snappy/zlib method could also be put up in the disk_dump_header?

Dave

makedumpfile.c renamed from makedumpfile.c.orig +0 -0
@@ -5162,6 +5162,16 @@ write_kdump_header(void)
     dh->bitmap_blocks  = divideup(info->len_bitmap, dh->block_size);
     memcpy(&dh->timestamp, &info->timestamp, sizeof(dh->timestamp));
     memcpy(&dh->utsname, &info->system_utsname, sizeof(dh->utsname));

makedumpfile.c write_kdump_header(void) · line 5174

Quoted 10 lines
makedumpfile.c renamed from makedumpfile.c.orig +10 -0
+    if (info->flag_compress & DUMP_DH_COMPRESSED_ZLIB)
+        dh->status |= DUMP_DH_COMPRESSED_ZLIB;
+#ifdef USELZO
+    else if (info->flag_compress & DUMP_DH_COMPRESSED_LZO)
+        dh->status |= DUMP_DH_COMPRESSED_LZO)
+#endif
+#ifdef USESNAPPY
+    else if (info->flag_compress & DUMP_DH_COMPRESSED_SNAPPY)
+        dh->status |= DUMP_DH_COMPRESSED_SNAPPY)
+#endif

dh->status will not guarantee that *all pages* are compressed by the check below:

Quoted 10 lines
makedumpfile.c renamed from makedumpfile.c.orig +0 -0
 #ifdef USELZO
                 } else if (info->flag_lzo_support
                            && (info->flag_compress &
                            DUMP_DH_COMPRESSED_LZO)
                            && ((size_out = info->page_size),
                                lzo1x_1_compress(buf, info->page_size,
                                buf_out,
                                                 &size_out, wrkmem) ==
                                                 LZO_E_OK)
                            && (size_out < info->page_size)) {
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^

When there is no effect of compression, the page will not be compressed.

Quoted 8 lines
Quoted 6 lines
Quoted 4 lines
                         pd.flags = DUMP_DH_COMPRESSED_LZO;
                         pd.size  = size_out;
                         memcpy(buf, buf_out, pd.size);

#endif

If we are on the same page, I'll accept your patch.

Right, I understand.

I think the primary intention is to let crash determine early if it is able to open a given dump or not. And that's a good idea!

From a semantic point of view, the status field in the header should be a logical OR of all compression methods used in the coredump. AFAICS, Dave's patch would actually work for the current implementation.

Did I miss something?

Petr T

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec

#5 lore raw

----- Original Message -----

Quoted 95 lines
Quoted 92 lines · On Fri, 20 Sep 2013 09:04:40 -0400 (EDT) Dave Anderson <[email protected]>…

----- Original Message -----

Quoted 89 lines

Hello Dave,

Quoted 54 lines · Dave Anderson

With the advent of LZO and snappy compression support, it would be helpful if it could be readily determined which type of compression was used to create the dumpfile.

The zlib/LZO/snappy indicator is only found down in the bowels of each page_desc structure:

   /* page flags */
   #define DUMP_DH_COMPRESSED_ZLIB 0x1     /* page is compressed with zlib
   */
   #define DUMP_DH_COMPRESSED_LZO  0x2     /* paged is compressed with lzo
   */
   #define DUMP_DH_COMPRESSED_SNAPPY       0x4
                                           /* paged is compressed with snappy */

/* descriptor of each page for vmcore */
typedef struct page_desc {

           off_t                   offset;         /* the offset of the page data */
           unsigned int            size;           /* the size of this dump page */
           unsigned int            flags;          /* flags */
           unsigned long long      page_flags;     /* page flags */

} page_desc_t;

The compressed kdump's header does contain a "status" flag:

struct disk_dump_header {

           char                    signature[SIG_LEN];     /* = "KDUMP   " */
           int                     header_version; /* Dump header version  */
           struct new_utsname      utsname;        /* copy of system_utsname */
           struct timeval          timestamp;      /* Time stamp */
           unsigned int            status;         /* Above flags */

...

which has these 3 flags:

   #define DUMP_HEADER_COMPLETED   0
   #define DUMP_HEADER_INCOMPLETED 1
   #define DUMP_HEADER_COMPRESSED  8

But they are all remnants of the old "diskdump" facility, and none of them are ever used by makedumpfile.

Any chance that the LZO/snappy/zlib method could also be put up in the disk_dump_header?

Dave

makedumpfile.c renamed from makedumpfile.c.orig +0 -0
@@ -5162,6 +5162,16 @@ write_kdump_header(void)
     dh->bitmap_blocks  = divideup(info->len_bitmap, dh->block_size);
     memcpy(&dh->timestamp, &info->timestamp, sizeof(dh->timestamp));
     memcpy(&dh->utsname, &info->system_utsname, sizeof(dh->utsname));

makedumpfile.c write_kdump_header(void) · line 5174

Quoted 10 lines
makedumpfile.c renamed from makedumpfile.c.orig +10 -0
+    if (info->flag_compress & DUMP_DH_COMPRESSED_ZLIB)
+        dh->status |= DUMP_DH_COMPRESSED_ZLIB;
+#ifdef USELZO
+    else if (info->flag_compress & DUMP_DH_COMPRESSED_LZO)
+        dh->status |= DUMP_DH_COMPRESSED_LZO)
+#endif
+#ifdef USESNAPPY
+    else if (info->flag_compress & DUMP_DH_COMPRESSED_SNAPPY)
+        dh->status |= DUMP_DH_COMPRESSED_SNAPPY)
+#endif

dh->status will not guarantee that *all pages* are compressed by the check below:

Quoted 10 lines
makedumpfile.c renamed from makedumpfile.c.orig +0 -0
 #ifdef USELZO
                 } else if (info->flag_lzo_support
                            && (info->flag_compress &
                            DUMP_DH_COMPRESSED_LZO)
                            && ((size_out = info->page_size),
                                lzo1x_1_compress(buf, info->page_size,
                                buf_out,
                                                 &size_out, wrkmem) ==
                                                 LZO_E_OK)
                            && (size_out < info->page_size)) {
                                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^

When there is no effect of compression, the page will not be compressed.

Quoted 4 lines
                         pd.flags = DUMP_DH_COMPRESSED_LZO;
                         pd.size  = size_out;
                         memcpy(buf, buf_out, pd.size);

#endif

Quoted 6 lines
Quoted 3 lines
Quoted 1 line

If we are on the same page, I'll accept your patch.

Right, I understand.

I think the primary intention is to let crash determine early if it is able to open a given dump or not. And that's a good idea!

The primary intention is informational only, either for "help -[nD]" or for the "crash -d# ..." display which dumps the header contents immediately
after reading it. As it is now, if crash is unable to support the compression
type because the relevant library was not built in, it will accept the vmcore, but will fail upon the first read attempt with a message "uncompress failed: no lzo compression support" (or snappy).

Quoted 7 lines

From a semantic point of view, the status field in the header should be a logical OR of all compression methods used in the coredump. AFAICS, Dave's patch would actually work for the current implementation.

Did I miss something?

Petr T

I don't think so -- Atsushi's point was simply that there is a good probability that there will be a mixture of compressed and uncompressed pages.

Dave

_______________________________________________
kexec mailing list
[email protected]
http://lists.infradead.org/mailman/listinfo/kexec