Skip to contentSkip to conversation

Clean up of gzip decompressor

PATCH In review Daniel P. Smith 3 participants 23 messages Apr 24, 2024 – May 9, 2024

lore
Share & tools
lore
#0 lore raw

v3 cover Clean up of gzip decompressor

An issue ran into by hyperlaunch was the need to use the gzip decompressor multiple times. The current implementation fails when reused due to tainting of decompressor state from a previous usage. This series seeks to colocate the gzip unit files under a single directory similar to the other decompression algorithms. To enable the refactoring of the state tracking, the code is then cleaned up in line with Xen coding style.

Changes in v3:
- patch "xen/gzip: Drop huffman code table tracking" was merged
- patch "xen/gzip: Remove custom memory allocator" was merged
- patch "xen/gzip: Drop unused define checks" was merged
- move of the decompressor state into struct was broken up to ease review
- expanded macros that were either only used once or obsfucated the logic
- adjusted variable types as needed to be more appropriate for their usage

Changes in v2:
- patch "xen/gzip: Colocate gunzip code files" was merged
- dropped ifdef chunks that are never enabled
- addressed formatting changes missed in v1
- replaced custom memory allocator with xalloc_bytes()
- renamed gzip_data to gzip_state
- moved gzip_state to being dynamically allocated
- changed crc table to the explicit size of uint32_t
- instead of moving huffman tracking into state, dropped altogether

Daniel P. Smith (8):
gzip: clean up comments and fix code alignment
gzip: refactor and expand macros
gzip: refactor the gunzip window into common state
gzip: move window pointer into gunzip state
gzip: move input buffer handling into gunzip state
gzip: move output buffer into gunzip state
gzip: move bitbuffer into gunzip state
gzip: move crc state into gunzip state

 xen/common/gzip/gunzip.c  |  84 ++--
 xen/common/gzip/inflate.c | 948 +++++++++++++++++++-------------------
 2 files changed, 516 insertions(+), 516 deletions(-)
Signature · 1 line
2.30.2
#1 lore raw

PATCH v3 1/8 gzip: clean up comments and fix code alignment

This commit cleans up the comments and fixes the code alignment using Xen coding style. This is done to make the code more legible before refactoring.

---
 xen/common/gzip/gunzip.c  |  14 +-
 xen/common/gzip/inflate.c | 787 +++++++++++++++++++-------------------
 2 files changed, 406 insertions(+), 395 deletions(-)
xen/common/gzip/gunzip.c +11 -3
index 53cee9ee178a..d07c451cd875 100644
@@ -54,11 +54,10 @@ static __init void error(const char *x)
 
 static __init int fill_inbuf(void)
 {
-        error("ran out of input data");
-        return 0;
+    error("ran out of input data");
+    return 0;
 }
 
-
 #include "inflate.c"
 
 static __init void flush_window(void)
@@ -122,3 +121,12 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len)
 
     return rc;
 }
+
+/*
+ * Local variables:
+ * mode: C
+ * c-file-style: "BSD"
+ * c-basic-offset: 4
+ * indent-tabs-mode: nil
+ * End:
+ */
xen/common/gzip/inflate.c +395 -392

1,042 lines not on this page · raw message

Signature · 1 line
2.30.2
#3 lore raw

PATCH v3 2/8 gzip: refactor and expand macros

In reply to Daniel P. Smith

This commit refactors macros into proper static functions. It in-place expands the flush_output macro, allowing the clear removal of the dead code underneath the underrun label.

---
 xen/common/gzip/gunzip.c  | 14 +++++----
 xen/common/gzip/inflate.c | 61 ++++++++++++++-------------------------
 2 files changed, 30 insertions(+), 45 deletions(-)
xen/common/gzip/gunzip.c +8 -6
index d07c451cd875..b7cadadcca8b 100644
@@ -25,8 +25,6 @@ typedef unsigned char   uch;
 typedef unsigned short  ush;
 typedef unsigned long   ulg;
 
-#define get_byte()      (inptr < insize ? inbuf[inptr++] : fill_inbuf())
-
 /* Diagnostic functions */
 #ifdef DEBUG
 #  define Assert(cond, msg) do { if (!(cond)) error(msg); } while (0)
@@ -52,10 +50,14 @@ static __init void error(const char *x)
     panic("%s\n", x);
 }
 
-static __init int fill_inbuf(void)
-{
-    error("ran out of input data");
-    return 0;
+static __init uch get_byte(void) {
+    if ( inptr >= insize )
+    {
+        error("ran out of input data");
+        return 0; /* should never reach */
+    }
+
+    return inbuf[inptr++];
 }
 
 #include "inflate.c"
xen/common/gzip/inflate.c +22 -39

139 lines not on this page · raw message

Signature · 1 line
2.30.2
#4 lore raw

Re: [PATCH v3 2/8] gzip: refactor and expand macros

Quoted 3 lines · Daniel P. Smith

This commit refactors macros into proper static functions. It in-place expands the flush_output macro, allowing the clear removal of the dead code underneath the underrun label.

But it's NEXTBYTE() which uses the label, not flush_output(). I'm actually unconvinced of its expanding / removal.

Quoted 12 lines
xen/common/gzip/gunzip.c +0 -2
@@ -25,8 +25,6 @@ typedef unsigned char   uch;
 typedef unsigned short  ush;
 typedef unsigned long   ulg;
 
-#define get_byte()      (inptr < insize ? inbuf[inptr++] : fill_inbuf())
-
 /* Diagnostic functions */
 #ifdef DEBUG
 #  define Assert(cond, msg) do { if (!(cond)) error(msg); } while (0)

xen/common/gzip/gunzip.c static __init void error(const char *x) · line 58

Quoted 7 lines
xen/common/gzip/gunzip.c +0 -4
@@ -52,10 +50,14 @@ static __init void error(const char *x)
     panic("%s\n", x);
 }
 
-static __init int fill_inbuf(void)
-{
-    error("ran out of input data");
-    return 0;

I'm not convinced of the removal of this as a separate function. It only calling error() right now could change going forward, so I'd at least expect a little bit of a justification.

xen/common/gzip/gunzip.c static __init void error(const char *x) · line 53

Quoted 1 line
xen/common/gzip/gunzip.c +1 -0
+static __init uch get_byte(void) {

Nit: Brace goes on its own line. Also for (effectively) new code it would be nice if __init (and alike) would be placed "canonically", i.e. between type and identifier.

Quoted 6 lines
xen/common/gzip/inflate.c +0 -0
@@ -119,6 +119,18 @@ static char rcsid[] = "#Id: inflate.c,v 0.14 1993/06/10 13:27:04 jloup Exp #";
 
 #endif /* !__XEN__ */
 

xen/common/gzip/inflate.c static char rcsid[] = "#Id: inflate.c,v 0.14 1993/06/10 13:27:04 jloup Exp #"; · line 131

Quoted 10 lines
xen/common/gzip/inflate.c +10 -0
+/*
+ * The inflate algorithm uses a sliding 32 K byte window on the uncompressed
+ * stream to find repeated byte strings.  This is implemented here as a
+ * circular buffer.  The index is updated simply by incrementing and then
+ * ANDing with 0x7fff (32K-1).
+ *
+ * It is left to other modules to supply the 32 K area.  It is assumed
+ * to be usable as if it were declared "uch slide[32768];" or as just
+ * "uch *slide;" and then malloc'ed in the latter case.  The definition
+ * must be in unzip.h, included above.

Nit: s/definition/declaration/ ?

Quoted 9 lines
xen/common/gzip/inflate.c +2 -0
+ */
+#define wp outcnt
 #define slide window
 
 /*
@@ -150,21 +162,6 @@ static int inflate_dynamic(void);
 static int inflate_block(int *);
 static int inflate(void);
 

xen/common/gzip/inflate.c static int inflate_dynamic(void); · line 162

Quoted 10 lines
xen/common/gzip/inflate.c +0 -10
-/*
- * The inflate algorithm uses a sliding 32 K byte window on the uncompressed
- * stream to find repeated byte strings.  This is implemented here as a
- * circular buffer.  The index is updated simply by incrementing and then
- * ANDing with 0x7fff (32K-1).
- *
- * It is left to other modules to supply the 32 K area.  It is assumed
- * to be usable as if it were declared "uch slide[32768];" or as just
- * "uch *slide;" and then malloc'ed in the latter case.  The definition
- * must be in unzip.h, included above.

Oh, an earlier comment just moves up. Is there really a need for this extra churn?

xen/common/gzip/inflate.c static const ush mask_bits[] = { · line 224

Quoted 6 lines
xen/common/gzip/inflate.c +1 -1
@@ -224,7 +221,7 @@ static const ush mask_bits[] = {
     0x01ff, 0x03ff, 0x07ff, 0x0fff, 0x1fff, 0x3fff, 0x7fff, 0xffff
 };
 
-#define NEXTBYTE()  ({ int v = get_byte(); if (v < 0) goto underrun; (uch)v; })
+#define NEXTBYTE()  (get_byte()) /* get_byte will panic on failure */

Nit: No need for the outer parentheses.

xen/common/gzip/inflate.c static int __init gunzip(void) · line 1139

Quoted 8 lines
xen/common/gzip/inflate.c +2 -2
@@ -1148,8 +1135,8 @@ static int __init gunzip(void)
     NEXTBYTE();
     NEXTBYTE();
 
-    (void)NEXTBYTE();  /* Ignore extra flags for the moment */
-    (void)NEXTBYTE();  /* Ignore OS type for the moment */
+    NEXTBYTE();  /* Ignore extra flags for the moment */
+    NEXTBYTE();  /* Ignore OS type for the moment */

In Misra discussions there were indications that such casts may need (re-) introducing. Perhaps better leave this alone, the more when it's not really fitting the patch's purpose?

Jan

#5 lore raw

PATCH v3 3/8 gzip: refactor the gunzip window into common state

In reply to Daniel P. Smith

Begin moving core state, in this case the gunzip window, into struct gunzip_state to allow a per decompression instance. In doing so, drop the define aliasing of window to slide.

---
 xen/common/gzip/gunzip.c  | 21 ++++++++----
 xen/common/gzip/inflate.c | 68 +++++++++++++++++++--------------------
 2 files changed, 48 insertions(+), 41 deletions(-)
xen/common/gzip/gunzip.c +14 -7
index b7cadadcca8b..e47f10ae19ad 100644
@@ -4,10 +4,12 @@
 #include <xen/lib.h>
 #include <xen/mm.h>
 
-static unsigned char *__initdata window;
-
 #define WSIZE           0x80000000U
 
+struct gunzip_state {
+    unsigned char *window;
+};
+
 static unsigned char *__initdata inbuf;
 static unsigned int __initdata insize;
 
@@ -43,7 +45,7 @@ typedef unsigned long   ulg;
 #endif
 
 static long __initdata bytes_out;
-static void flush_window(void);
+static void flush_window(struct gunzip_state *s);
 
 static __init void error(const char *x)
 {
@@ -62,7 +64,7 @@ static __init uch get_byte(void) {
 
 #include "inflate.c"
 
-static __init void flush_window(void)
+static __init void flush_window(struct gunzip_state *s)
 {
     /*
      * The window is equal to the output buffer therefore only need to
@@ -72,7 +74,7 @@ static __init void flush_window(void)
     unsigned int n;
     unsigned char *in, ch;
 
-    in = window;
+    in = s->window;
     for ( n = 0; n < outcnt; n++ )
     {
         ch = *in++;
@@ -99,12 +101,17 @@ __init int gzip_check(char *image, unsigned long image_len)
 
 __init int perform_gunzip(char *output, char *image, unsigned long image_len)
 {
+    struct gunzip_state *s;
     int rc;
 
     if ( !gzip_check(image, image_len) )
         return 1;
 
-    window = (unsigned char *)output;
+    s = (struct gunzip_state *)malloc(sizeof(struct gunzip_state));
+    if ( !s )
+        return -ENOMEM;
+
+    s->window = (unsigned char *)output;
     inbuf = (unsigned char *)image;
     insize = image_len;
     inptr = 0;
@@ -112,7 +119,7 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len)
 
     makecrc();
 
-    if ( gunzip() < 0 )
+    if ( gunzip(s) < 0 )
     {
         rc = -EINVAL;
     }
xen/common/gzip/inflate.c +34 -34

196 lines not on this page · raw message

Signature · 1 line
2.30.2
#6 lore raw

Re: [PATCH v3 3/8] gzip: refactor the gunzip window into common state

Quoted 58 lines · Daniel P. Smith

Begin moving core state, in this case the gunzip window, into struct gunzip_state to allow a per decompression instance. In doing so, drop the define aliasing of window to slide.

---
  xen/common/gzip/gunzip.c  | 21 ++++++++----
  xen/common/gzip/inflate.c | 68 +++++++++++++++++++--------------------
  2 files changed, 48 insertions(+), 41 deletions(-)
xen/common/gzip/gunzip.c +7 -5
index b7cadadcca8b..e47f10ae19ad 100644
@@ -4,10 +4,12 @@
  #include <xen/lib.h>
  #include <xen/mm.h>
 
-static unsigned char *__initdata window;
-
  #define WSIZE           0x80000000U
 
+struct gunzip_state {
+    unsigned char *window;
+};
+
  static unsigned char *__initdata inbuf;
  static unsigned int __initdata insize;
 
@@ -43,7 +45,7 @@ typedef unsigned long   ulg;
  #endif
 
  static long __initdata bytes_out;
-static void flush_window(void);
+static void flush_window(struct gunzip_state *s);
 
  static __init void error(const char *x)
  {
@@ -62,7 +64,7 @@ static __init uch get_byte(void) {
 
  #include "inflate.c"
 
-static __init void flush_window(void)
+static __init void flush_window(struct gunzip_state *s)
  {
      /*
       * The window is equal to the output buffer therefore only need to
@@ -72,7 +74,7 @@ static __init void flush_window(void)
      unsigned int n;
      unsigned char *in, ch;
 
-    in = window;
+    in = s->window;
      for ( n = 0; n < outcnt; n++ )
      {
          ch = *in++;
@@ -99,12 +101,17 @@ __init int gzip_check(char *image, unsigned long image_len)
 

xen/common/gzip/gunzip.c __init int gzip_check(char *image, unsigned long image_len) · line 110

Quoted 10 lines
xen/common/gzip/gunzip.c +2 -1
  __init int perform_gunzip(char *output, char *image, unsigned long image_len)
  {
+    struct gunzip_state *s;
      int rc;
 
      if ( !gzip_check(image, image_len) )
          return 1;
 
-    window = (unsigned char *)output;
+    s = (struct gunzip_state *)malloc(sizeof(struct gunzip_state));

Looks like I inadvertently dropped the corresponding free when breaking up the monolithic patch.

v/r,
dps

#7 lore raw

PATCH v3 4/8 gzip: move window pointer into gunzip state

In reply to Daniel P. Smith

Move the window pointer, outcnt/wp, into struct gunzip_data. It was erroneously labeled as outcnt and then define aliased to wp, this eliminates the aliasing and only refers to as wp, the window pointer.

---
 xen/common/gzip/gunzip.c  | 11 +++++------
 xen/common/gzip/inflate.c | 17 ++++++++---------
 2 files changed, 13 insertions(+), 15 deletions(-)
xen/common/gzip/gunzip.c +5 -6
index e47f10ae19ad..11bfe45d8222 100644
@@ -8,6 +8,8 @@
 
 struct gunzip_state {
     unsigned char *window;
+    /* window pointer: */
+    unsigned int wp;
 };
 
 static unsigned char *__initdata inbuf;
@@ -16,9 +18,6 @@ static unsigned int __initdata insize;
 /* Index of next byte to be processed in inbuf: */
 static unsigned int __initdata inptr;
 
-/* Bytes in output buffer: */
-static unsigned int __initdata outcnt;
-
 #define malloc(a)       xmalloc_bytes(a)
 #define free(a)         xfree(a)
 #define memzero(s, n)   memset((s), 0, (n))
@@ -75,15 +74,15 @@ static __init void flush_window(struct gunzip_state *s)
     unsigned char *in, ch;
 
     in = s->window;
-    for ( n = 0; n < outcnt; n++ )
+    for ( n = 0; n < s->wp; n++ )
     {
         ch = *in++;
         c = crc_32_tab[((int)c ^ ch) & 0xff] ^ (c >> 8);
     }
     crc = c;
 
-    bytes_out += (unsigned long)outcnt;
-    outcnt = 0;
+    bytes_out += (unsigned long)s->wp;
+    s->wp = 0;
 }
 
 __init int gzip_check(char *image, unsigned long image_len)
xen/common/gzip/inflate.c +8 -9
index 5fa5c039c6d1..78b2f20a97ba 100644
@@ -130,7 +130,6 @@ static char rcsid[] = "#Id: inflate.c,v 0.14 1993/06/10 13:27:04 jloup Exp #";
  * "uch *window;" and then malloc'ed in the latter case.  The definition
  * must be in unzip.h, included above.
  */
-#define wp outcnt
 
 /*
  * Huffman code lookup table entry--this entry is four bytes for machines
@@ -556,7 +555,7 @@ static int __init inflate_codes(
     /* make local copies of globals */
     b = bb;                       /* initialize bit buffer */
     k = bk;
-    w = wp;                       /* initialize window position */
+    w = s->wp;                    /* initialize window position */
 
     /* inflate the coded data */
     ml = mask_bits[bl];           /* precompute masks for speed */
@@ -579,7 +578,7 @@ static int __init inflate_codes(
             Tracevv((stderr, "%c", s->window[w-1]));
             if (w == WSIZE)
             {
-                wp = w;
+                s->wp = w;
                 flush_window(s);
                 w = 0;
             }
@@ -627,7 +626,7 @@ static int __init inflate_codes(
                     } while (--e);
                 if (w == WSIZE)
                 {
-                    wp = w;
+                    s->wp = w;
                     flush_window(s);
                     w = 0;
                 }
@@ -636,7 +635,7 @@ static int __init inflate_codes(
     }
 
     /* restore the globals from the locals */
-    wp = w;                       /* restore global window pointer */
+    s->wp = w;                    /* restore global window pointer */
     bb = b;                       /* restore global bit buffer */
     bk = k;
 
@@ -657,7 +656,7 @@ static int __init inflate_stored(struct gunzip_state *s)
     /* make local copies of globals */
     b = bb;                       /* initialize bit buffer */
     k = bk;
-    w = wp;                       /* initialize window position */
+    w = s->wp;                    /* initialize window position */
 
 
     /* go to byte boundary */
@@ -681,7 +680,7 @@ static int __init inflate_stored(struct gunzip_state *s)
         s->window[w++] = (uch)b;
         if (w == WSIZE)
         {
-            wp = w;
+            s->wp = w;
             flush_window(s);
             w = 0;
         }
@@ -689,7 +688,7 @@ static int __init inflate_stored(struct gunzip_state *s)
     }
 
     /* restore the globals from the locals */
-    wp = w;                       /* restore global window pointer */
+    s->wp = w;                    /* restore global window pointer */
     bb = b;                       /* restore global bit buffer */
     bk = k;
 
@@ -1004,7 +1003,7 @@ static int __init inflate(struct gunzip_state *s)
     int r;                /* result code */
 
     /* initialize window, bit buffer */
-    wp = 0;
+    s->wp = 0;
     bk = 0;
     bb = 0;
 
Signature · 1 line
2.30.2
Acked-by on v3 4/8
#8 lore raw

Re: [PATCH v3 4/8] gzip: move window pointer into gunzip state

Quoted 5 lines · Daniel P. Smith

Move the window pointer, outcnt/wp, into struct gunzip_data. It was erroneously labeled as outcnt and then define aliased to wp, this eliminates the aliasing and only refers to as wp, the window pointer.

#9 lore raw

PATCH v3 5/8 gzip: move input buffer handling into gunzip state

In reply to Daniel P. Smith

Move the input buffer handling, buffer pointer(inbuf), size(insize), and index(inptr), into gunzip state. Adjust functions and macros that consumed the input buffer to accept a struct gunzip_state reference.

---
 xen/common/gzip/gunzip.c  | 23 +++++-----
 xen/common/gzip/inflate.c | 92 +++++++++++++++++++--------------------
 2 files changed, 57 insertions(+), 58 deletions(-)
xen/common/gzip/gunzip.c +11 -12
index 11bfe45d8222..3fb9589b069e 100644
@@ -10,13 +10,12 @@ struct gunzip_state {
     unsigned char *window;
     /* window pointer: */
     unsigned int wp;
-};
-
-static unsigned char *__initdata inbuf;
-static unsigned int __initdata insize;
 
-/* Index of next byte to be processed in inbuf: */
-static unsigned int __initdata inptr;
+    unsigned char *inbuf;
+    size_t insize;
+    /* Index of next byte to be processed in inbuf: */
+    unsigned int inptr;
+};
 
 #define malloc(a)       xmalloc_bytes(a)
 #define free(a)         xfree(a)
@@ -51,14 +50,14 @@ static __init void error(const char *x)
     panic("%s\n", x);
 }
 
-static __init uch get_byte(void) {
-    if ( inptr >= insize )
+static __init uch get_byte(struct gunzip_state *s) {
+    if ( s->inptr >= s->insize )
     {
         error("ran out of input data");
         return 0; /* should never reach */
     }
 
-    return inbuf[inptr++];
+    return s->inbuf[s->inptr++];
 }
 
 #include "inflate.c"
@@ -111,9 +110,9 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len)
         return -ENOMEM;
 
     s->window = (unsigned char *)output;
-    inbuf = (unsigned char *)image;
-    insize = image_len;
-    inptr = 0;
+    s->inbuf = (unsigned char *)image;
+    s->insize = image_len;
+    s->inptr = 0;
     bytes_out = 0;
 
     makecrc();
xen/common/gzip/inflate.c +46 -46

234 lines not on this page · raw message

Signature · 1 line
2.30.2
#10 lore raw

Re: [PATCH v3 5/8] gzip: move input buffer handling into gunzip state

Quoted 7 lines · Daniel P. Smith
xen/common/gzip/gunzip.c +0 -1
@@ -10,13 +10,12 @@ struct gunzip_state {
     unsigned char *window;
     /* window pointer: */
     unsigned int wp;
-};

xen/common/gzip/gunzip.c struct gunzip_state { · line 17

Quoted 10 lines
xen/common/gzip/gunzip.c +4 -5
-
-static unsigned char *__initdata inbuf;
-static unsigned int __initdata insize;
 
-/* Index of next byte to be processed in inbuf: */
-static unsigned int __initdata inptr;
+    unsigned char *inbuf;
+    size_t insize;
+    /* Index of next byte to be processed in inbuf: */
+    unsigned int inptr;

I'm puzzled by the (suddenly) different types, seeing that ...

xen/common/gzip/gunzip.c static __init void error(const char *x) · line 54

Quoted 12 lines
xen/common/gzip/gunzip.c +3 -2
+};
 
 #define malloc(a)       xmalloc_bytes(a)
 #define free(a)         xfree(a)
@@ -51,14 +50,14 @@ static __init void error(const char *x)
     panic("%s\n", x);
 }
 
-static __init uch get_byte(void) {
-    if ( inptr >= insize )
+static __init uch get_byte(struct gunzip_state *s) {
+    if ( s->inptr >= s->insize )

... both are compared with one another.

Quoted 15 lines
xen/common/gzip/gunzip.c +4 -6
@@ -1129,29 +1129,29 @@ static int __init gunzip(struct gunzip_state *s)
         error("Input has invalid flags");
         return -1;
     }
-    NEXTBYTE(); /* Get timestamp */
-    NEXTBYTE();
-    NEXTBYTE();
-    NEXTBYTE();
+    NEXTBYTE(s); /* Get timestamp */
+    NEXTBYTE(s);
+    NEXTBYTE(s);
+    NEXTBYTE(s);
 
-    NEXTBYTE();  /* Ignore extra flags for the moment */
-    NEXTBYTE();  /* Ignore OS type for the moment */

xen/common/gzip/gunzip.c static int __init gunzip(struct gunzip_state *s) · line 1143

Quoted 10 lines
xen/common/gzip/gunzip.c +5 -3
+    NEXTBYTE(s);  /* Ignore extra flags for the moment */
+    NEXTBYTE(s);  /* Ignore OS type for the moment */
 
     if ((flags & EXTRA_FIELD) != 0) {
-        unsigned len = (unsigned)NEXTBYTE();
-        len |= ((unsigned)NEXTBYTE())<<8;
-        while (len--) (void)NEXTBYTE();
+        unsigned len = (unsigned)NEXTBYTE(s);
+        len |= ((unsigned)NEXTBYTE(s))<<8;
+        while (len--) (void)NEXTBYTE(s);

Would you mind moving the body of this while() to its own line, as you touch this anyway?

Quoted 4 lines
xen/common/gzip/gunzip.c +0 -0
     }
 
     /* Get original file name if it was truncated */
     if ((flags & ORIG_NAME) != 0) {

xen/common/gzip/gunzip.c static int __init gunzip(struct gunzip_state *s) · line 1155

Quoted 10 lines
xen/common/gzip/gunzip.c +2 -2
         /* Discard the old name */
-        while (NEXTBYTE() != 0) /* null */ ;
+        while (NEXTBYTE(s) != 0) /* null */ ;
     }
 
     /* Discard file comment if any */
     if ((flags & COMMENT) != 0) {
-        while (NEXTBYTE() != 0) /* null */ ;
+        while (NEXTBYTE(s) != 0) /* null */ ;
     }

For these two doing the same may help, too.

Jan

#11 lore raw

PATCH v3 6/8 gzip: move output buffer into gunzip state

In reply to Daniel P. Smith
---
 xen/common/gzip/gunzip.c  | 7 ++++---
 xen/common/gzip/inflate.c | 2 +-
 2 files changed, 5 insertions(+), 4 deletions(-)
xen/common/gzip/gunzip.c +4 -3
index 3fb9589b069e..95d924d36726 100644
@@ -15,6 +15,8 @@ struct gunzip_state {
     size_t insize;
     /* Index of next byte to be processed in inbuf: */
     unsigned int inptr;
+
+    unsigned long bytes_out;
 };
 
 #define malloc(a)       xmalloc_bytes(a)
@@ -42,7 +44,6 @@ typedef unsigned long   ulg;
 #  define Tracecv(c, x)
 #endif
 
-static long __initdata bytes_out;
 static void flush_window(struct gunzip_state *s);
 
 static __init void error(const char *x)
@@ -80,7 +81,7 @@ static __init void flush_window(struct gunzip_state *s)
     }
     crc = c;
 
-    bytes_out += (unsigned long)s->wp;
+    s->bytes_out += (unsigned long)s->wp;
     s->wp = 0;
 }
 
@@ -113,7 +114,7 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len)
     s->inbuf = (unsigned char *)image;
     s->insize = image_len;
     s->inptr = 0;
-    bytes_out = 0;
+    s->bytes_out = 0;
 
     makecrc();
 
xen/common/gzip/inflate.c +1 -1
index f1a3b04cef8f..bec8801df487 100644
@@ -1196,7 +1196,7 @@ static int __init gunzip(struct gunzip_state *s)
         error("crc error");
         return -1;
     }
-    if (orig_len != bytes_out) {
+    if (orig_len != s->bytes_out) {
         error("length error");
         return -1;
     }
Signature · 1 line
2.30.2
#12 lore raw

Re: [PATCH v3 6/8] gzip: move output buffer into gunzip state

xen/common/gzip/gunzip.c struct gunzip_state { · line 20

Quoted 9 lines · Daniel P. Smith
xen/common/gzip/gunzip.c +2 -0
@@ -15,6 +15,8 @@ struct gunzip_state {
     size_t insize;
     /* Index of next byte to be processed in inbuf: */
     unsigned int inptr;
+
+    unsigned long bytes_out;
 };

The conversion to unsigned long from ...

xen/common/gzip/gunzip.c typedef unsigned long ulg; · line 45

Quoted 5 lines
xen/common/gzip/gunzip.c +0 -1
@@ -42,7 +44,6 @@ typedef unsigned long   ulg;
 #  define Tracecv(c, x)
 #endif
 
-static long __initdata bytes_out;

... this originally wants justifying in the (then no longer empty) description. It's not a lot that needs saying, but such a type change cannot go entirely silently.

Jan

#13 lore raw

PATCH v3 7/8 gzip: move bitbuffer into gunzip state

In reply to Daniel P. Smith
---
 xen/common/gzip/gunzip.c  |  3 +++
 xen/common/gzip/inflate.c | 43 ++++++++++++++++++---------------------
 2 files changed, 23 insertions(+), 23 deletions(-)
xen/common/gzip/gunzip.c +3 -0
index 95d924d36726..0043ff8ac886 100644
@@ -17,6 +17,9 @@ struct gunzip_state {
     unsigned int inptr;
 
     unsigned long bytes_out;
+
+    unsigned long bb;      /* bit buffer */
+    unsigned int  bk;      /* bits in bit buffer */
 };
 
 #define malloc(a)       xmalloc_bytes(a)
xen/common/gzip/inflate.c +20 -23

109 lines not on this page · raw message

Signature · 1 line
2.30.2
Acked-by on v3 7/8
#14 lore raw

Re: [PATCH v3 7/8] gzip: move bitbuffer into gunzip state

Quoted 1 line · Daniel P. Smith
Quoted 5 lines
xen/common/gzip/inflate.c +0 -0
index bec8801df487..8da14880cfbe 100644

xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1020

Quoted 9 lines
xen/common/gzip/inflate.c +2 -2
@@ -1017,8 +1014,8 @@ static int __init inflate(struct gunzip_state *s)
     /* Undo too much lookahead. The next read will be byte aligned so we
      * can discard unused bits in the last meaningful byte.
      */
-    while (bk >= 8) {
-        bk -= 8;
+    while (s->bk >= 8) {
+        s->bk -= 8;
         s->inptr--;
     }

Isn't it just me, but isn't this just:

    s->inptr -= (s->bk >> 3);
    s->bk &= 7;

?

~Andrew

#15 lore raw

Re: [PATCH v3 7/8] gzip: move bitbuffer into gunzip state

Quoted 16 lines · Andrew Cooper

xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1020

Quoted 12 lines · Daniel P. Smith
xen/common/gzip/inflate.c +2 -2
@@ -1017,8 +1014,8 @@ static int __init inflate(struct gunzip_state *s)
     /* Undo too much lookahead. The next read will be byte aligned so we
      * can discard unused bits in the last meaningful byte.
      */
-    while (bk >= 8) {
-        bk -= 8;
+    while (s->bk >= 8) {
+        s->bk -= 8;
         s->inptr--;
     }

Isn't it just me, but isn't this just:

Quoted 4 lines

    s->inptr -= (s->bk >> 3);
    s->bk &= 7;

?

I'd say yes, if only there wasn't the comment talking of just a single byte, and even there supposedly some of the bits.

Jan

#16 lore raw

Re: [PATCH v3 7/8] gzip: move bitbuffer into gunzip state

Quoted 17 lines · Jan Beulich
Quoted 16 lines · Andrew Cooper

xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1020

Quoted 12 lines · Daniel P. Smith
xen/common/gzip/inflate.c +2 -2
@@ -1017,8 +1014,8 @@ static int __init inflate(struct gunzip_state *s)
     /* Undo too much lookahead. The next read will be byte aligned so we
      * can discard unused bits in the last meaningful byte.
      */
-    while (bk >= 8) {
-        bk -= 8;
+    while (s->bk >= 8) {
+        s->bk -= 8;
         s->inptr--;
     }

Isn't it just me, but isn't this just:

Quoted 7 lines
Quoted 4 lines

    s->inptr -= (s->bk >> 3);
    s->bk &= 7;

?

I'd say yes, if only there wasn't the comment talking of just a single byte, and even there supposedly some of the bits.

Talking of the comment: Considering what patch 1 supposedly does, how come that isn't Xen-style (yet)?

Jan

#17 lore raw

Re: [PATCH v3 7/8] gzip: move bitbuffer into gunzip state

Quoted 17 lines · Jan Beulich
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Andrew Cooper

xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1020

Quoted 12 lines · Daniel P. Smith
xen/common/gzip/inflate.c +2 -2
@@ -1017,8 +1014,8 @@ static int __init inflate(struct gunzip_state *s)
     /* Undo too much lookahead. The next read will be byte aligned so we
      * can discard unused bits in the last meaningful byte.
      */
-    while (bk >= 8) {
-        bk -= 8;
+    while (s->bk >= 8) {
+        s->bk -= 8;
         s->inptr--;
     }

Isn't it just me, but isn't this just:

Quoted 6 lines
Quoted 6 lines
Quoted 4 lines

    s->inptr -= (s->bk >> 3);
    s->bk &= 7;

?

I'd say yes, if only there wasn't the comment talking of just a single byte, and even there supposedly some of the bits.

The comments in this file leave a lot to be desired...

I'm reasonably confident they were not adjusted when a piece of userspace code was imported into Linux.

This cannot ever have been just a byte's worth of bits, seeing as the bit count is from 8 or more.  Right now it's an unsigned long's worth of bits.

Quoted 2 lines

Talking of the comment: Considering what patch 1 supposedly does, how come that isn't Xen-style (yet)?

I had been collecting some minor fixes like this to fold into patch 1 when I committed the whole series.

I'll see if I can fold them elsewhere.

~Andrew

#18 lore raw

PATCH v3 8/8 gzip: move crc state into gunzip state

In reply to Daniel P. Smith

Move the crc and its state into struct gunzip_state. In the process, expand the only use of CRC_VALUE as it is hides what is being compared.

---
 xen/common/gzip/gunzip.c  | 11 +++++++----
 xen/common/gzip/inflate.c | 14 +++++---------
 2 files changed, 12 insertions(+), 13 deletions(-)
xen/common/gzip/gunzip.c +7 -4
index 0043ff8ac886..26d2a4aa9d36 100644
@@ -20,6 +20,9 @@ struct gunzip_state {
 
     unsigned long bb;      /* bit buffer */
     unsigned int  bk;      /* bits in bit buffer */
+
+    uint32_t crc_32_tab[256];
+    uint32_t crc;
 };
 
 #define malloc(a)       xmalloc_bytes(a)
@@ -72,7 +75,7 @@ static __init void flush_window(struct gunzip_state *s)
      * The window is equal to the output buffer therefore only need to
      * compute the crc.
      */
-    unsigned long c = crc;
+    unsigned int c = s->crc;
     unsigned int n;
     unsigned char *in, ch;
 
@@ -80,9 +83,9 @@ static __init void flush_window(struct gunzip_state *s)
     for ( n = 0; n < s->wp; n++ )
     {
         ch = *in++;
-        c = crc_32_tab[((int)c ^ ch) & 0xff] ^ (c >> 8);
+        c = s->crc_32_tab[((int)c ^ ch) & 0xff] ^ (c >> 8);
     }
-    crc = c;
+    s->crc = c;
 
     s->bytes_out += (unsigned long)s->wp;
     s->wp = 0;
@@ -119,7 +122,7 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len)
     s->inptr = 0;
     s->bytes_out = 0;
 
-    makecrc();
+    makecrc(s);
 
     if ( gunzip(s) < 0 )
     {
xen/common/gzip/inflate.c +5 -9
index 8da14880cfbe..dc940e59d853 100644
@@ -1032,16 +1032,12 @@ static int __init inflate(struct gunzip_state *s)
  *
  **********************************************************************/
 
-static ulg __initdata crc_32_tab[256];
-static ulg __initdata crc;  /* initialized in makecrc() so it'll reside in bss */
-#define CRC_VALUE (crc ^ 0xffffffffUL)
-
 /*
  * Code to compute the CRC-32 table. Borrowed from
  * gzip-1.0.3/makecrc.c.
  */
 
-static void __init makecrc(void)
+static void __init makecrc(struct gunzip_state *s)
 {
 /* Not copyrighted 1990 Mark Adler */
 
@@ -1058,7 +1054,7 @@ static void __init makecrc(void)
     for (i = 0; i < sizeof(p)/sizeof(int); i++)
         e |= 1L << (31 - p[i]);
 
-    crc_32_tab[0] = 0;
+    s->crc_32_tab[0] = 0;
 
     for (i = 1; i < 256; i++)
     {
@@ -1069,11 +1065,11 @@ static void __init makecrc(void)
             if (k & 1)
                 c ^= e;
         }
-        crc_32_tab[i] = c;
+        s->crc_32_tab[i] = c;
     }
 
     /* this is initialized here so this code could reside in ROM */
-    crc = (ulg)0xffffffffUL; /* shift register contents */
+    s->crc = (ulg)~0u; /* shift register contents */
 }
 
 /* gzip flag byte */
@@ -1189,7 +1185,7 @@ static int __init gunzip(struct gunzip_state *s)
     orig_len |= (ulg) NEXTBYTE(s) << 24;
 
     /* Validate decompression */
-    if (orig_crc != CRC_VALUE) {
+    if (orig_crc != (s->crc ^ ~0u)) {
         error("crc error");
         return -1;
     }
Signature · 1 line
2.30.2
#19 lore raw

Re: [PATCH v3 8/8] gzip: move crc state into gunzip state

Quoted 2 lines · Daniel P. Smith

Move the crc and its state into struct gunzip_state. In the process, expand the only use of CRC_VALUE as it is hides what is being compared.

"All variables here should be uint32_t rather than unsigned long, which halves the storage space required."

Quoted 5 lines
xen/common/gzip/inflate.c +0 -0
index 8da14880cfbe..dc940e59d853 100644

xen/common/gzip/inflate.c static void __init makecrc(void) · line 1072

Quoted 10 lines
xen/common/gzip/inflate.c +2 -2
@@ -1069,11 +1065,11 @@ static void __init makecrc(void)
             if (k & 1)
                 c ^= e;
         }
-        crc_32_tab[i] = c;
+        s->crc_32_tab[i] = c;
     }
 
     /* this is initialized here so this code could reside in ROM */
-    crc = (ulg)0xffffffffUL; /* shift register contents */
+    s->crc = (ulg)~0u; /* shift register contents */

The (ulg) wasn't ever necessary, and can be dropped as part of this cleanup.

Can fix on commit.

~Andrew

#20 lore raw

Re: [PATCH v3 8/8] gzip: move crc state into gunzip state

In reply to Daniel P. Smith
Quoted 2 lines · Daniel P. Smith

Move the crc and its state into struct gunzip_state. In the process, expand the only use of CRC_VALUE as it is hides what is being compared.

Andrew mentioned uint32_t only for the description, but I think you want (maybe even need) to go further and actually use that type also, e.g.

xen/common/gzip/gunzip.c struct gunzip_state { · line 26

Quoted 10 lines
xen/common/gzip/gunzip.c +3 -0
@@ -20,6 +20,9 @@ struct gunzip_state {
 
     unsigned long bb;      /* bit buffer */
     unsigned int  bk;      /* bits in bit buffer */
+
+    uint32_t crc_32_tab[256];
+    uint32_t crc;
 };

... not just here, but also ...

xen/common/gzip/gunzip.c static __init void flush_window(struct gunzip_state *s) · line 78

Quoted 6 lines
xen/common/gzip/gunzip.c +1 -1
@@ -72,7 +75,7 @@ static __init void flush_window(struct gunzip_state *s)
      * The window is equal to the output buffer therefore only need to
      * compute the crc.
      */
-    unsigned long c = crc;
+    unsigned int c = s->crc;

... here.

xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1037

Quoted 9 lines
xen/common/gzip/inflate.c +0 -3
@@ -1032,16 +1032,12 @@ static int __init inflate(struct gunzip_state *s)
  *
  **********************************************************************/
 
-static ulg __initdata crc_32_tab[256];
-static ulg __initdata crc;  /* initialized in makecrc() so it'll reside in bss */
-#define CRC_VALUE (crc ^ 0xffffffffUL)

Note how this is _not_ same as ~0u, also ...

xen/common/gzip/inflate.c static void __init makecrc(void) · line 1072

Quoted 11 lines
xen/common/gzip/inflate.c +2 -2
@@ -1069,11 +1065,11 @@ static void __init makecrc(void)
             if (k & 1)
                 c ^= e;
         }
-        crc_32_tab[i] = c;
+        s->crc_32_tab[i] = c;
     }
 
     /* this is initialized here so this code could reside in ROM */
-    crc = (ulg)0xffffffffUL; /* shift register contents */
+    s->crc = (ulg)~0u; /* shift register contents */

... applicable here then: sizeof(int) >= 4, not ==.

As Andrew indicates, the cast previously wasn't needed here. Keeping it is at best misleading, when s->crc isn't of that type anymore.

Finally please stick to upper-case number suffixes; see all the related Misra changes, which (iirc) put in place only upper-case ones.

Jan

#21 lore raw
In reply to Daniel P. Smith
Quoted 7 lines · Daniel P. Smith

An issue ran into by hyperlaunch was the need to use the gzip decompressor multiple times. The current implementation fails when reused due to tainting of decompressor state from a previous usage. This series seeks to colocate the gzip unit files under a single directory similar to the other decompression algorithms. To enable the refactoring of the state tracking, the code is then cleaned up in line with Xen coding style.

Forgot to add this comment to the cover letter.

Concern was raised about taking updates from original source to the code. In this case the original source is from the Linux kernel, which can be found in lib/inflate.c, and added to Xen in 2009. The last time there was a logic change to that code in the Linux kernel was in 2008, before it was added to Xen. Since then, it has only been updated for changes made to included headers. If fact, as far as I can see, while the file is still in place, nothing uses it. For zlib decompression, three is a new lib/zlib_deflate code base that is used. The scope of this work is not to completely fix/replace zlib decompression for Xen, but to stabilize it to allow hyperlaunch to decompress more than one zlib compressed kernel.

V/r,
Daniel P. Smith

#22 lore raw
In reply to Daniel P. Smith
Quoted 26 lines · Daniel P. Smith

An issue ran into by hyperlaunch was the need to use the gzip decompressor multiple times. The current implementation fails when reused due to tainting of decompressor state from a previous usage. This series seeks to colocate the gzip unit files under a single directory similar to the other decompression algorithms. To enable the refactoring of the state tracking, the code is then cleaned up in line with Xen coding style.

Changes in v3:
- patch "xen/gzip: Drop huffman code table tracking" was merged
- patch "xen/gzip: Remove custom memory allocator" was merged
- patch "xen/gzip: Drop unused define checks" was merged
- move of the decompressor state into struct was broken up to ease review
- expanded macros that were either only used once or obsfucated the logic
- adjusted variable types as needed to be more appropriate for their usage

Changes in v2:
- patch "xen/gzip: Colocate gunzip code files" was merged
- dropped ifdef chunks that are never enabled
- addressed formatting changes missed in v1
- replaced custom memory allocator with xalloc_bytes()
- renamed gzip_data to gzip_state
- moved gzip_state to being dynamically allocated
- changed crc table to the explicit size of uint32_t
- instead of moving huffman tracking into state, dropped altogether

Quoted 9 lines

Daniel P. Smith (8):
gzip: clean up comments and fix code alignment
gzip: refactor and expand macros
gzip: refactor the gunzip window into common state
gzip: move window pointer into gunzip state
gzip: move input buffer handling into gunzip state
gzip: move output buffer into gunzip state
gzip: move bitbuffer into gunzip state
gzip: move crc state into gunzip state

In hindsight my suggestion that lead to "refactor and expand macros" wasn't great.

We want to keep the underrun labels for a future when the error handling isn't panic().  After that, expanding flush_window() is better folded into "refactor the gunzip window into common state" to reduce the churn.

As that was my fault, and unpicking it is reasonably hard, and we're on a tight deadline for 4.19 now, I've gone ahead and unpicked this.

Also I've rebased over Jan's patch addressing the memory leaks reported by Coverity, as the two sets of fixes collide, along with various other minor notes.

I've conferred with Daniel who is happy with the aformentioned changes.

This area of code still has a lot of work needing doing on it, but removing the use of global state is a concrete improvement in the status quo.

~Andrew