Clean up of gzip decompressor
v3 cover Clean up of gzip decompressor
An issue ran into by hyperlaunch was the need to use the gzip decompressor multiple times. The current implementation fails when reused due to tainting of decompressor state from a previous usage. This series seeks to colocate the gzip unit files under a single directory similar to the other decompression algorithms. To enable the refactoring of the state tracking, the code is then cleaned up in line with Xen coding style.
Changes in v3:
- patch "xen/gzip: Drop huffman code table tracking" was merged
- patch "xen/gzip: Remove custom memory allocator" was merged
- patch "xen/gzip: Drop unused define checks" was merged
- move of the decompressor state into struct was broken up to ease review
- expanded macros that were either only used once or obsfucated the logic
- adjusted variable types as needed to be more appropriate for their usage
Changes in v2:
- patch "xen/gzip: Colocate gunzip code files" was merged
- dropped ifdef chunks that are never enabled
- addressed formatting changes missed in v1
- replaced custom memory allocator with xalloc_bytes()
- renamed gzip_data to gzip_state
- moved gzip_state to being dynamically allocated
- changed crc table to the explicit size of uint32_t
- instead of moving huffman tracking into state, dropped altogether
Daniel P. Smith (8):
gzip: clean up comments and fix code alignment
gzip: refactor and expand macros
gzip: refactor the gunzip window into common state
gzip: move window pointer into gunzip state
gzip: move input buffer handling into gunzip state
gzip: move output buffer into gunzip state
gzip: move bitbuffer into gunzip state
gzip: move crc state into gunzip state
xen/common/gzip/gunzip.c | 84 ++-- xen/common/gzip/inflate.c | 948 +++++++++++++++++++------------------- 2 files changed, 516 insertions(+), 516 deletions(-)
Signature · 1 line
2.30.2
PATCH v3 1/8 gzip: clean up comments and fix code alignment
This commit cleans up the comments and fixes the code alignment using Xen coding style. This is done to make the code more legible before refactoring.
--- xen/common/gzip/gunzip.c | 14 +- xen/common/gzip/inflate.c | 787 +++++++++++++++++++------------------- 2 files changed, 406 insertions(+), 395 deletions(-)
xen/common/gzip/gunzip.c +11 -3
@@ -54,11 +54,10 @@ static __init void error(const char *x)static __init int fill_inbuf (void ) { -error ("ran out of input data" ); -return 0 ; +error ("ran out of input data" ); +return 0 ; } -#include "inflate.c" static __init void flush_window (void )
@@ -122,3 +121,12 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len)return rc; } + +/* +* Local variables: +* mode: C +* c-file-style: "BSD" +* c-basic-offset: 4 +* indent-tabs-mode: nil +* End: +*/
xen/common/gzip/inflate.c +395 -392
1,042 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH v3 1/8] gzip: clean up comments and fix code alignment
Quoted 4 lines · Daniel P. Smith
This commit cleans up the comments and fixes the code alignment using Xen coding style. This is done to make the code more legible before refactoring.
Signed-off-by Daniel P. Smith <[email protected]>
PATCH v3 2/8 gzip: refactor and expand macros
In reply to Daniel P. SmithThis commit refactors macros into proper static functions. It in-place expands the flush_output macro, allowing the clear removal of the dead code underneath the underrun label.
--- xen/common/gzip/gunzip.c | 14 +++++---- xen/common/gzip/inflate.c | 61 ++++++++++++++------------------------- 2 files changed, 30 insertions(+), 45 deletions(-)
xen/common/gzip/gunzip.c +8 -6
@@ -25,8 +25,6 @@ typedef unsigned char uch;typedef unsigned short ush ;typedef unsigned long ulg ; -#define get_byte () (inptr < insize ? inbuf[inptr++] : fill_inbuf()) -/* Diagnostic functions */ #ifdef DEBUG # define Assert (cond, msg) do { if (!(cond)) error(msg); } while (0)
@@ -52,10 +50,14 @@ static __init void error(const char *x)panic ("%s\n" , x); } -static __init int fill_inbuf (void ) -{ -error ("ran out of input data" ); -return 0 ; +static __init uch get_byte (void ) { +if ( inptr >= insize ) + { +error ("ran out of input data" ); +return 0 ;/* should never reach */ + } + +return inbuf[inptr++]; }#include "inflate.c"
xen/common/gzip/inflate.c +22 -39
139 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH v3 2/8] gzip: refactor and expand macros
Quoted 3 lines · Daniel P. Smith
This commit refactors macros into proper static functions. It in-place expands the
flush_outputmacro, allowing the clear removal of the dead code underneath theunderrunlabel.
But it's NEXTBYTE() which uses the label, not flush_output(). I'm actually unconvinced of its expanding / removal.
Quoted 12 lines
xen/common/gzip/gunzip.c +0
-2@@ -25,8 +25,6 @@ typedef unsigned char uch;typedef unsigned short ush ;typedef unsigned long ulg ; -#define get_byte () (inptr < insize ? inbuf[inptr++] : fill_inbuf()) -/* Diagnostic functions */ #ifdef DEBUG # define Assert (cond, msg) do { if (!(cond)) error(msg); } while (0)
xen/common/gzip/gunzip.c static __init void error(const char *x) · line 58
Quoted 7 lines
xen/common/gzip/gunzip.c +0
-4@@ -52,10 +50,14 @@ static __init void error(const char *x)panic ("%s\n" , x); } -static __init int fill_inbuf (void ) -{ -error ("ran out of input data" ); -return 0 ;
I'm not convinced of the removal of this as a separate function. It only calling error() right now could change going forward, so I'd at least expect a little bit of a justification.
xen/common/gzip/gunzip.c static __init void error(const char *x) · line 53
Quoted 1 line
xen/common/gzip/gunzip.c +1
-0+static __init uch get_byte (void ) {
Nit: Brace goes on its own line. Also for (effectively) new code it would be nice if __init (and alike) would be placed "canonically", i.e. between type and identifier.
Quoted 6 lines
xen/common/gzip/inflate.c +0
-0@@ -119,6 +119,18 @@ static char rcsid[] = "#Id: inflate.c,v 0.14 1993/06/10 13:27:04 jloup Exp #";#endif /* !__XEN__ */
xen/common/gzip/inflate.c static char rcsid[] = "#Id: inflate.c,v 0.14 1993/06/10 13:27:04 jloup Exp #"; · line 131
Quoted 10 lines
xen/common/gzip/inflate.c +10
-0+/* +* The inflate algorithm uses a sliding 32 K byte window on the uncompressed +* stream to find repeated byte strings. This is implemented here as a +* circular buffer. The index is updated simply by incrementing and then +* ANDing with 0x7fff (32K-1). +* +* It is left to other modules to supply the 32 K area. It is assumed +* to be usable as if it were declared "uch slide[32768];" or as just +* "uch *slide;" and then malloc'ed in the latter case. The definition +* must be in unzip.h, included above.
Nit: s/definition/declaration/ ?
Quoted 9 lines
xen/common/gzip/inflate.c +2
-0+*/ +#define wp outcnt#define slide window/* @@ -150,21 +162,6 @@ static int inflate_dynamic(void);static int inflate_block (int *);static int inflate (void );
xen/common/gzip/inflate.c static int inflate_dynamic(void); · line 162
Quoted 10 lines
xen/common/gzip/inflate.c +0
-10-/* -* The inflate algorithm uses a sliding 32 K byte window on the uncompressed -* stream to find repeated byte strings. This is implemented here as a -* circular buffer. The index is updated simply by incrementing and then -* ANDing with 0x7fff (32K-1). -* -* It is left to other modules to supply the 32 K area. It is assumed -* to be usable as if it were declared "uch slide[32768];" or as just -* "uch *slide;" and then malloc'ed in the latter case. The definition -* must be in unzip.h, included above.
Oh, an earlier comment just moves up. Is there really a need for this extra churn?
xen/common/gzip/inflate.c static const ush mask_bits[] = { · line 224
Quoted 6 lines
xen/common/gzip/inflate.c +1
-1@@ -224,7 +221,7 @@ static const ush mask_bits[] = { 0x01ff, 0x03ff, 0x07ff, 0x0fff, 0x1fff, 0x3fff, 0x7fff, 0xffff }; -#define NEXTBYTE () ({ int v = get_byte(); if (v < 0) goto underrun; (uch)v; }) +#define NEXTBYTE () (get_byte())/* get_byte will panic on failure */
Nit: No need for the outer parentheses.
xen/common/gzip/inflate.c static int __init gunzip(void) · line 1139
Quoted 8 lines
xen/common/gzip/inflate.c +2
-2@@ -1148,8 +1135,8 @@ static int __init gunzip(void)NEXTBYTE ();NEXTBYTE (); - (void )NEXTBYTE ();/* Ignore extra flags for the moment */ - (void )NEXTBYTE ();/* Ignore OS type for the moment */ +NEXTBYTE ();/* Ignore extra flags for the moment */ +NEXTBYTE ();/* Ignore OS type for the moment */
In Misra discussions there were indications that such casts may need (re-) introducing. Perhaps better leave this alone, the more when it's not really fitting the patch's purpose?
Jan
PATCH v3 3/8 gzip: refactor the gunzip window into common state
In reply to Daniel P. SmithBegin moving core state, in this case the gunzip window, into struct gunzip_state to allow a per decompression instance. In doing so, drop the define aliasing of window to slide.
--- xen/common/gzip/gunzip.c | 21 ++++++++---- xen/common/gzip/inflate.c | 68 +++++++++++++++++++-------------------- 2 files changed, 48 insertions(+), 41 deletions(-)
xen/common/gzip/gunzip.c +14 -7
@@ -4,10 +4,12 @@#include <xen/lib.h> #include <xen/mm.h> -static unsigned char *__initdata window; -#define WSIZE 0x80000000U +struct gunzip_state { +unsigned char *window; +}; +static unsigned char *__initdata inbuf;static unsigned int __initdata insize;
@@ -43,7 +45,7 @@ typedef unsigned long ulg;#endif static long __initdata bytes_out; -static void flush_window (void ); +static void flush_window (struct gunzip_state *s);static __init void error (const char *x) {
@@ -62,7 +64,7 @@ static __init uch get_byte(void) {#include "inflate.c" -static __init void flush_window (void ) +static __init void flush_window (struct gunzip_state *s) {/* * The window is equal to the output buffer therefore only need to
@@ -72,7 +74,7 @@ static __init void flush_window(void)unsigned int n;unsigned char *in, ch; - in = window; + in = s->window;for ( n =0 ; n < outcnt; n++ ) { ch = *in++;
@@ -99,12 +101,17 @@ __init int gzip_check(char *image, unsigned long image_len)__init int perform_gunzip (char *output,char *image,unsigned long image_len) { +struct gunzip_state *s;int rc;if ( !gzip_check (image, image_len) )return 1 ; - window = (unsigned char *)output; + s = (struct gunzip_state *)malloc (sizeof (struct gunzip_state )); +if ( !s ) +return -ENOMEM ; + + s->window = (unsigned char *)output; inbuf = (unsigned char *)image; insize = image_len; inptr =0 ;
@@ -112,7 +119,7 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len)makecrc (); -if (gunzip () <0 ) +if (gunzip (s) <0 ) { rc = -EINVAL ; }
xen/common/gzip/inflate.c +34 -34
196 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH v3 3/8] gzip: refactor the gunzip window into common state
Quoted 58 lines · Daniel P. Smith
Begin moving core state, in this case the gunzip window, into struct
gunzip_stateto allow a per decompression instance. In doing so, drop the define aliasing of window to slide.Signed-off-by Daniel P. Smith <[email protected]>--- xen/common/gzip/gunzip.c | 21 ++++++++---- xen/common/gzip/inflate.c | 68 +++++++++++++++++++-------------------- 2 files changed, 48 insertions(+), 41 deletions(-)xen/common/gzip/gunzip.c +7
-5@@ -4,10 +4,12 @@#include <xen/lib.h> #include <xen/mm.h> -static unsigned char *__initdata window; -#define WSIZE 0x80000000U +struct gunzip_state { +unsigned char *window; +}; +static unsigned char *__initdata inbuf;static unsigned int __initdata insize;@@ -43,7 +45,7 @@ typedef unsigned long ulg;#endif static long __initdata bytes_out; -static void flush_window (void ); +static void flush_window (struct gunzip_state *s);static __init void error (const char *x) {@@ -62,7 +64,7 @@ static __init uch get_byte(void) {#include "inflate.c" -static __init void flush_window (void ) +static __init void flush_window (struct gunzip_state *s) {/* * The window is equal to the output buffer therefore only need to @@ -72,7 +74,7 @@ static __init void flush_window(void)unsigned int n;unsigned char *in, ch; - in = window; + in = s->window;for ( n =0 ; n < outcnt; n++ ) { ch = *in++;@@ -99,12 +101,17 @@ __init int gzip_check(char *image, unsigned long image_len)
xen/common/gzip/gunzip.c __init int gzip_check(char *image, unsigned long image_len) · line 110
Quoted 10 lines
xen/common/gzip/gunzip.c +2
-1__init int perform_gunzip (char *output,char *image,unsigned long image_len) { +struct gunzip_state *s;int rc;if ( !gzip_check (image, image_len) )return 1 ; - window = (unsigned char *)output; + s = (struct gunzip_state *)malloc (sizeof (struct gunzip_state ));
Looks like I inadvertently dropped the corresponding free when breaking up the monolithic patch.
v/r,
dps
PATCH v3 4/8 gzip: move window pointer into gunzip state
In reply to Daniel P. SmithMove the window pointer, outcnt/wp, into struct gunzip_data. It was erroneously labeled as outcnt and then define aliased to wp, this eliminates the aliasing and only refers to as wp, the window pointer.
--- xen/common/gzip/gunzip.c | 11 +++++------ xen/common/gzip/inflate.c | 17 ++++++++--------- 2 files changed, 13 insertions(+), 15 deletions(-)
xen/common/gzip/gunzip.c +5 -6
@@ -8,6 +8,8 @@struct gunzip_state {unsigned char *window; +/* window pointer: */ +unsigned int wp; };static unsigned char *__initdata inbuf;
@@ -16,9 +18,6 @@ static unsigned int __initdata insize;/* Index of next byte to be processed in inbuf: */ static unsigned int __initdata inptr; -/* Bytes in output buffer: */ -static unsigned int __initdata outcnt; -#define malloc (a) xmalloc_bytes(a)#define free (a) xfree(a)#define memzero (s, n) memset((s), 0, (n))
@@ -75,15 +74,15 @@ static __init void flush_window(struct gunzip_state *s)unsigned char *in, ch; in = s->window; -for ( n =0 ; n < outcnt; n++ ) +for ( n =0 ; n < s->wp; n++ ) { ch = *in++; c = crc_32_tab[((int )c ^ ch) &0xff ] ^ (c >>8 ); } crc = c; - bytes_out += (unsigned long )outcnt; - outcnt =0 ; + bytes_out += (unsigned long )s->wp; + s->wp =0 ; }__init int gzip_check (char *image,unsigned long image_len)
xen/common/gzip/inflate.c +8 -9
@@ -130,7 +130,6 @@ static char rcsid[] = "#Id: inflate.c,v 0.14 1993/06/10 13:27:04 jloup Exp #";* "uch *window;" and then malloc'ed in the latter case. The definition * must be in unzip.h, included above. */ -#define wp outcnt/* * Huffman code lookup table entry--this entry is four bytes for machines
@@ -556,7 +555,7 @@ static int __init inflate_codes(/* make local copies of globals */ b = bb;/* initialize bit buffer */ k = bk; - w = wp;/* initialize window position */ + w = s->wp;/* initialize window position */ /* inflate the coded data */ ml = mask_bits[bl];/* precompute masks for speed */
@@ -579,7 +578,7 @@ static int __init inflate_codes(Tracevv ((stderr,"%c" , s->window[w-1 ]));if (w ==WSIZE ) { - wp = w; + s->wp = w;flush_window (s); w =0 ; }
@@ -627,7 +626,7 @@ static int __init inflate_codes( }while (--e);if (w ==WSIZE ) { - wp = w; + s->wp = w;flush_window (s); w =0 ; }
@@ -636,7 +635,7 @@ static int __init inflate_codes( }/* restore the globals from the locals */ - wp = w;/* restore global window pointer */ + s->wp = w;/* restore global window pointer */ bb = b;/* restore global bit buffer */ bk = k;
@@ -657,7 +656,7 @@ static int __init inflate_stored(struct gunzip_state *s)/* make local copies of globals */ b = bb;/* initialize bit buffer */ k = bk; - w = wp;/* initialize window position */ + w = s->wp;/* initialize window position */ /* go to byte boundary */
@@ -681,7 +680,7 @@ static int __init inflate_stored(struct gunzip_state *s) s->window[w++] = (uch )b;if (w ==WSIZE ) { - wp = w; + s->wp = w;flush_window (s); w =0 ; }
@@ -689,7 +688,7 @@ static int __init inflate_stored(struct gunzip_state *s) }/* restore the globals from the locals */ - wp = w;/* restore global window pointer */ + s->wp = w;/* restore global window pointer */ bb = b;/* restore global bit buffer */ bk = k;
@@ -1004,7 +1003,7 @@ static int __init inflate(struct gunzip_state *s)int r;/* result code */ /* initialize window, bit buffer */ - wp =0 ; + s->wp =0 ; bk =0 ; bb =0 ;
Signature · 1 line
2.30.2
Re: [PATCH v3 4/8] gzip: move window pointer into gunzip state
Quoted 5 lines · Daniel P. Smith
Move the window pointer, outcnt/wp, into struct
gunzip_data. It was erroneously labeled as outcnt and then define aliased to wp, this eliminates the aliasing and only refers to as wp, the window pointer.Signed-off-by Daniel P. Smith <[email protected]>
PATCH v3 5/8 gzip: move input buffer handling into gunzip state
In reply to Daniel P. SmithMove the input buffer handling, buffer pointer(inbuf), size(insize), and index(inptr), into gunzip state. Adjust functions and macros that consumed the input buffer to accept a struct gunzip_state reference.
--- xen/common/gzip/gunzip.c | 23 +++++----- xen/common/gzip/inflate.c | 92 +++++++++++++++++++-------------------- 2 files changed, 57 insertions(+), 58 deletions(-)
xen/common/gzip/gunzip.c +11 -12
@@ -10,13 +10,12 @@ struct gunzip_state {unsigned char *window;/* window pointer: */ unsigned int wp; -}; - -static unsigned char *__initdata inbuf; -static unsigned int __initdata insize; -/* Index of next byte to be processed in inbuf: */ -static unsigned int __initdata inptr; +unsigned char *inbuf; +size_t insize; +/* Index of next byte to be processed in inbuf: */ +unsigned int inptr; +};#define malloc (a) xmalloc_bytes(a)#define free (a) xfree(a)
@@ -51,14 +50,14 @@ static __init void error(const char *x)panic ("%s\n" , x); } -static __init uch get_byte (void ) { -if ( inptr >= insize ) +static __init uch get_byte (struct gunzip_state *s) { +if ( s->inptr >= s->insize ) {error ("ran out of input data" );return 0 ;/* should never reach */ } -return inbuf[inptr++]; +return s->inbuf[s->inptr++]; }#include "inflate.c"
@@ -111,9 +110,9 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len)return -ENOMEM ; s->window = (unsigned char *)output; - inbuf = (unsigned char *)image; - insize = image_len; - inptr =0 ; + s->inbuf = (unsigned char *)image; + s->insize = image_len; + s->inptr =0 ; bytes_out =0 ;makecrc ();
xen/common/gzip/inflate.c +46 -46
234 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH v3 5/8] gzip: move input buffer handling into gunzip state
Quoted 7 lines · Daniel P. Smith
xen/common/gzip/gunzip.c +0
-1@@ -10,13 +10,12 @@ struct gunzip_state {unsigned char *window;/* window pointer: */ unsigned int wp; -};
xen/common/gzip/gunzip.c struct gunzip_state { · line 17
Quoted 10 lines
xen/common/gzip/gunzip.c +4
-5- -static unsigned char *__initdata inbuf; -static unsigned int __initdata insize; -/* Index of next byte to be processed in inbuf: */ -static unsigned int __initdata inptr; +unsigned char *inbuf; +size_t insize; +/* Index of next byte to be processed in inbuf: */ +unsigned int inptr;
I'm puzzled by the (suddenly) different types, seeing that ...
xen/common/gzip/gunzip.c static __init void error(const char *x) · line 54
Quoted 12 lines
xen/common/gzip/gunzip.c +3
-2+};#define malloc (a) xmalloc_bytes(a)#define free (a) xfree(a)@@ -51,14 +50,14 @@ static __init void error(const char *x)panic ("%s\n" , x); } -static __init uch get_byte (void ) { -if ( inptr >= insize ) +static __init uch get_byte (struct gunzip_state *s) { +if ( s->inptr >= s->insize )
... both are compared with one another.
Quoted 15 lines
xen/common/gzip/gunzip.c +4
-6@@ -1129,29 +1129,29 @@ static int __init gunzip(struct gunzip_state *s)error ("Input has invalid flags" );return -1 ; } -NEXTBYTE ();/* Get timestamp */ -NEXTBYTE (); -NEXTBYTE (); -NEXTBYTE (); +NEXTBYTE (s);/* Get timestamp */ +NEXTBYTE (s); +NEXTBYTE (s); +NEXTBYTE (s); -NEXTBYTE ();/* Ignore extra flags for the moment */ -NEXTBYTE ();/* Ignore OS type for the moment */
xen/common/gzip/gunzip.c static int __init gunzip(struct gunzip_state *s) · line 1143
Quoted 10 lines
xen/common/gzip/gunzip.c +5
-3+NEXTBYTE (s);/* Ignore extra flags for the moment */ +NEXTBYTE (s);/* Ignore OS type for the moment */ if ((flags &EXTRA_FIELD ) !=0 ) { -unsigned len = (unsigned )NEXTBYTE (); - len |= ((unsigned )NEXTBYTE ())<<8 ; -while (len--) (void )NEXTBYTE (); +unsigned len = (unsigned )NEXTBYTE (s); + len |= ((unsigned )NEXTBYTE (s))<<8 ; +while (len--) (void )NEXTBYTE (s);
Would you mind moving the body of this while() to its own line, as you touch this anyway?
Quoted 4 lines
xen/common/gzip/gunzip.c +0
-0}/* Get original file name if it was truncated */ if ((flags &ORIG_NAME ) !=0 ) {
xen/common/gzip/gunzip.c static int __init gunzip(struct gunzip_state *s) · line 1155
Quoted 10 lines
xen/common/gzip/gunzip.c +2
-2/* Discard the old name */ -while (NEXTBYTE () !=0 )/* null */ ; +while (NEXTBYTE (s) !=0 )/* null */ ; }/* Discard file comment if any */ if ((flags &COMMENT ) !=0 ) { -while (NEXTBYTE () !=0 )/* null */ ; +while (NEXTBYTE (s) !=0 )/* null */ ; }
For these two doing the same may help, too.
Jan
PATCH v3 6/8 gzip: move output buffer into gunzip state
In reply to Daniel P. Smith--- xen/common/gzip/gunzip.c | 7 ++++--- xen/common/gzip/inflate.c | 2 +- 2 files changed, 5 insertions(+), 4 deletions(-)
xen/common/gzip/gunzip.c +4 -3
@@ -15,6 +15,8 @@ struct gunzip_state {size_t insize;/* Index of next byte to be processed in inbuf: */ unsigned int inptr; + +unsigned long bytes_out; };#define malloc (a) xmalloc_bytes(a)
@@ -42,7 +44,6 @@ typedef unsigned long ulg;# define Tracecv (c, x)#endif -static long __initdata bytes_out;static void flush_window (struct gunzip_state *s);static __init void error (const char *x)
@@ -80,7 +81,7 @@ static __init void flush_window(struct gunzip_state *s) } crc = c; - bytes_out += (unsigned long )s->wp; + s->bytes_out += (unsigned long )s->wp; s->wp =0 ; }
@@ -113,7 +114,7 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len) s->inbuf = (unsigned char *)image; s->insize = image_len; s->inptr =0 ; - bytes_out =0 ; + s->bytes_out =0 ;makecrc ();
xen/common/gzip/inflate.c +1 -1
@@ -1196,7 +1196,7 @@ static int __init gunzip(struct gunzip_state *s)error ("crc error" );return -1 ; } -if (orig_len != bytes_out) { +if (orig_len != s->bytes_out) {error ("length error" );return -1 ; }
Signature · 1 line
2.30.2
Re: [PATCH v3 6/8] gzip: move output buffer into gunzip state
xen/common/gzip/gunzip.c struct gunzip_state { · line 20
Quoted 9 lines · Daniel P. Smith
xen/common/gzip/gunzip.c +2
-0@@ -15,6 +15,8 @@ struct gunzip_state {size_t insize;/* Index of next byte to be processed in inbuf: */ unsigned int inptr; + +unsigned long bytes_out; };
The conversion to unsigned long from ...
xen/common/gzip/gunzip.c typedef unsigned long ulg; · line 45
Quoted 5 lines
xen/common/gzip/gunzip.c +0
-1@@ -42,7 +44,6 @@ typedef unsigned long ulg;# define Tracecv (c, x)#endif -static long __initdata bytes_out;
... this originally wants justifying in the (then no longer empty) description. It's not a lot that needs saying, but such a type change cannot go entirely silently.
Jan
PATCH v3 7/8 gzip: move bitbuffer into gunzip state
In reply to Daniel P. Smith--- xen/common/gzip/gunzip.c | 3 +++ xen/common/gzip/inflate.c | 43 ++++++++++++++++++--------------------- 2 files changed, 23 insertions(+), 23 deletions(-)
xen/common/gzip/gunzip.c +3 -0
@@ -17,6 +17,9 @@ struct gunzip_state {unsigned int inptr;unsigned long bytes_out; + +unsigned long bb;/* bit buffer */ +unsigned int bk;/* bits in bit buffer */ };#define malloc (a) xmalloc_bytes(a)
xen/common/gzip/inflate.c +20 -23
109 lines not on this page · raw message
Signature · 1 line
2.30.2
Re: [PATCH v3 7/8] gzip: move bitbuffer into gunzip state
Quoted 1 line · Daniel P. Smith
Signed-off-by Daniel P. Smith <[email protected]>
Quoted 5 lines
xen/common/gzip/inflate.c +0
-0
xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1020
Quoted 9 lines
xen/common/gzip/inflate.c +2
-2@@ -1017,8 +1014,8 @@ static int __init inflate(struct gunzip_state *s)/* Undo too much lookahead. The next read will be byte aligned so we * can discard unused bits in the last meaningful byte. */ -while (bk >=8 ) { - bk -=8 ; +while (s->bk >=8 ) { + s->bk -=8 ; s->inptr--; }
Isn't it just me, but isn't this just:
s->inptr -= (s->bk >> 3);
s->bk &= 7;
?
~Andrew
Re: [PATCH v3 7/8] gzip: move bitbuffer into gunzip state
Quoted 16 lines · Andrew Cooper
xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1020
Quoted 12 lines · Daniel P. Smith
xen/common/gzip/inflate.c +2
-2@@ -1017,8 +1014,8 @@ static int __init inflate(struct gunzip_state *s)/* Undo too much lookahead. The next read will be byte aligned so we * can discard unused bits in the last meaningful byte. */ -while (bk >=8 ) { - bk -=8 ; +while (s->bk >=8 ) { + s->bk -=8 ; s->inptr--; }Isn't it just me, but isn't this just:
Quoted 4 lines
s->inptr-= (s->bk>> 3);
s->bk&= 7;?
I'd say yes, if only there wasn't the comment talking of just a single byte, and even there supposedly some of the bits.
Jan
Re: [PATCH v3 7/8] gzip: move bitbuffer into gunzip state
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Andrew Cooper
xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1020
Quoted 12 lines · Daniel P. Smith
xen/common/gzip/inflate.c +2
-2@@ -1017,8 +1014,8 @@ static int __init inflate(struct gunzip_state *s)/* Undo too much lookahead. The next read will be byte aligned so we * can discard unused bits in the last meaningful byte. */ -while (bk >=8 ) { - bk -=8 ; +while (s->bk >=8 ) { + s->bk -=8 ; s->inptr--; }Isn't it just me, but isn't this just:
Quoted 7 lines
Quoted 4 lines
s->inptr-= (s->bk>> 3);
s->bk&= 7;?
I'd say yes, if only there wasn't the comment talking of just a single byte, and even there supposedly some of the bits.
Talking of the comment: Considering what patch 1 supposedly does, how come that isn't Xen-style (yet)?
Jan
Re: [PATCH v3 7/8] gzip: move bitbuffer into gunzip state
Quoted 17 lines · Jan Beulich
Quoted 16 lines · Jan Beulich
Quoted 15 lines · Andrew Cooper
xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1020
Quoted 12 lines · Daniel P. Smith
xen/common/gzip/inflate.c +2
-2@@ -1017,8 +1014,8 @@ static int __init inflate(struct gunzip_state *s)/* Undo too much lookahead. The next read will be byte aligned so we * can discard unused bits in the last meaningful byte. */ -while (bk >=8 ) { - bk -=8 ; +while (s->bk >=8 ) { + s->bk -=8 ; s->inptr--; }Isn't it just me, but isn't this just:
Quoted 6 lines
Quoted 6 lines
Quoted 4 lines
s->inptr-= (s->bk>> 3);
s->bk&= 7;?
I'd say yes, if only there wasn't the comment talking of just a single byte, and even there supposedly some of the bits.
The comments in this file leave a lot to be desired...
I'm reasonably confident they were not adjusted when a piece of userspace code was imported into Linux.
This cannot ever have been just a byte's worth of bits, seeing as the bit count is from 8 or more. Right now it's an unsigned long's worth of bits.
Quoted 2 lines
Talking of the comment: Considering what patch 1 supposedly does, how come that isn't Xen-style (yet)?
I had been collecting some minor fixes like this to fold into patch 1 when I committed the whole series.
I'll see if I can fold them elsewhere.
~Andrew
PATCH v3 8/8 gzip: move crc state into gunzip state
In reply to Daniel P. SmithMove the crc and its state into struct gunzip_state. In the process, expand the only use of CRC_VALUE as it is hides what is being compared.
--- xen/common/gzip/gunzip.c | 11 +++++++---- xen/common/gzip/inflate.c | 14 +++++--------- 2 files changed, 12 insertions(+), 13 deletions(-)
xen/common/gzip/gunzip.c +7 -4
@@ -20,6 +20,9 @@ struct gunzip_state {unsigned long bb;/* bit buffer */ unsigned int bk;/* bits in bit buffer */ + +uint32_t crc_32_tab[256 ]; +uint32_t crc; };#define malloc (a) xmalloc_bytes(a)
@@ -72,7 +75,7 @@ static __init void flush_window(struct gunzip_state *s)* The window is equal to the output buffer therefore only need to * compute the crc. */ -unsigned long c = crc; +unsigned int c = s->crc;unsigned int n;unsigned char *in, ch;
@@ -80,9 +83,9 @@ static __init void flush_window(struct gunzip_state *s)for ( n =0 ; n < s->wp; n++ ) { ch = *in++; - c = crc_32_tab[((int )c ^ ch) &0xff ] ^ (c >>8 ); + c = s->crc_32_tab[((int )c ^ ch) &0xff ] ^ (c >>8 ); } - crc = c; + s->crc = c; s->bytes_out += (unsigned long )s->wp; s->wp =0 ;
@@ -119,7 +122,7 @@ __init int perform_gunzip(char *output, char *image, unsigned long image_len) s->inptr =0 ; s->bytes_out =0 ; -makecrc (); +makecrc (s);if (gunzip (s) <0 ) {
xen/common/gzip/inflate.c +5 -9
@@ -1032,16 +1032,12 @@ static int __init inflate(struct gunzip_state *s)* **********************************************************************/ -static ulg __initdata crc_32_tab[256 ]; -static ulg __initdata crc;/* initialized in makecrc() so it'll reside in bss */ -#define CRC_VALUE (crc ^ 0xffffffffUL) -/* * Code to compute the CRC-32 table. Borrowed from * gzip-1.0.3/makecrc.c. */ -static void __init makecrc (void ) +static void __init makecrc (struct gunzip_state *s) {/* Not copyrighted 1990 Mark Adler */
@@ -1058,7 +1054,7 @@ static void __init makecrc(void)for (i =0 ; i <sizeof (p)/sizeof (int ); i++) e |=1L << (31 - p[i]); - crc_32_tab[0 ] =0 ; + s->crc_32_tab[0 ] =0 ;for (i =1 ; i <256 ; i++) {
@@ -1069,11 +1065,11 @@ static void __init makecrc(void)if (k &1 ) c ^= e; } - crc_32_tab[i] = c; + s->crc_32_tab[i] = c; }/* this is initialized here so this code could reside in ROM */ - crc = (ulg )0xffffffffUL ;/* shift register contents */ + s->crc = (ulg )~0u ;/* shift register contents */ }/* gzip flag byte */
@@ -1189,7 +1185,7 @@ static int __init gunzip(struct gunzip_state *s) orig_len |= (ulg )NEXTBYTE (s) <<24 ;/* Validate decompression */ -if (orig_crc !=CRC_VALUE ) { +if (orig_crc != (s->crc ^ ~0u )) {error ("crc error" );return -1 ; }
Signature · 1 line
2.30.2
Re: [PATCH v3 8/8] gzip: move crc state into gunzip state
Quoted 2 lines · Daniel P. Smith
Move the crc and its state into struct
gunzip_state. In the process, expand the only use ofCRC_VALUEas it is hides what is being compared.
"All variables here should be uint32_t rather than unsigned long, which halves the storage space required."
Quoted 5 lines
xen/common/gzip/inflate.c +0
-0
xen/common/gzip/inflate.c static void __init makecrc(void) · line 1072
Quoted 10 lines
xen/common/gzip/inflate.c +2
-2@@ -1069,11 +1065,11 @@ static void __init makecrc(void)if (k &1 ) c ^= e; } - crc_32_tab[i] = c; + s->crc_32_tab[i] = c; }/* this is initialized here so this code could reside in ROM */ - crc = (ulg )0xffffffffUL ;/* shift register contents */ + s->crc = (ulg )~0u ;/* shift register contents */
The (ulg) wasn't ever necessary, and can be dropped as part of this cleanup.
Can fix on commit.
~Andrew
Re: [PATCH v3 8/8] gzip: move crc state into gunzip state
In reply to Daniel P. SmithQuoted 2 lines · Daniel P. Smith
Move the crc and its state into struct
gunzip_state. In the process, expand the only use ofCRC_VALUEas it is hides what is being compared.
Andrew mentioned uint32_t only for the description, but I think you want (maybe even need) to go further and actually use that type also, e.g.
xen/common/gzip/gunzip.c struct gunzip_state { · line 26
Quoted 10 lines
xen/common/gzip/gunzip.c +3
-0@@ -20,6 +20,9 @@ struct gunzip_state {unsigned long bb;/* bit buffer */ unsigned int bk;/* bits in bit buffer */ + +uint32_t crc_32_tab[256 ]; +uint32_t crc; };
... not just here, but also ...
xen/common/gzip/gunzip.c static __init void flush_window(struct gunzip_state *s) · line 78
Quoted 6 lines
xen/common/gzip/gunzip.c +1
-1@@ -72,7 +75,7 @@ static __init void flush_window(struct gunzip_state *s)* The window is equal to the output buffer therefore only need to * compute the crc. */ -unsigned long c = crc; +unsigned int c = s->crc;
... here.
xen/common/gzip/inflate.c static int __init inflate(struct gunzip_state *s) · line 1037
Quoted 9 lines
xen/common/gzip/inflate.c +0
-3@@ -1032,16 +1032,12 @@ static int __init inflate(struct gunzip_state *s)* **********************************************************************/ -static ulg __initdata crc_32_tab[256 ]; -static ulg __initdata crc;/* initialized in makecrc() so it'll reside in bss */ -#define CRC_VALUE (crc ^ 0xffffffffUL)
Note how this is _not_ same as ~0u, also ...
xen/common/gzip/inflate.c static void __init makecrc(void) · line 1072
Quoted 11 lines
xen/common/gzip/inflate.c +2
-2@@ -1069,11 +1065,11 @@ static void __init makecrc(void)if (k &1 ) c ^= e; } - crc_32_tab[i] = c; + s->crc_32_tab[i] = c; }/* this is initialized here so this code could reside in ROM */ - crc = (ulg )0xffffffffUL ;/* shift register contents */ + s->crc = (ulg )~0u ;/* shift register contents */
... applicable here then: sizeof(int) >= 4, not ==.
As Andrew indicates, the cast previously wasn't needed here. Keeping it is at best misleading, when s->crc isn't of that type anymore.
Finally please stick to upper-case number suffixes; see all the related Misra changes, which (iirc) put in place only upper-case ones.
Jan
Quoted 7 lines · Daniel P. Smith
An issue ran into by hyperlaunch was the need to use the gzip decompressor multiple times. The current implementation fails when reused due to tainting of decompressor state from a previous usage. This series seeks to colocate the gzip unit files under a single directory similar to the other decompression algorithms. To enable the refactoring of the state tracking, the code is then cleaned up in line with Xen coding style.
Forgot to add this comment to the cover letter.
Concern was raised about taking updates from original source to the code. In this case the original source is from the Linux kernel, which can be found in lib/inflate.c, and added to Xen in 2009. The last time there was a logic change to that code in the Linux kernel was in 2008, before it was added to Xen. Since then, it has only been updated for changes made to included headers. If fact, as far as I can see, while the file is still in place, nothing uses it. For zlib decompression, three is a new lib/zlib_deflate code base that is used. The scope of this work is not to completely fix/replace zlib decompression for Xen, but to stabilize it to allow hyperlaunch to decompress more than one zlib compressed kernel.
V/r,
Daniel P. Smith
Quoted 26 lines · Daniel P. Smith
An issue ran into by hyperlaunch was the need to use the gzip decompressor multiple times. The current implementation fails when reused due to tainting of decompressor state from a previous usage. This series seeks to colocate the gzip unit files under a single directory similar to the other decompression algorithms. To enable the refactoring of the state tracking, the code is then cleaned up in line with Xen coding style.
Changes in v3:
- patch "xen/gzip: Drop huffman code table tracking" was merged
- patch "xen/gzip: Remove custom memory allocator" was merged
- patch "xen/gzip: Drop unused define checks" was merged
- move of the decompressor state into struct was broken up to ease review
- expanded macros that were either only used once or obsfucated the logic
- adjusted variable types as needed to be more appropriate for their usageChanges in v2:
- patch "xen/gzip: Colocate gunzip code files" was merged
- dropped ifdef chunks that are never enabled
- addressed formatting changes missed in v1
- replaced custom memory allocator withxalloc_bytes()
- renamedgzip_datatogzip_state
- movedgzip_stateto being dynamically allocated
- changed crc table to the explicit size ofuint32_t
- instead of moving huffman tracking into state, dropped altogether
Quoted 9 lines
Daniel P. Smith (8):
gzip: clean up comments and fix code alignment
gzip: refactor and expand macros
gzip: refactor the gunzip window into common state
gzip: move window pointer into gunzip state
gzip: move input buffer handling into gunzip state
gzip: move output buffer into gunzip state
gzip: move bitbuffer into gunzip state
gzip: move crc state into gunzip state
In hindsight my suggestion that lead to "refactor and expand macros" wasn't great.
We want to keep the underrun labels for a future when the error handling isn't panic(). After that, expanding flush_window() is better folded into "refactor the gunzip window into common state" to reduce the churn.
As that was my fault, and unpicking it is reasonably hard, and we're on a tight deadline for 4.19 now, I've gone ahead and unpicked this.
Also I've rebased over Jan's patch addressing the memory leaks reported by Coverity, as the two sets of fixes collide, along with various other minor notes.
I've conferred with Daniel who is happy with the aformentioned changes.
This area of code still has a lot of work needing doing on it, but removing the use of global state is a concrete improvement in the status quo.
~Andrew