mtd-utils: adding support for lzma in jffs2reader via liblzma
Subsystems (2)
cover mtd-utils: adding support for lzma in jffs2reader via liblzma
Hi,
This is my first time sending patch, I probably do some things wrong somewhere.
A while ago I, unfortunately, removed a file from my gl-inet 4g openwrt router. That file is on a jffs2 partition, with LZMA compression.
LZMA support on OpenWRT is activated via a patch which seems never made it to the mainline : jffs2_make_lzma_available.patch. There might be a problem with the jffs2 integration of LZMA, if it was not integrated into the kernel? The filesystem support should probably be first integrated in the main kernel before these patch in the user tools...
I think openwrt/LEDE is a big use case for jffs2, with maybe a lot of them with LZMA supports (I'm cannot say how much though...) So it would be cool to have this integrated so people can recover the files they lost.
There are two opensource LZMA libraries that I found on Ubuntu 20.04...
- The source SDK from https://www.7-zip.org/sdk.html . On ubuntu, it's in the p7zip source package, which provides p7zip & p7zip-full package. p7zip-full provides a library, /usr/lib/p7zip/7z.so, but it's not integrated the Linux/GNU way...
- liblzma: from https://tukaani.org/xz/, which integrates an older version of the LZMA SDK provided by www.7-zip.org, but on a standard Linux/GNU library.
It seems LEDE already has patches for mtd-utils, but it seems they integrate source files from www.7-zip.org in a static way.
https://patchwork.ozlabs.org/project/lede/patch/[email protected]/
I have to fiddle around to integrate with liblzma, as the low-level decoder is not exposed, but I got help from the XZ project author https://sourceforge.net/p/lzmautils/discussion/708858/thread/d02ebb9386/
I pushed the patches on https://github.com/fredericgermain/mtd-utils too, main branch. It might just help some people to recover their file anyway.
---
There is also a problem that jffs2reader selects the earliest version of an inode instead of the latest? I've made a patch to take the latest one with data. But I still needed to patch the mtd dump to remove the DIRENT entry that remove the file from the FS.
Maybe it would be nice that jffs2reader takes inode/version of the file to dump instead of a path? It's easy to get these parameters with jffs2dump.
Best regards,
Frederic Germain (4):
.gitignore: add new ubiscan utility
Fix warning about unaligned pointer in jffs2reader
jffsX-utils: add lzma support on jffs2reader find_raw_inode returns latest version with data of inode
.gitignore | 1 + Makefile.am | 4 +++ configure.ac | 24 +++++++++++++++ include/linux/jffs2.h | 2 ++ include/mtd/jffs2-user.h | 1 + jffsX-utils/Makemodule.am | 2 +- jffsX-utils/jffs2reader.c | 63 ++++++++++++++++++++++++++++++++++----- 7 files changed, 88 insertions(+), 9 deletions(-)
Signature · 6 lines
2.25.1 ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/
PATCH 1/4 .gitignore: add new ubiscan utility
--- .gitignore | 1 + 1 file changed, 1 insertion(+)
.gitignore +1 -0
@@ -100,6 +100,7 @@ ubinize ubirename ubirmvol ubirsvol +ubiscan ubiupdatevol volrefcnt mtdlib_test
Signature · 6 lines
2.25.1 ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/
PATCH 2/4 Fix warning about unaligned pointer in jffs2reader
In reply to Frederic Germaingcc 9.3.0-17ubuntu1~20.04 warning on time_t ctime value is :
Fix taking address of packed member of ‘struct jffs2_raw_inode’ may result in an unaligned pointer value [-Waddress-of-packed-member].
--- jffsX-utils/jffs2reader.c | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-)
jffsX-utils/jffs2reader.c +3 -2
@@ -336,8 +336,9 @@ static void printdir(char *o, size_t size, struct dir *d, const char *path, d = d->next;continue ; } - - filetime =ctime ((const time_t *) &(ri->ctime)); +time_t _ctime; +memcpy (&_ctime, &(ri->ctime),sizeof (time_t)); + filetime =ctime (&_ctime); age =time (NULL ) -je32_to_cpu (ri->ctime); mode.v32 = ri->mode.m; printf("%s %-4d %-8d %-8d ", mode_string(je32_to_cpu(mode)),
Signature · 6 lines
2.25.1 ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/
PATCH 3/4 jffsX-utils: add lzma support on jffs2reader
In reply to Frederic Germain--- Makefile.am | 4 ++++ configure.ac | 24 +++++++++++++++++++++ include/linux/jffs2.h | 2 ++ include/mtd/jffs2-user.h | 1 + jffsX-utils/Makemodule.am | 2 +- jffsX-utils/jffs2reader.c | 45 +++++++++++++++++++++++++++++++++++++++ 6 files changed, 77 insertions(+), 1 deletion(-)
Makefile.am +4 -0
@@ -11,6 +11,10 @@ if WITHOUT_LZO AM_CPPFLAGS += -DWITHOUT_LZO endif +if WITHOUT_LZMA +AM_CPPFLAGS += -DWITHOUT_LZMA +endif + if WITHOUT_ZSTD AM_CPPFLAGS += -DWITHOUT_ZSTD endif
configure.ac +24 -0
@@ -65,6 +65,7 @@ need_clock_gettime="no" need_pthread="no" need_uuid="no" need_zlib="no" +need_lzma="" need_lzo="no" need_zstd="no" need_xattr="no"
@@ -158,6 +159,14 @@ AC_ARG_WITH([lzo], *) AC_MSG_ERROR([bad value ${withval} for --without-lzo]) ;; esac]) +AC_ARG_WITH([lzma], + [AS_HELP_STRING([--without-lzma], [Disable support for LZMA compression])], + [case "${withval}" in + yes) need_lzma="yes";; + no) need_lzma="no" ;; + *) AC_MSG_ERROR([bad value ${withval} for --without-lzma]) ;; + esac]) + AC_ARG_WITH([zstd], [AS_HELP_STRING([--without-zstd], [Disable support for ZSTD compression])], [case "${withval}" in
@@ -190,6 +199,7 @@ clock_gettime_missing="no" pthread_missing="no" uuid_missing="no" zlib_missing="no" +lzma_missing="no" lzo_missing="no" zstd_missing="no" xattr_missing="no"
@@ -219,6 +229,14 @@ if test "x$need_pthread" = "xyes"; then AX_PTHREAD([], [pthread_missing="yes"]) fi +if test "x$need_lzma" != "xno"; then + AC_ARG_VAR([LZMA_CFLAGS], [C compiler flags for lzma]) + AC_ARG_VAR([LZMA_LIBS], [linker flags for lzma]) + AC_CHECK_LIB([lzma], [lzma_alone_decoder], [LZMA_LIBS="-llzma"], + [lzma_missing="yes"] + ) +fi + if test "x$need_lzo" = "xyes"; then AC_ARG_VAR([LZO_CFLAGS], [C compiler flags for lzo]) AC_ARG_VAR([LZO_LIBS], [linker flags for lzo])
@@ -287,6 +305,11 @@ if test "x$zlib_missing" = "xyes"; then dep_missing="yes" fi +if test "x$need_lzma" = "xyes" -a "x$lzma_missing" = "xyes"; then + AC_MSG_WARN([cannot find LZMA library required by options]) + dep_missing="yes" +fi + if test "x$lzo_missing" = "xyes"; then AC_MSG_WARN([cannot find LZO library required for mkfs programs]) AC_MSG_NOTICE([mtd-utils can optionally be built without mkfs.ubifs])
@@ -339,6 +362,7 @@ fi ##### generate output ##### AM_CONDITIONAL([WITHOUT_LZO], [test "x$need_lzo" != "xyes"]) +AM_CONDITIONAL([WITHOUT_LZMA], [test "x$need_lzma" == "xno" -o "x$missing_lzma" == "xyes" ]) AM_CONDITIONAL([WITHOUT_ZSTD], [test "x$need_zstd" != "xyes"]) AM_CONDITIONAL([WITHOUT_XATTR], [test "x$need_xattr" != "xyes"]) AM_CONDITIONAL([WITH_SELINUX], [test "x$need_selinux" == "xyes"])
include/linux/jffs2.h +2 -0
@@ -47,6 +47,8 @@#define JFFS2_COMPR_DYNRUBIN 0x05#define JFFS2_COMPR_ZLIB 0x06#define JFFS2_COMPR_LZO 0x07 +// diff --git a/target/linux/generic/patches-4.1/530-jffs2_make_lzma_available.patch b/target/linux/generic/patches-4.1/530-jffs2_make_lzma_available.patch +#define JFFS2_COMPR_LZMA 0x08 // https://git.openwrt.org/project/ubus.git;git://git.openwrt.org/project/ubox.git?p=openwrt/openwrt.git;a=commitdiff;h=a0c49ef46f7caf5eb02c635d446218201008ecff;hp=518ab154e059c9f39684f8c78159c9caf333716b/* Compatibility flags. */ #define JFFS2_COMPAT_MASK 0xc000/* What do to if an unknown nodetype is found */ #define JFFS2_NODE_ACCURATE 0x2000
include/mtd/jffs2-user.h +1 -0
@@ -36,6 +36,7 @@ extern int target_endian;#define le32_to_cpu (x) (__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_32(x))#define cpu_to_le16 (x) (__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_16(x))#define cpu_to_le32 (x) (__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_32(x)) +#define cpu_to_le64 (x) (__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_64(x))/* XATTR/POSIX-ACL related definition */ /* Namespaces copied from xattr.h and posix_acl_xattr.h */
jffsX-utils/Makemodule.am +1 -1
@@ -16,7 +16,7 @@ mkfs_jffs2_LDADD = libmtd.a $(ZLIB_LIBS) $(LZO_LIBS) mkfs_jffs2_CPPFLAGS = $(AM_CPPFLAGS) $(ZLIB_CFLAGS) $(LZO_CFLAGS) jffs2reader_SOURCES = jffsX-utils/jffs2reader.c include/mtd/jffs2-user.h -jffs2reader_LDADD = libmtd.a $(ZLIB_LIBS) $(LZO_LIBS) +jffs2reader_LDADD = libmtd.a $(ZLIB_LIBS) $(LZO_LIBS) $(LZMA_LIBS) jffs2reader_CPPFLAGS = $(AM_CPPFLAGS) $(ZLIB_CFLAGS) $(LZO_CFLAGS) jffs2dump_SOURCES = jffsX-utils/jffs2dump.c include/mtd/jffs2-user.h
jffsX-utils/jffs2reader.c +45 -0
57 lines not on this page · raw message
Signature · 6 lines
2.25.1 ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/
PATCH 4/4 find_raw_inode returns latest version with data of inode
In reply to Frederic Germain--- jffsX-utils/jffs2reader.c | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-)
jffsX-utils/jffs2reader.c +7 -6
@@ -484,18 +484,19 @@ static struct jffs2_raw_inode *find_raw_inode(char *o, size_t size,if (n < e &&je16_to_cpu (n->u.magic) ==JFFS2_MAGIC_BITMASK ) {if (je16_to_cpu (n->u.nodetype) ==JFFS2_NODETYPE_INODE && -je32_to_cpu (n->i.ino) == ino && (v =je32_to_cpu (n->i.version)) > vcur) { +je32_to_cpu (n->i.ino) == ino && (v =je32_to_cpu (n->i.version)) > vcur &&je32_to_cpu (n->i.csize)) {/* XXX crc check */ -if (vmaxt < v) +if (vmaxt < v) { vmaxt = v; -if (vmint > v) { - vmint = v; mp = n; } +if (vmint > v) { + vmint = v; + } -if (v == (vcur +1 )) -return (&(n->i)); +// if (v == (vcur + 1)) +// return (&(n->i)); }ADD_BYTES (n, ((je32_to_cpu (n->u.totlen) +3 ) & ~3 ));
Signature · 6 lines
2.25.1 ______________________________________________________ Linux MTD discussion mailing list http://lists.infradead.org/mailman/listinfo/linux-mtd/