Skip to contentSkip to conversation

mtd-utils: adding support for lzma in jffs2reader via liblzma

PATCH New Frederic Germain 1 participant 5 messages Nov 9, 2021

Subsystems (2)
lore
Share & tools
lore
#0 lore raw

cover mtd-utils: adding support for lzma in jffs2reader via liblzma

Hi,

This is my first time sending patch, I probably do some things wrong somewhere.

A while ago I, unfortunately, removed a file from my gl-inet 4g openwrt router. That file is on a jffs2 partition, with LZMA compression.

LZMA support on OpenWRT is activated via a patch which seems never made it to the mainline : jffs2_make_lzma_available.patch. There might be a problem with the jffs2 integration of LZMA, if it was not integrated into the kernel? The filesystem support should probably be first integrated in the main kernel before these patch in the user tools...

I think openwrt/LEDE is a big use case for jffs2, with maybe a lot of them with LZMA supports (I'm cannot say how much though...) So it would be cool to have this integrated so people can recover the files they lost.

There are two opensource LZMA libraries that I found on Ubuntu 20.04...
- The source SDK from https://www.7-zip.org/sdk.html . On ubuntu, it's in the p7zip source package, which provides p7zip & p7zip-full package. p7zip-full provides a library, /usr/lib/p7zip/7z.so, but it's not integrated the Linux/GNU way...
- liblzma: from https://tukaani.org/xz/, which integrates an older version of the LZMA SDK provided by www.7-zip.org, but on a standard Linux/GNU library.

It seems LEDE already has patches for mtd-utils, but it seems they integrate source files from www.7-zip.org in a static way.
https://patchwork.ozlabs.org/project/lede/patch/[email protected]/

I have to fiddle around to integrate with liblzma, as the low-level decoder is not exposed, but I got help from the XZ project author https://sourceforge.net/p/lzmautils/discussion/708858/thread/d02ebb9386/

I pushed the patches on https://github.com/fredericgermain/mtd-utils too, main branch. It might just help some people to recover their file anyway.

---
There is also a problem that jffs2reader selects the earliest version of an inode instead of the latest? I've made a patch to take the latest one with data. But I still needed to patch the mtd dump to remove the DIRENT entry that remove the file from the FS.

Maybe it would be nice that jffs2reader takes inode/version of the file to dump instead of a path? It's easy to get these parameters with jffs2dump.

Best regards,

Frederic Germain (4):
.gitignore: add new ubiscan utility
Fix warning about unaligned pointer in jffs2reader
jffsX-utils: add lzma support on jffs2reader find_raw_inode returns latest version with data of inode

 .gitignore                |  1 +
 Makefile.am               |  4 +++
 configure.ac              | 24 +++++++++++++++
 include/linux/jffs2.h     |  2 ++
 include/mtd/jffs2-user.h  |  1 +
 jffsX-utils/Makemodule.am |  2 +-
 jffsX-utils/jffs2reader.c | 63 ++++++++++++++++++++++++++++++++++-----
 7 files changed, 88 insertions(+), 9 deletions(-)
Signature · 6 lines
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
#1 lore raw

PATCH 1/4 .gitignore: add new ubiscan utility

---
 .gitignore | 1 +
 1 file changed, 1 insertion(+)
.gitignore +1 -0
index 1644ef0..8c51ee5 100644
@@ -100,6 +100,7 @@ ubinize
 ubirename
 ubirmvol
 ubirsvol
+ubiscan
 ubiupdatevol
 volrefcnt
 mtdlib_test
Signature · 6 lines
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
#2 lore raw

PATCH 2/4 Fix warning about unaligned pointer in jffs2reader

In reply to Frederic Germain

gcc 9.3.0-17ubuntu1~20.04 warning on time_t ctime value is :

Fix taking address of packed member of ‘struct jffs2_raw_inode’ may result in an unaligned pointer value [-Waddress-of-packed-member].

---
 jffsX-utils/jffs2reader.c | 5 +++--
 1 file changed, 3 insertions(+), 2 deletions(-)
jffsX-utils/jffs2reader.c +3 -2
index 083500e..33c5577 100644
@@ -336,8 +336,9 @@ static void printdir(char *o, size_t size, struct dir *d, const char *path,
 			d = d->next;
 			continue;
 		}
-
-		filetime = ctime((const time_t *) &(ri->ctime));
+		time_t _ctime;
+		memcpy(&_ctime, &(ri->ctime), sizeof(time_t));
+		filetime = ctime(&_ctime);
 		age = time(NULL) - je32_to_cpu(ri->ctime);
 		mode.v32 = ri->mode.m;
 		printf("%s %-4d %-8d %-8d ", mode_string(je32_to_cpu(mode)),
Signature · 6 lines
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
#3 lore raw

PATCH 3/4 jffsX-utils: add lzma support on jffs2reader

In reply to Frederic Germain
---
 Makefile.am               |  4 ++++
 configure.ac              | 24 +++++++++++++++++++++
 include/linux/jffs2.h     |  2 ++
 include/mtd/jffs2-user.h  |  1 +
 jffsX-utils/Makemodule.am |  2 +-
 jffsX-utils/jffs2reader.c | 45 +++++++++++++++++++++++++++++++++++++++
 6 files changed, 77 insertions(+), 1 deletion(-)
Makefile.am +4 -0
index 9da38af..02a3073 100644
@@ -11,6 +11,10 @@ if WITHOUT_LZO
 AM_CPPFLAGS += -DWITHOUT_LZO
 endif
 
+if WITHOUT_LZMA
+AM_CPPFLAGS += -DWITHOUT_LZMA
+endif
+
 if WITHOUT_ZSTD
 AM_CPPFLAGS += -DWITHOUT_ZSTD
 endif
configure.ac +24 -0
index f47100f..e5035c0 100644
@@ -65,6 +65,7 @@ need_clock_gettime="no"
 need_pthread="no"
 need_uuid="no"
 need_zlib="no"
+need_lzma=""
 need_lzo="no"
 need_zstd="no"
 need_xattr="no"
@@ -158,6 +159,14 @@ AC_ARG_WITH([lzo],
 	*) AC_MSG_ERROR([bad value ${withval} for --without-lzo]) ;;
 	esac])
 
+AC_ARG_WITH([lzma],
+	[AS_HELP_STRING([--without-lzma], [Disable support for LZMA compression])],
+	[case "${withval}" in
+	yes) need_lzma="yes";;
+	no) need_lzma="no" ;;
+	*) AC_MSG_ERROR([bad value ${withval} for --without-lzma]) ;;
+	esac])
+
 AC_ARG_WITH([zstd],
 	[AS_HELP_STRING([--without-zstd], [Disable support for ZSTD compression])],
 	[case "${withval}" in
@@ -190,6 +199,7 @@ clock_gettime_missing="no"
 pthread_missing="no"
 uuid_missing="no"
 zlib_missing="no"
+lzma_missing="no"
 lzo_missing="no"
 zstd_missing="no"
 xattr_missing="no"
@@ -219,6 +229,14 @@ if test "x$need_pthread" = "xyes"; then
 	AX_PTHREAD([], [pthread_missing="yes"])
 fi
 
+if test "x$need_lzma" != "xno"; then
+	AC_ARG_VAR([LZMA_CFLAGS], [C compiler flags for lzma])
+	AC_ARG_VAR([LZMA_LIBS], [linker flags for lzma])
+	AC_CHECK_LIB([lzma], [lzma_alone_decoder], [LZMA_LIBS="-llzma"],
+		[lzma_missing="yes"]
+	)
+fi
+
 if test "x$need_lzo" = "xyes"; then
 	AC_ARG_VAR([LZO_CFLAGS], [C compiler flags for lzo])
 	AC_ARG_VAR([LZO_LIBS], [linker flags for lzo])
@@ -287,6 +305,11 @@ if test "x$zlib_missing" = "xyes"; then
 	dep_missing="yes"
 fi
 
+if test "x$need_lzma" = "xyes" -a "x$lzma_missing" = "xyes"; then
+	AC_MSG_WARN([cannot find LZMA library required by options])
+	dep_missing="yes"
+fi
+
 if test "x$lzo_missing" = "xyes"; then
 	AC_MSG_WARN([cannot find LZO library required for mkfs programs])
 	AC_MSG_NOTICE([mtd-utils can optionally be built without mkfs.ubifs])
@@ -339,6 +362,7 @@ fi
 ##### generate output #####
 
 AM_CONDITIONAL([WITHOUT_LZO], [test "x$need_lzo" != "xyes"])
+AM_CONDITIONAL([WITHOUT_LZMA], [test "x$need_lzma" == "xno" -o "x$missing_lzma" == "xyes" ])
 AM_CONDITIONAL([WITHOUT_ZSTD], [test "x$need_zstd" != "xyes"])
 AM_CONDITIONAL([WITHOUT_XATTR], [test "x$need_xattr" != "xyes"])
 AM_CONDITIONAL([WITH_SELINUX], [test "x$need_selinux" == "xyes"])
include/linux/jffs2.h +2 -0
index 7306f86..47abf62 100644
@@ -47,6 +47,8 @@
 #define JFFS2_COMPR_DYNRUBIN	0x05
 #define JFFS2_COMPR_ZLIB	0x06
 #define JFFS2_COMPR_LZO		0x07
+// diff --git a/target/linux/generic/patches-4.1/530-jffs2_make_lzma_available.patch b/target/linux/generic/patches-4.1/530-jffs2_make_lzma_available.patch
+#define JFFS2_COMPR_LZMA	0x08 // https://git.openwrt.org/project/ubus.git;git://git.openwrt.org/project/ubox.git?p=openwrt/openwrt.git;a=commitdiff;h=a0c49ef46f7caf5eb02c635d446218201008ecff;hp=518ab154e059c9f39684f8c78159c9caf333716b
 /* Compatibility flags. */
 #define JFFS2_COMPAT_MASK 0xc000      /* What do to if an unknown nodetype is found */
 #define JFFS2_NODE_ACCURATE 0x2000
include/mtd/jffs2-user.h +1 -0
index bc5d99a..7228b5f 100644
@@ -36,6 +36,7 @@ extern int target_endian;
 #define le32_to_cpu(x)	(__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_32(x))
 #define cpu_to_le16(x)	(__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_16(x))
 #define cpu_to_le32(x)	(__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_32(x))
+#define cpu_to_le64(x)	(__BYTE_ORDER==__LITTLE_ENDIAN ? (x) : bswap_64(x))
 
 /* XATTR/POSIX-ACL related definition */
 /* Namespaces copied from xattr.h and posix_acl_xattr.h */
jffsX-utils/Makemodule.am +1 -1
index 96389f6..0c93c18 100644
@@ -16,7 +16,7 @@ mkfs_jffs2_LDADD = libmtd.a $(ZLIB_LIBS) $(LZO_LIBS)
 mkfs_jffs2_CPPFLAGS = $(AM_CPPFLAGS) $(ZLIB_CFLAGS) $(LZO_CFLAGS)
 
 jffs2reader_SOURCES = jffsX-utils/jffs2reader.c	include/mtd/jffs2-user.h
-jffs2reader_LDADD = libmtd.a $(ZLIB_LIBS) $(LZO_LIBS)
+jffs2reader_LDADD = libmtd.a $(ZLIB_LIBS) $(LZO_LIBS) $(LZMA_LIBS)
 jffs2reader_CPPFLAGS = $(AM_CPPFLAGS) $(ZLIB_CFLAGS) $(LZO_CFLAGS)
 
 jffs2dump_SOURCES = jffsX-utils/jffs2dump.c include/mtd/jffs2-user.h
jffsX-utils/jffs2reader.c +45 -0

57 lines not on this page · raw message

Signature · 6 lines
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
#4 lore raw

PATCH 4/4 find_raw_inode returns latest version with data of inode

In reply to Frederic Germain
---
 jffsX-utils/jffs2reader.c | 13 +++++++------
 1 file changed, 7 insertions(+), 6 deletions(-)
jffsX-utils/jffs2reader.c +7 -6
index 2aa2137..c0c61cf 100644
@@ -484,18 +484,19 @@ static struct jffs2_raw_inode *find_raw_inode(char *o, size_t size,
 
 		if (n < e && je16_to_cpu(n->u.magic) == JFFS2_MAGIC_BITMASK) {
 			if (je16_to_cpu(n->u.nodetype) == JFFS2_NODETYPE_INODE &&
-				je32_to_cpu(n->i.ino) == ino && (v = je32_to_cpu(n->i.version)) > vcur) {
+				je32_to_cpu(n->i.ino) == ino && (v = je32_to_cpu(n->i.version)) > vcur && je32_to_cpu(n->i.csize)) {
 				/* XXX crc check */
 
-				if (vmaxt < v)
+				if (vmaxt < v) {
 					vmaxt = v;
-				if (vmint > v) {
-					vmint = v;
 					mp = n;
 				}
+					if (vmint > v) {
+					vmint = v;
+				}
 
-				if (v == (vcur + 1))
-					return (&(n->i));
+//				if (v == (vcur + 1))
+//					return (&(n->i));
 			}
 
 			ADD_BYTES(n, ((je32_to_cpu(n->u.totlen) + 3) & ~3));
Signature · 6 lines
2.25.1


______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/