JFFS2 Garbage collection issue
Hi all,
we are currently facing a problem with the Garbage-collection on a JFFS2 file system.
We have an embedded linux system based on an ARM9 AT91RM9200 processor with an Intel / Micron JS28F256 NOR-flash of 32MB in size. The Flash holds the U-Boot loader and its environment and in the main part (30MB) of the chip a JFFS2-based root-file-system. Our Linux kernel version is 2.6.24.3
Some minutes after startup of the system, the garbage collector (gc.c) does a scan of the file system, gathering information for the garbage collection procedures that may be initiated based upon that data.
On some file systems, especially such with long runtime and therefore many file operations, this scan consumes a lot of time. On some systems it stays in that scan for more that 10 minutes. The problem with this behaviour is the fact, that during the scan the file system is locked, and therefore all file operations are suspended. This leads to a crash of the system, when the application program hangs for a long time, waiting for a file operation to return from a system call, and the watchdog is not triggered.
In a test system without watchdog, after the scan, the system runs normally. The flash chip itself shows no errors, so it seems to be a problem of the file system.
Any idea or suggestion to solve the problem would be very welcome!
Harald Albrecht
"linux-mtd" <[email protected]> wrote on 2013/11/27 11:11:04:
Quoted 25 lines
Hi all,
we are currently facing a problem with the Garbage-collection on a JFFS2
file system.
We have an embedded linux system based on an ARM9 AT91RM9200 processor with an Intel / Micron JS28F256 NOR-flash of 32MB in size. The Flash holds the U-Boot loader and its environment and in the main part (30MB) of the chip a JFFS2-based root-file-system.
Our Linux kernel version is 2.6.24.3Some minutes after startup of the system, the garbage collector (
gc.c) does a scan of the file system, gathering information for the garbage collection procedures that may be initiated based upon that data. On some file systems, especially such with long runtime and therefore many file operations, this scan consumes a lot of time. On some systems it stays in that scan for more that 10 minutes. The problem with this behaviour is the fact, that during the scan the file system is locked, and therefore all file operations are suspended. This leads to a crash of the system, when the application program hangs for a long time, waiting for a file operation to return from a system call, and the watchdog is not triggered.
Quoted 1 line
In a test system without watchdog, after the scan, the system runs
normally.
Quoted 4 lines
The flash chip itself shows no errors, so it seems to be a problem of the file system.
Any idea or suggestion to solve the problem would be very welcome!
The simple fix is to send a signal(SIGCONT?) to the GC process regularly to do GC
over time. That way you don't get all GC at boot time.
Jocke
"linux-mtd" <[email protected]> wrote on 2013/11/27 11:23:25:
Quoted 7 lines
"linux-mtd" <[email protected]> wrote on 2013/11/27 11:11:04:
Quoted 4 lines
Hi all,
we are currently facing a problem with the Garbage-collection on a
JFFS2
Quoted 10 lines
Quoted 10 lines
file system.
We have an embedded linux system based on an ARM9 AT91RM9200 processor
with an Intel / Micron JS28F256 NOR-flash of 32MB in size. The Flash holds the U-Boot loader and its environment and in the main part (30MB) of the chip a JFFS2-based root-file-system. Our Linux kernel version is 2.6.24.3
Quoted 6 lines
Quoted 6 lines
Some minutes after startup of the system, the garbage collector (
gc.c)does a scan of the file system, gathering information for the garbage collection procedures that may be initiated based upon that data. On some file systems, especially such with long runtime and therefore many file operations, this scan consumes a lot of time. On some
systems
Quoted 14 lines
Quoted 10 lines
it stays in that scan for more that 10 minutes. The problem with this behaviour is the fact, that during the scan the file system is locked,
and therefore all file operations are suspended. This leads to a crash
of the system, when the application program hangs for a long time, waiting for a file operation to return from a system call, and the watchdog is not triggered.
In a test system without watchdog, after the scan, the system runs
normally.
Quoted 3 lines
The flash chip itself shows no errors, so it seems to be a problem of the file system.
Quoted 8 lines
Quoted 1 line
Any idea or suggestion to solve the problem would be very welcome!
The simple fix is to send a signal(SIGCONT?) to the GC process regularly
to do GC
over time. That way you don't get all GC at boot time.Jocke
BTW, I guess you got logging to file such as syslog on your system? JFFS2 really don't do well with small incremental writes. I recommend you turn such logging off.
Hi Jocke:
Am 27.11.13 11:30 schrieb(en) Joakim Tjernlund:
Quoted 1 line
BTW, I guess you got logging to file such as syslog on your system? JFFS2 really don't do well with small incremental writes. I recommend you turn such logging off.
Does the delay by the gc process happen only when the jffs2 fs is mounted, or also during the normal operation when many updates occur?
Background: I had a similar issue as Harald with a NVRAM which is used to hold quickly changing event buffers and logs. My solution was to wait for the gc process during the startup of the system, before the "main" application is launched. However, if this could also happen *after* the initial scan has finished, this would be bad news for me, as I then probably would need the SIG(STOP|CONT) approach...
Thanks,
Albrecht.
Albrecht Dreß <[email protected]> wrote on 2013/11/27 20:34:06:
Quoted 5 lines
Hi Jocke:
Am 27.11.13 11:30 schrieb(en) Joakim Tjernlund:
Quoted 1 line
BTW, I guess you got logging to file such as syslog on your system?
JFFS2 really don't do well with small incremental writes. I recommend you turn such logging off.
Quoted 2 lines
Does the delay by the gc process happen only when the jffs2 fs is
mounted, or also during the normal operation when many updates occur?
Only at mount AFAIK. During initial scanning lots of GC can be triggered but during normal
operation not so much. The GC algo. is a bit naive and you have to help it if you got
lots of small writes. By sending the correct signal to the GC thread you trigger one GC.
Quoted 2 lines
Background: I had a similar issue as Harald with a NVRAM which is used
to hold quickly changing event buffers and logs. My solution was to wait for the gc process during the startup of the system, before the "main" application is launched. However, if this could also happen *after* the initial scan has finished, this would be bad news for me, as I then probably would need the SIG(STOP|CONT) approach...
Quoted 3 lines
Thanks,
Albrecht.[attachment "attq2f2p.dat" deleted by Joakim
Tjernlund/Transmode]
Hi All,
thank you for the fast reply.
What we figured out is that the jffs2_garbage_collect_pass loops for several minutes in the first for(;;) loop. It always went into the if statement after jffs2_get_ino_cache. In this situation it's not interruptable by any signals.
Regards
Harald
"linux-mtd" <[email protected]> wrote on 2013/11/27 11:11:04:
Quoted 40 lines
Quoted 23 lines
Hi all,
we are currently facing a problem with the Garbage-collection on a JFFS2 file system.
We have an embedded linux system based on an ARM9 AT91RM9200 processor with an Intel / Micron JS28F256 NOR-flash of 32MB in size. The Flash holds the U-Boot loader and its environment and in the main part (30MB) of the chip a JFFS2-based root-file-system.
Our Linux kernel version is 2.6.24.3Some minutes after startup of the system, the garbage collector (
gc.c) does a scan of the file system, gathering information for the garbage collection procedures that may be initiated based upon that data. On some file systems, especially such with long runtime and therefore many file operations, this scan consumes a lot of time. On some systems it stays in that scan for more that 10 minutes. The problem with this behaviour is the fact, that during the scan the file system is locked, and therefore all file operations are suspended. This leads to a crash of the system, when the application program hangs for a long time, waiting for a file operation to return from a system call, and the watchdog is not triggered.Quoted 1 line
In a test system without watchdog, after the scan, the system runs
normally.
Quoted 4 lines
The flash chip itself shows no errors, so it seems to be a problem of the file system.
Any idea or suggestion to solve the problem would be very welcome!
The simple fix is to send a signal(SIGCONT?) to the GC process regularly to do GC
over time. That way you don't get all GC at boot time.Jocke
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
Quoted 8 lines
Hi All,
thank you for the fast reply.
What we figured out is that thejffs2_garbage_collect_passloops for several minutes in the first for(;;) loop. It always went into the if statement afterjffs2_get_ino_cache. In this situation it's not interruptable by any signals.
Yes, by then it is too late. You need to force GC during normal operation so that when you reboot there isn't much GC left to do.
Try sending the right signal to the GC thread, say every minute(just to test something)
Quoted 10 lines
Regards
Harald"linux-mtd" <[email protected]> wrote on 2013/11/27 11:11:04:
Quoted 3 lines
Quoted 3 lines
Hi all,
we are currently facing a problem with the Garbage-collection on a
JFFS2
Quoted 3 lines
Quoted 3 lines
Quoted 3 lines
file system.
We have an embedded linux system based on an ARM9 AT91RM9200
processor
Quoted 6 lines
Quoted 6 lines
Quoted 6 lines
with an Intel / Micron JS28F256 NOR-flash of 32MB in size. The Flash holds the U-Boot loader and its environment and in the main part (30MB) of the chip a JFFS2-based root-file-system. Our Linux kernel version is 2.6.24.3
Some minutes after startup of the system, the garbage collector
(gc.c)
Quoted 4 lines
Quoted 4 lines
Quoted 4 lines
does a scan of the file system, gathering information for the garbage collection procedures that may be initiated based upon that data. On some file systems, especially such with long runtime and therefore many file operations, this scan consumes a lot of time. On some
systems
Quoted 2 lines
Quoted 2 lines
Quoted 2 lines
it stays in that scan for more that 10 minutes. The problem with this behaviour is the fact, that during the scan the file system is
locked,
Quoted 1 line
Quoted 1 line
Quoted 1 line
and therefore all file operations are suspended. This leads to a
crash
Quoted 11 lines
Quoted 11 lines
Quoted 5 lines
of the system, when the application program hangs for a long time, waiting for a file operation to return from a system call, and the watchdog is not triggered.
In a test system without watchdog, after the scan, the system runs
normally.
Quoted 4 lines
The flash chip itself shows no errors, so it seems to be a problem of the file system.
Any idea or suggestion to solve the problem would be very welcome!
The simple fix is to send a signal(SIGCONT?) to the GC process
regularly
Quoted 15 lines
Quoted 10 lines
to do GC
over time. That way you don't get all GC at boot time.Jocke
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/
Hi Harald:
I had *exactly* the same issue with a board based on a MPC5200 (PPC). I use kernel 3.2, and the problematic file system is a jffs2 on a NVRAM which is used to hold quickly changing logs and event buffers (before they are shifted to flash or a CF card). Apparently the scan process is triggered immediately when the NVRAM is mounted by the startup script. Apart from that I had the same picture - the scan eats so much time that the hw wdt is triggered... (see my original post <http://lists.infradead.org/pipermail/linux-mtd/2013-April/046632.html> - I unfortunately never got an answer to it).
My solution was to add a task to the startup script which monitors the scan process (jffs2_gcd_mtd6 in my case) and waits until it apparently finished. The main application (which in turn activates the hardware wdt) is launched only *afterwards*; then everything seems to run just fine, even if there is much "traffic" on the nvram.
The monitor application basically identifies the pid of the jffs2_gcd_mtd6, and then repeatedly reads /proc/<pid>/stat as long as the sum of the utime, stime, cutime and cstime is still growing (of course with a maximum loop count, so the system doesn't deadlock).
Hope this helps,
Albrecht.
Am 27.11.13 11:11 schrieb(en) Albrecht, Harald:
Quoted 22 lines
Hi all,
we are currently facing a problem with the Garbage-collection on a JFFS2 file system.
We have an embedded linux system based on an ARM9 AT91RM9200 processor with an Intel / Micron JS28F256 NOR-flash of 32MB in size. The Flash holds the U-Boot loader and its environment and in the main part (30MB) of the chip a JFFS2-based root-file-system. Our Linux kernel version is 2.6.24.3
Some minutes after startup of the system, the garbage collector (
gc.c) does a scan of the file system, gathering information for the garbage collection procedures that may be initiated based upon that data.
On some file systems, especially such with long runtime and therefore many file operations, this scan consumes a lot of time. On some systems it stays in that scan for more that 10 minutes. The problem with this behaviour is the fact, that during the scan the file system is locked, and therefore all file operations are suspended. This leads to a crash of the system, when the application program hangs for a long time, waiting for a file operation to return from a system call, and the watchdog is not triggered.In a test system without watchdog, after the scan, the system runs normally. The flash chip itself shows no errors, so it seems to be a problem of the file system.
Any idea or suggestion to solve the problem would be very welcome!
Harald Albrecht
______________________________________________________
Linux MTD discussion mailing list
http://lists.infradead.org/mailman/listinfo/linux-mtd/