Skip to contentSkip to conversation

New: jbd2 write on disk each 5/6 seconds

Bug 76821 [email protected] 1 participant 9 messages May 24, 2014 – May 25, 2014

lore
Share & tools
lore

[email protected]

author
#0 lore raw

https://bugzilla.kernel.org/show_bug.cgi?id=76821

            Bug ID: 76821
           Summary: jbd2 write on disk each 5/6 seconds
           Product: File System
           Version: 2.5
    Kernel Version: 3.14.4-1-ARCH
          Hardware: All
                OS: Linux
              Tree: Mainline
            Status: NEW
          Severity: normal
          Priority: P1
         Component: ext4
          Assignee: [email protected]
          Reporter: [email protected]
        Regression: No

➜ ~ sudo iotop -obtqqq | grep jbd2
[sudo] password for hacko:
13:12:05 390 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.87 %
[jbd2/sda6-8]
13:12:06 390 be/3 root 0.00 B/s 31.45 K/s 0.00 % 2.48 %
[jbd2/sda6-8]
13:12:12 111 be/3 root 0.00 B/s 11.53 K/s 0.00 % 4.99 %
[jbd2/sda3-8]
13:12:19 390 be/3 root 0.00 B/s 3.84 K/s 0.00 % 2.98 %
[jbd2/sda6-8]
13:12:25 390 be/3 root 0.00 B/s 3.84 K/s 0.00 % 5.91 %
[jbd2/sda6-8]
13:12:25 111 be/3 root 0.00 B/s 23.03 K/s 0.00 % 5.71 %
[jbd2/sda3-8]
13:12:31 390 be/3 root 0.00 B/s 3.84 K/s 0.00 % 4.42 %
[jbd2/sda6-8]
13:12:32 390 be/3 root 0.00 B/s 0.00 B/s 0.00 % 0.03 %
[jbd2/sda6-8]
13:12:34 390 be/3 root 0.00 B/s 23.05 K/s 0.00 % 7.62 %
[jbd2/sda6-8]
13:12:39 390 be/3 root 0.00 B/s 26.86 K/s 0.00 % 4.40 %
[jbd2/sda6-8]
13:12:46 390 be/3 root 0.00 B/s 3.83 K/s 0.00 % 4.42 %
[jbd2/sda6-8]
13:12:53 390 be/3 root 0.00 B/s 11.49 K/s 0.00 % 4.31 %
[jbd2/sda6-8]
13:13:01 390 be/3 root 0.00 B/s 11.53 K/s 0.00 % 4.46 %
[jbd2/sda6-8]
13:13:07 390 be/3 root 0.00 B/s 7.69 K/s 0.00 % 4.55 %
[jbd2/sda6-8]
13:13:12 390 be/3 root 0.00 B/s 11.56 K/s 0.00 % 4.85 %
[jbd2/sda6-8]
13:13:19 390 be/3 root 0.00 B/s 11.56 K/s 0.00 % 4.27 %
[jbd2/sda6-8]
13:13:23 390 be/3 root 0.00 B/s 46.13 K/s 0.00 % 9.26 %
[jbd2/sda6-8]
13:13:26 390 be/3 root 0.00 B/s 7.54 K/s 0.00 % 6.05 %
[jbd2/sda6-8]
13:13:31 390 be/3 root 0.00 B/s 30.01 K/s 0.00 % 4.60 %
[jbd2/sda6-8]

Signature · 5 lines
You are receiving this mail because:
You are watching the assignee of the bug.--
To unsubscribe from this list: send the line "unsubscribe linux-ext4" in
the body of a message to [email protected]
More majordomo info at  http://vger.kernel.org/majordomo-info.html
author
#3 lore raw

[Bug 76821] jbd2 write on disk each 2/3 seconds

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

Abdelhak Bougouffa <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

            Summary|jbd2 write on disk each 5/6 |jbd2 write on disk each 2/3
                   |seconds                     |seconds
Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
author
#4 lore raw

[Bug 76821] jbd2 write on disk each 2/3 seconds

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

--- Comment #1 from Abdelhak Bougouffa <[email protected]> --- Hi,
I have observed that "kworker" is writing to the disk even when no program opened >.<, when I ran iotop -Po I see that "kworker" write more than jbd2 :-(

I have no idea how to solve this!

I searched the web and I found that many users have the same problem

https://bbs.archlinux.org/viewtopic.php?id=171608

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
author
#5 lore raw

[Bug 76821] jbd2 write on disk each 2/3 seconds

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

Theodore Tso <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------
CC| |[email protected]

--- Comment #2 from Theodore Tso <[email protected]> ---
If your kernel doesn't have ftrace enabled, please enable it by enabling the following kernel config options:

CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE

Once you do that, make sure debugfs is mounted:

mount -t debugfs debugfs /sys/kernel/debug

Now enable the ext4_mark_inode_dirty tracepoint and start tracing:

cd /sys/kernel/debug/tracing
echo 1 > events/ext4/ext4_mark_inode_dirty/enable
cat trace_pipe

You will start seeing lines like this:

kworker/u16:2-14718 [006] ...1 616239.342464: ext4_mark_inode_dirty: dev
252,0 ino 4203428 caller ext4_writepages+0x743/0x93e
BrowserBlocking-5875 [006] ...1 616241.972533: ext4_mark_inode_dirty: dev 8,3
ino 3545316 caller ext4_dirty_inode+0x40/0x59

The first line indicates that process 14718, which is a kernel thread named kworking/u16:2, dirtied inode 4203428 on device 252,0, which is /dev/dm-0 (which on my system has major number 252, minor 0, which you can see by the
command ls -l /dev/dm-0). This is probably dirty inode writeback, so that
doesn't help tell you who dirtied the file. You can find the filename using debugfs:

debugfs /dev/dm-0
debugfs: ncheck 4203428
Inode Pathname
4203428 /build/e2fsprogs-maint/tests/f_eofblocks.1.log

(and since /dev/dm-0 is mounted on /u1, I know the filename is /u1/build/e2fsprogs-maint/....)

The second line indicates the task id was 5875, and with a name like BrowserBlocking, it's almost certainly one of the Chrome threads. The ps program won't show individual threads by default, but you can also use the inode number to find which process has the file open by using lsof:

lsof | grep 3545316

Once you're done, don't forget to disable the tracing to save a bit on CPU and battery life:

cat /dev/null > /sys/kernel/debug/tracing/set_event

Anyway, I hope this helps.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
author
#6 lore raw

[Bug 76821] jbd2 write on disk each 2/3 seconds

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

--- Comment #3 from Abdelhak Bougouffa <[email protected]> --- Hi,
Thank you Mr. Theodore

So, I tried this commands, and I get a big number of process!

I observed that the kworker is not really the cause! the systemd-journal is writing all times to the disk!, When I traced one of his used inodes, I found it writing to /var/log/journal/****.journal, each time a file of 8MB.

http://pastebin.com/W3yu3SVH

It's notable that the debugging options:
CONFIG_FUNCTION_TRACER
CONFIG_FUNCTION_GRAPH_TRACER
CONFIG_STACK_TRACER
CONFIG_DYNAMIC_FTRACE

Are enabled in my working kernel (I installed it using the Arch repository)

Thanks!

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.
author
#8 lore raw

[Bug 76821] jbd2 write on disk each 2/3 seconds

In reply to [email protected]

https://bugzilla.kernel.org/show_bug.cgi?id=76821

Theodore Tso <[email protected]> changed:

What |Removed |Added
----------------------------------------------------------------------------

             Status|NEW                         |RESOLVED
         Resolution|---                         |INVALID

--- Comment #5 from Theodore Tso <[email protected]> ---
You can try running "journalctl -f" and see what kind of log messages you are seeing. But in any case, this isn't an ext4 bug, but rather a systemd, journald, or some system daemon or distro-level bug.

If userspace shovels stuff for us to write, we have to write it! So for that
reason, I've closed out this bug. If this isn't a configuration issue, please open a bug with the appropriate distribution or systemd bug tracking system.

Signature · 2 lines
You are receiving this mail because:
You are watching the assignee of the bug.