Skip to contentSkip to conversation

Tracking actual disk write sources instead of flush thread

DISCUSSION Phillip Susi 7 participants 17 messages Apr 16, 2014 – Apr 28, 2014 VFS

lore
Share & tools
lore
#0 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

A lot of disk writes, especially when they are small individual files being written by several different processes, are hidden behind the flush thread. Is there no way to properly track the process actually responsible for the IO, even when it is the flush thread that initiates the writeout?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTTeRzAAoJEI5FoCIzSKrw9WUH/jTozXvpeN7A0uN8KeafAhdk agVupehoAYUDzDdYP3+JDkCmp5vgymzZydFasrPyLHmVqGsnojr/s/r6BowiQT73 Nks90Po+5wsburECDiuRz91POBbE5PTu0laK0fJWpTqeX+gZUlZBuNQlz8B2s2dr wwtnN7tSVPc+H7TVrJz5CI/zW+rVjADXuA5GEshXg3/jfh9UnA/W5kKEVSbBvo7t SwLH0jQ173qbTjWMu4DQIo4mAJ+JC0fYpxnmsIDZlIdNQ3JvbnwG03eyKJiDTRE1 hWklKcxSvv6MDBWcb0fhbIcLhfnQGGFq16szkQO1KmBg1/MDe7HSEncc+MooQOc= =3C3I
-----END PGP SIGNATURE-----

#1 lore raw
Quoted 5 lines · Phillip Susi

A lot of disk writes, especially when they are small individual files being written by several different processes, are hidden behind the flush thread. Is there no way to properly track the process actually responsible for the IO, even when it is the flush thread that initiates the writeout?

Correct.

#2 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 8 lines · Matthew Wilcox
Quoted 5 lines · Phillip Susi

A lot of disk writes, especially when they are small individual files being written by several different processes, are hidden behind the flush thread. Is there no way to properly track the process actually responsible for the IO, even when it is the flush thread that initiates the writeout?

Correct.

Wow. If I understand things correctly, this also means that if process A dirties a ton of cache pages, then process B tries to write a relatively small amount, it can end up blocking in the synchronous flush path, and so it will appear that process B and flush are doing all of the writes, and not process A.

That seems like a severe defect. How can such a defect be tolerated in this day and age? Why does the io accounting not track how many pages the process dirties rather than how many it actually initiates the writeout for?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTTp6dAAoJEI5FoCIzSKrwBV0IAJGguGEYYWIMe0II9MtbQB1e S+ObSEygB1u6xt9GABLaUYhC7+e53vpx0/yNMbhQy+Zte79aBHXGYlP0fG4itazt 1QVjstMnSnmu6/9Q3wif3ldRgN56OkrA//L4zufLSNyWOQIRJ+I7HIg1S7ZqsvL8 Rvewat7i9jbCgT98FEqeUQp0iXqDkoZ+BVrS5s/bgBkSPdCLCiS0pRva+guabjSm B9r7n5XGrjMqRJsSGX2kofpv3sZ7NhyTBFhbryZaBe6WIChS121rFZxqAWBTqPbp Xm7N03djIekXD6Y+onLNloKV9Voe4k/1BLx1PLMjy1TRD4Cv6iTNWCB6x7ppTxM= =yp+R
-----END PGP SIGNATURE-----

#3 lore raw
Quoted 16 lines · Phillip Susi
Quoted 8 lines · Matthew Wilcox
Quoted 5 lines · Phillip Susi

A lot of disk writes, especially when they are small individual files being written by several different processes, are hidden behind the flush thread. Is there no way to properly track the process actually responsible for the IO, even when it is the flush thread that initiates the writeout?

Correct.

Wow. If I understand things correctly, this also means that if process A dirties a ton of cache pages, then process B tries to write a relatively small amount, it can end up blocking in the synchronous flush path, and so it will appear that process B and flush are doing all of the writes, and not process A.

Quoted 4 lines

That seems like a severe defect. How can such a defect be tolerated in this day and age? Why does the io accounting not track how many pages the process dirties rather than how many it actually initiates the writeout for?

For Lustre (which has the added difficulty that the thread doing the actual writeout is on a remote server) we track the last process that dirtied the inode in the Lustre-private part of the inode itself, and account the writes against that process. By default, we store the process name + UID in a 32-byte string in the inode, but this can also be changed to store a cluster-unique job identifier (jobid).

For us, recording the PID isn't very useful, since the PID will be different on each node in the cluster, but since it is just a string it would be possible to store any identifier that fits.

Cheers, Andreas

#4 lore raw
In reply to Phillip Susi
Quoted 6 lines

That seems like a severe defect. How can such a defect be tolerated in this day and age? Why does the io accounting not track how many pages the process dirties rather than how many it actually initiates the writeout for?

Typically File System services do not offer semantics of transactional isolation. Attempts to add that took place and were rejected. Therefore, we are speaking only about some potential performance penalty, right?

#5 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 4 lines · Boris

Typically File System services do not offer semantics of transactional isolation. Attempts to add that took place and were rejected. Therefore, we are speaking only about some potential performance penalty, right?

Not at all. This has nothing to do with transactions; it is simply a question of being able to identify what process is causing all of the disk IO, like via iotop. By counting writes in the process that initiates the writeout of dirty pages, rather than the process that dirties the page, it renders the write io tracking largely inaccurate to the point of being useless at times, since often the process dirtying the pages, even with an actual write() system call, is not the process that initiates the writeout.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTTslkAAoJEI5FoCIzSKrwb4MH/AzwLC8GXD0huUYsU7xqEIDj ZciTTSFMVTXa47tK8jIIbPLW3vTHyf0Dm/F1rnjn8jnJwhSqnVFfYKfUTrNQy/ze 7hJQXZ2oORZxp7jwPo3bHR1Oir0T1yXQdE5Ao7/W6wWCXY+JHEEvK1nyCPIClq+T 7TCLbkA3L0qHlGpaJMU7FkE9VaT2EI/8SKrQ05Hy2pLcksiAePnerFvxgCGIjSGU uSXPEqamzl42eqU6pyTdTMM7yJDyIhluCH02he7y56tCX7IU5oEG6LYsDYKfsVLd mEQeO2gCwcgKSrSQsjguPwTC5vyaomrYWvFt37jvvBWjcQwWj7A/kX6N9SBQOqY= =4zjT
-----END PGP SIGNATURE-----

#6 lore raw
Quoted 7 lines · Boris
Quoted 4 lines

Typically File System services do not offer semantics of transactional isolation. Attempts to add that took place and were rejected. Therefore, we are speaking only about some potential performance penalty, right?

Quoted 7 lines

Not at all. This has nothing to do with transactions; it is simply a question of being able to identify what process is causing all of the disk IO, like via iotop. By counting writes in the process that initiates the writeout of dirty pages, rather than the process that dirties the page, it renders the write io tracking largely inaccurate to the point of being useless at times, since often the process dirtying the pages, even with an actual write() system call, is not the process that initiates the writeout.

[bz:]
In such case I'd rather have lighter implementation of caching layer, than ability to track disk IOs per process. IOs per process can be tracked by other tools...

#7 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 3 lines · Boris

[bz:] In such case I'd rather have lighter implementation of caching layer, than ability to track disk IOs per process. IOs per process can be tracked by other tools...

What other tools?

I can envision blktrace being extended to hook into the mm layer to track the real source of writes, but without any overhead when the trace is not running.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTTtm3AAoJEI5FoCIzSKrwPPkIAJKnDv4uJfBAufqRlQbRPmcM T3F6Dapzs2XSiMaBCaMeZQx4Zp7wxn79IM76GQeNUekHrV74kFqwLF7ezLpAviEm IBdnhWv3uJXS6uX40os0DOqlRj5H49EhPzbLN6f5z4ARds06Zc+Z9flauCadZyce j/1tinzm0cNYUxAIL9L+m1x7kj8Qzpq5LYyOEw7M8AL93xMGPJLgt4xWUP0JwP3l cgiZKGD6H/ySDAFhckynCjmLdaDDNMOg8uWhyDNk8VQQnWuAzEwK3QgpfnLNSfkJ vo1lvqMp86doRHc7pBoqY6cCU/iIrmXiMGMjVCXExkVFK4GrBHYlqlsBHnY5qys= =mtN2
-----END PGP SIGNATURE-----

#15 lore raw
In reply to Phillip Susi
Quoted 5 lines · Phillip Susi

That seems like a severe defect. How can such a defect be tolerated in this day and age? Why does the io accounting not track how many pages the process dirties rather than how many it actually initiates the writeout for?

There is no per process dirty counter anyways.

The exiting proces write counter is legacy and generally useless.

If you want to track syscalls there are already plenty of ways to do that per process. There's no way to track background mmap flushes and it may be meaningless in this case anyways.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only
#8 lore raw
In reply to Phillip Susi
Quoted 10 lines · Phillip Susi
Quoted 3 lines · Boris

[bz:] In such case I'd rather have lighter implementation of caching layer, than ability to track disk IOs per process. IOs per process can be tracked by other tools...

What other tools?

I can envision blktrace being extended to hook into the mm layer to track the real source of writes, but without any overhead when the trace is not running.

How do you 'envision' this zero-overhead trace, exactly?

I don't understand your high-level goal, which makes suggesting low-overhead solutions hard. Can you tolerate a certain amount of ambiguity, for example? Do you really only want to track back to the UID that is causing the I/O? With shared mmaps, are you OK attributing the I/O to one of the processes that has written to it, or do you need to attribute the write to all the processes that have written to that page?

You're coming off kind of condescending, which isn't a great approach when you're asking for a new feature to be implemented.

#9 lore raw

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 1 line · Matthew Wilcox

How do you 'envision' this zero-overhead trace, exactly?

The same way blktrace currently works: the accounting code is not run until enabled. When disabled, nothing more runs than does now. Obviously while actively tracing, there would be overhead.

Quoted 7 lines

I don't understand your high-level goal, which makes suggesting low-overhead solutions hard. Can you tolerate a certain amount of ambiguity, for example? Do you really only want to track back to the UID that is causing the I/O? With shared mmaps, are you OK attributing the I/O to one of the processes that has written to it, or do you need to attribute the write to all the processes that have written to that page?

I suppose the first process that dirties the page would be fine. It isn't very often that more than one process is writing to the same data at the same time.

Quoted 2 lines

You're coming off kind of condescending, which isn't a great approach when you're asking for a new feature to be implemented.

I'm just a little flabbergasted that this regression ( I'm sure there was a time when the current io accounting mechanism did work, probably before the buffer_head -> page cache transition way, *way* back when ) went unfixed for so long, especially when it is kind of a vital tool for a sysadmin trying to figure out why his system is slow. I think every sysadmin out there takes it for granted that running iotop should let them spot what process or processes are the source of all the IO, so I almost can't believe that it doesn't really work.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2.0.17 (MingW32)
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBAgAGBQJTWBbnAAoJEI5FoCIzSKrwJaUH/RG3kn8pDxtoqvijAZj6BgnT iuI/ptnFhaYo22p3coRggqYHKI0Nu6MmNJx9Iq2g6lfoFTCY02Bb9fZ9r05Svg5h pQaLvk0dsK1vNwYTHW573KkMiyeUTvi7nUeRUB9MTarhHO6HveqvENd/jEiviCE4 zOqyT15V9Riwm78L5ytQFNsq43wNtZ4d9MUmw0182f/IRtpHn/G1B0UroqjZgs/+ a5hIudxajar5oVfR6O0A7+guKkJa4b8ibUHns4zgbEo1HbO7taOcn6rNoNV/C3NK lN5/0lcHNZwvk0JN9diiEP4812KWZyJIFm7E8FlvchkXNCeHG5hNHjg7FUB1t30= =Cgep
-----END PGP SIGNATURE-----

#10 lore raw
Quoted 3 lines · Phillip Susi

I'm just a little flabbergasted that this regression ( I'm sure there was a time when the current io accounting mechanism did work, probably before the buffer_head -> page cache transition way, *way* back when )

Using what tool? Task-level io accounting is new compared to the buffer -> page cache transition, as far as I can remember.

- Ted

#12 lore raw
In reply to Phillip Susi
Quoted 9 lines · Phillip Susi
Quoted 7 lines · Matthew Wilcox

I don't understand your high-level goal, which makes suggesting low-overhead solutions hard. Can you tolerate a certain amount of ambiguity, for example? Do you really only want to track back to the UID that is causing the I/O? With shared mmaps, are you OK attributing the I/O to one of the processes that has written to it, or do you need to attribute the write to all the processes that have written to that page?

Quoted 3 lines

I suppose the first process that dirties the page would be fine. It isn't very often that more than one process is writing to the same data at the same time.

I think that adding a pointer or integer per page would meet resistance, but I think it is pretty reasonable to track this on a per-inode basis. It is fairly uncommon to have multiple threads writing to the same file, and I would guess it is vanishingly rare that different applications are writing to the same file at one time.

Storing {current->comm}.{pid} would take 20 bytes of space per inode, but would be much more useful than just storing {pid}, since a process may be long gone by the time that the blocks are even submitted to disk due to delayed allocation and such.

It would be possible to store a refcounted struct with this info pointed to from the inode, since it would only be useful on inodes being written, but that has to be balanced against the complexity of maintaining that struct and the potential of saving 12 bytes per inode (since there would still need to be a pointer in the inode).

There are potentially a number of other fields in struct inode that are only used during writes (i_size_seqcount, dirtied_when (how did that avoid getting an i_ prefix?), i_wb_list, i_writecount) that might also be moved to a separate struct that is allocated only for files being written (add 8-byte pointer, subtract 32 bytes for fields). That would have the benefit of slimming down the majority of files not currently being written, and make the addition "write source" information less costly.

Cheers, Andreas

#11 lore raw
In reply to Theodore Ts'o

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 8 lines · Theodore Ts'o
Quoted 4 lines · Phillip Susi

I'm just a little flabbergasted that this regression ( I'm sure there was a time when the current io accounting mechanism did work, probably before the buffer_head -> page cache transition way, *way* back when )

Using what tool? Task-level io accounting is new compared to the buffer -> page cache transition, as far as I can remember.

Using iotop. I thought that task level io accounting went all the way back to bsd?

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTWGbZAAoJEI5FoCIzSKrwAL0H/1X5SLCuLytZFmZ/EGFvpXwB 1GuMwxwwyRR+fNVTAiCvk7mHI1i1k/qfPujAc1aRtnJaAxmSSsFYo2YAMocE0Xl+ AOave6IYDaLDZruagPA/rYcfcYDrJtR6eBrVkroPGUNzkCLULOIveUYEB4WtzGft hYn5U1hgkTwQWrsaZurjoCWj9Dxd751ohsGVzp/+6f7/G/UWwdgTHwMfNerlE4g5 tzCAud/WcIMGcxO9JQoBvY9qxp3NEHVYAlVE1kcsTjdA46u1CxRm86+vpkWM86qR vOQ7eBfBngCgiw8iaTp3tIj+22E3TzH9FzwUB1aPmO7KpFIRa1Qb5AwN0MYgdWA= =UCmL
-----END PGP SIGNATURE-----

#13 lore raw
In reply to Andreas Dilger

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512

Quoted 6 lines · Andreas Dilger

I think that adding a pointer or integer per page would meet resistance, but I think it is pretty reasonable to track this on a per-inode basis. It is fairly uncommon to have multiple threads writing to the same file, and I would guess it is vanishingly rare that different applications are writing to the same file at one time.

Wherever the current counter is should be fine, the question is when it gets updated. Rather than updating it on sync it just needs updated on actual write() / page dirty.

Quoted 10 lines

Storing {current->comm}.{pid} would take 20 bytes of space per inode, but would be much more useful than just storing {pid}, since a process may be long gone by the time that the blocks are even submitted to disk due to delayed allocation and such.

It would be possible to store a refcounted struct with this info pointed to from the inode, since it would only be useful on inodes being written, but that has to be balanced against the complexity of maintaining that struct and the potential of saving 12 bytes per inode (since there would still need to be a pointer in the inode).

That might be a bit overkill. I believe the current interface just has a counter of bytes of IO hung off the task, so you can't catch very short lived processes. That's probably not too bad. I don't think it needs tracked on a per inode basis; just being able to sort out that this process is doing the writing instead of some other random process or the flush task that actually ends up submitting the bio.

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1
Comment: Using GnuPG with Thunderbird - http://www.enigmail.net/

iQEcBAEBCgAGBQJTWGs7AAoJEI5FoCIzSKrwbiUH/iCQYRn74zTt/G1pSVEKiNCj +j0AL004tkU8UHZkgUvTedYzMyaqy5gs59IahwuPX+u/U6jRIUElrkBpecL5E+q/ OgfL+g/HybC0YbMJDKWedjRshxwHyjLLhRuPxTPSLXLxOPj8ZCvhzg2Hc2UkswJ5 /mIiWlKt70Ezxf2sq4Xnna8nGk6iuTlNqCc/VkB/AOu+aSsYcdIkoi1T/O+vUMHz uRUwVuo+grn85NdRjpL0l7sXT1FJaJsbQkjy72akiBEDWN4J/3w48ZaoHr1Gv7tB uCgWmZgPdQYeftkK0gnU7MSbWplcr7VrwdnYr9BdzEluKaX6I0vxejbpPYcAwOo= =p1rv
-----END PGP SIGNATURE-----

reviewer
#16 lore raw
In reply to Phillip Susi
Quoted 11 lines · Phillip Susi

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Quoted 7 lines · Matthew Wilcox
Quoted 5 lines · Phillip Susi

A lot of disk writes, especially when they are small individual files being written by several different processes, are hidden behind the flush thread. Is there no way to properly track the process actually responsible for the IO, even when it is the flush thread that initiates the writeout?

Quoted 7 lines
Quoted 1 line

Correct.

Wow. If I understand things correctly, this also means that if process A dirties a ton of cache pages, then process B tries to write a relatively small amount, it can end up blocking in the synchronous flush path, and so it will appear that process B and flush are doing all of the writes, and not process A.

There is nothing like synchronous flush path anymore (since 3.1 or
something like that). All flushing is done by the flusher works these days (and sync(2), fsync(2) obviously). And you are right we have no way to attribute particular dirty pages in page cache to a process. However we do track how many pages each process has dirtied and use this information in throttling processes in balance_dirty_pages() (the process waits in that function for a time proportional to the number of dirtied pages since it last entered that function).

However that dirty counter isn't very useful for administrative purposes because it gets zeroed out in balance_dirty_pages(). But as others have pointed out for tracking originators of dirty pages, we would like something which survives longer than a process anyway.

Honza

Signature · 2 lines
Jan Kara <[email protected]>
SUSE Labs, CR
#14 lore raw
In reply to Andreas Dilger
Quoted 3 lines · Andreas Dilger

I think that adding a pointer or integer per page would meet resistance, but I think it is pretty reasonable to track this on a per-inode basis.

That's a really bad hot cache line in the making.

Quoted 3 lines

It is fairly uncommon to have multiple threads writing to the same file, and I would guess it is vanishingly rare that different applications are writing to the same file at one time.

Sounds like a bad assumption to me.
Likely would be a scalability disaster.

-Andi

Signature · 1 line
[email protected] -- Speaking for myself only