Potentially unbounded allocations in seq_read?
Hi all,
It seems that the buffer allocation in seq_read can double in size indefinitely, at least I've seen that in practice with /proc/<pid>/smaps (attempting to double m->size to 4M on a read of 1000 bytes). This produces an ugly WARN_ON_ONCE, which should perhaps be avoided? (given that it can be triggered by userspace at will)
Quoted 1 line
From the top comment in
seq_file.cone would think that it is a
fundamental limitation of the current code that everything which will be read (even if in chunks) needs to be in the kernel side buffer at the same time?
If that is true then only way to fix it would be to completely re-design the seq_file interface, just silencing the allocation failure with __GFP_NOWARN perhaps as a temporary measure.
As an alternative, since it does sound a bit pathological, perhaps users for seq_file who know can be printing out such huge amounts of text should just use a different (new?) facility?
Thanks,
Tvrtko
Quoted 8 lines · Tvrtko Ursulin
Hi all,
It seems that the buffer allocation in
seq_readcan double in size indefinitely, at least I've seen that in practice with /proc/<pid>/smaps (attempting to doublem->sizeto 4M on a read of 1000 bytes). This produces an uglyWARN_ON_ONCE, which should perhaps be avoided? (given that it can be triggered by userspace at will)
Quoted 4 lines
From the top comment in
seq_file.cone would think that it is a fundamental limitation of the current code that everything which will be read (even if in chunks) needs to be in the kernel side buffer at the same time?
Oh-oh, seems that m->size is doubled on every read. So if app is reading with a buffer smaller than data available, it can do nine reads before it hits a >MAX_ORDER allocation. Not good. :)
Regards,
Tvrtko
Al Viro
maintainerQuoted 9 lines · Tvrtko Ursulin
Quoted 8 lines · Tvrtko Ursulin
Hi all,
It seems that the buffer allocation in
seq_readcan double in size indefinitely, at least I've seen that in practice with /proc/<pid>/smaps (attempting to doublem->sizeto 4M on a read of 1000 bytes). This produces an uglyWARN_ON_ONCE, which should perhaps be avoided? (given that it can be triggered by userspace at will)
Quoted 8 lines
Quoted 4 lines
From the top comment in
seq_file.cone would think that it is a fundamental limitation of the current code that everything which will be read (even if in chunks) needs to be in the kernel side buffer at the same time?Oh-oh, seems that
m->sizeis doubled on every read. So if app is reading with a buffer smaller than data available, it can do nine reads before it hits a >MAX_ORDER allocation. Not good. :)
Huh? Is that from observation or from reading the code? If it's the former, I would really like to see details, if it's the latter... you are misreading it. m->size is doubled until it's large enough to hold ->show() output; size argument of seq_read() has nothing to do with that. Once the damn thing is large enough, read() is served from it. So are subsequent reads, until you manage to eat all that had been generated. Then the same buffer is used for the next entry; again, no doubling unless that next entry is even bigger and won't fit. Doubling on each read(2) takes really strange iterator to trigger and you'll need ->show() spewing bigger and bigger entries. Again - details, please...
Al Viro
maintainerQuoted 7 lines · Tvrtko Ursulin
Hi all,
It seems that the buffer allocation in
seq_readcan double in size indefinitely, at least I've seen that in practice with /proc/<pid>/smaps (attempting to doublem->sizeto 4M on a read of 1000 bytes). This produces an uglyWARN_ON_ONCE, which should perhaps be avoided? (given that it can be triggered by userspace at will)
An entry in /proc/<pid>/smaps that did not fit into 2Mb? Seriously? How in hell has that happened? If you can trigger that at will, please post the reproducer.
Quoted 5 lines
Quoted 1 line
From the top comment in
seq_file.cone would think that it is afundamental limitation of the current code that everything which will be read (even if in chunks) needs to be in the kernel side buffer at the same time?
Quoted 7 lines
If that is true then only way to fix it would be to completely re-design the
seq_fileinterface, just silencing the allocation failure with__GFP_NOWARNperhaps as a temporary measure.As an alternative, since it does sound a bit pathological, perhaps users for
seq_filewho know can be printing out such huge amounts of text should just use a different (new?) facility?
If a seq_file user is attempting to spew a couple of megs of text in one ->show() call, there's definitely something misused. Either they ought to use a different iterator (might be feasible if that monster entry is produced by some kind of loop) or just not use seq_file at all.
I'm very surprised that /proc/*/smaps has managed to step into that, though - show_pid_smap() shouldn't be able to do so, AFAICS...
Quoted 9 lines · Al Viro
Quoted 7 lines · Tvrtko Ursulin
Hi all,
It seems that the buffer allocation in
seq_readcan double in size indefinitely, at least I've seen that in practice with /proc/<pid>/smaps (attempting to doublem->sizeto 4M on a read of 1000 bytes). This produces an uglyWARN_ON_ONCE, which should perhaps be avoided? (given that it can be triggered by userspace at will)
Quoted 3 lines
An entry in /proc/<pid>/smaps that did not fit into 2Mb? Seriously? How in hell has that happened? If you can trigger that at will, please post the reproducer.
Yeah, no, wrong assumption. It was not about the size but the number of reads. For example:
open("/proc/3131/smaps", O_RDONLY|O_LARGEFILE) = 3 fstat64(1, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...}) = 0
brk(0xf9019000) = 0xf9019000read(3, "be483000-be4ab000 rw-s 00000000 "..., 4096) = 4054 write(1, "be483000-be4ab000 rw-s 00000000 "..., 4054) = 4054 read(3, "be5c3000-be5c5000 rw-s 10da3f000"..., 4096) = 3656 write(1, "be5c3000-be5c5000 rw-s 10da3f000"..., 3656) = 3656 read(3, "be6b1000-be6b2000 rw-s 10da48000"..., 4096) = 3661 write(1, "be6b1000-be6b2000 rw-s 10da48000"..., 3661) = 3661 read(3, "be6b9000-be6ba000 rw-s 10da38000"..., 4096) = 3599 write(1, "be6b9000-be6ba000 rw-s 10da38000"..., 3599) = 3599 read(3, "be6d6000-be7b6000 rw-s 10d889000"..., 4096) = 3599 write(1, "be6d6000-be7b6000 rw-s 10d889000"..., 3599) = 3599 read(3, "be884000-be885000 rw-s 10d85c000"..., 4096) = 3661 write(1, "be884000-be885000 rw-s 10d85c000"..., 3661) = 3661 read(3, "be88d000-be8de000 rw-p 00000000 "..., 4096) = 4007 write(1, "be88d000-be8de000 rw-p 00000000 "..., 4007) = 4007 read(3, "bea29000-bea4d000 r-xp 00000000 "..., 4096) = 4057 write(1, "bea29000-bea4d000 r-xp 00000000 "..., 4057) = 4057 read(3, "beab3000-bead0000 r--p 00000000 "..., 4096) = 2092 write(1, "beab3000-bead0000 r--p 00000000 "..., 2092) = 2092read(3, 0xf9017030, 4096) = -1 ENOMEM (Out of memory)
Regards,
Tvrtko
Al Viro
maintainerQuoted 31 lines · Tvrtko Ursulin
Quoted 9 lines · Al Viro
Quoted 7 lines · Tvrtko Ursulin
Hi all,
It seems that the buffer allocation in
seq_readcan double in size indefinitely, at least I've seen that in practice with /proc/<pid>/smaps (attempting to doublem->sizeto 4M on a read of 1000 bytes). This produces an uglyWARN_ON_ONCE, which should perhaps be avoided? (given that it can be triggered by userspace at will)Quoted 3 lines
An entry in /proc/<pid>/smaps that did not fit into 2Mb? Seriously? How in hell has that happened? If you can trigger that at will, please post the reproducer.
Yeah, no, wrong assumption. It was not about the size but the number of reads. For example:
open("/proc/3131/smaps", O_RDONLY|O_LARGEFILE)= 3fstat64(1, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...})= 0
brk(0xf9019000) =0xf9019000read(3, "be483000-be4ab000 rw-s 00000000 "..., 4096)= 4054write(1, "be483000-be4ab000 rw-s 00000000 "..., 4054)= 4054read(3, "be5c3000-be5c5000 rw-s 10da3f000"..., 4096)= 3656write(1, "be5c3000-be5c5000 rw-s 10da3f000"..., 3656)= 3656read(3, "be6b1000-be6b2000 rw-s 10da48000"..., 4096)= 3661write(1, "be6b1000-be6b2000 rw-s 10da48000"..., 3661)= 3661read(3, "be6b9000-be6ba000 rw-s 10da38000"..., 4096)= 3599write(1, "be6b9000-be6ba000 rw-s 10da38000"..., 3599)= 3599read(3, "be6d6000-be7b6000 rw-s 10d889000"..., 4096)= 3599write(1, "be6d6000-be7b6000 rw-s 10d889000"..., 3599)= 3599read(3, "be884000-be885000 rw-s 10d85c000"..., 4096)= 3661
Quoted 8 lines
write(1, "be884000-be885000 rw-s 10d85c000"..., 3661)= 3661read(3, "be88d000-be8de000 rw-p 00000000 "..., 4096)= 4007write(1, "be88d000-be8de000 rw-p 00000000 "..., 4007)= 4007read(3, "bea29000-bea4d000 r-xp 00000000 "..., 4096)= 4057write(1, "bea29000-bea4d000 r-xp 00000000 "..., 4057)= 4057read(3, "beab3000-bead0000 r--p 00000000 "..., 4096)= 2092write(1, "beab3000-bead0000 r--p 00000000 "..., 2092)= 2092read(3, 0xf9017030, 4096)= -1 ENOMEM (Out of memory)
OK, so you've got ENOMEM somewhere, but where had it come from? The buffer from previous read() ought to have sufficed for this one, unless the next entry had been much longer than usual...
Quoted 41 lines · Al Viro
Quoted 31 lines · Tvrtko Ursulin
Quoted 9 lines · Al Viro
Quoted 7 lines · Tvrtko Ursulin
Hi all,
It seems that the buffer allocation in
seq_readcan double in size indefinitely, at least I've seen that in practice with /proc/<pid>/smaps (attempting to doublem->sizeto 4M on a read of 1000 bytes). This produces an uglyWARN_ON_ONCE, which should perhaps be avoided? (given that it can be triggered by userspace at will)Quoted 3 lines
An entry in /proc/<pid>/smaps that did not fit into 2Mb? Seriously? How in hell has that happened? If you can trigger that at will, please post the reproducer.
Yeah, no, wrong assumption. It was not about the size but the number of reads. For example:
open("/proc/3131/smaps", O_RDONLY|O_LARGEFILE)= 3fstat64(1, {st_mode=S_IFCHR|0666, st_rdev=makedev(1, 3), ...})= 0
brk(0xf9019000) =0xf9019000read(3, "be483000-be4ab000 rw-s 00000000 "..., 4096)= 4054write(1, "be483000-be4ab000 rw-s 00000000 "..., 4054)= 4054read(3, "be5c3000-be5c5000 rw-s 10da3f000"..., 4096)= 3656write(1, "be5c3000-be5c5000 rw-s 10da3f000"..., 3656)= 3656read(3, "be6b1000-be6b2000 rw-s 10da48000"..., 4096)= 3661write(1, "be6b1000-be6b2000 rw-s 10da48000"..., 3661)= 3661read(3, "be6b9000-be6ba000 rw-s 10da38000"..., 4096)= 3599write(1, "be6b9000-be6ba000 rw-s 10da38000"..., 3599)= 3599read(3, "be6d6000-be7b6000 rw-s 10d889000"..., 4096)= 3599write(1, "be6d6000-be7b6000 rw-s 10d889000"..., 3599)= 3599read(3, "be884000-be885000 rw-s 10d85c000"..., 4096)= 3661Quoted 8 lines
write(1, "be884000-be885000 rw-s 10d85c000"..., 3661)= 3661read(3, "be88d000-be8de000 rw-p 00000000 "..., 4096)= 4007write(1, "be88d000-be8de000 rw-p 00000000 "..., 4007)= 4007read(3, "bea29000-bea4d000 r-xp 00000000 "..., 4096)= 4057write(1, "bea29000-bea4d000 r-xp 00000000 "..., 4057)= 4057read(3, "beab3000-bead0000 r--p 00000000 "..., 4096)= 2092write(1, "beab3000-bead0000 r--p 00000000 "..., 2092)= 2092read(3, 0xf9017030, 4096)= -1 ENOMEM (Out of memory)
Quoted 3 lines
OK, so you've got ENOMEM somewhere, but where had it come from? The buffer from previous
read()ought to have sufficed for this one, unless the next entry had been much longer than usual...
So this is the story... task_mmu.c:show_map_vma() calls seq_path. There we have a d_path call which returns -ENAMETOOLONG and keeps doing so even though the buffer grows to huge proportions. It is something on tmpfs, don't know what.
But in the meantime, shouldn't seq_path be a bit more considerate on this particular error and not mark the state as "could not fit" forever? Perhaps it would make sense to limit it a bit?
Or even more so, on errors _other_ than -ENAMETOOLONG it will at the moment mark the result as "need more space". That also sounds broken to me.
Regards,
Tvrtko
Al Viro
maintainerQuoted 5 lines · Tvrtko Ursulin
So this is the story... task_mmu.c:show_map_vma() calls
seq_path. There we have ad_pathcall which returns-ENAMETOOLONGand keeps doing so even though the buffer grows to huge proportions. It is something on tmpfs, don't know what.
Quoted 7 lines
But in the meantime, shouldn't
seq_pathbe a bit more considerate on this particular error and not mark the state as "could not fit" forever? Perhaps it would make sense to limit it a bit?Or even more so, on errors _other_ than
-ENAMETOOLONGit will at the moment mark the result as "need more space". That also sounds broken to me.
a) *what* errors other than -ENAMETOOLONG?
b) d_path() not fitting into 2Mb is definitely a bug. If you really have managed to get a dentry tree 1 million levels deep, you have much worse problems.
c) which kernel version it is?
Quoted 11 lines · Al Viro
Quoted 9 lines · Tvrtko Ursulin
So this is the story... task_mmu.c:show_map_vma() calls
seq_path. There we have ad_pathcall which returns-ENAMETOOLONGand keeps doing so even though the buffer grows to huge proportions. It is something on tmpfs, don't know what.But in the meantime, shouldn't
seq_pathbe a bit more considerate on this particular error and not mark the state as "could not fit" forever? Perhaps it would make sense to limit it a bit?
Quoted 5 lines
Quoted 3 lines
Or even more so, on errors _other_ than
-ENAMETOOLONGit will at the moment mark the result as "need more space". That also sounds broken to me.a) *what* errors other than
-ENAMETOOLONG?
Is this your way of saying there can't be any other errors from d_path?
Quoted 4 lines
b)
d_path()not fitting into 2Mb is definitely a bug. If you really have managed to get a dentry tree 1 million levels deep, you have much worse problems.
c) which kernel version it is?
3.10. I can't imagine this is an actual dentry tree somewhere, probably just a bug of some sort. I'll probably hunt it down completely some time next week, time permitting.
Regards,
Tvrtko
Al Viro
maintainerQuoted 3 lines · Tvrtko Ursulin
Quoted 1 line
a) *what* errors other than
-ENAMETOOLONG?Is this your way of saying there can't be any other errors from
d_path?
Check yourself... It is the only error that makes sense there and yes, it is the only one being returned.
Quoted 8 lines
Quoted 4 lines
b)
d_path()not fitting into 2Mb is definitely a bug. If you really have managed to get a dentry tree 1 million levels deep, you have much worse problems.
c) which kernel version it is?3.10. I can't imagine this is an actual dentry tree somewhere, probably just a bug of some sort. I'll probably hunt it down completely some time next week, time permitting.
Sounds like missing backport of 118b23 ("cope with potentially long ->d_dname() output for shmem/hugetlb"). It *is* in -stable (linux-3.10.y has it since 3.10.17 as commit ad4c3c), but if your tree doesn't have it, that's the one to try first...
Quoted 8 lines · Al Viro
Quoted 3 lines · Tvrtko Ursulin
Quoted 1 line
a) *what* errors other than
-ENAMETOOLONG?Is this your way of saying there can't be any other errors from
d_path?Check yourself... It is the only error that makes sense there and yes, it is the only one being returned.
Cool then, thats why I e-mailed you in the first place, because you know this area very well.
Quoted 5 lines
Quoted 5 lines
Quoted 4 lines
b)
d_path()not fitting into 2Mb is definitely a bug. If you really have managed to get a dentry tree 1 million levels deep, you have much worse problems.
c) which kernel version it is?
Quoted 8 lines
Quoted 3 lines
3.10. I can't imagine this is an actual dentry tree somewhere, probably just a bug of some sort. I'll probably hunt it down completely some time next week, time permitting.
Sounds like missing backport of 118b23 ("cope with potentially long ->d_dname() output for shmem/hugetlb"). It *is* in -stable (linux-3.10.y has it since 3.10.17 as commit ad4c3c), but if your tree doesn't have it, that's the one to try first...
Yes that fixed it, thanks!
Regards,
Tvrtko