Skip to contentSkip to conversation

DOS by unprivileged user

DISCUSSION Ferry Toth 7 participants 14 messages Apr 19, 2018 – Apr 30, 2018 3 messages not in the archive

lore
Share & tools
lore

Ferry Toth

author
#0 lore raw

It appears any ordinary user can easily create a DOS on linux.

One sure way to reproduce this is to open gitk on the linux kernel repo (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core
+ hyperthreading. But I will be easy enough to get the same effect with more RAM, other fs etc.

In this case gitk allocates more and more memory (until my system freezes 6.5GB of 7.5GB avaiable), the system starts swapping or writing to tmp files (can't investigate as there is no time until it freezes) and the io wait goes to 100% on all cores. At this point it is impossible to login from remote and local keyboard and mouse are frozen. Hard reset is the only way out at this point.

IMHO there is something wrong in how the kernel hands out resources, in this case memory, CPU time and disk accesses. It should be easily measurable that a single application is allocating all memory and consequently all CPU time is spent in io wait. I'm pretty sure that if the kernel would stop or reduce allocating CPU time to the hogging (user) process causing excessive io wait, the machine would continue just fine. And probably a single process should not even be allowed to allocate this amount of memory. Or at least should not force other applications to get swapped out.

In effect gitk should have beeb forced to run slow or be killed off (which doesn't happen because probably the kernel itself doesn't receive enough CPU time).

As a positive side effect maybe someone would dig into gitk and make it less memory hungry (qgit is half as hungry, still a lot).

Signature · 1 line
Ferry Toth
#1 lore raw
Quoted 7 lines · Ferry Toth

It appears any ordinary user can easily create a DOS on linux.

One sure way to reproduce this is to open gitk on the linux kernel repo (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core
+ hyperthreading. But I will be easy enough to get the same effect with more RAM, other fs etc.

Quoted 6 lines

In this case gitk allocates more and more memory (until my system freezes 6.5GB of 7.5GB avaiable), the system starts swapping or writing to tmp files (can't investigate as there is no time until it freezes) and the io wait goes to 100% on all cores. At this point it is impossible to login from remote and local keyboard and mouse are frozen. Hard reset is the only way out at this point.

datapoint: my i4790/ext4 box running master.yesterday booted mem=8G became highly unpleasant to use, but I retained control, and the all cores going to 100% thing did not happen at any time.

I didn't try constraining on the gitk user, just turned it loose a few times to see if it managed to render box effectively dead. It failed to kill my box, but (expectedly) did make it suck rocks.

-Mike

author
#2 lore raw

Op vrijdag 20 april 2018 06:46:58 CEST schreef Mike Galbraith:

Quoted 19 lines · Ferry Toth
Quoted 7 lines

It appears any ordinary user can easily create a DOS on linux.

One sure way to reproduce this is to open gitk on the linux kernel repo (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core
+ hyperthreading. But I will be easy enough to get the same effect with more RAM, other fs etc.

Quoted 6 lines

In this case gitk allocates more and more memory (until my system freezes 6.5GB of 7.5GB avaiable), the system starts swapping or writing to tmp files (can't investigate as there is no time until it freezes) and the io wait goes to 100% on all cores. At this point it is impossible to login from remote and local keyboard and mouse are frozen. Hard reset is the only way out at this point.

datapoint: my i4790/ext4 box running master.yesterday booted mem=8G became highly unpleasant to use, but I retained control, and the all cores going to 100% thing did not happen at any time.

Quoted 6 lines

I didn't try constraining on the gitk user, just turned it loose a few times to see if it managed to render box effectively dead. It failed to kill my box, but (expectedly) did make it suck rocks.

-Mike

Yes, might be less dramatic with ext4 than with btrfs (COW icw fsync on hdd's destroys performance of things like virtualbox, databases, dpkg).

Nevertheless I feel one process should not be allowed to harm other processes by denying them resources. Even if when btrfs makes it easy abuse I think the scheduler should have throttled gitk.

#3 lore raw
Quoted 4 lines · Ferry Toth

Nevertheless I feel one process should not be allowed to harm other processes by denying them resources. Even if when btrfs makes it easy abuse I think the scheduler should have throttled gitk.

Memory management is not in it's job description.

-Mike

#4 lore raw
In reply to Ferry Toth
Quoted 6 lines · Ferry Toth

It appears any ordinary user can easily create a DOS on linux.

One sure way to reproduce this is to open gitk on the linux kernel repo (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core
+ hyperthreading. But I will be easy enough to get the same effect with more RAM, other fs etc.

You may want to disable swap.

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html
#5 lore raw
Quoted 10 lines · Pavel Machek
Quoted 6 lines · Ferry Toth

It appears any ordinary user can easily create a DOS on linux.

One sure way to reproduce this is to open gitk on the linux kernel repo (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core
+ hyperthreading. But I will be easy enough to get the same effect with more RAM, other fs etc.

You may want to disable swap.

I run without swap on my laptops, and still observe long periods of thrashing on the road towards OOM. What seems to occur is the active file-backed mappings of executables/libraries become a sort of swap area, repeatedly being discarded and faulted back in as the context switches occur.

If there's any good way to prevent this, I'd like to know.

Regards,
Vito Caputo

#6 lore raw
Quoted 12 lines · On Sun 22-04-18 10:43:00, [email protected] wrote:
Quoted 10 lines · Pavel Machek
Quoted 6 lines · Ferry Toth

It appears any ordinary user can easily create a DOS on linux.

One sure way to reproduce this is to open gitk on the linux kernel repo (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core
+ hyperthreading. But I will be easy enough to get the same effect with more RAM, other fs etc.

You may want to disable swap.

Quoted 7 lines

I run without swap on my laptops, and still observe long periods of thrashing on the road towards OOM. What seems to occur is the active file-backed mappings of executables/libraries become a sort of swap area, repeatedly being discarded and faulted back in as the context switches occur.

If there's any good way to prevent this, I'd like to know.

I am afraid there is none yet. Johannes had some ground work for
page cache trashing detection https://marc.info/?i=20170727153010.23347-1-hannes%40cmpxchg.org
but there was no version of the patchseries for quite some time and
there was no integration into the oom detection which would be
non-trivial as well.

I realize this sucks. But the reality is that this is far from trivial to resolve without introducing pre-mature OOM killer invocations.

Signature · 2 lines
Michal Hocko
SUSE Labs
#7 lore raw
Quoted 21 lines · Michal Hocko
Quoted 12 lines · On Sun 22-04-18 10:43:00, [email protected] wrote:
Quoted 10 lines · Pavel Machek
Quoted 6 lines · Ferry Toth

It appears any ordinary user can easily create a DOS on linux.

One sure way to reproduce this is to open gitk on the linux kernel repo (SIC) on a machine with 8GB RAM 16 GB swap on a HDD with btrfs and quad core
+ hyperthreading. But I will be easy enough to get the same effect with more RAM, other fs etc.

You may want to disable swap.

Quoted 7 lines

I run without swap on my laptops, and still observe long periods of thrashing on the road towards OOM. What seems to occur is the active file-backed mappings of executables/libraries become a sort of swap area, repeatedly being discarded and faulted back in as the context switches occur.

If there's any good way to prevent this, I'd like to know.

Quoted 8 lines

I am afraid there is none yet. Johannes had some ground work for
page cache trashing detection https://marc.info/?i=20170727153010.23347-1-hannes%40cmpxchg.org
but there was no version of the patchseries for quite some time and
there was no integration into the oom detection which would be
non-trivial as well.

I realize this sucks. But the reality is that this is far from trivial to resolve without introducing pre-mature OOM killer invocations.

Another problem is that what "unusable machine" in X/web browser situation may be normal load for build server...

I guess one way would be "hey, this is my X server; if it is waiting for disk for more than 10 seconds, you probably want to OOM kill someone. Ouch and same goes for my window manager".

Pavel

Signature · 2 lines
(english) http://www.livejournal.com/~pavelmachek
(cesky, pictures) http://atrey.karlin.mff.cuni.cz/~pavel/picture/horses/blog.html

A message that is not in the archive (sent off-list or not yet received): 4285098.DEWjdbWF2X@delfion

A message that is not in the archive (sent off-list or not yet received): [email protected]

A message that is not in the archive (sent off-list or not yet received): 6057755.ozdVOybsI6@delfion

#11 lore raw
Quoted 6 lines · Ferry Toth
Quoted 6 lines

Yes your memory hog scenario thoroughly wrecks the user experience, but the process scheduler in not the source of that wreckage, it's a memory management issue. With no constraints in place, anybody can just keep on allocating until the entire system starts grinding itself to dust.

-Mike

Quoted 8 lines


That is exactly the issue I think. It is not just a user experience, they is no distinction between crashing the kernel and grinding it to dust. The effect we have is: any user on a multi user system can crash the system.

Memory management / constraints can't solve the problem either: it might be intentional to alloc such amounts of memory.

Constraints definitely can solve this particular problem instance. Plain old ulimit can stop the memory hog (wish) dead in its tracks, or you can use memory cgroup controller to constrain it in a more friendly manner. I started gitk in a 4G constrained cgroup, which redirected its greedy fingers to the swap bin for the remainder of its needs.

But yes, there is currently no wonderful one size fits all fully automatic solution to running low on memory that doesn't involve running to the store.

Quoted 2 lines

I think memory allocation and io waits can't be decoupled from scheduling as they are now.

The scheduler is not decoupled from either, it is intimately involved in both. However, none of the decision making smarts for either reside in the scheduler, nor should they.

-Mike

#12 lore raw
Quoted 6 lines
Quoted 2 lines

I think memory allocation and io waits can't be decoupled from scheduling as they are now.

The scheduler is not decoupled from either, it is intimately involved in both. However, none of the decision making smarts for either reside in the scheduler, nor should they.

It belongs in both.

Classical Unix systems never had this problem because they respond to thrashing by ensuring that all processes consumed CPU and made some progress. Linux handles it by thrashing itself to dealth while BSD always handled it by moving from paging more towards swapping and behaving like a swap bound batch machine.

Linux thrashes itself to death, the classic BSD algorithn instead throws fairness out of the window under extreme load to prevent it. It might take a few seconds but at least you will get your prompt back.

Alan

#13 lore raw
Quoted 8 lines · Alan Cox
Quoted 6 lines
Quoted 2 lines

I think memory allocation and io waits can't be decoupled from scheduling as they are now.

The scheduler is not decoupled from either, it is intimately involved in both. However, none of the decision making smarts for either reside in the scheduler, nor should they.

It belongs in both.

If mm decision making belongs within the process scheduler, it follows that IO requests, dirty page writeback etc. do as well. Nope, I don't think we want to create a squid-uler, with tentacles extending all over the dang kernel.

The thrashing problem could use some attention, but we'll have to agree to disagree about the scheduler growing mm, io (etc) smarts.

-Mike

#14 lore raw
In reply to Alan Cox
Quoted 6 lines · Alan Cox

Classical Unix systems never had this problem because they respond to thrashing by ensuring that all processes consumed CPU and made some progress. Linux handles it by thrashing itself to dealth while BSD always handled it by moving from paging more towards swapping and behaving like a swap bound batch machine.

Memcg constrained the gitk hog nicely, forcing it do dig into swap after hitting its limit. Dunno if there are any userspace bits that can use it wisely though.

-Mike

author
#15 lore raw
In reply to Alan Cox

Op woensdag 25 april 2018 16:54:59 CEST schreef Alan Cox:

Quoted 15 lines
Quoted 6 lines
Quoted 2 lines

I think memory allocation and io waits can't be decoupled from scheduling as they are now.

The scheduler is not decoupled from either, it is intimately involved in both. However, none of the decision making smarts for either reside in the scheduler, nor should they.

It belongs in both.

Classical Unix systems never had this problem because they respond to thrashing by ensuring that all processes consumed CPU and made some progress. Linux handles it by thrashing itself to dealth while BSD always handled it by moving from paging more towards swapping and behaving like a swap bound batch machine.

Quoted 6 lines

Linux thrashes itself to death, the classic BSD algorithn instead throws fairness out of the window under extreme load to prevent it. It might take a few seconds but at least you will get your prompt back.

Alan

I havent tried BSD.

But when I was young I allocated 10MB on a HP9000 (UX) with 1MB of RAM. People wanted to launch me out of the window (18th floor).

I did not want to say Unix was better, only with so much emphasis on security I'm surprised how easy it is for a regular user to bring linux to on its knees.

#16 lore raw
Quoted 20 lines · Ferry Toth

Op woensdag 25 april 2018 16:54:59 CEST schreef Alan Cox:

Quoted 19 lines
Quoted 6 lines
Quoted 2 lines

I think memory allocation and io waits can't be decoupled from scheduling as they are now.

The scheduler is not decoupled from either, it is intimately involved in both. However, none of the decision making smarts for either reside in the scheduler, nor should they.

It belongs in both.

Classical Unix systems never had this problem because they respond to thrashing by ensuring that all processes consumed CPU and made some progress. Linux handles it by thrashing itself to dealth while BSD always handled it by moving from paging more towards swapping and behaving like a swap bound batch machine.

Linux thrashes itself to death, the classic BSD algorithn instead throws fairness out of the window under extreme load to prevent it. It might take a few seconds but at least you will get your prompt back.

Quoted 7 lines
Quoted 2 lines

Alan

I haven t tried BSD.

But when I was young I allocated 10MB on a HP9000 (UX) with 1MB of RAM. People wanted to launch me out of the window (18th floor).

I did not want to say Unix was better, only with so much emphasis on security I' m surprised how easy it is for a regular user to bring linux to on it s knees.

While it is true that things can be improved/tweaked for typical desktop/single user usage; this isn't really a security issue. For shared systems, there are a few ways to soft/hard limit resources:
nice, *limit, cgroups, systemd limits, containers/VMs...

Cheers,
Miguel