From: Vedran Furač on
David Rientjes wrote:

> Right, because in Vedran's latest oom log it shows that Xorg is preferred
> more than any other thread other than the memory hogging test program with
> your patch than without. I pointed out a clear distinction in the killing
> order using both total_vm and rss in that log and in my opinion killing
> Xorg as opposed to krunner would be undesireable.

But then you should rename OOM killer to TRIPK:
Totally Random Innocent Process Killer

If you have OOM situation and Xorg is the first, that means it's leaking
memory badly and the system is probably already frozen/FUBAR. Killing
krunner in that situation wouldn't do any good. From a user perspective,
nothing changes, system is still FUBAR and (s)he would probably reboot
cursing linux in the process.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: David Rientjes on
On Thu, 29 Oct 2009, Vedran Furac wrote:

> [ 1493.064458] Out of memory: kill process 6304 (kdeinit4) score 1190231
> or a child
> [ 1493.064467] Killed process 6409 (konqueror)
> [ 1493.261149] knotify4 invoked oom-killer: gfp_mask=0x201da, order=0,
> oomkilladj=0
> [ 1493.261166] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1493.276528] Out of memory: kill process 6304 (kdeinit4) score 1161265
> or a child
> [ 1493.276538] Killed process 6411 (krusader)
> [ 1499.221160] akregator invoked oom-killer: gfp_mask=0x201da, order=0,
> oomkilladj=0
> [ 1499.221178] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1499.236431] Out of memory: kill process 6304 (kdeinit4) score 1067593
> or a child
> [ 1499.236441] Killed process 6412 (irexec)
> [ 1499.370192] firefox-bin invoked oom-killer: gfp_mask=0x201da,
> order=0, oomkilladj=0
> [ 1499.370209] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1499.385417] Out of memory: kill process 6304 (kdeinit4) score 1066861
> or a child
> [ 1499.385427] Killed process 6420 (xchm)
> [ 1499.458304] kio_file invoked oom-killer: gfp_mask=0x201da, order=0,
> oomkilladj=0
> [ 1499.458333] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1499.458367] [<ffffffff81120900>] ? d_kill+0x5c/0x7c
> [ 1499.473573] Out of memory: kill process 6304 (kdeinit4) score 1043690
> or a child
> [ 1499.473582] Killed process 6425 (kio_file)
> [ 1500.250746] korgac invoked oom-killer: gfp_mask=0x201da, order=0,
> oomkilladj=0
> [ 1500.250765] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1500.266186] Out of memory: kill process 6304 (kdeinit4) score 1020350
> or a child
> [ 1500.266196] Killed process 6464 (icedove)
> [ 1500.349355] syslog-ng invoked oom-killer: gfp_mask=0x201da, order=0,
> oomkilladj=0
> [ 1500.349371] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1500.364689] Out of memory: kill process 6304 (kdeinit4) score 1019864
> or a child
> [ 1500.364699] Killed process 6477 (kio_http)
> [ 1500.452151] kded4 invoked oom-killer: gfp_mask=0x201da, order=0,
> oomkilladj=0
> [ 1500.452167] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1500.452196] [<ffffffff81120900>] ? d_kill+0x5c/0x7c
> [ 1500.467307] Out of memory: kill process 6304 (kdeinit4) score 993142
> or a child
> [ 1500.467316] Killed process 6478 (kio_http)
> [ 1500.780222] akregator invoked oom-killer: gfp_mask=0x201da, order=0,
> oomkilladj=0
> [ 1500.780239] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1500.796280] Out of memory: kill process 6304 (kdeinit4) score 966331
> or a child
> [ 1500.796290] Killed process 6484 (kio_http)
> [ 1501.065374] syslog-ng invoked oom-killer: gfp_mask=0x201da, order=0,
> oomkilladj=0
> [ 1501.065390] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1501.080579] Out of memory: kill process 6304 (kdeinit4) score 939434
> or a child
> [ 1501.080587] Killed process 6486 (kio_http)
> [ 1501.381188] knotify4 invoked oom-killer: gfp_mask=0x201da, order=0,
> oomkilladj=0
> [ 1501.381204] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1501.396338] Out of memory: kill process 6304 (kdeinit4) score 912691
> or a child
> [ 1501.396346] Killed process 6487 (firefox-bin)
> [ 1502.661294] icedove-bin invoked oom-killer: gfp_mask=0x201da,
> order=0, oomkilladj=0
> [ 1502.661311] [<ffffffff810d6dd7>] ? oom_kill_process+0x9a/0x264
> [ 1502.676563] Out of memory: kill process 7580 (test) score 708945 or a
> child
> [ 1502.676575] Killed process 7580 (test)
>

Ok, so this is the forkbomb problem by adding half of each child's
total_vm into the badness score of the parent. We should address this
completely seperately by addressing that specific part of the heuristic,
not changing what we consider to be a baseline.

The rationale is quite simple: we'll still experience the same problem
with rss as we did with total_vm in the forkbomb scenario above on certain
workloads (maybe not yours, but others). The oom killer always kills a
child first if it has a different mm than the selected parent, so the
amount of memory freeing as a result of that is entirely dependent on the
order of the child list. It may be very little, but killed because its
siblings had large total_vm values.

So instead of focusing on rss, we simply need to find a better heuristic
for the forkbomb issue which I've already proposed a very trivial solution
for. Then, afterwards, we can debate about how the scoring heuristic can
be changed to select better tasks (and perhaps remove a lot of the clutter
that's there currently!).

> > Can you explain why Xorg is preferred as a baseline to kill rather than
> > krunner in your example?
>
> Krunner is a small app for running other apps and do similar things. It
> shouldn't use a lot of memory. OTOH, Xorg has to hold all the pixmaps
> and so on. That was expected result. Fist Xorg, then firefox and
> thunderbird.
>

You're making all these claims and assertions based _solely_ on the theory
that killing the application with the most resident RAM is always the
optimal solution. That's just not true, especially if we're just
allocating small numbers of order-0 memory.

Much better is to allow the user to decide at what point, regardless of
swap usage, their application is using much more memory than expected or
required. They can do that right now pretty well with /proc/pid/oom_adj
without this outlandish claim that they should be expected to know the rss
of their applications at the time of oom to effectively tune oom_adj.

What would you suggest? A script that sits in a loop checking each task's
current rss from /proc/pid/stat or their current oom priority though
/proc/pid/oom_score and adjusting oom_adj preemptively just in case the
oom killer is invoked in the next second?

And that "small app" has 30MB of rss which could be freed, if killed, and
utilized for subsequent page allocations.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: David Rientjes on
On Thu, 29 Oct 2009, Vedran Furac wrote:

> But then you should rename OOM killer to TRIPK:
> Totally Random Innocent Process Killer
>

The randomness here is the order of the child list when the oom killer
selects a task, based on the badness score, and then tries to kill a child
with a different mm before the parent.

The problem you identified in http://pastebin.com/f3f9674a0, however, is a
forkbomb issue where the badness score should never have been so high for
kdeinit4 compared to "test". That's directly proportional to adding the
scores of all disjoint child total_vm values into the badness score for
the parent and then killing the children instead.

That's the problem, not using total_vm as a baseline. Replacing that with
rss is not going to solve the issue and reducing the user's ability to
specify a rough oom priority from userspace is simply not an option.

> If you have OOM situation and Xorg is the first, that means it's leaking
> memory badly and the system is probably already frozen/FUBAR. Killing
> krunner in that situation wouldn't do any good. From a user perspective,
> nothing changes, system is still FUBAR and (s)he would probably reboot
> cursing linux in the process.
>

It depends on what you're running, we need to be able to have the option
of protecting very large tasks on production servers. Imagine if "test"
here is actually a critical application that we need to protect, its
not solely mlocked anonymous memory, but still kill if it is leaking
memory beyond your approximate 2.5GB. How do you do that when using rss
as the baseline?
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: KAMEZAWA Hiroyuki on
On Thu, 29 Oct 2009 12:53:42 -0700 (PDT)
David Rientjes <rientjes(a)google.com> wrote:

> > If you have OOM situation and Xorg is the first, that means it's leaking
> > memory badly and the system is probably already frozen/FUBAR. Killing
> > krunner in that situation wouldn't do any good. From a user perspective,
> > nothing changes, system is still FUBAR and (s)he would probably reboot
> > cursing linux in the process.
> >
>
> It depends on what you're running, we need to be able to have the option
> of protecting very large tasks on production servers. Imagine if "test"
> here is actually a critical application that we need to protect, its
> not solely mlocked anonymous memory, but still kill if it is leaking
> memory beyond your approximate 2.5GB. How do you do that when using rss
> as the baseline?

As I wrote repeatedly,

- OOM-Killer itselfs is bad thing, bad situation.
- The kernel can't know the program is bad or not. just guess it.
- Then, there is no "correct" OOM-Killer other than fork-bomb killer.
- User has a knob as oom_adj. This is very strong.

Then, there is only "reasonable" or "easy-to-understand" OOM-Kill.
"Current biggest memory eater is killed" sounds reasonable, easy to
understand. And if total_vm works well, overcommit_guess should catch it.
Please improve overcommit_guess if you want to stay on total_vm.


Thanks,
-Kame

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: David Rientjes on
On Fri, 30 Oct 2009, KAMEZAWA Hiroyuki wrote:

> As I wrote repeatedly,
>
> - OOM-Killer itselfs is bad thing, bad situation.

Not necessarily, the memory controller and cpusets uses it quite often to
enforce it's policy and is standard runtime behavior. We'd like to
imagine that our cpuset will never be too small to run all the attached
jobs, but that happens and we can easily recover from it by killing a
task.

> - The kernel can't know the program is bad or not. just guess it.

Totally irrelevant, given your fourth point about /proc/pid/oom_adj. We
can tell the kernel what we'd like the oom killer behavior should be if
the situation arises.

> - Then, there is no "correct" OOM-Killer other than fork-bomb killer.

Well of course there is, you're seeing this is a WAY too simplistic
manner. If we are oom, we want to be able to influence how the oom killer
behaves and respond to that situation. You are proposing that we change
the baseline for how the oom killer selects tasks which we use CONSTANTLY
as part of our normal production environment. I'd appreciate it if you'd
take it a little more seriously.

> - User has a knob as oom_adj. This is very strong.
>

Agreed.

> Then, there is only "reasonable" or "easy-to-understand" OOM-Kill.
> "Current biggest memory eater is killed" sounds reasonable, easy to
> understand. And if total_vm works well, overcommit_guess should catch it.
> Please improve overcommit_guess if you want to stay on total_vm.
>

I don't necessarily want to stay on total_vm, but I also don't want to
move to rss as a baseline, as you would probably agree.

We disagree about a very fundamental principle: you are coming from a
perspective of always wanting to kill the biggest resident memory eater
even for a single order-0 allocation that fails and I'm coming from a
perspective of wanting to ensure that our machines know how the oom killer
will react when it is used. Moving to rss reduces the ability of the user
to specify an expected oom priority other than polarizing it by either
disabling it completely with an oom_adj value of -17 or choosing the
definite next victim with +15. That's my objection to it: the user cannot
possibly be expected to predict what proportion of each application's
memory will be resident at the time of oom.

I understand you want to totally rewrite the oom killer for whatever
reason, but I think you need to spend a lot more time understanding the
needs that the Linux community has for its behavior instead of insisting
on your point of view.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/