fs: limit maximum concurrent coredumps
in [Kernel]
|
Prev: [PATCH 2/5] isdn/gigaset: correct CAPI voice connection encoding
Next: [PATCH net-next 2/2] cxgb3: request 7.10 firmware
From: Roland McGrath on 21 Jun 2010 21:30 A core dump is just an instance of a process suddenly reading lots of its address space and doing lots of filesystem writes, producing the kinds of thrashing that any such instance might entail. It really seems like the real solution to this kind of problem will be in some more general kind of throttling of processes (or whatever manner of collections thereof) when they got hog-wild on page-ins or filesystem writes, or whatever else. I'm not trying to get into the details of what that would be. But I have to cite this hack as the off-topic kludge that it really is. That said, I do certainly sympathize with the desire for a quick hack that addresses the scenario you experience. For the case you described, it seems to me that constraining concurrency per se would be better than punting core dumps when too concurrent. That is, you should not skip the dump when you hit the limit. Rather, you should block in do_coredump() until the next dump already in progress finishes. (It should be possible to use TASK_KILLABLE so that those dumps in waiting can be aborted with a follow-on SIGKILL. But Oleg will have to check on the signals details being right for that.) That won't make your crashers each complete quickly, but it will prevent the thrashing. Instead of some crashers suddenly not producing dumps at all, they'll just all queue up waiting to finish crashing but not using any CPU or IO resources. That way you don't lose any core dumps unless you want to start SIGKILL'ing things (which oom_kill might do if need be), you just don't die in flames trying to do nothing but dump cores. Thanks, Roland -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andrew Morton on 21 Jun 2010 21:50 On Mon, 21 Jun 2010 18:23:03 -0700 (PDT) Roland McGrath <roland(a)redhat.com> wrote: > A core dump is just an instance of a process suddenly reading lots of its > address space and doing lots of filesystem writes, producing the kinds of > thrashing that any such instance might entail. It really seems like the > real solution to this kind of problem will be in some more general kind of > throttling of processes (or whatever manner of collections thereof) when > they got hog-wild on page-ins or filesystem writes, or whatever else. I'm > not trying to get into the details of what that would be. But I have to > cite this hack as the off-topic kludge that it really is. That said, I do > certainly sympathize with the desire for a quick hack that addresses the > scenario you experience. yup. > For the case you described, it seems to me that constraining concurrency > per se would be better than punting core dumps when too concurrent. That > is, you should not skip the dump when you hit the limit. Rather, you > should block in do_coredump() until the next dump already in progress > finishes. (It should be possible to use TASK_KILLABLE so that those dumps > in waiting can be aborted with a follow-on SIGKILL. But Oleg will have to > check on the signals details being right for that.) yup. Might be able to use semaphores for this. Use sema_init(), down_killable() and up(). Modifying the max concurrency value would require a loop of up()s and down()s, probably all surrounded by a mutex_lock. Which is a bit ugly, and should be done in kernel/semaphore.c I guess. > That won't make your crashers each complete quickly, but it will prevent > the thrashing. Instead of some crashers suddenly not producing dumps at > all, they'll just all queue up waiting to finish crashing but not using any > CPU or IO resources. That way you don't lose any core dumps unless you > want to start SIGKILL'ing things (which oom_kill might do if need be), > you just don't die in flames trying to do nothing but dump cores. A global knob is a bit old-school. Perhaps it should be a per-memcg knob or something. otoh, one could perhaps toss all these tasks into a blkio_cgroup and solve this problem with the block IO controller. After all, that's what it's for. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: KOSAKI Motohiro on 21 Jun 2010 22:20 > A moderately large number of large core files being dumped simultaneously > can impose severe latency penalties on other processes due to IO load. > > For example, a common configuration for PHP web-servers includes apache's > prefork MPM, mod_php and a PHP opcode cache utilizing shared memory. In > certain failure modes, all requests serviced by PHP result in a segfault. > Enabling coredumps might lead to 10-20 coredumps per second, all attempting > to write a 150-200MB core file. This leads to the whole system becoming > entirely unresponsive for many minutes. > > The ability to limit concurrent coredumps allows dumping core to be safely > enabled in these situations without affecting responsiveness of the system > as a whole. > > I have several servers running with this patch applied (actually backported > to v2.6.26) and it has allowed me to deal successfully with the situation > described above. > > In do_coredump I have pulled dump_count back out to the top-level scope > and core_dump_count is now incremented in the normal path. > > Added sysctl parameter for tuning core_max_concurrency. > > checkpatch complains about "extern ... core_max_concurrency" but this seems > to be an acceptable exception based on the preponderance of other extern > variables in kernel/sysctl.c To be honest, My customers have very similar problem for long time. If HPC MPI job or apache prefork MPM processces die, The system load to make large concurrent core dump I/O and consume all I/O band width. Eventually, all other service stall long time and HA cluster software shutdown such node forcely. My case used two technique, 1) limit concurrent coredump (but not skip as sugessted Roland) 2) reduce I/O priority automatically in do_coredump(). I don't know my way was correct way. but I believe we need something case. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: H. Peter Anvin on 22 Jun 2010 00:00 On 06/21/2010 06:23 PM, Roland McGrath wrote: > A core dump is just an instance of a process suddenly reading lots of its > address space and doing lots of filesystem writes, producing the kinds of > thrashing that any such instance might entail. It is, although with one possibly important difference: the process has had an involuntary state transition, which may mean that its priority settings that it had as a "live" process are no longer applicable. It would certainly seem appropriate to give the administrator the option of altering the priority parameters of coredumping processes. -hpa -- H. Peter Anvin, Intel Open Source Technology Center I work for Intel. I don't speak on their behalf. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Jiri Kosina on 22 Jun 2010 04:50
On Mon, 21 Jun 2010, Roland McGrath wrote: > A core dump is just an instance of a process suddenly reading lots of its > address space and doing lots of filesystem writes, producing the kinds of > thrashing that any such instance might entail. But priority settings don't apply any more for core dumping process, do they? -- Jiri Kosina SUSE Labs, Novell Inc. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |