Call cond_resched() at bottom of main look in balance_pgdat()
in [Kernel]
|
Prev: oom: unify CAP_SYS_RAWIO check into other superuser check
Next: [PATCHv7 09/11] USB: gadget: g_multi: added documentation and INF files
From: KOSAKI Motohiro on 21 Jun 2010 07:50 > We are seeing a problem where kswapd gets stuck and hogs the CPU on a > small single CPU system when an OOM kill should occur. When this > happens swap space has been exhausted and the pagecache has been shrunk > to zero. Once kswapd gets the CPU it never gives it up because at least > one zone is below high. Adding a single cond_resched() at the end of > the main loop in balance_pgdat() fixes the problem by allowing the > watchdog and tasks to run and eventually do an OOM kill which frees up > the resources. Thank you. this seems regression. > Mem-Info: > DMA per-cpu: > CPU 0: hi: 0, btch: 1 usd: 0 > Normal per-cpu: > CPU 0: hi: 186, btch: 31 usd: 152 > active_anon:54902 inactive_anon:54849 isolated_anon:32 > active_file:0 inactive_file:25 isolated_file:0 > unevictable:660 dirty:0 writeback:6 unstable:0 > free:1172 slab_reclaimable:1969 slab_unreclaimable:8322 > mapped:196 shmem:801 pagetables:1300 bounce:0 > ... > Normal free:2672kB min:2764kB low:3452kB high:4144kB > ... > 21729 total pagecache pages > 20240 pages in swap cache > Swap cache stats: add 468211, delete 447971, find 12560445/12560936 > Free swap = 0kB > Total swap = 1015800kB > 128720 pages RAM > 0 pages HighMem > 3223 pages reserved > 1206 pages shared > 121413 pages non-shared > zero free swap. then, vmscan don't try to scan anon pages. but file pages are almost zero. then, shrink_page_list() was not called enough frequently.... I guess it is caused following commit (by me). commit bb3ab596832b920c703d1aea1ce76d69c0f71fb7 Author: KOSAKI Motohiro <kosaki.motohiro(a)jp.fujitsu.com> Date: Mon Dec 14 17:58:55 2009 -0800 vmscan: stop kswapd waiting on congestion when the min watermark is not being met Very thanks your effort. your analysis seems perfect. btw, I reformat your patch a bit. your previous email is a bit akpm unfriendly. ============================================================= Subject: [PATCH] Call cond_resched() at bottom of main look in balance_pgdat() From: Larry Woodman <lwoodman(a)redhat.com> We are seeing a problem where kswapd gets stuck and hogs the CPU on a small single CPU system when an OOM kill should occur. When this happens swap space has been exhausted and the pagecache has been shrunk to zero. Once kswapd gets the CPU it never gives it up because at least one zone is below high. Adding a single cond_resched() at the end of the main loop in balance_pgdat() fixes the problem by allowing the watchdog and tasks to run and eventually do an OOM kill which frees up the resources. kosaki note: This seems regression caused by commit bb3ab59683 (vmscan: stop kswapd waiting on congestion when the min watermark is not being met) Signed-off-by: Larry Woodman <lwoodman(a)redhat.com> Reviewed-by: KOSAKI Motohiro <kosaki.motohiro(a)jp.fujitsu.com> --- mm/vmscan.c | 1 + 1 files changed, 1 insertions(+), 0 deletions(-) diff --git a/mm/vmscan.c b/mm/vmscan.c index 9c7e57c..c5c46b7 100644 --- a/mm/vmscan.c +++ b/mm/vmscan.c @@ -2182,6 +2182,7 @@ loop_again: */ if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX) break; + cond_resched(); } out: /* -- 1.6.5.2 -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: KOSAKI Motohiro on 21 Jun 2010 22:30 > > ============================================================= > > Subject: [PATCH] Call cond_resched() at bottom of main look in balance_pgdat() > > From: Larry Woodman <lwoodman(a)redhat.com> > > > > We are seeing a problem where kswapd gets stuck and hogs the CPU on a > > small single CPU system when an OOM kill should occur. When this > > happens swap space has been exhausted and the pagecache has been shrunk > > to zero. Once kswapd gets the CPU it never gives it up because at least > > one zone is below high. Adding a single cond_resched() at the end of > > the main loop in balance_pgdat() fixes the problem by allowing the > > watchdog and tasks to run and eventually do an OOM kill which frees up > > the resources. > > > > kosaki note: This seems regression caused by commit bb3ab59683 > > (vmscan: stop kswapd waiting on congestion when the min watermark is > > not being met) > > > > Signed-off-by: Larry Woodman <lwoodman(a)redhat.com> > > Reviewed-by: KOSAKI Motohiro <kosaki.motohiro(a)jp.fujitsu.com> > > --- > > mm/vmscan.c | 1 + > > 1 files changed, 1 insertions(+), 0 deletions(-) > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index 9c7e57c..c5c46b7 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -2182,6 +2182,7 @@ loop_again: > > */ > > if (sc.nr_reclaimed >= SWAP_CLUSTER_MAX) > > break; > > + cond_resched(); > > } > > out: > > /* > > -- > > 1.6.5.2 > > Kosaki's patch's goal is that kswap doesn't yield cpu if the zone doesn't meet its > min watermark to avoid failing atomic allocation. > But this patch could yield kswapd's time slice at any time. > Doesn't the patch break your goal in bb3ab59683? No. it don't break. Typically, kswapd periodically call shrink_page_list() and it call cond_resched() even if bb3ab59683 case. Larry observed very exceptional situation. his system don't have reclaimable pages at all, then eventually shrink_page_list() was not called very long time. His patch only change such very rare situation, I think it's safe. Thanks. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Rik van Riel on 22 Jun 2010 14:30 On 06/21/2010 07:45 AM, KOSAKI Motohiro wrote: > kosaki note: This seems regression caused by commit bb3ab59683 > (vmscan: stop kswapd waiting on congestion when the min watermark is > not being met) > > Signed-off-by: Larry Woodman<lwoodman(a)redhat.com> > Reviewed-by: KOSAKI Motohiro<kosaki.motohiro(a)jp.fujitsu.com> Reviewed-by: Rik van Riel <riel(a)redhat.com> -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Johannes Weiner on 22 Jun 2010 17:40
On Tue, Jun 22, 2010 at 01:29:17PM +0900, Minchan Kim wrote: > On Tue, Jun 22, 2010 at 12:23 PM, KOSAKI Motohiro > <kosaki.motohiro(a)jp.fujitsu.com> wrote: > >> >> Kosaki's patch's goal is that kswap doesn't yield cpu if the zone doesn't meet its > >> >> min watermark to avoid failing atomic allocation. > >> >> But this patch could yield kswapd's time slice at any time. > >> >> Doesn't the patch break your goal in bb3ab59683? > >> > > >> > No. it don't break. > >> > > >> > Typically, kswapd periodically call shrink_page_list() and it call > >> > cond_resched() even if bb3ab59683 case. > >> > >> Hmm. If it is, bb3ab59683 is effective really? > >> > >> The bb3ab59683's goal is prevent CPU yield in case of free < min_watermark. > >> But shrink_page_list can yield cpu from kswapd at any time. > >> So I am not sure what is bb3ab59683's benefit. > >> Did you have any number about bb3ab59683's effectiveness? > >> (Of course, I know it's very hard. Just out of curiosity) > >> > >> As a matter of fact, when I saw this Larry's patch, I thought it would > >> be better to revert bb3ab59683. Then congestion_wait could yield CPU > >> to other process. > >> > >> What do you think about? > > > > No. The goal is not prevent CPU yield. The goal is avoid unnecessary > > _long_ sleep (i.e. congestion_wait(BLK_RW_ASYNC, HZ/10)). > > I meant it. > > > Anyway we can't refuse CPU yield on UP. it lead to hangup ;) > > > > What do you mean the number? If it mean how much reduce congestion_wait(), > > it was posted a lot of time. If it mean how much reduce page allocation > > failure bug report, I think it has been observable reduced since half > > years ago. > > I meant second. > Hmm. I doubt it's observable since at that time, Mel had posted many > patches to reduce page allocation fail. bb3ab59683 was just one of > them. > > > > > If you have specific worried concern, can you please share it? > > > > My concern is that I don't want to add new band-aid on uncertain > feature to solve > regression of uncertain feature.(Sorry for calling Larry's patch as band-aid.). > If we revert bb3ab59683, congestion_wait in balance_pgdat could yield > cpu from kswapd. > > If you insist on bb3ab59683's effective and have proved it at past, I > am not against it. > > And If it's regression of bb3ab59683, Doesn't it make sense following as? > It could restore old behavior. > > --- > * OK, kswapd is getting into trouble. Take a nap, then take > * another pass across the zones. > */ > if (total_scanned && (priority < DEF_PRIORITY - 2)) { > if (has_under_min_watermark_zone) { > count_vm_event(KSWAPD_SKIP_CONGESTION_WAIT); > /* allowing CPU yield to go on > watchdog or OOMed task */ > cond_resched(); We have two things here: one is waiting for some IO to complete, which we skip if we are in a hurry. The other thing is that we have a potentially long-running loop with no garuanteed rescheduling point in it. I would rather not mix up those two and let this cond_resched() for #2 stand on it's own and be self-explanatory. So, Acked-by: Johannes Weiner <hannes(a)cmpxchg.org> to Larry's patch (or KOSAKI-san's version of it for that matter). -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |