s2disk hang update
in [Kernel]
|
Prev: trivial: capitalization and other minor fixes in pnp documentation
Next: nfs: clear_commit_release incorrectly handle truncated page
From: Rafael J. Wysocki on 22 Feb 2010 14:20 On Monday 22 February 2010, Alan Jenkins wrote: > Rafael J. Wysocki wrote: > > On Friday 19 February 2010, Alan Jenkins wrote: > > > >> On 2/18/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > >> > >>> On Thursday 18 February 2010, Alan Jenkins wrote: > >>> > >>>> On 2/17/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > >>>> > >>>>> On Wednesday 17 February 2010, Alan Jenkins wrote: > >>>>> > >>>>>> On 2/16/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > >>>>>> > >>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote: > >>>>>>> > >>>>>>>> On 2/16/10, Alan Jenkins <sourcejedi.lkml(a)googlemail.com> wrote: > >>>>>>>> > >>>>>>>>> On 2/15/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > >>>>>>>>> > >>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote: > >>>>>>>>>> > >>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too many > >>>>>>>>>>> applications. The first hibernation fails with "not enough > >>>>>>>>>>> swap" > >>>>>>>>>>> as > >>>>>>>>>>> expected, but the second or third attempt hangs (with the same > >>>>>>>>>>> backtrace > >>>>>>>>>>> as before). > >>>>>>>>>>> > >>>>>>>>>>> The patch definitely helps though. Without the patch, I see a > >>>>>>>>>>> hang > >>>>>>>>>>> the > >>>>>>>>>>> first time I try to hibernate with too many applications > >>>>>>>>>>> running. > >>>>>>>>>>> > >>>>>>>>>> Well, I have an idea. > >>>>>>>>>> > >>>>>>>>>> Can you try to apply the appended patch in addition and see if > >>>>>>>>>> that > >>>>>>>>>> helps? > >>>>>>>>>> > >>>>>>>>>> Rafael > >>>>>>>>>> > >>>>>>>>> It doesn't seem to help. > >>>>>>>>> > >>>>>>>> To be clear: It doesn't stop the hang when I hibernate with too many > >>>>>>>> applications. > >>>>>>>> > >>>>>>>> It does stop the same hang in a different case though. > >>>>>>>> > >>>>>>>> 1. boot with init=/bin/bash > >>>>>>>> 2. run s2disk > >>>>>>>> 3. cancel the s2disk > >>>>>>>> 4. repeat steps 2&3 > >>>>>>>> > >>>>>>>> With the patch, I can run 10s of iterations, with no hang. > >>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as > >>>>>>>> always). > >>>>>>>> > >>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no problem. > >>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an > >>>>>>>> allocation > >>>>>>>> failure after a couple of iterations ("kthreadd: page allocation > >>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the same > >>>>>>>> stop_machine thread allocation failure that causes the hang. > >>>>>>>> > >>>>>>> Have you tested it alone or on top of the previous one? If you've > >>>>>>> tested it > >>>>>>> alone, please apply the appended one in addition to it and retest. > >>>>>>> > >>>>>>> Rafael > >>>>>>> > >>>>>> I did test with both patches applied together - > >>>>>> > >>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and > >>>>>> resume > >>>>>> 2. "reducing the number of pages that we're going to keep preallocated > >>>>>> by > >>>>>> 20%" > >>>>>> > >>>>> In that case you can try to reduce the number of preallocated pages even > >>>>> more, > >>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch. > >>>>> > >>>> It still hangs if I try to hibernate a couple of times with too many > >>>> applications. > >>>> > >>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit system and > >>> how much RAM is there in the box? > >>> > >>> Rafael > >>> > >> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD. > >> > > > > Hmm. I'd try to make free_unnecessary_pages() free all of the preallocated > > pages and see what happens. > > > > It still hangs in hibernation_snapshot() / disable_nonboot_cpus(). > After apparently freeing over 400Mb / 100,000 pages of preallocated ram. > > > > There is a change which I missed before. When I applied your first > patch ("Force GFP_NOIO during suspend" etc.), it did change the hung > task backtraces a bit. I don't know if it tells us anything. > > Without the patch, there were two backtraces. The first backtrace > suggested a problem allocating pages for a kernel thread (at > copy_process() / try_to_free_pages()). The second showed that this > problem was blocking s2disk (at hibernation_snapshot() / > disable_nonboot_cpus() / stop_machine_create()). > > With the GFP_NOIO patch, I see only the s2disk backtrace. Can you please post this backtrace? Rafael -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Alan Jenkins on 23 Feb 2010 09:30 On 2/22/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > On Monday 22 February 2010, Alan Jenkins wrote: >> Rafael J. Wysocki wrote: >> > On Friday 19 February 2010, Alan Jenkins wrote: >> > >> >> On 2/18/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: >> >> >> >>> On Thursday 18 February 2010, Alan Jenkins wrote: >> >>> >> >>>> On 2/17/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: >> >>>> >> >>>>> On Wednesday 17 February 2010, Alan Jenkins wrote: >> >>>>> >> >>>>>> On 2/16/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: >> >>>>>> >> >>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote: >> >>>>>>> >> >>>>>>>> On 2/16/10, Alan Jenkins <sourcejedi.lkml(a)googlemail.com> wrote: >> >>>>>>>> >> >>>>>>>>> On 2/15/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: >> >>>>>>>>> >> >>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote: >> >>>>>>>>>> >> >>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too >> >>>>>>>>>>> many >> >>>>>>>>>>> applications. The first hibernation fails with "not enough >> >>>>>>>>>>> swap" >> >>>>>>>>>>> as >> >>>>>>>>>>> expected, but the second or third attempt hangs (with the same >> >>>>>>>>>>> backtrace >> >>>>>>>>>>> as before). >> >>>>>>>>>>> >> >>>>>>>>>>> The patch definitely helps though. Without the patch, I see a >> >>>>>>>>>>> hang >> >>>>>>>>>>> the >> >>>>>>>>>>> first time I try to hibernate with too many applications >> >>>>>>>>>>> running. >> >>>>>>>>>>> >> >>>>>>>>>> Well, I have an idea. >> >>>>>>>>>> >> >>>>>>>>>> Can you try to apply the appended patch in addition and see if >> >>>>>>>>>> that >> >>>>>>>>>> helps? >> >>>>>>>>>> >> >>>>>>>>>> Rafael >> >>>>>>>>>> >> >>>>>>>>> It doesn't seem to help. >> >>>>>>>>> >> >>>>>>>> To be clear: It doesn't stop the hang when I hibernate with too >> >>>>>>>> many >> >>>>>>>> applications. >> >>>>>>>> >> >>>>>>>> It does stop the same hang in a different case though. >> >>>>>>>> >> >>>>>>>> 1. boot with init=/bin/bash >> >>>>>>>> 2. run s2disk >> >>>>>>>> 3. cancel the s2disk >> >>>>>>>> 4. repeat steps 2&3 >> >>>>>>>> >> >>>>>>>> With the patch, I can run 10s of iterations, with no hang. >> >>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as >> >>>>>>>> always). >> >>>>>>>> >> >>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no >> >>>>>>>> problem. >> >>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an >> >>>>>>>> allocation >> >>>>>>>> failure after a couple of iterations ("kthreadd: page allocation >> >>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the >> >>>>>>>> same >> >>>>>>>> stop_machine thread allocation failure that causes the hang. >> >>>>>>>> >> >>>>>>> Have you tested it alone or on top of the previous one? If you've >> >>>>>>> tested it >> >>>>>>> alone, please apply the appended one in addition to it and retest. >> >>>>>>> >> >>>>>>> Rafael >> >>>>>>> >> >>>>>> I did test with both patches applied together - >> >>>>>> >> >>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and >> >>>>>> resume >> >>>>>> 2. "reducing the number of pages that we're going to keep >> >>>>>> preallocated >> >>>>>> by >> >>>>>> 20%" >> >>>>>> >> >>>>> In that case you can try to reduce the number of preallocated pages >> >>>>> even >> >>>>> more, >> >>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch. >> >>>>> >> >>>> It still hangs if I try to hibernate a couple of times with too many >> >>>> applications. >> >>>> >> >>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit >> >>> system and >> >>> how much RAM is there in the box? >> >>> >> >>> Rafael >> >>> >> >> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD. >> >> >> > >> > Hmm. I'd try to make free_unnecessary_pages() free all of the >> > preallocated >> > pages and see what happens. >> > >> >> It still hangs in hibernation_snapshot() / disable_nonboot_cpus(). >> After apparently freeing over 400Mb / 100,000 pages of preallocated ram. >> >> >> >> There is a change which I missed before. When I applied your first >> patch ("Force GFP_NOIO during suspend" etc.), it did change the hung >> task backtraces a bit. I don't know if it tells us anything. >> >> Without the patch, there were two backtraces. The first backtrace >> suggested a problem allocating pages for a kernel thread (at >> copy_process() / try_to_free_pages()). The second showed that this >> problem was blocking s2disk (at hibernation_snapshot() / >> disable_nonboot_cpus() / stop_machine_create()). >> >> With the GFP_NOIO patch, I see only the s2disk backtrace. > > Can you please post this backtrace? Sure. It's rather like the one I posted before, except a) it only shows the one hung task (s2disk) b) this time I had lockdep enabled c) this time most of the lines don't have question marks. Kernel verson: - mainline v2.6.33-rc8-164-gaea187c, with the one patch "Force GFP_NOIO..." Image: http://picasaweb.google.com/lh/photo/f9KRZT2l9wCmVt-Jdggd9g?feat=directlink INFO: task s2disk:1916 blocked for more than 120 seconds ... Call Trace: ? _raw_spin_unlock schedule_timeout+0x22 (timer.c:1366) ? mark_held_locks ? _raw_spin_unlock_irq ? trace_hardirqs_on_caller ? trace_hardirqs_on wait_for_common+0xb8 (sched.c:5844) ? default_wake_function wait_for_completion+0x12 (sched.c:5879) kthread_create+0x75 (kthread.c:133) ? worker_thread+0x0 create_workqueue_thread+0x38 (workqueue.c:921) ? worker_thread+0x0 __create_workqueue_key+0x156 (workqueue.c:1006) stop_machine_create+0x32 (stop_machine.c:121) disable_nonboot_cpus+0xe (cpu.c:370) hibernation_snapshot+0x94 (hibernate.c:266) snapshot_ioctl+0x21b (user.c:256) ... sys_ioctl+0x41 (ioctl.c:624) Thanks Alan -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Rafael J. Wysocki on 23 Feb 2010 16:20 On Tuesday 23 February 2010, Alan Jenkins wrote: > On 2/22/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > > On Monday 22 February 2010, Alan Jenkins wrote: > >> Rafael J. Wysocki wrote: > >> > On Friday 19 February 2010, Alan Jenkins wrote: > >> > > >> >> On 2/18/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > >> >> > >> >>> On Thursday 18 February 2010, Alan Jenkins wrote: > >> >>> > >> >>>> On 2/17/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > >> >>>> > >> >>>>> On Wednesday 17 February 2010, Alan Jenkins wrote: > >> >>>>> > >> >>>>>> On 2/16/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > >> >>>>>> > >> >>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote: > >> >>>>>>> > >> >>>>>>>> On 2/16/10, Alan Jenkins <sourcejedi.lkml(a)googlemail.com> wrote: > >> >>>>>>>> > >> >>>>>>>>> On 2/15/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > >> >>>>>>>>> > >> >>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote: > >> >>>>>>>>>> > >> >>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too > >> >>>>>>>>>>> many > >> >>>>>>>>>>> applications. The first hibernation fails with "not enough > >> >>>>>>>>>>> swap" > >> >>>>>>>>>>> as > >> >>>>>>>>>>> expected, but the second or third attempt hangs (with the same > >> >>>>>>>>>>> backtrace > >> >>>>>>>>>>> as before). > >> >>>>>>>>>>> > >> >>>>>>>>>>> The patch definitely helps though. Without the patch, I see a > >> >>>>>>>>>>> hang > >> >>>>>>>>>>> the > >> >>>>>>>>>>> first time I try to hibernate with too many applications > >> >>>>>>>>>>> running. > >> >>>>>>>>>>> > >> >>>>>>>>>> Well, I have an idea. > >> >>>>>>>>>> > >> >>>>>>>>>> Can you try to apply the appended patch in addition and see if > >> >>>>>>>>>> that > >> >>>>>>>>>> helps? > >> >>>>>>>>>> > >> >>>>>>>>>> Rafael > >> >>>>>>>>>> > >> >>>>>>>>> It doesn't seem to help. > >> >>>>>>>>> > >> >>>>>>>> To be clear: It doesn't stop the hang when I hibernate with too > >> >>>>>>>> many > >> >>>>>>>> applications. > >> >>>>>>>> > >> >>>>>>>> It does stop the same hang in a different case though. > >> >>>>>>>> > >> >>>>>>>> 1. boot with init=/bin/bash > >> >>>>>>>> 2. run s2disk > >> >>>>>>>> 3. cancel the s2disk > >> >>>>>>>> 4. repeat steps 2&3 > >> >>>>>>>> > >> >>>>>>>> With the patch, I can run 10s of iterations, with no hang. > >> >>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), as > >> >>>>>>>> always). > >> >>>>>>>> > >> >>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no > >> >>>>>>>> problem. > >> >>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an > >> >>>>>>>> allocation > >> >>>>>>>> failure after a couple of iterations ("kthreadd: page allocation > >> >>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the > >> >>>>>>>> same > >> >>>>>>>> stop_machine thread allocation failure that causes the hang. > >> >>>>>>>> > >> >>>>>>> Have you tested it alone or on top of the previous one? If you've > >> >>>>>>> tested it > >> >>>>>>> alone, please apply the appended one in addition to it and retest. > >> >>>>>>> > >> >>>>>>> Rafael > >> >>>>>>> > >> >>>>>> I did test with both patches applied together - > >> >>>>>> > >> >>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation and > >> >>>>>> resume > >> >>>>>> 2. "reducing the number of pages that we're going to keep > >> >>>>>> preallocated > >> >>>>>> by > >> >>>>>> 20%" > >> >>>>>> > >> >>>>> In that case you can try to reduce the number of preallocated pages > >> >>>>> even > >> >>>>> more, > >> >>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch. > >> >>>>> > >> >>>> It still hangs if I try to hibernate a couple of times with too many > >> >>>> applications. > >> >>>> > >> >>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit > >> >>> system and > >> >>> how much RAM is there in the box? > >> >>> > >> >>> Rafael > >> >>> > >> >> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" SSD. > >> >> > >> > > >> > Hmm. I'd try to make free_unnecessary_pages() free all of the > >> > preallocated > >> > pages and see what happens. > >> > > >> > >> It still hangs in hibernation_snapshot() / disable_nonboot_cpus(). > >> After apparently freeing over 400Mb / 100,000 pages of preallocated ram. > >> > >> > >> > >> There is a change which I missed before. When I applied your first > >> patch ("Force GFP_NOIO during suspend" etc.), it did change the hung > >> task backtraces a bit. I don't know if it tells us anything. > >> > >> Without the patch, there were two backtraces. The first backtrace > >> suggested a problem allocating pages for a kernel thread (at > >> copy_process() / try_to_free_pages()). The second showed that this > >> problem was blocking s2disk (at hibernation_snapshot() / > >> disable_nonboot_cpus() / stop_machine_create()). > >> > >> With the GFP_NOIO patch, I see only the s2disk backtrace. > > > > Can you please post this backtrace? > > Sure. It's rather like the one I posted before, except > > a) it only shows the one hung task (s2disk) > b) this time I had lockdep enabled > c) this time most of the lines don't have question marks. Well, it still looks like we're waiting for create_workqueue_thread() to return, which probably is trying to allocate memory for the thread structure. My guess is that the preallocated memory pages freed by free_unnecessary_pages() go into a place from where they cannot be taken for subsequent NOIO allocations. I have no idea why that happens though. To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in kernel/power/suspend.c for completness). Rafael -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: KAMEZAWA Hiroyuki on 23 Feb 2010 20:30 On Tue, 23 Feb 2010 22:13:56 +0100 "Rafael J. Wysocki" <rjw(a)sisk.pl> wrote: > Well, it still looks like we're waiting for create_workqueue_thread() to > return, which probably is trying to allocate memory for the thread > structure. > > My guess is that the preallocated memory pages freed by > free_unnecessary_pages() go into a place from where they cannot be taken for > subsequent NOIO allocations. I have no idea why that happens though. > > To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the > calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in > kernel/power/suspend.c for completness). > If allocation of kernel threads for stop_machine_run() is the problem, What happens when 1. use CONIFG_4KSTACK or 2. make use of stop_machine_create(), stop_machine_destroy(). A new interface added by this commit. http://git.kernel.org/?p=linux/kernel/git/torvalds/ linux-2.6.git;a=commit;h=9ea09af3bd3090e8349ca2899ca2011bd94cda85 You can do no-fail stop_machine_run(). Thanks, -Kame -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Alan Jenkins on 24 Feb 2010 11:30
On 2/23/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: > On Tuesday 23 February 2010, Alan Jenkins wrote: >> On 2/22/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: >> > On Monday 22 February 2010, Alan Jenkins wrote: >> >> Rafael J. Wysocki wrote: >> >> > On Friday 19 February 2010, Alan Jenkins wrote: >> >> > >> >> >> On 2/18/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: >> >> >> >> >> >>> On Thursday 18 February 2010, Alan Jenkins wrote: >> >> >>> >> >> >>>> On 2/17/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: >> >> >>>> >> >> >>>>> On Wednesday 17 February 2010, Alan Jenkins wrote: >> >> >>>>> >> >> >>>>>> On 2/16/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: >> >> >>>>>> >> >> >>>>>>> On Tuesday 16 February 2010, Alan Jenkins wrote: >> >> >>>>>>> >> >> >>>>>>>> On 2/16/10, Alan Jenkins <sourcejedi.lkml(a)googlemail.com> >> >> >>>>>>>> wrote: >> >> >>>>>>>> >> >> >>>>>>>>> On 2/15/10, Rafael J. Wysocki <rjw(a)sisk.pl> wrote: >> >> >>>>>>>>> >> >> >>>>>>>>>> On Tuesday 09 February 2010, Alan Jenkins wrote: >> >> >>>>>>>>>> >> >> >>>>>>>>>>> Perhaps I spoke too soon. I see the same hang if I run too >> >> >>>>>>>>>>> many >> >> >>>>>>>>>>> applications. The first hibernation fails with "not enough >> >> >>>>>>>>>>> swap" >> >> >>>>>>>>>>> as >> >> >>>>>>>>>>> expected, but the second or third attempt hangs (with the >> >> >>>>>>>>>>> same >> >> >>>>>>>>>>> backtrace >> >> >>>>>>>>>>> as before). >> >> >>>>>>>>>>> >> >> >>>>>>>>>>> The patch definitely helps though. Without the patch, I >> >> >>>>>>>>>>> see a >> >> >>>>>>>>>>> hang >> >> >>>>>>>>>>> the >> >> >>>>>>>>>>> first time I try to hibernate with too many applications >> >> >>>>>>>>>>> running. >> >> >>>>>>>>>>> >> >> >>>>>>>>>> Well, I have an idea. >> >> >>>>>>>>>> >> >> >>>>>>>>>> Can you try to apply the appended patch in addition and see >> >> >>>>>>>>>> if >> >> >>>>>>>>>> that >> >> >>>>>>>>>> helps? >> >> >>>>>>>>>> >> >> >>>>>>>>>> Rafael >> >> >>>>>>>>>> >> >> >>>>>>>>> It doesn't seem to help. >> >> >>>>>>>>> >> >> >>>>>>>> To be clear: It doesn't stop the hang when I hibernate with >> >> >>>>>>>> too >> >> >>>>>>>> many >> >> >>>>>>>> applications. >> >> >>>>>>>> >> >> >>>>>>>> It does stop the same hang in a different case though. >> >> >>>>>>>> >> >> >>>>>>>> 1. boot with init=/bin/bash >> >> >>>>>>>> 2. run s2disk >> >> >>>>>>>> 3. cancel the s2disk >> >> >>>>>>>> 4. repeat steps 2&3 >> >> >>>>>>>> >> >> >>>>>>>> With the patch, I can run 10s of iterations, with no hang. >> >> >>>>>>>> Without the patch, it soon hangs, (in disable_nonboot_cpus(), >> >> >>>>>>>> as >> >> >>>>>>>> always). >> >> >>>>>>>> >> >> >>>>>>>> That's what happens on 2.6.33-rc7. On 2.6.30, there is no >> >> >>>>>>>> problem. >> >> >>>>>>>> On 2.6.31 and 2.6.32 I don't get a hang, but dmesg shows an >> >> >>>>>>>> allocation >> >> >>>>>>>> failure after a couple of iterations ("kthreadd: page >> >> >>>>>>>> allocation >> >> >>>>>>>> failure. order:1, mode:0xd0"). It looks like it might be the >> >> >>>>>>>> same >> >> >>>>>>>> stop_machine thread allocation failure that causes the hang. >> >> >>>>>>>> >> >> >>>>>>> Have you tested it alone or on top of the previous one? If >> >> >>>>>>> you've >> >> >>>>>>> tested it >> >> >>>>>>> alone, please apply the appended one in addition to it and >> >> >>>>>>> retest. >> >> >>>>>>> >> >> >>>>>>> Rafael >> >> >>>>>>> >> >> >>>>>> I did test with both patches applied together - >> >> >>>>>> >> >> >>>>>> 1. [Update] MM / PM: Force GFP_NOIO during suspend/hibernation >> >> >>>>>> and >> >> >>>>>> resume >> >> >>>>>> 2. "reducing the number of pages that we're going to keep >> >> >>>>>> preallocated >> >> >>>>>> by >> >> >>>>>> 20%" >> >> >>>>>> >> >> >>>>> In that case you can try to reduce the number of preallocated >> >> >>>>> pages >> >> >>>>> even >> >> >>>>> more, >> >> >>>>> ie. change "/ 5" to "/ 2" (for example) in the second patch. >> >> >>>>> >> >> >>>> It still hangs if I try to hibernate a couple of times with too >> >> >>>> many >> >> >>>> applications. >> >> >>>> >> >> >>> Hmm. I guess I asked that before, but is this a 32-bit or 64-bit >> >> >>> system and >> >> >>> how much RAM is there in the box? >> >> >>> >> >> >>> Rafael >> >> >>> >> >> >> EeePC 701. 32 bit. 512Mb RAM. 350Mb swap file, on a "first-gen" >> >> >> SSD. >> >> >> >> >> > >> >> > Hmm. I'd try to make free_unnecessary_pages() free all of the >> >> > preallocated >> >> > pages and see what happens. >> >> > >> >> >> >> It still hangs in hibernation_snapshot() / disable_nonboot_cpus(). >> >> After apparently freeing over 400Mb / 100,000 pages of preallocated >> >> ram. >> >> >> >> >> >> >> >> There is a change which I missed before. When I applied your first >> >> patch ("Force GFP_NOIO during suspend" etc.), it did change the hung >> >> task backtraces a bit. I don't know if it tells us anything. >> >> >> >> Without the patch, there were two backtraces. The first backtrace >> >> suggested a problem allocating pages for a kernel thread (at >> >> copy_process() / try_to_free_pages()). The second showed that this >> >> problem was blocking s2disk (at hibernation_snapshot() / >> >> disable_nonboot_cpus() / stop_machine_create()). >> >> >> >> With the GFP_NOIO patch, I see only the s2disk backtrace. >> > >> > Can you please post this backtrace? >> >> Sure. It's rather like the one I posted before, except >> >> a) it only shows the one hung task (s2disk) >> b) this time I had lockdep enabled >> c) this time most of the lines don't have question marks. > > Well, it still looks like we're waiting for create_workqueue_thread() to > return, which probably is trying to allocate memory for the thread > structure. > > My guess is that the preallocated memory pages freed by > free_unnecessary_pages() go into a place from where they cannot be taken for > subsequent NOIO allocations. I have no idea why that happens though. > > To test that theory you can try to change GFP_IOFS to GFP_KERNEL in the > calls to clear_gfp_allowed_mask() in kernel/power/hibernate.c (and in > kernel/power/suspend.c for completness). Effectively forcing GFP_NOWAIT, so the allocation should fail instead of hanging? It seems to stop the hang, but I don't see any other difference - the hibernation process isn't stopped earlier, and I don't get any new kernel messages about allocation failures. I wonder if it's because GFP_NOWAIT triggers ALLOC_HARDER. I have other evidence which argues for your theory: [ successful s2disk, with forced NOIO (but not NOWAIT), and test code as attached ] Freezing remaining freezable tasks ... (elapsed 0.01 seconds) done. 1280 GFP_NOWAIT allocations of order 0 are possible 640 GFP_NOWAIT allocations of order 1 are possible 320 GFP_NOWAIT allocations of order 2 are possible [ note - 1280 pages is the maximum test allocation used here. The test code is only accurate when talking about smaller numbers of free pages ] 1280 GFP_KERNEL allocations of order 0 are possible 640 GFP_KERNEL allocations of order 1 are possible 320 GFP_KERNEL allocations of order 2 are possible PM: Preallocating image memory... 212 GFP_NOWAIT allocations of order 0 are possible 102 GFP_NOWAIT allocations of order 1 are possible 50 GFP_NOWAIT allocations of order 2 are possible Freeing all 90083 preallocated pages (and 0 highmem pages, out of 0) 190 GFP_NOWAIT allocations of order 0 are possible 102 GFP_NOWAIT allocations of order 1 are possible 50 GFP_NOWAIT allocations of order 2 are possible 1280 GFP_KERNEL allocations of order 0 are possible 640 GFP_KERNEL allocations of order 1 are possible 320 GFP_KERNEL allocations of order 2 are possible done (allocated 90083 pages) It looks like you're right and the freed pages are not accessible with GFP_NOWAIT for some reason. I also tried a number of test runs with too many applications, and saw this: Freeing all 104006 preallocated pages ... 65 GFP_NOWAIT allocations of order 0 ... 18 GFP_NOWAIT allocations of order 1 ... 9 GFP_NOWAIT allocations of order 2 ... 0 GFP_KERNEL allocations of order 0 are possible .... Disabling nonboot cpus ... .... PM: Hibernation image created Force enabled HPET at resume PM: early thaw of devices complete after ... msecs <hang, no backtrace visible even after 120 seconds> I'm not bothered by the new hang; the test code will inevitably have some side effects. I'm not sure why GFP_KERNEL allocations would fail in this scenario though... perhaps the difference is that we've swapped out the entire userspace so GFP_IO doesn't help. Regards Alan |