From: Ingo Molnar on

* Linus Torvalds <torvalds(a)linux-foundation.org> wrote:

> [...] And filesystem writeback performance issues, etc. It's all good.

> Christoph Hellwig (11):
> writeback: fix writeback completion notifications
> writeback: queue work on stack in writeback_inodes_sb
> writeback: enforce s_umount locking in writeback_inodes_sb
> writeback: fix writeback_inodes_wb from writeback_inodes_sb
> writeback: simplify wakeup_flusher_threads
> writeback: simplify and split bdi_start_writeback
> writeback: add missing requeue_io in writeback_inodes_wb
> writeback: fix pin_sb_for_writeback

Hm, those writeback patches are not looking overly good here - the kernel
crashes quite a lot during bootup:

BUG: unable to handle kernel paging request at 000000000528aa7e
IP: [<ffffffff8112d514>] wb_clear_pending+0x3b/0x66
PGD 0
Oops: 0002 [#1] PREEMPT SMP DEBUG_PAGEALLOC
last sysfs file: /sys/class/net/eth0/broadcast
CPU 1
Modules linked in:

Pid: 1637, comm: flush-8:0 Not tainted 2.6.35-rc4-tip-01075-g0c1880a-dirty #15306 A8N-E/System Product Name
RIP: 0010:[<ffffffff8112d514>] [<ffffffff8112d514>] wb_clear_pending+0x3b/0x66
RSP: 0018:ffff88003b899d90 EFLAGS: 00010286
RAX: 000000000528aa7e RBX: ffff880038e3d5f8 RCX: ffff88003b952000
RDX: ffff88003dd8c468 RSI: ffff88003dd8c420 RDI: ffff88003dd8c408
RBP: ffff88003b899da0 R08: ffffffff825bf870 R09: 0000000000000000
R10: ffff88003b899cf0 R11: 0000000000000002 R12: ffff88003dd8c408
R13: 0000000000000000 R14: ffff880038e3d5f8 R15: ffff88003dd8c468
FS: 00007f2d706b36f0(0000) GS:ffff880003200000(0000) knlGS:00000000f77bb6c0
CS: 0010 DS: 0000 ES: 0000 CR0: 000000008005003b
CR2: 000000000528aa7e CR3: 0000000001d1e000 CR4: 00000000000006a0
DR0: ffffffff82ceebbc DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000600
Process flush-8:0 (pid: 1637, threadinfo ffff88003b898000, task ffff88003b952000)
Stack:
ffff88003dd8c3a8 0000000000000000 ffff88003b899e30 ffffffff8112d5d7
<0> ffffffff8112d5de ffffffff823ae2a0 0000000000000000 ffff88003b899de0
<0> 00000000000009c4 ffff88003b899fd8 000000000000000a 0000000000000000
Call Trace:
[<ffffffff8112d5d7>] wb_do_writeback+0x98/0x1d7
[<ffffffff8112d5de>] ? wb_do_writeback+0x9f/0x1d7
[<ffffffff8112d755>] bdi_writeback_task+0x3f/0xfc
[<ffffffff810809a7>] ? bit_waitqueue+0x17/0xa9
[<ffffffff810ee2d0>] ? bdi_start_fn+0x0/0xda
[<ffffffff810ee346>] bdi_start_fn+0x76/0xda
[<ffffffff810ee2d0>] ? bdi_start_fn+0x0/0xda
[<ffffffff810805e9>] kthread+0x7f/0x87
[<ffffffff8102ac04>] kernel_thread_helper+0x4/0x10
[<ffffffff8105af79>] ? finish_task_switch+0x70/0xdc
[<ffffffff818674c5>] ? _raw_spin_unlock_irq+0x4f/0x6c
[<ffffffff81867994>] ? restore_args+0x0/0x30
[<ffffffff8108056a>] ? kthread+0x0/0x87
[<ffffffff8102ac00>] ? kernel_thread_helper+0x0/0x10
Code: ff 4e 28 0f 94 c0 84 c0 74 47 4c 8b 67 10 49 81 c4 10 02 00 00 4c 89 e7 e8 d5 92 73 00 48 8b 43 08 48 8b 13 4c 89 e7 48 89 42 08 <48> 89 10 48 b8 00 02 20 00 00 00 ad de 48 89 43 08 e8 4d a0 73
RIP [<ffffffff8112d514>] wb_clear_pending+0x3b/0x66

It appears to be a race of some sorts and is not reproducible consistently: it
occurs with an about 2-10% likelyhood in my test runs and i've seen about ~5
similar crashes so far. They appeared after you pulled those writeback
patches. I had a long and successful testrun of 1000+ kernels with no crashes,
shortly before those wb patches went upstream.

Given the small linear chance for the crash [i'd have to do a series of 100
reboots per bisection point to be reasonably sure about goodness/badness] i'd
rather not try to bisect it - but can try fix patches or can try specific
reverts as well.

Config and full crashlog attached. (Note, the log is from
tip-2.6.35-rc4-tip-01075-g0c1880a but the crash clearly came in from upstream,
the patterns are consistent and nothing remotely related happened in -tip in
that timeframe.)

Thanks,

Ingo