From: Ingo Molnar on

Dave,

FYI, this boot crash in arp_error_report() started triggering in -tip testing:

[ 113.285384] BUG: unable to handle kernel paging request at 6b6b6b87
[ 113.285384] IP: [<c14c8237>] arp_error_report+0x1d/0x32
[ 113.285384] *pdpt = 00000000340fd001 *pde = 0000000000000000
[ 113.285384] Oops: 0000 [#1] SMP
[ 113.285384] last sysfs file: /sys/class/net/eth0/address
[ 113.285384] Modules linked in:
[ 113.285384]
[ 113.285384] Pid: 0, comm: swapper Not tainted 2.6.34-08129-gcc106eb-dirty #257 A8N-E/System Product Name
[ 113.285384] EIP: 0060:[<c14c8237>] EFLAGS: 00010202 CPU: 0
[ 113.285384] EIP is at arp_error_report+0x1d/0x32
[ 113.285384] EAX: 6b6b6b6b EBX: f4c820b0 ECX: 00000000 EDX: c1867a24
[ 113.285384] ESI: f40ff074 EDI: f4c820b0 EBP: c2401f34 ESP: c2401f30
[ 113.285384] DS: 007b ES: 007b FS: 00d8 GS: 00e0 SS: 0068
[ 113.285384] Process swapper (pid: 0, ti=c2401000 task=c185e3c0 task.ti=c1853000)
[ 113.285384] Stack:
[ 113.285384] f40ff000 c2401f4c c147bca4 f40ff024 f40ff000 00000000 ffffb6db c2401f64
[ 113.285384] <0> c147cfde f40ff024 00000100 f40ff0ec c2401f8c c2401f98 c10447ab f40ff000
[ 113.285384] <0> c147ce8c f40ff0a4 c22104c4 00000000 c1804b08 00000000 00000000 f40ff0a4
[ 113.285384] Call Trace:
[ 113.285384] [<c147bca4>] ? neigh_invalidate+0x6d/0x84
[ 113.285384] [<c147cfde>] ? neigh_timer_handler+0x152/0x1f9
[ 113.285384] [<c10447ab>] ? call_timer_fn+0x6f/0xed
[ 113.285384] [<c147ce8c>] ? neigh_timer_handler+0x0/0x1f9
[ 113.285384] [<c1044962>] ? run_timer_softirq+0x139/0x16f
[ 113.285384] [<c147ce8c>] ? neigh_timer_handler+0x0/0x1f9
[ 113.285384] [<c103ed4f>] ? __do_softirq+0xc9/0x18d
[ 113.285384] [<c103ec86>] ? __do_softirq+0x0/0x18d
[ 113.285384] <IRQ>
[ 113.285384] [<c103e9ab>] ? irq_exit+0x3a/0x6d
[ 113.285384] [<c1016895>] ? smp_apic_timer_interrupt+0x6c/0x7a
[ 113.285384] [<c15664d6>] ? apic_timer_interrupt+0x36/0x40
[ 113.285384] [<c1009740>] ? poll_idle+0x31/0x5d
[ 113.285384] [<c100255e>] ? cpu_idle+0xab/0xc5
[ 113.285384] [<c1529925>] ? rest_init+0xa1/0xa6
[ 113.285384] [<c18e6a2d>] ? start_kernel+0x3ed/0x3f2
[ 113.285384] [<c18e60ca>] ? i386_start_kernel+0xca/0xd1
[ 113.285384] Code: 0c 89 43 70 31 c0 8d 65 f4 5b 5e 5f 5d c3 55 89 e5 53 0f 1f 44 00 00 89 d3 89 d0 e8 fa fb ff ff 85 c0 74 12 8b 40 40 85 c0 74 0b <8b> 50 1c 85 d2 74 04 89 d8 ff d2 89 d8 e8 9b 50 fa ff 5b 5d c3
[ 113.285384] EIP: [<c14c8237>] arp_error_report+0x1d/0x32 SS:ESP 0068:c2401f30
[ 113.285384] CR2: 000000006b6b6b87
[ 113.493016] ---[ end trace e94352dcdbc4a293 ]---

Config and full crashlog attached.

Thanks,

Ingo
From: Linus Torvalds on


On Thu, 27 May 2010, Ingo Molnar wrote:
>
> FYI, this boot crash in arp_error_report() started triggering in -tip testing:
>
> [ 113.285384] BUG: unable to handle kernel paging request at 6b6b6b87

That's the POISON_FREE signature, with an offset of 28 (0x1c).

And it looks like the whole function got captured in the Code: sequence.
It looks like this:

0: 55 push %ebp
1: 89 e5 mov %esp,%ebp
3: 53 push %ebx
4: 0f 1f 44 00 00 nopl 0x0(%eax,%eax,1)
9: 89 d3 mov %edx,%ebx
b: 89 d0 mov %edx,%eax
d: e8 fa fb ff ff call 0xfffffc0c # skb_dst()
12: 85 c0 test %eax,%eax # dst
14: 74 12 je 0x28
16: 8b 40 40 mov 0x40(%eax),%eax # dst->ops
19: 85 c0 test %eax,%eax
1b: 74 0b je 0x28
1d:* 8b 50 1c mov 0x1c(%eax),%edx <-- trapping instruction
20: 85 d2 test %edx,%edx
22: 74 04 je 0x28
24: 89 d8 mov %ebx,%eax
26: ff d2 call *%edx # dst->ops->link_failure()
28: 89 d8 mov %ebx,%eax
2a: e8 9b 50 fa ff call 0xfffa50ca # skb_free()
2f: 5b pop %ebx
30: 5d pop %ebp
31: c3 ret

Where most of it is "dst_link_failure()" being inlined (that last "callq"
is the call to kfree_skb().

Looks like 'dst' points to free'd memory, so when we load a pointer from
it (the dst->ops) field, we get 0x6b6b6b6b, and then when we try to load
dst->ops->link_failure it oopses.

tl;dr: that

struct dst_entry *dst = skb_dst(skb);

in dst_link_failure seems to result in a stale skb.

Linus
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Eric Dumazet on
Le jeudi 27 mai 2010 à 12:27 -0700, Linus Torvalds a écrit :
>
> On Thu, 27 May 2010, Ingo Molnar wrote:
> >
> > FYI, this boot crash in arp_error_report() started triggering in -tip testing:
> >
> > [ 113.285384] BUG: unable to handle kernel paging request at 6b6b6b87
>
> That's the POISON_FREE signature, with an offset of 28 (0x1c).
>
> And it looks like the whole function got captured in the Code: sequence.
> It looks like this:
>
> 0: 55 push %ebp
> 1: 89 e5 mov %esp,%ebp
> 3: 53 push %ebx
> 4: 0f 1f 44 00 00 nopl 0x0(%eax,%eax,1)
> 9: 89 d3 mov %edx,%ebx
> b: 89 d0 mov %edx,%eax
> d: e8 fa fb ff ff call 0xfffffc0c # skb_dst()
> 12: 85 c0 test %eax,%eax # dst
> 14: 74 12 je 0x28
> 16: 8b 40 40 mov 0x40(%eax),%eax # dst->ops
> 19: 85 c0 test %eax,%eax
> 1b: 74 0b je 0x28
> 1d:* 8b 50 1c mov 0x1c(%eax),%edx <-- trapping instruction
> 20: 85 d2 test %edx,%edx
> 22: 74 04 je 0x28
> 24: 89 d8 mov %ebx,%eax
> 26: ff d2 call *%edx # dst->ops->link_failure()
> 28: 89 d8 mov %ebx,%eax
> 2a: e8 9b 50 fa ff call 0xfffa50ca # skb_free()
> 2f: 5b pop %ebx
> 30: 5d pop %ebp
> 31: c3 ret
>
> Where most of it is "dst_link_failure()" being inlined (that last "callq"
> is the call to kfree_skb().
>
> Looks like 'dst' points to free'd memory, so when we load a pointer from
> it (the dst->ops) field, we get 0x6b6b6b6b, and then when we try to load
> dst->ops->link_failure it oopses.
>
> tl;dr: that
>
> struct dst_entry *dst = skb_dst(skb);
>
> in dst_link_failure seems to result in a stale skb.
>
> Linus
> --

I am looking at this bug report, as I am probably at fault, please give
me one or two hour ;)

Thanks


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Eric Dumazet on
Le jeudi 27 mai 2010 à 21:47 +0200, Eric Dumazet a écrit :

> I am looking at this bug report, as I am probably at fault, please give
> me one or two hour ;)

I believe problem comes from commit 7fee226ad2
(net: add a noref bit on skb dst)

We probably should add a WARN in __skb_queue_tail() and similar enqueue
functions to catch other problems. I'll post a followup.

Thanks !

[PATCH] net: fix __neigh_event_send()

commit 7fee226ad23 (net: add a noref bit on skb dst) missed one spot
where an skb is enqueued, with a possibly not refcounted dst entry.

__neigh_event_send() inserts skb into arp_queue, so we must make sure
dst entry is refcounted, or dst entry can be freed by garbage collector
after caller exits from rcu protected section.

Reported-by: Ingo Molnar <mingo(a)elte.hu>
Signed-off-by: Eric Dumazet <eric.dumazet(a)gmail.com>
---
net/core/neighbour.c | 1 +
1 file changed, 1 insertion(+)

diff --git a/net/core/neighbour.c b/net/core/neighbour.c
index bff3790..6ba1c0e 100644
--- a/net/core/neighbour.c
+++ b/net/core/neighbour.c
@@ -934,6 +934,7 @@ int __neigh_event_send(struct neighbour *neigh, struct sk_buff *skb)
kfree_skb(buff);
NEIGH_CACHE_STAT_INC(neigh->tbl, unres_discards);
}
+ skb_dst_force(skb);
__skb_queue_tail(&neigh->arp_queue, skb);
}
rc = 1;


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/