x86 NMI-safe INT3 and Page Fault
in [Kernel]
|
From: Jeffrey Merkey on 16 Jul 2010 18:50 On Fri, Jul 16, 2010 at 4:22 PM, Linus Torvalds <torvalds(a)linux-foundation.org> wrote: > On Fri, Jul 16, 2010 at 3:02 PM, Jeffrey Merkey <jeffmerkey(a)gmail.com> wrote: >> >> So Linus, my understanding of Intel's processor design is that the >> processor will NEVER singal a nested NMI until it sees an iret from >> the first NMI exception. > > Wrong. > > I like x86, but it has warts. The NMI blocking is one of them. > > The NMI's will be nested until the _next_ "iret", but it has no > nesting. So if you take a fault during the NMI (debug, page table > fixup, whatever), the iret in the faulthandler will re-enable NMI's > even though we're still busy with the original NMI. There is no > nesting, or any way to say that "this is a NMI-releasing iret". They > could even do it still - make a new "iret that doesn't clear NMI" by > adding a segment override prefix to iret or whatever. But it's not > going to happen, and it's just one of those ugly special cases that > has various historical reasons (recursive faults during NMI sure as > hell didn't make sense back in the real-mode 8086 days). > > So we have to handle it in software. Or not ever trap at all inside > the NMI handler. > > The original patch - and the patch I detest - is to make the normal > fault paths use a "popf + ret" to emulate iret, but without the NMI > release. > > Now, I could live with that if it's the only solution, but it _is_ > pretty damn ugly. > > If somebody shows that it's actually faster to do "popf + ret" when > retuning to kernel space (a poor mans special-case iret), maybe it > would be worth it, but the really critical code sequence is actually > not "return to kernel space", but the "return to user space" case that > really wants the iret. And I just think it's disgusting to add extra > tests to that path. > > The other alternative would be to just make the rule be "NMI can never > take traps". It's possible to do that, but quite frankly, it's a pain. > It's a pain for page faults due to the whole vmalloc thing, and it's a > pain if you ever want to debug an NMI in any way (or put a breakpoint > on anything that is accessed from an NMI, which could potentially be > quite a lot of things). > > If it was just the debug issue, I'd say "neener neener, debuggers are > for wimps", but it's clearly not just about debug. It's a whole lot of > other thigs. Random percpu datastructures used for tracing, kernel > pointer verification code, yadda yadda. > > � � � � � � � � �Linus > Well, the way I handled this problem on NetWare SMP and that other kernel was to create a pool of TSS descriptors and reload each during the exception to swap stacks before any handlers were called. Allowed it to nest until I ran out of TSS descriptors (64 levels). Not sure that's the way to go here though but it worked on that case. Jeff -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Jeffrey Merkey on 16 Jul 2010 19:00 > > If it was just the debug issue, I'd say "neener neener, debuggers are > for wimps", but it's clearly not just about debug. It's a whole lot of > other thigs. Random percpu datastructures used for tracing, kernel > pointer verification code, yadda yadda. > > � � � � � � � � �Linus > I guess I am a wimp then ... :-) Jeff -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Linus Torvalds on 16 Jul 2010 21:20 On Fri, Jul 16, 2010 at 3:41 PM, Andi Kleen <andi(a)firstfloor.org> wrote: > > Maybe I'm misunderstanding everything (and it has been a lot of emails > in the thread), but the case I was thinking of would be if the second NMI > faults too, and then another one comes in after the IRET etc. No, the nested NMI cannot fault, because it never even enters C code. It literally just returns immediately after having noticed it is nested (and corrupted the stack of the original one, so that the original NMI will re-do itself at return).. So the nested NMI will use some few tens of bytes of stack. In fact, it will use the stack "above" the stack that the original NMI handler is using, because it will reset the stack pointer back to the top of the NMI stack. So in a very real sense, it is not even extending the stack, it is just re-using a small part of the same stack that the original NMI used (and that we copied away so that it doesn't matter that it gets re-used) As to another small but important detail: the _nested_ NMI actually returns using "popf+ret", leaving NMI's blocked again. Thus guaranteeing forward progress and lack of NMI storms. To summarize: - the "original" (first-level) NMI can take faults (like the page fault to fill in vmalloc pages lazily, or debug faults). That will actually cause two stack frames (or three, if you debug a page fault that happened while NMI was active). So there is certainly exception nesting going on, but we're talking _much_ less stack than normal stack usage where the nesting can be deep and in complex routines. - any "nested" NMI's will not actually use any more stack at all than a non-nested one, because we've pre-reserved space for them (and we _had_ to reserve space for them due to IST) - even if we get NMI's during the execution of the original NMI, there can be only one such "spurious" NMI per nested exception. So if we take a single page fault, that exception will re-enable NMI (because it returns with "iret"), and as a result we may take a _single_ new nested NMI until we disable NMI's again. In other words, the approach is not all that different from doing "lazy irq disable" like powerpc does for regular interrupts. For NMI's, we do it because it's impossible (on x86) to disable NMI's without actually taking one. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 18 Jul 2010 05:30 On 07/16/2010 10:30 PM, Andi Kleen wrote: > We already have infrastructure for kprobes to prevent breakpoints > on critical code (the __kprobes section). In principle kgdb/kdb > could be taught about honoring those too. > > It doesn't help with NMI code calling other functions, or with data breakpoints. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Peter Zijlstra on 4 Aug 2010 05:50
On Fri, 2010-07-16 at 11:05 -0700, H. Peter Anvin wrote: > > I really hope noone ever gets the idea of touching user space from an > NMI handler, though, and expecting it to work... Perf actually already does that to unwind user-space stacks... ;-) See arch/x86/kernel/cpu/perf_event.c:copy_from_user_nmi() and its users. What we do is a manual page table walk (using __get_user_pages_fast) and simply bail when the page is not available. That said, I think that the thing that started the whole per-cpu-per-context temp stack-frame storage story also means that that function is now broken and can lead to kmap_atomic corruption. I really should brush up that stack based kmap_atomic thing, last time I got stuck on FRV wanting things. Linus should I refresh that whole series and give a FRV a slow but working implementation and then let David Howells sort out things if he cares about that? -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |