2.6.33: ftrace triggers soft lockup
in [Kernel]
|
Prev: linux-next: build warning after merge of the mfd tree
Next: linux-next: build failure after merge of the driver-core tree
From: Steven Rostedt on 5 Mar 2010 13:40 On Fri, 2010-03-05 at 18:16 +0100, Frederic Weisbecker wrote: > It's true it has a high overhead, but not to the point of > making the whole system unusable. We are supposed to be even > far from that. I'm currently able to turn on the function graph > tracer and use firefox without problems. It's just a bit slower > but it's far from a visible starvation. > > And Li seems to see the same thing. > For now I can not test, but I will try this week-end. Americo said he's seen the issue as far back as 2.6.32. So perhaps some CPUs take a bigger hit from the function graph tracer than others. I have several different boxes that I can try. I've seen noticeable slow downs but never something that cripples the box. The only time that I've seen it cripple the box is when LOCKDEP_DEBUG was set (which according to Americo's config it was not). But that's because LOCKDEP_DEBUG updates a global variable every time interrupts are enabled or disabled. This caused a huge cache line bouncing with the function graph tracer since it caused this variable to be updated 4 times for every function call! -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Frederic Weisbecker on 5 Mar 2010 13:50 On Fri, Mar 05, 2010 at 01:35:00PM -0500, Steven Rostedt wrote: > On Fri, 2010-03-05 at 18:16 +0100, Frederic Weisbecker wrote: > > > It's true it has a high overhead, but not to the point of > > making the whole system unusable. We are supposed to be even > > far from that. I'm currently able to turn on the function graph > > tracer and use firefox without problems. It's just a bit slower > > but it's far from a visible starvation. > > > > And Li seems to see the same thing. > > For now I can not test, but I will try this week-end. > > Americo said he's seen the issue as far back as 2.6.32. So perhaps some > CPUs take a bigger hit from the function graph tracer than others. I > have several different boxes that I can try. I've seen noticeable slow > downs but never something that cripples the box. > > The only time that I've seen it cripple the box is when LOCKDEP_DEBUG > was set (which according to Americo's config it was not). But that's > because LOCKDEP_DEBUG updates a global variable every time interrupts > are enabled or disabled. This caused a huge cache line bouncing with the > function graph tracer since it caused this variable to be updated 4 > times for every function call! Ouch...that's the hardirqs_off_events/redundant_hardirqs_off variables? Those should be clearly made per cpu. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Steven Rostedt on 5 Mar 2010 14:10 On Fri, 2010-03-05 at 19:43 +0100, Frederic Weisbecker wrote: > Ouch...that's the hardirqs_off_events/redundant_hardirqs_off variables? > Those should be clearly made per cpu. Yeah, probably all the debug_atomic_inc(*) variables should be made per-cpu. Should not be too hard since they are already wrapped in macros. -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Américo Wang on 7 Mar 2010 21:40 On Fri, Mar 5, 2010 at 11:06 PM, Steven Rostedt <rostedt(a)goodmis.org> wrote: > On Fri, 2010-03-05 at 15:16 +0800, Américo Wang wrote: >> On Fri, Mar 5, 2010 at 12:14 PM, Américo Wang <xiyou.wangcong(a)gmail.com> wrote: >> > On Thu, Mar 4, 2010 at 9:54 PM, Steven Rostedt <rostedt(a)goodmis.org> wrote: >> >> On Wed, 2010-03-03 at 14:04 +0800, Américo Wang wrote: >> >>> I am not sure if this is ftrace's fault, but it is ftrace who triggers >> >>> the soft lockup. On my machine, it is pretty easy, just run: >> >>> >> >>> echo function_graph > current_tracer >> >>> > >> > >> > I can't say that because I didn't try -rc6. >> > >> >> Sigh, 2.6.33-rc6 doesn't work, even 2.6.32 doesn't work... > > So basically you are saying that the function_graph tracer, when enabled > has a high overhead? Well, unfortunately, that's expected. A unusable system is expected? ;-) Here my machine is too slow to use. > > So my question to you is, have you seen the function graph perform > better with the same configs in previous kernels? Also, the function > graph makes other debugging (like lockdep) have a greater impact to > performance than they usually do. Oh, sorry, I still don't find a working kernel, I will try today. > > Now some things you can do to help performance. One is not to trace > functions that are known to have a high hit rate. You can do this with > the set_ftrace_notrace file, or add "ftrace_notrace=func1,func2,func3" > to the command line where func1,func2,func3 are the functions you do not > want to trace. This just adds these by default to the set_ftrace_notrace > and can be removed at runtime. > > > The functions I commonly write to are: > > echo '*spin_lock*' '*spin_unlock*' '*spin_try*' '*rcu_read*' > set_ftace_notrace > > since these functions are hit quite intensively, by not tracing them it > helps a bit with performance. I will try this now. Thanks! -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Li Zefan on 8 Mar 2010 02:40
>>>>>> I am not sure if this is ftrace's fault, but it is ftrace who triggers >>>>>> the soft lockup. On my machine, it is pretty easy, just run: >>>>>> >>>>>> echo function_graph > current_tracer >>>>>> >>>> I can't say that because I didn't try -rc6. >>>> >>> Sigh, 2.6.33-rc6 doesn't work, even 2.6.32 doesn't work... >> So basically you are saying that the function_graph tracer, when enabled >> has a high overhead? Well, unfortunately, that's expected. >> >> The function_graph tracer traces the start and end of every function. It >> uses the same mechanism as function tracer to trace the start of the >> function (mcount), but to trace the exit of a function, in the enter of >> the function it hijacks the return address and replaces it to call a >> trampoline. This trampoline will do the trace and then jump back to the >> original return address. >> >> Doing this breaks branch prediction in the CPU, as the CPU uses call/ret >> as part of its branch prediction analysis. So function graph tracing is >> not just twice as slow as function tracing, it actually has a bigger >> impact than that. > > > It's true it has a high overhead, but not to the point of > making the whole system unusable. We are supposed to be even > far from that. I'm currently able to turn on the function graph > tracer and use firefox without problems. It's just a bit slower > but it's far from a visible starvation. > > And Li seems to see the same thing. Yes, and more than this. I can see segmentation fault while testing, both my testing threads and kernel building threads that are running at the same time can get segfault. > For now I can not test, but I will try this week-end. > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |