2.6.33: ftrace triggers soft lockup
in [Kernel]
|
Prev: linux-next: build warning after merge of the mfd tree
Next: linux-next: build failure after merge of the driver-core tree
From: Steven Rostedt on 4 Mar 2010 09:00 On Wed, 2010-03-03 at 14:04 +0800, Am�rico Wang wrote: > I am not sure if this is ftrace's fault, but it is ftrace who triggers > the soft lockup. On my machine, it is pretty easy, just run: > > echo function_graph > current_tracer > > I found this when 2.6.33-rc7, but forgot to report here. I just tried > today's > Linus tree, it still has this problem. So this is not present in 2.6.33-rc6? If you know approximately when it appeared, would you be able to do a git bisect on it? Thanks, -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Américo Wang on 4 Mar 2010 23:20 On Thu, Mar 4, 2010 at 9:54 PM, Steven Rostedt <rostedt(a)goodmis.org> wrote: > On Wed, 2010-03-03 at 14:04 +0800, Américo Wang wrote: >> I am not sure if this is ftrace's fault, but it is ftrace who triggers >> the soft lockup. On my machine, it is pretty easy, just run: >> >> echo function_graph > current_tracer >> >> I found this when 2.6.33-rc7, but forgot to report here. I just tried >> today's >> Linus tree, it still has this problem. > > So this is not present in 2.6.33-rc6? If you know approximately when it > appeared, would you be able to do a git bisect on it? > I can't say that because I didn't try -rc6. Ok, let me try to do a bisect, but since weekend is coming, I can't do any testing during weekend, so will try it next Monday. Thanks! -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Américo Wang on 5 Mar 2010 02:20 On Fri, Mar 5, 2010 at 12:14 PM, Américo Wang <xiyou.wangcong(a)gmail.com> wrote: > On Thu, Mar 4, 2010 at 9:54 PM, Steven Rostedt <rostedt(a)goodmis.org> wrote: >> On Wed, 2010-03-03 at 14:04 +0800, Américo Wang wrote: >>> I am not sure if this is ftrace's fault, but it is ftrace who triggers >>> the soft lockup. On my machine, it is pretty easy, just run: >>> >>> echo function_graph > current_tracer >>> >>> I found this when 2.6.33-rc7, but forgot to report here. I just tried >>> today's >>> Linus tree, it still has this problem. >> >> So this is not present in 2.6.33-rc6? If you know approximately when it >> appeared, would you be able to do a git bisect on it? >> > > I can't say that because I didn't try -rc6. > Sigh, 2.6.33-rc6 doesn't work, even 2.6.32 doesn't work... -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Steven Rostedt on 5 Mar 2010 10:10 On Fri, 2010-03-05 at 15:16 +0800, Am�rico Wang wrote: > On Fri, Mar 5, 2010 at 12:14 PM, Am�rico Wang <xiyou.wangcong(a)gmail.com> wrote: > > On Thu, Mar 4, 2010 at 9:54 PM, Steven Rostedt <rostedt(a)goodmis.org> wrote: > >> On Wed, 2010-03-03 at 14:04 +0800, Am�rico Wang wrote: > >>> I am not sure if this is ftrace's fault, but it is ftrace who triggers > >>> the soft lockup. On my machine, it is pretty easy, just run: > >>> > >>> echo function_graph > current_tracer > >>> > > > > I can't say that because I didn't try -rc6. > > > > Sigh, 2.6.33-rc6 doesn't work, even 2.6.32 doesn't work... So basically you are saying that the function_graph tracer, when enabled has a high overhead? Well, unfortunately, that's expected. The function_graph tracer traces the start and end of every function. It uses the same mechanism as function tracer to trace the start of the function (mcount), but to trace the exit of a function, in the enter of the function it hijacks the return address and replaces it to call a trampoline. This trampoline will do the trace and then jump back to the original return address. Doing this breaks branch prediction in the CPU, as the CPU uses call/ret as part of its branch prediction analysis. So function graph tracing is not just twice as slow as function tracing, it actually has a bigger impact than that. So my question to you is, have you seen the function graph perform better with the same configs in previous kernels? Also, the function graph makes other debugging (like lockdep) have a greater impact to performance than they usually do. Now some things you can do to help performance. One is not to trace functions that are known to have a high hit rate. You can do this with the set_ftrace_notrace file, or add "ftrace_notrace=func1,func2,func3" to the command line where func1,func2,func3 are the functions you do not want to trace. This just adds these by default to the set_ftrace_notrace and can be removed at runtime. The functions I commonly write to are: echo '*spin_lock*' '*spin_unlock*' '*spin_try*' '*rcu_read*' > set_ftace_notrace since these functions are hit quite intensively, by not tracing them it helps a bit with performance. -- Steve -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Frederic Weisbecker on 5 Mar 2010 12:20
On Fri, Mar 05, 2010 at 10:06:43AM -0500, Steven Rostedt wrote: > On Fri, 2010-03-05 at 15:16 +0800, Am�rico Wang wrote: > > On Fri, Mar 5, 2010 at 12:14 PM, Am�rico Wang <xiyou.wangcong(a)gmail.com> wrote: > > > On Thu, Mar 4, 2010 at 9:54 PM, Steven Rostedt <rostedt(a)goodmis.org> wrote: > > >> On Wed, 2010-03-03 at 14:04 +0800, Am�rico Wang wrote: > > >>> I am not sure if this is ftrace's fault, but it is ftrace who triggers > > >>> the soft lockup. On my machine, it is pretty easy, just run: > > >>> > > >>> echo function_graph > current_tracer > > >>> > > > > > > > I can't say that because I didn't try -rc6. > > > > > > > Sigh, 2.6.33-rc6 doesn't work, even 2.6.32 doesn't work... > > So basically you are saying that the function_graph tracer, when enabled > has a high overhead? Well, unfortunately, that's expected. > > The function_graph tracer traces the start and end of every function. It > uses the same mechanism as function tracer to trace the start of the > function (mcount), but to trace the exit of a function, in the enter of > the function it hijacks the return address and replaces it to call a > trampoline. This trampoline will do the trace and then jump back to the > original return address. > > Doing this breaks branch prediction in the CPU, as the CPU uses call/ret > as part of its branch prediction analysis. So function graph tracing is > not just twice as slow as function tracing, it actually has a bigger > impact than that. It's true it has a high overhead, but not to the point of making the whole system unusable. We are supposed to be even far from that. I'm currently able to turn on the function graph tracer and use firefox without problems. It's just a bit slower but it's far from a visible starvation. And Li seems to see the same thing. For now I can not test, but I will try this week-end. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |