Prev: virtio_net: fix oom handling on tx
Next: Aerospace and linux
From: Jiri Kosina on 16 Jun 2010 11:10
On Thu, 10 Jun 2010, Brian Bloniarz wrote:

> On 06/10/2010 02:10 PM, Chris Wedgwood wrote:
> > (sorry if this reponse isn't on target, i was just pointed to this
> > thread a few minutes ago)
> >
> >
> > On Thu, Jun 10, 2010 at 10:25:36AM -0700, Linus Torvalds wrote:
> >
> >> I thought we long since (ie back last fall) fixed the latency
> >> problems with pty's, but there does seem to be something very fishy
> >> going on there still.
> >
> > this might not be related, but i have slow serial ports with NOHZ that
> > goes away when i revert 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad.
>
> Unrelated or not, I think Chris is right about this. Somewhere before
> -rc1, the emulated serial console on my KVM instance became slow
> to echo input. I just tested with the commit reverted and it's
> back to normal.

So let's CC Mike then.

>
> > commit 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad
> > Author: Mike Galbraith <efault(a)gmx.de>
> > Date: Thu Mar 11 17:17:13 2010 +0100
> >
> > sched: Rate-limit nohz
> >
> > Entering nohz code on every micro-idle is costing ~10% throughput for netperf
> > TCP_RR when scheduling cross-cpu. Rate limiting entry fixes this, but raises
> > ticks a bit. On my Q6600, an idle box goes from ~85 interrupts/sec to 128.
> >
> > The higher the context switch rate, the more nohz entry costs. With this patch
> > and some cycle recovery patches in my tree, max cross cpu context switch rate is
> > improved by ~16%, a large portion of which of which is this ratelimiting.
> >
> > and looking at the only two interesting hunks it's not clear why:
> >
> > +int nohz_ratelimit(int cpu)
> > +{
> > + struct rq *rq = cpu_rq(cpu);
> > + u64 diff = rq->clock - rq->nohz_stamp;
> > +
> > + rq->nohz_stamp = rq->clock;
> > +
> > + return diff < (NSEC_PER_SEC / HZ) >> 1;
> > +}
> >
> > + if (nohz_ratelimit(cpu))
> > + goto end;
> > +
> >
> > network latnecy is fine, and if i create lots of wakeups (network IO
> > is fine) then the serial port latency is noticable

--
Jiri Kosina
SUSE Labs, Novell Inc.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Mike Galbraith on 16 Jun 2010 11:20
On Wed, 2010-06-16 at 17:03 +0200, Jiri Kosina wrote:
> On Thu, 10 Jun 2010, Brian Bloniarz wrote:
>
> > On 06/10/2010 02:10 PM, Chris Wedgwood wrote:
> > > (sorry if this reponse isn't on target, i was just pointed to this
> > > thread a few minutes ago)
> > >
> > >
> > > On Thu, Jun 10, 2010 at 10:25:36AM -0700, Linus Torvalds wrote:
> > >
> > >> I thought we long since (ie back last fall) fixed the latency
> > >> problems with pty's, but there does seem to be something very fishy
> > >> going on there still.
> > >
> > > this might not be related, but i have slow serial ports with NOHZ that
> > > goes away when i revert 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad.
> >
> > Unrelated or not, I think Chris is right about this. Somewhere before
> > -rc1, the emulated serial console on my KVM instance became slow
> > to echo input. I just tested with the commit reverted and it's
> > back to normal.
>
> So let's CC Mike then.

Chris already gave me a heads up, it's on my todo. The old P4 box I use
for a serial console box is exploding on boot, or I would have already
had a look.

> > > and looking at the only two interesting hunks it's not clear why:

Complete mystery to me.

> > > +int nohz_ratelimit(int cpu)
> > > +{
> > > + struct rq *rq = cpu_rq(cpu);
> > > + u64 diff = rq->clock - rq->nohz_stamp;
> > > +
> > > + rq->nohz_stamp = rq->clock;
> > > +
> > > + return diff < (NSEC_PER_SEC / HZ) >> 1;
> > > +}
> > >
> > > + if (nohz_ratelimit(cpu))
> > > + goto end;
> > > +
> > >
> > > network latnecy is fine, and if i create lots of wakeups (network IO
> > > is fine) then the serial port latency is noticable

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Mike Galbraith on 17 Jun 2010 02:40
On Wed, 2010-06-16 at 17:16 +0200, Mike Galbraith wrote:
> On Wed, 2010-06-16 at 17:03 +0200, Jiri Kosina wrote:
> > On Thu, 10 Jun 2010, Brian Bloniarz wrote:
> >
> > > On 06/10/2010 02:10 PM, Chris Wedgwood wrote:
> > > > (sorry if this reponse isn't on target, i was just pointed to this
> > > > thread a few minutes ago)
> > > >
> > > >
> > > > On Thu, Jun 10, 2010 at 10:25:36AM -0700, Linus Torvalds wrote:
> > > >
> > > >> I thought we long since (ie back last fall) fixed the latency
> > > >> problems with pty's, but there does seem to be something very fishy
> > > >> going on there still.
> > > >
> > > > this might not be related, but i have slow serial ports with NOHZ that
> > > > goes away when i revert 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad.
> > >
> > > Unrelated or not, I think Chris is right about this. Somewhere before
> > > -rc1, the emulated serial console on my KVM instance became slow
> > > to echo input. I just tested with the commit reverted and it's
> > > back to normal.
> >
> > So let's CC Mike then.
>
> Chris already gave me a heads up, it's on my todo. The old P4 box I use
> for a serial console box is exploding on boot, or I would have already
> had a look.

(Removing filth and re-seating ram seems to have revived poor old P4)

I'm not seeing any problem with serial console here, seems to work just
fine P4->Q6600, both running NOHZ kernels with nohz_ratelimit(), 33.5 on
the P4, and tip.today on the Q6600.

Eyeballing it, perhaps we need to proceed downward if any needs_cpu
condition is true, despite having just been here a wee bit ago.

Does this help anyone's woes?

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 5f171f0..ec72fad 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -262,7 +262,7 @@ void tick_nohz_stop_sched_tick(int inidle)
ktime_t last_update, expires, now;
struct clock_event_device *dev = __get_cpu_var(tick_cpu_device).evtdev;
u64 time_delta;
- int cpu;
+ int cpu, cpu_needed;

local_irq_save(flags);

@@ -315,7 +315,9 @@ void tick_nohz_stop_sched_tick(int inidle)
goto end;
}

- if (nohz_ratelimit(cpu))
+ cpu_needed = rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) || arch_needs_cpu(cpu);
+
+ if (!cpu_needed && nohz_ratelimit(cpu))
goto end;

ts->idle_calls++;
@@ -327,8 +329,7 @@ void tick_nohz_stop_sched_tick(int inidle)
time_delta = timekeeping_max_deferment();
} while (read_seqretry(&xtime_lock, seq));

- if (rcu_needs_cpu(cpu) || printk_needs_cpu(cpu) ||
- arch_needs_cpu(cpu)) {
+ if (cpu_needed) {
next_jiffies = last_jiffies + 1;
delta_jiffies = 1;
} else {


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Mike Galbraith on 17 Jun 2010 03:10
On Thu, 2010-06-17 at 08:39 +0200, Mike Galbraith wrote:

> I'm not seeing any problem with serial console here, seems to work just
> fine P4->Q6600, both running NOHZ kernels with nohz_ratelimit(), 33.5 on
> the P4, and tip.today on the Q6600.

Of course, as soon as I say that, the problem appeared. Hopefully,
It'll stick around a while.

-Mike

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Mike Galbraith on 17 Jun 2010 07:00
On Thu, 2010-06-17 at 09:00 +0200, Mike Galbraith wrote:
> On Thu, 2010-06-17 at 08:39 +0200, Mike Galbraith wrote:
>
> > I'm not seeing any problem with serial console here, seems to work just
> > fine P4->Q6600, both running NOHZ kernels with nohz_ratelimit(), 33.5 on
> > the P4, and tip.today on the Q6600.
>
> Of course, as soon as I say that, the problem appeared. Hopefully,
> It'll stick around a while.

Actually, it's fully reproducible, I just have _way_ too many (49)
kernels to choose from.

I had to go back to virgin 34, apply 39c0cbe and fixlet to fully test,
as git/tip network isn't working quite right for me atm. At any rate,
the below fixed it up for me, and cross-cpu throughput gain is intact.

sched: do not ratelimit NOHZ when the tick is stopped.

Chris Wedgwood reports that 39c0cbe sched: Rate-limit nohz causes a serial
console regression, unresponsiveness, and indeed it does. The below fixes
it by not skipping out when the tick has been stopped.

Tested that the throughput benefit of ratelimiting is still intact. It is.

Signed-off-by: Mike Galbraith <efault(a)gmx.de>
Reported-by: Chris Wedgwood <cw(a)f00f.org>
LKML-Reference: <new-submission>

diff --git a/kernel/time/tick-sched.c b/kernel/time/tick-sched.c
index 5f171f0..83c5129 100644
--- a/kernel/time/tick-sched.c
+++ b/kernel/time/tick-sched.c
@@ -315,7 +315,7 @@ void tick_nohz_stop_sched_tick(int inidle)
goto end;
}

- if (nohz_ratelimit(cpu))
+ if (!ts->tick_stopped && nohz_ratelimit(cpu))
goto end;

ts->idle_calls++;


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: virtio_net: fix oom handling on tx
Next: Aerospace and linux