Slow pty's (was Re: libdivecomputer interfaces?)
in [Kernel]
|
From: Linus Torvalds on 10 Jun 2010 13:30 Greg, Alan, Hirofumi-san, I thought we long since (ie back last fall) fixed the latency problems with pty's, but there does seem to be something very fishy going on there still. On Thu, Jun 10, 2010 at 8:01 AM, Linus Torvalds <torvalds(a)linux-foundation.org> wrote: >On Sat, May 29, 2010 at 12:53 PM, Jef Driesen <jefdriesen(a)telenet.be> wrote >> BTW, now that I have your attention, could you maybe help me with a linux >> kernel problem I'm experiencing in this area? I reported the problem on LKML >> but got no response: >> >> http://www.divesoftware.org/libdc/simulator.html >> http://groups.google.com/group/linux.kernel/browse_thread/thread/5a2b00e35b0864a7 > > [ Hmm.. Testing.. ] > > Yeah, it's slow. Your test thing takes one and a quarter minutes for > me. That's ridiculous. > > And no, we shouldn't need the low-latency flag, we're supposed to do > this all automatically correctly. I'll talk to the tty people. This is clearly not a regression (it's been going on forever, I suspect), but taking over a minute to transfer just over half a MB of data over a pty seems crazy. Maybe it's not a kernel problem, and it's something done wrong by rx/sx/socat, I haven't looked at what they do. But since setting low_latency apparently helps (I didn't test that part, but I did test "ridiculously slow"), it sounds very much like something is still wrong in the kernel unless there is some really subtle timing issue in user space. From Jef's original lkml report linked to above: > You can reproduce the problem by running these commands in three > different terminals: > > # Terminal 1: Setup the pty's. > socat PTY,link=/tmp/ttyS0 PTY,link=/tmp/ttyS1 > # Terminal 2: Send some data. > dd if=/dev/urandom of=input.bin bs=538368 count=1 > sx input.bin >>/tmp/ttyS0 </tmp/ttyS0 > # Terminal 2: Receive the data data. > time rx output.bin >/tmp/ttyS1 </tmp/ttyS1 and yeah, it's pretty clear to see. A "perf report" on that receiving side just shows queue_delayed_work_on(), but that doesn't mean much. It's clearly just sleeping all the time... Any ideas? Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: OGAWA Hirofumi on 10 Jun 2010 14:10 Linus Torvalds <torvalds(a)linux-foundation.org> writes: >>From Jef's original lkml report linked to above: > >> You can reproduce the problem by running these commands in three >> different terminals: >> >> # Terminal 1: Setup the pty's. >> socat PTY,link=/tmp/ttyS0 PTY,link=/tmp/ttyS1 >> # Terminal 2: Send some data. >> dd if=/dev/urandom of=input.bin bs=538368 count=1 >> sx input.bin >>/tmp/ttyS0 </tmp/ttyS0 >> # Terminal 2: Receive the data data. >> time rx output.bin >/tmp/ttyS1 </tmp/ttyS1 > > and yeah, it's pretty clear to see. A "perf report" on that receiving > side just shows queue_delayed_work_on(), but that doesn't mean much. > It's clearly just sleeping all the time... I'm not reading all of this thread yet, so sorry if I'm missing the point of this thread. FWIW, I remember there was the unnecessary waiting related to background flusher. It was in input_available_p(). I'm not sure at all whether this is related to the problem, and totally untested patch (this patch calls tty_flush_to_ldisc() only when data is unavailable). Someone can test the following or something (is there any change)? Well, anyway, I'll read this thread at this weekend, and will see if someone didn't already fix it. I.e. the following or something, static inline int input_available_p(struct tty_struct *tty, int amt) { int try = 0; retry: if (tty->icanon) { if (tty->canon_data) return 1; } else if (tty->read_cnt >= (amt ? amt : 1)) return 1; if (!checked) { tty_flush_to_ldisc(tty); try = 1; goto retry; } return 0; } -- OGAWA Hirofumi <hirofumi(a)mail.parknet.co.jp> -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Chris Wedgwood on 10 Jun 2010 14:30 (sorry if this reponse isn't on target, i was just pointed to this thread a few minutes ago) On Thu, Jun 10, 2010 at 10:25:36AM -0700, Linus Torvalds wrote: > I thought we long since (ie back last fall) fixed the latency > problems with pty's, but there does seem to be something very fishy > going on there still. this might not be related, but i have slow serial ports with NOHZ that goes away when i revert 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad. commit 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad Author: Mike Galbraith <efault(a)gmx.de> Date: Thu Mar 11 17:17:13 2010 +0100 sched: Rate-limit nohz Entering nohz code on every micro-idle is costing ~10% throughput for netperf TCP_RR when scheduling cross-cpu. Rate limiting entry fixes this, but raises ticks a bit. On my Q6600, an idle box goes from ~85 interrupts/sec to 128. The higher the context switch rate, the more nohz entry costs. With this patch and some cycle recovery patches in my tree, max cross cpu context switch rate is improved by ~16%, a large portion of which of which is this ratelimiting. and looking at the only two interesting hunks it's not clear why: +int nohz_ratelimit(int cpu) +{ + struct rq *rq = cpu_rq(cpu); + u64 diff = rq->clock - rq->nohz_stamp; + + rq->nohz_stamp = rq->clock; + + return diff < (NSEC_PER_SEC / HZ) >> 1; +} + if (nohz_ratelimit(cpu)) + goto end; + network latnecy is fine, and if i create lots of wakeups (network IO is fine) then the serial port latency is noticable -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Brian Bloniarz on 10 Jun 2010 18:30 On 06/10/2010 02:10 PM, Chris Wedgwood wrote: > (sorry if this reponse isn't on target, i was just pointed to this > thread a few minutes ago) > > > On Thu, Jun 10, 2010 at 10:25:36AM -0700, Linus Torvalds wrote: > >> I thought we long since (ie back last fall) fixed the latency >> problems with pty's, but there does seem to be something very fishy >> going on there still. > > this might not be related, but i have slow serial ports with NOHZ that > goes away when i revert 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad. Unrelated or not, I think Chris is right about this. Somewhere before -rc1, the emulated serial console on my KVM instance became slow to echo input. I just tested with the commit reverted and it's back to normal. > commit 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad > Author: Mike Galbraith <efault(a)gmx.de> > Date: Thu Mar 11 17:17:13 2010 +0100 > > sched: Rate-limit nohz > > Entering nohz code on every micro-idle is costing ~10% throughput for netperf > TCP_RR when scheduling cross-cpu. Rate limiting entry fixes this, but raises > ticks a bit. On my Q6600, an idle box goes from ~85 interrupts/sec to 128. > > The higher the context switch rate, the more nohz entry costs. With this patch > and some cycle recovery patches in my tree, max cross cpu context switch rate is > improved by ~16%, a large portion of which of which is this ratelimiting. > > and looking at the only two interesting hunks it's not clear why: > > +int nohz_ratelimit(int cpu) > +{ > + struct rq *rq = cpu_rq(cpu); > + u64 diff = rq->clock - rq->nohz_stamp; > + > + rq->nohz_stamp = rq->clock; > + > + return diff < (NSEC_PER_SEC / HZ) >> 1; > +} > > + if (nohz_ratelimit(cpu)) > + goto end; > + > > network latnecy is fine, and if i create lots of wakeups (network IO > is fine) then the serial port latency is noticable > -- > To unsubscribe from this list: send the line "unsubscribe linux-kernel" in > the body of a message to majordomo(a)vger.kernel.org > More majordomo info at http://vger.kernel.org/majordomo-info.html > Please read the FAQ at http://www.tux.org/lkml/ -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Linus Torvalds on 10 Jun 2010 18:40
On Thu, Jun 10, 2010 at 3:25 PM, Brian Bloniarz <bmb(a)athenacr.com> wrote: > On 06/10/2010 02:10 PM, Chris Wedgwood wrote: >> >> this might not be related, but i have slow serial ports with NOHZ that >> goes away when i revert 39c0cbe2150cbd848a25ba6cdb271d1ad46818ad. > > Unrelated or not, I think Chris is right about this. Somewhere before > -rc1, the emulated serial console on my KVM instance became slow > to echo input. I just tested with the commit reverted and it's > back to normal. I suspect it's related. The tty subsystem clearly ends up depending on the timer tick to move things forward even if there are pending readers. So I suspect the slowness is very much related to the same issue. Linus -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |