From: Ed W on
Hi, my network connection looks like 500Kbits with a round trip latency
of perhaps 1s+ (it's a satellite link).

From what I can see the linux initial congestion window is signficantly
limiting me here, with slow start taking many many seconds to open up
the window wide enough to get the data flowing? For protocols like http
this is really hurting with all the short lived connections never really
getting up to speed. (throw in some random packet loss and things
really screech to a halt)

Reading around there appear to be several previous attempts to modify
the kernel to start with a slightly wider initial congestion window, say
10 packets. (Seems even google did some work on this and agreed that a
small of initial cwd to 10 ish, would help even many non satellite
users?) However, all the work I can find is quite old and doesn't seem
to give me much of a leg up in terms of experimenting with such changes
on a modern kernel?

Does someone have some pointers on where to look to modify initial
congestion window please?


Thanks

Ed W
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Bill Davidsen on
Ed W wrote:
> Hi, my network connection looks like 500Kbits with a round trip latency
> of perhaps 1s+ (it's a satellite link).
>
> From what I can see the linux initial congestion window is signficantly
> limiting me here, with slow start taking many many seconds to open up
> the window wide enough to get the data flowing? For protocols like http
> this is really hurting with all the short lived connections never really
> getting up to speed. (throw in some random packet loss and things
> really screech to a halt)
>
> Reading around there appear to be several previous attempts to modify
> the kernel to start with a slightly wider initial congestion window, say
> 10 packets. (Seems even google did some work on this and agreed that a
> small of initial cwd to 10 ish, would help even many non satellite
> users?) However, all the work I can find is quite old and doesn't seem
> to give me much of a leg up in terms of experimenting with such changes
> on a modern kernel?
>
> Does someone have some pointers on where to look to modify initial
> congestion window please?
>
Are you sure that's the issue? The backlog is in incoming, is it not?

Having dealt with moderately long delays push TB between timezones, have you set
your window size up? Set /proc/sys/net/ipv4/tcp_adv_win_scale to 5 or 6 and see
if that helps. You may have to go into /proc/sys/net/core and crank up the
rmem_* settings, depending on your distribution.

This allows the server to push a lot of data without an ack, which is what you
want, the ack will be delayed by the long latency, so this helps. You can
calculate how large to make the setting, but "make it bigger until tcpdump never
shows the window size < 2k" is the best way, even a bit larger than that wont
hurt, although it will take a bit of memory.

--
Bill Davidsen <davidsen(a)tmr.com>
"We have more to fear from the bungling of the incompetent than from
the machinations of the wicked." - from Slashdot
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: David Miller on
From: Bill Davidsen <davidsen(a)tmr.com>
Date: Wed, 14 Jul 2010 11:21:15 -0400

> You may have to go into /proc/sys/net/core and crank up the
> rmem_* settings, depending on your distribution.

You should never, ever, have to touch the various networking sysctl
values to get good performance in any normal setup. If you do, it's a
bug, report it so we can fix it.

I cringe every time someone says to do this, so please do me a favor
and don't spread this further. :-)

For one thing, TCP dynamically adjusts the socket buffer sizes based
upon the behavior of traffic on the connection.

And the TCP memory limit sysctls (not the core socket ones) are sized
based upon available memory. They are there to protect you from
situations such as having so much memory dedicated to socket buffers
that there is none left to do other things effectively. It's a
protective limit, rather than a setting meant to increase or improve
performance. So like the others, leave these alone too.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Ed W on

>> Does someone have some pointers on where to look to modify initial
>> congestion window please?
>>
> Are you sure that's the issue? The backlog is in incoming, is it not?

Well, I was simplifying a little bit, actually I have a bunch of
protocols in use, http is one of them


> Having dealt with moderately long delays push TB between timezones,
> have you set your window size up? Set
> /proc/sys/net/ipv4/tcp_adv_win_scale to 5 or 6 and see if that helps.
> You may have to go into /proc/sys/net/core and crank up the rmem_*
> settings, depending on your distribution.
>
> This allows the server to push a lot of data without an ack, which is
> what you want, the ack will be delayed by the long latency, so this helps.

I think I'm misunderstanding something fundamental here:

- Surely the limited congestion window is what throttles me at
connection initialisation time and this will not be affected by changing
the params you mention above? For sure the sliding window will be
relevant vs my bandwidth delay product once the tcp connection reaches
steady state, but I'm mostly worried here about performance right at the
creation of the connection?

- Both you and Alan mention that the bulk of the traffic is "incoming" -
this implies you think it's relevant? Obviously I'm missing something
fundamental here because my understanding is that the congestion window
shuts us down in both directions (at the start of the connection?)

Thanks for the replies - I will take it over to netdev

Ed W
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Ed W on
On 14/07/2010 19:15, David Miller wrote:
> From: Bill Davidsen<davidsen(a)tmr.com>
> Date: Wed, 14 Jul 2010 11:21:15 -0400
>
>
>> You may have to go into /proc/sys/net/core and crank up the
>> rmem_* settings, depending on your distribution.
>>
> You should never, ever, have to touch the various networking sysctl
> values to get good performance in any normal setup. If you do, it's a
> bug, report it so we can fix it.
>

Just checking the basics here because I don't think this is a bug so
much as a, less common installation that differs from the "normal" case.

- When we create a tcp connection we always start with tcp slow start
- This sets the congestion window to effectively 4 packets?
- This applies in both directions?
- Remote sender responds to my hypothetical http request with the first
4 packets of data
- We need to wait one RTT for the ack to come back and now we can send
the next 8 packets,
- Wait for the next ack and at 16 packets we are now moving at a
sensible fraction of the bandwidth delay product?

So just to be clear:
- We don't seem to have any user-space tuning knobs to influence this
right now?
- In this age of short attention spans, a couple of extra seconds
between clicking something and it responding is worth optimising (IMHO)
- I think I need to take this to netdev, but anyone else with any ideas
happy to hear them?

Thanks

Ed W
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/