From: Heinz Diehl on
On 23.07.2010, Vivek Goyal wrote:

> This is V4 of the patchset which implements a new tunable group_idle and also
> implements IOPS mode for group fairness. Following are changes since V3.
[....]

Just for information: this patchset, applied to 2.6.35-rc6, gives about
20-25% increase in speed/throughput on my desktop system
(Phenom 2.5GHz Quadcore, 3 disks) with the tunables set according
to what you've used/reported here (the setup with slice_idle set to 0),
and it's measurable with fs_mark, too.

After 2 hours of hard testing, the machine remains stable and responsive.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Vivek Goyal on
On Fri, Jul 23, 2010 at 04:03:43PM +0200, Heinz Diehl wrote:
> On 23.07.2010, Vivek Goyal wrote:
>
> > This is V4 of the patchset which implements a new tunable group_idle and also
> > implements IOPS mode for group fairness. Following are changes since V3.
> [....]
>
> Just for information: this patchset, applied to 2.6.35-rc6, gives about
> 20-25% increase in speed/throughput on my desktop system
> (Phenom 2.5GHz Quadcore, 3 disks) with the tunables set according
> to what you've used/reported here (the setup with slice_idle set to 0),
> and it's measurable with fs_mark, too.
>
> After 2 hours of hard testing, the machine remains stable and responsive.

Thanks for some testing Heinz. I am assuming you are not using cgroups
and blkio controller.

In that case, you are seeing improvements probably due to first patch
where we don't idle on service tree if slice_idle=0. Hence we cut down on
overall idling and can see throughput incrase.

What kind of configuration these 3 disks are on your system? Some Hardare
RAID or software RAID ?

Thanks
Vivek
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Heinz Diehl on
On 23.07.2010, Vivek Goyal wrote:

> Thanks for some testing Heinz. I am assuming you are not using cgroups
> and blkio controller.

Not at all.

> In that case, you are seeing improvements probably due to first patch
> where we don't idle on service tree if slice_idle=0. Hence we cut down on
> overall idling and can see throughput incrase.

Hmm, in any case it's not getting worse by setting slice_idle to 8.

My main motivation to test your patches was that I thought
the other way 'round, and was just curious on how this patchset
will affect machines which are NOT a high end server/storage system :-)

> What kind of configuration these 3 disks are on your system? Some Hardare
> RAID or software RAID ?

Just 3 SATA disks plugged into the onboard controller, no RAID or whatsoever.

I used fs_mark for testing:
"fs_mark -S 1 -D 10000 -N 100000 -d /home/htd/fsmark/test -s 65536 -t 1 -w 4096 -F"

These are the results with plain cfq (2.6.35-rc6) and the settings which
gave the best speed/throughput on my machine:

low_latency = 0
slice_idle = 4
quantum = 32

Setting slice_idle to 0 didn't improve anything, I tried this before.

FSUse% Count Size Files/sec App Overhead
27 1000 65536 360.3 34133
27 2000 65536 384.4 34657
27 3000 65536 401.1 32994
27 4000 65536 394.3 33781
27 5000 65536 406.8 32569
27 6000 65536 401.9 34001
27 7000 65536 374.5 33192
27 8000 65536 398.3 32839
27 9000 65536 405.2 34110
27 10000 65536 398.9 33887
27 11000 65536 402.3 34111
27 12000 65536 398.1 33652
27 13000 65536 412.9 32443
27 14000 65536 408.1 32197


And this is after applying your patchset, with your settings
(and slice_idle = 0):

FSUse% Count Size Files/sec App Overhead
27 1000 65536 600.7 29579
27 2000 65536 568.4 30650
27 3000 65536 522.0 29171
27 4000 65536 534.1 29751
27 5000 65536 550.7 30168
27 6000 65536 521.7 30158
27 7000 65536 493.3 29211
27 8000 65536 495.3 30183
27 9000 65536 587.8 29881
27 10000 65536 469.9 29602
27 11000 65536 482.7 29557
27 12000 65536 486.6 30700
27 13000 65536 516.1 30243


There's some 2-3% further improvement on my system with these settings,
which after som fiddling turned out to give most performance here
(don't need the group settings, of course):

group_idle = 0
group_isolation = 0
low_latency = 1
quantum = 8
slice_idle = 8

Thanks,
Heinz.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Vivek Goyal on
On Fri, Jul 23, 2010 at 04:56:31PM +0200, Heinz Diehl wrote:
> On 23.07.2010, Vivek Goyal wrote:
>
> > Thanks for some testing Heinz. I am assuming you are not using cgroups
> > and blkio controller.
>
> Not at all.
>
> > In that case, you are seeing improvements probably due to first patch
> > where we don't idle on service tree if slice_idle=0. Hence we cut down on
> > overall idling and can see throughput incrase.
>
> Hmm, in any case it's not getting worse by setting slice_idle to 8.
>
> My main motivation to test your patches was that I thought
> the other way 'round, and was just curious on how this patchset
> will affect machines which are NOT a high end server/storage system :-)
>
> > What kind of configuration these 3 disks are on your system? Some Hardare
> > RAID or software RAID ?
>
> Just 3 SATA disks plugged into the onboard controller, no RAID or whatsoever.
>
> I used fs_mark for testing:
> "fs_mark -S 1 -D 10000 -N 100000 -d /home/htd/fsmark/test -s 65536 -t 1 -w 4096 -F"
>
> These are the results with plain cfq (2.6.35-rc6) and the settings which
> gave the best speed/throughput on my machine:
>
> low_latency = 0
> slice_idle = 4
> quantum = 32
>
> Setting slice_idle to 0 didn't improve anything, I tried this before.
>
> FSUse% Count Size Files/sec App Overhead
> 27 1000 65536 360.3 34133
> 27 2000 65536 384.4 34657
> 27 3000 65536 401.1 32994
> 27 4000 65536 394.3 33781
> 27 5000 65536 406.8 32569
> 27 6000 65536 401.9 34001
> 27 7000 65536 374.5 33192
> 27 8000 65536 398.3 32839
> 27 9000 65536 405.2 34110
> 27 10000 65536 398.9 33887
> 27 11000 65536 402.3 34111
> 27 12000 65536 398.1 33652
> 27 13000 65536 412.9 32443
> 27 14000 65536 408.1 32197
>
>
> And this is after applying your patchset, with your settings
> (and slice_idle = 0):
>
> FSUse% Count Size Files/sec App Overhead
> 27 1000 65536 600.7 29579
> 27 2000 65536 568.4 30650
> 27 3000 65536 522.0 29171
> 27 4000 65536 534.1 29751
> 27 5000 65536 550.7 30168
> 27 6000 65536 521.7 30158
> 27 7000 65536 493.3 29211
> 27 8000 65536 495.3 30183
> 27 9000 65536 587.8 29881
> 27 10000 65536 469.9 29602
> 27 11000 65536 482.7 29557
> 27 12000 65536 486.6 30700
> 27 13000 65536 516.1 30243
>

I think that above improvement is due to first patch and changes in
cfq_should_idle(). cfq_should_idle() used to return 1 even if slice_idle=0
and that created bottlenecks at some places like in select_queue() we
will not expire a queue till request from that queue completed. This
stopped a new queue from dispatching requests etc...

Anyway, for fs_mark problem, can you give following patch a try.

https://patchwork.kernel.org/patch/113061/

Above patch should improve your fs_mark numbers even without setting
slice_idle=0.

Thanks
Vivek
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Heinz Diehl on
On 23.07.2010, Vivek Goyal wrote:

> Anyway, for fs_mark problem, can you give following patch a try.
> https://patchwork.kernel.org/patch/113061/

Ported it to 2.6.35-rc6, and these are my results using the same fs_mark
call as before:

slice_idle = 0

FSUse% Count Size Files/sec App Overhead
28 1000 65536 241.6 39574
28 2000 65536 231.1 39939
28 3000 65536 230.4 39722
28 4000 65536 243.2 39646
28 5000 65536 227.0 39892
28 6000 65536 224.1 39555
28 7000 65536 228.2 39761
28 8000 65536 235.3 39766
28 9000 65536 237.3 40518
28 10000 65536 225.7 39861
28 11000 65536 227.2 39441


slice_idle = 8

FSUse% Count Size Files/sec App Overhead
28 1000 65536 502.2 30545
28 2000 65536 407.6 29406
28 3000 65536 381.8 30152
28 4000 65536 438.1 30038
28 5000 65536 447.5 30477
28 6000 65536 422.0 29610
28 7000 65536 383.1 30327
28 8000 65536 415.3 30102
28 9000 65536 397.6 31013
28 10000 65536 401.4 29201
28 11000 65536 408.8 29720
28 12000 65536 391.2 29157

Huh...there's quite a difference! It's definitely the slice_idle settings
which affect the results here. Besides, this patch gives noticeably bad
desktop interactivity on my system.

Don't know if this is related, but I'm not quite shure if XFS (which I use
exclusively) uses the jbd/jbd2 journaling layer at all.

Thanks,
Heinz.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/