Prev: Another set of ioctl bkl pushdown, almost the end
Next: sched: change nohz idle load balancing logic to push model
From: Zan Lynx on 20 May 2010 20:50
On 5/20/10 5:48 PM, KOSAKI Motohiro wrote:
> Hi
>
> CC to Nick and Jan
>
>> We've seen multiple performance regressions linked to the lower(20%)
>> dirty_ratio. When performing enough IO to overwhelm the background
>> flush daemons the percent of dirty pagecache memory quickly climbs
>> to the new/lower dirty_ratio value of 20%. At that point all writing
>> processes are forced to stop and write dirty pagecache pages back to disk.
>> This causes performance regressions in several benchmarks as well as causing
>> a noticeable overall sluggishness. We all know that the dirty_ratio is
>> an integrity vs performance trade-off but the file system journaling
>> will cover any devastating effects in the event of a system crash.
>>
>> Increasing the dirty_ratio to 40% will regain the performance loss seen
>> in several benchmarks. Whats everyone think about this???
>
> In past, Jan Kara also claim the exactly same thing.
>
> Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
> Date: Wed, 24 Feb 2010 15:34:42 +0100
>
> > (*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be
> > with old kernels because customers running e.g. LDAP (using BerkelyDB
> > heavily) were complaining about performance problems.
>
> So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly
> same distro specific patch. because we can easily imazine other users will face the same
> issue in the future.

On desktop systems the low dirty limits help maintain interactive feel.
Users expect applications that are saving data to be slow. They do not
like it when every application in the system randomly comes to a halt
because of one program stuffing data up to the dirty limit.

The cause and effect for the system slowdown is clear when the dirty
limit is low. "I saved data and now the system is slow until it is
done." When the dirty page ratio is very high, the cause and effect is
disconnected. "I was just web surfing and the system came to a halt."

I think we should expect server admins to do more tuning than desktop
users, so the default limits should stay low in my opinion.

--
Zan Lynx
zlynx(a)acm.org

"Knowledge is Power. Power Corrupts. Study Hard. Be Evil."
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: KOSAKI Motohiro on 20 May 2010 21:20
> > So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly
> > same distro specific patch. because we can easily imazine other users will face the same
> > issue in the future.
>
> On desktop systems the low dirty limits help maintain interactive feel.
> Users expect applications that are saving data to be slow. They do not
> like it when every application in the system randomly comes to a halt
> because of one program stuffing data up to the dirty limit.

really?
Do you mean our per-task dirty limit wouldn't works?

If so, I think we need fix it. IOW sane per-task dirty limitation seems independent issue
from per-system dirty limit.


> The cause and effect for the system slowdown is clear when the dirty
> limit is low. "I saved data and now the system is slow until it is
> done." When the dirty page ratio is very high, the cause and effect is
> disconnected. "I was just web surfing and the system came to a halt."
>
> I think we should expect server admins to do more tuning than desktop
> users, so the default limits should stay low in my opinion.


--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: David Miller on 21 May 2010 02:20
From: Larry Woodman <lwoodman(a)redhat.com>
Date: Thu, 20 May 2010 07:20:42 -0400

> Increasing the dirty_ratio to 40% will regain the performance loss
> seen in several benchmarks. Whats everyone think about this???

I've been making this change via sysctl on every single system I have,
and have been doing so for quite some time.

When doing a lot of GIT operations to a non-SSD disk the kernel simply
can't submit the writes early enough to prevent everything getting
backlogged, and then processes pile up being forced to sleep on I/O
for several seconds at a time.

I therefore totally support making this the default, but I know some
people will be against it :-)

Acked-by: David S. Miller <davem(a)davemloft.net>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Jan Kara on 21 May 2010 12:00
Hi,

On Fri 21-05-10 08:48:57, KOSAKI Motohiro wrote:
> CC to Nick and Jan
Thanks.

> > We've seen multiple performance regressions linked to the lower(20%)
> > dirty_ratio. When performing enough IO to overwhelm the background
> > flush daemons the percent of dirty pagecache memory quickly climbs
> > to the new/lower dirty_ratio value of 20%. At that point all writing
> > processes are forced to stop and write dirty pagecache pages back to disk.
> > This causes performance regressions in several benchmarks as well as causing
> > a noticeable overall sluggishness. We all know that the dirty_ratio is
> > an integrity vs performance trade-off but the file system journaling
> > will cover any devastating effects in the event of a system crash.
> >
> > Increasing the dirty_ratio to 40% will regain the performance loss seen
> > in several benchmarks. Whats everyone think about this???
>
> In past, Jan Kara also claim the exactly same thing.
>
> Subject: [LSF/VM TOPIC] Dynamic sizing of dirty_limit
> Date: Wed, 24 Feb 2010 15:34:42 +0100
>
> > (*) We ended up increasing dirty_limit in SLES 11 to 40% as it used to be
> > with old kernels because customers running e.g. LDAP (using BerkelyDB
> > heavily) were complaining about performance problems.
>
> So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly
> same distro specific patch. because we can easily imazine other users will face the same
> issue in the future.
>
> Acked-by: KOSAKI Motohiro <kosaki.motohiro(a)jp.fujitsu.com>
>
> Nick, Jan, if the above is too old and your distro have been dropped the patch, please
> correct me.
No, SLE11 SP1 still has a patch that increases dirty_ratio to 40. But on
the other hand I agree with Zan that for desktop, 40% of memory for dirty
data is a lot these days and takes a long time to write out (it could
easily be 30s - 1m). On a desktop the memory is much better used as
a read-only pagecache or for memory hungry apps like Firefox or Acrobat
Reader. So I believe for a desktop the current setting (20) is a better
choice. So until we find a way how to dynamically size the dirty limit, we
have to decide whether we want to have a default setting for a server or
for a desktop... Personally, I don't care very much and I feel my time
would be better spent thinking about dynamic limit sizing rather than
arguing what is better default ;).

Honza
--
Jan Kara <jack(a)suse.cz>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Jan Kara on 21 May 2010 12:10
On Fri 21-05-10 10:11:59, KOSAKI Motohiro wrote:
> > > So, I'd prefer to restore the default rather than both Redhat and SUSE apply exactly
> > > same distro specific patch. because we can easily imazine other users will face the same
> > > issue in the future.
> >
> > On desktop systems the low dirty limits help maintain interactive feel.
> > Users expect applications that are saving data to be slow. They do not
> > like it when every application in the system randomly comes to a halt
> > because of one program stuffing data up to the dirty limit.
>
> really?
> Do you mean our per-task dirty limit wouldn't works?
>
> If so, I think we need fix it. IOW sane per-task dirty limitation seems
> independent issue from per-system dirty limit.
Well, I don't know about any per-task dirty limits. What function
implements them? Any application that dirties a single page can be caught
and forced to call balance_dirty_pages() and do writeback.
But generally what we observe on a desktop with lots of dirty memory is
that application needs to allocate memory (either private one or for page
cache) and that triggers direct reclaim so the allocation takes a long
time to finish and thus the application is sluggish...

Honza
--
Jan Kara <jack(a)suse.cz>
SUSE Labs, CR
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: Another set of ioctl bkl pushdown, almost the end
Next: sched: change nohz idle load balancing logic to push model