From: Andi Kleen on
Gleb Natapov <gleb(a)redhat.com> writes:
>
> The patch below allows to patch ticket spinlock code to behave similar to
> old unfair spinlock when hypervisor is detected. After patching unlocked

The question is what happens when you have a system with unfair
memory and you run the hypervisor on that. There it could be much worse.

Your new code would starve again, right?

There's a reason the ticket spinlocks were added in the first place.

-Andi

--
ak(a)linux.intel.com -- Speaking for myself only.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Gleb Natapov on
On Tue, Jun 01, 2010 at 05:53:09PM +0200, Andi Kleen wrote:
> Gleb Natapov <gleb(a)redhat.com> writes:
> >
> > The patch below allows to patch ticket spinlock code to behave similar to
> > old unfair spinlock when hypervisor is detected. After patching unlocked
>
> The question is what happens when you have a system with unfair
> memory and you run the hypervisor on that. There it could be much worse.
>
How much worse performance hit could be?

> Your new code would starve again, right?
>
Yes, of course it may starve with unfair spinlock. Since vcpus are not
always running there is much smaller chance then vcpu on remote memory
node will starve forever. Old kernels with unfair spinlocks are running
fine in VMs on NUMA machines with various loads.

> There's a reason the ticket spinlocks were added in the first place.
>
I understand that reason and do not propose to get back to old spinlock
on physical HW! But with virtualization performance hit is unbearable.

--
Gleb.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Andi Kleen on
On Tue, Jun 01, 2010 at 07:24:14PM +0300, Gleb Natapov wrote:
> On Tue, Jun 01, 2010 at 05:53:09PM +0200, Andi Kleen wrote:
> > Gleb Natapov <gleb(a)redhat.com> writes:
> > >
> > > The patch below allows to patch ticket spinlock code to behave similar to
> > > old unfair spinlock when hypervisor is detected. After patching unlocked
> >
> > The question is what happens when you have a system with unfair
> > memory and you run the hypervisor on that. There it could be much worse.
> >
> How much worse performance hit could be?

It depends on the workload. Overall it means that a contended
lock can have much higher latencies.

If you want to study some examples see the locking problems the
RT people have with their heavy weight mutex-spinlocks.

But the main problem is that in the worst case you
can see extremly long stalls (upto a second has been observed),
which then turns in a correctness issue.
>
> > Your new code would starve again, right?
> >
> Yes, of course it may starve with unfair spinlock. Since vcpus are not
> always running there is much smaller chance then vcpu on remote memory
> node will starve forever. Old kernels with unfair spinlocks are running
> fine in VMs on NUMA machines with various loads.

Try it on a NUMA system with unfair memory.

> > There's a reason the ticket spinlocks were added in the first place.
> >
> I understand that reason and do not propose to get back to old spinlock
> on physical HW! But with virtualization performance hit is unbearable.

Extreme unfairness can be unbearable too.

-Andi
--
ak(a)linux.intel.com -- Speaking for myself only.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on
On 06/01/2010 07:38 PM, Andi Kleen wrote:
>>> Your new code would starve again, right?
>>>
>>>
>> Yes, of course it may starve with unfair spinlock. Since vcpus are not
>> always running there is much smaller chance then vcpu on remote memory
>> node will starve forever. Old kernels with unfair spinlocks are running
>> fine in VMs on NUMA machines with various loads.
>>
> Try it on a NUMA system with unfair memory.
>

We are running everything on NUMA (since all modern machines are now
NUMA). At what scale do the issues become observable?

>> I understand that reason and do not propose to get back to old spinlock
>> on physical HW! But with virtualization performance hit is unbearable.
>>
> Extreme unfairness can be unbearable too.
>

Well, the question is what happens first. In our experience, vcpu
overcommit is a lot more painful. People will never see the NUMA
unfairness issue if they can't use kvm due to the vcpu overcommit problem.

What I'd like to see eventually is a short-term-unfair, long-term-fair
spinlock. Might make sense for bare metal as well. But it won't be
easy to write.

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Andi Kleen on
On Tue, Jun 01, 2010 at 07:52:28PM +0300, Avi Kivity wrote:
> We are running everything on NUMA (since all modern machines are now NUMA).
> At what scale do the issues become observable?

On Intel platforms it's visible starting with 4 sockets.

>
>>> I understand that reason and do not propose to get back to old spinlock
>>> on physical HW! But with virtualization performance hit is unbearable.
>>>
>> Extreme unfairness can be unbearable too.
>>
>
> Well, the question is what happens first. In our experience, vcpu
> overcommit is a lot more painful. People will never see the NUMA
> unfairness issue if they can't use kvm due to the vcpu overcommit problem.

You really have to address both, if you don't fix them both
users will eventually into one of them and be unhappy.

-Andi

--
ak(a)linux.intel.com -- Speaking for myself only.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/