VM performance issue in KVM guests.
in [Kernel]
|
Prev: Google Summer of Code 2010 - Please ASSIGN (not propose) mentors to the students
Next: Google Summer of Code 2010 - CORRECTION: Only organization admin can assign, so please PROPOSE being a mentor for a student
From: Zhang, Xiantao on 13 Apr 2010 23:30 Avi Kivity wrote: > On 04/13/2010 03:50 AM, Zhang, Xiantao wrote: >> Avi Kivity wrote: >> >>> On 04/12/2010 05:04 AM, Zhang, Xiantao wrote: >>> >>>> >>>>> What was the performance hit? What was your I/O setup (image >>>>> format, using aio?) >>>>> >>>>> >>>> The issue only happens when vcpu number is over-committed(e.g. >>>> vcpu/pcpu>2) and physical cpus are saturated. For example, when >>>> run webbench in windows OS in this case, its performance drops by >>>> 80%. In our experiment, we are using image file through virtio, >>>> and I think aio should be used by default also. >>>> >>>> >>> Is this on a machine that does pause-loop exits? The current >>> handing of PLE is very suboptimal. With proper directed yield we >>> should be much better there. >>> >>> Without PLE, we need paravirtualized spinlocks, no way around it. >>> >> PLE has the ability to eliminate the issue at some extent, and pv >> solution should be helpful also. But for windows guests running on >> machines without PLE, we still needs to enhance host side to resolve >> the issue. >> > > Well, was this on a machine with PLE or without PLE? I am saying the machine has no PLE feature support. Even with PLE feature support, there is still performance loss due to PLE's cost. >>> Spin loops need to be addressed first, they are known to kill >>> performance in overcommit situations. >>> >> Even in overcommit case, if vcpu threads of one qemu are not >> scheduled or pulled to the same logical processor, the performance >> drop is tolerant like Xen's case today. But for KVM, it has to >> suffer from additional performance loss, since host's scheduler >> actively pulls these vcpu threads together. >> >> > Can you quantify this loss? Give examples of what happens? For example, one machine is configured with 2 pCPUs and there are two Windows guests running on the machine, and each guest is cconfigured with 2 vcpus and one webbench server runs in it. If use host's default scheduler, webbench's performance is very bad, but if pin each geust's vCPU0 to pCPU0 and vCPU1 to pCPU1, we can see 5-10X performance improvement with same CPU utilization. In addition, we also see kvm's perf scalability is also impacted in large systems, for some performance experiments, kvm's perf begins to drop when vCPU is overcommitted and pCPU are saturated, but once the wake_up_affine feature is switched off in scheduler, kvm's perf can keep rising in this case. Xiantao -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 14 Apr 2010 04:20 On 04/14/2010 06:24 AM, Zhang, Xiantao wrote: > >>>> Spin loops need to be addressed first, they are known to kill >>>> performance in overcommit situations. >>>> >>>> >>> Even in overcommit case, if vcpu threads of one qemu are not >>> scheduled or pulled to the same logical processor, the performance >>> drop is tolerant like Xen's case today. But for KVM, it has to >>> suffer from additional performance loss, since host's scheduler >>> actively pulls these vcpu threads together. >>> >>> >>> >> Can you quantify this loss? Give examples of what happens? >> > For example, one machine is configured with 2 pCPUs and there are two Windows guests running on the machine, and each guest is cconfigured with 2 vcpus and one webbench server runs in it. > If use host's default scheduler, webbench's performance is very bad, but if pin each geust's vCPU0 to pCPU0 and vCPU1 to pCPU1, we can see 5-10X performance improvement with same CPU utilization. > In addition, we also see kvm's perf scalability is also impacted in large systems, for some performance experiments, kvm's perf begins to drop when vCPU is overcommitted and pCPU are saturated, but once the wake_up_affine feature is switched off in scheduler, kvm's perf can keep rising in this case. > Ok. This is probably due to spinlock contention. When vcpus are pinned to pcpus, there is a 50% chance that a guest's vcpus will be co-scheduled and spinlocks will perform will. When vcpus are not pinned, but affine wakeups are disabled, there is a 33% chance that vcpus will be co-scheduled. When vcpus are not pinned and affine wakeups are enabled there is a 0% chance that vcpus will be co-scheduled. Keeping both vcpus on the same core actually makes sense since they can communicate through the local cache faster than across cores. What we need is to make sure that they don't spin. Windows 2008 can report spinlock spinning through a hypercall. Can you hook to that interface and see if it happens regularly? Altenatively use a PLE capable host and trace the kvm_vcpu_on_spin() function. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 15 Apr 2010 04:20 On 04/15/2010 07:58 AM, Srivatsa Vaddagiri wrote: > On Sun, Apr 11, 2010 at 11:40 PM, Avi Kivity <avi(a)redhat.com > <mailto:avi(a)redhat.com>> wrote: > > The current handing of PLE is very suboptimal. With proper > directed yield we should be much better there. > > > > Hi Avi, > By directed yield, do you mean transfer the timeslice of > one thread (which is contending for a lock) to another thread (which > is holding a lock)? It's a priority transfer (in CFS terms, vruntime) (we don't know who holds the lock, so we pick a co-vcpu at random). > If at that point in time, the lock-holder thread/VCPU is actually not > running currently, ie it is at the back of the runqueue, would it help > much? In such case, it will take time for the lock holder to run again > and the default timeslice it would have got could have been sufficient > to release the lock? The idea is to increase the chances to the target vcpu to run, and to decrease the changes of the spinner to run (hopefully they change places). > > I am also working on a prototype for some other technique here - to > avoid preempting guest threads/VCPUs in the middle of their > (spin-lock) critical section. This requires guest to hint host when > there are in such a section. [1] has shown 33% improvement to an > apache benchmark based on this idea. > Certainly that has even greater potential for Linux guests. Note that we spin on mutexes now, so we need to prevent preemption while the lock owner is running. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Peter Zijlstra on 15 Apr 2010 09:40 On Thu, 2010-04-15 at 11:18 +0300, Avi Kivity wrote: > > Certainly that has even greater potential for Linux guests. Note that > we spin on mutexes now, so we need to prevent preemption while the lock > owner is running. either that, or disable spinning on (para) virt kernels. Para virt kernels could possibly extend the thing by also checking to see if the owner's vcpu is running. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Srivatsa Vaddagiri on 15 Apr 2010 12:50
On Thu, Apr 15, 2010 at 03:33:18PM +0200, Peter Zijlstra wrote: > On Thu, 2010-04-15 at 11:18 +0300, Avi Kivity wrote: > > > > Certainly that has even greater potential for Linux guests. Note that > > we spin on mutexes now, so we need to prevent preemption while the lock > > owner is running. > > either that, or disable spinning on (para) virt kernels. Para virt > kernels could possibly extend the thing by also checking to see if the > owner's vcpu is running. I suspect we will need a combination of both approaches, given that we will not be able to avoid preempting guests in their critical section always (too long critical sections or real-time tasks wanting to preempt). Other idea is to gang-schedule VCPUs of the same guest as much as possible? - vatsa -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |