VM performance issue in KVM guests.
in [Kernel]
|
Prev: Google Summer of Code 2010 - Please ASSIGN (not propose) mentors to the students
Next: Google Summer of Code 2010 - CORRECTION: Only organization admin can assign, so please PROPOSE being a mentor for a student
From: Avi Kivity on 10 Apr 2010 15:30 (copying lkml and some scheduler folk) On 04/10/2010 11:16 AM, Zhang, Xiantao wrote: > Hi, all > We are working on the scalability work for KVM guests, and found one big issue exists in linux scheduler and it may impact guest's performance and scalability a lot for some special workloads running in VM. In the current Linux scheduler, there are some features to enhance App's performance which are defined in the file kvm.git/kernel/sched_features.h. Certainly, they are mostly beneficial optimizations to improve system's performance, but unluckily, some of them may hurt VM's performance and scalablity in KVM case > We know that if two or more vcpus of one guests are scheduled to one same logical processor, same CPU utilization may generate less valid output due mutual lock in VM's OS than that are scheduled to different logical processors .And we also know that VM's vcpus are emulated or executed through the threads of Qemu for KVM. If the vcpu threads of qemu are often pulled to one same logical processor by some features of Linux scheduler, kvm guests'performance may be hurt a lot. In our performance testing, the results also show this performance bottleneck due to this issue. What was the performance hit? What was your I/O setup (image format, using aio?) > After analysis about Linux scheduler, we found it is indeed caused by the known features of Linux schduler, such as AFFINE_WAKEUPS, SYNC_WAKEUPS etc. With these features on, linux schduler often tries to schedule the vcpu threads of one guests to one same logical processor when vcpus are over-committed and logical processors are saturated. Once the vcpu threads of one VM are scheduled to the same LP, system performance drops dramatically with some workloads(like webbench running in windows OS). > Were the affine wakeups due to the kernel (emulated guest IPIs) or qemu? > To verify this finding, we also worked out a simple patch attached in the mail to dynamially switch off the two sheduler features mentioned above when scheduler knows the scheduling tasks are vcpu threads, and we found the the whole system's performance and scalability are improved a lot. Certatinly, this patch is not good for upstream, but it can enlighten us to think how to optimize Linux scheduler and we also want to initiate the discussion about how to make LINUX's scheduler more friendly to virtualization. Besides, this issue maybe not only kvm's special issue, insteadly it should be a common issue for host-based VMs, and we also expect that we can have an elegant solution to thoroughly resolve the performance or scalability gap compared with hypervisor-based VMs. > Most likely it also hits non-virtualized loads as well. If the scheduler pulls two long-running threads to the same cpu, performance will take a hit. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Zhang, Xiantao on 11 Apr 2010 22:10 Avi Kivity wrote: > (copying lkml and some scheduler folk) > > On 04/10/2010 11:16 AM, Zhang, Xiantao wrote: >> Hi, all >> We are working on the scalability work for KVM guests, and found >> one big issue exists in linux scheduler and it may impact guest's >> performance and scalability a lot for some special workloads running >> in VM. In the current Linux scheduler, there are some features to >> enhance App's performance which are defined in the file >> kvm.git/kernel/sched_features.h. Certainly, they are mostly >> beneficial optimizations to improve system's performance, but >> unluckily, some of them may hurt VM's performance and scalablity in >> KVM case We know that if two or more vcpus of one guests are >> scheduled to one same logical processor, same CPU utilization may >> generate less valid output due mutual lock in VM's OS than that are >> scheduled to different logical processors .And we also know that >> VM's vcpus are emulated or executed through the threads of Qemu for >> KVM. If the vcpu threads of qemu are often pulled to one same >> logical processor by some features of Linux scheduler, kvm >> guests'performance may be hurt a lot. In our performance testing, >> the results also show this performance bottleneck due to this issue. > > What was the performance hit? What was your I/O setup (image format, > using aio?) The issue only happens when vcpu number is over-committed(e.g. vcpu/pcpu>2) and physical cpus are saturated. For example, when run webbench in windows OS in this case, its performance drops by 80%. In our experiment, we are using image file through virtio, and I think aio should be used by default also. >> After analysis about Linux scheduler, we found it is indeed caused >> by the known features of Linux schduler, such as AFFINE_WAKEUPS, >> SYNC_WAKEUPS etc. With these features on, linux schduler often tries >> to schedule the vcpu threads of one guests to one same logical >> processor when vcpus are over-committed and logical processors are >> saturated. Once the vcpu threads of one VM are scheduled to the same >> LP, system performance drops dramatically with some workloads(like >> webbench running in windows OS). >> > > Were the affine wakeups due to the kernel (emulated guest IPIs) or > qemu? We have basic guesses about the reasone, one is wakeup affinity between vcpu threads due to IPI, and the other is wakeup affinity between io theads and vcpu threads. >> To verify this finding, we also worked out a simple patch >> attached in the mail to dynamially switch off the two sheduler >> features mentioned above when scheduler knows the scheduling tasks >> are vcpu threads, and we found the the whole system's performance >> and scalability are improved a lot. Certatinly, this patch is not >> good for upstream, but it can enlighten us to think how to optimize >> Linux scheduler and we also want to initiate the discussion about >> how to make LINUX's scheduler more friendly to virtualization. >> Besides, this issue maybe not only kvm's special issue, insteadly it >> should be a common issue for host-based VMs, and we also expect that >> we can have an elegant solution to thoroughly resolve the >> performance or scalability gap compared with hypervisor-based VMs. >> > > Most likely it also hits non-virtualized loads as well. If the > scheduler pulls two long-running threads to the same cpu, performance > will take a hit. Since the hit only happens when physical cpus are saturated, and sheduling non-virtualized multiple threads of one process to same processor can benefit the performance due to cache share or other affinities, but you know it hurts performance a lot once schedule two vcpu theads to a same processor due to mutual spin-lock in guests. Xiantao -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 12 Apr 2010 02:50 On 04/12/2010 05:04 AM, Zhang, Xiantao wrote: > >> What was the performance hit? What was your I/O setup (image format, >> using aio?) >> > The issue only happens when vcpu number is over-committed(e.g. vcpu/pcpu>2) and physical cpus are saturated. For example, when run webbench in windows OS in this case, its performance drops by 80%. In our experiment, we are using image file through virtio, and I think aio should be used by default also. > Is this on a machine that does pause-loop exits? The current handing of PLE is very suboptimal. With proper directed yield we should be much better there. Without PLE, we need paravirtualized spinlocks, no way around it. >>> After analysis about Linux scheduler, we found it is indeed caused >>> by the known features of Linux schduler, such as AFFINE_WAKEUPS, >>> SYNC_WAKEUPS etc. With these features on, linux schduler often tries >>> to schedule the vcpu threads of one guests to one same logical >>> processor when vcpus are over-committed and logical processors are >>> saturated. Once the vcpu threads of one VM are scheduled to the same >>> LP, system performance drops dramatically with some workloads(like >>> webbench running in windows OS). >>> >>> >> Were the affine wakeups due to the kernel (emulated guest IPIs) or >> qemu? >> > We have basic guesses about the reasone, one is wakeup affinity between vcpu threads due to IPI, and the other is wakeup affinity between io theads and vcpu threads. > It would be good to find out. >> Most likely it also hits non-virtualized loads as well. If the >> scheduler pulls two long-running threads to the same cpu, performance >> will take a hit. >> > Since the hit only happens when physical cpus are saturated, and sheduling non-virtualized multiple threads of one process to same processor can benefit the performance due to cache share or other affinities, but you know it hurts performance a lot once schedule two vcpu theads to a same processor due to mutual spin-lock in guests. > Spin loops need to be addressed first, they are known to kill performance in overcommit situations. -- I have a truly marvellous patch that fixes the bug which this signature is too narrow to contain. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Zhang, Xiantao on 12 Apr 2010 21:00 Avi Kivity wrote: > On 04/12/2010 05:04 AM, Zhang, Xiantao wrote: >> >>> What was the performance hit? What was your I/O setup (image >>> format, using aio?) >>> >> The issue only happens when vcpu number is over-committed(e.g. >> vcpu/pcpu>2) and physical cpus are saturated. For example, when run >> webbench in windows OS in this case, its performance drops by 80%. >> In our experiment, we are using image file through virtio, and I >> think aio should be used by default also. >> > > Is this on a machine that does pause-loop exits? The current handing > of PLE is very suboptimal. With proper directed yield we should be > much better there. > > Without PLE, we need paravirtualized spinlocks, no way around it. PLE has the ability to eliminate the issue at some extent, and pv solution should be helpful also. But for windows guests running on machines without PLE, we still needs to enhance host side to resolve the issue. >>>> After analysis about Linux scheduler, we found it is indeed caused >>>> by the known features of Linux schduler, such as AFFINE_WAKEUPS, >>>> SYNC_WAKEUPS etc. With these features on, linux schduler often >>>> tries to schedule the vcpu threads of one guests to one same >>>> logical processor when vcpus are over-committed and logical >>>> processors are saturated. Once the vcpu threads of one VM are >>>> scheduled to the same LP, system performance drops dramatically >>>> with some workloads(like webbench running in windows OS). >>> >> Since the hit only happens when physical cpus are saturated, and >> sheduling non-virtualized multiple threads of one process to same >> processor can benefit the performance due to cache share or other >> affinities, but you know it hurts performance a lot once schedule >> two vcpu theads to a same processor due to mutual spin-lock in >> guests. >> > Spin loops need to be addressed first, they are known to kill > performance in overcommit situations. Even in overcommit case, if vcpu threads of one qemu are not scheduled or pulled to the same logical processor, the performance drop is tolerant like Xen's case today. But for KVM, it has to suffer from additional performance loss, since host's scheduler actively pulls these vcpu threads together. Xiantao -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 13 Apr 2010 02:50
On 04/13/2010 03:50 AM, Zhang, Xiantao wrote: > Avi Kivity wrote: > >> On 04/12/2010 05:04 AM, Zhang, Xiantao wrote: >> >>> >>>> What was the performance hit? What was your I/O setup (image >>>> format, using aio?) >>>> >>>> >>> The issue only happens when vcpu number is over-committed(e.g. >>> vcpu/pcpu>2) and physical cpus are saturated. For example, when run >>> webbench in windows OS in this case, its performance drops by 80%. >>> In our experiment, we are using image file through virtio, and I >>> think aio should be used by default also. >>> >>> >> Is this on a machine that does pause-loop exits? The current handing >> of PLE is very suboptimal. With proper directed yield we should be >> much better there. >> >> Without PLE, we need paravirtualized spinlocks, no way around it. >> > PLE has the ability to eliminate the issue at some extent, and pv solution should be helpful also. But for windows guests running on machines without PLE, we still needs to enhance host side to resolve the issue. > Well, was this on a machine with PLE or without PLE? >> Spin loops need to be addressed first, they are known to kill >> performance in overcommit situations. >> > Even in overcommit case, if vcpu threads of one qemu are not scheduled or pulled to the same logical processor, the performance drop is tolerant like Xen's case today. But for KVM, it has to suffer from additional performance loss, since host's scheduler actively pulls these vcpu threads together. > > Can you quantify this loss? Give examples of what happens? -- Do not meddle in the internals of kernels, for they are subtle and quick to panic. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |