Unify KVM kernel-space and user-space code into a single project
in [Kernel]
|
Prev: Irish 2010 Grant Winner
Next: [PATCH] staging: winbond: mds_f.h whitespace and CamelCase corrections.
From: Ingo Molnar on 18 Mar 2010 06:20 * Alexander Graf <agraf(a)suse.de> wrote: > > On 18.03.2010, at 09:56, Ingo Molnar wrote: > > > > > * Avi Kivity <avi(a)redhat.com> wrote: > > > >> On 03/17/2010 10:10 AM, Ingo Molnar wrote: > >>> > >>>> It's about who owns the user interface. > >>>> > >>>> If qemu owns the user interface, than we can satisfy this in a very > >>>> simple way by adding a perf monitor command. If we have to support third > >>>> party tools, then it significantly complicates things. > >>> > >>> Of course illogical modularization complicates things 'significantly'. > >> > >> Who should own the user interface then? > > > > If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or > > series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and > > tools/perf/. > > It's not a 1:1 connection. There are more users of the KVM interface. To > name a few I'm aware of: > > - Mac-on-Linux (PPC) > - Dolphin (PPC) > - Xenner (x86) > - Kuli (s390) There must be a misunderstanding here: tools/perf/ still has a clear userspace interface and ABI. There's external projects making use of it: sysprof and libpfm (and probably more i dont know about it). Those projects are also contributing back. Still it's _very_ useful to have a single reference implementation under tools/perf/ where we concentrate the best of the code. That is where we make sure that each new kernel feature is appropriately implemented in user-space as well, that the combination works well together and is releasable to users. That is what keeps us all honest: the latency of features is much lower, and there's no ping-pong of blame going on between the two components in case of bugs or in case of misfeatures. Same goes for KVM+Qemu: it would be so much nicer to have a single, well-focused reference implementation under tools/kvm/ and have improvements flow into that code base. That way KVM developers cannot just shrug "well, GUI suckage is a user-space problem" - like the answers i got in the KVM usability thread ... The buck will stop here. And if someone thinks he can do better an external project can be started anytime. (it may even replace the upstream thing if it's better) > Having a clear userspace interface is the only viable solution there. And if > you're interested, look at my MOL enabling patch. It's less than 500 lines > of code. Why do you suppose that what i propose is an "either or" scenario? It isnt. I just suggested that instead of letting core KVM fragment its limbs into an external entity, put your name behind one good all-around solution and focus the development model into a single project. I.e. do what KVM has done originally in the kernel space to begin with - and where it was so much better than Xen: single focus. Learn from what KVM has done so well in the initial years and use the concept on the user-space components as well. The very same arguments that caused KVM to integrate into the upstream kernel (instead of being a separate project) are a valid basis to integrate the user-space components into tools/kvm/. Dont forget your roots and dont assume all your design decisions were correct. > The kernel/userspace interface really isn't the difficult part. Getting > device emulation working properly, easily and fast is. The kernel/userspace ABI is not difficult at all. Getting device emulation working properly, easily and fast indeed is. And my experience is that it is not working properly nor quickly at the moment, at all. (see the 'KVM usability' thread) Getting device emulation working properly often involves putting certain pieces that are currently done in Qemu into kernel-space. That kind of 'movement of emulation technology' from user-space component into the kernel-space component [or back] would very clearly be helped if those two components were in the same repository. And i have first-hand experience there: we had (and have) similar scenarios with tools/perf routinely. We did some aspects in user-space, then decided to do it in kernel-space. Sometimes we moved kernel bits to user-space. It was very easy and there were no package and version complications as it's a single project. Sometimes we even moved bits back and forth until we found the right balance. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 18 Mar 2010 06:20 On 03/18/2010 10:56 AM, Ingo Molnar wrote: > * Avi Kivity<avi(a)redhat.com> wrote: > > >> On 03/17/2010 10:10 AM, Ingo Molnar wrote: >> >>> >>>> It's about who owns the user interface. >>>> >>>> If qemu owns the user interface, than we can satisfy this in a very >>>> simple way by adding a perf monitor command. If we have to support third >>>> party tools, then it significantly complicates things. >>>> >>> Of course illogical modularization complicates things 'significantly'. >>> >> Who should own the user interface then? >> > If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or > series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and > tools/perf/. > We would have exactly the same issues, only they would be in a single repository. The only difference is that we could ignore potential alternatives to qemu, libvirt, and RHEV-M. But that's not how kernel ABIs are developed, we try to make them general, not suited to just one consumer that happens to be close to our heart. > Numerous times did we have patches to kernel/perf_event.c that fixed some > detail, also accompanied by a tools/perf/ patch fixing another detail. Having > a single 'culture of contribution' is a powerful way to develop. > In fact kvm started out in a single repo, and it certainly made it easy to bring it up in baby steps. But we've long outgrown that. Maybe the difference is that perf is still new and thus needs tight cooperation. If/when perf gains a real GUI, I doubt more than 1% of the patches will touch both kernel and userspace. > It turns out kernel developers can be pretty good user-space developers as > well and user-space developers can be pretty good kernel developers as well. > Some like to do both - as long as it's all within a single project. > Very childish of them. If someone wants to contribute to a userspace project, they can swallow their pride and send patches to a non-kernel mailing list and repository. > The moment any change (be it as trivial as fixing a GUI detail or as complex > as a new feature) involves two or more packages, development speed slows down > to a crawl - while the complexity of the change might be very low! > Why is that? I the maintainers of all packages are cooperative and responsive, then the patches will get accepted quickly. If they aren't, development will be slow. It isn't any different from contributing to two unrelated kernel subsystems (which are in fact in different repositories until the next merge window). > Also, there's the harmful process that people start categorizing themselves > into 'I am a kernel developer' and 'I am a user space programmer' stereotypes, > which limits the scope of contributions artificially. > You're encouraging this with your proposal. You're basically using the glory of kernel development to attract people to userspace. >>> Fast forward to 2010. The kernel side of KVM is maximum goodness - by far >>> the worst-quality remaining aspects of KVM are precisely in areas that you >>> mention: 'if we have to support third party tools, then it significantly >>> complicates things'. You kept Qemu as an external 'third party' entity to >>> KVM, and KVM is clearly hurting from that - just see the recent KVM >>> usability thread for examples about suckage. >>> >> Any qemu usability problems are because developers (or their employers) are >> not interested in fixing them, not because of the repository location. Most >> kvm developer interest is in server-side deployment (even for desktop >> guests), so there is limited effort in implementing a virtualbox-style GUI. >> > The same has been said of oprofile as well: 'it somewhat sucks because we are > too server centric', 'nobody is interested in good usability and oprofile is > fine for the enterprises'. Ironically, the same has been said of Xen usability > as well, up to the point KVM came around. > > What was the core of the problem was a bad design and a split kernel-side > user-side tool landscape. > I can accept the bad design (not knowing any of the details), but how can the kernel/user split affect usability? > In fact i think saying that 'our developers only care about the server' is > borderline dishonest, when at the same time you are making it doubly sure (by > inaction) that it stays so: by leaving an artificial package wall between > kernel-side KVM and user-side KVM and not integrating the two technologies. > The wall is maybe four nanometers high. Please be serious. If someone wants to work on qemu usability all they have to do is to clone the repository and start sending patches to qemu-devel@. What's gained by putting it in the kernel repository? You're saving a minute's worth of clone, and that only for people who already happen to be kernel developers. > You'll never know what heights you could achieve if you leave that wall there > ... > I truly don't know. What highly usable GUIs were developed in the kernel? > Furthermore, what should be realized is that bad usability hurts "server > features" just as much. Most of the day-to-day testing is done on the desktop > by desktop oriented testers/developers. _Not_ by enterprise shops - they tend > to see the code years down the line to begin with ... > > Yes, a particular feature might be server oriented, but a good portion of our > testing is on the desktop and everyone is hurting from bad usability and this > puts limits on contribution efficiency. > I'm not saying that improved usability isn't a good thing, but time spent on improving the GUI is time not spent on the features that we really want. Desktop oriented users also rarely test 16 vcpu guests with tons of RAM exercising 10Gb NICs and a SAN. Instead they care about graphics performance for 2vcpu/1GB guests. > As the patch posted in _this very thread demonstrates it_, it is doubly more > difficult to contribute a joint KVM+Qemu feature, because it's two separate > code bases, two contribution guidelines, two release schedules. While to the > user it really is just one and the same thing. It should be so for the > developer as well. > It's hard to contribute a patch that goes against the architecture of the system, where kvm deals with cpu virtualization, qemu (or theoretically another tool) manages a guest, and libvirt (or another tool) manages the host. You want a list of guests to be provided by qemu or the kernel, and that simply isn't how the system works. > Put in another way: KVM's current split design is making it easy to contribute > server features (because the kernel side is clean and cool), but also makes it > artificially hard to contribute desktop features: because the tooling side > (Qemu) is 'just another package', is separated by a package and maintenance > wall Most server oriented patches in qemu/kvm have gone into qemu, not kvm (simply because it sees many more patches overall). It isn't hard to contribute to 'just another package', I have 1700 packages installed on my desktop and only one of them is a kernel. Anyway your arguments apply equally well to gedit. > and is made somewhat uncool by a (as some KVM developers have pointed out > in this thread) quirky codebase. > The qemu codebase is in fact quirky, but cp won't solve it. Only long patchsets to qemu-devel@. > (the rest of your points are really a function of this fundamental > disagreement) > I disagree. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar on 18 Mar 2010 06:30 * Avi Kivity <avi(a)redhat.com> wrote: > On 03/18/2010 10:56 AM, Ingo Molnar wrote: > >* Avi Kivity<avi(a)redhat.com> wrote: > > > >>On 03/17/2010 10:10 AM, Ingo Molnar wrote: > >>>>It's about who owns the user interface. > >>>> > >>>>If qemu owns the user interface, than we can satisfy this in a very > >>>>simple way by adding a perf monitor command. If we have to support third > >>>>party tools, then it significantly complicates things. > >>>Of course illogical modularization complicates things 'significantly'. > >>Who should own the user interface then? > >If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or > >series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and > >tools/perf/. > > We would have exactly the same issues, only they would be in a single > repository. The only difference is that we could ignore potential > alternatives to qemu, libvirt, and RHEV-M. But that's not how kernel ABIs > are developed, we try to make them general, not suited to just one consumer > that happens to be close to our heart. Not at all - as i replied to in a previous mail, tools/perf/ still has a clear userspace interface and ABI, and external projects are making use of it. So there's no problem with the ABI at all. In fact our experience has been the opposite: the perf ABI is markedly better _because_ there's an immediate consumer of it in the form of tools/perf/. It gets tested better and external projects can get their ABI tweaks in as well and can provide a reference implementation for tools/perf. This has happened a couple of times. It's a win-win scenario. So the exact opposite of what you suggest is happening in practice. Thanks, Ingo -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 18 Mar 2010 06:30 On 03/18/2010 12:10 PM, Ingo Molnar wrote: > >> It's not a 1:1 connection. There are more users of the KVM interface. To >> name a few I'm aware of: >> >> - Mac-on-Linux (PPC) >> - Dolphin (PPC) >> - Xenner (x86) >> - Kuli (s390) >> > There must be a misunderstanding here: tools/perf/ still has a clear userspace > interface and ABI. There's external projects making use of it: sysprof and > libpfm (and probably more i dont know about it). Those projects are also > contributing back. > So it seems it is possible to scale the package wall. > Still it's _very_ useful to have a single reference implementation under > tools/perf/ where we concentrate the best of the code. That is where we make > sure that each new kernel feature is appropriately implemented in user-space > as well, that the combination works well together and is releasable to users. > That is what keeps us all honest: the latency of features is much lower, and > there's no ping-pong of blame going on between the two components in case of > bugs or in case of misfeatures. > That would make sense for a truly minimal userspace for kvm: we once had a tool called kvmctl which was used to run tests (since folded into qemu). It didn't contain a GUI and was unable to run a general purpose guest. It was a few hundred lines of code, and indeed patches to kvmctl had a much closer correspondence to patches with kvm (though still low, as most kvm patches don't modify the ABI). > Same goes for KVM+Qemu: it would be so much nicer to have a single, > well-focused reference implementation under tools/kvm/ and have improvements > flow into that code base. > > That way KVM developers cannot just shrug "well, GUI suckage is a user-space > problem" - like the answers i got in the KVM usability thread ... > > The buck will stop here. > Suppose we copy qemu tomorrow into tools/. All the problems will be copied with it. Someone still has to write patches to fix them. Who will it be? >> The kernel/userspace interface really isn't the difficult part. Getting >> device emulation working properly, easily and fast is. >> > The kernel/userspace ABI is not difficult at all. Getting device emulation > working properly, easily and fast indeed is. And my experience is that it is > not working properly nor quickly at the moment, at all. (see the 'KVM > usability' thread) > > Getting device emulation working properly often involves putting certain > pieces that are currently done in Qemu into kernel-space. That kind of > 'movement of emulation technology' from user-space component into the > kernel-space component [or back] would very clearly be helped if those two > components were in the same repository. > Moving emulation into the kernel is indeed a problem. Not because it's difficult, but because it indicates that the interfaces exposed to userspace are insufficient to obtain good performance. We had that with vhost-net and I'm afraid we'll have that with vhost-blk. > And i have first-hand experience there: we had (and have) similar scenarios > with tools/perf routinely. We did some aspects in user-space, then decided to > do it in kernel-space. Sometimes we moved kernel bits to user-space. It was > very easy and there were no package and version complications as it's a single > project. Sometimes we even moved bits back and forth until we found the right > balance. > That's reasonable in the first iterations of a project. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Avi Kivity on 18 Mar 2010 06:40
On 03/18/2010 11:22 AM, Ingo Molnar wrote: > * Avi Kivity<avi(a)redhat.com> wrote: > > >>> - move a clean (and minimal) version of the Qemu code base to tools/kvm/, >>> in the upstream kernel repo, and work on that from that point on. >>> >> I'll ignore the repository location which should be immaterial to a serious >> developer and concentrate on the 'clean and minimal' aspect, since it has >> some merit. [...] >> > To the contrary, experience shows that repository location, and in particular > a shared repository for closely related bits is very much material! > > It matters because when there are two separate projects, even a "serious > developer" is finding it double and triple difficult to contribute even > trivial changes. > > It becomes literally a nightmare if you have to touch 3 packages: kernel, a > library and an app codebase. It takes _forever_ to get anything useful done. > You can't be serious. I find that the difficulty in contributing a patch has mostly to do with writing the patch, and less with figuring out which email address to send it to. > Also, 'focus on a single thing' is a very basic aspect of humans, especially > those who do computer programming. Working on two code bases in two > repositories at once can be very challenging physically and psychically. > Indeed, working simultaneously on two different projects is difficult. I usually work for a while on one, and then 'cd', physically and psychically, to the other. Then switch back. Sort of like the scheduler on a uniprocessor machine. > So what i've seen is that OSS programmers tend to pick a side, pretty much > randomly, and then rationalize it in hindsight why they prefer that side ;-) > > Most of them become either a kernel developer or a user-space package > developer - and then they specialize on that field and shy away from changes > that involve both. It's a basic human thing to avoid the hassle that comes > with multi-package changes. (One really has to be outright stupid, fanatic or > desperate to even attempt such changes these days - such are the difficulties > for a comparatively low return.) > We have a large number of such stupid, fanatic, desperate developers in the qemu and kvm communities. > The solution is to tear down such artificial walls of contribution where > possible. And tearing down the wall between KVM and qemu-kvm seems very much > possible and the advantages would be numerous. > > Unless by "serious developer" you meant: "developer willing to [or forced to] > waste time and effort on illogically structured technology". > By "serious developer" I mean - someone who is interested in contributing, not in getting their name into the kernel commits list - someone who is willing to read the wiki page and find out where the repository and mailing list for a project is - someone who will spend enough time on the project so that the time to clone two repositories will not be a factor in their contributions - someone who will work on the uncool stuff like fixing bugs and providing interfaces to other tools >> [...] >> >> Do you really think the echo'n'cat school of usability wants to write a GUI? >> In linux-2.6.git? >> > Then you'll be surprised to hear that it's happening as we speak and the > commits are there in linux-2.6.git. Both a TUI and GUI is in the works. > > Furthermore, the numbers show that half of the usability fixes to tools/perf/ > came not from regular perf contributors but from random kernel developers and > testers who when they build the latest kernel and try out perf at the same > time (it's very easy because you already have it in the kernel repository - no > separate download, no installation, etc. necessary). > > I had literally zero such contributions when (the precursor to) 'perf' was > still a separate user-space project. > > You could have the same effect for Qemu: the latest bits in tools/kvm/ would > be built by regular kernel testers and developers. The integration benefits > dont just extend to developers, a unified project is vastly easier to test as > well. > > Let's wait and see then. If the tools/perf/ experience has really good results, we can reconsider this at a later date. -- error compiling committee.c: too many arguments to function -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |