From: Avi Kivity on
On 03/17/2010 10:10 AM, Ingo Molnar wrote:
>
>> It's about who owns the user interface.
>>
>> If qemu owns the user interface, than we can satisfy this in a very simple
>> way by adding a perf monitor command. If we have to support third party
>> tools, then it significantly complicates things.
>>
> Of course illogical modularization complicates things 'significantly'.
>

Who should own the user interface then?

> Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the
> worst-quality remaining aspects of KVM are precisely in areas that you
> mention: 'if we have to support third party tools, then it significantly
> complicates things'. You kept Qemu as an external 'third party' entity to KVM,
> and KVM is clearly hurting from that - just see the recent KVM usability
> thread for examples about suckage.
>

Any qemu usability problems are because developers (or their employers)
are not interested in fixing them, not because of the repository
location. Most kvm developer interest is in server-side deployment
(even for desktop guests), so there is limited effort in implementing a
virtualbox-style GUI.

> - move a clean (and minimal) version of the Qemu code base to tools/kvm/, in
> the upstream kernel repo, and work on that from that point on.
>

I'll ignore the repository location which should be immaterial to a
serious developer and concentrate on the 'clean and minimal' aspect,
since it has some merit. Qemu development does have a tension between
the needs of kvm and tcg. For kvm we need fine-grained threading to
improve performance and tons of RAS work. For tcg these are mostly
meaningless, and the tcg code has sufficient inertia to reduce the rate
at which we can develop.

Nevertheless, the majority of developers feel that we'll lose more by a
fork (the community) than we gain by it (reduced constraints).

> - co-develop new features within the same patch. Release new versions of
> kvm-qemu and the kvm bits at the same time (together with the upstream
> kernel), at well defined points in time.
>

The majority of patches to qemu don't require changes to kvm, and vice
versa. The interface between qemu and kvm is fairly narrow, and most of
the changes are related to save/restore and guest debugging, hardly
areas of great interest to the causal user.

> - encourage kernel-space and user-space KVM developers to work on both
> user-space and kernel-space bits as a single unit. It's one project and a
> single experience to the user.
>

When a feature is developed that requires both kernel and qemu changes,
the same developer makes the changes in both projects. Having them in
different repositories does not appear to be a problem.

> - [ and probably libvirt should go there too ]
>

Let's make a list of projects who don't need to be in the kernel
repository, it will probably be shorted.

Seriously, libvirt is a cross-platform cross-hypervisor library, it has
no business near the Linux kernel.

> If KVM's hypervisor and guest kernel code can enjoy the benefits of a single
> repository,

In fact I try hard not to rely too much on that. While both kvm guest
and host code are in the same repo, there is an ABI barrier between them
because we need to support any guest version on any host version. When
designing, writing, or reading guest or host code that interacts across
that barrier we need to keep forward and backward compatibility in
mind. It's very different from normal kernel APIs that we can adapt
whenever the need arises.

> why cannot the rest of KVM enjoy the same developer goodness? Only
> fixing that will bring the break-through in quality - not more manpower
> really.
>

I really don't understand why you believe that. You seem to want a
virtualbox-style GUI, and lkml is probably the last place in the world
to develop something like that. The developers here are mostly
uninterested in GUI and usability problems, remember these are people
who thing emacs xor vi is a great editor.

> Yes, i've read a thousand excuses for why this is an absolutely impossible and
> a bad thing to do, and none of them was really convincing to me - and you also
> have become rather emotional about all the arguments so it's hard to argue
> about it on a technical basis.
>
> We made a similar (admittedly very difficult ...) design jump from oprofile to
> perf, and i can tell you from that experience that it's day and night, both in
> terms of development and in terms of the end result!
>

Maybe it was due to better design and implementation choices.

> ( We recently also made another, kernel/kernel unification that had a very
> positive result: we unified the 32-bit and 64-bit x86 architectures. Even
> within the same repo the unification of technology is generally a good
> thing. The KVM/Qemu situation is different - it's more similar to the perf
> design. )
>
> Not having to fight artificial package boundaries and forced package
> separation is very refreshing experience to a developer - and very rewarding
> and flexible to develop on. ABI compatibility is _easier_ to maintain in such
> a model. It's quite similar to the jump from Xen hacking to KVM hacking (i did
> both). It's a bit like the jump from CVS to Git. Trust me, you _cannot_ know
> the difference if you havent tried a similar jump with Qemu.
>

Why is ABI compatibility easier to maintain in a single repo?

> Anyway, you made your position about this rather clear and you are clearly
> uncompromising, so i just wanted to post this note to the list: you'll waste
> years of your life on a visibly crappy development model that has been unable
> to break through a magic usability barrier for the past 2-3 years - just like
> the Xen mis-design has wasted so many people's time and effort in kernel
> space.
>

Do you really think the echo'n'cat school of usability wants to write a
GUI? In linux-2.6.git?

--
error compiling committee.c: too many arguments to function

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Jes Sorensen on
On 03/17/10 09:10, Ingo Molnar wrote:
> I wish both you and Avi looked back 3-4 years and realized what made KVM so
> successful back then and why the hearts and minds of virtualization developers
> were captured by KVM almost overnight.

Ingo,

What made KVM so successful was that the core kernel of the hypervisor
was designed the right way, as a kernel module where it belonged. It was
obvious to anyone who had been exposed to the main competition at the
time, Xen, that this was the right approach. What has ended up killing
Xen in the end is the not-invented-here approach of copying everything
over, reformatting it, and rewriting half of it, which made it
impossible to maintain and support as a single codebase. At my previous
employer we ended up dropping all Xen efforts exactly because it was
like maintaining two separate operating system kernels. The key to KVM
is that once you have Linux, you practically have KVM as well.

> Fast forward to 2010. The kernel side of KVM is maximum goodness - by far the
> worst-quality remaining aspects of KVM are precisely in areas that you
> mention: 'if we have to support third party tools, then it significantly
> complicates things'. You kept Qemu as an external 'third party' entity to KVM,
> and KVM is clearly hurting from that - just see the recent KVM usability
> thread for examples about suckage.
>
> So a similar 'complication' is the crux of the matter behind KVM quality
> problems: you've not followed through with the original KVM vision and you
> have not applied that concept to Qemu!

Well there are two ways to go about this. Either you base the KVM
userland on top of an existing project, like QEMU, _or_ you rewrite it
all from scratch. However, there is far more to it than just a couple of
ioctls, for example the stack of reverse device-drivers is a pretty
significant code base, rewriting that and maintaining it is not a
trivial task. It is certainly my belief that the benefit we get from
sharing that with QEMU by far outweighs the cost of forking it and
keeping our own fork in the kernel tree. In fact it would result in
exactly the same problems I mentioned above wrt Xen.

> If you want to jump to the next level of technological quality you need to fix
> this attitude and you need to go back to the design roots of KVM. Concentrate
> on Qemu (as that is the weakest link now), make it a first class member of the
> KVM repo and simplify your development model by having a single repo:
>
> - move a clean (and minimal) version of the Qemu code base to tools/kvm/, in
> the upstream kernel repo, and work on that from that point on.

With this you have just thrown away all the benefits of having the QEMU
repository shared with other developers who will actively fix bugs in
components we do care about for KVM.

> - encourage kernel-space and user-space KVM developers to work on both
> user-space and kernel-space bits as a single unit. It's one project and a
> single experience to the user.

This is already happening and a total non issue.

> - [ and probably libvirt should go there too ]

Now that would be interesting, next we'll have to include things like
libxml in the kernel git tree as well, to make sure libvirt doesn't get
out of sync with the version supplied by your distribution vendor.

> Yes, i've read a thousand excuses for why this is an absolutely impossible and
> a bad thing to do, and none of them was really convincing to me - and you also
> have become rather emotional about all the arguments so it's hard to argue
> about it on a technical basis.

So far your argument would justify pulling all of gdb into the kernel
git tree as well, to support the kgdb efforts, or gcc so we can get rid
of the gcc version quirks in the kernel header files, e2fsprogs and
equivalent for _all_ file systems included in the kernel so we can make
sure our fs tools never get out of sync with whats supported in the
kernel......

> We made a similar (admittedly very difficult ...) design jump from oprofile to
> perf, and i can tell you from that experience that it's day and night, both in
> terms of development and in terms of the end result!

The user components for perf vs oprofile are _tiny_ projects compared to
the portions of QEMU that are actually used by KVM.

Oh and you completely forgot SeaBIOS. KVM+QEMU rely on SeaBIOS too, so
from what you're saying we should pull that into the kernel git
repository as well. Never mind the fact that we share SeaBIOS with the
coreboot project which is very actively adding features to it that
benefit us as well.....

Sorry, but there are times when unification make sense, and there are
times where having a reasonably well designed split makes sense. KVM
had problems with QEMU in the past which resulted in the qemu-kvm branch
of it, which proved to be a major pain to deal with, but that is
fortunately improving and qemu-kvm should go away completely at some
point.

Cheers,
Jes
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar on

* Avi Kivity <avi(a)redhat.com> wrote:

> On 03/17/2010 10:10 AM, Ingo Molnar wrote:
> >
> >> It's about who owns the user interface.
> >>
> >> If qemu owns the user interface, than we can satisfy this in a very
> >> simple way by adding a perf monitor command. If we have to support third
> >> party tools, then it significantly complicates things.
> >
> > Of course illogical modularization complicates things 'significantly'.
>
> Who should own the user interface then?

If qemu was in tools/kvm/ then we wouldnt have such issues. A single patch (or
series of patches) could modify tools/kvm/, arch/x86/kvm/, virt/ and
tools/perf/.

Numerous times did we have patches to kernel/perf_event.c that fixed some
detail, also accompanied by a tools/perf/ patch fixing another detail. Having
a single 'culture of contribution' is a powerful way to develop.

It turns out kernel developers can be pretty good user-space developers as
well and user-space developers can be pretty good kernel developers as well.
Some like to do both - as long as it's all within a single project.

The moment any change (be it as trivial as fixing a GUI detail or as complex
as a new feature) involves two or more packages, development speed slows down
to a crawl - while the complexity of the change might be very low!

Also, there's the harmful process that people start categorizing themselves
into 'I am a kernel developer' and 'I am a user space programmer' stereotypes,
which limits the scope of contributions artificially.

> > Fast forward to 2010. The kernel side of KVM is maximum goodness - by far
> > the worst-quality remaining aspects of KVM are precisely in areas that you
> > mention: 'if we have to support third party tools, then it significantly
> > complicates things'. You kept Qemu as an external 'third party' entity to
> > KVM, and KVM is clearly hurting from that - just see the recent KVM
> > usability thread for examples about suckage.
>
> Any qemu usability problems are because developers (or their employers) are
> not interested in fixing them, not because of the repository location. Most
> kvm developer interest is in server-side deployment (even for desktop
> guests), so there is limited effort in implementing a virtualbox-style GUI.

The same has been said of oprofile as well: 'it somewhat sucks because we are
too server centric', 'nobody is interested in good usability and oprofile is
fine for the enterprises'. Ironically, the same has been said of Xen usability
as well, up to the point KVM came around.

What was the core of the problem was a bad design and a split kernel-side
user-side tool landscape.

In fact i think saying that 'our developers only care about the server' is
borderline dishonest, when at the same time you are making it doubly sure (by
inaction) that it stays so: by leaving an artificial package wall between
kernel-side KVM and user-side KVM and not integrating the two technologies.

You'll never know what heights you could achieve if you leave that wall there
....

Furthermore, what should be realized is that bad usability hurts "server
features" just as much. Most of the day-to-day testing is done on the desktop
by desktop oriented testers/developers. _Not_ by enterprise shops - they tend
to see the code years down the line to begin with ...

Yes, a particular feature might be server oriented, but a good portion of our
testing is on the desktop and everyone is hurting from bad usability and this
puts limits on contribution efficiency.

As the patch posted in _this very thread demonstrates it_, it is doubly more
difficult to contribute a joint KVM+Qemu feature, because it's two separate
code bases, two contribution guidelines, two release schedules. While to the
user it really is just one and the same thing. It should be so for the
developer as well.

Put in another way: KVM's current split design is making it easy to contribute
server features (because the kernel side is clean and cool), but also makes it
artificially hard to contribute desktop features: because the tooling side
(Qemu) is 'just another package', is separated by a package and maintenance
wall and is made somewhat uncool by a (as some KVM developers have pointed out
in this thread) quirky codebase.

(the rest of your points are really a function of this fundamental
disagreement)

Thanks,

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar on

* Avi Kivity <avi(a)redhat.com> wrote:

> > - move a clean (and minimal) version of the Qemu code base to tools/kvm/,
> > in the upstream kernel repo, and work on that from that point on.
>
> I'll ignore the repository location which should be immaterial to a serious
> developer and concentrate on the 'clean and minimal' aspect, since it has
> some merit. [...]

To the contrary, experience shows that repository location, and in particular
a shared repository for closely related bits is very much material!

It matters because when there are two separate projects, even a "serious
developer" is finding it double and triple difficult to contribute even
trivial changes.

It becomes literally a nightmare if you have to touch 3 packages: kernel, a
library and an app codebase. It takes _forever_ to get anything useful done.

Also, 'focus on a single thing' is a very basic aspect of humans, especially
those who do computer programming. Working on two code bases in two
repositories at once can be very challenging physically and psychically.

So what i've seen is that OSS programmers tend to pick a side, pretty much
randomly, and then rationalize it in hindsight why they prefer that side ;-)

Most of them become either a kernel developer or a user-space package
developer - and then they specialize on that field and shy away from changes
that involve both. It's a basic human thing to avoid the hassle that comes
with multi-package changes. (One really has to be outright stupid, fanatic or
desperate to even attempt such changes these days - such are the difficulties
for a comparatively low return.)

The solution is to tear down such artificial walls of contribution where
possible. And tearing down the wall between KVM and qemu-kvm seems very much
possible and the advantages would be numerous.

Unless by "serious developer" you meant: "developer willing to [or forced to]
waste time and effort on illogically structured technology".

> [...]
>
> Do you really think the echo'n'cat school of usability wants to write a GUI?
> In linux-2.6.git?

Then you'll be surprised to hear that it's happening as we speak and the
commits are there in linux-2.6.git. Both a TUI and GUI is in the works.

Furthermore, the numbers show that half of the usability fixes to tools/perf/
came not from regular perf contributors but from random kernel developers and
testers who when they build the latest kernel and try out perf at the same
time (it's very easy because you already have it in the kernel repository - no
separate download, no installation, etc. necessary).

I had literally zero such contributions when (the precursor to) 'perf' was
still a separate user-space project.

You could have the same effect for Qemu: the latest bits in tools/kvm/ would
be built by regular kernel testers and developers. The integration benefits
dont just extend to developers, a unified project is vastly easier to test as
well.

Thanks,

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Ingo Molnar on

* Jes Sorensen <Jes.Sorensen(a)redhat.com> wrote:

> On 03/17/10 09:10, Ingo Molnar wrote:
>
> > I wish both you and Avi looked back 3-4 years and realized what made KVM
> > so successful back then and why the hearts and minds of virtualization
> > developers were captured by KVM almost overnight.
>
> Ingo,
>
> What made KVM so successful was that the core kernel of the hypervisor was
> designed the right way, as a kernel module where it belonged. It was obvious
> to anyone who had been exposed to the main competition at the time, Xen,
> that this was the right approach. What has ended up killing Xen in the end
> is the not-invented-here approach of copying everything over, reformatting
> it, and rewriting half of it, which made it impossible to maintain and
> support as a single codebase. [...]

Yes, exactly.

I was part of that nightmare so i know.

> [...]
>
> At my previous employer we ended up dropping all Xen efforts exactly because
> it was like maintaining two separate operating system kernels. The key to
> KVM is that once you have Linux, you practically have KVM as well.

Yes. Please realize that what is behind it is a strikingly simple argument:

"Once you have a single project to develop and maintain all is much better."

> > Fast forward to 2010. The kernel side of KVM is maximum goodness - by far
> > the worst-quality remaining aspects of KVM are precisely in areas that you
> > mention: 'if we have to support third party tools, then it significantly
> > complicates things'. You kept Qemu as an external 'third party' entity to
> > KVM, and KVM is clearly hurting from that - just see the recent KVM
> > usability thread for examples about suckage.
> >
> > So a similar 'complication' is the crux of the matter behind KVM quality
> > problems: you've not followed through with the original KVM vision and you
> > have not applied that concept to Qemu!
>
> Well there are two ways to go about this. Either you base the KVM userland
> on top of an existing project, like QEMU, _or_ you rewrite it all from
> scratch. [...]

Btw., i made similar arguments to Avi about 3 years ago when it was going
upstream, that qemu should be unified with KVM. This is more true today than
ever.

> [...] However, there is far more to it than just a couple of ioctls, for
> example the stack of reverse device-drivers is a pretty significant code
> base, rewriting that and maintaining it is not a trivial task. It is
> certainly my belief that the benefit we get from sharing that with QEMU by
> far outweighs the cost of forking it and keeping our own fork in the kernel
> tree. In fact it would result in exactly the same problems I mentioned above
> wrt Xen.

I do not suggest forking Qemu at all, i suggest using the most natural
development model for the KVM+Qemu shared project: a single repository.

> > If you want to jump to the next level of technological quality you need to
> > fix this attitude and you need to go back to the design roots of KVM.
> > Concentrate on Qemu (as that is the weakest link now), make it a first
> > class member of the KVM repo and simplify your development model by having
> > a single repo:
> >
> > - move a clean (and minimal) version of the Qemu code base to tools/kvm/,
> > in the upstream kernel repo, and work on that from that point on.
>
> With this you have just thrown away all the benefits of having the QEMU
> repository shared with other developers who will actively fix bugs in
> components we do care about for KVM.

Not if it's a unified project.

> > - encourage kernel-space and user-space KVM developers to work on both
> > user-space and kernel-space bits as a single unit. It's one project and
> > a single experience to the user.
>
> This is already happening and a total non issue.

My experience as an external observer of the end result contradicts this.

Seemingly trivial usability changes to the KVM+Qemu combo are not being done
often because they involve cross-discipline changes.

( _In this very thread_ there has been a somewhat self-defeating argument by
Anthony that multi-package scenario would 'significantly complicate'
matters. What more proof do we need to state the obvious? Keeping what
has become one piece of technology over the years in two separate halves is
obviously bad. )

> > - [ and probably libvirt should go there too ]
>
> Now that would be interesting, next we'll have to include things like libxml
> in the kernel git tree as well, to make sure libvirt doesn't get out of sync
> with the version supplied by your distribution vendor.

The way we have gone about this in tools/perf/ is similar to the route picked
by Git: we only use very lowlevel libraries available everywhere, and we
provide optional wrappers to the rest.

We are also using the kernel's libraries so we rarely need to go outside to
get some functionality.

I.e. it's a non-issue in practice and despite perf having an (optional)
dependency on xmlto and docbook we dont include those packages nor do we force
users to install particular versions of them.

> > Yes, i've read a thousand excuses for why this is an absolutely impossible
> > and a bad thing to do, and none of them was really convincing to me - and
> > you also have become rather emotional about all the arguments so it's hard
> > to argue about it on a technical basis.
>
> So far your argument would justify pulling all of gdb into the kernel git
> tree as well, to support the kgdb efforts, or gcc so we can get rid of the
> gcc version quirks in the kernel header files, e2fsprogs and equivalent for
> _all_ file systems included in the kernel so we can make sure our fs tools
> never get out of sync with whats supported in the kernel......

gdb and gcc is clearly extrinsic to the kernel so why would we move them
there?

I was talking about tools that are closely related to the kernel - where much
of the development and actual use is in combination with the Linux kernel.

90%+ of the Qemu usecases are combined with Linux. (Yes, i know that you can
run Qemu without KVM, and no, i dont think it matters in the grand scheme of
things and most investment into Qemu comes from the KVM angle these days. In
particular it for sure does not justify handicapping future KVM evolution so
drastically.)

> > We made a similar (admittedly very difficult ...) design jump from
> > oprofile to perf, and i can tell you from that experience that it's day
> > and night, both in terms of development and in terms of the end result!
>
> The user components for perf vs oprofile are _tiny_ projects compared to the
> portions of QEMU that are actually used by KVM.

I know the size and scope of Qemu, i even hacked it - still my points remain.
(my arguments are influenced and strengthened by that past hacking experience)

> Oh and you completely forgot SeaBIOS. KVM+QEMU rely on SeaBIOS too, so from
> what you're saying we should pull that into the kernel git repository as
> well. Never mind the fact that we share SeaBIOS with the coreboot project
> which is very actively adding features to it that benefit us as well.....

SeaBIOS is in essence a firmware, so it could either be loaded as such.

Just look at the qemu source code - the BIOSes are .bin images in
qemu/pc-bios/ imported externally in essence.

Moving qemu to tools/kvm/ would not change that much. The firmware could
become part of /lib/firmware/*.bin.

( That would probably be a more intelligent approach to the BIOS image import
problem as well. )

> Sorry, but there are times when unification make sense, and there are times
> where having a reasonably well designed split makes sense. KVM had problems
> with QEMU in the past which resulted in the qemu-kvm branch of it, which
> proved to be a major pain to deal with, but that is fortunately improving
> and qemu-kvm should go away completely at some point.

qemu-kvm branch is not similar to my proposal at all: it made KVM _more_
fragmented, not more unified. I.e. it was a move in the exact opposite
direction and i'd expect such a move to fail.

In fact the failure of qemu-kvm supports my point rather explicitly: it
demonstrates that extra packages and split development are actively harmful.

I speak about this as a person who has done successful unifications of split
codebases and in my judgement this move would be significantly beneficial to
KVM.

You cannot really validly reject this proposal with "It wont work" as it
clearly has worked in other, comparable cases. You could only reject this with
"I have tried it and it didnt work".

Think about it: a clean and hackable user-space component in tools/kvm/. It's
very tempting :-)

Ingo
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/