From: Zach Pfeffer on
On Wed, Jul 14, 2010 at 11:05:36PM +0100, Russell King - ARM Linux wrote:
> On Wed, Jul 14, 2010 at 01:11:49PM -0700, Zach Pfeffer wrote:
> > If the DMA-API contained functions to allocate virtual space separate
> > from physical space and reworked how chained buffers functioned it
> > would probably work - but then things start to look like the VCM API
> > which does graph based map management.
>
> Every additional virtual mapping of a physical buffer results in
> additional cache aliases on aliasing caches, and more workload for
> developers to sort out the cache aliasing issues.
>
> What does VCM to do mitigate that?

The VCM ensures that all mappings that map a given physical buffer:
IOMMU mappings, CPU mappings and one-to-one device mappings all map
that buffer using the same (or compatible) attributes. At this point
the only attribute that users can pass is CACHED. In the absence of
CACHED all accesses go straight through to the physical memory.

The architecture of the VCM allows these sorts of consistency checks
to be made since all mappers of a given physical resource are
tracked. This is feasible because the physical resources we're
tracking are typically large.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Zach Pfeffer on
On Thu, Jul 15, 2010 at 08:07:28AM +0900, FUJITA Tomonori wrote:
> On Wed, 14 Jul 2010 13:11:49 -0700
> Zach Pfeffer <zpfeffer(a)codeaurora.org> wrote:
>
> > On Wed, Jul 14, 2010 at 10:59:38AM +0900, FUJITA Tomonori wrote:
> > > On Tue, 13 Jul 2010 05:14:21 -0700
> > > Zach Pfeffer <zpfeffer(a)codeaurora.org> wrote:
> > >
> > > > > You mean that you want to specify this alignment attribute every time
> > > > > you create an IOMMU mapping? Then you can set segment_boundary_mask
> > > > > every time you create an IOMMU mapping. It's odd but it should work.
> > > >
> > > > Kinda. I want to forget about IOMMUs, devices and CPUs. I just want to
> > > > create a mapping that has the alignment I specify, regardless of the
> > > > mapper. The mapping is created on a VCM and the VCM is associated with
> > > > a mapper: a CPU, an IOMMU'd device or a direct mapped device.
> > >
> > > Sounds like you can do the above with the combination of the current
> > > APIs, create a virtual address and then an I/O address.
> > >
> >
> > Yes, and that's what the implementation does - and all the other
> > implementations that need to do this same thing. Why not solve the
> > problem once?
>
> Why we we need a new abstraction layer to solve the problem that the
> current API can handle?

The current API can't really handle it because the DMA API doesn't
separate buffer allocation from buffer mapping. To use the DMA API a
scatterlist would need to be synthesized from the physical buffers
that we've allocated.

For instance: I need 10, 1 MB physical buffers and a 64 KB physical
buffer. With the DMA API I need to allocate 10*1MB/PAGE_SIZE + 64
KB/PAGE_SIZE scatterlist elements, fix them all up to follow the
chaining specification and then go through all of them again to fix up
their virtual mappings for the mapper that's mapping the physical
buffer. If I want to share the buffer with another device I have to
make a copy of the entire thing then fix up the virtual mappings for
the other device I'm sharing with. The VCM splits the two things up so
that I do a physical allocation, then 2 virtual allocations and then
map both.

>
> The above two operations don't sound too complicated. The combination
> of the current API sounds much simpler than your new abstraction.
>
> Please show how the combination of the current APIs doesn't
> work. Otherwise, we can't see what's the benefit of your new
> abstraction.

See above.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Zach Pfeffer on
On Wed, Jul 14, 2010 at 06:29:58PM -0700, Zach Pfeffer wrote:
> On Wed, Jul 14, 2010 at 11:05:36PM +0100, Russell King - ARM Linux wrote:
> > On Wed, Jul 14, 2010 at 01:11:49PM -0700, Zach Pfeffer wrote:
> > > If the DMA-API contained functions to allocate virtual space separate
> > > from physical space and reworked how chained buffers functioned it
> > > would probably work - but then things start to look like the VCM API
> > > which does graph based map management.
> >
> > Every additional virtual mapping of a physical buffer results in
> > additional cache aliases on aliasing caches, and more workload for
> > developers to sort out the cache aliasing issues.
> >
> > What does VCM to do mitigate that?
>
> The VCM ensures that all mappings that map a given physical buffer:
> IOMMU mappings, CPU mappings and one-to-one device mappings all map
> that buffer using the same (or compatible) attributes. At this point
> the only attribute that users can pass is CACHED. In the absence of
> CACHED all accesses go straight through to the physical memory.
>
> The architecture of the VCM allows these sorts of consistency checks
> to be made since all mappers of a given physical resource are
> tracked. This is feasible because the physical resources we're
> tracking are typically large.

A few more things...

In addition to CACHED, the VCMM can support different cache policies
as long as the architecture can support it - they get passed down
through the device map call.

In addition, handling physical mappings in the VCMM enables it to
perform refcounting on the physical chunks (ie, to see how many
virtual spaces it's been mapped to, including the kernel's). This
allows it to turn on any coherency protocols that are available in
hardware (ie, setting the shareable bit on something that is mapped to
more than one virtual space). That same attribute can be left off on a
buffer that has only one virtual mapping (ie, scratch buffers used by
one device only). It is then up to the underlying system to deal with
that shared attribute - to enable redirection if it's supported, or to
force something to be non-cacheable, etc. Doing it all through the
VCMM allows all these mechanisms be worked out once per architecture
and then reused.

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Zach Pfeffer on
On Wed, Jul 14, 2010 at 06:47:34PM -0700, Eric W. Biederman wrote:
> Zach Pfeffer <zpfeffer(a)codeaurora.org> writes:
>
> > On Wed, Jul 14, 2010 at 11:05:36PM +0100, Russell King - ARM Linux wrote:
> >> On Wed, Jul 14, 2010 at 01:11:49PM -0700, Zach Pfeffer wrote:
> >> > If the DMA-API contained functions to allocate virtual space separate
> >> > from physical space and reworked how chained buffers functioned it
> >> > would probably work - but then things start to look like the VCM API
> >> > which does graph based map management.
> >>
> >> Every additional virtual mapping of a physical buffer results in
> >> additional cache aliases on aliasing caches, and more workload for
> >> developers to sort out the cache aliasing issues.
> >>
> >> What does VCM to do mitigate that?
> >
> > The VCM ensures that all mappings that map a given physical buffer:
> > IOMMU mappings, CPU mappings and one-to-one device mappings all map
> > that buffer using the same (or compatible) attributes. At this point
> > the only attribute that users can pass is CACHED. In the absence of
> > CACHED all accesses go straight through to the physical memory.
> >
> > The architecture of the VCM allows these sorts of consistency checks
> > to be made since all mappers of a given physical resource are
> > tracked. This is feasible because the physical resources we're
> > tracking are typically large.
>
> On x86 this is implemented in the pat code, and could reasonably be
> generalized to be cross platform.
>
> This is controlled by HAVE_PFNMAP_TRACKING and with entry points
> like track_pfn_vma_new.
>
> Given that we already have an implementation that tracks the cached
> vs non-cached attribute using the dma api. I don't see that the
> API has to change. An implementation of the cached vs non-cached
> status for arm and other architectures is probably appropriate.
>
> It is definitely true that getting your mapping caching attributes
> out of sync can be a problem.

Sure, but we're still stuck with needing lots of scatterlist list
elements and needing to copy them to share physical buffers.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Larry Bassel on
On 16 Jul 10 08:58, Russell King - ARM Linux wrote:
> On Thu, Jul 15, 2010 at 08:48:36PM -0400, Tim HRM wrote:
> > Interesting, since I seem to remember the MSM devices mostly conduct
> > IO through regions of normal RAM, largely accomplished through
> > ioremap() calls.
> >
> > Without more public domain documentation of the MSM chips and AMSS
> > interfaces I wouldn't know how to avoid this, but I can imagine it
> > creates a bit of urgency for Qualcomm developers as they attempt to
> > upstream support for this most interesting SoC.
>
> As the patch has been out for RFC since early April on the linux-arm-kernel
> mailing list (Subject: [RFC] Prohibit ioremap() on kernel managed RAM),
> and no comments have come back from Qualcomm folk.

We are investigating the impact of this change on us, and I
will send out more detailed comments next week.

>
> The restriction on creation of multiple V:P mappings with differing
> attributes is also fairly hard to miss in the ARM architecture
> specification when reading the sections about caches.
>

Larry Bassel

--
Sent by an employee of the Qualcomm Innovation Center, Inc.
The Qualcomm Innovation Center, Inc. is a member of the Code Aurora Forum.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/