USB transfer_buffer allocations on 64bit systems
in [Kernel]
|
Prev: mfd: Initialise WM831x IRQ masks on chip even if interrupts not in use
Next: INFO: task apache2:7601 blocked for more than 120 seconds.
From: Robert Hancock on 10 Apr 2010 13:10 On Sat, Apr 10, 2010 at 2:34 AM, Daniel Mack <daniel(a)caiaq.de> wrote: > On Fri, Apr 09, 2010 at 05:38:13PM -0600, Robert Hancock wrote: >> On Fri, Apr 9, 2010 at 10:50 AM, Sarah Sharp >> <sarah.a.sharp(a)linux.intel.com> wrote: >> > What makes you think that? �I've seen URB buffers with 64-bit DMA >> > addresses. �I can tell when the debug polling loop runs and I look at >> > the DMA addresses the xHCI driver is feeding to the hardware: >> > >> > Dev 1 endpoint ring 0: >> > xhci_hcd 0000:05:00.0: @71a49800 01000680 00080000 00000008 00000841 >> > >> > So the TRB at address 71a49800 is pointing to a buffer at address >> > 0x0008000001000680. >> >> I'm not sure why the address would be that huge, unless it's not >> actually a physical address, or there's some kind of remapping going >> on? >> >> > >> > If I'm setting a DMA mask wrong somewhere, or doing something else to >> > limit the DMA to 32-bit, then please let me know. >> >> The DMA mask for the controller isn't being set anywhere (in the >> version that's in Linus' current git anyway). In that case it'll >> default to 32-bit and any DMA mappings above 4GB will need to be >> remapped. The controller driver doesn't do the mapping itself but the >> USB core does, passing in the controller device as the one doing the >> DMA, so if the controller's DMA mask is set to 32-bit then the buffers >> passed in will get remapped/bounced accordingly. > > So if we're seeing physical addresses in the log above, and the xHCI > driver does not explicitly allow the USB core to use addresses above > 4GB, why shouldn't the same thing be true as well for EHCI? > (Which would then be exactly the case we're seeing) That is a bit suspicious, yes. With the DMA mask at default I don't expect that should happen. Sarah, what kind of traffic was happening when you saw that (bulk, isochronous, etc)? If this apparent EHCI problem is reproducible, it might be useful to add some code that warns if anything non-zero gets written to the hw_bufp_hi and hw_buf_hi members in the descriptors. Also, booting with mem=4096M on an affected system would also tell you if it's related to using memory above 4GB. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andi Kleen on 12 Apr 2010 05:10 Alan Stern <stern(a)rowland.harvard.edu> writes: >> >> The fix is to use usb_buffer_alloc() for that purpose which ensures >> memory that is suitable for DMA. And on x86_64, this also means that the >> upper 32 bits of the address returned are all 0's. > > That is not a good fix. usb_buffer_alloc() provides coherent memory, > which is not what we want. I believe the correct fix is to specify the > GFP_DMA32 flag in the kzalloc() call. The traditional way to handle this is to leave it to swiotlb in pci_map_*. pci_map_* is needed anyways if you run with a IOMMU. Also note at least on x86 systems coherent memory is the same as non coherent memory. And GFP_DMA32 is a x86 specific flag, doesn't necessarily do any good anywhere else. So if you add x86isms anyways you could as well use dma_alloc_coherent() directly which is actually better at this than a simple GFP_DMA32 and as a bonus handles the IOMMUs correctly too. Or just use GFP_KERNEL and pci_map_* later. -Andi -- ak(a)linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Daniel Mack on 12 Apr 2010 06:50 On Fri, Apr 09, 2010 at 04:11:52PM -0600, Robert Hancock wrote: > On Fri, Apr 9, 2010 at 3:23 PM, Alan Stern <stern(a)rowland.harvard.edu> wrote: > > On Fri, 9 Apr 2010, Konrad Rzeszutek Wilk wrote: > > > >> On Fri, Apr 09, 2010 at 03:34:06PM -0400, Alan Stern wrote: > >> > On Fri, 9 Apr 2010, Pedro Ribeiro wrote: > >> > > >> > > > The DMA pointers do indeed look sane. I wanted to take a deeper look at > >> > > > this and set up a 64bit system today. However, I fail to see the problem > >> > > > here. Pedro, how much RAM does your machine have installed? > >> > > >> > > It has 4 GB. > >> > > >> > That means DMA mapping cannot be the cause of the problem. �:-( > >> > >> That isn't entirely true. The BIOS usually allocates a 256 MB ACPI/PCI hole > >> that is under the 4GB. > >> > >> So end up with 3.7 GB, then the 256MB hole, and then right above the 4GB > >> you the the remaining memory: 4.3GB. > > > > How can Pedro find out what physical addresses are in use on his > > system? > > If you have 4GB of RAM then almost certainly you have memory located > at addresses over 4GB. If you look at the e820 memory map printed at > the start of dmesg on bootup and see entries with addresses of > 100000000 or higher reported as usable, then this is the case. Pedro, can you provide your dmesg output, please? I installed 5GB or RAM to my machine now, and even with your .config, I can't see the problem. Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Daniel Mack on 12 Apr 2010 07:20 On Mon, Apr 12, 2010 at 10:59:57AM +0200, Andi Kleen wrote: > Alan Stern <stern(a)rowland.harvard.edu> writes: > >> > >> The fix is to use usb_buffer_alloc() for that purpose which ensures > >> memory that is suitable for DMA. And on x86_64, this also means that the > >> upper 32 bits of the address returned are all 0's. > > > > That is not a good fix. usb_buffer_alloc() provides coherent memory, > > which is not what we want. I believe the correct fix is to specify the > > GFP_DMA32 flag in the kzalloc() call. > > The traditional way to handle this is to leave it to swiotlb in > pci_map_*. pci_map_* is needed anyways if you run with a IOMMU. > > Also note at least on x86 systems coherent memory is the same as non coherent > memory. And GFP_DMA32 is a x86 specific flag, doesn't necessarily > do any good anywhere else. > > So if you add x86isms anyways you could as well use dma_alloc_coherent() > directly which is actually better at this than a simple GFP_DMA32 > and as a bonus handles the IOMMUs correctly too. Which is exactly what usb_buffer_alloc() does already. So at least for x86 you say this is the right thing to do? However, we don't necessarily need coherent memory on other platforms, which is why I hessitate to enforce that type of memory for all transfer_buffer allocations. > Or just use GFP_KERNEL and pci_map_* later. The USB core does this already, but at least on Pedro's machine, this seems unsufficient. Unfortunately, I can't reproduce the issue yet, even with more than 4GB of RAM installed. Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Andi Kleen on 12 Apr 2010 08:00
> Which is exactly what usb_buffer_alloc() does already. So at least for > x86 you say this is the right thing to do? However, we don't necessarily > need coherent memory on other platforms, which is why I hessitate to > enforce that type of memory for all transfer_buffer allocations. Yes today it's faster at least. Probably we should have better interfaces for this, but we don't. > > > Or just use GFP_KERNEL and pci_map_* later. > > The USB core does this already, but at least on Pedro's machine, this > seems unsufficient. Unfortunately, I can't reproduce the issue yet, even > with more than 4GB of RAM installed. Then something must be broken in Pedro's system and likely other drivers will also not work. I don't think it should be worked around in the USB layer. -Andi -- ak(a)linux.intel.com -- Speaking for myself only. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |