USB transfer_buffer allocations on 64bit systems
in [Kernel]
|
Prev: mfd: Initialise WM831x IRQ masks on chip even if interrupts not in use
Next: INFO: task apache2:7601 blocked for more than 120 seconds.
From: Robert Hancock on 9 Apr 2010 18:20 On Fri, Apr 9, 2010 at 3:23 PM, Alan Stern <stern(a)rowland.harvard.edu> wrote: > On Fri, 9 Apr 2010, Konrad Rzeszutek Wilk wrote: > >> On Fri, Apr 09, 2010 at 03:34:06PM -0400, Alan Stern wrote: >> > On Fri, 9 Apr 2010, Pedro Ribeiro wrote: >> > >> > > > The DMA pointers do indeed look sane. I wanted to take a deeper look at >> > > > this and set up a 64bit system today. However, I fail to see the problem >> > > > here. Pedro, how much RAM does your machine have installed? >> > >> > > It has 4 GB. >> > >> > That means DMA mapping cannot be the cause of the problem. �:-( >> >> That isn't entirely true. The BIOS usually allocates a 256 MB ACPI/PCI hole >> that is under the 4GB. >> >> So end up with 3.7 GB, then the 256MB hole, and then right above the 4GB >> you the the remaining memory: 4.3GB. > > How can Pedro find out what physical addresses are in use on his > system? If you have 4GB of RAM then almost certainly you have memory located at addresses over 4GB. If you look at the e820 memory map printed at the start of dmesg on bootup and see entries with addresses of 100000000 or higher reported as usable, then this is the case. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Robert Hancock on 9 Apr 2010 19:40 On Fri, Apr 9, 2010 at 10:50 AM, Sarah Sharp <sarah.a.sharp(a)linux.intel.com> wrote: >> >>I don't know if the comment is still true, but until the "#if 0" is >> >>removed, ehci-hcd won't make use of 64-bit DMA. >> > >> >I think someone tried to remove it recently, but I wouldn't let them :) >> > >> >What a mess, hopefully xhci will just take over and save the world from >> >this whole thing... > > I hate to break it to you, but 64-bit DMA support is optional for an > xHCI implementation. �There's a bit in HCCPARAMS that tells whether the > host supports it (see the HCC_64BIT_ADDR macro in xhci.h). �The xHCI > driver currently doesn't do anything with that bit, although it should. > All the implementations I've seen do 64-bit DMA. > >> True.. except for the fact that the xhci driver currently doesn't do >> 64-bit DMA either > > What makes you think that? �I've seen URB buffers with 64-bit DMA > addresses. �I can tell when the debug polling loop runs and I look at > the DMA addresses the xHCI driver is feeding to the hardware: > > Dev 1 endpoint ring 0: > xhci_hcd 0000:05:00.0: @71a49800 01000680 00080000 00000008 00000841 > > So the TRB at address 71a49800 is pointing to a buffer at address > 0x0008000001000680. I'm not sure why the address would be that huge, unless it's not actually a physical address, or there's some kind of remapping going on? > > If I'm setting a DMA mask wrong somewhere, or doing something else to > limit the DMA to 32-bit, then please let me know. The DMA mask for the controller isn't being set anywhere (in the version that's in Linus' current git anyway). In that case it'll default to 32-bit and any DMA mappings above 4GB will need to be remapped. The controller driver doesn't do the mapping itself but the USB core does, passing in the controller device as the one doing the DMA, so if the controller's DMA mask is set to 32-bit then the buffers passed in will get remapped/bounced accordingly. You should likely be setting the DMA mask for the controller device to 64-bit if the HCC_64BIT_ADDR flag is set, similar to the code in ehci-hcd.c which is currently #if 0'ed out. You can see the currently set mask in sysfs under /sys/devices/pci(whatever)/dma_mask_bits. > >> nor does it support MSI even though the HW >> supports it (surprisingly enough the NEC Windows driver does, MSI-X >> even). > > There's a patch from AMD to enable MSI-X. �The code was there, just > commented out because the early prototypes didn't do MSI-X. Ah, I see it. That could presumably be tested now (the NEC D720200F1 chip on the Asus U3S6 card I have seems to support MSI-X with 8 vectors). > >> At this point only Intel likely knows how to do this >> properly, though, since AFAICS the spec isn't publicly available >> yet. > > I have tried very hard to fix this, and will continue to do so. > > Sarah Sharp > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Daniel Mack on 10 Apr 2010 04:40 On Fri, Apr 09, 2010 at 05:38:13PM -0600, Robert Hancock wrote: > On Fri, Apr 9, 2010 at 10:50 AM, Sarah Sharp > <sarah.a.sharp(a)linux.intel.com> wrote: > > What makes you think that? �I've seen URB buffers with 64-bit DMA > > addresses. �I can tell when the debug polling loop runs and I look at > > the DMA addresses the xHCI driver is feeding to the hardware: > > > > Dev 1 endpoint ring 0: > > xhci_hcd 0000:05:00.0: @71a49800 01000680 00080000 00000008 00000841 > > > > So the TRB at address 71a49800 is pointing to a buffer at address > > 0x0008000001000680. > > I'm not sure why the address would be that huge, unless it's not > actually a physical address, or there's some kind of remapping going > on? > > > > > If I'm setting a DMA mask wrong somewhere, or doing something else to > > limit the DMA to 32-bit, then please let me know. > > The DMA mask for the controller isn't being set anywhere (in the > version that's in Linus' current git anyway). In that case it'll > default to 32-bit and any DMA mappings above 4GB will need to be > remapped. The controller driver doesn't do the mapping itself but the > USB core does, passing in the controller device as the one doing the > DMA, so if the controller's DMA mask is set to 32-bit then the buffers > passed in will get remapped/bounced accordingly. So if we're seeing physical addresses in the log above, and the xHCI driver does not explicitly allow the USB core to use addresses above 4GB, why shouldn't the same thing be true as well for EHCI? (Which would then be exactly the case we're seeing) Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Daniel Mack on 10 Apr 2010 08:50 On Fri, Apr 09, 2010 at 07:19:22PM +0100, Pedro Ribeiro wrote: > On 9 April 2010 19:09, Daniel Mack <daniel(a)caiaq.de> wrote: > > On Fri, Apr 09, 2010 at 12:01:27PM -0400, Alan Stern wrote: > >> I don't see anything suspicious. �The transfer_buffer addresses repeat > >> every 32 URBs, and the DMA addresses cycle almost entirely uniformly > >> from 0x20000000 to 0x23ffffff in units of 0x2000 (there are a few gaps > >> where the interval is a little bigger). > > > > The DMA pointers do indeed look sane. I wanted to take a deeper look at > > this and set up a 64bit system today. However, I fail to see the problem > > here. Pedro, how much RAM does your machine have installed? > > > > Daniel > > > > > > It has 4 GB. Upgraded my machine now to 4GB, but I still can't reproduce this bug. Pedro, can you send your config, please? Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Pedro Ribeiro on 10 Apr 2010 09:30
On 10 April 2010 13:49, Daniel Mack <daniel(a)caiaq.de> wrote: > On Fri, Apr 09, 2010 at 07:19:22PM +0100, Pedro Ribeiro wrote: >> On 9 April 2010 19:09, Daniel Mack <daniel(a)caiaq.de> wrote: >> > On Fri, Apr 09, 2010 at 12:01:27PM -0400, Alan Stern wrote: >> >> I don't see anything suspicious. The transfer_buffer addresses repeat >> >> every 32 URBs, and the DMA addresses cycle almost entirely uniformly >> >> from 0x20000000 to 0x23ffffff in units of 0x2000 (there are a few gaps >> >> where the interval is a little bigger). >> > >> > The DMA pointers do indeed look sane. I wanted to take a deeper look at >> > this and set up a 64bit system today. However, I fail to see the problem >> > here. Pedro, how much RAM does your machine have installed? >> > >> > Daniel >> > >> > >> >> It has 4 GB. > > Upgraded my machine now to 4GB, but I still can't reproduce this bug. > Pedro, can you send your config, please? > > Daniel > Here it is. It is configured for realtime and TuxOnIce patches. However, the bug still occurs without these kernel patches. Pedro |