USB transfer_buffer allocations on 64bit systems
in [Kernel]
|
Prev: mfd: Initialise WM831x IRQ masks on chip even if interrupts not in use
Next: INFO: task apache2:7601 blocked for more than 120 seconds.
From: Daniel Mack on 12 Apr 2010 14:10 On Mon, Apr 12, 2010 at 07:22:33PM +0200, Andi Kleen wrote: > On Mon, Apr 12, 2010 at 07:15:07PM +0200, Daniel Mack wrote: > > Given that - at least for non-64-aware host controllers - we want memory > > <4GB anyway for USB transfers to avoid DMA bouncing buffers, maybe we > > should just do that and fix the problem at this level? I already started > > to implement usb_[mz]alloc() and use it in some USB drivers. > > If the area is not mapped correctly it will fail in other situations, > e.g. with an IOMMU active or in virtualized setups. So the bug > has to be eventually tracked down. Ok, agreed. But we all agree to the fact that we need an interface for such allocations to avoid bounce buffers? If that is the case, we could already start to implement that while we're tracking down the actual bug. [...] > > Can anyone explain which is the right way to go? > > The right thing would be to define a proper interface for it. > > I had an attempt for it a couple of years ago with the mask allocator, > but it didn't make it into the tree. Any plans to continue on this? Or can you dig it out again so others can pick up the idea? Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Sarah Sharp on 12 Apr 2010 15:10 On Sat, Apr 10, 2010 at 11:02:53AM -0600, Robert Hancock wrote: > On Sat, Apr 10, 2010 at 2:34 AM, Daniel Mack <daniel(a)caiaq.de> wrote: > > On Fri, Apr 09, 2010 at 05:38:13PM -0600, Robert Hancock wrote: > >> On Fri, Apr 9, 2010 at 10:50 AM, Sarah Sharp > >> <sarah.a.sharp(a)linux.intel.com> wrote: > >> > What makes you think that? �I've seen URB buffers with 64-bit DMA > >> > addresses. �I can tell when the debug polling loop runs and I look at > >> > the DMA addresses the xHCI driver is feeding to the hardware: > >> > > >> > Dev 1 endpoint ring 0: > >> > xhci_hcd 0000:05:00.0: @71a49800 01000680 00080000 00000008 00000841 > >> > > >> > So the TRB at address 71a49800 is pointing to a buffer at address > >> > 0x0008000001000680. > >> > >> I'm not sure why the address would be that huge, unless it's not > >> actually a physical address, or there's some kind of remapping going > >> on? > >> > >> > > >> > If I'm setting a DMA mask wrong somewhere, or doing something else to > >> > limit the DMA to 32-bit, then please let me know. > >> > >> The DMA mask for the controller isn't being set anywhere (in the > >> version that's in Linus' current git anyway). In that case it'll > >> default to 32-bit and any DMA mappings above 4GB will need to be > >> remapped. The controller driver doesn't do the mapping itself but the > >> USB core does, passing in the controller device as the one doing the > >> DMA, so if the controller's DMA mask is set to 32-bit then the buffers > >> passed in will get remapped/bounced accordingly. > > > > So if we're seeing physical addresses in the log above, and the xHCI > > driver does not explicitly allow the USB core to use addresses above > > 4GB, why shouldn't the same thing be true as well for EHCI? > > (Which would then be exactly the case we're seeing) > > That is a bit suspicious, yes. With the DMA mask at default I don't > expect that should happen. Sarah, what kind of traffic was happening > when you saw that (bulk, isochronous, etc)? Ring 0 is the default control ring, so it must be control transfers. That's the first control transfer on the ring (and it didn't fill), so it must have come from the USB core. I was running the USB mass storage driver, and the bulk endpoint rings don't have the high 32-bits of the address set. It mostly uses the scatter-gather interface, which calls usb_buffer_map_sg(), so that's not surprising. Sarah Sharp -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Robert Hancock on 12 Apr 2010 16:50 On Mon, Apr 12, 2010 at 12:56 PM, Sarah Sharp <sarah.a.sharp(a)linux.intel.com> wrote: > On Sat, Apr 10, 2010 at 11:02:53AM -0600, Robert Hancock wrote: >> On Sat, Apr 10, 2010 at 2:34 AM, Daniel Mack <daniel(a)caiaq.de> wrote: >> > On Fri, Apr 09, 2010 at 05:38:13PM -0600, Robert Hancock wrote: >> >> On Fri, Apr 9, 2010 at 10:50 AM, Sarah Sharp >> >> <sarah.a.sharp(a)linux.intel.com> wrote: >> >> > What makes you think that? �I've seen URB buffers with 64-bit DMA >> >> > addresses. �I can tell when the debug polling loop runs and I look at >> >> > the DMA addresses the xHCI driver is feeding to the hardware: >> >> > >> >> > Dev 1 endpoint ring 0: >> >> > xhci_hcd 0000:05:00.0: @71a49800 01000680 00080000 00000008 00000841 >> >> > >> >> > So the TRB at address 71a49800 is pointing to a buffer at address >> >> > 0x0008000001000680. >> >> >> >> I'm not sure why the address would be that huge, unless it's not >> >> actually a physical address, or there's some kind of remapping going >> >> on? >> >> >> >> > >> >> > If I'm setting a DMA mask wrong somewhere, or doing something else to >> >> > limit the DMA to 32-bit, then please let me know. >> >> >> >> The DMA mask for the controller isn't being set anywhere (in the >> >> version that's in Linus' current git anyway). In that case it'll >> >> default to 32-bit and any DMA mappings above 4GB will need to be >> >> remapped. The controller driver doesn't do the mapping itself but the >> >> USB core does, passing in the controller device as the one doing the >> >> DMA, so if the controller's DMA mask is set to 32-bit then the buffers >> >> passed in will get remapped/bounced accordingly. >> > >> > So if we're seeing physical addresses in the log above, and the xHCI >> > driver does not explicitly allow the USB core to use addresses above >> > 4GB, why shouldn't the same thing be true as well for EHCI? >> > (Which would then be exactly the case we're seeing) >> >> That is a bit suspicious, yes. With the DMA mask at default I don't >> expect that should happen. Sarah, what kind of traffic was happening >> when you saw that (bulk, isochronous, etc)? > > Ring 0 is the default control ring, so it must be control transfers. > That's the first control transfer on the ring (and it didn't fill), so > it must have come from the USB core. > > I was running the USB mass storage driver, and the bulk endpoint rings > don't have the high 32-bits of the address set. �It mostly uses the > scatter-gather interface, which calls usb_buffer_map_sg(), so that's not > surprising. Is this machine using an IOMMU or something which would be remapping physical addresses? That address 0x0008000001000680 seems huge, I don't see how it could even be a valid bus address otherwise.. -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Sarah Sharp on 12 Apr 2010 17:10 On Mon, Apr 12, 2010 at 02:39:04PM -0600, Robert Hancock wrote: > On Mon, Apr 12, 2010 at 12:56 PM, Sarah Sharp > <sarah.a.sharp(a)linux.intel.com> wrote: > > On Sat, Apr 10, 2010 at 11:02:53AM -0600, Robert Hancock wrote: > >> On Sat, Apr 10, 2010 at 2:34 AM, Daniel Mack <daniel(a)caiaq.de> wrote: > >> > On Fri, Apr 09, 2010 at 05:38:13PM -0600, Robert Hancock wrote: > >> >> On Fri, Apr 9, 2010 at 10:50 AM, Sarah Sharp > >> >> <sarah.a.sharp(a)linux.intel.com> wrote: > >> >> > What makes you think that? �I've seen URB buffers with 64-bit DMA > >> >> > addresses. �I can tell when the debug polling loop runs and I look at > >> >> > the DMA addresses the xHCI driver is feeding to the hardware: > >> >> > > >> >> > Dev 1 endpoint ring 0: > >> >> > xhci_hcd 0000:05:00.0: @71a49800 01000680 00080000 00000008 00000841 > >> >> > > >> >> > So the TRB at address 71a49800 is pointing to a buffer at address > >> >> > 0x0008000001000680. > >> >> > >> >> I'm not sure why the address would be that huge, unless it's not > >> >> actually a physical address, or there's some kind of remapping going > >> >> on? > >> >> > >> >> > > >> >> > If I'm setting a DMA mask wrong somewhere, or doing something else to > >> >> > limit the DMA to 32-bit, then please let me know. > >> >> > >> >> The DMA mask for the controller isn't being set anywhere (in the > >> >> version that's in Linus' current git anyway). In that case it'll > >> >> default to 32-bit and any DMA mappings above 4GB will need to be > >> >> remapped. The controller driver doesn't do the mapping itself but the > >> >> USB core does, passing in the controller device as the one doing the > >> >> DMA, so if the controller's DMA mask is set to 32-bit then the buffers > >> >> passed in will get remapped/bounced accordingly. > >> > > >> > So if we're seeing physical addresses in the log above, and the xHCI > >> > driver does not explicitly allow the USB core to use addresses above > >> > 4GB, why shouldn't the same thing be true as well for EHCI? > >> > (Which would then be exactly the case we're seeing) > >> > >> That is a bit suspicious, yes. With the DMA mask at default I don't > >> expect that should happen. Sarah, what kind of traffic was happening > >> when you saw that (bulk, isochronous, etc)? > > > > Ring 0 is the default control ring, so it must be control transfers. > > That's the first control transfer on the ring (and it didn't fill), so > > it must have come from the USB core. > > > > I was running the USB mass storage driver, and the bulk endpoint rings > > don't have the high 32-bits of the address set. �It mostly uses the > > scatter-gather interface, which calls usb_buffer_map_sg(), so that's not > > surprising. > > Is this machine using an IOMMU or something which would be remapping > physical addresses? That address 0x0008000001000680 seems huge, I > don't see how it could even be a valid bus address otherwise.. Oh, shoot, nevermind. That TRB has a slightly different format, where the setup data for the control transfer is substituted for the DMA buffer address. I'll have to look through my logs to see if there's any real 64-bit DMA addresses in there, and fix xHCI's DMA mask. Sarah Sharp -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Daniel Mack on 14 Apr 2010 06:20
On Wed, Apr 14, 2010 at 12:46:42AM +0100, Pedro Ribeiro wrote: > On 13 April 2010 19:22, Daniel Mack <daniel(a)caiaq.de> wrote: > > Some more ideas to nail this down would be to boot the machine with > > mem=4096M and mem=2048M to see whether this makes any difference. Also, > > I think it would be interesting to know whether the patch below helps. > > > > As the real reason seems entirely obfuscated, there's unfortunately need > > for some trial error approach. Pedro, as you're the only one who can > > actually reproduce the problem, do you see any chance to do that? > > > > Thanks, > > Daniel > > > > > > diff --git a/sound/usb/caiaq/audio.c b/sound/usb/caiaq/audio.c > > index 4328cad..26013be 100644 > > --- a/sound/usb/caiaq/audio.c > > +++ b/sound/usb/caiaq/audio.c > > @@ -560,7 +560,7 @@ static struct urb **alloc_urbs(struct snd_usb_caiaqdev *dev, int dir, int *ret) > > � � � � � � � �} > > > > � � � � � � � �urbs[i]->transfer_buffer = > > - � � � � � � � � � � � kmalloc(FRAMES_PER_URB * BYTES_PER_FRAME, GFP_KERNEL); > > + � � � � � � � � � � � kmalloc(FRAMES_PER_URB * BYTES_PER_FRAME, GFP_KERNEL | GFP_DMA); > > � � � � � � � �if (!urbs[i]->transfer_buffer) { > > � � � � � � � � � � � �log("unable to kmalloc() transfer buffer, OOM!?\n"); > > � � � � � � � � � � � �*ret = -ENOMEM; > > > > > > Good news, I can't trigger the interference with either: > - mem=2048m > - mem=4096m > - your patch Thanks! So the only thing I can do for now is submit exactly this patch. At least, it helps you and it shouldn't break anything. The question remains whether this type of memory should be used for all transfer_buffers. > Any idea why is mem=4096m different than a regular boot since I have 4GB anyway? On Fri, Apr 09, 2010 at 04:11:52PM -0600, Robert Hancock wrote: > If you have 4GB of RAM then almost certainly you have memory located > at addresses over 4GB. If you look at the e820 memory map printed at > the start of dmesg on bootup and see entries with addresses of > 100000000 or higher reported as usable, then this is the case. Could you post the these e820 line from your dmesg when booted with mem=4096? Daniel -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |