x86, pci: Handle fallout pci devices with peer root bus
in [Kernel]
|
Prev: TCP connections on 'lo' interface randomly stall
Next: vmscan: Setup pagevec as late as possible in shrink_inactive_list()
From: Bjorn Helgaas on 14 Jun 2010 14:40 On Monday, June 14, 2010 11:47:59 am Yinghai Lu wrote: > > Graham bisected > | commit 3e3da00c01d050307e753fb7b3e84aefc16da0d0 > | x86/pci: AMD one chain system to use pci read out res > > cause the SND_HDA_INTEL doesn't work anymore. > > https://bugzilla.kernel.org/show_bug.cgi?id=16007 > > It turns out that his system with via chipset only have one hypertransport > chain, but does have one extra orphan device 80:01.0 > > PCI: Probing PCI hardware (bus 00) > PCI: Discovered primary peer bus 80 [IRQ] > > node 0 link 0: io port [1000, ffffff] > TOM: 0000000080000000 aka 2048M > node 0 link 0: mmio [e0000000, efffffff] > node 0 link 0: mmio [a0000, bffff] > node 0 link 0: mmio [80000000, ffffffff] > bus: [00, ff] on node 0 link 0 > > Try to make peer root buses to share same mmio/io resources if those peer root > buses fall into the same bus range. Yinghai, did you read https://bugzilla.kernel.org/show_bug.cgi?id=16007#c15 ? I made the point there that an HT chain may contain multiple HT/PCI host bridges, but you are stuck on the idea that "one HT chain == one PCI root bus." I have not found the "one PCI host bridge per HT chain" requirement in the HT spec (if you find it, please point me to it). If an HT chain may contain multiple HT/PCI host bridges, then it's obvious that the HT host bridge registers read by amd_bus.c don't contain enough information to correctly assign address space to the PCI root buses. > Also need to update insert_resource to avoid insert same resource two times. > > We need this patch for 2.6.34 stable. No, we don't! Not yet, anyway. We need to find the root cause of this problem, not just paper over it and wait for it to pop up again somewhere else. > Reported-by: Graham Ramsey <ramsey.graham(a)ntlworld.com> > Bisected-by: Graham Ramsey <ramsey.graham(a)ntlworld.com> > Tested-by: Graham Ramsey <ramsey.graham(a)ntlworld.com> > Signed-off-by: Yinghai Lu <yinghai(a)kernel.org> > Cc: stable(a)kernel.org > > --- > arch/x86/pci/bus_numa.c | 3 ++- > kernel/resource.c | 2 +- > 2 files changed, 3 insertions(+), 2 deletions(-) > > Index: linux-2.6/arch/x86/pci/bus_numa.c > =================================================================== > --- linux-2.6.orig/arch/x86/pci/bus_numa.c > +++ linux-2.6/arch/x86/pci/bus_numa.c > @@ -22,7 +22,8 @@ void x86_pci_root_bus_res_quirks(struct > return; > > for (i = 0; i < pci_root_num; i++) { > - if (pci_root_info[i].bus_min == b->number) > + if (pci_root_info[i].bus_min <= b->number && > + pci_root_info[i].bus_max >= b->number) > break; > } > > Index: linux-2.6/kernel/resource.c > =================================================================== > --- linux-2.6.orig/kernel/resource.c > +++ linux-2.6/kernel/resource.c > @@ -451,7 +451,7 @@ static struct resource * __insert_resour > if (!first) > return first; > > - if (first == parent) > + if (first == parent || first == new) > return first; > > if ((first->start > new->start) || (first->end < new->end)) > -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Bjorn Helgaas on 14 Jun 2010 15:50 On Monday, June 14, 2010 12:39:54 pm H. Peter Anvin wrote: > On 06/14/2010 11:34 AM, Bjorn Helgaas wrote: > > > > I made the point there that an HT chain may contain multiple HT/PCI > > host bridges, but you are stuck on the idea that "one HT chain == one > > PCI root bus." > > > > I have not found the "one PCI host bridge per HT chain" requirement > > in the HT spec (if you find it, please point me to it). > > > > If an HT chain may contain multiple HT/PCI host bridges, then it's > > obvious that the HT host bridge registers read by amd_bus.c don't > > contain enough information to correctly assign address space to the > > PCI root buses. > > A HT-to-PCI bridge appears as a PCI-to-PCI bridge (i.e. a Header Type 1 > device), not as a host bridge (a Header Type 0 device). > > That is at least the software model as defined. Certainly that's what the HT I/O Link spec (v3.10, sec 7.4) suggests, and I think I saw hints that AMD chipsets do that. I can't tell from the HT I/O spec whether it would be an actual defect to use host bridges instead of PCI-to-PCI bridges, and I can imagine why one might want to leave an existing PCI host bridge design alone and merely glue on an HT interface, rather than redesign the bridge register set. In any case, the VIA chipset in Graham's machine does not have a PCI-to-PCI bridge leading to bus 80 (see https://bugzilla.kernel.org/show_bug.cgi?id=16007#c14). However, ACPI *does* report a PCI host bridge leading to bus 80, and the apertures it reports seem to be correct (see https://bugzilla.kernel.org/show_bug.cgi?id=16007#c6). Bjorn -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Bjorn Helgaas on 14 Jun 2010 16:10 On Monday, June 14, 2010 12:55:44 pm Yinghai Lu wrote: > On 06/14/2010 11:39 AM, H. Peter Anvin wrote: > > On 06/14/2010 11:34 AM, Bjorn Helgaas wrote: > >> > >> I made the point there that an HT chain may contain multiple HT/PCI > >> host bridges, but you are stuck on the idea that "one HT chain == one > >> PCI root bus." > > should be. > > >> I have not found the "one PCI host bridge per HT chain" requirement > >> in the HT spec (if you find it, please point me to it). > > according to my experience with LinuxBIOS. AMD chipset, nvidia and serverworks (broadcom) I'm afraid I'm still not convinced. > >> If an HT chain may contain multiple HT/PCI host bridges, then it's > >> obvious that the HT host bridge registers read by amd_bus.c don't > >> contain enough information to correctly assign address space to the > >> PCI root buses. > > the host bridges is on AMD CPUs, Don't confuse the HT host bridge with the PCI host bridge. The HT I/O spec is quite clear that it uses "host bridge" to refer to the HT host bridge, i.e., the interface between CPUs and a HyperTransport link. I agree that the *HT host bridge* is indeed on the AMD CPU. But that is certainly not the same as the PCI host bridge that bridges between an HT link and a PCI bus. See sections 4.9.4 (HT host bridge) and 7.4 (HT/PCI host bridge), for example. Bjorn -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Bjorn Helgaas on 14 Jun 2010 16:30 On Monday, June 14, 2010 02:08:37 pm H. Peter Anvin wrote: > On 06/14/2010 01:00 PM, Bjorn Helgaas wrote: > >> > >> the host bridges is on AMD CPUs, > > > > Don't confuse the HT host bridge with the PCI host bridge. The HT I/O spec > > is quite clear that it uses "host bridge" to refer to the HT host bridge, > > i.e., the interface between CPUs and a HyperTransport link. > > > > I agree that the *HT host bridge* is indeed on the AMD CPU. But that is > > certainly not the same as the PCI host bridge that bridges between an HT > > link and a PCI bus. > > > > See sections 4.9.4 (HT host bridge) and 7.4 (HT/PCI host bridge), for > > example. > > From a software point of view the latter is [largely] a PCI-to-PCI > bridge, though; it's not a root-level host bridge in the classical sense > (as noted in section 7.4). OK, but Graham's system doesn't have anything resembling a PCI-to-PCI bridge leading to bus 80. So while I agree that in an ideal world, HT/PCI host bridges might always look like PCI-to-PCI bridges, it seems this is not the case in practice. > Incidentally, in my copy of HT 3.10b, section 7.4 is marked > "HyperTransport Bridge Headers", and does not use the term "host bridge" > to refer to a secondary PCI bus. Section 4.9.4 is simply marked "Host > Bridge". As such, I think the HT spec is pretty consistent about > unambiguously referring to the HT host bridge when using the term "host > bridge". Yes, absolutely. My point is that what the HT spec means by "host bridge" is not the same as what the PCI spec and Linux mean by "PCI host bridge". Those are two completely separate functions, and I think Yinghai is confusing them when he says "the host bridge is on the AMD CPU and amd_bus.c uses its config to determine PCI root bus resources." Bjorn -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Bjorn Helgaas on 14 Jun 2010 22:00
On Monday, June 14, 2010 03:10:20 pm H. Peter Anvin wrote: > On 06/14/2010 01:20 PM, Bjorn Helgaas wrote: > > > > OK, but Graham's system doesn't have anything resembling a PCI-to-PCI > > bridge leading to bus 80. So while I agree that in an ideal world, > > HT/PCI host bridges might always look like PCI-to-PCI bridges, it > > seems this is not the case in practice. > > Invisible PCI bridges have been known to occur in pure PCI space, too. Are you talking about PCI host bridges that don't appear in PCI config space? I suppose those could be described as "invisible," but since host bridges aren't architected and their primary interface isn't PCI, it seems only natural that we'd discover them by a non-PCI mechanism. They're invisible in PCI terms, but obviously perfectly discoverable and configurable via ACPI. If you ask me, it's weird that most x86 chipsets put PCI host bridge configuration in PCI config space -- it may be convenient in some ways, but still architecturally strange. > > Yes, absolutely. My point is that what the HT spec means by "host bridge" > > is not the same as what the PCI spec and Linux mean by "PCI host bridge". > > Actually, they're *exactly* the same thing. If HT is identical to PCI, I agree "HT host bridge" means the same as "PCI host bridge" (that's almost too trivial to say :-)). I guess I'm still dubious that HT is identical to PCI. Since Graham's box has a single HT change, we know that all his devices are on HT chain A. If HT is identical to PCI, that chain must be bus 00. Here are the relevant parts of the box: 00:00.0 Host bridge: VIA K8T890CF Host Bridge 00:00.1 Host bridge: VIA VT3351 Host Bridge 00:00.2 Host bridge: VIA VT3351 Host Bridge 00:00.3 Host bridge: VIA VT3351 Host Bridge 00:00.4 Host bridge: VIA VT3351 Host Bridge 00:00.7 Host bridge: VIA VT3351 Host Bridge 00:02.0 PCI bridge: VIA K8T890 PCI to PCI Bridge [to bus 06] 00:03.0 PCI bridge: VIA K8T890 PCI to PCI Bridge [to bus 05] 00:03.1 PCI bridge: VIA K8T890 PCI to PCI Bridge [to bus 04] 00:03.2 PCI bridge: VIA K8T890 PCI to PCI Bridge [to bus 03] 00:03.3 PCI bridge: VIA K8T890 PCI to PCI Bridge [to bus 02] 00:11.7 Host bridge: VIA VT8251 Ultra VLINK Controller 00:13.0 Host bridge: VIA VT8237A Host Bridge 00:18.0 Host bridge: AMD HyperTransport Technology Configuration 80:01.0 Audio device: VIA VT1708/A High Definition Audio Controller The question is "how do we get to bus 80?" If everything behind the AMD HT host bridge is PCI and can be understood solely in terms of PCI specs, there must be a P2P bridge from bus 00 to bus 80. We clearly don't have that. I suppose one could argue that there's a non-standard P2P bridge from bus 00 to bus 80, but I can't imagine anybody doing that. An OS would have to have vendor-specific code just to do PCI resource management, and that really misses the point of PCI. It seems more likely to me that one of the VIA host bridges leads to bus 80. PCI host bridges are not architected, so if this bridge lives on HT chain 00, and we can think of HT as "not quite PCI," then it seems natural that the host bridge would be VIA-specific, just like it was in pre-HT days. The underlying question for all of this is "what's the future of amd_bus.c?" or stated another way, "does AMD HT config and standard PCI P2P bridge config tell us everything we need to know about address space routing?" On this machine, I claim the answer is "no," and therefore we must use ACPI to discover and configure the host bridges, i.e., we have to turn on "pci=use_crs". We currently turn it on automatically for machines from 2008 and newer. I think we need to do it for older machines, too, perhaps even whenever we use ACPI at all. Bjorn -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |