From: Mike Travis on


Jesse Barnes wrote:
> On Fri, 14 May 2010 15:34:01 -0700
> Mike Travis <travis(a)sgi.com> wrote:
>
>>
>> Jesse Barnes wrote:
>>> On Thu, 13 May 2010 14:02:30 -0600
>>> Bjorn Helgaas <bjorn.helgaas(a)hp.com> wrote:
>>>>>> This issue is not specific to x86, so I don't really like having
>>>>>> the implementation be x86-specific.
>>>>> I agree this isn't a x86 specific issue but given the 'norom'
>>>>> cmdline option is basically doing the same thing (but for pci
>>>>> Expansion ROM BARs) this code was modeled after it.
>>>> IMHO, we should fix both.
>>> Yeah, that would be good. Mike, have you looked at this at all?
>>>
>>> Also, to clarify, this isn't affecting users today, right? Or do you
>>> need all this I/O space for multiple IOHs and the drivers that bind to
>>> them in current UV systems?
>> We have customers that want to install more than 16 PCI-e cards right
>> now. Our window of opportunity closes very soon (days), so either this
>> patch makes it in as is (or something close), or we wait for another
>> release cycle. UV shipments start this month.
>>
>> [I wouldn't mind working on an improvement for later.]
>
> Wow and they're using cards that want to use I/O space? Funky. It's
> too late to get this into 2.6.34, but that can't be what you were
> expecting... I don't see a problem with getting something like this in
> for 2.6.35.

2.6.35 would be fine. It's the acceptance that's the key.

And yes, we're using standard cards like everyone else... ;-)

[The message is "UV" is just a really, really big PC. ;-)]

I would appreciate however, some more detail on what's the goal of the
updates to "fix both". Thanks!

>
>>> Fundamentally, until we have real dynamic PCI resource management (i.e.
>>> driver hooks for handling relocation, lazy allocation of resources at
>>> driver bind time, etc.) we're going to continue to need hacks like
>>> this. However, we could make them slightly more automated by making
>>> "nobar" and "norom" the default on systems that typically need them,
>>> maybe with a DMI table.
>> It seems that BIOS changes are much more difficult. The real solution
>> to this problem is for Card Vendors to not request I/O Bars if they
>> won't be using them. But that's the hardest option of all to accomplish.
>
> Right.
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Mike Travis on


Jesse Barnes wrote:
> On Fri, 14 May 2010 15:59:11 -0700
> Mike Travis <travis(a)sgi.com> wrote:
>
>>
>> Jesse Barnes wrote:
>>> On Fri, 14 May 2010 15:34:01 -0700
>>> Mike Travis <travis(a)sgi.com> wrote:
>>>
>>>> Jesse Barnes wrote:
>>>>> On Thu, 13 May 2010 14:02:30 -0600
>>>>> Bjorn Helgaas <bjorn.helgaas(a)hp.com> wrote:
>>>>>>>> This issue is not specific to x86, so I don't really like having
>>>>>>>> the implementation be x86-specific.
>>>>>>> I agree this isn't a x86 specific issue but given the 'norom'
>>>>>>> cmdline option is basically doing the same thing (but for pci
>>>>>>> Expansion ROM BARs) this code was modeled after it.
>>>>>> IMHO, we should fix both.
>>>>> Yeah, that would be good. Mike, have you looked at this at all?
>>>>>
>>>>> Also, to clarify, this isn't affecting users today, right? Or do you
>>>>> need all this I/O space for multiple IOHs and the drivers that bind to
>>>>> them in current UV systems?
>>>> We have customers that want to install more than 16 PCI-e cards right
>>>> now. Our window of opportunity closes very soon (days), so either this
>>>> patch makes it in as is (or something close), or we wait for another
>>>> release cycle. UV shipments start this month.
>>>>
>>>> [I wouldn't mind working on an improvement for later.]
>>> Wow and they're using cards that want to use I/O space? Funky. It's
>>> too late to get this into 2.6.34, but that can't be what you were
>>> expecting... I don't see a problem with getting something like this in
>>> for 2.6.35.
>> 2.6.35 would be fine. It's the acceptance that's the key.
>>
>> And yes, we're using standard cards like everyone else... ;-)
>>
>> [The message is "UV" is just a really, really big PC. ;-)]
>>
>> I would appreciate however, some more detail on what's the goal of the
>> updates to "fix both". Thanks!
>
> As Bjorn noted, both the norom and nobar options are listed as x86
> specific in the documentation and use x86 specific flags and code to
> prevent their respective allocations.
>
> It would be good if we could move the flags and code to the common PCI
> layer in drivers/pci so that other arches could take advantage of it,
> and we could have less resource management fragmentation.
>

Thanks! I was hoping you weren't going to say we need to remove the
need for these options.

I think the dynamic provisioning thing would be a great feature for the
future, but it seems to me that it would entail quite a bit of coordinated
effort between BIOS and the kernel? Not to mention the final death of
all POST related initialization (at least in the case where the device
is not being used in Legacy mode)? [Heck, can we kill POST too?!?] I
haven't looked too closely at non-x86 arch's but unless they run an
x86 emulator, then the POST initialization is pretty much superfluous,
isn't it? The drivers will initialize when the device comes online?
I mean we do have more the 64k of memory now... ;-)]

Thanks,
Mike




--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: H. Peter Anvin on
On 05/14/2010 03:47 PM, Jesse Barnes wrote:
>
> Wow and they're using cards that want to use I/O space? Funky. It's
> too late to get this into 2.6.34, but that can't be what you were
> expecting... I don't see a problem with getting something like this in
> for 2.6.35.
>

Most cards on the market provide I/O BARs as a convenience to legacy
BIOSes; they don't need the I/O BAR functionality from inside a
full-featured OS. There are a few, key, exceptions, mainly in the form
of legacy-interface devices like UARTs and VGA (note that VGA has its
own routing bits and is therefore unaffected by this problem.)

-hpa
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: H. Peter Anvin on
On 05/14/2010 04:28 PM, Jesse Barnes wrote:
> On Fri, 14 May 2010 16:20:45 -0700
> "H. Peter Anvin" <hpa(a)zytor.com> wrote:
>
>> On 05/14/2010 03:47 PM, Jesse Barnes wrote:
>>>
>>> Wow and they're using cards that want to use I/O space? Funky. It's
>>> too late to get this into 2.6.34, but that can't be what you were
>>> expecting... I don't see a problem with getting something like this in
>>> for 2.6.35.
>>>
>>
>> Most cards on the market provide I/O BARs as a convenience to legacy
>> BIOSes; they don't need the I/O BAR functionality from inside a
>> full-featured OS. There are a few, key, exceptions, mainly in the form
>> of legacy-interface devices like UARTs and VGA (note that VGA has its
>> own routing bits and is therefore unaffected by this problem.)
>
> Yeah, it's the "legacy" part that I'm questioning. I'm just lamenting
> that it's dying off so slowly...
>
> And yes, VGA is an unfortunate standard.
>

I'm not lamenting that fact, because my experience is that what ends up
replacing it is often far worse. Consider UARTs -- no MMU dependencies
at all, can be accessed with four lines of assembly, and compare it to
EHCI debug port, the driver for which is over 900 lines in the Linux
kernel -- and that assumes that you're already in flat mode.

-hpa
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: H. Peter Anvin on
On 05/14/2010 04:33 PM, Jesse Barnes wrote:
>
> As for BIOS coordination for dynamic reallocation, yeah there'd be some
> of that. I think the basic principles would be:
> 1) use BIOS allocations wherever possible
> 2) get an accurate list of available resources from the BIOS for
> potential remapping later
> 3) allocate resources for BARs and devices as late as possible (e.g.
> at driver bind time) to avoid allocating more than we need
>
> But that's a good chunk of work, and as we've seen, PCs in particular
> are really sensitive to having resources moved around too much, so step
> (2) is probably the hardest part.
>

The real problem that I see, as outlined before, has nothing to do with
the BIOS, but rather the interdependencies between resources.

-hpa

--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/