From: nmm1 on
In article <gnnu67-i192.ln1(a)ntp.tmsw.no>,
Terje Mathisen <"terje.mathisen at tmsw.no"> wrote:
>Robert Myers wrote:
>> I'm slowly doing my catchup homework on what the national wisdom is on
>> bisection bandwidth. Not too surprisingly, there a plenty of people
>> out there who know that it's already a big problem, and that it is
>> only going to get bigger, as there is no Moore's Law for bandwidth.
>
>Huh?
>
>Sure there is, it is driven by the same size shrinks as regular ram and
>cpu chips have enjoyed.

Of course. But what he has probably reforgotten is that bisection
bandwidth is a completely damn-fool measure in the first place.

There are at least half a dozen 'obvious' definitions of it, some
of which are easy to get to scale and others are not so easy.
In general, the easier they are to get to scale, the less they are
correlated with application performance. Before discussing its
technical merits, it must be defined.

When I was procuring supercomputers, I found that I could usually
find the bisection bandwidth in GB/sec, but rarely find out which
definition had been used to measure it! In some cases, I was sure
that it was the marketdroids' choice, which is almost completely
irrelevant to overall performance.


Regards,
Nick Maclaren.
From: Robert Myers on
On Mar 14, 2:44 pm, n...(a)cam.ac.uk wrote:
> In article <gnnu67-i192....(a)ntp.tmsw.no>,
> Terje Mathisen  <"terje.mathisen at tmsw.no"> wrote:
>
> >Robert Myers wrote:
> >> I'm slowly doing my catchup homework on what the national wisdom is on
> >> bisection bandwidth.   Not too surprisingly, there a plenty of people
> >> out there who know that it's already a big problem, and that it is
> >> only going to get bigger, as there is no Moore's Law for bandwidth.
>
> >Huh?
>
> >Sure there is, it is driven by the same size shrinks as regular ram and
> >cpu chips have enjoyed.
>
> Of course.  But what he has probably reforgotten is that bisection
> bandwidth is a completely damn-fool measure in the first place.
>
> There are at least half a dozen 'obvious' definitions of it, some
> of which are easy to get to scale and others are not so easy.
> In general, the easier they are to get to scale, the less they are
> correlated with application performance.  Before discussing its
> technical merits, it must be defined.
>
> When I was procuring supercomputers, I found that I could usually
> find the bisection bandwidth in GB/sec, but rarely find out which
> definition had been used to measure it!  In some cases, I was sure
> that it was the marketdroids' choice, which is almost completely
> irrelevant to overall performance.
>
What? We have linpack supported as meaningful and bisection bandwidth
dismissed as a damn-fool measure?

I'm not worried about factors of two. No matter how you measure it,
the limiting bisection bandwidth per flop of Blue Gene and computers
with a similar topology is ****zero****, unless you increase the
bandwidth of each link as you add flops.

I spend enough of my time swimming against the tide. You are the only
one so far not to agree that bisection bandwidth is a useful proxy for
global FFT performance.

Robert.
From: nmm1 on
In article <2f8c0fa9-3840-495d-aaa6-20c2636afdc7(a)33g2000yqj.googlegroups.com>,
Robert Myers <rbmyersusa(a)gmail.com> wrote:
>On Mar 14, 2:44=A0pm, n...(a)cam.ac.uk wrote:
>> In article <gnnu67-i192....(a)ntp.tmsw.no>,
>> Terje Mathisen =A0<"terje.mathisen at tmsw.no"> wrote:
>>
>> >> I'm slowly doing my catchup homework on what the national wisdom is on
>> >> bisection bandwidth. =A0 Not too surprisingly, there a plenty of peopl=
>e
>> >> out there who know that it's already a big problem, and that it is
>> >> only going to get bigger, as there is no Moore's Law for bandwidth.
>>
>> >Huh?
>>
>> >Sure there is, it is driven by the same size shrinks as regular ram and
>> >cpu chips have enjoyed.
>>
>> Of course. =A0But what he has probably reforgotten is that bisection
>> bandwidth is a completely damn-fool measure in the first place.
>>
>What? We have linpack supported as meaningful and bisection bandwidth
>dismissed as a damn-fool measure?

Please don't be ridiculous. Linpack is meaningful, because it's well
defined, though it's not very useful. Bisection bandwidth is a damn
fool measure because it isn't well-defined.

>I'm not worried about factors of two. ...

And I'm not talking about factors of two. Try ones of 10-100. No,
I am not joking. I have told you before what the problem is.

The 'best' (marketdroid) bisection bandwidth comes from connecting
the nodes in pairs, optimally for performance and irrespective of
usefulness of the topology.

You can also measure the minimum over all such divisions, or the
average over all divisions, or the median, or .... And you can also
constrain the set of divisions that is considered in any way that
tickles your fancy.

It isn't rare for the ratio of the best to worst (and sometimes best
to average) to be in the range I mention above.


Regards,
Nick Maclaren.
From: Del Cecchi` on
MitchAlsup wrote:
> On Mar 14, 12:15 pm, Robert Myers <rbmyers...(a)gmail.com> wrote:
>
>>On Mar 14, 5:43 am, Terje Mathisen <"terje.mathisen at tmsw.no">
>>
>>>I guess the real problem is that you'd like the total bandwidth to scale
>>>not just with the link frequencies but even faster so that it also keeps
>>>up by the increasing total number of ports/nodes in the system, without
>>>overloading the central mesh?
>>
>>At the chip (or maybe chip carrier) level, there are interesting
>>things you can do because of decreased feature sizes, as we have
>>recently discussed.
>
>
> One achieves maximal "routable" bandwidth at the "frame" scale . With
> todays current board technologies, this "frame" scale occurs around 1
> cubic meter.
>
> Consider a 1/2 meter sq motherboard with "several" CPU nodes with 16
> bidirectionial (about) byte wide ports running at 6-10 GTs. Now
> consider a back plane that simply couples this 1/2 sq meter
> motherboard to another 1/2 sq meter DRAM carring board also with 16
> bidirectional (almost) bite wide ports running at the same
> frequencies. Except, this time, the DRAM boards are perpendicular to
> the CPU boards. With this arrangement, we have 16 CPU containing
> motherboards fully connected to 16 DRAM containing motherboards and
> 256 (almost) byte wide connections running at 6-10 GTs. 1 cubic meter,
> about the actual size of a refrigerator. {Incidentally, this kind of
> system would have about 4TB/s of bandwidth to about 4TB of actual
> memory}
>
> Once you get larger than this, all of the wires actualy have to exist
> as wires (between "frames"), not just traces of coper on a board or
> through a connector, and one becomes wire bound connecting frames.
>
> Mitch

I went looking for the network characteristics of Blue Gene/P. In the
process I found this paper that, while not discussing BG/P does have
some interesting data.

http://www.cs.rochester.edu/u/sandhya/csc258/seminars/Patel_HPC.pdf

And this one on BG/P

http://workshops.alcf.anl.gov/gs10/files/2010/01/Morozov-BlueGeneP-Architecture.pdf

which appears to have data that should allow the calculation of various
types of bandwidths.

del
From: Del Cecchi on

"Anton Ertl" <anton(a)mips.complang.tuwien.ac.at> wrote in message
news:2010Mar14.192558(a)mips.complang.tuwien.ac.at...
> MitchAlsup <MitchAlsup(a)aol.com> writes:
>>Consider a 1/2 meter sq motherboard with "several" CPU nodes with 16
>>bidirectionial (about) byte wide ports running at 6-10 GTs. Now
>>consider a back plane that simply couples this 1/2 sq meter
>>motherboard to another 1/2 sq meter DRAM carring board also with 16
>>bidirectional (almost) bite wide ports running at the same
>>frequencies. Except, this time, the DRAM boards are perpendicular to
>>the CPU boards. With this arrangement, we have 16 CPU containing
>>motherboards fully connected to 16 DRAM containing motherboards and
>>256 (almost) byte wide connections running at 6-10 GTs. 1 cubic
>>meter,
>>about the actual size of a refrigerator.
>
> I compute 1/2m x 1/2m x 1/2m = 1/8 m^3.
>
> Where have I misunderstood you?
>
> But that size is the size of a small freezer around here (typical
> width 55-60cm, depth about the same, and the height of the small
> ones
> is around the same, with the normal-sized ones at about 1m height).
>
> Hmm, couldn't you have DRAM boards on both sides of the mainboard
> (if
> you find a way to mount the mainboard in the middle and make it
> strong
> enough). Then you can have a computer like a normal-size fridge:-).
>
> - anton

Golly, how much memory do you want. If a DIMM holds 2GB, you could
put a whole bunch of memory on the board with the CPUs rather than
putting it way over there on the other side of the interconnect and
with a switch that has to arbitrate all those processors beating on
that memory.

Or is there something I missed here?

del