From: Larry on
On Mar 20, 9:08 pm, "Del Cecchi" <delcec...(a)> wrote:
> "Bernd Paysan" <> wrote in message

> Source routing and flow control.  This stuff was considered in SCI and
> InfiniBand. Perhaps even earlier.  What do you do with source routing
> if a node dies?
> God, I am starting to sound like Lynn.  :-)

All of this design space is pretty well laid out in Dally and Towles,
Principles and Practices of Interconnection Networks.

Even if you do not choose source routing, you have to be prepared for
recovery from hardware failures at the link and switch level. Once
you analyze the failure cases, and all the extra hardware to do
dynamic routing, source routing starts to look simple and effective.
Source routing also lets you schedule the network better when doing

But even without routing, the switches have to deal with link and
switch error detection, flow control, deadlock and livelock avoidance,

From: MitchAlsup on
On Mar 20, 4:46 pm, Bernd Paysan <> wrote:
> MitchAlsup wrote:
> > It appears to me that those routers collect an entire message before
> > shipping it forward. This is a pure performance loss (pure gain in
> > latency) compared to organizing the message where the first beat
> > contrains the routing information.
> I think the obvious thing was mentioned in the paper: "make the routers
> as simple as possible".  This means that the routing information
> contains a physical route (a sequence of turns - the most simple router
> is a butterfly router with two inputs, two outputs, and one bit of
> routing information), and the router just passes on packets as it knows
> the next hop from the first beat of the message.

The other option is the virtual router where the first beat of
information is the virtual route, and the router takes a beat to
lookup the physical port accociated wth the requests at hand. Adds one
clock of delay from pin to pin, solves a lot of problems.

From: nmm1 on
In article <80ms7jFaskU1(a)>,
Del Cecchi` <delcecchi(a)> wrote:
>Robert Myers wrote:
>Why are you so hostile towards me? SCI wasn't IBM at all. Infiniband
>sort of was but also a number of other companies.

Join the club :-)

InfiniBand was an example of something designed by a committee of
multinational companies - I am flabberghasted that it is as usable
at it is, given its complexity. But it assuredly wasn't IBM plus
small fry, as some standards have been.

>My point was, to the extent that I had one, that the industry has spent
>a lot of time working out many of the ways to build switches for
>processor to processor networks. So the tradeoffs are known. You just
>have to pick your poison as it were.

Yup. Whether the current ones are the right ones is a very different
matter - as is whether switched networks are the right solution.
Debating that would at least be architecture.

>What is it you propose folks should build as a supercomputer? Or do you
>just prefer to howl at the moon because the National Labs and others are
>(allegedly) spending money inefficiently to perform (allegedly)
>inaccurate simulations?

Humans being what they are, and the problems being what they are, it
would be flabberghasting if they weren't. But that doesn't mean
that aren't doing as well as can reasonably be expected. No comment
on that one.

> [ Blue Gene ]
>The original motivation was to do molecular simulations in the bio-tech
>field, hence the name. Sure, IBM seized on the desire of the National
>Labs for prestige and bomb simulation and used it to make a profit.

Yup. I looked at it, and my objections aren't because it isn't as
efficient as it should be, but because it it paid too much attention
to performance and not enough to usability. And MOST of that is a
remark about the software, though not all.

Nick Maclaren.
From: Anne & Lynn Wheeler on

Del Cecchi` <delcecchi(a)> writes:
> The original motivation was to do molecular simulations in the
> bio-tech field, hence the name. Sure, IBM seized on the desire of the
> National Labs for prestige and bomb simulation and used it to make a
> profit.

or seized on national labs (& numerical intensive) only as possible
walling off move into commercial at the same time.

old email

old post about jan92 moving into commercial also

a few weeks before being told it was transferred and couldn't work on
anything with more than four processors.

old email, a couple days/hrs ... before the hammer fell

discussing the national lab scenario (I had to skip a LLNL meeting
because of other commitments ... but some of the people at the
meeting dropped by afterwards to bring me up to date).

then the press item shortly after the hammer fell (17feb92)

and another press item later that summer (we were both gone within a few

the kingston engineering & scientific had been doing molecular
simulation with numerous Floating Point Systems boxes tied to 3090 with
vector facility.

In 1980, I had done some HYPERChannel work to allow overflow in the
Santa Teresa lab. (300 people from IMS group) to be moved to offsite
bldg ... but getting local interactive performance using HYPERChannel as
mainframe channel extension. Then basically did the same installation
for large IMS field support group in boulder. recent reference

The person that I worked with for the Boulder installation then moved to
Kingston to manage the Kingston E&S operation. I worked with him there
to do high-speed HYPERChannel satellite link between Kingston E&S lab
and the west coast. This was somewhat totally unrelated to the
operation that was supposedly designing their own numerical intensive
supercomputer and also providing funding for Steve Chen's effort.
recent post with a little more of the gory details: Happy DEC-10 Day

The above tended to have some LLNL ties, in part because early backing
for FCS was standards moving to fiber-optics ... something that LLNL had
installed in serial-copper form.

The SCI stuff was with Gustafson out of SLAC.

Later one of the sparc-10 engineers was at another chip-shop and
designing a fast/inexpensive SCI subset ... and tried to interest me
into taking over the SUN SPRING/DOE operating system effort and adapting
it to a large distributed SCI infrastructure. This was about the time
SUN was shutting down the SPRING/DOE effort and transferring everybody
over to Java group.

42yrs virtualization experience (since Jan68), online at home since Mar1970
From: Bernd Paysan on
Del Cecchi wrote:
> Source routing and flow control. This stuff was considered in SCI and
> InfiniBand. Perhaps even earlier. What do you do with source routing
> if a node dies?

Well, source routing isn't necessary static. You have to ask someone
for the route, and this "someone" (the distributed routing table) will
change when a path or node dies. Usually, the "died" message would be
part of the flow control ("Hey, I can't send your packets any longer, my
next hop died"), and the reaction would be to look for an alternative
path or even an alternative destination.

It is stupid to make a highly critical part (the router must be as fast
and as simple as possible) more intelligent to handle rare exceptions
when you can handle that somewhere else, even though it might be handled
slower. It's an exception, it doesn't happen all the time.

Bernd Paysan
"If you want it done right, you have to do it yourself"