a potential lisp convert, and interpreting the shootout
in [Lisp]
|
Prev: Pocket Lisp Machine
Next: Next Generation of Language
From: Isaac Gouy on 1 Oct 2006 13:17 Henry Bigelow wrote: > Juho Snellman wrote: -snip- > > > p.s. what would you say to changing the shootout such that multiple > > > programs can be submitted, and the results shown based on whatever > > > selection criteria wanted? > > > > Sounds like a horrible idea. You want *more* crappy programs for > > people to whine about? :-) > > yes! :) Multiple programs can be submitted, and the results for all those programs are shown. Occasionally we go through and throw-out slower programs (unless they seem interesting). And then there's "FAQ: Who's working on similar projects?" http://shootout.alioth.debian.org/gp4/faq.php#similar
From: Jon Harrop on 1 Oct 2006 15:01 Henry Bigelow wrote: > hi mr. harrop! i was surprised to see your post over here on c.l.l! > did you start on lisp before o'caml? I started on SML and moved to OCaml. Then I toyed with some of the other FPLs, including Lisp. > so, is that shootout the best one > available, despite the grim lifecycle descriptions? Yes, AFAIK. > from your and juho's descriptions, the owners sound a bit deceitful! Only Isaac Gouy has tried to deceive people, AFAIK. I have two fundamental disagreements with the shootout: 1. Benchmarks should be a function of a non-trivial input to obviate arbitrary precomputation, e.g. computing the digits of pi is a stupid benchmark. 2. Program selection should be objective to evade any notion of "cheating", e.g. see the "Note"s in binary-trees. If I were to create a shootout, I would use benchmarks with variable inputs and I would try to remove all subjective criteria. For example, to test the performance and implementation of balanced binary trees in different languages I would try to define a benchmark that was best solved using such trees, rather than trying to force programmers to use trees. This might be something like: "use the given random number generator with seed "x" to generate "n" sets of "m" random numbers in different ranges and then compute the intersection of the sets, with "x", "n" and "m" given on the command line". (When the overlap of two sets is small, set-theoretic operations are faster with trees than with hash tables.) Unfortunately, my constructive criticisms fell on one of Isaac's deaf ears. ;-) -- Dr Jon D Harrop, Flying Frog Consultancy Objective CAML for Scientists http://www.ffconsultancy.com/products/ocaml_for_scientists
From: Isaac Gouy on 1 Oct 2006 17:41 Jon Harrop wrote: > Henry Bigelow wrote: > > hi mr. harrop! i was surprised to see your post over here on c.l.l! > > did you start on lisp before o'caml? > > I started on SML and moved to OCaml. Then I toyed with some of the other > FPLs, including Lisp. > > > so, is that shootout the best one > > available, despite the grim lifecycle descriptions? > > Yes, AFAIK. > > > from your and juho's descriptions, the owners sound a bit deceitful! > > Only Isaac Gouy has tried to deceive people, AFAIK. That is no more than malicious personal abuse - you contributed similar baseless personal attacks to the shootout mailing-list. > I have two fundamental disagreements with the shootout: > > 1. Benchmarks should be a function of a non-trivial input to obviate > arbitrary precomputation, e.g. computing the digits of pi is a stupid > benchmark. > > 2. Program selection should be objective to evade any notion of "cheating", > e.g. see the "Note"s in binary-trees. > > If I were to create a shootout, I would use benchmarks with variable inputs > and I would try to remove all subjective criteria. > > For example, to test the performance and implementation of balanced binary > trees in different languages I would try to define a benchmark that was > best solved using such trees, rather than trying to force programmers to > use trees. This might be something like: "use the given random number > generator with seed "x" to generate "n" sets of "m" random numbers in > different ranges and then compute the intersection of the sets, > with "x", "n" and "m" given on the command line". (When the overlap of two > sets is small, set-theoretic operations are faster with trees than with > hash tables.) > > Unfortunately, my constructive criticisms fell on one of Isaac's deaf > ears. ;-) I heard them the first and second time, but stopped listening as they were repeated month after month. Jack Andrews created "The shootin" partly in response to criticisms of the shootout. I think it's sad that in the past year you haven't taken the opportunity to be constructive and contribute to "The shootin". http://shootin.sourceforge.net/ > > -- > Dr Jon D Harrop, Flying Frog Consultancy > Objective CAML for Scientists > http://www.ffconsultancy.com/products/ocaml_for_scientists
From: Henry Bigelow on 1 Oct 2006 18:47 Isaac Gouy wrote: > Henry Bigelow wrote: > > Juho Snellman wrote: > -snip- > > > > > p.s. what would you say to changing the shootout such that multiple > > > > programs can be submitted, and the results shown based on whatever > > > > selection criteria wanted? > > > > > > Sounds like a horrible idea. You want *more* crappy programs for > > > people to whine about? :-) > > > > yes! :) > > Multiple programs can be submitted, and the results for all those > programs are shown. Occasionally we go through and throw-out slower > programs (unless they seem interesting). hi isaac, thanks for your reply. and thanks especially for maintaining the shootout. it was a very enlightening day when i first came across it a few years ago. i think the territory of benchmarks and contests is unavoidably contentious. and in some ways this is a positive thing. (although in others, unfortunately, negative) to mitigate the frustration of having a benchmark deleted, (even if it becomes invalid due to a change in requirements), would you consider a CVS tree, one for each benchmark/language combination? or would that be impractical? the branches or leaves could be labeled by benchmark criteria, and whether they satisfy it, and the running statistics, and possibly even the version of the compiler that compiled it. that way, it would be plain to everyone who wants to know whether a given version was producing the correct output, and in certain cases, what tweaks made something run faster. that would be interesting to me at least. of course, it would have the drawback of accumulating 'dead' leaves, maybe even spam. depending on how prevalent it was, i suppose this approach would become impractical. i don't know. also, i feel the concept of a shootout and shootin are orthgonal. i'd be interested to follow the development of both. thanks in advance, henry
From: Isaac Gouy on 1 Oct 2006 20:38
Henry Bigelow wrote: > Isaac Gouy wrote: > > Henry Bigelow wrote: > > > Juho Snellman wrote: > > -snip- > > > > > > > p.s. what would you say to changing the shootout such that multiple > > > > > programs can be submitted, and the results shown based on whatever > > > > > selection criteria wanted? > > > > > > > > Sounds like a horrible idea. You want *more* crappy programs for > > > > people to whine about? :-) > > > > > > yes! :) > > > > Multiple programs can be submitted, and the results for all those > > programs are shown. Occasionally we go through and throw-out slower > > programs (unless they seem interesting). > > hi isaac, > thanks for your reply. and thanks especially for maintaining the > shootout. it was a very enlightening day when i first came across it a > few years ago. > > i think the territory of benchmarks and contests is unavoidably > contentious. and in some ways this is a positive thing. (although in > others, unfortunately, negative) > > to mitigate the frustration of having a benchmark deleted, (even if it > becomes invalid due to a change in requirements), would you consider a > CVS tree, one for each benchmark/language combination? or would that > be impractical? > > the branches or leaves could be labeled by benchmark criteria, and > whether they satisfy it, and the running statistics, and possibly even > the version of the compiler that compiled it. that way, it would be > plain to everyone who wants to know whether a given version was > producing the correct output, and in certain cases, what tweaks made > something run faster. that would be interesting to me at least. > > of course, it would have the drawback of accumulating 'dead' leaves, > maybe even spam. depending on how prevalent it was, i suppose this > approach would become impractical. i don't know. > > also, i feel the concept of a shootout and shootin are orthgonal. i'd > be interested to follow the development of both. > > thanks in advance, > > henry It's not clear to me what problem your suggestion is supposed to solve. I think the place to understand "what tweaks made something run faster" is within a particular language community like this: http://www.haskell.org/hawiki/ShootoutEntry The old Doug Bagley benchmarks were mostly replaced by new benchmarks during autumn/winter 2005 - I haven't been keeping track but I don't think the benchmarks have been changed this year. |