My Library TaskSpeed tests updated
in [JavaScript]
|
From: David Mark on 19 Feb 2010 00:58 Scott Sauyet wrote: > On Feb 18, 5:09 pm, David Mark <dmark.cins...(a)gmail.com> wrote: >> Scott Sauyet wrote: >>> http://scott.sauyet.com/Javascript/Test/taskspeed/2010-02-15a/ >> I just ran it in (IETester) IE6. The only error I see is at the end >> (dojo is undefined). LOL. >> >> Of course, IETester is just some black box that could be doing anything >> behind the scenes. I normally test with a real IE6 installation. Just >> so happens that that is not convenient for me at the moment and I want >> to get to the bottom of this. So, what errors are you seeing? > > Unfortunately, all that shows up in the title of the cell is: > > [object Error] > > which is not at all useful. Right. Something screwy is going on there as I've had several reports of success. But all it takes is one failure to spoil the bunch. > > I made a simpler version just testing My Library in order to try to > run this down. It's at > > http://scott.sauyet.com/Javascript/Test/taskspeed/2010-02-17a/ I'll try it out, but I expect it will "pass" here again. > > It's happening for me on my home machine as well as my work one. I'm > using Multiple IEs [1], which installs multiple versions of IE on a > Windows machine. I've never run into any issues beyond the many > versions occasionally sharing some settings, but it could be a problem > with this version. Multiple IE versions can lead to screwy problems, but it galls me that my library is somehow tripping over this (and the others aren't). If you would, I would appreciate it if you would make a simpler test case. It will likely be a good learning exercise in any event (you've stumbled over something that is throwing me and I thought I'd seen it all in IE, multi or not). Paste the test function(s) into a page without the try-catch and see what it throws up. If you can tell me the error message and line, I'm sure I can figure it out and fix it without duplicating the issue here. I'm _very_ curious as to what the problem could be. Also, did you try IE7? I reviewed the related code and nothing jumped out at me, so I'm in the dark at this point. That's not somewhere I'm used to being when it comes to IE. It's particularly irritating as I had multi-IE (or something similar) on the test machine that went down recently. I wonder if the problem would have shown up on that one. Thanks again for your help Scott. Your efforts are _much_ appreciated!
From: Michael Wojcik on 18 Feb 2010 14:12 David Mark wrote: > Michael Wojcik wrote: >> >> I've been a professional developer for nearly a quarter of a century, >> and I'm not surprised at all when one person delivers a better >> solution than what a large team delivers. Even if the large team's >> total effort is much greater - which may or may not be the case here. > > I think I agree with all of this, but was confused by this statement. > What may or may not be the case? I'm one guy who took a two-year hiatus > from the library. Meanwhile hundreds (if not thousands) of people have > been filing tickets, arguing patches, actually applying come patches, > un-applying patches, maintaining repositories, testing browsers > (somewhat ineffectually), arguing about blog comments, etc. Make no > mistake that I did none of that. It's been me and Notepad and a few > weekends and evenings over the last month. Thanks to the handful of > people who gave feedback too. :) Now that you ask, I'm not sure why I added that final clause. It may be the result of overly-aggressive editing. Or I might have been thinking in general terms, but appended the "here" out of habit. -- Michael Wojcik Micro Focus Rhetoric & Writing, Michigan State University
From: David Mark on 19 Feb 2010 17:04 Scott Sauyet wrote: > On Feb 18, 5:09 pm, David Mark <dmark.cins...(a)gmail.com> wrote: >> Scott Sauyet wrote: >>> http://scott.sauyet.com/Javascript/Test/taskspeed/2010-02-15a/ >> I just ran it in (IETester) IE6. The only error I see is at the end >> (dojo is undefined). LOL. >> >> Of course, IETester is just some black box that could be doing anything >> behind the scenes. I normally test with a real IE6 installation. Just >> so happens that that is not convenient for me at the moment and I want >> to get to the bottom of this. So, what errors are you seeing? > > Unfortunately, all that shows up in the title of the cell is: > > [object Error] > > which is not at all useful. > > I made a simpler version just testing My Library in order to try to > run this down. It's at > > http://scott.sauyet.com/Javascript/Test/taskspeed/2010-02-17a/ > > It's happening for me on my home machine as well as my work one. I'm > using Multiple IEs [1], which installs multiple versions of IE on a > Windows machine. I've never run into any issues beyond the many > versions occasionally sharing some settings, but it could be a problem > with this version. I was actually thinking it was a problem with my build, but your suggestion seems more likely at this point (a problem with the multi-IE6). I created a simplified test page for you:- http://www.cinsoft.net/scotttest.html If it makes it all the way through, you will see a "good!" alert. There is no try-catch, so any error will be reported. I assume it will be the sethtml test as it is the first of the three that was reported as having a problem in your setup(s). Thanks again for your help! Will be very interesting to see what those errors are about.
From: David Mark on 19 Feb 2010 19:42 Cody Haines wrote: > Andrew Poulos <ap_prog(a)hotmail.com> wrote: >> On 15/02/2010 3:45 PM, David Mark wrote: >>> I ran it a few times. This is representative. The two versions >>> flip >>> flop randomly. Usually around a third of the purer tests. :) >> I'm not sure that if one person can write a library that's as good or >> better than libraries on which (I believe) teams of people have worked >> says a lot about one person's ability or not much about the others. >> > Both. He's one of the best there is, and he's also the first one to tell > you about how much the others suck. Thanks for the vote of confidence, Cody! I do try. >> I'm not meaning to be offensive, I'm just wondering how one person can >> appear to achieve so much. > I'm not sure if there was any way he could be offended there :) >> Andrew Poulos >> > P.S. I'm sending this from an iPhone app, so apologies if the formatting > is screwy > NP. Lots of people post from GG. :)
From: Richard Cornford on 21 Feb 2010 13:46
Scott Sauyet wrote: >On Feb 17, 11:04 am, Richard Cornford wrote: >> On Feb 16, 8:57 pm, Scott Sauyet wrote: > >>> I think that testing the selector engine is part of testing >>> the library. >> >> Obviously it is, if the 'library' has a selector engine, but that >> is a separate activity from testing the library's ability to >> carry out tasks as real world tasks don't necessitate any >> selector engine. > > Perhaps it's only because the test framework was built testing against > libraries that had both DOM manipulation and selector engines, Haven't we already agreed that the test framework was adapted directly from one that was designed to test selector engines, and so must have been for libraries with selector engines? > but these seem a natural fit. ? > I don't believe this was meant to be a DOM manipulation test > in particular. The name "taskspeed" implies pretty much that; that we are testing actual tasks. > My understanding (and I was not involved in any of the > original design, so take this with a grain of salt) is > that this was meant to be a more general test of how the > libraries were used, which involved DOM manipulation and > selector- based querying. Some of the libraries do their DOM manipulation via selector based querying. For them there is no alternative. But DOM manipulation tasks do not necessitate the use of selector engines (else DOM manipulation did not happen prior to about 2006). It is disingenuous to predicate DOM manipulation tasks on selector engine use. It makes much more sense to see how each library competes doing realistic tasks in whatever way best suites them, be it with selector engines or not. > If it seemed at all feasible, the framework would > probably have included event handler manipulation tests, > as well. So all those - addEventListener - and - attachEvent - calls are not event handler manipulations then? > If the libraries had all offered classical OO infrastructures > the way MooTools and Prototype do, that would probably also > be tested. Then that would have been another mistake as "offering classical OO" is not necessary for any real world tasks either. > Why the scare quotes around "library"? Is there a better > term -- "toolkit"? -- that describes the systems being tested? The "scare quotes" are an expression of 'so called'. "Library" is a perfectly acceptable name for these things, but tends to get used in these contexts with connotations that exclude many of the things that could also reasonably be considered to be libraries. For example, when John Resig wrote:- <URL: http://groups.google.fr/group/comp.lang.javascript/msg/56d22b30b168fbb5 > | Yawwwwnnnnn.... this is, quite possibly, the most inane | set of ramblings set up by someone who has obviously never | created a JavaScript library of any kind, nor has used | JavaScript code in any sort of production environment. | ... | Sigh. Where's your ultra-documented, totally-awesome, | perfect-in-every- way JavaScript library? That's what I | thought - you don't have one, nor does one exist. | ... | The same argument, over and over, is getting incredibly | old. Where's your ultra-popular library that is 100% | future proof? - his notion of what a javascript library would be is such that many of the things that I would categorise with that term would not be recognised as such by him. There is a notion of a 'library' being a specific thing, and the creation of that thing being an achievement. I prefer to question these notions, and make two observations. The first being that when asked for the rational justification for general purpose javascript libraries the answers given tend towards the superficial and the unthinking. Mostly when I ask the question no attempt to answer is given at all. A few people bother to assert that the answer is "obvious" (dubious because when the reason for something really is obvious it is also trivial to state that reason; meaning the fact that it has not been stated when asked for implies that it is not "obvious"). Sometimes the next position is that having pre-existing general libraries are is good idea in other languages so they must be a good idea in javascript, which denies the context of application, as downloading/compiling the equivalent of Java's 16 megabytes of libraries every time you want view a web page clearly would not be a good idea. Secondly, on the rare occasions when the discussion of the rationale for general purpose libraries gets beyond the superficial the position that ends up having the most substance is the code re-use argument. Trivially expressed as 'it is better to use a library than write everything from scratch each time you need it'. Trivial because it should be fairly obvious that only a complete fool would write everything from scratch each time they needed it, so in reality the people who do not use any of the 'popular' general purpose javascript libraries are not suffering from the absence of code re-use in their work (but rather employing a different approach to achieving it). Code re-use is important to me. I don't want to be doing the same work over and over again, and I do want to be able to produce things quickly, and so to have the solutions to the re-occurring problems at hand. So I have spent (a significant part of) the last 9 years thinking about, discussing/arguing, and experimenting with, strategies for achieving optimum code re-use in the interestingly problematic area of cross-browser scripting (and mostly in the public forum of this newsgroup). My conclusions, the position that I have settled into and now employ on a daily basis, is, in the broadest sense, a library. But its nature, its architecture, is such that it almost cannot be recognised as a 'library' by the likes of the authors of those 'popular' javascript libraries. I responded to the post I cited above by John Resig by asking him, among other things, why there were not more general-purpose javascript libraries. If you take the position that the creation of such a thing is the manifestation of the apex of browser scripting understanding then shouldn't each and every significant contributor to the field have published one at some point or another? Inevitably, no answer was forthcoming. Consider who these people are; they are, for example, the people who worked on and promoted feature detection as an alternative to browser detection (around the turn of the century), (with the possible exception of Douglas Crockford, as he has some responsibility for YUI) the people who promoted and popularised the use of closures (as the last of the pre-ES3 environments (which did not have closures) died out around 2003), and the small group who picked up on the idea of using the inline execution of a function expression to achieve some notion of 'private'/encapsulation and pushed/developed its applications (to include what was later christened "the module pattern", and pretty much every other variation on the theme)(2003-2004). So a group of individual who's influence can be seen in pretty much any modern browser script, and certainly any 'popular' javascript library. So, counting only the direct participants in the above, we should have at least a dozen published general purpose javascript libraries from those individuals. Even if significant factors were laziness, being too busy, etc. you would still expect to see 3 or 4 examples from that group. Unless something more fundamental were getting in the way of their taking that step. Such as, their conclusions about code re-use strategies being along the same lines as mine and so their rejection of the notion of a general purpose javascript library as an appropriate architecture to address code re-use. Thus they have not created these things because their experience has led them to perceive such an action as a mistake rather than as an achievement. >> (Remember that common hardware and browser performance was >> not sufficient for any sort of selector engine even to look >> like a viable idea before about the middle of 2005, but >> (even quite extreme) DOM manipulation was long established >> by that time.) > > Really? Very interesting. I didn't realize that it was a > system performance issue. I just thought it was a new way > of doing things that people started trying around then. The thing with trying something before it is viable is that when you find out that it is not viable you are not then going to waste time promoting the idea. Recall, though, that in the early days of JQuery a great deal of work went into making its selector engine faster. That would not have been necessary if it had not been on the very edge of being viable at the time. >> The 'pure DOM' tests, as a baseline for comparison, don't >> necessarily need a selector engine to perform any given >> task (beyond the fact that the tasks themselves have been >> designed around a notion of 'selectors'). So making selector >> engine testing part of the 'task' tests acts to impose >> arbitrary restrictions on the possible code used, > > Absolutely. A pure selector engine would also not be testable, Why not? Given a selector and a document all you have to do is verify that the correct number of nodes were created and that they all are the expected nodes. > nor would a drag-and-drop toolkit. Automatically testing a system that relies on human interaction is inherently problematic. > We are restricted to systems that can manipulate the DOM > and find the size of certain collections of elements. But why find the size of collections of elements? That is not a task that is common in browser scripting tasks. Even if you need to iterate over some collection of elements with something like a - for - loop you don't care how large that collection is, only that you can read whatever size it is in order to constrain the loop. >> biases the results, > >In what way? It forces the 'pure DOM' code to do things that are not necessary for real-world tasks, thus constraining the potential implementations of the tests to code that is needlessly inefficient in how it addresses the tasks. Thus the 'libraries' never get compared with against what real DOM scripting is capable of, in which case why bother with making the comparison at all? >> and ultimately negates the significance of the entire >> exercise. > > I just don't see it. There is clearly much room for > improvement, but I think the tests as they stand have > significant value. In making comparisons between the libraries, at doing selector engine based tasks (that is, forcing everyone else to play the game JQuery's way) they may have some value. But there is no real comparison against a 'baseline' unless the baseline is free to do whatever needs doing by any means available and where the tasks being preformed are realistically related to the sorts of things that actually need doing, as opposed to being tied up with arbitrary element counting. >>> Although this is not the same as the SlickSpeed >>> selectors test, >> >> Comparing the selector engines in libraries that have selector >> engines seems like a fairly reasonable thing to do. Suggesting >> that a selector engine is an inevitable prerequisite for >> carrying out DOM manipulation tasks is self evident BS. > > Note that these results don't require that the library actually > use a CSS-style selector engine, only that it can for instance > find the number of elements of a certain type, the set of which > if often most easily described via a CSS selector. So why is the element retrieval for the 'pure DOM' code done with a simplified selector engine that receives CSS selector strings are its argument? > When the "table" function is defined to return "the length of > the query 'tr td'," we can interpret that as counting the results > of running the selector "tr td" in the context of the document > if we have a selector engine, but as "the number of distinct TD > elements in the document which descend from TR > elements"if not. We can also observe that in formally valid HTML TD elements are required to descend from TR elements and so that the set we are after is actually nothing more than all the TD elements in the document, and so wonder why the code used in the 'pure DOM' is:- | tr = body.getElementsByTagName("tr"); | i = tr.length; | for(var total = 0; i;) | total += tr[--i].getElementsByTagName("td").length | ; | return total; (code that will produce a seriously faulty result if there were nested tables in the document as some TD would end up being counted twice.) - instead of just:- return body.getElementsByTagName("td").length; - or better yet, counting the number of TDs in the document before adding another 80 (so when the document is smaller and so faster to search) and then returning that number plus 80 for the number of TDs added gets the job done. I.E.:- .... var total = body.getElementsByTagName("td").length; .... //loop in which elements are added return total + 80; And then, when you start down that path, you know the document and so you know it started with 168 TDs and so adding 80 results in 248, so just return that number instead. It is completely reasonable for DOM scripts to be written for the context in which they are used, and so for them to employ information about that context which is gathered at the point of writing the code. This comes down to a question of verification; is this code here to verify the document structure after the modifications, or to announce how many TDs there are in the DOM? If it is for verification then that should not be being done inside the test function, and it should not be being done differently for each 'library', because where it is done impacts on the performance that is supposed to be the subject of the tests, and how it is done impacts on its reliability. > Being able to find such elements has been an > important part of most of the DOM manipulation > I've done. Finding elements is an important aspect of DOM scripting, but how often do you actually care about how many you have found (at least beyond the question of were any found at all)? > PureDOM does all this without any particular CSS selector > engine, so it's clear that one is not required to pass > the tests. > >>> it should subsume that one. So I don't object >>> to testing selector speed. The verification, though, >>> is a different story. It's quite easy to switch testing >>> documents, but it is presumably not so easy to verify >>> all the results of all the manipulations. >> >> Why not (at least in most cases)? code could be written >> to record the changes to a DOM that resulted from running >> a test function. You know what you expect the test function >> to do so verifying that it did do it shouldn't be too hard. > > The document to test has been fairly static, and I suppose > one could go through it, analyzing its structure, and > calculating the expected results. But the document is > included as a stand-alone file, used with this PHP: <snip> I don't see any of that as relevant. For the TD example above, all the verification code has to do is get a count of the number of TDs in the DOM before it is modified, run the test, and then count the number afterward in order to verify that 80 were added. Even that is more than the current 'verification' attempts. To mirror the current set-up all you would have to do is have some external code count some collection of elements from the document's DOM after the test has been timed. A simple selector engine (which is passed the document to search as an argument) would be fine for that, each DOM would be subject to the same test code and its performance would not matter as it would not be part of the operation being timed. > Another file could easily be substituted, and it might > well be worthwhile doing. Adding this sort of analysis > would make it much more time-consuming to test against > a different document. Why does how long the test takes to run matter? Is this a short attention span thing; worried that people will get bored waiting? That isn't a good enough reason to compromise a test system. >> Granted there are cases like the use of - addEventListener >> - where positive verification becomes a lot more difficult, >> but as it is the existing tests aren't actually verifying >> that listeners were added. > > Are there any good techniques you know of that would make it > straightforward to actually test this from within the > browser's script engine? It would be great to be able > to test this. I don't know of anything simple. I think that the designers of - addEventListener - fell down badly here. It would have been so easy for that method to return a boolean; true for success; then if you attempted to add a non-supported listener it could return false from which you would know that your listener was going to be ineffective. Still, that is what happens when the people designing the specs have negligible practical experience in the field. >>> The compromise that TaskSpeed inherits >>> from SlickSpeed is, I think, fairly reasonable. >> >> I don't. TaskSpeed's validity is compromised in the process. >> >>> Make all the libraries report their results, and note >>> if there is any disagreement. >> >> But reporting result is not part of any genuinely >> representative task, and so it should not be timed along >> with any given task. The task itself should be timed in >> isolation, and any verification employed separately. [ ... ] > > I think this critique is valid only if you assume that the > infrastructure is designed only to test DOM Manipulation. I > don't buy that assumption. The infrastructure should be designed only to test DOM manipulation. Selector engine performance can be (and is) tested separately, and while for a library that must use one for element retrieval that performance will impact on DOM manipulation, selector engine performance is not an inevitable aspect of DOM scripting. Elements may be retrieved by any means possible, and it is the DOM manipulation itself that represents the work of a browser script. >>> They could, of course, all be wrong and >>> yet all have the same values, but that seems >>> relatively unlikely. >> >> Unlikely, but not impossible, and an issue that can >> easily be entirely avoided. > > Easily for a single document, and even then only with > some real work in finding the expected results and > devising a way to test them. I don't see it as that difficult at all. >>> There is an approach that I doubt I'd bother trying, but >>> which is quite interesting: Add a url query parameter, >>> which would serve as a seed for a randomizing function. >>> If the server does not get one, it chooses a random value >>> and redirects to a page with that random seed. Then, based >>> upon random numbers derived from that seed, a document is >>> generated with some flexible structure, and a test script >>> is generated that runs a random some sequence of the >>> predefined test cases against each library. >> >> I can see how this might make sense in selector speed >> testing (though presumably you would run up against many >> cases where the reported duration of the test would be >> zero millisecond, despite our knowing that nothing happens >> in zero time) > > In another thread [1], I discuss an updated version of > slickspeed, which counts repeated tests over a 250ms span > to more accurately time the selectors. Way too short. If a browser's clock is +/-56 milliseconds that is more than 20% of your total timing. Even if it is the observably common +/-16 milliseconds then that is 5%. I would want to see this sort of testing loop pushed up to over 2 seconds. >> but for task testing randomly generating the document acted >> upon would be totally the wrong approach. If you did that you >> would bias against the baseline pure DOM tests as then they >> would have to handle issues arising from the general case, >> which are not issues inherent in DOM scripting because >> websites are not randomly generated. > > I was not expecting entirely random documents. Instead, I > would expect to generate one in which the supplied tests > generally have meaningful results. So for this test > > "attr" : function(){ > // find all ul elements in the page. > // generate an array of their id's > // return the length of that array > }, That is not a hugely realistic test in itself. What exactly would anyone do with an array of element IDs? If you were going to use them to look up the elements in the DOM why not collect the array of elements and save yourself the trouble later? > I might want to randomly determine the level of nesting at > which ULs appear, randomly determine how many are included > in the document, and perhaps randomly choose whether some > of them do not actually have ids. There would probably be > some small chance that there were no ULs at all. Judging whether that is a realistic variance to impose on the document would depend on why you needed this information in the first place. It is realistic to propose that in real-world web pages a server side script may be generating something like a tree structure made up of nested ULs and that some of its nodes would have IDs where others would not. But now, given server side scripting, we have the possibility of the server knowing the list of IDs and directly writing it into the document somewhere so that it did not need looking up from the DOM with client-side scripts, and if the reason for collecting the IDs was to send them back to the server we might also wonder whether the server could not keep the information in its session and never necessitate client-side script doing anything. Depending on what the task is actually supposed to be, for the 'pure DOM' alternative in context there may be no task here to perform at all. Giving a zero timing (for real, not just as a result of measuring inaccuracies). >> In any real web site/web application employment of scripting, >> somewhere between something and everything is known about the >> documents that are being scripted. Thus DOM scripts do not need >> to deal with general issues in browser scripting, but rather >> only need to deal with the issues that are known to exist in >> their specific context. > > Absolutely. I definitely wouldn't try to build entirely random > documents, only documents for which the results of the tests > should be meaningful. The reason I said I probably wouldn't > do this is that, while it is by no means impossible, it is also > a far from trivial exercise. There is certainly much room for improvement in these testing frameworks before moving to that point. >> In contrast, it is an inherent problem in general purpose >> library code that they must address (or attempt to address) >> all the issues that occur in a wide range of context (at >> minimum, all the common contexts). There are inevitably >> overheads in doing this, with those overheads increasing >> as the number of contexts accommodated increases. > > Yes, this is true. But it is precisely these general purpose > libraries that are under comparison in these tests. Not exclusively if you want to compare them with a 'pure DOM' baseline. > Being able to compare their performance and the code each > one uses are the only reason these tests exist. So the reason for having a 'pure DOM' baseline is to be able to compare their performance/code with what could be achieved without the overheads imposed by the need to be general. >> [ ... ] >>> Verification might be tricky, but should be doable. >>> This might make it more difficult for libraries to >>> design their tests around the particulars of the >>> document and/or the ordering of the tests. While >>> I think this would work, it sounds like more >>> effort than I'm willing to put in right now. >> >> Given that javascript source is available if anyone want >> to look at it, any library author attempting to optimise >> for a specific test (rather than, say, optimising for a >> common case) is likely to be spotted doing so, and see >> their reputation suffer as a result. > > I would hope so, but as I said in the post to which you > initially responded, I see a fair bit of what could > reasonably be considered optomising for the test, and > I only really looked at jQuery's, YUI's, and My Library's > test code. I wouldn't be surprised to find more in > the others. But that is in the implementation for the test functions, not the libraries themselves. I don't see any reason why the test functions cannot be written to exploit the strengths of the individual libraries. Of course that should be fair, so you don't have a regular JQuery optimiser write all the test and have everyone else suffer for his not knowing how best to use the other libraries. The biggest point for improvement would be in the specification for the tasks. They should be more realistic, more specific (and more accurate), and probably agreed by all interested parties (as realistic and implementable) and then given to the representatives of each library/version to implement as best they can. That way nobody can be cheated and the results would be the best representation possible of what can be achieved by each library. Granted, the re-working of the specs for the tasks would not be easy, as it would have to address questions such as whether saying an event listener should be attached to all of a particular set of elements left room for attaching to a common ancestor and delegating the event handling. This is one of the reasons that specifying realistic tasks should be the approach taken as for a realistic task the question is whether delegation is viable, and if it is, and a library can do it, there is no reason why it should not. Richard. |