From: Scott Sauyet on
I needed a technique to aggregate the results of several asynchronous
calls. Obviously there are many ways of doing this. I wrote an
Aggregator constructor function and was wondering if the API this
presents makes sense to people here.

Here is some demo code that uses this function:

var agg = new Aggregator();

var myHandler = function(data, statuses, errors) {
// statuses = {"blue": "completed", "red": "completed",
// "yellow": "completed"}
// errors = {} //
// data = {"red": { /* ... */}, "blue": { /* ... */ },
// "yellow": { /* ... */ }}

// do something with data.blue, data.red, data.yellow
}

agg.onComplete(myHandler);

// Or var agg = new Aggregator(myHandler);

agg.start("blue");
agg.start("red");
agg.start("yellow");

getJsonAjax({
url: "blue/",
success: function(data, status) {
agg.complete("blue", data);
},
failure: function(status, err) {
agg.error("blue", status);
}
});

getJsonAjax({
url: "red/",
success: function(data, status) {
agg.complete("red", data);
},
failure: function(status, err) {
agg.error("red", status);
}
});

getJsonAjax({
url: "yellow/",
success: function(data, status) {
agg.complete("yellow", data);
},
failure: function(status, err) {
agg.error("yellow", status);
}
});

(I don't actually have a getJsonAjax() function, but it should be
obvious enough what it's meant to do.)

The user associates each asynchronous process with a String (above
"red", "blue", and "yellow"). When each process completes, the
relevant returned data is supplied with the complete() call.

The completion handlers accept three parameters. The first represents
the actual results of the various asynchronous calls, the second, the
status flags of each call ("started", "completed", "error", possibly
"aborted"), and the third, any error messages generated by these
calls. Any actual combination of the results is left to the caller,
presumably in the completion handlers. You can pass such handlers to
the onComplete() method or to the constructor.

I do have an implementation of this, but at the moment, I'm mostly
concerned with whether the API makes sense. Is there a cleaner way to
do this? Is this general enough? Are the String identifiers for the
calls robust enough? Should the constructor also accept the
identifiers? What suggestions do you have?

--
Scott
From: Asen Bozhilov on
Scott Sauyet wrote:
> I needed a technique to aggregate the results of several asynchronous
> calls.  Obviously there are many ways of doing this.  I wrote an
> Aggregator constructor function and was wondering if the API this
> presents makes sense to people here.
> Here is some demo code that uses this function:
>
>     var agg = new Aggregator();
>
>     var myHandler = function(data, statuses, errors) {
>       // statuses = {"blue": "completed", "red": "completed",
>       //             "yellow": "completed"}
>       // errors = {} //
>       // data = {"red": { /* ... */}, "blue": { /* ... */ },
>       //         "yellow": { /* ... */ }}
>
>       // do something with data.blue, data.red, data.yellow
>     }
>
>     agg.onComplete(myHandler);
>
>     // Or var agg = new Aggregator(myHandler);
>
>     agg.start("blue");
>     agg.start("red");
>     agg.start("yellow");
>     getJsonAjax({
>       url: "blue/",
>       success: function(data, status) {
>         agg.complete("blue", data);
>       },
>       failure: function(status, err) {
>         agg.error("blue", status);
>       }
>     });

Leave `Aggregator' to care about your requests. I do not see any
reasons for:

agg.start("yellow");

First I should push in aggregated request and after that I should
explicitly inform `Aggregator' for completion of that request:

agg.complete("blue", data);

That is a little bit inconsistency.

The following is my point of view of the problem.

function Aggregator() {
var that = this;
this._aggregated = [];
this._count = 0;

this._handler = function (req, data) {
that._count--;
if (that._count == 0) {
that.onComplete();
}
};
}

Aggregator.prototype = {
onComplete : function () {},

push : function (reqWrapper) {
reqWrapper.success = this._handler;
this._count++;
this._aggregated.push(reqWraper);
},

start : function () {
for (var i = 0, len = this._aggregated.length; i < len; i++) {
this._aggregated[i].send();
}
}
};

function RequestWrapper() {}
RequestWrapper.prototype.send = function () {
//...
};

var agg = new Aggregator();

agg.onComplete = function () {
//...
};

agg.push(new RequestWrapper());
agg.push(new RequestWrapper());
agg.push(new RequestWrapper());

agg.start();



From: Stefan Weiss on
On 29/06/10 22:57, Scott Sauyet wrote:
> I needed a technique to aggregate the results of several asynchronous
> calls. Obviously there are many ways of doing this. I wrote an
> Aggregator constructor function and was wondering if the API this
> presents makes sense to people here.

(snip)

> I do have an implementation of this, but at the moment, I'm mostly
> concerned with whether the API makes sense. Is there a cleaner way to
> do this? Is this general enough? Are the String identifiers for the
> calls robust enough? Should the constructor also accept the
> identifiers? What suggestions do you have?


I don't see any major problems with the API you suggested. If you ask 10
people to design an Aggregator object, you'll very likely end up with 10
different designs, and the one you come up with yourself will always
feel the most natural.
But since you asked for feedback, here are some random unsorted thoughts
that came to my mind:

-

If "agg = new Aggregator(handler)" and "agg.onComplete(handler)" are
equivalent, then the onComplete method is misnamed, because the handler
will also get called on failures (and possibly other status changes).
"setHandler" might be a better name.

-

One thing I've learned in a project where we had very complex and highly
automated forms with more than one layer of "aggregators" before the
actual Ajax transport layer, is that the handler function won't always
be interested in the complete current state of all requests, but in the
changes. From the handler's perspective, it would ask itself

- why am I being called now?
- has a job completed?
- has a job failed?
- is there a new job to consider?
etc.

To answer these questions it would have to keep an internal copy of the
previous state for comparison, but that would only duplicate what's
already available in the Aggregator object. You could find a way to pass
delta information to the handler. It could still get at the complete
state information if it has a reference to the Aggregator (in any of
several possible ways, or maybe even passed as an argument in the
callback), and the Aggregator exposes methods to query its state.

-

I don't know how general you want to keep the Aggregator. Code reuse was
never my strong suit; I tend to copy and adjust instead of writing one
perfect class/object/function and leaving it untouched afterwards. From
that standpoint, I would probably integrate the Ajax requests into the
Aggregator. In your example, they all look similar, so with an internal
getJsonAjax function in the Aggregator, the calls could be changed from

agg.start("blue");
agg.start("red");
...
getJsonAjax({
url: "blue/",
success: function(data, status) {
agg.complete("blue", data);
},
failure: function(status, err) {
agg.error("blue", status);
}
});
getJsonAjax({
url: "red/",
success: function(data, status) {
agg.complete("red", data);
},
failure: function(status, err) {
agg.error("red", status);
}
});
...

to

agg.start("blue", "blue/");
agg.start("red", "red/");

-

I guess there are parts of the API you haven't posted. From what I've
seen, the Aggregator could almost be considered as semantic sugar for a
simple hash-like registry object:

// create "Aggregator"
var agg = {};

// start "red" job
agg.red = { status: "pending" };
myHandler(agg);

getJsonAjax({
url: "red/",
success: function(data, status) {
agg.red = { status: "completed", data: data };
myHandler(agg);
},
failure: function(status, err) {
agg.red = { status: "failed", error: err };
myHandler(agg);
}
});

Okay, that's a little too simplistic, and not very pretty, either ;-)

At the other end of the spectrum, you could go in a more object oriented
direction and create Job objects which would know how they should be
transported, and which an Aggregator could then run and manage. This
could also help to eliminate the "stringiness" of your job identifiers.


PS: I just saw Asen's reply. It looks like his RequestWrapper objects
are similar to what I was trying to say in my last paragraph.


--
stefan
From: Scott Sauyet on
Asen Bozhilov wrote:
> Scott Sauyet wrote:
>> I needed a technique to aggregate the results of several asynchronous
>> calls.  Obviously there are many ways of doing this.  I wrote an
>> Aggregator constructor function and was wondering if the API this
>> presents makes sense to people here. [ ... ]

> Leave `Aggregator' to care about your requests. I do not see any
> reasons for:
>
> agg.start("yellow");
>
> First I should push in aggregated request and after that I should
> explicitly inform `Aggregator' for completion of that request:
>
> agg.complete("blue", data);
>
> That is a little bit inconsistency.

First off, thank you Asen for taking the time to reply.

I'm not sure I seen an inconsistency here, but it certainly might seem
awkward.



> The following is my point of view of the problem.
>
> function Aggregator() {
>     var that = this;
>     this._aggregated = [];
>     this._count = 0;
>
>     this._handler = function (req, data) {
>         that._count--;
>         if (that._count == 0) {
>             that.onComplete();
>         }
>     };
>
> }
>
> Aggregator.prototype = {
>     onComplete : function () {},
>
>     push : function (reqWrapper) {
>         reqWrapper.success = this._handler;
>         this._count++;
>         this._aggregated.push(reqWraper);
>     },
>
>     start : function () {
>         for (var i = 0, len = this._aggregated.length; i < len; i++) {
>             this._aggregated[i].send();
>         }
>     }
>
> };
>
> function RequestWrapper() {}
> RequestWrapper.prototype.send = function () {
>     //...
>
> };
>
> var agg = new Aggregator();
>
> agg.onComplete = function () {
>     //...
>
> };
>
> agg.push(new RequestWrapper());
> agg.push(new RequestWrapper());
> agg.push(new RequestWrapper());
>
> agg.start();

(Sorry for the long quote, can't find any way to trim it without
removing something essential.)

This would certainly be clearer and cleaner to use for the problem I
presented in my demo.

It probably is enough for my current needs, too. But there is a real
possibility that I will need some additional features I didn't mention
in my initial message, but which did prompt the API I used. First of
all, sometimes the asynchronous process I'm waiting for might be user
input rather than an AJAX call. Second, some calls might lead me to
make additional ones, and I want my wrap-up function to run only after
all the results are in. I might still be able to get away without
listing the calls, though, and only counting, but I don't think I
could use the push-push-push-start system, but rather one in which
each process is started and the aggregator checks after each one
completes whether there are still any processes still running.

I will look into whether I need the labels at all. I might be able to
avoid them.

Thanks again,

--
Scott
From: Scott Sauyet on
Stefan Weiss wrote:
> On 29/06/10 22:57, Scott Sauyet wrote:
>
>> I needed a technique to aggregate the results of several asynchronous
>> calls.  [ ... ]
>> I do have an implementation of this, but at the moment, I'm mostly
>> concerned with whether the API makes sense.  Is there a cleaner way to
>> do this?  Is this general enough?  Are the String identifiers for the
>> calls robust enough?  Should the constructor also accept the
>> identifiers?  What suggestions do you have?
>
> I don't see any major problems with the API you suggested. If you ask 10
> people to design an Aggregator object, you'll very likely end up with 10
> different designs, and the one you come up with yourself will always
> feel the most natural.

Of course. That's why when I don't have others to immediately work
with an API, I like to ask knowledgeable people if the API makes
sense.

> But since you asked for feedback, here are some random unsorted thoughts
> that came to my mind:
>
> If "agg = new Aggregator(handler)" and "agg.onComplete(handler)" are
> equivalent, then the onComplete method is misnamed, because the handler
> will also get called on failures (and possibly other status changes).
> "setHandler" might be a better name.

Absolutely right. I changed my stop(name, data) method to
complete(name, data) at the last minute because I was using the status
values of "started", "completed", or "errored" for the various tasks
and it made sense to make the verbs match these statuses. I didn't
consider that this is too close to the onComplete() method. I'm not
sure I like setHandler() because I'm allowing multiple callback
functions (quite possibly for no good reason.) I think I'll keep the
onComplete() and rename complete(name, data), but I'm not sure to
what. Perhaps I should use startTask() and stopTask(), although
"errorTask()" does not roll off the tongue properly.


> One thing I've learned in a project where we had very complex and highly
> automated forms with more than one layer of "aggregators" before the
> actual Ajax transport layer, is that the handler function won't always
> be interested in the complete current state of all requests, but in the
> changes. From the handler's perspective, it would ask itself
>
>   - why am I being called now?
>   - has a job completed?
>   - has a job failed?
>   - is there a new job to consider?
> etc.
>
> To answer these questions it would have to keep an internal copy of the
> previous state for comparison, but that would only duplicate what's
> already available in the Aggregator object. You could find a way to pass
> delta information to the handler. It could still get at the complete
> state information if it has a reference to the Aggregator (in any of
> several possible ways, or maybe even passed as an argument in the
> callback), and the Aggregator exposes methods to query its state.

That is a fascinating concept. I've had systems where writing this in
a general way would have been very useful. For my current needs and
those anticipated relatively soon, this would be overkill, but I can
almost see how it could be done elegantly with a fairly simple API. I
think if I find a little spare time, I might try that just for the fun
of it.


> I don't know how general you want to keep the Aggregator. Code reuse was
> never my strong suit; I tend to copy and adjust instead of writing one
> perfect class/object/function and leaving it untouched afterwards. From
> that standpoint, I would probably integrate the Ajax requests into the
> Aggregator.

Oh, I definitely want it more general than that. One AJAX call might
actually add a new task to the aggregator. Other asynchronous
processes for the aggregator might involve waiting for user input.
The main thing is that I want a simple way to know when all the data
needed for processing, however it's gathered from disparate sources,
is available. So it should be fairly general.


> I guess there are parts of the API you haven't posted. From what I've
> seen, the Aggregator could almost be considered as semantic sugar for a
> simple hash-like registry object:
>
>   // create "Aggregator"
>   var agg = {};
>
>   // start "red" job
>   agg.red = { status: "pending" };
>   myHandler(agg);
>
>   getJsonAjax({
>       url: "red/",
>       success: function(data, status) {
>         agg.red = { status: "completed", data: data };
>         myHandler(agg);
>       },
>       failure: function(status, err) {
>         agg.red = { status: "failed", error: err };
>         myHandler(agg);
>       }
>   });
>
> Okay, that's a little too simplistic, and not very pretty, either ;-)

To some extent that's right. But the trouble with such a hash is the
lack of knowledge of overall completion. In such a system, that
calculation would have to be pushed down into the handler function.
Avoiding that is really the motivator for this. I've done such things
fairly often, but the scenarios are getting more complicated, and if
certain expected requirements come through, they will get far more
complicated.


> At the other end of the spectrum, you could go in a more object oriented
> direction and create Job objects which would know how they should be
> transported, and which an Aggregator could then run and manage. This
> could also help to eliminate the "stringiness" of your job identifiers.
>
> PS: I just saw Asen's reply. It looks like his RequestWrapper objects
> are similar to what I was trying to say in my last paragraph.

Yes, I hadn't really considered going that far, but it might really be
useful. I'm not quite sure of the API, though. Are you thinking
something like this?:

var agg = new Aggregator(myHandler);
var job1 = new Job(function() {
getJsonAjax({
url: "red/",
success: function(data, status) {job1.stop(data);},
failure: function(status, err) {job1.error(status);}
});
});
agg.add(job1);


Thank you for your thoughtful and detailed reply,

--
Scott