Prev: Rake versus Ruby?
Next: Read text file and set the values in the Variables
From: Tony Arcieri on 19 May 2010 01:25
[Note: parts of this message were removed to make it a legal post.]

On Tue, May 18, 2010 at 10:26 PM, Bill Kelly <billk(a)cts.com> wrote:

> Sorry if I'm being redundant, but I wanted to point out that EventMachine
> *can* support non-inverted control semantics, with a Fiber-based wrapper
> layer.


And I'm pretty sure I was the first person to ever implement a Ruby
Fiber-based wrapper which provides normal flow control semantics on top of
an IoC-driven event-based framework with Revactor, for what it's worth.

Even so, there's been little success in actually applying that to an
asynchronous HTTP framework. Cramp and Rainbows are all that come to mind.
Although Revactor did support concurrent I/O alongside HTTP request
processing with Mongrel.

--
Tony Arcieri
Medioh! A Kudelski Brand

From: Daniel DeLorme on 18 May 2010 21:56
Daniel N wrote:
> On 18 May 2010 16:56, Daniel DeLorme <dan-ml(a)dan42.com> wrote:
>
>> Daniel N wrote:
>>
>>> On 13 May 2010 17:37, Daniel DeLorme <dan-ml(a)dan42.com> wrote:
>>>
>>> Does anyone know how to do the following, but without threads, purely
>>>> with
>>>> asynchronous IO?
>>>>
>>>> website = Thread.new{ Net::HTTP.get(URI.parse(url)) }
>>>> template = compute_lots_of_stuff()
>>>> puts template.sub("<content goes here>", website.value)
>>>>
>>> Event machine is perfect for this kind of stuff. Weather it fits with the
>>> rest of your web framework is more likely the thing that makes it an
>>> unlikely selection (if you're using anything rack based for example)
>>>
>> If you could show me how to use EventMachine in this case I'd be grateful.
>> I couldn't figure out how to run compute_lots_of_stuff() while the http
>> requests are executing.
>>
>
> What context are you trying to do this in? Is it inside a rack request
> (rails / merb / sinatra /pancake / other)? or is this in a stand alone
> script?
>
> Could you perhaps provide a bit of context for what you're trying to achive?

Ok, here's the context. I didn't put this in my OP because I figured
it would just bore everybody to tears.

This is inside a rack request. The idea is that I'm assembling a web
page by doing a bunch of sub-requests for the various parts of the
page. So I'll have something like:

action "index" do
@news = subreq("http://news.server")
@ad = subreq("http://ad.server")
@blog = subreq("http://blog.server")
@forum = subreq("http://forum.server")
end

All these sub-requests are launched asynchronously and, while they are
executing, the app generates the layout within which the output of the
subrequests will be embedded. So I'll have something like:

response = ['<html><body>',
'<div>',@ad,'</div>',
'<div>',@news,'</div>',
'<div>',@blog,'</div>',
'<div>',@forum,'</div>',
'</body></html>']

And when rack finally outputs the response to the client it will block
on the various subrequests unless/until they have completed.

What I can't figure out with EventMachine is how to have the "main
thread" generate the layout while the subrequests are executing.

From: Brian Candler on 19 May 2010 03:20
Daniel DeLorme wrote:
> I didn't post that code without testing it. If you look at it a bit more
> carefully maybe you'll understand how it works. The HTTP request is sent
> after the first Fiber#resume but the fiber yields before attempting to
> read the response.

Oh yes, sorry about that. I'd digested one of the method_missing
sections but not the other.

It still seems like unnecessary complexity when ruby threads are cheap,
but it achieves what you want.
--
Posted via http://www.ruby-forum.com/.

From: Daniel N on 18 May 2010 22:54
[Note: parts of this message were removed to make it a legal post.]

On 19 May 2010 11:56, Daniel DeLorme <dan-ml(a)dan42.com> wrote:

> Daniel N wrote:
>
>> On 18 May 2010 16:56, Daniel DeLorme <dan-ml(a)dan42.com> wrote:
>>
>> Daniel N wrote:
>>>
>>> On 13 May 2010 17:37, Daniel DeLorme <dan-ml(a)dan42.com> wrote:
>>>>
>>>> Does anyone know how to do the following, but without threads, purely
>>>>
>>>>> with
>>>>> asynchronous IO?
>>>>>
>>>>> website = Thread.new{ Net::HTTP.get(URI.parse(url)) }
>>>>> template = compute_lots_of_stuff()
>>>>> puts template.sub("<content goes here>", website.value)
>>>>>
>>>>> Event machine is perfect for this kind of stuff. Weather it fits with
>>>> the
>>>> rest of your web framework is more likely the thing that makes it an
>>>> unlikely selection (if you're using anything rack based for example)
>>>>
>>>> If you could show me how to use EventMachine in this case I'd be
>>> grateful.
>>> I couldn't figure out how to run compute_lots_of_stuff() while the http
>>> requests are executing.
>>>
>>>
>> What context are you trying to do this in? Is it inside a rack request
>> (rails / merb / sinatra /pancake / other)? or is this in a stand alone
>> script?
>>
>> Could you perhaps provide a bit of context for what you're trying to
>> achive?
>>
>
> Ok, here's the context. I didn't put this in my OP because I figured
> it would just bore everybody to tears.
>
> This is inside a rack request. The idea is that I'm assembling a web
> page by doing a bunch of sub-requests for the various parts of the
> page. So I'll have something like:
>
> action "index" do
> @news = subreq("http://news.server")
> @ad = subreq("http://ad.server")
> @blog = subreq("http://blog.server")
> @forum = subreq("http://forum.server")
> end
>
> All these sub-requests are launched asynchronously and, while they are
> executing, the app generates the layout within which the output of the
> subrequests will be embedded. So I'll have something like:
>
> response = ['<html><body>',
> '<div>',@ad,'</div>',
> '<div>',@news,'</div>',
> '<div>',@blog,'</div>',
> '<div>',@forum,'</div>',
> '</body></html>']
>
> And when rack finally outputs the response to the client it will block
> on the various subrequests unless/until they have completed.
>
> What I can't figure out with EventMachine is how to have the "main
> thread" generate the layout while the subrequests are executing.
>
>
Ok now we're talking. So with rack you can't do true async with a callback.
Rack is callstack based, meaning that you have to return the value as the
response the the .call method on your application. This means that any
callback based async actually needs to block in order for the rack
application you're in to return it's result. You _could_ do it by returning
a custom object in the rack response that renders as much as possible while
it waits for the response, and then renders that when it can, but that
option may not be available depending on what framework you're using.

There are a couple of other things that could help you here that immediately
come to mind.

You can use http://github.com/pauldix/typhoeus which can fetch all the
resources in parallell, and then block until all the responses come in.
This is probably going to be relatively easy to implement, and means that
the total request time for the resources is only very slightly higher than
the longest single response.

You can use ESI. There's an esi for rack project on github by Joshua Hull
which could be useful to you. http://github.com/joshbuddy/esi-for-rack You
can also use esi outside of the rack request in apache or nginx, by
responding first with a layout file containing esi tags pointing to the
content to use. Ngins, Apache or the esi rack project can then assemble the
page for you using the resources specified.

Alternatively if you're not married to hard to rack, you can take a look at
something like cramp, or node.js for a true async server environment.

HTH
Daniel

From: Bill Kelly on 19 May 2010 00:26
Tony Arcieri wrote:
> On Tue, May 18, 2010 at 7:56 PM, Daniel DeLorme <dan-ml(a)dan42.com> wrote:
>
>> What I can't figure out with EventMachine is how to have the "main
>> thread" generate the layout while the subrequests are executing.
>
> The problem here is inversion of control. EventMachine inverts control on
> you, and it sucks. You can't just do subreq(...) and expect it to return a
> value. In the best case, you have subreq call a block when it completes.
> The familiar pattern of "call function, get value" no longer applies.

Sorry if I'm being redundant, but I wanted to point out that EventMachine
*can* support non-inverted control semantics, with a Fiber-based wrapper
layer.

For example, the following is an excerpt from a test for a single-threaded
EventMachine application. Most of the methods below are being invoked on
remote server(s), but nothing ever blocks the thread. (Other Fibers
on the same EM thread will still be responding to UI events, etc.)

def test_add_documents_to_catalog
@app.reset_to_init_state

catalog1_path = @app.testsv_uri + URI.encode("/@default-catalog-path/catalog1")
catalogs = @app.catalog_manager
cat = catalogs.open_catalog(catalog1_path, :delete_existing=>true)

num_docs = cat.query_num_documents
assert_equal(0, num_docs)

catalogs.active_catalog = cat
assert_equal( cat, catalogs.active_catalog )

doc_paths = imageset_paths(1) + imageset_paths(2)
per_dir_doc_paths = @app.partition_filelist_per_directory(doc_paths)
records = @app.fetch_metadata_for_partitioned_filelist(per_dir_doc_paths)
assert_equal( doc_paths.length, records.length )

# run the 'store' test twice, to make sure the
# "INSERT OR REPLACE" is working...
2.times do
cat.store_document_metadata(records)
num_docs = cat.query_num_documents
num_docs_expected = doc_paths.length
assert_equal(num_docs_expected, num_docs)
end

# try some searches
paths = cat.search("caption" => "World Series")
assert_equal( 1, paths.length )
assert_equal( imageset_paths(2)[0], paths[0] )

# etc.
end


Anyway, dunno if this adds anything to the topic. Apologies if not.


Regards,

Bill




 |  Next  |  Last
Pages: 1 2 3
Prev: Rake versus Ruby?
Next: Read text file and set the values in the Variables