Do named pipes have disk I/O ?? [Unix Programming]

Prev: Learn about proxy sites and how to use them to open blocked sites unlimited downloads from RapidShare and megaupload and increase the speed of the Internet with new sites for free
Next: Why can't I catch a SIGBUS on my Mac (when strlen(NULL) is used)?

From: Rainer Weikusat on 6 Apr 2010 08:05

"Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
> I have to have the 3.5 minute thread run. I have to have the
> 50 ms thread have absolute priority over the 3.5 minute
> thread. Ideally I would like the 50 ms thread to preempt the
> 3.5 minute thread and have the 3.5 minute thread pick up
> exactly where it left off the next time it gets scheduled.
>
> I want the design to be as simple as possible.

Provided that you can ensure that your high-priority thread does not
use any ressources also used by the low-priority thread (which
includes the usual, useless C library facilities such as stdio or
malloc), the easy way to do this is to create the high-priority thread
with a suitable scheduling policy (SCHED_FIFO). Anything else is
automatic. If your threads do use shared ressources, the worst thing
which is going to happen is that your high-priority thread will block
for a short amount of time until the low-priority thread has
released a particular lock. It is entirely conceivable that this
approach will work for you even if you insist on using stdio and/or
malloc for some weird 'religious' reason. It certainly will if you
don't.

From: Peter Olcott on 6 Apr 2010 08:36

"David Schwartz" <davids(a)webmaster.com> wrote in message
news:ab8e4158-566d-4a4d-b7ad-6814122a0ac1(a)r18g2000yqd.googlegroups.com...
On Apr 5, 9:07 pm, "Peter Olcott" <NoS...(a)OCR4Screen.com>
wrote:

> The only lock that I will need is when the customer's
> account balance is updated, this could be an exception to
> preempting.

--So your threads will never allocate memory? Never access
files?
They will only allocate private memory for their own use,
likewise for files. They will open files for their own use.

> I must do the 3.5 minute job, but every 50 ms job can
> preempt it. It is more important that the 50 ms job gets
> done (in as best as can be accomplished) within 500 ms
> response time, than it is that the 3.5 minute job achieve
> less than 12 hour response time.

--So code that. Don't try to use some clever trick. If you
want the 50ms
--job to run and the 3.5 minute job to yield, then write the
code that
--runs the 3.5 minute job such that it waits if there's a
higher-
--priority task.

I don't want it to just wait, I want it to stop whatever its
doing and wait, and then resume exactly where it left off.

> I always want the 50 ms jobs to have all of the CPU
> resources, one whole CPU core to itself.

--Right, but that's the wrong way to think about it. Don't
think "I have
--to make the 50 ms job get a CPU by itself" think "I have
to write my
--threads so that they do the work I want done, and the
scheduler will
--find the most efficient way to do that (which I suspect is
giving it
--its own CPU)".

--You're trying to design across levels of abstraction. That
makes
--things much harder and is much less likely to work.

--For example, suppose you have four cores and five
high-priority tasks.

Then because of strict FIFO order the fifth job must wait. I
could have for cores and 185 jobs, 181 of these jobs will
wait their turn.

--Suppose the memory allocator lock is held by a
lower-priority task. If

There is no need for a memory allocator lock because memory
will not be shared across threads.

--you have locked all four cores exclusively to the
high-priority tasks,
--and all high-priority tasks are blocked on memory
allocation, further
--forward progress will be *impossible*.

--So you typically want the higher-priority tasks to get
dedicated
--cores, but you definitely don't want to actually enforce
that. Because
--that's a *means* to the end you want and not the end you
want. Code
--the end, not the means.
--
--DS

Strict FIFO order will enforce this, and this strict FIFO
order must be enforced.

From: Peter Olcott on 6 Apr 2010 08:39

The random number generator was not as random as I thought.
When this fact was taken into account, then the pattern of
ten-fold faster performance when memory would fit into cache
emerged. You know about temporal and spatial locality of
reference don't you?

"David Schwartz" <davids(a)webmaster.com> wrote in message
news:b769f013-88d1-4d9a-bf22-496a8e0809e2(a)z6g2000yqz.googlegroups.com...
On Apr 5, 9:09 pm, "Peter Olcott" <NoS...(a)OCR4Screen.com>
wrote:

> If I have 8MB of cache then random access to a 2 GB memory
> space will have little benefit from the 8MB cache. I know
> this from empirical testing and analysis. Do you disagree?

I totally disagree. In fact, you presented your own counter
example.
You yourself had a case where what the programmer thought
was random
access actually degenerated into a tight loop that
completely fit in
cache. Your access was random. Your access was to a 2 GB
memory space.
Yet it benefited massively from cache.

DS

From: Peter Olcott on 6 Apr 2010 08:52

"Ersek, Laszlo" <lacos(a)caesar.elte.hu> wrote in message
news:Pine.LNX.4.64.1004061137040.32334(a)login01.caesar.elte.hu...
> On Mon, 5 Apr 2010, David Schwartz wrote:
>
>> If you don't want a thread to be doing the 3.5 minute
>> job, because there's something more important to do, for
>> the love of god DON'T CODE IT TO DO THAT JOB.
>>
>> What you're trying to do is code the thread to do one
>> thing and then use some clever manipulation to get it to
>> do something else. Just code the thread to do the work
>> you want done and you won't have to find some way to
>> pre-empt it or otherwise "trick" it.
>
> One way to do this, I guess, is to enable all threads to
> handle all kinds of tasks, both low priority and high
> priority. A thread executing a low priority job should
> quickly notice the availability of a high priority thread,
> put back the half-baked low prio job to the low prio work
> queue, and switch to the high prio job.
>
> This would mean the following:
>
> - The state of the low prio job cannot be kept on the
> stack (or more precisely, in auto variables) for the
> entire duration between the start of that job and the
> completion of the job. When the switch(es) to the high
> prio job(s) occur(s), the low prio job must be
> "serialized" (at least conceptually) and put back to the
> low prio work queue.

I was envisioning an array of structs (or a std::vector of
objects) such that all of the data for the thread is
contained in this struct (or object). The context switch
would involve changing the subscript into this array. If the
thread is in a tight loop some mechanism would be in place
to store this state, and later restore this state.

>
> - A way must be devised to notify, more or less
> asynchronously, the thread(s) working on low prio jobs:
>
> (a) Asynchronous thread cancellation would kind of defeat
> the whole idea.
>
> (b) Checking periodically for a locked resource (eg. the
> head of the high prio work queue) could work on Linux
> without incurring a huge performance penalty, because I'm
> aware (on a gossip level) of mutexes being implemented
> there with futexes, ie. in user space. I guess this would
> be a good use case for the pthread_mutex_trylock()
> function.
>
> (c) Atomic (lockless) access to a boolean variable could
> be specified in a non-portable way. (Eg. in assembly or
> maybe with gcc-specific types or attributes.)
>
> (d) A signal could be generated for the worker thread
> specifically. The handler would set a volatile
> sig_atomic_t object with boolean meaning. The handler
> setting the flag asynchronously and the code checking the
> flag would have to be executed by the same thread, see
> <http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2006/n2016.html>,
> "Existing portable uses" / 3.
>
> (e) Perhaps thread-specific data might be summoned in (d).
> I'm led to the idea by a recent reddit comment
> <http://www.reddit.com/r/programming/comments/bmnkl/pthreads_as_a_case_study_of_good_api_design/c0nj62j>
> stating how fast accesses to thread-specific data on
> common platforms are. Perhaps they exhibit some atomic
> properties too.
>
> lacos

I haven't decided upon the means of notification yet. I was
leaning towards (d).

From: Peter Olcott on 6 Apr 2010 08:57

"Rainer Weikusat" <rweikusat(a)mssgmbh.com> wrote in message
news:874ojokcr3.fsf(a)fever.mssgmbh.com...
> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>> I have to have the 3.5 minute thread run. I have to have
>> the
>> 50 ms thread have absolute priority over the 3.5 minute
>> thread. Ideally I would like the 50 ms thread to preempt
>> the
>> 3.5 minute thread and have the 3.5 minute thread pick up
>> exactly where it left off the next time it gets
>> scheduled.
>>
>> I want the design to be as simple as possible.
>
> Provided that you can ensure that your high-priority
> thread does not
> use any ressources also used by the low-priority thread
> (which
> includes the usual, useless C library facilities such as
> stdio or
> malloc), the easy way to do this is to create the
> high-priority thread
> with a suitable scheduling policy (SCHED_FIFO). Anything
> else is

I don't yet know the details of this, but a FIFO is what I
have in mind for the order of the jobs within a priority
level.

> automatic. If your threads do use shared ressources, the
> worst thing
> which is going to happen is that your high-priority thread
> will block
> for a short amount of time until the low-priority thread
> has
> released a particular lock. It is entirely conceivable
> that this
> approach will work for you even if you insist on using
> stdio and/or
> malloc for some weird 'religious' reason. It certainly
> will if you
> don't.
>
>

The idea that I have in mind is that the high priority
thread preempt the lower priority thread.
What about std::vector?