Do named pipes have disk I/O ?? [Unix Programming]

Prev: Learn about proxy sites and how to use them to open blocked sites unlimited downloads from RapidShare and megaupload and increase the speed of the Internet with new sites for free
Next: Why can't I catch a SIGBUS on my Mac (when strlen(NULL) is used)?

From: Chris Friesen on 7 Apr 2010 16:33

On 04/07/2010 12:32 PM, Peter Olcott wrote:

> Likewise with my scheduling of my
> processes as compared to the OS scheduling them for me.

The OS has knowledge of what else the system is doing that you don't have.

> Is there any way to tell the hardware cache to load specific
> data?

Generally yes, but it requires assembly code.

Chris

From: Scott Lurndal on 7 Apr 2010 16:51

"Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>If I am wrong (and I really don't think that I am, I have
>benchmarking to support my hypotheses) I can always use the

Peter,

Many of the folks you are arguing with create operating systems
or highly sophisticated application stacks for a living. You
should listen a bit more carefully.

Your benchmarks, such as you've described to this point,
tell you basically _nothing_; Trust me, the current system
that I've architected and built (192 cores, 1TB memory) requires
very substantial and extremely sophisticated modelling and
run-time performance analysis to determine the memory access
characterstics. Your simple test, which several have shown
to be defective, tells you exactly nothing about the memory
or cache subsystems performance.

>simple mechanism that you proposed. I would benchmark these
>against each other. Since I will make sure that there are no
>locks between processes that makes all of these

You don't get this choice. For example, the OS will take locks
on your behalf. If you use threads, so will the standard C library.

I think you need to do the following:

1) re-evaluate your requirements.

- Do you really need real-time response for a batch OCR service?
- Do you really need to pre-empt a longer running job? Why?
- Do you seriously think you could run a pay-for-service server
with a one single-core box (hint: your SLA will suck).

2) Start listening to the professionals who are attempting to help
you, instead of blindly disregarding, ignoring or misrepresenting
their input.

scott

From: Scott Lurndal on 7 Apr 2010 16:52

Chris Friesen <cbf123(a)mail.usask.ca> writes:
>On 04/07/2010 12:32 PM, Peter Olcott wrote:
>
>> Likewise with my scheduling of my
>> processes as compared to the OS scheduling them for me.
>
>The OS has knowledge of what else the system is doing that you don't have.
>
>> Is there any way to tell the hardware cache to load specific
>> data?
>
>Generally yes, but it requires assembly code.

And generally it is not a good idea. The processor has a better
idea of what to prefetch than most application programmers.

scott

From: Scott Lurndal on 7 Apr 2010 16:53

"Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>
>"Phil Carmody" <thefatphil_demunged(a)yahoo.co.uk> wrote in
>message news:87k4sjhke4.fsf(a)kilospaz.fatphil.org...
>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>>> "Rainer Weikusat" <rweikusat(a)mssgmbh.com> wrote in
>>> message
>>> news:87d3ycnwcl.fsf(a)fever.mssgmbh.com...
>>>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>>>>> "Scott Lurndal" <scott(a)slp53.sl.home>: wrote in message
>>>>>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>>>>
>>>> [...]
>>>>
>>>>>>>I am only focusing on the design of the preemptive
>>>>>>>scheduling now.
>>>>>>>Also the initial hardware will only have a single core
>>>>>>>with
>>>>>>>hyperthreading.
>>>>>>>
>>>>>>
>>>>>> You are making this far more complicated than
>>>>>> necessary.
>>>>>> Have fun.
>>>>>>
>>>>>> scott
>>>>>
>>>>> So then do you have a simple way to implement
>>>>> preemptive
>>>>> scheduling?
>>>>
>>>> If you are convinced that this is the solution to your
>>>> problem, the
>>>> kernel already provides exactly that. You just have to
>>>> use
>>>> the
>>>> existing facilities.
>>>
>>> So the OS will put a thread to sleep, saving its
>>> multi-megabyte state upon a process triggered event?
>>
>> What, *precisely*, do you mean by "saving" the state,
>> and why are you so worried by that?
>>
>>> I would save the state with a simple change of the
>>> integer
>>> subscript into an array of each threads data. Could the
>>> OS
>>> be told that this is what is needed?
>>
>> In which case you have to simultaniously maintain the
>> multi-megabyte
>> state of _all_ of the "thread data".
>
>David has mostly convinced me that my 3.5 minute job is best
>off as its own process. The only issue is that the minimum
>sleep period seems to be one second, I could really use it
>to be 100 ms. I might have to build my own sleep system for
>this process.

man nanosleep.

In addition, learn how to use man -k (e.g. man -k sleep).

scott

From: Scott Lurndal on 7 Apr 2010 16:54

"Peter Olcott" <NoSpam(a)OCR4Screen.com> writes:
>
>"Jens Thoms Toerring" <jt(a)toerring.de> wrote in message
>news:823jm7F41dU1(a)mid.uni-berlin.de...
>> In comp.unix.programmer Peter Olcott
>> <NoSpam(a)ocr4screen.com> wrote:
>>> David has mostly convinced me that my 3.5 minute job is
>>> best
>>> off as its own process. The only issue is that the
>>> minimum
>>> sleep period seems to be one second, I could really use
>>> it
>>> to be 100 ms. I might have to build my own sleep system
>>> for
>>> this process.
>>
>> For the sleep() function that's correct. For shorter sleep
>> periods use nanosleep(), that should give you a resolution
>> of at least 10 ms.
>> Regards, Jens
>
>Yes I forgot about that.
>
>>
>> PS: Please stop crossposting this to
>> linux.development.system,
>> that group is about "Linux kernels, device drivers,
>> modules"
>> and thus the whole stuff discussed here is off-topic
>> over
>> there.
>
>People define systems programming in different ways. I need
>to speak to people that do the kind of programming that you
>have referred to, to make sure that my designs work the way
>that I expect them to under the covers, as implemented in
>the kernel.
>
>Here is a specific concrete example:
>I am assuming that all pwrite(), pread() and append()
>operations are atomic specifically because the kernel forces
>these to be executed sequentially. Is this the means that
>the kernel uses to make these operations atomic?

The kernel does not "force them to be executed sequentially".

The kernel does use mutual exclusion techniques to ensure that
the current file position pointer is only updated atomically.

scott