From: Joseph M. Newcomer on
On Thu, 8 Apr 2010 21:10:25 -0500, "Peter Olcott" <NoSpam(a)> wrote:

>"Joseph M. Newcomer" <newcomer(a)> wrote in
>message news:561tr5dgf4lptedsnbavf3frg7rk20r0dj(a)
>> See below...
>> On Thu, 8 Apr 2010 18:19:31 -0500, "Peter Olcott"
>> <NoSpam(a)> wrote:
>>>Was your trouble with Windows named pipes? (I won't be
>>>What IPC did you end up choosing? (I like named pipes
>>>because their buffer can grow to any length).
>> ****
>> Actually, nowhere does it say this. In fact, the linux
>> documentation seems to suggest
>> that a fifo can be blocking on write. And if there is a
>> failure of any process connected
>> to the pipe fails, what happens to the data in the pipe?
>> (Hint: it is lost) So it is not
>> a particularly "reliable" mechanism unless there is the
>> equivalent of transactions
>> confirming receipt of a block of information.
>Yes that is it. I don't even acknowledge receipt of the
>request until it is committed to the transaction log.
>Anything at all that prevents this write also prevents the
>acknowledgement of receipt. So basically I never say "I
>heard you" until the point where nothing can prevent
>completing the transaction.
OK, this is a good specification. I'm not sure how the current proposal, which doesn't
have anything resembling a reliable log, accomplishes it.
>Then in the event that I do not receive the HTTP
>acknowledgement of final receipt of the output data, I roll
>the whole transaction back. If the reason for the failure is
>anything at all on my end I roll the charges back, but, let
>the customer keep the output data for free. If the
>connection was lost, then this data is waiting for them the
>next time they log in.
And you guarantee this exactly HOW? Oh yes, with the transacted database (I thought this
had been eliminated from the design). And you have designed the recovery code? You have
the state machine diagram of the entire workflow and know what happens at each of the
cut-points where failure can occur (which is essentially at any arc of the DFA)? I don't
recall any acknowledgement that this was part of your implmentation design.
>One of the Linux/Unix people is recommending MySQL InnoDB
>storage engine because it has very good crash recovery.
Crash recovery of the database is NOT the same as having a recovery policy for your
workflow; all it guarantees is a certain amount of trust of what is in the database, What
you DO with that information is what is crtical! When you discover the database
accurarely reflects a state of handling a request, you have to have an idea of what you
are going to do for EVERY such state that is accurately reflected in the database!
Joseph M. Newcomer [MVP]
email: newcomer(a)
MVP Tips:
From: Joseph M. Newcomer on
See beklow...
On Thu, 8 Apr 2010 20:40:34 -0500, "Peter Olcott" <NoSpam(a)> wrote:

>"Joseph M. Newcomer" <newcomer(a)> wrote in
>message news:68vsr5dlv7053hcg4es22v0obf1tilc7ee(a)
>> See below...
>> On Thu, 8 Apr 2010 14:57:19 -0500, "Peter Olcott"
>> <NoSpam(a)> wrote:
>>>If my understanding is correct fsync() is supposed to
>>>It might be the case that I must use the low level open()
>>>command so that there are no application buffers.
>> ****
>> fflush() will flush application buffers if you are using
>> stdio. and fsync(), if it is
>> implemented (did you see the section of that SQLLITE
>> discussion that says that it is not
>> always implemented correctly?)
>No, missed that.
>> You may not be able to turn off the onboard disk cache
>> buffering. That's part of the
>> problem I was referring to. (And yes, it kills hard drive
>> performance)
>> ****
>Solution is new vendor where disk caching can be turned off.
>Vendor says that disk caching can be turned off. Vendor rep
>may be guessing.
Note also that the existence of an ATAPI command to invoke an action does not guarantee
the existence of an API that will send that ATAPI command. So you need a guarantee that
the OS and/or the DBMS can actually activate this feature!

We discovered that even though our system wanted to take advantage of certain features of
a particular vendor's disk drive, we could not invoke them (for example, the SCSI
pass-through was broken in the SCSI device driver!). So EVERY component of the system,
from the application through the OS through the low-level disk drivers through the
hardware on the disk drive must support the ability to invoke some state in the hardware.

But hey, what does this matter? If the vendor says the device has the feature, that
should be good enough to predicate success on, right?
>>>Also the experts seem to be saying that the drive's own
>>>onboard cache is not much of an issue if there is UPS.
>>>There are some ways to force some drives to empty their
>>>onboard cache. The only way that is supposed to always
>>>is to turn write buffering off. This can really hurt
>>>hard-drive performance.
>> ****
>> Power failure is not much of an issue if you have a UPS,
>> so worrying about what happens
>> under power failure is not a really high priority in real
>> life.
>> ****
>But required OS reboots are, right? Still need all writes to
>go straight to the platters.
Note that when the OS reboots, the reboot procedure has, as one of its effects, the
completion of pending writes to the disk. When you say "required reboot", or course, you
are referring to the kinds of reboots that happen after updates to software, or any other
reboot invoked by some kind of "reboot-and-restart" command or API. And these actually
notify the file system (at least in Windows) that all pending I/O MUST be flushed NOW.
Plug-and-Play, a variety of IRP_MJ_PNP notifications with various minor functions like
IRP_MJ_PNP:IRP_MN_REMOVE_DEVICE guarantee that the buffers are flushed. Or did you read
the specs for building file systems? Since you clearly pretend to have expertise on the
implementation of file systems, it would be nice if you could back it up with a little
>>>>>It helps to have a single point I/O controller, but how
>>>>>are you
>>>>>planning to use this thread? How will you talk to it?
>>>>>IOW, now you
>>>>>really need to make sure you have synchronization.
>>>> ****
>>>> If one thread handles the file, then no
>>>> "synchronization"
>>>> is required because all requests
>>>> serialze through this one thread. It is an approach
>>>> called the "agent pattern".
>>>It looks like clarification from the Linux/Unix experts
>>>indicate that this would be required for my transaction
>> ****
>> Of course, you still have to flush application buffers and
>> flush kernel buffers; putting
>> it in a single thread still does not guarantee
>> transactional integrity.
>> You have to decide where your "start transaction" and "end
>> transaction" points are.
>> Oh yes, it really is hard on the disk drive; I killed on
>> disk drive by running a large
>> number of tests on a transacted database; it just stopped
>> seeking. But during the tests,
>> it was seeking ferociously as it made sure the directory
>> blocks were consistent with the
>> file contents.
>Good reason for hot swappable RAID, then.
I presume you mean RAID-5. And that you will maintain a set of spare hard drives in their
carriers for this contingency (I do)
>>>> Apparently, he thinks that a database can't a FIFO queue
>>>> because he once read that SQLLITE
>>>> doesn't have a record number, or something else silly
>>>> like
>>>> that. He missed the idea thata
>>>> a FIFO queue is a FIFO queue and ANY stream-oriented
>>>> protocol (including TCP/IP to the
>>>> local machine!) could be a valid implementation;
>>>> instead,
>>>> he fastened on one
>>>And its buffer would automatically grow to any required
>>>length and automatically shorten as items are removed?
>> ****
>> Yep. That's EXACTLY what happens. And only and undefined
>> and indeterminate points does
>> the file system manage to get these updated blocks out to
>> the hard drive (unless you have
>> a way to force synchronization of the buffers with the
>> magnetic surfaces). So imagine
>> that you have deleted records in page 1 and added records
>> to page 7. When you delete
>> records, the other records are "shuffled down" to fill the
>> space. These pages are
>> committed to disk in opportunistic order, so what is on
>> the platters represents a snapshot
>> of the in-memory buffers at random states, and the pages
>> on the disk may be inconsistent
>> with the pages in memory. So you can end up with
>> duplicate records, missing records at
>> the end, etc.
>> ****
>Make sure the flush to disk then.
The OS just crashed. Or your unlikely power-failure scenario just happened. So the files
are flushed to disk exactly HOW?
>>>I think that many of these issues may go away by using two
>>>half-duplex named pipes one in each direction. No one has
>>>yet pointed out any issues with Unix/Linux named pipes. I
>>>like named pipes because the implement the FIFO
>>>with minimal learning curve.
>> ****
>> No, in fact, NONE of them change, at all. Whether you are
>> using two half-duplex pipes
>> (which is all linux supports, even as named pipes) or a
>> full-duplex pipe (as is supported
>> in Windows).
>Unix/Linux groups say that any issues with named pipes must
>be on Windows because Windows named pipes are borked.
The typical sort of asinine response I expect from a linuxoid. They have no idea what
they are talking about, because the ONLY think Windows named pipes and linux named pipes
have in common is the spelling 'n-a-m-e-d p-i-p-e-s'. They are not even the same CONCEPT.

But since linux is perfect, and every other operating system is less than perfect, OF
COURSE linux got it right.

And do you know the joke about what happens if you try to delete all the files (not
directories) in the root directory (where there should actually be no files) and one of
them has the valid Unix filename "-r"? Long out of print, The Unix-Haters Handbook is a
hoot, mostly because those of us who used Unix for years got bit by one or more of the
failures described. And linux is bug-for-bug compatible with Unix, so the same problems
exist. Note that many of these problems existed in the open-source programs from GNU,
BSD, and others, which linux adopted.
>> If either the server app or the app it spawns fail, the
>> contents of the name pipe will be
>> lost. Just because nobody bothered to point out the
>> obvious does not mean the problem
>> does not exist. Low learning curve does not immediately
>> map to robust transacted data
>> transfer!
>I already solved this issue with my very early design. That
>is the purpose of my persistent disk file based FIFO queue.
>As soon as it gets to this file, then even a server crash
>will not prevent the job from getting completed correctly.
>We may lose the way to send it back to the user's screen (it
>will still be in their account when they log in) but we did
>not lose the actual transaction even if the server crashes.
As I recall, because SQLLITE could not have record numbers and do a seek, you abandoned
the persisten disk-based FIFO queue in favor of named pipes (which have ZERO robusteness
under most failure scenarios, but can add infinite amounts of kernel memory so they can
keep growing, so they have some advantages; run out of memory, just create a pipe and
write to it until you get more memory added by magic)

In fact, the last I saw, you had FOUR of these queues, all badly designed, all working
with processes that were using the worst possible approach to handling prioritization,
minimizing concurrency, maximizing response time. Not clear that this is forward
>Alternatively if we lose any part of the process before we
>get an HTTP acknowledgement that they received their
>results, we roll the whole transaction back.
Actually, you either lose all of a process, or none of it. What you are trying to say, I
think, is that if you suffer a failure at any point in the workflow state machine, there
is some totally magical means that gets you back to the magically constructed recovery
software that restarts the workflow at some point.

I love this approach. Sadly, it would not work for me, because my customers actually want
results. But I defrauded them by billing for many hours spent solving these problems,
when I could have just waved my magic wand and gotten a solution that worked!

>> joe
>> ****
>> Joseph M. Newcomer [MVP]
>> email: newcomer(a)
>> Web:
>> MVP Tips:
Joseph M. Newcomer [MVP]
email: newcomer(a)
MVP Tips:
From: Hector Santos on
Joseph M. Newcomer wrote:

> See below...
> On Thu, 08 Apr 2010 22:21:31 -0400, Hector Santos <sant9442(a)> wrote:
>> Joseph M. Newcomer wrote:
>>> The same way ANY CGI-based process notifies its parent that it has completed. I believe
>>> this is by closing stdout.
>> Ultimately, the process ending (which must end) is the deciding factor
>> which the parent is waiting on (with a idle timeout in redirected data).
> ****
> As I indicated, it is whatever the normal criterion is. My recollection of CGI was the
> closing of stdout meant that there could be no future output, and that was the determining
> factor, but I haven't look at this in a decade.

Most CGIs just write to the standard output device using the std
output functions, i.e. printf(), while others might open the std
output handle and use it in other output functions, in which case,
they should be closing it when they open. Since all handles will be
closed when the process ends anyway, you need to check for the child
process to end.

However, a good web server processing a script map or any spawned
child with a std I/O redirection, needs to do four things overall:

- Graceful Process Ending

- Client side socket drop or disconnect.

- Global Timeout, no CGI should be active "forever".
Our default is 5 mins.

- An output idle timeout, optional. This can be shorter
than 5 minutes if used. It should not be enforced because
you don't know if a particular CGI is just a little slow to

The child process is terminated if a timeout occurs or the client
connection drops which can easily happen when users switch to new
pages and/or clicking another link when the browser hour glass is
active waiting for a response.

> The problem with the OP is that since he
> doesn't actually look into the details, he hypothesizes how the details MUST work by
> falling back on some form of mysticism, and if he sees a blank wall, makes some guesses
> based on inadequate information and if the guesses don't work, figures there must not be
> such a mechanism. Hence the overconcern with a non-problem, notifying a Web server that
> the process has completed its action. The VERY FIRST instance of a Web server launching a
> script had such a mechanism, but since he doesn't see it, it must be an unsolved problem!

Right. The main thing I was trying to get across is that HTTP is a
client-request, server-response system. The thread that handles
processing the request is the same thread to wait for a response
generated from the processing.

Whether he realizes it or not, I lean on not, his Many Threads to 1
FIFO OCR thread design (he now says four, 1 for each type of request),
requires a thread handle to send back a result to the request
thread waiting for a response.

In short, he can just get a request, a new thread is started, queue to
1 FIFO queue and EXIT without a response. With no response, the
socket is close and the browser sets that as a 404. A response is
required on the SAME SOCKET connection.

He has not shown he understand this repeated idea to him.

> There must be a Platonic Ideal mechanism; he guesses about its existence. From flickering
> shadows on the wall. The rest of us read the documentation or the code.
> I get so tired of this...

Yeah. Me too. :)

What gets me, is that is simple - just follow standard practice.

I said this before, there is only one reason for this odd behavior,
and I honestly believe this:

He really has not control over this "ocr code" or he lacks
understanding how to change it, i.e. make thread safe.

You can imagine that whatever is done is all using global variables,
etc. So he honestly appears to be stuck on wrapping everything else
around a single process handling all the queued request as they come in.

Just consider the stated boundary conditions:

100 TPS (Transactions Per Second)
100 ms PPT (Processing time Per Transaction)

That right there dictates a minimum of 10 handlers for an ideal steady
state operation with no pressure points. It doesn't matter if its 10
threads in 1 process or 10 processes on one or multiple machines, he
needs 10 handlers.

He having realized this after it was pointed out, he seems to now say
that 100 ms is the worst case and that 10 ms is really the fastest
process time. Did he calculate 10ms to satisfy the 1 single FIFO OCR
model he seems to be stuck with?

One wonders, but there is no way on earth, even with the great Linux,
he can't handle:


- incoming request, start thread
- log the request translation
- save posted data (images, I presume) to unique folder
- check user database for authentication and ACL, credit baloney
- queue it for the OCR processor
- signal the OCR processor with thread handle (see below)
- wait for response
- send http response

all in 10 milliseconds. I'm sorry. :)

and remember the OCR processor is a single thread:


- Wait for named piped to be signaled
- read fixed length pipe data
- update database state
- process the images from unique folder
- generate response
- send response to requesting thread
- update database state
- goto top

There is NO WAY he do both parts in 10 ms! Even with the benefit of
the doubt he can do it in 11ms, the queue pressure will build.

You can actually model this with this simple formula:

TPS = N * 1000/(RPT + PPT)


TPS = Transactions per second
N = number of handlers
RPT = Request handling Time per Transaction (ms)
PPT = processing time per Transaction (ms)

So for

TPS = 100
PPT = 10 ms
N = 1

100 = 1*1000/(RPT+10)


RPT = 0.

which is not reality. The reality is to solve for N, given a TPS
goal and estimated total processing time.

N = TPS*(RPT + PPT)/1000

Assume RPT = 40 ms, we have

N = 100*(40+10)/1000

N = 5 handlers!

If your worst case is RPT+PPT = 100 ms, then you need 10 handlers.

Now, he should probably be more realistic about the 100 TPS. I doubt
he will get that, 6000 Request per minute, 360,000 per hour! Come on!
he has a niche product. Expect lower TPS to start and go from there.

Consider 3600 per hour (you should be so lucky), 60 per min, 1 per
second, with just 1 machine

1 = 1*(RPT+PPT)/1000

RPT+PPT = 1000 ms or 1 second!

Now he can jungle the performance (and methods) of all both parts.

Even if he feels with a good engineering guess that the turnaround
time is 500 ms with 1 processor:

1 = TPS*(500)/1000

Solve for TPS, and he can handle transactions at a rate of:

2 per second,
120 per min,
7200 per hour,
57,600 per day (8 hrs per day)
13,824,000 per year (20*12 workdays)

DUDE, be realistic! Don't be greedy! Keep your eye on the prize! Your
customer base will start small and that is what you MUST consider. If
you do have something, they will come and now you will be able to
get the capital and maybe VC interest who will treasure businesses
with an establish customer base and PROVE there is a market.

Stop wasting your time on issues that only made you never finish this
10 year endeavor of yours and will continue for another 10 years the
way you are approaching this.

From: Hector Santos on
Peter Olcott wrote:

> I will be looking into all these details, I bought a bunch
> of books. I want to narrow down exactly which details that I
> need to look into.

For Windows or Linux?

From: Peter Olcott on

"Hector Santos" <sant9442(a)> wrote in message
> Peter Olcott wrote:
>> I will be looking into all these details, I bought a
>> bunch of books. I want to narrow down exactly which
>> details that I need to look into.
> For Windows or Linux?
> --

Linux/Unix MySQL SQLite HTTP