Can extra processing threads help in this case?
in [MFC]
|
From: Joseph M. Newcomer on 10 Apr 2010 00:08 See below... On Fri, 9 Apr 2010 11:08:21 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:00jur51advnvpktje1f529vocjgan09u67(a)4ax.com... >> See below.... >> On Thu, 8 Apr 2010 20:24:48 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>>OK what is the simplest possible way to make sure that I >>>never ever lose the customer's ten bucks, even if the >>>server >>>crashes before my next backup and this data is lost in the >>>crash? >> *** >> I would use a transacted database system, and record >> several states of the job submitted >> "In the queue", "being processed". "processing completed", >> "results succesfully sent to >> customer", "billing completed". Perhaps fewer states are >> required. Then, upon recovery, >> I would examing these states and determine what recovery >> action was required. Note that >> what I'm talking about here is the specification document; >> the requirements document >> merelly says "Shall not bill customer for undelivered >> results" and stops right there. >> >> Then, in the specification document, the state machine for >> billing would be laid out, and >> a suggestion of a transacted databased to maintain the >> state, and so on. Only when you >> got to the implementation would you worry about a >> transacted database as the >> implementation strategy, or the details of the state >> management and error recovery. >> joe > >That is all well and good, and required, but, not exactly >the scenario that I was envisioning. What you proposed would >probably provide good recovery in the event of a power >failure. > >The scenario that I was envisioning is all that great >transaction stuff that you just said, and then the server >crashes overwriting the great transacted database data with >garbage data. It does this in-between periodic backups of >this data. How do I protect against this scenario? **** It is the nature of transacted files that the situation you describe can never happen, because the people who wrote the code anticipate the possibility, and make sure it doesn't ever occur. This requires GREAT CARE in how overwrites are done, the "transactional dance" I mentioned. One thing you can be very certain of with transacted databases: they do not fail because the database has been garbaged due to any kind of failure, including crashes in the server itself! But then, you wouldn't know that. We trust our transacted file systems; we DO NOT trust beginners who have never written one to get ANYTHING right. Do not assume your transacted database was written by a beginner. I once worked on a system with a fully-transacted file system (IBM's AIX system) and if you did the right things about indicating transaction start and commit, you never could lose ANYTHING, EVER. And otherwise, you treat backup events as cut-points in the workflow analysis, and you use simple, existing technology to handle backup and restore. And what you backup and restore has to be treated as properly transacted with respect to the file system. Fortunately, transacted databases usually come with this software as part of the product. At least INGRES did, and I think ORACLE does. I haven't looked at SQL Server, ever, or MySQL or any other database in the last 15 years. Because I stopped caring about database technology about 15 years ago. joe **** > >> **** >>> >>>> **** >>>>> >>>>> >>>> Joseph M. Newcomer [MVP] >>>> email: newcomer(a)flounder.com >>>> Web: http://www.flounder.com >>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 10 Apr 2010 00:16 See below... On Fri, 9 Apr 2010 20:49:05 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:mmnur5122gmolfpaf17h0d7088tlkuv1lg(a)4ax.com... >> See below... >> On Thu, 8 Apr 2010 21:51:37 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>>(1) One web server that inherently has by its own design >>>one >>>thread per HTTP request >>>(2) Four named pipes corresponding to four OCR processes, >>>one of these has much higher process priority than the >>>rest. >> *** >> In other words, a design which GUARATEES maximum response >> time and fails utterly to >> provide ANY form of concurrency on important requests! >> WOW! Let's see if it is possible >> to create an even WORSE design (because it is so much >> easier to create a better design >> that is no fun) >> **** >>>(3) The web server threads place items in each of the FIFO >>>queues. >> *** >> Silly. A priority-ordered queue with >> anti-priority-inversion policies makes a LOT more >> sense! >> **** >>>(4) The OCR processes work on one job at a time from each >>>of >>>the four queues. >> **** >> Let's see, can we make this worse? I don't see how, given >> how bad a design this is, but >> why take the challenge away? Or, make a better design: >> multiple servers, a single >> priority-ordered queue. No, that is too simple and too >> obvious! All it does is minimize >> response time and maximize concurrency, and what possible >> value could that have in a >> design? > >Oh right now I know why that wont work. the 3.5 minute jobs >(whenever they occur) would completely kill my 100 MS goal >for the high priority jobs. **** THis is called "priority inversion". Of course, if you used an ordered queue and multiple servers, you would have several other servers to handle the other jobs. **** > I could keep killing these jobs >whenever a high priority job arrives, but, I might not ever >have a whole 3.5 minutes completely free so this 3.5 minute >jobs may never get executed. Giving these 3.5 minutes jobs a >tiny time slice is the simplest way to solve this problem. **** No, and if you think it is so, you have missed the point entirely. Ultimately, you have to schedule things appropriately, and delegating this to the OS scheduler is NOT going to solve this problem! And handling multiple queues one at a time is not going to solve this problem. You need to look at how priority-inversion-protection systems work and go that way (for example, the way you schedule in the one-and-only queue is that you never let more than N large jobs get in the queue in sequence). This problem has been around for as long as realtime systems have been around, and the solutions form an entire body of realtime literature. RTOS systems deal with this, and they have documented the RTOS algorithms for preventing priority inversion. Spent a little time actually reading about these systems you claim expertise on, and you will find how to order your queue to prevent priority inversion situations from arising. And it is all done in a single queue. **** > >A possibly better way to handle this would be to have the >3.5 minute job completely yield to the high priority jobs, >and then pick up exactly where it left off. **** No, the better way is to use priority-inversion-prevention algorithms as documented in the realtime literature. I went to a lecture on these last year some time, and learned in 40 minutes what progress had been made in priority-inversion-prevention. Clever stuff. **** > >>>Just the pipe name itself it part of the disk, nothing >>>else >>>hits the disk. There are many messages about this on the >>>Unix/Linux groups, I stated a whole thread on this: >> **** >> And the pipe grows until what point? It runs out of >> memory? Ohh, this is a new >> interpretation of "unlimited pipe growth" of which I have >> been previously unaware! > >It does not ever simply discard input at some arbitrary >queue length such as five items. One of the FIFO models did >just that. **** Well, that is not exactly "robust" or "reliable" now, is it? So it either blocks, or discards data. If it blocks, it doesn't matter whether it grows or not. If it discards data, it is not a very good design. joe **** > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 10 Apr 2010 10:02 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:l2uvr5lb02sj377lok83q08bdj9se5cr0r(a)4ax.com... > See below... > On Fri, 9 Apr 2010 11:08:21 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >> >>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>message news:00jur51advnvpktje1f529vocjgan09u67(a)4ax.com... >>> See below.... >>> On Thu, 8 Apr 2010 20:24:48 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>>OK what is the simplest possible way to make sure that I >>>>never ever lose the customer's ten bucks, even if the >>>>server >>>>crashes before my next backup and this data is lost in >>>>the >>>>crash? >>> *** >>> I would use a transacted database system, and record >>> several states of the job submitted >>> "In the queue", "being processed". "processing >>> completed", >>> "results succesfully sent to >>> customer", "billing completed". Perhaps fewer states >>> are >>> required. Then, upon recovery, >>> I would examing these states and determine what recovery >>> action was required. Note that >>> what I'm talking about here is the specification >>> document; >>> the requirements document >>> merelly says "Shall not bill customer for undelivered >>> results" and stops right there. >>> >>> Then, in the specification document, the state machine >>> for >>> billing would be laid out, and >>> a suggestion of a transacted databased to maintain the >>> state, and so on. Only when you >>> got to the implementation would you worry about a >>> transacted database as the >>> implementation strategy, or the details of the state >>> management and error recovery. >>> joe >> >>That is all well and good, and required, but, not exactly >>the scenario that I was envisioning. What you proposed >>would >>probably provide good recovery in the event of a power >>failure. >> >>The scenario that I was envisioning is all that great >>transaction stuff that you just said, and then the server >>crashes overwriting the great transacted database data >>with >>garbage data. It does this in-between periodic backups of >>this data. How do I protect against this scenario? > **** > It is the nature of transacted files that the situation > you describe can never happen, > because the people who wrote the code anticipate the > possibility, and make sure it doesn't > ever occur. This requires GREAT CARE in how overwrites > are done, the "transactional > dance" I mentioned. One thing you can be very certain of > with transacted databases: they > do not fail because the database has been garbaged due to > any kind of failure, including > crashes in the server itself! > > But then, you wouldn't know that. OK then I will reinterpret this to mean that you don't consider this scenario likely enough to be concerned about. I know for sure that a bug in the Operating System, or corrupted Operating System executable file could cause this. I am sure that whomever my payment processor is will keep records of all customer transactions. Because of this I will already have at least one backup of the one type of most crucial transaction, (the customer adds money to their account). This plus your advice that it is nothing to worry about will be sufficient. Also I will go ahead and use a transacted database for all persistent storage. SQLite is supposed to be good for up to 500 transactions per second, and I only need 100. I just have to make sure that the reliability caveats that SQLite mentions are covered. I am guessing that SQLite might even be smart enough to cover the one efficiency aspect that I was concerned about. It may be able to directly seek using record number and record size to derive the file byte offset. In any case I won't worry about this either. > > We trust our transacted file systems; we DO NOT trust > beginners who have never written one > to get ANYTHING right. Do not assume your transacted > database was written by a beginner. > > I once worked on a system with a fully-transacted file > system (IBM's AIX system) and if > you did the right things about indicating transaction > start and commit, you never could > lose ANYTHING, EVER. > > And otherwise, you treat backup events as cut-points in > the workflow analysis, and you use > simple, existing technology to handle backup and restore. > And what you backup and restore > has to be treated as properly transacted with respect to > the file system. Fortunately, > transacted databases usually come with this software as > part of the product. > > At least INGRES did, and I think ORACLE does. I haven't > looked at SQL Server, ever, or > MySQL or any other database in the last 15 years. Because > I stopped caring about database > technology about 15 years ago. > joe > **** >> >>> **** >>>> >>>>> **** >>>>>> >>>>>> >>>>> Joseph M. Newcomer [MVP] >>>>> email: newcomer(a)flounder.com >>>>> Web: http://www.flounder.com >>>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>>> >>> Joseph M. Newcomer [MVP] >>> email: newcomer(a)flounder.com >>> Web: http://www.flounder.com >>> MVP Tips: http://www.flounder.com/mvp_tips.htm >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Peter Olcott on 10 Apr 2010 10:22 "Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in message news:peuvr59s085vrj20t8b7ddse13f71sv4qo(a)4ax.com... > See below... > On Fri, 9 Apr 2010 20:49:05 -0500, "Peter Olcott" > <NoSpam(a)OCR4Screen.com> wrote: > >>Oh right now I know why that wont work. the 3.5 minute >>jobs >>(whenever they occur) would completely kill my 100 MS goal >>for the high priority jobs. > **** > THis is called "priority inversion". Of course, if you > used an ordered queue and multiple > servers, you would have several other servers to handle > the other jobs. > **** That is not in the budget in the foreseeable future. One single core server with Hyperthreading. >> I could keep killing these jobs >>whenever a high priority job arrives, but, I might not >>ever >>have a whole 3.5 minutes completely free so this 3.5 >>minute >>jobs may never get executed. Giving these 3.5 minutes jobs >>a >>tiny time slice is the simplest way to solve this problem. > **** > No, and if you think it is so, you have missed the point > entirely. Ultimately, you have > to schedule things appropriately, and delegating this to > the OS scheduler is NOT going to > solve this problem! And handling multiple queues one at a > time is not going to solve this > problem. You need to look at how > priority-inversion-protection systems work and go that > way (for example, the way you schedule in the one-and-only > queue is that you never let > more than N large jobs get in the queue in sequence). > This problem has been around for as Even one large job take 2,100- fold too long. > long as realtime systems have been around, and the > solutions form an entire body of > realtime literature. RTOS systems deal with this, and > they have documented the RTOS > algorithms for preventing priority inversion. Spent a > little time actually reading about > these systems you claim expertise on, and you will find > how to order your queue to prevent > priority inversion situations from arising. And it is all > done in a single queue. I did and see no possible way the priority inversion could possibly occur in either of the two designs that I am considering: (1) Four queues each with their own OCR process, one of these processes has much more process priority than the rest. (2) My original design four processes that immediately yield putting themselves to sleep as soon as a higher priority process occurs. The highest one of these never yields. Also I don't see that there is much difference between four queues one for each priority level and a single priority ordered queue. The priority ordered queue would have a more complex insert. Looking for a higher priority job would be easier, only look at the head of the queue. Named pipes can not handle this. > **** >> >>A possibly better way to handle this would be to have the >>3.5 minute job completely yield to the high priority jobs, >>and then pick up exactly where it left off. > **** > No, the better way is to use priority-inversion-prevention > algorithms as documented in the > realtime literature. I went to a lecture on these last > year some time, and learned in 40 > minutes what progress had been made in > priority-inversion-prevention. Clever stuff. > **** From what I have been told, the only possible way that priority inversion can possibly occur is if there is some sort of dependency on a shared resource. I don't see any shared resource in my new design with four different independent processes. >> >>>>Just the pipe name itself it part of the disk, nothing >>>>else >>>>hits the disk. There are many messages about this on the >>>>Unix/Linux groups, I stated a whole thread on this: >>> **** >>> And the pipe grows until what point? It runs out of >>> memory? Ohh, this is a new >>> interpretation of "unlimited pipe growth" of which I >>> have >>> been previously unaware! >> >>It does not ever simply discard input at some arbitrary >>queue length such as five items. One of the FIFO models >>did >>just that. > **** > Well, that is not exactly "robust" or "reliable" now, is > it? So it either blocks, or > discards data. If it blocks, it doesn't matter whether it > grows or not. If it discards > data, it is not a very good design. > joe The fundamental OS IPC mechanisms are what is discarding the data. I think that this pertained to TCP sockets discarding data because their buffer of very few elements had been exceeded. > **** >> > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Jerry Coffin on 10 Apr 2010 10:47
In article <2IydnTUFcvDQ2CLWnZ2dnUVZ_rmdnZ2d(a)giganews.com>, NoSpam(a)OCR4Screen.com says... [ ... ] > I am going to use four processes for the four different > levels of process priority. The High priority jobs will be > assigned something like 80% of the CPU (relative to the low > priority jobs) and the low priority jobs will each get > something like 7% of the CPU. Each of these processes will > pull jobs from each of four FIFO queues. Some sort of port > access might be better than a named pipe because each port > access could be explicitly acknowledged when it occurs. > Shoving something in a named pipe does not provide this > benefit. This is a really bad idea. What you almost certainly want is a priority queue to hold the incoming tasks, with the "priority" for the queue based on a combination of the input priority and the arrival time. This will let a new input move ahead of some lower priority items as long as they arrived recently enough (for some definition of "recently enough") but also guarantees that a low priority task won't sit in the queue indefinitely -- at some point, it'll get to the front of the queue and be processed. This is simple, straightforward to implement, and fairly easy to be sure it works correctly. Despite it's *seeming* simplicity, the method you've outlined is none of the above -- quite the contrary, it's a recipe for putting another 10 years (or more) of work into getting your process synchronization to work. -- Later, Jerry. |