Can extra processing threads help in this case?
in [MFC]
|
From: Joseph M. Newcomer on 28 Mar 2010 22:01 See below... On Thu, 25 Mar 2010 09:48:24 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:tmomq5tj59i46ufn9hvv2pf9gho5tedhoa(a)4ax.com... >> See below... >> On Wed, 24 Mar 2010 23:15:49 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>message news:snnlq5lfv1a59n003k5gkk8a6plb41ruj1(a)4ax.com... >>>> See below... >>>> >>>> On Tue, 23 Mar 2010 14:11:28 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>>>Ah so this is the code that you were suggesting? >>>>>I won't be able to implement multi-threading until >>>>>volume >>>>>grows out of what a single core processor can >>>>>accomplish. >>>>>I was simply going to use MySQL for the inter-process >>>>>communication, building and maintaining my FIFO queue. >>>> **** >>>> Well, I can think of worse ways. For example, writing >>>> the >>>> data to a floppy disk. Or >>>> punching it to paper tape and asking the user to >>>> re-insert >>>> the paper tape. MySQL for >>>> interprocess communication? Get serious! >>> >>>This was suggested on the comp.programming,threads group >>>Cross Platform FIFO Queue >> **** >> Which apparently is inhabited by equally clueless people. >> Or by people who clearly have >> no interest in performance! Or maybe they decided to give >> you a facile answer so you >> would stop bothering them....one which was just pulling >> your chain. >> >> Or they were testing the limits of your credulity. >> Reminds me of the Calvin & Hobbs >> cartoon: The family is in the card. Calvin: "Dad, how do >> they determine the weight limit >> of a bridge?" Dad: "They run bigger and bigger trucks >> over it until it collapses, then >> they rebuild it exactly and post the weight limit" >> >> Can you explain why you would accept, without question, >> such a patently absurd suggestion >> from one newsgroup while ignoring all the good advice >> you've been getting in this one? >> joe > >Can you provide any other basis for maximizing fault >tolerance that does not require some sort of persistent >storage? *** Once I know what YOU mean by "fault" and "tolerance", I will know what form of "persistent" storage is required. For example, if you let the client time out on failure, then the "persistent" storage is the active transaction on the client. If the client goes down, nothing about this response time matters. Or it could be RAM on the server; if it sends a request off to a process to handle, and the process fails, it just reinitiates the request. Of course, none of these meet the 500ms goal, but I think you have made it pretty clear this "non-negotiable" requirement was just pure noise with no content. Because everything you've added to the set of non-negotiable requirements (MySQL, PayPal) are insonsistent with that requirement. joe **** > >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 22:05 See below... On Thu, 25 Mar 2010 14:29:40 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message >news:eAZSFJEzKHA.404(a)TK2MSFTNGP02.phx.gbl... >> >> "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message >> news:aLadnYhyYOIk6jbWnZ2dnUVZ_vWdnZ2d(a)giganews.com... >>> Can you provide any other basis for maximizing fault >>> tolerance that does not require some sort of persistent >>> storage? >> >> Clustering or redundancy... >> >> -Pete >> > >When I speak of fault tolerance and I talking about yanking >the power code at any point during execution. **** This doesn't happen. Not in the real world. And power in what? And you have still failed to default "fault" and "tolerance". **** > >I don't see how Clustering or redundancy could recover from >this. There are many other cases where clustering and >redundancy would make a system more fault tolerant, but, not >on a transaction by transaction basis. **** DUH! And what, exactly, do you think the purpose of clustering actually IS? Have you read ANYTHING on how clustering works? And how places like google actually use their own homebuilt clustering systems? OMG, it's another "Let's teach Peter basic concepts, while he tells us we don't know what we're talking about" and then starts quoting from other newsgroups to prove we are all wrong. You clearly know NOTHING about what clustering is or does, and how, for example, financial institutions use it to build truly fault-tolerant systems that don't lose either your money or theirs. Oh, yes, on a transaction-by-transaction basis. Oh well, here we go again... joe **** > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 22:11 See below... On Thu, 25 Mar 2010 09:23:58 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >news:e5udsXBzKHA.2552(a)TK2MSFTNGP04.phx.gbl... >> Joseph M. Newcomer wrote: >> >>> See below... >>> >>> On Tue, 23 Mar 2010 14:11:28 -0500, "Peter Olcott" >>> <NoSpam(a)OCR4Screen.com> wrote: >>> >>>> Ah so this is the code that you were suggesting? >>>> I won't be able to implement multi-threading until >>>> volume grows out of what a single core processor can >>>> accomplish. >>>> I was simply going to use MySQL for the inter-process >>>> communication, building and maintaining my FIFO queue. >>> **** >>> Well, I can think of worse ways. For example, writing >>> the data to a floppy disk. Or >>> punching it to paper tape and asking the user to >>> re-insert the paper tape. MySQL for >>> interprocess communication? Get serious! >> >> Crazy huh? :) And all this talk about performance, >> worrying about chip caches, and throws in this elephant? >> >> Even though, given his limitations by his own design, he >> picked the wrong software - he wants SQLITE3! It fits in >> perfectly with a single ACCESSOR queuing concept. :) >> >> -- >> HLS > >You got any other idea for a 100% portable FIFO queue that >has maximum fault tolerance? *** Did you look into the Microsoft Queing system? It's a transacted persistent-memory FIFO queue. Of course, by all reports, it is flaky as all-get-out, but you never mentioned it. **** > >(By fault tolerance I mean yank the power plug from the wall >and when the machine is re-started it (as much as possible) >picks up right where it left off) **** This is a stupid idea. If a machine fails, whether from power failure, application exception, or whatever, the real goal of fault tolerance is to make sure the transaction completes, and this RARELY, IF EVER depends on the machine coming up again! The assumption is the machine failure is permanent, but the transaction STILL has to go through! I used to work in bank software, and I know something about how they had to implement fault tolerance, and NOTHING you said would have EVER made sense to those people (I did the device driver for the MICR check-reader and didn't work in the fault tolerance level; I only ate lunch with the people who did). And I've been to seminars on Microsoft clustering, and nothing you said would be recognizable to those people. Clueless, and denser than depleted uranium. joe **** > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 28 Mar 2010 22:21 Joseph M. Newcomer wrote: >> When I speak of fault tolerance and I talking about yanking >> the power code at any point during execution. > **** > This doesn't happen. Not in the real world. And power in what? And you have still > failed to default "fault" and "tolerance". > **** He really meant was not paying this electric bill and his power is cut off. :) -- HLS
From: Joseph M. Newcomer on 28 Mar 2010 22:21
See below... On Thu, 25 Mar 2010 10:35:27 -0400, Hector Santos <sant9442(a)nospam.gmail.com> wrote: >Peter Olcott wrote: > > >>> Even though, given his limitations by his own design, he >>> picked the wrong software - he wants SQLITE3! It fits in >>> perfectly with a single ACCESSOR queuing concept. :) >>> >>> -- >>> HLS >> >> You got any other idea for a 100% portable FIFO queue that >> has maximum fault tolerance? > > >Yes, the world has already put aside this Kindergarden concept! and NO >ONE IN THE WORLD uses SQL for a FIFO! If they do, they are PRIMITIVE. **** Banking software uses features in products like ORACLE to guarantee transacted access to the database of pending transactions, and the audit trail that follows these transactions. So while they aren't always using SQL, they are using transacted databases. Slow as hell, of course, but utterly robust, because the transactions either get completed or rolled back, and the multiple servers that work on the queue can handle the fact that if server 1 removes the request, and then crashes, the transaction that removed the request is rolled back and the pending transaction reappears in the database. But this requirement is incosnsitent with the requirement that there be no disk accesses. But we expect consistent requirements? SIlly us! **** > >I already provided you an example C++ class but you can also use >standard file I/O or an ISAM file or better yet a Memory Mapped FILE! > >> (By fault tolerance I mean yank the power plug from the wall >> and when the machine is re-started it (as much as possible) >> picks up right where it left off) > > >Then you have NO CHOICE but to avoid (reduce) Cache and buffer I/O, >i.e. keep all your memory and data virtualized on disk to minimize >your integrity issue and to maximize restarts with least data lost. **** The whole scenario is completley silly. Plugs don't get yanked, and the only way to handle this is with a transacted database. But the delta-T under a crash scenario moves response time from 500ms to something probably more on the order of 5 minutes (counting his 3.5 minute startup time, Windows reboot time, server startup time, etc.) But it is really weird that he is creating requirements that are completely inconsistent with each other. Or maybe not. **** > >KIDDIE STUFF! **** Fault tolerance is not kiddie stuff, it is HARD. And it COSTS! It costs in complexity, and it costs in performance. And it impacts lots of pieces of the world. But he acts as if he can build something that adds a completely inconsistent requirement and just magically make it happen, with no effort and without any performance impact. I've talked to the people who build fault tolerance, including the people who have invented transacted file systems. I've sat through their lectures. This stuff is nontrivial, but Peter thinks it is going to happen if we waves his hands in some magical way. And has said explicitly that he wants to compromise reliability for performance, and is unwilling to compromise reliability no matter what the performance cost. Say what? joe **** Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm |