Can extra processing threads help in this case?
in [MFC]
|
From: Peter Olcott on 25 Mar 2010 16:57 "Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message news:%23Qq40gFzKHA.928(a)TK2MSFTNGP05.phx.gbl... > > "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message > news:E46dnUbV0LzZJzbWnZ2dnUVZ_judnZ2d(a)giganews.com... >> >> "Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in >> message news:ORrRkLEzKHA.928(a)TK2MSFTNGP05.phx.gbl... >>> >>> >>> "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message >>> news:EqmdnZkWypVi7DbWnZ2dnUVZ_rednZ2d(a)giganews.com... >>>> (By fault tolerance I mean yank the power plug from the >>>> wall and when the machine is re-started it (as much as >>>> possible) picks up right where it left off) >>> >>> ...take a look at transactional NTFS. >>> >>> http://msdn.microsoft.com/en-us/library/aa365738(VS.85).aspx >>> >>> -Pete >>> >> >> Which I bet requires some sort of persistent storage, yup >> it does. > > Peter, > You did not, in this particular portion of a very > convoluted thread, mention that you wanted fault tolerance > without persistant storage. I apologize if I have read > your mind incorrectly, but you simply asked for fault > tolerance and I gave you a method that you could actually > use with *very little additional knowledge required*! > Additionally, I believe that your interpretation of > "fault tolerance" is that a catastrophic event could > happen to your system and you application would not lose > *any* data. Is this the definition that you are using? > > > -Pete I am looking for fault tolerance implemented entirely in software, I can not afford much hardware fault tolerance at the current time. The most hardware fault tolerance that I will have, initially is RAID 1 on a single core single server. I think that I figured out how to implement the sort of fault tolerance that I need. The essence is that a request is not acknowledged as fully accepted until after it is committed to disk in the transaction log. Every fault after that can recover using this basis. I am not sure of the underlying mechanisms, but, most every database system can handle commits and rollbacks of transactions. I am not sure exactly what happens when someone actually pulls the plug during a database update. Certainly some of the data is garbled and must be repaired. I don't really know the exact extent that this repair can be fully automated.
From: Peter Olcott on 25 Mar 2010 17:08 "Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message news:ei$3u1FzKHA.2644(a)TK2MSFTNGP04.phx.gbl... > > "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message > news:SOqdnbRxhpnk4DbWnZ2dnUVZ_radnZ2d(a)giganews.com... >> >> I was making a conservative estimate, actual measurement >> indicated zero page faults after all data was loaded, >> even after waiting 12 hours. > > Are you: > > a) Running under the debugger by any chance? > b) Allowing the system to hibernate during your 12 hour > run? > c) Doing anything special to lock the data in memory? > > I would expect that the SuperFetch service as well as the > working set manager to work against keeping your large > data file in memory if you are simply using the standard > malloc or new to reserve space for your data within your > program. In fact, I built a test system with Windows 7 x64 > and 8GB of memory and Quad processor to test your > assertion, but I noticed immedietly that even with the > amount of memory installed on my system and NO additional > programs beyond the OS, I still have a non-zero pagefault > delta when I simply switch between applications. While > that certainly is not an exhaustive test, it indicates the > type of behavior that I expected. > > If you would like to supply me with your test exectuable > and data file, I'd be happy to test on my system to see if > I get the same results as you do as I do not mind > reimaging the machine. > > -Pete > > You have to look for additional page faults after all of the data is loaded, that is why I explicitly put a little Hit any key, pause in.
From: Pete Delgado on 25 Mar 2010 17:28 "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message news:q9mdncghSqVDTTbWnZ2dnUVZ_rCdnZ2d(a)giganews.com... > You have to look for additional page faults after all of the data is > loaded, that is why I explicitly put a little Hit any key, pause in. You didn't answer any of my questions... -Pete
From: Hector Santos on 25 Mar 2010 17:30 Peter Olcott wrote: >> Peter Delgado wrote: >> >> You did not, in this particular portion of a very >> convoluted thread, mention that you wanted fault tolerance >> without persistant storage. I apologize if I have read >> your mind incorrectly, but you simply asked for fault >> tolerance and I gave you a method that you could actually >> use with *very little additional knowledge required*! >> Additionally, I believe that your interpretation of >> "fault tolerance" is that a catastrophic event could >> happen to your system and you application would not lose >> *any* data. Is this the definition that you are using? > I am looking for fault tolerance implemented entirely in > software, I can not afford much hardware fault tolerance at > the current time. The most hardware fault tolerance that I > will have, initially is RAID 1 on a single core single > server. Once again, the only way to maximize your recovery at the pure software level is to: - reducing your caching and buffer I/O AKA flushing, Committing > I think that I figured out how to implement the sort of > fault tolerance that I need. The essence is that a request > is not acknowledged as fully accepted until after it is > committed to disk in the transaction log. Every fault after > that can recover using this basis. > > I am not sure of the underlying mechanisms, Its called: - reducing your caching and buffer I/O AKA flushing, Committing > but, most every > database system can handle commits and rollbacks of > transactions. Your application doesn't call for a SQL transactional database system, and EVEN if you wanted to use SQL, for you, use something like SQLITE3. Why? Because if your FOCUS is now disaster recovery, coupled with last week's high performance and turnkey machine operations needs i.e isolated to your OCR processing, and don't wish to introduce another point of component failure with MYSQL, then you need a light weight, low footprint SQL engine that can be EMBEDDED into your package - SQLITE3 is that! So now you have: SQLITE3+MONGOOSE+OCR The only reason you will want to go with MySQL is if you want multi-thread client integrity - there we go again - MULTI-THREADS! While SQLITE3 will give work in a multi-thread environment, it fits your single ACCESS FIFO access design. When WRITING to the SQLITE3 database, it is exclusive access operation - all other access is locked. But since a WEB SERVER will do this very fast, you can use a ReaderWriter logic to handle this. The "Speed Lost" is gain by the lesser overhead you will need to have a RECORD LEVEL locking system with MYSQL. > I am not sure exactly what happens when > someone actually pulls the plug during a database update. In a system that supports crash recovery by minimizes lost, guess what it uses? MEMORY MAPPED FILES!! There we go again - Multi-Threads, Memory Maps, etc! There is a SPECIAL MEMORY MAP called the system file journal hat records ALL file I/O events, it doesn't SAVE YOUR DATA, just the EVENT! If you want to use this, you still need to use a non-buffered I/O file writer. > Certainly some of the data is garbled and must be repaired. > I don't really know the exact extent that this repair can be > fully automated. It depends on what you write to it. A typical method is to have a MAGIC HEADER in between records fixed or varied: --------------------------- Magic Header Record --------------------------- Magic Header Record --------------------------- Magic Header Record --------------------------- . . --------------------------- Magic Header Record --------------------------- So any repair process will simply recover all blocks with proper magic header blocks. The magic header includes a unique quid or number that never changes and it MAY include a cyclic redundancy check and/or hash such as MD5, SHA1 of the record so it can be compared again to check for integrity. -- HLS
From: Joseph M. Newcomer on 25 Mar 2010 18:01
See below... On Thu, 25 Mar 2010 10:09:29 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:agqmq5hvh7d7e99ekhbrjp1snta9hm630p(a)4ax.com... >> See below... >> On Thu, 25 Mar 2010 00:07:00 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >> >>> >>>"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >>>message news:sjolq5dp8kkg42mubvr9kroebcimut3blo(a)4ax.com... >>>> SEe below... >>>> On Tue, 23 Mar 2010 15:53:36 -0500, "Peter Olcott" >>>> <NoSpam(a)OCR4Screen.com> wrote: >>>> >>>> >>>>>> Run a 2nd instance and you begin to see faults. You >>>>>> saw >>>>>> that. You proved that. You told is that. It is why >>>>>> this >>>>>> thread got started. >>>>> >>>>>Four instances of 1.5 GB RAM and zero page faults after >>>>>the >>>>>data is loaded. >>>>> >>>>>You never know a man with a billion dollars in the bank >>>>>just >>>>>might panic and sell all of his furniture just in case >>>>>he >>>>>loses the billion dollars and won't be able to afford to >>>>>pay >>>>>his electric bill. >>>> **** >>>> There are people who behave this way. Custodial care >>>> and >>>> psychoactive drugs (like >>>> lithium-based drugs) usually help them. SSRIs sometimes >>>> help (selective serotonin >>>> reuptake inhibitors). I don't know what an SSRI or >>>> lithium equivalent is for an app that >>>> becomes depressed. >>> >>>Ah so then paging out a process or its data when loads of >>>RAM is still available is crazy right? >> **** >> No, lots of operating systems do it. Or did you miss that >> part of my explanation of the > >It have never occurred with my process. **** And you know this because? Oh, the "Page anticipation counter", which is a completely different concept than the page fault counter (I don't know if there is such a thing, but note that pre-pageout is NOT something that causes a page fault, but you clearly missed the distinction here!) **** > >> two-timer linux page-marking method? >> >> You still persist in believing your fantasies. >> >> Essentially, what the OS is doing is the euivalent of >> putting its money into an >> interest-bearing account! It is doing this while >> maximizing the liquidity of its assets. >> That isn't crazy. NOT doing it is crazy! But as >> operating systems programmers, we > >If the most RAM it can possibly need is 1 GB, and it has 4 >GB then it seems crazy to page anything out. How is this not >crazy? ***** Because I just cited an operating system (linux) that DOES pre-pageout pages. You may call it crazy, but some designer somewhere discovered empirically that it improves overall system performance. The difference is all you have is a p-baked opinion (for p < 0.1) of what you think an operating system should do, and the people who did this have real data that supports this as a strategy. **** > >> learning this in the 1970s. We even wrote papers about >> it. And books. I not only read >> those papers and books, I helped write some of them. You >> will find me acknowledged in >> some of them. > >Sure and back then 64K was loads of RAM. I worked on an >application that calculated the water bills for the City of >Council Bluffs IA, on a machine with 4K RAM. **** 64K was loads of RAM on small machines. But in that era, I was working on machines that had between 1 and 8 megabytes of memory (when you can afford to spend $250,000 on your memory add-on, you can put LOTS of memory on your computer). Sadly, you are confusing toy personal computers with real mainframes. Sorry, I was there. Our IBM/360/67 (1967) had 8MB of RAM, our first DECSYSTEM-10 (1969) had 1MB of RAM (well, 256K WORDS, which is how we measured it then). The first 4Meg memory system that I was responsible for budying required a forklift to remove it from the truck, and cost about $300,000. (Ampex add-on for a DECSystem-20, 1982). Actually, that was 4Meg WORDS, or about 16MB. So I really don't care about 4K systems (e.g., the IBM 1401/1440 series entry-level systems; ours had 16K) or toys that ran DR-DOS. Because we had real systems with real memory, and they were paged, we had to build real software that ran on them. And the paging disks were DEAD SLOW compared to modern hard drives (50ms seek time for 1-cylinder seek, for example); DEC RP02 drives that held 40MB or RP03s that held 80MB, not to mention the klunky 2MB drives on PDP-11s. So we HAD to squeeze every iota of performance out of these systems, and I spent a nontrivial part of my life figuring out how to do this. I even wrote a performance tool that could accurately measure page transitions on code so we could link functions so the calls happened in the same page. I rewrote it for MS-DOS with extended memory and code paging back in 1987. So don't tell me I don't understand virtual memory or how it works. I've been doing it since 1967. TSS/360, OS/360, DECSystem-10, DECSYstem-20, Mac, Apollo under the Aegis system, Vax/VMS,PDP-11, including PDP-11s under our Hydra multiprocessor operating system (which I helped write) and getting reports on the PDP-11s under STAROS (a NUMA-based system of 128 Micro-11s and its competing project whose name I forget), x86s under MS-DOS, Win16, and Win32; Unix on a variety of platforms (PDP-11, Vax, RISC/6000 AIX, I've seen a LOT of different algorithms for paging, and studied a lot more, and Peter Denning pretty much nailed it in his classic paper on paging; and Lazlo Belady's famous LRU paper. not to mention the more recent work done by IBM Research, John Ousterhout's work on file systems (John and I were students together at CMU) or the work of Mahadev Satyanarayanan at CMU on the Andrew File system and his successor work. (Satya and I were also students together at CMU, and I used AFS in 1990-1991). But never mind, you know a lot more about this because you say you THINK about ideal behavior but never, ever actually DO anything. Note that all the people I'm citing here did REAL experiments and collected REAL data, then fed this back into the operating system design process. But why should you believe them; they only are the key players in the history of the field, people who set out the design goals that all modern operating systems and file systems now use? What could they POSSIBLY have learned by collecting REAL data instead of just sitting thinking about what they might have read in some book on operating system design? joe **** > >> >> Sadly, you persist in believing what you want to believe >> instead of understanding how real >> systems work. >> joe >> >> **** >> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm |