Can extra processing threads help in this case?
in [MFC]
|
From: Joseph M. Newcomer on 28 Mar 2010 15:03 See below... On Sat, 27 Mar 2010 22:42:12 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:31htq5tcpos7o7vah75d0tp02p6768q8ai(a)4ax.com... >> See below... >> On Fri, 26 Mar 2010 22:44:24 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: >>>OK so I am convinced. It is a problem that I do have to >>>solve. I shouldn't have skipped that last half of the >>>book, >>>but, I was going for maximum grades, and instead focused >>>my >>>time elsewhere. I still have the original copyright 1985 >>>book, will this be good enough or could I learn more >>>efficiently elsewhere? >>> Operating System Concepts by Peterson and Silberschatz >>> >> **** >> Almost every introduction-to-operating-systems book gives >> only general principles and >> simplified-for-students overviews. Real systems are never >> as simple as the textbooks > >But then (I guess I will ask explicitly) How much has this >stuff changed in 25 years? >My book is 25 years old (it was new when I took the course). *** As "general principles", nothing much has changed. In the infinite details of real operating systems, they have changed a LOT. For example, did you know that Vista and Win7 have new kernel storage allocators that are lock-free and multiprocessor-safe? The technique, which I read about on some Microsoft blog, is called "speculative allocation". You won't find that in your textbook. HUNDREDS of details like this are the difference between textbook descriptions and real systems. And each system evolves its own techniques. Look at the earliest file systems (NFS) vs. the later, secured file systems (AFS, done at CMU in the late 1980s). Not at all the same; yet unless you read the AFS papers, you will not learn how they did it, what problems they had to solve, etc. (for example, AFS used Kerberos security to protect the data flowing on the network, and that had a cost) joe > >The only thing that I really need to know about VM right >now, is how to minimize its impact on my performance. **** Something we have been trying to explain to you for close to two weeks. **** > >#include <sys/mman.h> >int mlock(const void *addr, size_t len); >int munlock(const void *addr, size_t len); >int mlockall(int flags); >int munlockall(void); > >Is there anything like this in Windows? **** How would I know? I have no idea what that is supposed to do. Seeing an interface spec by itself conveys no useful information. This looks suspiciously like a locked memory manager, and yes, Windows has it at application level, and I just told you the kernel has a whole new method called "speculative allocation" that eliminates the cost of multiprocessor and multithread memory locks. So why should I care about that header file? Note that header files in the sys subdirectory are usually platform-specific application-level files. And solve platform-specific issues. It doesn't mean they are for the operating system code itself. So the name of the header file conveys nothing. So unless you tell me what those functions are supposed to do, I can't tell you if there is an equivalent in Windows! Perhaps you are referring to VirtualLock, which is a pretty obvious API if you simply read the MSDN, but has its own problems in terms of usability. Read the fine print, and the fine print on SetProcessWorkingSetSize. It might improve your program's performance, at the expense of eveything else (including the Web server you are using, thus degrading your respons time). Maybe it will work, maybe not. This is why you have to run actual experiments on the configuration you are going to use, not just try to guess at what might happen! Or draw conclusions from experiments unrelated to what you might actually do (for example, coming to a conclusion about multithreading by running multiple processes, which is clearly an invalid experiment to predict multithreaded behavior, which we have been trying to tell you!) As I pointed out, in your fantasy world, the ONLY process you consider is your app running, and you ignore forms of reality such as (a) the file system is written in terms of paging (b) the executable image is handled by memory-mapped files paging in pieces of the executable image (c) there are other processes, owned by the OS, that are running (d) parts of the OS itself are paged. But hey, why should you let reality intrude into your fantasy? it is, after all, YOUR fantasy, and it can be anything you want it to be (the fact that it becomes more and more distant from reality is not a major concern here, obviuosly) joe **** > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 15:36 See below... On Sun, 28 Mar 2010 08:07:58 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Pete Delgado" <Peter.Delgado(a)NoSpam.com> wrote in message >news:eeYnLekzKHA.2644(a)TK2MSFTNGP04.phx.gbl... >> >> >> "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote in message >> news:GpGdnebGT-QCRDPWnZ2dnUVZ_h2dnZ2d(a)giganews.com... >>> Here is how you tell Windows not to ever swap this memory >>> out. >>> >>> http://msdn.microsoft.com/en-us/library/aa366895(VS.85).aspx >> >> Weren't you the one who insisted that your application >> wasn't using *virtual memory*? So why would you use this >> API for your program that doesn't use virtual memory? >> <grin> >> > >I am the one that continues to insist that my application >experiences no page faults after all of its data is loaded. >Joe continues to insist that I can not count on this >behavior. I think that Joe may be wrong, but, this is a >backup plan. *** I'm right. Run ten instances of your app. Or twenty. Essentially, you are predicating you success on a temporary piece of good luck. Not the way to build robust systems. joe **** > >> PS: I don't suppose you read the "remarks" section that >> explained about the maximum number of pages a process can >> lock? > >SetProcessWorkingSetSize *** And did you read the fine print in it? the part that says ============================ Remarks .... If the values of either dwMinimumWorkingSetSize or dwMaximumWorkingSetSize are greater than the process' current working set sizes, the specified process must have the SE_INC_WORKING_SET_NAME privilege. All users generally have this privilege. For more information about security privileges, see Privileges. Windows Server 2003 and Windows XP/2000: The specified process must have the SE_INC_BASE_PRIORITY_NAME privilege. Users in the Administrators and Power Users groups generally have this privilege. The operating system allocates working set sizes on a first-come, first-served basis. For example, if an application successfully sets 40 megabytes as its minimum working set size on a 64-megabyte system, and a second application requests a 40-megabyte working set size, the operating system denies the second application's request. ****Using the SetProcessWorkingSetSize function to set an application's minimum and maximum working set sizes does not guarantee that the requested memory will be reserved, or that it will remain resident at all times. When the application is idle, or a low-memory situation causes a demand for memory, the operating system can reduce the application's working set.***** An application can use the VirtualLock function to lock ranges of the application's virtual address space in memory; however, that can potentially degrade the performance of the system. [**** emphasis added] When you increase the working set size of an application, you are taking away physical memory from the rest of the system. This can degrade the performance of other applications and the system as a whole. It can also lead to failures of operations that require physical memory to be present (for example, creating processes, threads, and kernel pool). Thus, you must use the SetProcessWorkingSetSize function carefully. You must always consider the performance of the whole system when you are designing an application. ============================================= > >> >> >> -Pete >> > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 15:59 I had forgotten that! And I DID tell him about VirtualLock! Denser than depleted uranium. What is amazing is his persistence in using technically incorrect terminology even after it has been carefully explained to him what is going on. For example, a desire to allocate contiguous physical memory should have been the first clue that he had no idea what virtual memory was. And his persistence in equating virtual memory (which, at its essensce, is merely a mapping mechanism to physical memory) with paging activity (an add-on to virtual memory that allows oversubscription to physical memory). Because he once heard about it 25 years ago in an introductory course and misremembered what was going on! And his insistence on closed-form analytic solutions to problems for which this is impossible, and boiling things down to their essence, which he doens't do but claims to have done! It must be some form of OCD that keeps me coming back here. If I had one form, I'd keep washing my hands over and over, hoping I'll get them clean; my form is that I seem to believe that if I keep telling Peter the same thing over and over, perhaps he'll get it! I should go get treatment (all good programmers are obsessive in some way; that's how we get to be good; a friend one reported he was under treatment for his OCD, and when I told him that his obsession with detail made him one of the best programmers I'd ever known, he said "yes, but it was spilling over into my life outside programming". I should find out who his doctor is. Then maybe I could stop hanging out here anwering Peter's bizarre questions) (And what I was saying back then was RUN THE %&*ING EXPERIMENT!) joe On Sun, 28 Mar 2010 03:44:56 -0400, Hector Santos <sant9442(a)nospam.gmail.com> wrote: >Peter Olcott wrote: > > >> Here is how you tell Windows not to ever swap this memory >> out. >> http://msdn.microsoft.com/en-us/library/aa366895(VS.85).aspx >> > > >Peter, people told you this many times. I know I did and also >indicated AWE to you and guess what so did Joe. So please act like >you discovered something new, when in in fact, this whole thing seems >just another duplicate episode. > >Folks, I'm surprise Joe doesn't remembers, but check out this >incredible Peter O. vs the World thread back in December 2009! > >http://groups.google.com/group/microsoft.public.vc.mfc/browse_frm/thread/661f9ebd4aadcd03 > >and a more flatter version to read: > >http://www.eggheadcafe.com/software/aspnet/35500506/is-there-a-maximum-contig.aspx > >I see people provided test code, even JOE did testing for him on >VirtualLock! > >Same old stuff, same incompetence. Many of the same characters too! > >WOW! Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Joseph M. Newcomer on 28 Mar 2010 16:09 See below... On Sat, 27 Mar 2010 22:55:14 -0500, "Peter Olcott" <NoSpam(a)OCR4Screen.com> wrote: > >"Joseph M. Newcomer" <newcomer(a)flounder.com> wrote in >message news:e9jtq5pav683bb9qgcfrvmkg439nribmno(a)4ax.com... >> Seee below... >> >> On Thu, 25 Mar 2010 17:27:37 -0500, "Peter Olcott" >> <NoSpam(a)OCR4Screen.com> wrote: > >>> >>>So I have been convinced for several days now that >>>multi-threads could be a good idea. I still see no benefit >>>of adding memory mapped files to this mix. You have >>>explicitly failed to point out any incremental benefit >>>that >>>I get from memory mapped files over multiple threads >>>sharing >>>the same memory in the case where data remains resident in >>>memory as my testing has shown that it does. (I will also >>>test this on Linux). >> *** >> I believe I pointed out that it is likely to result in >> FEWER page faults, since parts of > >As I have said over a dozen times; GIVEN the immutable >premise that there are no page faults, then within the >explicit context of this immutable premise ONLY, how can >memory mapped files help? AND apparently the answer is they >can not help in this case. **** Why is this premise immutable? That's the difference between religion and science. Religion assumes all premises are immutable, and science assumes that every premise is merely a temporary state subject to change based on observed data. I'm saying that if you abandon your bizarre model and start doing REAL engineering, you could end up with significantly improved performance! **** > >If you don't answer this time I will give up on you as >playing head games with me. **** The only "head game" I'm trying to play with you is called "EDUCATION". I'm trying to teach you how to ENGINEER systems instead of just tossing them together, applying unreleasitic models of behavior, and getting bent out of shape when anyone questions your "immutable" premises. **** > >> the DFA you don't access don't actually have to be brought >> in. And the initialization is >> going to be faster than several minutes, because there's >> nothing to "read in" and no >> storage allocation is going to be required! >> >> Is this not screamingly obvious? Just from basic >> principles? >> **** > >It is screamingly utterly pointless when as I have said the >initialization is a three minutes every three months thing. >It is the 24/7/365 that needs to be optimized, NOT the three >minutes every three months. ***** Yet there are still unresolved issues about how the OS manages these instances, and the MMF model means that what the OS has to manage is much smaller, resulting in overall better throughput under particular scenarios (such as the multiprocess model that most Web servers assume). As such, it can simplify your engineering to have a model that allows a simpler Web service architecture which gives the performance leverage that a MMF will give. If you stopped to actually THINK about any of this instead of giving knee-jerk reactions when it violates your "immutable" premises, you'd realize what is going on here. **** > >>> >>>What you do instead is continue to insist that data does >>>not >>>remain resident in memory. I will test this in Linux very >>>soon. >> **** >> Interesting that in a later post you reveal that it >> DOESN'T remain resident in memory, > >I don't remember ever saying this. **** What was that post about linux slowing down after 500MB or so? Or did I misread that? >I am considering this now. I tested Linux again and was >somewhat surprised that my process only gets a little less >than 500MB of the 2 GB before paging kicks in. to which I replied "Working Set". Did I misread this? **** > >Substantial testing has confirmed that ALL the data from my >process does remain in RAM for both Windows and Linux. AND >if nothing else, in Linux I can issue a simple command to >forbid the OS from paging out my data. I can apply this same >command to every aspect of the process including stack >space. ***** This seems to contradict your original post. **** > >> proving that I was right; I stated that you have no >> guarantees. Or are those numbers >> about how linux slows down pure fiction? I think they >> prove what I said, and that it is >> true in linux (and it looks like a Working Set issue, >> which I pointed out somewhere in my >> first response to your post). >> joe >> **** >>> >>>>> >>>> Joseph M. Newcomer [MVP] >>>> email: newcomer(a)flounder.com >>>> Web: http://www.flounder.com >>>> MVP Tips: http://www.flounder.com/mvp_tips.htm >>> >> Joseph M. Newcomer [MVP] >> email: newcomer(a)flounder.com >> Web: http://www.flounder.com >> MVP Tips: http://www.flounder.com/mvp_tips.htm > Joseph M. Newcomer [MVP] email: newcomer(a)flounder.com Web: http://www.flounder.com MVP Tips: http://www.flounder.com/mvp_tips.htm
From: Hector Santos on 28 Mar 2010 17:36
Joe, I think it can be best summarize to the one statement he made in Dec 2009. It would be really great if this problem did not exist because I would then be able to process Chinese glyphs efficiently. The current process is estimated to require about 2.0 TB RAM. I am working on redesigning the process to eliminate this restriction. He thought that by getting a 8GB QUAD computer or bigger and if need be, more of them, the problem would no longer exist. No extra programming required beyond what he already obtained from public open source where he was lucky enough to be able to get to compile without much issue. He probably brought the computer over Christmas and found he still had a problem. So he repeated the question hoping the answer will change over the last four months - classic definition of insanity. -- HLS Joseph M. Newcomer wrote: > I had forgotten that! And I DID tell him about VirtualLock! > > Denser than depleted uranium. > > What is amazing is his persistence in using technically incorrect terminology even after > it has been carefully explained to him what is going on. For example, a desire to > allocate contiguous physical memory should have been the first clue that he had no idea > what virtual memory was. And his persistence in equating virtual memory (which, at its > essensce, is merely a mapping mechanism to physical memory) with paging activity (an > add-on to virtual memory that allows oversubscription to physical memory). Because he > once heard about it 25 years ago in an introductory course and misremembered what was > going on! And his insistence on closed-form analytic solutions to problems for which this > is impossible, and boiling things down to their essence, which he doens't do but claims to > have done! > > It must be some form of OCD that keeps me coming back here. If I had one form, I'd keep > washing my hands over and over, hoping I'll get them clean; my form is that I seem to > believe that if I keep telling Peter the same thing over and over, perhaps he'll get it! I > should go get treatment (all good programmers are obsessive in some way; that's how we get > to be good; a friend one reported he was under treatment for his OCD, and when I told him > that his obsession with detail made him one of the best programmers I'd ever known, he > said "yes, but it was spilling over into my life outside programming". I should find out > who his doctor is. Then maybe I could stop hanging out here anwering Peter's bizarre > questions) > > (And what I was saying back then was RUN THE %&*ING EXPERIMENT!) > joe > > On Sun, 28 Mar 2010 03:44:56 -0400, Hector Santos <sant9442(a)nospam.gmail.com> wrote: > >> Peter Olcott wrote: >> >> >>> Here is how you tell Windows not to ever swap this memory >>> out. >>> http://msdn.microsoft.com/en-us/library/aa366895(VS.85).aspx >>> >> >> Peter, people told you this many times. I know I did and also >> indicated AWE to you and guess what so did Joe. So please act like >> you discovered something new, when in in fact, this whole thing seems >> just another duplicate episode. >> >> Folks, I'm surprise Joe doesn't remembers, but check out this >> incredible Peter O. vs the World thread back in December 2009! >> >> http://groups.google.com/group/microsoft.public.vc.mfc/browse_frm/thread/661f9ebd4aadcd03 >> >> and a more flatter version to read: >> >> http://www.eggheadcafe.com/software/aspnet/35500506/is-there-a-maximum-contig.aspx >> >> I see people provided test code, even JOE did testing for him on >> VirtualLock! >> >> Same old stuff, same incompetence. Many of the same characters too! >> >> WOW! > Joseph M. Newcomer [MVP] > email: newcomer(a)flounder.com > Web: http://www.flounder.com > MVP Tips: http://www.flounder.com/mvp_tips.htm |