2 questions about memory mapped files
in [Win32 Kernel]
|
From: Alexander Grigoriev on 22 Mar 2010 19:48 Handles closed implicitly by ExitProcess/TerminateProcess are closed for as good as with CloseFile. "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:O2xDD5dyKHA.5036(a)TK2MSFTNGP02.phx.gbl... > Hugo gleaves(a)hotmail.com> <hugh wrote: > >> Finally read this sentence from a microsoft document about MMFs: "Changes >> made to a memory-mapped file through a view of the file, other than the >> system pagefile, are automatically written to disk when the view is >> unmapped or when the file-mapping object is deleted." >> >> This is taken from: http://msdn.microsoft.com/en-us/library/ms810613.aspx >> >> I think you have to admit, that if one refers to officially published >> material, then I am correct about this whole issue. Now if I am wrong >> (and I may be) then we must accept that that official system >> documentation is wrong. > > Right, but the above is insight into giving you the engineering > requirement to prepare for it. In other words, it told programmers what > will happen. The programmer now needs to prepare to unmap or delete the > file map object to get the system to auto-flush for you. It doesn't mean > that if a program ends (gracefully) without properly unmapped and deleting > the object that the system will do this for you. > > Is it implied? I don't think so, I wouldn't trust it. You need to close > your handles properly. > > Lets say your code is running and it has an file mapping object open, and > has not be flushed yet. If your process get a WM_QUERYENDSESSION and/or > don't handle a system broadcasted exit or shutdown event and don't begin a > cleanup phase, what happens to the unflushed data? Are you sure objects > that are not programmatically closed by your process, the OS will be > quaranteed to be flushed for you? > > Can we simulate this with quick test? > > - create a small console program with a file map > - add SMALL AMOUNTS OF data to the mapping view, no flushing. > - pause by calling _getch() > > now abort the process by hitting control C. > > See if the data was FLUSHED to the disk file. > > I would also try this with a file across a network. > > > -- > HLS
From: Hector Santos on 23 Mar 2010 01:03 Can you elaborate? Alexander Grigoriev wrote: > Handles closed implicitly by ExitProcess/TerminateProcess are closed for as > good as with CloseFile. > > "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message > news:O2xDD5dyKHA.5036(a)TK2MSFTNGP02.phx.gbl... >> Hugo gleaves(a)hotmail.com> <hugh wrote: >> >>> Finally read this sentence from a microsoft document about MMFs: "Changes >>> made to a memory-mapped file through a view of the file, other than the >>> system pagefile, are automatically written to disk when the view is >>> unmapped or when the file-mapping object is deleted." >>> >>> This is taken from: http://msdn.microsoft.com/en-us/library/ms810613.aspx >>> >>> I think you have to admit, that if one refers to officially published >>> material, then I am correct about this whole issue. Now if I am wrong >>> (and I may be) then we must accept that that official system >>> documentation is wrong. >> Right, but the above is insight into giving you the engineering >> requirement to prepare for it. In other words, it told programmers what >> will happen. The programmer now needs to prepare to unmap or delete the >> file map object to get the system to auto-flush for you. It doesn't mean >> that if a program ends (gracefully) without properly unmapped and deleting >> the object that the system will do this for you. >> >> Is it implied? I don't think so, I wouldn't trust it. You need to close >> your handles properly. >> >> Lets say your code is running and it has an file mapping object open, and >> has not be flushed yet. If your process get a WM_QUERYENDSESSION and/or >> don't handle a system broadcasted exit or shutdown event and don't begin a >> cleanup phase, what happens to the unflushed data? Are you sure objects >> that are not programmatically closed by your process, the OS will be >> quaranteed to be flushed for you? >> >> Can we simulate this with quick test? >> >> - create a small console program with a file map >> - add SMALL AMOUNTS OF data to the mapping view, no flushing. >> - pause by calling _getch() >> >> now abort the process by hitting control C. >> >> See if the data was FLUSHED to the disk file. >> >> I would also try this with a file across a network. >> >> >> -- >> HLS > > -- HLS
From: Hugo gleaves on 23 Mar 2010 09:37 Well this is actually old stuff, goes back to earliest design of NT. The OS will call CloseHandle etc for any process when it exits, so it is almost a unimportant issue. There is no way any OS would last very long if it required all apps to behave strictly. Obviously one should close handles etc in non-trivial apps, else one could (in say a big multithreaded network server) gradually consume handles or something. But there is no strict requirement for any app to explicity close handles. The OS always cleans up internal resiurces used by any process, when that process exits/dies. To recap, one can only lose updates to file-backed mapped memory under these situations: 1) Updates made, power fails. 2) Updates made, OS crashes (BSD or locks up) 3) Updates made, system gets a hardware reset. 4) Updates made, disk controller/disk drive fails. 5) Updates made, other serious hardware failure. Other than that, updates cannot be "lost". If its a single process and it dies/exits then all handles/views are closed for you. The system will then write (asynchronously) all modified pages to the disk medium at some time; if the system is shutdown then the cache manager will immediately write all cached updates to disk. If its a single process and it calls FlushViewOfFile then it will flush data to the disk immediately (if file opened with WRITE_THROUGH then it will not return from Flush until data has been sent to the disk drive itself). If there are multiple sharing processes and one exits/dies then the system will not asynchronously write updates to the disk (but updated pages might get paged out by other system activity). When mapped pages are still in use by other sharers the system leaves these pages alone. Only when all sharers exit/die does the system then begin the asynchronous write. The asynchronous writing is done by the cache mamager and it will do this when there are modified pages in cache that are no longer in use (i.e. when all handles from all processes are closed). I have done tests with many gigs of mapped data, updated all that data then have app sit in a wait. If I then do a shutdown, the shutdown is slow, because it kills the app and then the cache manager is told "we're shutting down, so please write all modified cached pages to disk before I switch off box" and the cache manager does so, and it takes a while with several gig of data, but it does it and you can see it takes far longer than a normal shutdown and disk drive light is very busy. This is what goes on inside, this is how it works, this is documented on web and inside OS internals books from MS/Sysinternals people and device driver books. None of this recommends that we dont Flush, all this tells us is that updated data wont be lost ever UNLESS a fault/crash occurs. Of course killing apps/threads that use shared memory is a bad idea because one can leave data partially updated or corrupted and one might even leave locks held if on uses interlock functions for concurrency. So there are sound principles to follow with mapping/sharing but the OP questions were not about this aspect of the subject. Hugo "Hector Santos" wrote: > Can you elaborate? > > Alexander Grigoriev wrote: > > > Handles closed implicitly by ExitProcess/TerminateProcess are closed for as > > good as with CloseFile. > > > > "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message > > news:O2xDD5dyKHA.5036(a)TK2MSFTNGP02.phx.gbl... > >> Hugo gleaves(a)hotmail.com> <hugh wrote: > >> > >>> Finally read this sentence from a microsoft document about MMFs: "Changes > >>> made to a memory-mapped file through a view of the file, other than the > >>> system pagefile, are automatically written to disk when the view is > >>> unmapped or when the file-mapping object is deleted." > >>> > >>> This is taken from: http://msdn.microsoft.com/en-us/library/ms810613.aspx > >>> > >>> I think you have to admit, that if one refers to officially published > >>> material, then I am correct about this whole issue. Now if I am wrong > >>> (and I may be) then we must accept that that official system > >>> documentation is wrong. > >> Right, but the above is insight into giving you the engineering > >> requirement to prepare for it. In other words, it told programmers what > >> will happen. The programmer now needs to prepare to unmap or delete the > >> file map object to get the system to auto-flush for you. It doesn't mean > >> that if a program ends (gracefully) without properly unmapped and deleting > >> the object that the system will do this for you. > >> > >> Is it implied? I don't think so, I wouldn't trust it. You need to close > >> your handles properly. > >> > >> Lets say your code is running and it has an file mapping object open, and > >> has not be flushed yet. If your process get a WM_QUERYENDSESSION and/or > >> don't handle a system broadcasted exit or shutdown event and don't begin a > >> cleanup phase, what happens to the unflushed data? Are you sure objects > >> that are not programmatically closed by your process, the OS will be > >> quaranteed to be flushed for you? > >> > >> Can we simulate this with quick test? > >> > >> - create a small console program with a file map > >> - add SMALL AMOUNTS OF data to the mapping view, no flushing. > >> - pause by calling _getch() > >> > >> now abort the process by hitting control C. > >> > >> See if the data was FLUSHED to the disk file. > >> > >> I would also try this with a file across a network. > >> > >> > >> -- > >> HLS > > > > > > > > -- > HLS > . >
From: Alexander Grigoriev on 23 Mar 2010 11:29 You said you don't trust ExitProcess path to properly close any open handles for the mapped file to flush properly. I said ExitProcess path is no worse than your code. "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:%232vT6YkyKHA.1796(a)TK2MSFTNGP02.phx.gbl... > Can you elaborate? > > Alexander Grigoriev wrote: > >> Handles closed implicitly by ExitProcess/TerminateProcess are closed for >> as good as with CloseFile. >> >> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >> news:O2xDD5dyKHA.5036(a)TK2MSFTNGP02.phx.gbl... >>> Hugo gleaves(a)hotmail.com> <hugh wrote: >>> >>>> Finally read this sentence from a microsoft document about MMFs: >>>> "Changes made to a memory-mapped file through a view of the file, other >>>> than the system pagefile, are automatically written to disk when the >>>> view is unmapped or when the file-mapping object is deleted." >>>> >>>> This is taken from: >>>> http://msdn.microsoft.com/en-us/library/ms810613.aspx >>>> >>>> I think you have to admit, that if one refers to officially published >>>> material, then I am correct about this whole issue. Now if I am wrong >>>> (and I may be) then we must accept that that official system >>>> documentation is wrong. >>> Right, but the above is insight into giving you the engineering >>> requirement to prepare for it. In other words, it told programmers what >>> will happen. The programmer now needs to prepare to unmap or delete the >>> file map object to get the system to auto-flush for you. It doesn't mean >>> that if a program ends (gracefully) without properly unmapped and >>> deleting the object that the system will do this for you. >>> >>> Is it implied? I don't think so, I wouldn't trust it. You need to close >>> your handles properly. >>> >>> Lets say your code is running and it has an file mapping object open, >>> and has not be flushed yet. If your process get a WM_QUERYENDSESSION >>> and/or don't handle a system broadcasted exit or shutdown event and >>> don't begin a cleanup phase, what happens to the unflushed data? Are >>> you sure objects that are not programmatically closed by your process, >>> the OS will be quaranteed to be flushed for you? >>> >>> Can we simulate this with quick test? >>> >>> - create a small console program with a file map >>> - add SMALL AMOUNTS OF data to the mapping view, no flushing. >>> - pause by calling _getch() >>> >>> now abort the process by hitting control C. >>> >>> See if the data was FLUSHED to the disk file. >>> >>> I would also try this with a file across a network. >>> >>> >>> -- >>> HLS >> >> > > > > -- > HLS
From: Hector Santos on 23 Mar 2010 13:04
You mean the OS shutdown process calling ExitProcess() on an application does is not listening to WM_QUERYENDSESSION? Alexander Grigoriev wrote: > You said you don't trust ExitProcess path to properly close any open handles > for the mapped file to flush properly. I said ExitProcess path is no worse > than your code. > > "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message > news:%232vT6YkyKHA.1796(a)TK2MSFTNGP02.phx.gbl... >> Can you elaborate? >> >> Alexander Grigoriev wrote: >> >>> Handles closed implicitly by ExitProcess/TerminateProcess are closed for >>> as good as with CloseFile. >>> >>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >>> news:O2xDD5dyKHA.5036(a)TK2MSFTNGP02.phx.gbl... >>>> Hugo gleaves(a)hotmail.com> <hugh wrote: >>>> >>>>> Finally read this sentence from a microsoft document about MMFs: >>>>> "Changes made to a memory-mapped file through a view of the file, other >>>>> than the system pagefile, are automatically written to disk when the >>>>> view is unmapped or when the file-mapping object is deleted." >>>>> >>>>> This is taken from: >>>>> http://msdn.microsoft.com/en-us/library/ms810613.aspx >>>>> >>>>> I think you have to admit, that if one refers to officially published >>>>> material, then I am correct about this whole issue. Now if I am wrong >>>>> (and I may be) then we must accept that that official system >>>>> documentation is wrong. >>>> Right, but the above is insight into giving you the engineering >>>> requirement to prepare for it. In other words, it told programmers what >>>> will happen. The programmer now needs to prepare to unmap or delete the >>>> file map object to get the system to auto-flush for you. It doesn't mean >>>> that if a program ends (gracefully) without properly unmapped and >>>> deleting the object that the system will do this for you. >>>> >>>> Is it implied? I don't think so, I wouldn't trust it. You need to close >>>> your handles properly. >>>> >>>> Lets say your code is running and it has an file mapping object open, >>>> and has not be flushed yet. If your process get a WM_QUERYENDSESSION >>>> and/or don't handle a system broadcasted exit or shutdown event and >>>> don't begin a cleanup phase, what happens to the unflushed data? Are >>>> you sure objects that are not programmatically closed by your process, >>>> the OS will be quaranteed to be flushed for you? >>>> >>>> Can we simulate this with quick test? >>>> >>>> - create a small console program with a file map >>>> - add SMALL AMOUNTS OF data to the mapping view, no flushing. >>>> - pause by calling _getch() >>>> >>>> now abort the process by hitting control C. >>>> >>>> See if the data was FLUSHED to the disk file. >>>> >>>> I would also try this with a file across a network. >>>> >>>> >>>> -- >>>> HLS >>> >> >> >> -- >> HLS > > -- HLS |