2 questions about memory mapped files
in [Win32 Kernel]
|
From: Jonathan de Boyne Pollard on 19 Mar 2010 13:06 >>>>> >>>>>> even if an app dies and the OS begins to automatically flush, the >>>>>> logoff will be forced to hang until its done. >>>>> >>>>> M. Lebedinsky has explained that that is a falsehood. WINLOGON >>>>> doesn't do any magic at logout to flush dirty pages. >>>>> >>> He didn't say that exactly. >>> >> Yes, xe did. Go and read all of what xe wrote. >> >>> [...] I simply logged out and logged back in, the file was there [...] >>> >> Now try attempting to demonstrate the forcible hang and flush of >> dirty pages to disc that you claimed would occur, instead of >> something quite different. >> > Jonathan the only circumstances I know of, that can lead to missing > data [...] > For goodness' sake read! Read what you yourself claimed would happen, that M. Lebedinsky had earlier explained to be a falsehood. And stop repeatedly trying to prove something quite different. > I'm not looking for a bickering contest Jonathan, the OP asked an > important question and it is important that we all try to stick with > facts and not let our ego intrude. > Then stick with them, and stop using ad hominems like this when it's pointed out that some things are not facts but falsehoods.
From: Hugo gleaves on 22 Mar 2010 09:51 "Jonathan de Boyne Pollard" wrote: > > >>>>> > >>>>>> even if an app dies and the OS begins to automatically flush, the > >>>>>> logoff will be forced to hang until its done. > >>>>> > >>>>> M. Lebedinsky has explained that that is a falsehood. WINLOGON > >>>>> doesn't do any magic at logout to flush dirty pages. > >>>>> > >>> He didn't say that exactly. > >>> > >> Yes, xe did. Go and read all of what xe wrote. > >> > >>> [...] I simply logged out and logged back in, the file was there [...] > >>> > >> Now try attempting to demonstrate the forcible hang and flush of > >> dirty pages to disc that you claimed would occur, instead of > >> something quite different. > >> > > Jonathan the only circumstances I know of, that can lead to missing > > data [...] > > > For goodness' sake read! Read what you yourself claimed would happen, > that M. Lebedinsky had earlier explained to be a falsehood. And stop > repeatedly trying to prove something quite different. > > > I'm not looking for a bickering contest Jonathan, the OP asked an > > important question and it is important that we all try to stick with > > facts and not let our ego intrude. > > > Then stick with them, and stop using ad hominems like this when it's > pointed out that some things are not facts but falsehoods. > > . > OK I will correct one error I made earlier, and apologize if this caused a misunderstanding. When I referred to "logoff" I should have written "shutdown". So to recap, if you have a single process, it creates a 1GB (for example) mapping along with a new file for it. Then the process populates that data block with some data, a Gigabyte - so that every mapped page has been modified. Then if you shutdown the machine, and this cause the app to be killed, ALL THE DATA WILL BE AUTOMATICALLY WRITTEN TO DISK before the machines shuts off. In the case of logging off, I was incorrect because in that scenario the modified pages are asynchronously written to the file (eventually) by the OS in its own time, shutdown of course forces the system to flush cached pages. Now I am not and have not advocated "not flushing" I am aware and understand the risks of relying on the machinery and OS in this way, and one should of course always take steps to minimize the risk of data loss. But that does not alter the fact that unless there is a powerfail, OS crash, reset or hardware failure - memory mapped pages are always written to the backing file at some point in time, usually failry quickly if apps die and the file is no longer in use. Now is this something you take issue with or can we at least agree on this? Hugo
From: Hugo gleaves on 22 Mar 2010 10:26 "Jonathan de Boyne Pollard" wrote: > > >>>>> > >>>>>> even if an app dies and the OS begins to automatically flush, the > >>>>>> logoff will be forced to hang until its done. > >>>>> > >>>>> M. Lebedinsky has explained that that is a falsehood. WINLOGON > >>>>> doesn't do any magic at logout to flush dirty pages. > >>>>> > >>> He didn't say that exactly. > >>> > >> Yes, xe did. Go and read all of what xe wrote. > >> > >>> [...] I simply logged out and logged back in, the file was there [...] > >>> > >> Now try attempting to demonstrate the forcible hang and flush of > >> dirty pages to disc that you claimed would occur, instead of > >> something quite different. > >> > > Jonathan the only circumstances I know of, that can lead to missing > > data [...] > > > For goodness' sake read! Read what you yourself claimed would happen, > that M. Lebedinsky had earlier explained to be a falsehood. And stop > repeatedly trying to prove something quite different. > > > I'm not looking for a bickering contest Jonathan, the OP asked an > > important question and it is important that we all try to stick with > > facts and not let our ego intrude. > > > Then stick with them, and stop using ad hominems like this when it's > pointed out that some things are not facts but falsehoods. > > . > Finally read this sentence from a microsoft document about MMFs: "Changes made to a memory-mapped file through a view of the file, other than the system pagefile, are automatically written to disk when the view is unmapped or when the file-mapping object is deleted." This is taken from: http://msdn.microsoft.com/en-us/library/ms810613.aspx I think you have to admit, that if one refers to officially published material, then I am correct about this whole issue. Now if I am wrong (and I may be) then we must accept that that official system documentation is wrong. Hugh
From: Hector Santos on 22 Mar 2010 12:39 Hugo gleaves(a)hotmail.com> <hugh wrote: > Finally read this sentence from a microsoft document about MMFs: > > "Changes made to a memory-mapped file through a view of the file, other than > the system pagefile, are automatically written to disk when the view is > unmapped or when the file-mapping object is deleted." > > This is taken from: http://msdn.microsoft.com/en-us/library/ms810613.aspx > > I think you have to admit, that if one refers to officially published > material, then I am correct about this whole issue. Now if I am wrong (and I > may be) then we must accept that that official system documentation is wrong. Right, but the above is insight into giving you the engineering requirement to prepare for it. In other words, it told programmers what will happen. The programmer now needs to prepare to unmap or delete the file map object to get the system to auto-flush for you. It doesn't mean that if a program ends (gracefully) without properly unmapped and deleting the object that the system will do this for you. Is it implied? I don't think so, I wouldn't trust it. You need to close your handles properly. Lets say your code is running and it has an file mapping object open, and has not be flushed yet. If your process get a WM_QUERYENDSESSION and/or don't handle a system broadcasted exit or shutdown event and don't begin a cleanup phase, what happens to the unflushed data? Are you sure objects that are not programmatically closed by your process, the OS will be quaranteed to be flushed for you? Can we simulate this with quick test? - create a small console program with a file map - add SMALL AMOUNTS OF data to the mapping view, no flushing. - pause by calling _getch() now abort the process by hitting control C. See if the data was FLUSHED to the disk file. I would also try this with a file across a network. -- HLS
From: Hugo gleaves on 22 Mar 2010 18:22
"Hector Santos" wrote: > Hugo gleaves(a)hotmail.com> <hugh wrote: > > > Finally read this sentence from a microsoft document about MMFs: > > > > "Changes made to a memory-mapped file through a view of the file, other than > > the system pagefile, are automatically written to disk when the view is > > unmapped or when the file-mapping object is deleted." > > > > This is taken from: http://msdn.microsoft.com/en-us/library/ms810613.aspx > > > > I think you have to admit, that if one refers to officially published > > material, then I am correct about this whole issue. Now if I am wrong (and I > > may be) then we must accept that that official system documentation is wrong. > > Right, but the above is insight into giving you the engineering > requirement to prepare for it. In other words, it told programmers > what will happen. The programmer now needs to prepare to unmap or > delete the file map object to get the system to auto-flush for you. > It doesn't mean that if a program ends (gracefully) without properly > unmapped and deleting the object that the system will do this for you. > > Is it implied? I don't think so, I wouldn't trust it. You need to > close your handles properly. > > Lets say your code is running and it has an file mapping object open, > and has not be flushed yet. If your process get a WM_QUERYENDSESSION > and/or don't handle a system broadcasted exit or shutdown event and > don't begin a cleanup phase, what happens to the unflushed data? Are > you sure objects that are not programmatically closed by your process, > the OS will be quaranteed to be flushed for you? > > Can we simulate this with quick test? > > - create a small console program with a file map > - add SMALL AMOUNTS OF data to the mapping view, no flushing. > - pause by calling _getch() > > now abort the process by hitting control C. > > See if the data was FLUSHED to the disk file. > > I would also try this with a file across a network. > > > -- > HLS > . > Well I've done such test many times, including forcibly killing the process, delibertaley raising a fatal exception inside the process and shutting down the machine when the process does no shutdown-handling. In every case every page is correctly written to the disk. When a process terminates (no matter how) the OS actually does the terminating and executes an orderly sequence of cleanup steps, what we see as a brutal crash in an app is not seen like that by the OS. The OS must free up kernel space that has been used to store process and thread descriptors, IO related kernel objects and many other things. This involves automatically closing all handles opened by the app (handles MUST be cleaned up, else other apps might think a file is in use etc) so in short, unless the OS is prevented from operating (OS crash, powerfail etc) then it doesn't matter what you do, die, crash, exit, close some handles, no handles, all handles, unmap or not, all modified pages will be written to disk. Now I recognize that such modified pages, (when all sharers have exited) will be written asynchronously and that this involves a degree of uncertainty, so I never once advocated NOT doing a flush. If you don't flush and the app dies, and then ten seconds later the OS crashes, then yes, some modified pages may be lost I accept that. Doing a flush reduces the size of this window, but then again the OS might crash 5 seconds before you do a flush! So doing a flush is a very good idea (in most cases) and I have not said otherwise, but the OP asks some very direct questions and I wanted to give my view on this, I have used mapping and sharing a great deal in my career. Hugh |