2 questions about memory mapped files
in [Win32 Kernel]
|
From: gartaud on 11 Mar 2010 10:26 Thanks. This makes sense. "Alexander Grigoriev" wrote: > > "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message > news:uXMrJtIwKHA.3952(a)TK2MSFTNGP06.phx.gbl... > > gartaud wrote: > > > > No, you have to flush it. At least that has been our practice. For a > > 365x24x7 system, we make exclusive use of mapped files and if thye > > customer's machine aborted or some idiot killed the process in some > > automated maintenance thing and didn't follow our own automated > > shutdown/restart procedure, they ended up calling support about file > > integrity issues requiring them to run our repair tool. > > > > In the last few years or so, we decided to minimize the support issues and > > now flush every (record) write to avoid such issues. We watched for > > complaints of performance. None, and thats probably because machines and > > hard drives are much faster than the time our software was first design > > many moons ago. > > > > I think you're mistaken. Whatever the application puts to the mapped memory > will be flushed to the file, even if the application dies. You problem was > more likely because an application killed in a wrong moment would leave > inconsistent state in the file. > > > . >
From: Hector Santos on 11 Mar 2010 11:32 Not quite. While all bets are off when a process is abnormally killed, AG failed to take into account maps over the network. Read the MSDN note on FlushViewOfFile() regarding its networking notes, and thats if you do use it. If you don't, you have no clue whether it is was flushed or not. In fact, an option in our file mapping class allows a different approach to FlushViewOfFile() - unmap the view, write to disk, flush it, and then recreate the view. This is how we get better reliability in a different customer environments where different storage arrangements are possible. We have enough field operations over thousands of installations to know that different network driver provider can have different networking caching than what you see on a local map. However, I will note that over time, with OS and driver corrections, even speed, etc, its rare to see the problems as we use to. --- HLS gartaud wrote: > Thanks. This makes sense. > > "Alexander Grigoriev" wrote: > >> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >> news:uXMrJtIwKHA.3952(a)TK2MSFTNGP06.phx.gbl... >>> gartaud wrote: >>> >>> No, you have to flush it. At least that has been our practice. For a >>> 365x24x7 system, we make exclusive use of mapped files and if thye >>> customer's machine aborted or some idiot killed the process in some >>> automated maintenance thing and didn't follow our own automated >>> shutdown/restart procedure, they ended up calling support about file >>> integrity issues requiring them to run our repair tool. >>> >>> In the last few years or so, we decided to minimize the support issues and >>> now flush every (record) write to avoid such issues. We watched for >>> complaints of performance. None, and thats probably because machines and >>> hard drives are much faster than the time our software was first design >>> many moons ago. >>> >> I think you're mistaken. Whatever the application puts to the mapped memory >> will be flushed to the file, even if the application dies. You problem was >> more likely because an application killed in a wrong moment would leave >> inconsistent state in the file. >> >> >> . >> -- HLS
From: Alexander Grigoriev on 11 Mar 2010 23:11 A mapped file is essentially just a cached file with its cache accessible to the application. In fact, the file cache itself is implemented through the file mapping. With that, the same guarantees apply to the file mapping, as to a cached file. Mapping a network file creates another failure point, which is unreliable nature of network connections. If you lose connectivity, your data can't be written back. But in any case, the data is flushed when the file mapping object is ultimately closed. How far flushed, depends on other attributes. For example, if you open the handle without FILE_FLAG_NO_BUFFERING, the file may also be picked by the cache manager, and then you may face undefined write-back latency even after the file is closed. "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message news:OGiGqhTwKHA.3564(a)TK2MSFTNGP05.phx.gbl... > Not quite. While all bets are off when a process is abnormally killed, AG > failed to take into account maps over the network. > > Read the MSDN note on FlushViewOfFile() regarding its networking notes, > and thats if you do use it. If you don't, you have no clue whether it is > was flushed or not. > > In fact, an option in our file mapping class allows a different approach > to FlushViewOfFile() - unmap the view, write to disk, flush it, and then > recreate the view. This is how we get better reliability in a different > customer environments where different storage arrangements are possible. > > We have enough field operations over thousands of installations to know > that different network driver provider can have different networking > caching than what you see on a local map. However, I will note that over > time, with OS and driver corrections, even speed, etc, its rare to see the > problems as we use to. > > --- > HLS > > gartaud wrote: > >> Thanks. This makes sense. >> >> "Alexander Grigoriev" wrote: >> >>> "Hector Santos" <sant9442(a)nospam.gmail.com> wrote in message >>> news:uXMrJtIwKHA.3952(a)TK2MSFTNGP06.phx.gbl... >>>> gartaud wrote: >>>> >>>> No, you have to flush it. At least that has been our practice. For a >>>> 365x24x7 system, we make exclusive use of mapped files and if thye >>>> customer's machine aborted or some idiot killed the process in some >>>> automated maintenance thing and didn't follow our own automated >>>> shutdown/restart procedure, they ended up calling support about file >>>> integrity issues requiring them to run our repair tool. >>>> >>>> In the last few years or so, we decided to minimize the support issues >>>> and now flush every (record) write to avoid such issues. We watched >>>> for complaints of performance. None, and thats probably because >>>> machines and hard drives are much faster than the time our software was >>>> first design many moons ago. >>>> >>> I think you're mistaken. Whatever the application puts to the mapped >>> memory will be flushed to the file, even if the application dies. You >>> problem was more likely because an application killed in a wrong moment >>> would leave inconsistent state in the file. >>> >>> . >>> > > > > -- > HLS
From: Pavel Lebedinsky [MSFT] on 12 Mar 2010 01:52 > But in any case, the data is flushed when the file mapping object is > ultimately closed. There are two ways a modified mapped page can be written to disk: 1. As a result of an explicit FlushViewOfFile call. 2. After a page has been trimmed or unmapped from all working sets, the mapped page writer thread will (eventually) write it to disk. The OS makes no guarantees as to how long this will take, but eventally all such pages will be written to their backing storage, unless the system crashes or loses power, or the backing storage becomes unavailable (e.g. network is disconnected). Closing the file mapping object has no effect on when the pages will be written out. -- Pavel Lebedinsky/Windows Fundamentals Test This posting is provided "AS IS" with no warranties, and confers no rights.
From: Alexander Grigoriev on 12 Mar 2010 09:41
Does cache manager guarantee that the pages are flushed for a remote file, when a logon session is being closed (an user logs out)? "Pavel Lebedinsky [MSFT]" <pavel(a)online.microsoft.com> wrote in message news:%23K76VCbwKHA.4636(a)TK2MSFTNGP06.phx.gbl... >> But in any case, the data is flushed when the file mapping object is >> ultimately closed. > > There are two ways a modified mapped page can be written to disk: > > 1. As a result of an explicit FlushViewOfFile call. > > 2. After a page has been trimmed or unmapped from all working sets, > the mapped page writer thread will (eventually) write it to disk. > The OS makes no guarantees as to how long this will take, but > eventally all such pages will be written to their backing storage, > unless the system crashes or loses power, or the backing storage > becomes unavailable (e.g. network is disconnected). > > Closing the file mapping object has no effect on when the pages will > be written out. > > -- > Pavel Lebedinsky/Windows Fundamentals Test > This posting is provided "AS IS" with no warranties, and confers no > rights. > |