Flushing file writes to disk with 100% reliability
in [Unix Programming]
|
Prev: Learn about proxy sites and how to use them to open blocked sites unlimited downloads from RapidShare and megaupload and increase the speed of the Internet with new sites for free
Next: How reliable are named pipes?
From: Ian Collins on 8 Apr 2010 19:54 On 04/ 9/10 11:48 AM, Peter Olcott wrote: > > I am trying to have completely reliable writes to a > transaction log. This transaction log includes financial > transactions. Even if someone pulls the plug in the middle > of a transaction I want to only lose this single transaction > and not have and other missing or corrupted data. > > One aspect of the solution to this problem is to make sure > that all disk write are immediately flushed to the actual > platters. It is this aspect of this problem that I am > attempting to solve in this thread. Can't you rely on your database to manage this for you? -- Ian Collins
From: Peter Olcott on 8 Apr 2010 19:55 "Moi" <root(a)invalid.address.org> wrote in message news:aa4e5$4bbe6bd6$5350c024$25809(a)cache100.multikabel.net... > On Thu, 08 Apr 2010 18:43:40 -0500, Peter Olcott wrote: > >> "Ian Collins" <ian-news(a)hotmail.com> wrote in message >> news:827764Fp8jU12(a)mid.individual.net... >>> On 04/ 9/10 11:13 AM, Chris Friesen wrote: >>>> On 04/08/2010 10:58 AM, Peter Olcott wrote: >>>>> Is there a completely certain way that a write to a >>>>> file can be >>>>> flushed to the disk that encompasses every possible >>>>> memory buffer, including the hard drives onboard >>>>> cache? I >>>>> want to be able to yank the power cord at any moment >>>>> and not >>>>> get corrupted data other than the most recent single >>>>> transaction. >>>> >>>> Do you know what OS, OS version, filesystem, disk >>>> controller, and hard >>>> disk you're using? >>>> >>>> If not, then the generic answer is that you need to >>>> disable the hard >>>> drive cache. >>>> >>>> fsync() is supposed to work even with the cache >>>> enabled, but depending >>>> on the OS and version it may not guarantee that the >>>> data has hit the >>>> platter. >>> >>> Or the drive (or array controller) firmware. There are >>> plenty of >>> points between application code and the platter that can >>> lie about >>> honouring a sync request. >>> >>> -- >>> Ian Collins >> >> This is exactly why I made this post. I want to >> categorically handle all >> of these. This is what I have so far: >> >> Turn drive write caching off then >> open with O_SYNC, then >> write() followed by fsync() > > It is equivalent. If you open with O_SYNC or O_DATASYNC, > the system acts *as if* > every write were followed by a sync(). > > But that is only at the OS level. The disk driver will > commit it to the > disk, and the disk firmware may decide to let it wait a > few rotations > before the block hits the platter. > > AvK > I was told to use fsync() not sync().
From: Peter Olcott on 8 Apr 2010 19:56 "Moi" <root(a)invalid.address.org> wrote in message news:aa4e5$4bbe6bd6$5350c024$25809(a)cache100.multikabel.net... > On Thu, 08 Apr 2010 18:43:40 -0500, Peter Olcott wrote: > >> "Ian Collins" <ian-news(a)hotmail.com> wrote in message >> news:827764Fp8jU12(a)mid.individual.net... >>> On 04/ 9/10 11:13 AM, Chris Friesen wrote: >>>> On 04/08/2010 10:58 AM, Peter Olcott wrote: >>>>> Is there a completely certain way that a write to a >>>>> file can be >>>>> flushed to the disk that encompasses every possible >>>>> memory buffer, including the hard drives onboard >>>>> cache? I >>>>> want to be able to yank the power cord at any moment >>>>> and not >>>>> get corrupted data other than the most recent single >>>>> transaction. >>>> >>>> Do you know what OS, OS version, filesystem, disk >>>> controller, and hard >>>> disk you're using? >>>> >>>> If not, then the generic answer is that you need to >>>> disable the hard >>>> drive cache. >>>> >>>> fsync() is supposed to work even with the cache >>>> enabled, but depending >>>> on the OS and version it may not guarantee that the >>>> data has hit the >>>> platter. >>> >>> Or the drive (or array controller) firmware. There are >>> plenty of >>> points between application code and the platter that can >>> lie about >>> honouring a sync request. >>> >>> -- >>> Ian Collins >> >> This is exactly why I made this post. I want to >> categorically handle all >> of these. This is what I have so far: >> >> Turn drive write caching off then >> open with O_SYNC, then >> write() followed by fsync() > > It is equivalent. If you open with O_SYNC or O_DATASYNC, > the system acts *as if* > every write were followed by a sync(). > > But that is only at the OS level. The disk driver will > commit it to the > disk, and the disk firmware may decide to let it wait a > few rotations > before the block hits the platter. > > AvK > Oh yeah to solve the problem of disk drive onboard cache, simply turn off write caching.
From: Peter Olcott on 8 Apr 2010 20:00 "Ian Collins" <ian-news(a)hotmail.com> wrote in message news:8278l3Fp8jU16(a)mid.individual.net... > On 04/ 9/10 11:48 AM, Peter Olcott wrote: >> >> I am trying to have completely reliable writes to a >> transaction log. This transaction log includes financial >> transactions. Even if someone pulls the plug in the >> middle >> of a transaction I want to only lose this single >> transaction >> and not have and other missing or corrupted data. >> >> One aspect of the solution to this problem is to make >> sure >> that all disk write are immediately flushed to the actual >> platters. It is this aspect of this problem that I am >> attempting to solve in this thread. > > Can't you rely on your database to manage this for you? > > -- > Ian Collins Not for the transaction log because it will not be in a database. The transaction log file will be the primary means of IPC. Named pipes will provide event notification of changes to the log file, and the file offset of these changes.
From: Ian Collins on 8 Apr 2010 20:26
On 04/ 9/10 12:00 PM, Peter Olcott wrote: > "Ian Collins"<ian-news(a)hotmail.com> wrote in message > news:8278l3Fp8jU16(a)mid.individual.net... >> On 04/ 9/10 11:48 AM, Peter Olcott wrote: >>> >>> I am trying to have completely reliable writes to a >>> transaction log. This transaction log includes financial >>> transactions. Even if someone pulls the plug in the >>> middle >>> of a transaction I want to only lose this single >>> transaction >>> and not have and other missing or corrupted data. >>> >>> One aspect of the solution to this problem is to make >>> sure >>> that all disk write are immediately flushed to the actual >>> platters. It is this aspect of this problem that I am >>> attempting to solve in this thread. >> >> Can't you rely on your database to manage this for you? > > Not for the transaction log because it will not be in a > database. The transaction log file will be the primary > means of IPC. Named pipes will provide event notification of > changes to the log file, and the file offset of these > changes. It sounds very much (here and in other threads) like you are trying to reinvent database transactions. just sore everything in a database and signal watchers when data is updated. Databases had atomic transactions licked decades ago! -- Ian Collins |