Flushing file writes to disk with 100% reliability
in [Unix Programming]
|
Prev: Learn about proxy sites and how to use them to open blocked sites unlimited downloads from RapidShare and megaupload and increase the speed of the Internet with new sites for free
Next: How reliable are named pipes?
From: Ian Collins on 8 Apr 2010 22:49 On 04/ 9/10 02:37 PM, Peter Olcott wrote: > "Ian Collins"<ian-news(a)hotmail.com> wrote in message > news:827evjFp8jU20(a)mid.individual.net... >> On 04/ 9/10 01:21 PM, Peter Olcott wrote: >>> "Ian Collins"<ian-news(a)hotmail.com> wrote in message >>> news:827d0gFp8jU18(a)mid.individual.net... >>>> On 04/ 9/10 12:43 PM, Peter Olcott wrote: >>>>> "Ian Collins"<ian-news(a)hotmail.com> wrote: >>> Exactly how fault tolerant is it with the server's power >>> cord yanked from the wall? >> >> As good as any. If you want 5 nines reliability you have >> to go a lot further than synchronous writes. My main >> server has highly redundant raid (thinks to ZFS), >> redundant PSUs and a UPS. I'm not quite at the generator >> stage yet, our power here is very dependable :). > > Do you also have disk write caching turned off? I leave that call to the filesystem, see other post. -- Ian Collins
From: Peter Olcott on 8 Apr 2010 23:23 "Ian Collins" <ian-news(a)hotmail.com> wrote in message news:827iu3FgvmU2(a)mid.individual.net... > On 04/ 9/10 02:37 PM, Peter Olcott wrote: >> "Ian Collins"<ian-news(a)hotmail.com> wrote in message >> news:827evjFp8jU20(a)mid.individual.net... >>> On 04/ 9/10 01:21 PM, Peter Olcott wrote: >>>> "Ian Collins"<ian-news(a)hotmail.com> wrote in message >>>> news:827d0gFp8jU18(a)mid.individual.net... >>>>> On 04/ 9/10 12:43 PM, Peter Olcott wrote: >>>>>> "Ian Collins"<ian-news(a)hotmail.com> wrote: >>>> Exactly how fault tolerant is it with the server's >>>> power >>>> cord yanked from the wall? >>> >>> As good as any. If you want 5 nines reliability you >>> have >>> to go a lot further than synchronous writes. My main >>> server has highly redundant raid (thinks to ZFS), >>> redundant PSUs and a UPS. I'm not quite at the >>> generator >>> stage yet, our power here is very dependable :). >> >> Do you also have disk write caching turned off? > > I leave that call to the filesystem, see other post. > > -- > Ian Collins I am not sure that this can work this way. Maybe it can and so much the better. It may be hardware specific and not very well supported by the OS.
From: David Schwartz on 9 Apr 2010 15:35 On Apr 8, 4:56 pm, "Peter Olcott" <NoS...(a)OCR4Screen.com> wrote: > Oh yeah to solve the problem of disk drive onboard cache, > simply turn off write caching. If this really is a hard requirement, you will have to do one of two things: 1) Strictly control the hardware and software that is to be used. Unfortunately, in-between your call to 'fdatasync' and the physical platters, there are lots of places where the code can be lied to and told that the sync is completed when it's really not. 2) Use an architecture that inherently provides this guarantee by design. For example, if you commit a transaction to a separate storage system before you move on, you are guaranteed that the transaction will not be lost unless both this system and that separate system fail concurrently. I think it's fair to say that you will never get this to 100%, so if you need the overall system reliability to be high, one factor will have to be high hardware reliability. Even if you can only get this to 99%, if a power loss only occurs once a year, the system will, on average, only fail once per hundred years. You can achieve this with redundant power supplies plugged into separate UPSes. RAID 6 with hot spares helps too. (Tip: Make sure your controller is set to periodically *test* your hot spares!) DS
From: Peter Olcott on 9 Apr 2010 16:46 "David Schwartz" <davids(a)webmaster.com> wrote in message news:683c22d6-b73f-44cf-ae8d-8df79039bc4b(a)w42g2000yqm.googlegroups.com... On Apr 8, 4:56 pm, "Peter Olcott" <NoS...(a)OCR4Screen.com> wrote: > Oh yeah to solve the problem of disk drive onboard cache, > simply turn off write caching. --If this really is a hard requirement, you will have to do one of two --things: --1) Strictly control the hardware and software that is to be used. --Unfortunately, in-between your call to 'fdatasync' and the physical --platters, there are lots of places where the code can be lied to and --told that the sync is completed when it's really not. --2) Use an architecture that inherently provides this guarantee by --design. For example, if you commit a transaction to a separate storage --system before you move on, you are guaranteed that the transaction --will not be lost unless both this system and that separate system fail --concurrently. It looks like one piece that is often missing (according to one respondent) is that fsync() is often broken. From what I understand this makes the whole system much less reliable as this relates to committed transactions. The only way that I could think of to account for this is to provide some sort of transaction-by-transaction on-the-fly offsite backup. One simple way to do this (I don't know how reliable it would be) would be to simply email the transactions to myself. Another way would be to provide some sort of HTTP based web service that can accept and archive transactions from another HTTP web service. The main transactions that I want to never lose track of is whenever a customer adds money to their user account. All other transactions are less crucial. --I think it's fair to say that you will never get this to 100%, so if --you need the overall system reliability to be high, one factor will --have to be high hardware reliability. Even if you can only get this to --99%, if a power loss only occurs once a year, the system will, on --average, only fail once per hundred years. You can achieve this with --redundant power supplies plugged into separate UPSes. RAID 6 with hot --spares helps too. (Tip: Make sure your controller is set to --periodically *test* your hot spares!) DS
From: Golden California Girls on 9 Apr 2010 21:30
Peter Olcott wrote: > "David Schwartz" <davids(a)webmaster.com> wrote in message > news:683c22d6-b73f-44cf-ae8d-8df79039bc4b(a)w42g2000yqm.googlegroups.com... > On Apr 8, 4:56 pm, "Peter Olcott" <NoS...(a)OCR4Screen.com> > wrote: > >> Oh yeah to solve the problem of disk drive onboard cache, >> simply turn off write caching. > > --If this really is a hard requirement, you will have to do > one of two > --things: > > --1) Strictly control the hardware and software that is to > be used. > --Unfortunately, in-between your call to 'fdatasync' and the > physical > --platters, there are lots of places where the code can be > lied to and > --told that the sync is completed when it's really not. > > --2) Use an architecture that inherently provides this > guarantee by > --design. For example, if you commit a transaction to a > separate storage > --system before you move on, you are guaranteed that the > transaction > --will not be lost unless both this system and that separate > system fail > --concurrently. > > It looks like one piece that is often missing (according to > one respondent) is that fsync() is often broken. From what I > understand this makes the whole system much less reliable as > this relates to committed transactions. The only way that I > could think of to account for this is to provide some sort > of transaction-by-transaction on-the-fly offsite backup. > > One simple way to do this (I don't know how reliable it > would be) would be to simply email the transactions to > myself. Another way would be to provide some sort of HTTP > based web service that can accept and archive transactions > from another HTTP web service. The main transactions that I > want to never lose track of is whenever a customer adds > money to their user account. All other transactions are less > crucial. > > --I think it's fair to say that you will never get this to > 100%, so if > --you need the overall system reliability to be high, one > factor will > --have to be high hardware reliability. Even if you can only > get this to > --99%, if a power loss only occurs once a year, the system > will, on > --average, only fail once per hundred years. You can achieve > this with > --redundant power supplies plugged into separate UPSes. RAID > 6 with hot > --spares helps too. (Tip: Make sure your controller is set > to > --periodically *test* your hot spares!) I think there may be another issue as well, despite everything working for the file and its data, when does the kernel issue its write request to update the directory? Even if every buffer for the file is written, if the directory isn't updated because the kernel hasn't asked for it then you are still hosed. Of course there is a work around for this, don't use the file system. |