Flushing file writes to disk with 100% reliability
in [Unix Programming]
|
Prev: Learn about proxy sites and how to use them to open blocked sites unlimited downloads from RapidShare and megaupload and increase the speed of the Internet with new sites for free
Next: How reliable are named pipes?
From: Scott Lurndal on 21 Apr 2010 12:52 "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes: > >"Ian Collins" <ian-news(a)hotmail.com> wrote in message >news:836u94Fvj5U5(a)mid.individual.net... >> On 04/21/10 06:06 AM, Peter Olcott wrote: >>> >>> SSD have a limited life that is generally not compatible >>> with extremely high numbers of transactions. >>> >> Not any more. >> >> They are used in the most transaction intensive (cache and >> logs) roles in many ZFS storage configurations. They are >> used where a very high number of IOPs are required. >> >> -- >> Ian Collins > >100,000 writes per cell and the best ones are fried. > http://en.wikipedia.org:80/wiki/Solid-state_drive Wikipedia? How about calling up Intel and asking their opinion? scott
From: Rainer Weikusat on 21 Apr 2010 13:17 scott(a)slp53.sl.home (Scott Lurndal) writes: > "Peter Olcott" <NoSpam(a)OCR4Screen.com> writes: >>"Ian Collins" <ian-news(a)hotmail.com> wrote in message >>news:836u94Fvj5U5(a)mid.individual.net... >>> On 04/21/10 06:06 AM, Peter Olcott wrote: >>>> >>>> SSD have a limited life that is generally not compatible >>>> with extremely high numbers of transactions. >>>> >>> Not any more. >>> >>> They are used in the most transaction intensive (cache and >>> logs) roles in many ZFS storage configurations. They are >>> used where a very high number of IOPs are required. >>> >>> -- >>> Ian Collins >> >>100,000 writes per cell and the best ones are fried. >> http://en.wikipedia.org:80/wiki/Solid-state_drive > > Wikipedia? How about calling up Intel and asking their opinion? Amusingly, Intel doesn't provide detailed information about the actual flash chips it uses anymore. Judging from code I have seen in the past, the very fact that these devices contain 'super secret Intel proprietary software' whose purpose is to maintain the illusion of a reliable storage technology by detecting and automatically correcting errors in the stored data is a sufficient reason for me to have no desire using these devices. The opinion of 'Intel' is, of course, that I really should be Intel products, even in absence of relevant technical information.
From: Ersek, Laszlo on 21 Apr 2010 13:18 On Wed, 21 Apr 2010, Scott Lurndal wrote: > Wikipedia? How about calling up [the vendor] and asking their opinion? While Wikipedia is oftentimes not much better than folklore, I'm not sure if the vendor (any vendor) could withstand its urge to provide padded stats. Secondly, I'm not sure if anybody would talk to me from [big vendor] if I wanted to buy eg. two pieces of hardware. My suggestion would be researching tomshardware.com, phoronix.com and anandtech.com, for the caliber in question. lacos
From: Peter Olcott on 21 Apr 2010 13:25 "Ersek, Laszlo" <lacos(a)caesar.elte.hu> wrote in message news:Pine.LNX.4.64.1004211912140.13685(a)login01.caesar.elte.hu... > On Wed, 21 Apr 2010, Scott Lurndal wrote: > >> Wikipedia? How about calling up [the vendor] and asking >> their opinion? > > While Wikipedia is oftentimes not much better than > folklore, I'm not sure if the vendor (any vendor) could > withstand its urge to provide padded stats. Secondly, I'm > not sure if anybody would talk to me from [big vendor] if > I wanted to buy eg. two pieces of hardware. > > My suggestion would be researching tomshardware.com, > phoronix.com and anandtech.com, for the caliber in > question. > > lacos Tom's hardware is ggod.
From: Joe Beanfish on 21 Apr 2010 14:07
On 04/20/10 14:06, Peter Olcott wrote: > "Joe Beanfish"<joe(a)nospam.duh> wrote in message > news:hqkp2c$c98(a)news.thunderstone.com... >> On 04/20/10 08:38, Peter Olcott wrote: >>> "Tony Delroy"<tony_in_da_uk(a)yahoo.co.uk> wrote in >>> message >>> news:7ed1ecad-c641-4d32-bcef-6bd5b3bc5632(a)c20g2000prb.googlegroups.com... >>> On Apr 9, 1:58 am, "Peter Olcott"<NoS...(a)OCR4Screen.com> >>> wrote: >>>> Is there a completely certain way that a write to a file >>>> can be flushed to the disk that encompasses every >>>> possible >>>> memory buffer, including the hard drives onboard cache? >>>> I >>>> want to be able to yank the power cord at any moment and >>>> not >>>> get corrupted data other than the most recent single >>>> transaction. >>> >>> --You'd have to check the hard disk programming >>> documents, >>> you may be >>> --able to do direct I/O to ensure your data is written. >>> Even if the >>> --drive's onboard cache has not been flushed, it might >>> have >>> enough >>> --capacitance to flush during a power failure, or use >>> non-volatile >>> --memory that can be flushed when power returns. Above >>> that, the answer >>> --is highly OS dependent, and you've specified absolutely >>> nothing about >>> --your hardware, OS, programming language etc.... >>> -- >>> --Cheers, >>> --Tony >>> >>> It looks like that OS is not the big problem. The OS can >>> always be bypassed, by working directly with the >>> hardware. >>> The big problem is that for example Western Digital SATA >>> drives simply do not implement the "Flush Cache" ATA >>> command. >>> >>> Seagate drives do implement this command. It was >>> Seagate's >>> idea to create this command in 2001. Although it may >>> still >>> be possible to simply shut off write caching for these >>> drives, this will wear the drive out much more quickly, >>> and >>> drastically reduce performance. >> >> Have you considered solid state hard disks? Server >> quality, not the >> cheap desktop quality ones. >> >> IMHO, with a magnetic HD with a journalling filesystem and >> a good UPS >> with software to shutdown before battery runs out are all >> you need. >> Then you won't have to sacrifice speed trying to sync all >> the way to >> the hard media. > > I will be renting my system from my service provider, thus > no choices are available for hardware. Both UPS and backup > generators are provided by my service provider. > > SSD have a limited life that is generally not compatible > with extremely high numbers of transactions. > > Some drives might not even be smart enough to flush their > buffers even when UPS kicks in. I guess that you could force > a buffer flush for every drive by simply writing a file > larger than the buffer. If you make sure that this file is > not fragmented, it might even be fast enough to do this > after every transaction. > > Obviously the best way to do this would be to have a drive > that correctly implements some sort of "Flush Cache" command > such as the ATA command. You're missing my point. Reliable power can eliminate the need to flush cache thereby saving a lot of hardware specific headaches and keeping the speed high. It's not like the cache will sit unwritten for days or even hours. An orderly shutdown when the UPS nears death is all that's needed. OTOH if you're going to be paranoid about every possibility don't ignore the possibility of flushing your cache onto a bad sector that won't read back. Do you have data redundancy in your plan? |