Prev: KMail column display
Next: M.I 5-Persecu tion . t he BBC , te levision and rad io
From: Jim Howes on 31 Dec 2007 17:30
Folderol wrote:
> I have an application that will be run on a fairly robust industrial
> computer. The hardware side is pretty much cut and dried, but I have
> concerns about the software.
>
> The main problem is that I can absolutely guarantee that the system will
> be quite frequently just switched off at the supply while running.
>
> There is only one configuration text file that is saved (quite
> frequently) and it wouldn't be a disaster if this became corrupted and
> was dropped back to default settings. However, how can I protect
> against corruption of other parts of the system, and less important,
> avoid a long slow disc check next time the machine is rebooted.

One option is to use the 'sync' filesystem option, but we ran into
trouble with that when used on the root filesystem.

Careful rearranging of the filesystem meant that on our system we were
able to boot from a root fs mounted read-only (either from a local disk,
a M-Systems DiskOnChip, or an initial RamDisk which we never unmount),
and various other filesystems either mounted with the sync option,
recreated from scratch at boot prior to fsck/mount (for temporary
files), and for critical stuff we didn't want to lose, mounted via
NFS/CIFS/SMB/whatever from a server at the other end of some ethernet
string which is in a room where people DO read notices about turning
things off, and has a pretty huge ****-off UPS thankyouverymuch.

That job is made more difficult the more things are actually running on
the box. If it's X plus a single app, with none of the usual
cups/xinetd/whateverelsed stuff running in the background, then it's
usually just a matter of pointing /tmp, /var/tmp, /var/log and the home
directory of the user starting X in the right place (If you start X as a
user, rather than using something like Xdm/kdm, it's demands on the
filesystem are somewhat less, and for turnkey apps, you can probably
/dev/null most of it's log files)


In fact, there is possibly mileage in having the entire system boot from
the network, mount the root fs over the network and only use the local
disk for temporary files and swap. Less to go wrong and easier to
centrally managed. (And an NFS filesystem doesn't need checking)

Jim
From: Roger Leigh on 31 Dec 2007 17:19
Folderol <folderol(a)ukfsn.org> writes:

> There is only one configuration text file that is saved (quite
> frequently) and it wouldn't be a disaster if this became corrupted and
> was dropped back to default settings. However, how can I protect
> against corruption of other parts of the system, and less important,
> avoid a long slow disc check next time the machine is rebooted.

Mount *everything* read only. This way, it can't get corrupted.
Current systems can be made completely read only if you try hard
enough. Debian is introducing the /lib/init/rw tmpfs for read-write
in early boot to make / read-only, though this isn't totally done
yet. You'll probably want /var writable for logging, but for an
embedded system you might equally want to remove syslog and make it
read only as well.

For the one writable file, give it a dedicated writable, journalled,
filesystem. Use fdatasync() to create write barriers to ensure it
doesn't get corrupted during writing. This will depend upon how you
write the file (append only, multiple writers, etc.).


Regards,
Roger

--
.''`. Roger Leigh
: :' : Debian GNU/Linux http://people.debian.org/~rleigh/
`. `' Printing on GNU/Linux? http://gutenprint.sourceforge.net/
`- GPG Public Key: 0x25BFB848 Please GPG sign your mail.
From: Unruh on 31 Dec 2007 22:20
Roger Leigh <rleigh(a)whinlatter.ukfsn.org> writes:

>--=-=-=
>Content-Transfer-Encoding: quoted-printable

>Folderol <folderol(a)ukfsn.org> writes:

>> There is only one configuration text file that is saved (quite
>> frequently) and it wouldn't be a disaster if this became corrupted and
>> was dropped back to default settings. However, how can I protect
>> against corruption of other parts of the system, and less important,
>> avoid a long slow disc check next time the machine is rebooted.

>Mount *everything* read only. This way, it can't get corrupted.
>Current systems can be made completely read only if you try hard
>enough. Debian is introducing the /lib/init/rw tmpfs for read-write
>in early boot to make / read-only, though this isn't totally done
>yet. You'll probably want /var writable for logging, but for an
>embedded system you might equally want to remove syslog and make it
>read only as well.

>For the one writable file, give it a dedicated writable, journalled,
>filesystem. Use fdatasync() to create write barriers to ensure it
>doesn't get corrupted during writing. This will depend upon how you
>write the file (append only, multiple writers, etc.).

Alternativly put the log files onto tha remote computer which yuo can nfs
mount, or have syslog write the external system.
That way nothing on the machine itself need be writeable.

From: Gordon Henderson on 1 Jan 2008 07:02
In article <20071231211713.78fd7b51(a)ukfsn.org>,
Folderol <folderol(a)ukfsn.org> wrote:
>On Mon, 31 Dec 2007 19:58:05 +0000
>Darren Salt <news(a)youmustbejoking.demon.cu.invalid> wrote:
>
>Thanks for such a quick reply.
>
>> I demand that Folderol may or may not have written...
>>
>> [snip]
>> > The main problem is that I can absolutely guarantee that the system will
>> > be quite frequently just switched off at the supply while running.
>>
>> Two notices, one "press briefly to shut down" pointing at the power button,
>> one "use the computer's power button to shut down properly" at the plug? And
>> have a cluebat handy :-)
>>
>> I think that you want to make it easy to shut down from the power button
>> (requires acpid) and very awkward (at least, normally so) to switch off at
>> the plug or, if it has a switch, the PSU. And make sure that the BIOS's
>> power-off setting is _not_ "instant off"...
>
>Unfortunately this is not an option. In the first place, for HSE
>compliance there must be an easily accessible means of quickly
>removing *all* power to the machinery. This is usually necessary when
>the machine crashes (much more 'interesting' than a software crash).

I've worked in such an environment - many years ago - I was doing
"robotics" and factory automation and stuff. The big CNC machines were
good in that the newer ones all had non-volatile memory (bubble IIRC)
and the older ones were so dumb, they just read of punched tape...

And more recently I've developed a small number of "appliances" which
boot off linux off flash and run from RAM, and the issue of how to
"remember" various updates was foremost on my mind too..

One box - an Asterisk PBX system has 2 main "lumps" of data to store -
one is the voicemail - I consider this to be stored on a "reasonable
effort" basis. I chose ext2 rather than 3 and arranged a forced fsck
(should it need it) at boot time - the reason at the time was to
minimise writes to the flash device which ext3 would be constantly doing
(I've since found out how to minimise this with ext3 though, so it may
not be the problem I first thought it was)

The other thing I need to store is the system configuration files -
these are a small number of small text files, so I chose to 'tar' them
into an empty partition on the flash device. This is restored very early
in the boot sequence, so holds things like the IP address and so on.

There is still the issue of power failure during the actual 'tar' write
when it stores the updated data and there are techniques to overcome even
this (2nd partition, marking it 'clean', swapping to the 'clean'/active
one, etc.)

But where do you draw the line?

>> > There is only one configuration text file that is saved (quite frequently)
>> > and it wouldn't be a disaster if this became corrupted and was dropped back
>> > to default settings. However, how can I protect against corruption of other
>> > parts of the system, and less important, avoid a long slow disc check next
>> > time the machine is rebooted.
>>
>> Use journalled file systems. I'd go straight for ext3 (except, perhaps, for
>> very small partitions which it takes only a few seconds to check even with
>> ext2).
>
>Hmmm. I was thinking of using ext3 anyway. It looks pretty mature and
>well supported.

If you've got a real hard-drive in the system then you'll probably be
OK. When I was developing my appliances, I looked at other devices - the
Nokia 770 uses JFS (IBM's journalling filesystem), and I considered that
rather than ext2/3. I've also experience of XFS in other settings (multi
TB filestores), but wasn't happy enough with it for an "embedded"
system, so stuck with ext2/3 for the "live" side of things.

I'd like to know how things like the "Slug" and so (domestic ADSL
routers, etc.) on store their configurations, but have never had the
time to look closely. I suspect for most consumer devices, there is just
an area of non volatile RAM of some kind where they store byte values
and hope it doesn't get corrupted... (cf. PC BIOSes)

>> There's not much that you can do about deferred writes short of making the
>> flush to disk happen sooner (/proc/sys/vm/dirty_writeback_centisecs,
>> /proc/sys/vm/dirty_expire_centisecs), but if the power switch takes
>> sufficiently long to get to, this won't really matter.
>>
>> [snip]
>
>Would a 'sync' after each write take care of this?

fsync(2) on the file-handle before closing the file - but beware that
some disk devices return "written" when they really haven't. There were
some tests a while back about this. See the man page and do some
googling...

>Once started, the program takes over the whole machine but uses X GUI.
>If I disable logging, and shutdown almost all services, could I be
>reasonably confident the OS won't be writing anything to disc once the
>system is up and running?
>
>Must-haves are: display, mouse, keyboard, usb, RS232.

How about running it entirely in RAM?

You need to build an initrd.gz of exactly what you need, then arrange a
mechanism to overwrite a few config files at boot-time...

Good luck!

Gordon
(And a Happy New year!)
From: Folderol on 1 Jan 2008 07:11
On Mon, 31 Dec 2007 22:19:49 +0000
Roger Leigh <rleigh(a)whinlatter.ukfsn.org> wrote:

Thanks everyone for your replies.

> Folderol <folderol(a)ukfsn.org> writes:
>
> > There is only one configuration text file that is saved (quite
> > frequently) and it wouldn't be a disaster if this became corrupted and
> > was dropped back to default settings. However, how can I protect
> > against corruption of other parts of the system, and less important,
> > avoid a long slow disc check next time the machine is rebooted.
>
> Mount *everything* read only. This way, it can't get corrupted.
> Current systems can be made completely read only if you try hard
> enough. Debian is introducing the /lib/init/rw tmpfs for read-write
> in early boot to make / read-only, though this isn't totally done
> yet. You'll probably want /var writable for logging, but for an
> embedded system you might equally want to remove syslog and make it
> read only as well.

This looks like the best option for us. I didn't realise you could do a
read-only boot. I must be lagging behind :(

We had already thought of removing/disabling syslog.

> For the one writable file, give it a dedicated writable, journalled,
> filesystem. Use fdatasync() to create write barriers to ensure it
> doesn't get corrupted during writing. This will depend upon how you
> write the file (append only, multiple writers, etc.).
>
>
> Regards,
> Roger

Hmmm, didn't think of giving it a dedicated partition.

I wonder if there would be anything gained by using a solid-state
drive, both from a writing speed point of view and maybe less likelihood
of corruption if the power dies.

To those that suggested network solutions. This isn't an option for
several reasons - the most embarrassing being none of us have any
network experience!

This is our second refurb project at this factory (they seem to like
our work) and networking machines to the offices was briefly mentioned
in our early discussions. The owners were instantly hostile to the
idea. All their machines run completely independently.

Interestingly it is only the European manufactured kit they want
refurbed. These machines just plod on forever. Being massively
constructed and metric standard, almost all the mechanical parts are
easily replaced.

On this one, we are replacing a DOS system which in itself replaced a
TTL 'controller'!

They also have one machine which is an electronic engineers dream. It
still has in service the original variable reluctance DC motor
controller. The only concession to progress being the replacement of the
*massive* rectifiers with modern silicon ones.

--
Will J G
First  |  Prev  |  Next  |  Last
Pages: 1 2 3
Prev: KMail column display
Next: M.I 5-Persecu tion . t he BBC , te levision and rad io