From: Andi Kleen on
On Thu, Jun 10, 2010 at 12:38:10PM -0600, Brian Gordon wrote:
> > It's also a serious consideration for standard servers.
> Yes. Good point.
>
> > On server class systems with ECC memory hardware does that.
>
> > Normally server class hardware handles this and the kernel then reports
> > memory errors (e.g. through mcelog or through EDAC)
>
> Agreed. EDAC is a good and sane solution and most companies do this.

Sorry, but you mean ECC?

IMHO EDAC Is not a good solution for error reporting (however I'm biased
because I work on a better one)

> Some do not due to naivity or cost reduction. EDAC doesn't cover
> processor registers and I have fairly good solutions on how to deal
> with that in tiny "home-grown" tasking systems.

mcelog covers OS visible processor registers on x86 systems.

If your hardware doesn't support it it's hard for the general
case, although special cases are always possible.
>
> > Lower end systems which are optimized for cost generally ignore the
> > problem though and any flipped bit in memory will result
> > in a crash (if you're lucky) or silent data corruption (if you're unlucky)
>
> Right! And this is the area that I am interested in. Some people
> insist on lowering the cost of the hardware without considering these
> issues. One thing I want to do is to be as diligent as possible (even
> in these low cost situations) and do the best job I can in spite of
> the low cost hardware.

AFAIK there's no support for this in a standard Linux kernel.

That is some architectures do scrubbing in software,
but the basic ECC implementation is still hardware.

In general I suspect you'll need some application specific
strategy, if your hardware doesn't help you in this.

Having good hardware definitely helps, software is generally
not happy if it cannot trust its memory enough.

It's a bit like a human with no reliable air supply.

That is the existing memory error handling mechanisms (like hwpoison)
assume events are reliably detected and are relatively rare.

>
> So, some pages of RAM are going to be read-only and the data in those
> pages came from some source (file system?). Can anyone describe a
> high level strategy to occasionaly provide some coverage of this data?

Just for block data there's some support for checksumming
e.g, block integrity (needs special support in the device)
or file systems (e.g. btrfs)

However they all normally assume memory is reliable and
are more focussed on errors coming from storage.

>
> So far I have thought about page descriptors adding an MD5 hash
> whenever they are read-only and first being "loaded/mapped?" and then
> a background daemon could occasionaly verify.

In theory btrfs or block integrity could be probably extended
to regularly re check page cache. It would not be trivial

But to really catch errors before use you would need to recheck on
every access, and that's hard (or rather extremly slow) in some cases
(e.g. mmap)

And this still wouldn't help with r/w memory. Normally on most
workloads r/o (that is clean) memory is the only a small fraction of the
active memory.

-Andi

--
ak(a)linux.intel.com -- Speaking for myself only.
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Chris Friesen on
On 06/10/2010 12:38 PM, Brian Gordon wrote:

> On the more exotic end, I have also seen systems that have dual
> redundant processors / memories. Then they add compare logic between
> the redundant processors that compare most pins each clock cycle. If
> any pins are not identical at a clock cycle, then something has gone
> wrong (SEU, hardware failure, etc..)

Some phone switches do this. Some of them also have at least two copies
of everything in memory and will do transactional operations that can be
rolled back if there is a hardware glitch.

> So, some pages of RAM are going to be read-only and the data in those
> pages came from some source (file system?). Can anyone describe a
> high level strategy to occasionaly provide some coverage of this data?

> So far I have thought about page descriptors adding an MD5 hash
> whenever they are read-only and first being "loaded/mapped?" and then
> a background daemon could occasionaly verify.

Makes sense to me. You might also pick an on-disk format with extra
checksumming so you could compare the on-disk checksum with the
in-memory checksum.

Chris

--
The author works for GENBAND Corporation (GENBAND) who is solely
responsible for this email and its contents. All enquiries regarding
this email should be addressed to GENBAND. Nortel has provided the use
of the nortel.com domain to GENBAND in connection with this email solely
for the purpose of connectivity and Nortel Networks Inc. has no
liability for the email or its contents. GENBAND's web site is
http://www.genband.com
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Brian Gordon on
Thank you both. This has been very helpful for me.

I think I read two conclusions:
1) R/O is a small percentage of RAM
2) To cover this small precentage would be non-trivial

Thank you both very much for your time and knowledge, I'll move along now.



On Thu, Jun 10, 2010 at 12:46 PM, Chris Friesen <cfriesen(a)nortel.com> wrote:
> On 06/10/2010 12:38 PM, Brian Gordon wrote:
>
>> On the more exotic end, I have also seen systems that have dual
>> redundant processors / memories. �Then they add compare logic between
>> the redundant processors that compare most pins each clock cycle. � If
>> any pins are not identical at a clock cycle, then something has gone
>> wrong (SEU, hardware failure, etc..)
>
> Some phone switches do this. �Some of them also have at least two copies
> of everything in memory and will do transactional operations that can be
> rolled back if there is a hardware glitch.
>
>> So, some pages of RAM are going to be read-only and the data in those
>> pages came from some source (file system?). � Can anyone describe a
>> high level strategy to occasionaly provide some coverage of this data?
>
>> So far I have thought about page descriptors adding an MD5 hash
>> whenever they are read-only and first being "loaded/mapped?" and then
>> a background daemon could occasionaly verify.
>
> Makes sense to me. �You might also pick an on-disk format with extra
> checksumming so you could compare the on-disk checksum with the
> in-memory checksum.
>
> Chris
>
> --
> The author works for GENBAND Corporation (GENBAND) who is solely
> responsible for this email and its contents. All enquiries regarding
> this email should be addressed to GENBAND. Nortel has provided the use
> of the nortel.com domain to GENBAND in connection with this email solely
> for the purpose of connectivity and Nortel Networks Inc. has no
> liability for the email or its contents. GENBAND's web site is
> http://www.genband.com
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Massimiliano Galanti on

> What about .ro and .text sections of an executable? I would think
> kernel support for that would be required. If its application data,
> then all sorts of things are possible like you described. Ive also
> seen critical ram variables be stored in triplicate and then
> compared/voted just to ensure no silent SEU corruption.

Maybe slightly off topic but... if flash is safer than ram, what about
using XIP (where possible, e.g. on NORs)? That would not put .data
sections into ram, at least.

--
Massimiliano
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Brian Gordon on
Sorry, I take it back. This wont work for me because I wont have
NOR. Also, I only want the "in-place" to apply to read-only pages.
This looks like all reads and writes get passed to the underlying
storage and I can't suffer flash page erase/writes to update a
variable. :) The device will wear out and meaningful work would be
starved.

On Thu, Jun 10, 2010 at 1:37 PM, Brian Gordon <legerde(a)gmail.com> wrote:
> Yes.  Thats exactly what I am looking for.   Even if there is a speed
> penalty, I wouldn't mind so much.  However wikipedia says that XIP is
> filesystem dependent and im stuck with FAT32 or NTFS.   Wikipedia
> claims NTFS can do XIP.   Is this true under linux?
>
> On Thu, Jun 10, 2010 at 1:23 PM, Massimiliano Galanti
> <massiblue(a)libero.it> wrote:
>>
>>> What about .ro and .text sections of an executable?   I would think
>>> kernel support for that would be required.   If its application data,
>>> then all sorts of things are possible like you described.   Ive also
>>> seen critical ram variables be stored in triplicate and then
>>> compared/voted just to ensure no silent SEU corruption.
>>
>> Maybe slightly off topic but... if flash is safer than ram, what about using
>> XIP (where possible, e.g. on NORs)? That would not put .data sections into
>> ram, at least.
>>
>> --
>> Massimiliano
>>
>
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/