Prev: Calling library functions in Linux kernel code
Next: Fail-over: how data can be migrated from the failed server?
From: Ersek, Laszlo on 22 Jul 2010 17:23
On Thu, 22 Jul 2010, Scott Lurndal wrote:

> "Ersek, Laszlo" <lacos(a)caesar.elte.hu> writes:

>> How does one guarantee the correct alignment of mutex objects in shared
>> memory, portably?
>
> The processor ABI requires objects to be allocated at specified
> alignments.
>
> The mutex object is generally specified as a struct type, which IIRC is
> required to be aligned at a boundary consistent with the largest
> alignment required by any member (typically an 8-byte boundary for the
> x86_64 ABI).
>
> Therefore, the compiler will _align_ the mutex object at the correct
> alignment boundary.

Yes, when it is left up to the compiler (at compile time) to allocate
(align) a mutex object, ie. to generate "well-informed" code that will do
the space reservation at runtime. Like the allocation of objects with
static storage duration or automatic storage duration. Dynamic allocation
is different.

malloc() comes with a guarantee [0]:

"The pointer returned if the allocation succeeds shall be suitably aligned
so that it may be assigned to a pointer to any type of object and then
used to access such an object in the space allocated (until the space is
explicitly freed or reallocated)."

However, I can't seem to find such a guarantee for shmat() [1] or mmap()
[2]. In practice, such segments are probably aligned at page boundaries,
which are orders of (binary) magnitude coarser than any struct would
require. Nonetheless, this is not spelled out in the Standard. Thus I'm
not sure one can do this:

#define _XOPEN_SOURCE 700
#include <pthread.h>
#include <sys/shm.h>

struct scratchpad
{
pthread_mutex_t mutex;
/* ... */
};

int
main(void)
{
void *addr;

/* ... */
addr = shmat(shmid, 0, 0);
if ((void *)-1 != addr) {
pthread_mutexattr_t attr;

/* ... */
(void)pthread_mutex_init(&((struct scratchpad *)addr)->mutex, &attr);

/*
or, equivalently:
(void)pthread_mutex_init(addr, &attr);
*/

/* ... */
}
/* ... */
}

Because theoretically, shmat() could return anything.


I sense a vague possibility that one could allocate an auto or static
struct and use its address as an input parameter to shmat(), but I have no
idea how that could be used in a portable and reliable way:

- don't remap anything "before" or "after" the struct intended to be
shared (due to the shared segment being bigger [3] than sizeof(struct
scratchpad), and its base perhaps even rounded down [1]),

- make sure that shmat() [1] never returns with ((void *)-1) / EINVAL on
any XSI platform, for "shmaddr" being an invalid address for attaching
shared memory (wrt. the given "shmflg").


> The developers of the pthread library and processor ABI also take care
> to ensure that any field in the structure which may be the target of a
> lock prefix (on Intel/AMD architectures) never crosses a cache line
> boundary (a split lock).
>
> Split-locks are bad, bad, bad since they generate global bus locks (a
> lock prefix on a memory reference contained entirely within a single
> 64-byte cache line simply requires exclusive access by the cache
> coherency protocol and doesn't need to stop the entire system with a
> global bus lock; the same locked reference to a memory access that
> spans two cache lines requires a system-wide _global_ bus lock that
> serializes _all_ memory references in the system).

That's very educational, thanks.

lacos

[0] http://www.opengroup.org/onlinepubs/9699919799/functions/malloc.html
[1] http://www.opengroup.org/onlinepubs/9699919799/functions/shmat.html
[2] http://www.opengroup.org/onlinepubs/9699919799/functions/mmap.html
[3] http://www.opengroup.org/onlinepubs/9699919799/functions/shmget.html
From: Scott Lurndal on 22 Jul 2010 20:00
"Ersek, Laszlo" <lacos(a)caesar.elte.hu> writes:
>On Thu, 22 Jul 2010, Scott Lurndal wrote:

>> Therefore, the compiler will _align_ the mutex object at the correct
>> alignment boundary.
>
>Yes, when it is left up to the compiler (at compile time) to allocate
>(align) a mutex object, ie. to generate "well-informed" code that will do
>the space reservation at runtime. Like the allocation of objects with
>static storage duration or automatic storage duration. Dynamic allocation
>is different.

Good point.

>
>malloc() comes with a guarantee [0]:
>
>"The pointer returned if the allocation succeeds shall be suitably aligned
>so that it may be assigned to a pointer to any type of object and then
>used to access such an object in the space allocated (until the space is
>explicitly freed or reallocated)."
>
>However, I can't seem to find such a guarantee for shmat() [1] or mmap()
>[2]. In practice, such segments are probably aligned at page boundaries,
>which are orders of (binary) magnitude coarser than any struct would
>require. Nonetheless, this is not spelled out in the Standard. Thus I'm
>not sure one can do this:

Clean code could, if alignment mattered and wasn't guaranteed by the A[BP]I:

void *x = shmat(...);
(unsigned char *)x += (sizeof(unsigned long) - 1);
(unsighed char *)x &= ~(sizeof(unsigned long) - 1);

now x is properly aligned; if it was previously aligned, this
will be a no-op. I've seen similar code used in ancient production software
to ensure the return from shmat() is page aligned.

This should be portable to most non-BCD architectures.

That said, I think the assumption that mmap and shmat return ABI-aligned
regions is one that portable software can safely make, even without a
guarantee from the standards.

I don't recall, from my time on the X/Open committee, whether we discussed
the alignment requirements for mmap and shmat(); but that was over a decade ago
and has been pushed from the memory banks. Note that we were always careful
not to preclude oddball architectures, or require things like MMU's.

scott
From: Ersek, Laszlo on 23 Jul 2010 01:34
On Fri, 23 Jul 2010, Scott Lurndal wrote:

> That said, I think the assumption that mmap and shmat return ABI-aligned
> regions is one that portable software can safely make, even without a
> guarantee from the standards.
>
> I don't recall, from my time on the X/Open committee, whether we
> discussed the alignment requirements for mmap and shmat(); but that was
> over a decade ago and has been pushed from the memory banks. Note that
> we were always careful not to preclude oddball architectures, or require
> things like MMU's.

Thank you.
lacos
From: Natarajan Krishnaswami on 24 Jul 2010 11:22
On 2010-07-20, john <john(a)replyatnewsgroup.com> wrote:
> Anyway, I've followed the guidelines on using mutexes in shared memory:
>
> pthread_mutexattr_t hAttr;
[...]
> pthread_mutexattr_setpshared(&hAttr, PTHREAD_PROCESS_SHARED)
> pthread_mutex_init((pthread_mutex_t*)m_hMutex, NULL);
^^^^
Your mutex isn't process shared. Should be &hAttr there.

N.
First  |  Prev  | 
Pages: 1 2 3
Prev: Calling library functions in Linux kernel code
Next: Fail-over: how data can be migrated from the failed server?