From: Manfred Spraul on
Hi Luca,

On 06/26/2010 02:52 PM, Luca Tettamanti wrote:
> They don't seem really hung as before, I see two different behaviours:
> * Near the end of the run ab is frozen for a few seconds, but in the
> end all requests are processed; however I see a few "length" errors,
> meaning that the received page does not match the expected content
> (I'm testing a static page):
>
>
That's consistent with what I see:
If I run:
#./semtimedop 100 100&
#./semtimedop 100 100&
#./semtimedop 100 100&
#./semtimedop 100 100&

(i.e.: 4 times the attached test app concurrently), then the system
sometimes locks up for 10..20 seconds:
The keyboard is unresponsive, not even the numlock key is processed
(i.e.: the LED does not change anymore).
After 10 or 20 seconds, the keyboard reacts again (both to <enter> and
to Num Lock)
The stock Fedora 13 kernel (2.6.33.5) does not exhibit this behavior
The load average is 300 or so, that's expected.

I have no idea why and how to debug the behavior.
# CONFIG_PREEMPT_NONE is not set
CONFIG_PREEMPT_VOLUNTARY=y
# CONFIG_PREEMPT is not set
> strace on apache shows:
> [pid 3787] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> [pid 3789] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> [pid 3788] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> [pid 3784] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> [pid 3783] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> [pid 3782] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> [pid 3239] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> [pid 3233] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> [pid 3238] restart_syscall(<... resuming interrupted call ...> <unfinished ...>
> [pid 3237] restart_syscall(<... resuming interrupted call ...>
>

That can't be semop:
sysv ipc and msg are among the (broken) parts of the kernel that do not
honor SA_RESTART.

--
Manfred


From: Luca Tettamanti on
On Sat, Jun 26, 2010 at 5:47 PM, Manfred Spraul
<manfred(a)colorfullife.com> wrote:
> Hi Luca,
>
> On 06/26/2010 02:52 PM, Luca Tettamanti wrote:
>>
>> They don't seem really hung as before, I see two different behaviours:
>> * Near the end of the run ab is frozen for a few seconds, but in the
>> end all requests are processed; however I see a few "length" errors,
>> meaning that the received page does not match the expected content
>> (I'm testing a static page):
>>
>>
>
> That's consistent with what I see:
> If I run:
> #./semtimedop 100 100&
> #./semtimedop 100 100&
> #./semtimedop 100 100&
> #./semtimedop 100 100&
>
> (i.e.: 4 times the attached test app concurrently), then the system
> sometimes locks up for 10..20 seconds:
> The keyboard is unresponsive, not even the numlock key is processed (i.e.:
> the LED does not change anymore).
> After 10 or 20 seconds, the keyboard reacts again (both to <enter> and to
> Num Lock)
> The stock Fedora 13 kernel (2.6.33.5) does not exhibit this behavior
> The load average is 300 or so, that's expected.

Confirmed here: your test program freezes the system for a while under
2.6.35-rc3, while vanilla 2.6.34 copes fine.
sysrq-t was responsive during the freeze, so I took a snapshot during
it, file is attached.

> I have no idea why and how to debug the behavior.
> # CONFIG_PREEMPT_NONE is not set
> CONFIG_PREEMPT_VOLUNTARY=y
> # CONFIG_PREEMPT is not set

My kernel has PREEMPT enabled.

Luca
From: Manfred Spraul on
Hi Luca,

On 06/26/2010 06:47 PM, Luca Tettamanti wrote:
>
> Confirmed here: your test program freezes the system for a while under
> 2.6.35-rc3, while vanilla 2.6.34 copes fine.
> sysrq-t was responsive during the freeze, so I took a snapshot during
> it, file is attached.
>
>
Ignore my test program:
If the master thread is interrupted in the right place, then there are
400 runnable tasks in the runqueue.
It seems that the scheduler just processes these 400 tasks first instead
of the keventd/ksoftirqd that is necessary for the keyboard handling.

Attached is a new idea, could you try it with your httpd test?

Perhaps the race is actually a race in the user space:
The exit path of semtimedop() does not contain an explicit memory barrier.
For the kernel, it does not matter: It merely reads one integer value.
If sysret is also no memory barrier, then user space might observe stale
data.

Which cpu do you have? I was unable to show any misbehavior on a Phenom X4.

--
Manfred
From: Luca Tettamanti on
On Wed, Jun 30, 2010 at 9:07 PM, Manfred Spraul
<manfred(a)colorfullife.com> wrote:
> Attached is a new idea, could you try it with your httpd test?

Will test ASAP.

> Perhaps the race is actually a race in the user space:
> The exit path of semtimedop() does not contain an explicit memory barrier.
> For the kernel, it does not matter: It merely reads one integer value.
> If sysret is also no memory barrier, then user space might observe stale
> data.
>
> Which cpu do you have? I was unable to show any misbehavior on a Phenom X4.

Intel(R) Core(TM)2 Duo CPU T7500 @ 2.20GHz

I also have a Phenom X4, but I'm currently waiting a replacement for the PSU...

Luca
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/
From: Luca Tettamanti on
On Wed, Jun 30, 2010 at 9:07 PM, Manfred Spraul
<manfred(a)colorfullife.com> wrote:
> Attached is a new idea, could you try it with your httpd test?

With kernel 2.6.35-rc3 your patch does not make any difference.
2.6.35-rc4, however, works fine with either one of your patches (yes,
I've checked multiple times).

Luca
--
To unsubscribe from this list: send the line "unsubscribe linux-kernel" in
the body of a message to majordomo(a)vger.kernel.org
More majordomo info at http://vger.kernel.org/majordomo-info.html
Please read the FAQ at http://www.tux.org/lkml/