futex: race in lock and unlock&exit for robust futex with PI?
in [Kernel]
|
Prev: [PATCH] s6e63m0: set permissions on gamma_table file to 0444
Next: [2.6.33.4 PATCH] V4L/uvcvideo: Add support for Suyin Corp. Lenovo Webcam
From: Michal Hocko on 23 Jun 2010 05:20 Hi, attached you can find a simple test case which fails quite easily on the following glibc assert: "SharedMutexTest: pthread_mutex_lock.c:289: __pthread_mutex_lock: Assertion `(-(e)) != 3 || !robust' failed." " AFAIU, this assertion says that futex syscall cannot fail with ESRCH for robust futex because it should either succeed or fail with EOWNERDEAD. We have seen this problem on SLES11 and SLES11SP1 but I was able to reproduce it with the 2.6.34 kernel as well. The test case is quite easy. Executed with a parameter it creates a test file and initializes shared, robust pthread mutex (optionaly compile time configured with priority inheritance) backed by the mmapped test file. Without a parameter it mmaps the file and just locks, unlocks mutex and checks for EOWNERDEAD (this should never happen during the test as the process never dies with the lock held) in the loop. If I run this application for multiple users in parallel I can see the above assertion. However, if priority inheritance is turned off then there is no problem. I am not able to reproduce also if the test case is run under a single user. I am using the attached runSimple.sh script to run the test case like this: rm test.file simple for i in `seq 10` do sh runSimple.sh done To disable IP just comment out USE_PI variable in the script. You need to change USER1 and USER2 variables to match you system. You will need to run the script as root if you do not set any special setting to run su on behalf of those users. I have tried to look at futex_{un}lock_pi but it is really hard to understand. I assume that lookup_pi_state is the one which sets ESRCH after it is not able to find the pid of the current owner. This would suggest that we are racing with the unlock of the current lock holder but I don't see how is this possible as both lock and unlock paths hold fshared lock for all operations over the lock value. I have noticed that the lock path drops fshared if the current holder is dying but then it retries the whole process again. Any advice would be highly appreciated. Let me know if you need any further information Thanks -- Michal Hocko L3 team SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic
From: Darren Hart on 24 Jun 2010 22:50 On 06/23/2010 02:13 AM, Michal Hocko wrote: > Hi, Hi Michal, Thanks for reporting the issue and providing a testcase. > > attached you can find a simple test case which fails quite easily on the > following glibc assert: > "SharedMutexTest: pthread_mutex_lock.c:289: __pthread_mutex_lock: > Assertion `(-(e)) != 3 || !robust' failed." " I've run runSimple.sh in a tight loop for a couple hours (about 2k iterations so far) and haven't seen anything other than "Here we go" printed to the console. I had to add -D_GNU_SOURCE to get it to build on my system (RHEL5.2 + 2.6.34). Perhaps this is just a difference in the toolchain. > AFAIU, this assertion says that futex syscall cannot fail with ESRCH > for robust futex because it should either succeed or fail with > EOWNERDEAD. I'll have to think on that and review the libc source. We do need to confirm that the assert is even doing the right thing. > > We have seen this problem on SLES11 and SLES11SP1 but I was able to > reproduce it with the 2.6.34 kernel as well. What kind of system are you seeing this on? I've been running on a 4-way x86_64 blade. > The test case is quite easy. > > Executed with a parameter it creates a test file and initializes shared, > robust pthread mutex (optionaly compile time configured with priority > inheritance) backed by the mmapped test file. Without a parameter it > mmaps the file and just locks, unlocks mutex and checks for EOWNERDEAD > (this should never happen during the test as the process never dies with > the lock held) in the loop. Have you found the PI parameter to be required for reproducing the error? From the comments below I'm assuming so... just want to be sure. > > If I run this application for multiple users in parallel I can see the > above assertion. However, if priority inheritance is turned off then > there is no problem. I am not able to reproduce also if the test case is > run under a single user. > > I am using the attached runSimple.sh script to run the test case like > this: > > rm test.file simple > for i in `seq 10` > do > sh runSimple.sh > done > > To disable IP just comment out USE_PI variable in the script. > You need to change USER1 and USER2 variables to match you system. You > will need to run the script as root if you do not set any special > setting to run su on behalf of those users. > > I have tried to look at futex_{un}lock_pi but it is really hard to > understand. *grin* tell me about it... See Documentation/pi-futex.txt if you haven't already. > I assume that lookup_pi_state is the one which sets ESRCH > after it is not able to find the pid of the current owner. > > This would suggest that we are racing with the unlock of the current > lock holder but I don't see how is this possible as both lock and unlock > paths hold fshared lock for all operations over the lock value. I have > noticed that the lock path drops fshared if the current holder is dying > but then it retries the whole process again. > > Any advice would be highly appreciated. If I can reproduce this I should be able to get some trace points in there to get a better idea of the execution path leading up to the problem. This would be a great time to have those futex fault injection patches... -- Darren Hart IBM Linux Technology Center Real-Time Linux Team -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Michal Hocko on 25 Jun 2010 04:30 On Thu 24-06-10 19:42:50, Darren Hart wrote: > On 06/23/2010 02:13 AM, Michal Hocko wrote: > >Hi, Hi, > > Hi Michal, > > Thanks for reporting the issue and providing a testcase. > > > > >attached you can find a simple test case which fails quite easily on the > >following glibc assert: > >"SharedMutexTest: pthread_mutex_lock.c:289: __pthread_mutex_lock: > > Assertion `(-(e)) != 3 || !robust' failed." " > > I've run runSimple.sh in a tight loop for a couple hours (about 2k > iterations so far) and haven't seen anything other than "Here we go" > printed to the console. Maybe a higher load on CPUs would help (busy loop on other CPUs). > > I had to add -D_GNU_SOURCE to get it to build on my system (RHEL5.2 > + 2.6.34). Perhaps this is just a difference in the toolchain. I assume that you got PTHREAD_PRIO_INHERIT undeclared error, don't you? I have hacked around that by #define __USE_UNIX98 which worked on Debian and OpenSuse. But you are right _GNU_SOURCE is definitely better solution. > > >AFAIU, this assertion says that futex syscall cannot fail with ESRCH > >for robust futex because it should either succeed or fail with > >EOWNERDEAD. > > I'll have to think on that and review the libc source. We do need to > confirm that the assert is even doing the right thing. Sure. I have looked through the glibc lock implementation and it makes quite a good sense to me. A robust lock should never return with ESRCH. > > > > >We have seen this problem on SLES11 and SLES11SP1 but I was able to > >reproduce it with the 2.6.34 kernel as well. > > What kind of system are you seeing this on? I've been running on a > 4-way x86_64 blade. * Debian (squeeze/sid) with - Intel(R) Core(TM)2 CPU T5600 @ 1.83GHz - kernel: vanilla 2.6.34 - glibc: 2.11.1-3 - i386 * OpenSuse 11.2 with - Intel(R) Core(TM)2 Duo CPU E4500 @ 2.20GHz - kernel: distribution 2.6.31.12-0.2-desktop - glibc: 2.10.1-10.5.1 - i386 * SLES11SP1 - Dual-Core AMD Opteron(tm) Processor 1218 - kernel: distribution 2.6.32.12-0.3-default - glibc: 2.11.1-0.17.4 - x86_64 Each box shows a different number of asserts during 10 iterations. > > >The test case is quite easy. > > > >Executed with a parameter it creates a test file and initializes shared, > >robust pthread mutex (optionaly compile time configured with priority > >inheritance) backed by the mmapped test file. Without a parameter it > >mmaps the file and just locks, unlocks mutex and checks for EOWNERDEAD > >(this should never happen during the test as the process never dies with > >the lock held) in the loop. > > Have you found the PI parameter to be required for reproducing the > error? From the comments below I'm assuming so... just want to be > sure. Yes. If you comment out USE_PI variable in the script the problem is not shown at all. > > > > >If I run this application for multiple users in parallel I can see the > >above assertion. However, if priority inheritance is turned off then > >there is no problem. I am not able to reproduce also if the test case is > >run under a single user. > > > >I am using the attached runSimple.sh script to run the test case like > >this: > > > >rm test.file simple > >for i in `seq 10` > >do > > sh runSimple.sh > >done > > > >To disable IP just comment out USE_PI variable in the script. > >You need to change USER1 and USER2 variables to match you system. You > >will need to run the script as root if you do not set any special > >setting to run su on behalf of those users. > > > >I have tried to look at futex_{un}lock_pi but it is really hard to > >understand. > > *grin* tell me about it... > > See Documentation/pi-futex.txt if you haven't already. Will do. > > >I assume that lookup_pi_state is the one which sets ESRCH > >after it is not able to find the pid of the current owner. > > > >This would suggest that we are racing with the unlock of the current > >lock holder but I don't see how is this possible as both lock and unlock > >paths hold fshared lock for all operations over the lock value. I have > >noticed that the lock path drops fshared if the current holder is dying > >but then it retries the whole process again. > > > >Any advice would be highly appreciated. > > If I can reproduce this I should be able to get some trace points in > there to get a better idea of the execution path leading up to the > problem. Please make sure that you run the test case with two different users. I couldn't reproduce the issue with a single user. If you have some ideas about patches which I could try then just pass it to me. > > This would be a great time to have those futex fault injection patches... > > > -- > Darren Hart > IBM Linux Technology Center > Real-Time Linux Team Thanks for looking into it. -- Michal Hocko L3 team SUSE LINUX s.r.o. Lihovarska 1060/12 190 00 Praha 9 Czech Republic -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Darren Hart on 25 Jun 2010 14:00 On 06/25/2010 01:27 AM, Michal Hocko wrote: > On Thu 24-06-10 19:42:50, Darren Hart wrote: >> On 06/23/2010 02:13 AM, Michal Hocko wrote: >>> Hi, > > Hi, > >> >> Hi Michal, >> >> Thanks for reporting the issue and providing a testcase. >> >>> >>> attached you can find a simple test case which fails quite easily on the >>> following glibc assert: >>> "SharedMutexTest: pthread_mutex_lock.c:289: __pthread_mutex_lock: >>> Assertion `(-(e)) != 3 || !robust' failed." " >> >> I've run runSimple.sh in a tight loop for a couple hours (about 2k >> iterations so far) and haven't seen anything other than "Here we go" >> printed to the console. > > Maybe a higher load on CPUs would help (busy loop on other CPUs). Must have been a build issue. I can reproduce _something_ now. Within 10 iterations of runSimple.sh the test hangs. ps shows all the simple processes sitting in pause. (gdb) bt #0 0x0000003c0060e030 in __pause_nocancel () from /lib64/libpthread.so.0 #1 0x0000003c006085fc in __pthread_mutex_lock_full () from /lib64/libpthread.so.0 #2 0x0000000000400cd6 in main (argc=1, argv=0x7fffc016e508) at simple.c:101 There is only one call to pause* in pthread_mutex_lock.c: (line ~316): /* ESRCH can happen only for non-robust PI mutexes where the owner of the lock died. */ assert (INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust); /* Delay the thread indefinitely. */ while (1) pause_not_cancel (); Right now I'm thinking that NDEBUG is set in my build for whatever reason, but I think I'm seeing the same issue you are. I'll review the futex code and prepare a trace patch and see if I can reproduce with that. Note: confirmed, the glibc rpm has -DNDEBUG=1 -- Darren > >> >> I had to add -D_GNU_SOURCE to get it to build on my system (RHEL5.2 >> + 2.6.34). Perhaps this is just a difference in the toolchain. > > I assume that you got PTHREAD_PRIO_INHERIT undeclared error, don't you? > I have hacked around that by #define __USE_UNIX98 which worked on Debian > and OpenSuse. But you are right _GNU_SOURCE is definitely better > solution. > >> >>> AFAIU, this assertion says that futex syscall cannot fail with ESRCH >>> for robust futex because it should either succeed or fail with >>> EOWNERDEAD. >> >> I'll have to think on that and review the libc source. We do need to >> confirm that the assert is even doing the right thing. > > Sure. I have looked through the glibc lock implementation and it makes > quite a good sense to me. A robust lock should never return with ESRCH. > >> >>> >>> We have seen this problem on SLES11 and SLES11SP1 but I was able to >>> reproduce it with the 2.6.34 kernel as well. >> >> What kind of system are you seeing this on? I've been running on a >> 4-way x86_64 blade. > > * Debian (squeeze/sid) with > - Intel(R) Core(TM)2 CPU T5600 @ 1.83GHz > - kernel: vanilla 2.6.34 > - glibc: 2.11.1-3 > - i386 > > * OpenSuse 11.2 with > - Intel(R) Core(TM)2 Duo CPU E4500 @ 2.20GHz > - kernel: distribution 2.6.31.12-0.2-desktop > - glibc: 2.10.1-10.5.1 > - i386 > > * SLES11SP1 > - Dual-Core AMD Opteron(tm) Processor 1218 > - kernel: distribution 2.6.32.12-0.3-default > - glibc: 2.11.1-0.17.4 > - x86_64 > > Each box shows a different number of asserts during 10 iterations. > >> >>> The test case is quite easy. >>> >>> Executed with a parameter it creates a test file and initializes shared, >>> robust pthread mutex (optionaly compile time configured with priority >>> inheritance) backed by the mmapped test file. Without a parameter it >>> mmaps the file and just locks, unlocks mutex and checks for EOWNERDEAD >>> (this should never happen during the test as the process never dies with >>> the lock held) in the loop. >> >> Have you found the PI parameter to be required for reproducing the >> error? From the comments below I'm assuming so... just want to be >> sure. > > Yes. If you comment out USE_PI variable in the script the problem is not > shown at all. > >> >>> >>> If I run this application for multiple users in parallel I can see the >>> above assertion. However, if priority inheritance is turned off then >>> there is no problem. I am not able to reproduce also if the test case is >>> run under a single user. >>> >>> I am using the attached runSimple.sh script to run the test case like >>> this: >>> >>> rm test.file simple >>> for i in `seq 10` >>> do >>> sh runSimple.sh >>> done >>> >>> To disable IP just comment out USE_PI variable in the script. >>> You need to change USER1 and USER2 variables to match you system. You >>> will need to run the script as root if you do not set any special >>> setting to run su on behalf of those users. >>> >>> I have tried to look at futex_{un}lock_pi but it is really hard to >>> understand. >> >> *grin* tell me about it... >> >> See Documentation/pi-futex.txt if you haven't already. > > Will do. > >> >>> I assume that lookup_pi_state is the one which sets ESRCH >>> after it is not able to find the pid of the current owner. >>> >>> This would suggest that we are racing with the unlock of the current >>> lock holder but I don't see how is this possible as both lock and unlock >>> paths hold fshared lock for all operations over the lock value. I have >>> noticed that the lock path drops fshared if the current holder is dying >>> but then it retries the whole process again. >>> >>> Any advice would be highly appreciated. >> >> If I can reproduce this I should be able to get some trace points in >> there to get a better idea of the execution path leading up to the >> problem. > > Please make sure that you run the test case with two different users. I > couldn't reproduce the issue with a single user. > > If you have some ideas about patches which I could try then just pass it > to me. > >> >> This would be a great time to have those futex fault injection patches... >> >> >> -- >> Darren Hart >> IBM Linux Technology Center >> Real-Time Linux Team > > Thanks for looking into it. -- Darren Hart IBM Linux Technology Center Real-Time Linux Team -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/
From: Darren Hart on 25 Jun 2010 19:40
On 06/25/2010 10:53 AM, Darren Hart wrote: > On 06/25/2010 01:27 AM, Michal Hocko wrote: >> On Thu 24-06-10 19:42:50, Darren Hart wrote: >>> On 06/23/2010 02:13 AM, Michal Hocko wrote: >>>> attached you can find a simple test case which fails quite easily on >>>> the >>>> following glibc assert: >>>> "SharedMutexTest: pthread_mutex_lock.c:289: __pthread_mutex_lock: >>>> Assertion `(-(e)) != 3 || !robust' failed." " >>> >>> I've run runSimple.sh in a tight loop for a couple hours (about 2k >>> iterations so far) and haven't seen anything other than "Here we go" >>> printed to the console. >> >> Maybe a higher load on CPUs would help (busy loop on other CPUs). > > Must have been a build issue. I can reproduce _something_ now. Within 10 > iterations of runSimple.sh the test hangs. ps shows all the simple > processes sitting in pause. > > (gdb) bt > #0 0x0000003c0060e030 in __pause_nocancel () from /lib64/libpthread.so.0 > #1 0x0000003c006085fc in __pthread_mutex_lock_full () > from /lib64/libpthread.so.0 > #2 0x0000000000400cd6 in main (argc=1, argv=0x7fffc016e508) at simple.c:101 > > There is only one call to pause* in pthread_mutex_lock.c: (line ~316): > > /* ESRCH can happen only for non-robust PI mutexes where > the owner of the lock died. */ > assert (INTERNAL_SYSCALL_ERRNO (e, __err) != ESRCH || !robust); > > /* Delay the thread indefinitely. */ > while (1) > pause_not_cancel (); > > Right now I'm thinking that NDEBUG is set in my build for whatever > reason, but I think I'm seeing the same issue you are. I'll review the > futex code and prepare a trace patch and see if I can reproduce with that. > > Note: confirmed, the glibc rpm has -DNDEBUG=1 The simple tracing patch (below) confirms that we are indeed returning -ESRCH to userspace from futex_lock_pi(). Notice that the pids of the two "simple" processes lingering after the runSimple.sh script are the ones that return -ESRCH to userspace, and therefor end up in the pause_not_cancel() trap inside glibc. # trace-cmd record -p nop ./runSimple.sh <snip> # ps -eLo pid,comm,wchan | grep "simple " 20636 simple pause 20876 simple pause # trace-cmd report version = 6 CPU 0 is empty cpus=4 field->offset = 24 size=8 <...>-20636 [003] 1778.965860: bprint: futex_lock_pi_atomic : lookup_pi_state: -ESRCH <...>-20636 [003] 1778.965865: bprint: futex_lock_pi_atomic : ownerdied not detected, returning -ESRCH <...>-20636 [003] 1778.965866: bprint: futex_lock_pi_atomic : lookup_pi_state: -3 >>---> <...>-20636 [003] 1778.965867: bprint: futex_lock_pi : returning -ESRCH to userspace <...>-20876 [001] 1780.199394: bprint: futex_lock_pi_atomic : cmpxchg failed, retrying <...>-20876 [001] 1780.199400: bprint: futex_lock_pi_atomic : lookup_pi_state: -ESRCH <...>-20876 [001] 1780.199401: bprint: futex_lock_pi_atomic : ownerdied not detected, returning -ESRCH <...>-20876 [001] 1780.199402: bprint: futex_lock_pi_atomic : lookup_pi_state: -3 >>---> <...>-20876 [001] 1780.199403: bprint: futex_lock_pi : returning -ESRCH to userspace <...>-21316 [002] 1782.300695: bprint: futex_lock_pi_atomic : cmpxchg failed, retrying <...>-21316 [002] 1782.300698: bprint: futex_lock_pi_atomic : cmpxchg failed, retrying Attaching gdb to 20636, we can see the state of the mutex: (gdb) print (struct __pthread_mutex_s)*mutex $1 = {__lock = 0, __count = 1, __owner = 0, __nusers = 0, __kind = 176, __spins = 0, __list = {__prev = 0x0, __next = 0x0}} This is consistent with hex dump of the first bits of the backing file: # xxd test.file | head -n 3 0000000: 0000 0000 0100 0000 0000 0000 0000 0000 ................ 0000010: b000 0000 0000 0000 0000 0000 0000 0000 ................ 0000020: 0000 0000 0000 0000 0000 0000 0000 0000 ................ The futex (__lock) value is 0, indicating it is unlocked and has no waiters. The count being 1 however suggests a task has acquired it once, which, if I read the glibc source correctly, means the owner field and __lock fields should not be 0. This supports Michal's thought about lock racing with unlock, seeing it's held, then unable to find the owner (pi_state) as it has since been unlocked. Possibly some horkage with the WAITERS bit leading to glibc performing atomic acquistions/releases and rendering the mutex inconsistent with the kernel's view. This should be protected against, but that is the direction I am going to start looking. -- Darren Hart From 92014a07df73489460ff788274506255ff0f775d Mon Sep 17 00:00:00 2001 From: Darren Hart <dvhltc(a)us.ibm.com> Date: Fri, 25 Jun 2010 13:54:25 -0700 Subject: [PATCH] robust pi futex tracing --- kernel/futex.c | 24 ++++++++++++++++++++---- 1 files changed, 20 insertions(+), 4 deletions(-) diff --git a/kernel/futex.c b/kernel/futex.c index e7a35f1..24ac437 100644 --- a/kernel/futex.c +++ b/kernel/futex.c @@ -683,6 +683,8 @@ retry: */ if (unlikely(ownerdied || !(curval & FUTEX_TID_MASK))) { /* Keep the OWNER_DIED bit */ + if (ownerdied) + trace_printk("ownerdied, taking over lock\n"); newval = (curval & ~FUTEX_TID_MASK) | task_pid_vnr(task); ownerdied = 0; lock_taken = 1; @@ -692,14 +694,18 @@ retry: if (unlikely(curval == -EFAULT)) return -EFAULT; - if (unlikely(curval != uval)) + if (unlikely(curval != uval)) { + trace_printk("cmpxchg failed, retrying\n"); goto retry; + } /* * We took the lock due to owner died take over. */ - if (unlikely(lock_taken)) + if (unlikely(lock_taken)) { + trace_printk("ownerdied, lock acquired, return 1\n"); return 1; + } /* * We dont have the lock. Look up the PI state (or create it if @@ -710,13 +716,16 @@ retry: if (unlikely(ret)) { switch (ret) { case -ESRCH: + trace_printk("lookup_pi_state: -ESRCH\n"); /* * No owner found for this futex. Check if the * OWNER_DIED bit is set to figure out whether * this is a robust futex or not. */ - if (get_futex_value_locked(&curval, uaddr)) + if (get_futex_value_locked(&curval, uaddr)) { + trace_printk("get_futex_value_locked: -EFAULT\n"); return -EFAULT; + } /* * We simply start over in case of a robust @@ -724,10 +733,13 @@ retry: * and return happy. */ if (curval & FUTEX_OWNER_DIED) { + trace_printk("ownerdied, goto retry\n"); ownerdied = 1; goto retry; } + trace_printk("ownerdied not detected, returning -ESRCH\n"); default: + trace_printk("lookup_pi_state: %d\n", ret); break; } } @@ -1950,6 +1962,8 @@ retry_private: put_futex_key(fshared, &q.key); cond_resched(); goto retry; + case -ESRCH: + trace_printk("returning -ESRCH to userspace\n"); default: goto out_unlock_put_key; } @@ -2537,8 +2551,10 @@ void exit_robust_list(struct task_struct *curr) /* * Avoid excessively long or circular lists: */ - if (!--limit) + if (!--limit) { + trace_printk("excessively long list, aborting\n"); break; + } cond_resched(); } -- 1.7.0.4 -- Darren Hart IBM Linux Technology Center Real-Time Linux Team -- To unsubscribe from this list: send the line "unsubscribe linux-kernel" in the body of a message to majordomo(a)vger.kernel.org More majordomo info at http://vger.kernel.org/majordomo-info.html Please read the FAQ at http://www.tux.org/lkml/ |