Prev: DSOFile.dll
Next: IIS 7 - Cannot launch out of process component?
From: _richard on 9 Jan 2009 08:49
I have an intermittent problem with a web site hosted on IIS6 (w2k3 sp2).

I appears to occur randomly to users when they click on a hyperlink within a
page. The request is sent to the web server but a response is never
returned. If the user tries to navigate to another hyperlink they are not
able to (i.e. the web site appears to hang for that user). Other users of
the website at the time are not affected by this hang and if the user with
the problem opens a new http session (closing IE and opening the web site
again) they no long experience the hang.

I've placed a debugger (IISState) on the w3wp process with the following
output. Entries with "Thread is waiting for a lock to be released. Looking
for lock owner." look like they might be causing the issue. Can anyone tell
what lock the process is waiting on?

Thanks

------------------------------------------------------
Opened log file 'E:\Temp\__Richard\IISState2\output\IISState-3620.log'

***********************
Starting new log output
IISState version 3.3.1

Fri Jan 09 13:21:01 2009

OS = Windows 2003 Server
Executable: w3wp.exe
PID = 3620

Note: Thread times are formatted as HH:MM:SS.ms

***********************


IIS has crashed...
Beginning Analysis
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
E:\App\Oracle\Product\1020\bin\OraClient10.Dll -
DLL (!FunctionName) that failed: OraClient10!kpufhndl0




Thread ID: 17
System Thread ID: b00
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
*** WARNING: Unable to verify checksum for
E:\App\Oracle\Product\1020\bin\oci.dll
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
E:\App\Oracle\Product\1020\bin\oci.dll -
Thread Type: Other
# ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be
wrong.
00 0206fbc8 61ccb998 OraClient10!kpufhndl0+0x81a
01 0206fbdc 61c2602c OraClient10!kpufhndl+0x10
02 0206fbec 0c7e706c OraClient10!OCIHandleFree+0x1a
03 0206fbfc 01e0b4d1 oci!OCIHandleFree+0x18
04 0206fc80 0d314415 0x1e0b4d1
05 0206fca8 79203bd3 0xd314415
06 0206fd34 79203c95 mscorsvr!MethodTable::CallFinalizer+0xee
07 0206fd48 79203395 mscorsvr!CallFinalizer+0x84
08 0206fde0 79203f97 mscorsvr!CallFinalizer+0x255
09 0206fdf4 791cf976 mscorsvr!GCHeap::RestartEE+0x73
0a 0206fe3c 79203f6a mscorsvr!Thread::DoADCallBack+0x5c
0b 0206fedc 792034de mscorsvr!CallFinalizer+0x2fa
0c 0206ff24 7922ec3f mscorsvr!GCHeap::FinalizerThreadStart+0xc2
0d 0206ffb8 77e64829 mscorsvr!Thread::intermediateThreadProc+0x44
0e 0206ffec 00000000 kernel32!BaseThreadStart+0x34
Closing open log file E:\Temp\__Richard\IISState2\output\IISState-3620.log
Opened log file 'E:\Temp\__Richard\IISState2\output\IISState-3620.log'

***********************
Starting new log output
IISState version 3.3.1

Fri Jan 09 13:21:03 2009

OS = Windows 2003 Server
Executable: w3wp.exe
PID = 3620

Note: Thread times are formatted as HH:MM:SS.ms

***********************




Thread ID: 0
System Thread ID: 2dc
Kernel Time: 0:0:0.0
User Time: 0:0:0.93
Thread Status: Thread is in a WAIT state.
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 0006fc08 7c827d0b ntdll!KiFastSystemCallRet
01 0006fc0c 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0006fc7c 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 0006fc90 5a364662 kernel32!WaitForSingleObject+0x12
04 0006fca0 5a366e3f w3dt!WP_CONTEXT::RunMainThreadLoop+0x10
05 0006fca8 5a3af42d w3dt!UlAtqStartListen+0x2d
06 0006fcb8 5a3bc335 w3core!W3_SERVER::StartListen+0xbd
07 0006ff0c 0100187c w3core!UlW3Start+0x26e
08 0006ff44 01001a27 w3wp!wmain+0x22a
09 0006ffc0 77e6f23b w3wp!wmainCRTStartup+0x12f
0a 0006fff0 00000000 kernel32!BaseProcessStart+0x23




Thread ID: 1
System Thread ID: 678
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0063fea0 7c827cfb ntdll!KiFastSystemCallRet
01 0063fea4 7c80e5bb ntdll!NtWaitForMultipleObjects+0xc
02 0063ff48 7c80e4a2 ntdll!EtwpWaitForMultipleObjectsEx+0xf7
03 0063ffb8 77e64829 ntdll!EtwpEventPump+0x27f
04 0063ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 2
System Thread ID: 8c0
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00ccff9c 7c826f4b ntdll!KiFastSystemCallRet
01 00ccffa0 7c83d424 ntdll!NtDelayExecution+0xc
02 00ccffb8 77e64829 ntdll!RtlpTimerThread+0x47
03 00ccffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 3
System Thread ID: 230
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00d0ff70 7c8277db ntdll!KiFastSystemCallRet
01 00d0ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 00d0ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 00d0ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 4
System Thread ID: e18
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 00d9fe18 7c82783b ntdll!KiFastSystemCallRet
01 00d9fe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
02 00d9ff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
03 00d9ff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
04 00d9ffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
05 00d9ffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
06 00d9ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 5
System Thread ID: ff8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 00ddff08 7c826f4b ntdll!KiFastSystemCallRet
01 00ddff0c 77e41ed1 ntdll!NtDelayExecution+0xc
02 00ddff74 77e424ed kernel32!SleepEx+0x68
03 00ddff84 776bbb0f kernel32!Sleep+0xf
04 00ddff90 776bbab4 ole32!CROIDTable::WorkerThreadLoop+0x14
05 00ddffac 776b1704 ole32!CRpcThread::WorkerLoop+0x26
06 00ddffb8 77e64829 ole32!CRpcThreadCache::RpcWorkerThreadEntry+0x20
07 00ddffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 6
System Thread ID: ee8
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 00f9ff24 7c8277db ntdll!KiFastSystemCallRet
01 00f9ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 00f9ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 00f9ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 00f9ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 00f9ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 00f9ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 7
System Thread ID: ddc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 00fdff24 7c8277db ntdll!KiFastSystemCallRet
01 00fdff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 00fdff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 00fdff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 00fdffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 00fdffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 00fdffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 8
System Thread ID: 818
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0104ff24 7c8277db ntdll!KiFastSystemCallRet
01 0104ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 0104ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 0104ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 0104ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 0104ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 0104ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 9
System Thread ID: 8e8
Kernel Time: 0:0:0.0
User Time: 0:0:0.15
Thread Type: HTTP Listener
# ChildEBP RetAddr
00 0108ff24 7c8277db ntdll!KiFastSystemCallRet
01 0108ff28 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 0108ff54 5a30248e kernel32!GetQueuedCompletionStatus+0x29
03 0108ff8c 5a3026ac W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x33
04 0108ffa0 5a301da9 W3TP!THREAD_POOL_DATA::ThreadPoolThread+0x24
05 0108ffb8 77e64829 W3TP!THREAD_MANAGER::ThreadManagerThread+0x39
06 0108ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 10
System Thread ID: dc4
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0114fcec 7c827cfb ntdll!KiFastSystemCallRet
01 0114fcf0 7c83c78e ntdll!NtWaitForMultipleObjects+0xc
02 0114ffb8 77e64829 ntdll!RtlpWaitThread+0x161
03 0114ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 11
System Thread ID: 578
Kernel Time: 0:0:0.15
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: HTTP Compression Thread
# ChildEBP RetAddr
00 01b9fa84 7c827d0b ntdll!KiFastSystemCallRet
01 01b9fa88 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 01b9faf8 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 01b9fb0c 5a3b820f kernel32!WaitForSingleObject+0x12
04 01b9ffb8 77e64829 w3core!HTTP_COMPRESSION::CompressionThread+0x126
05 01b9ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 12
System Thread ID: f14
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Possible ASP page. Possible DCOM activity
Executing Page: ASP.dll symbols not found. Unable to locate ASP page.
Continuing with other analysis.

No remote call being made

# ChildEBP RetAddr
00 01bdfe18 7c82783b ntdll!KiFastSystemCallRet
01 01bdfe1c 77c885ac ntdll!NtReplyWaitReceivePortEx+0xc
02 01bdff84 77c88792 RPCRT4!LRPC_ADDRESS::ReceiveLotsaCalls+0x198
03 01bdff8c 77c8872d RPCRT4!RecvLotsaCallsWrapper+0xd
04 01bdffac 77c7b110 RPCRT4!BaseCachedThreadRoutine+0x9d
05 01bdffb8 77e64829 RPCRT4!ThreadStartRoutine+0x1b
06 01bdffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 13
System Thread ID: a40
Kernel Time: 0:0:0.265
User Time: 0:0:1.640
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 01d2fe78 7c827d0b ntdll!KiFastSystemCallRet
01 01d2fe7c 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 01d2feec 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 01d2ff00 791d6134 kernel32!WaitForSingleObject+0x12
04 01d2ff24 791fe95c mscorsvr!ThreadpoolMgr::WorkerThreadStart+0x3a
05 01d2ffb8 77e64829 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44
06 01d2ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 14
System Thread ID: 9dc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 01e7fe70 7c827cfb ntdll!KiFastSystemCallRet
01 01e7fe74 77e6202c ntdll!NtWaitForMultipleObjects+0xc
02 01e7ff1c 77e62fbe kernel32!WaitForMultipleObjectsEx+0x11a
03 01e7ff38 7922fce6 kernel32!WaitForMultipleObjects+0x18
04 01e7ffa0 7922fc5d mscorsvr!DebuggerRCThread::MainLoop+0x90
05 01e7ffb0 79232154 mscorsvr!DebuggerRCThread::ThreadProc+0x68
06 01e7ffb8 77e64829 mscorsvr!DebuggerRCThread::ThreadProcStatic+0xb
07 01e7ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 15
System Thread ID: 580
Kernel Time: 0:0:0.0
User Time: 0:0:0.62
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 01fefefc 7c827d0b ntdll!KiFastSystemCallRet
01 01feff00 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 01feff70 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 01feff84 7920369d kernel32!WaitForSingleObject+0x12
04 01feffac 79227e07 mscorsvr!gc_heap::gc_thread_function+0x2f
05 01feffb0 00000000 mscorsvr!SetupUnstartedThread




Thread ID: 16
System Thread ID: b54
Kernel Time: 0:0:0.0
User Time: 0:0:0.62
Thread Status: Thread is in a WAIT state.
Thread Type: Other
# ChildEBP RetAddr
00 0202fdfc 7c827d0b ntdll!KiFastSystemCallRet
01 0202fe00 77e61d1e ntdll!NtWaitForSingleObject+0xc
02 0202fe70 77e61c8d kernel32!WaitForSingleObjectEx+0xac
03 0202fe84 7920369d kernel32!WaitForSingleObject+0x12
04 0202feac 79227e07 mscorsvr!gc_heap::gc_thread_function+0x2f
05 0202feb0 00000000 mscorsvr!SetupUnstartedThread




Thread ID: 17
System Thread ID: b00
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
WARNING: Stack unwind information not available. Following frames may be
wrong.
00 0206fbc8 61ccb998 OraClient10!kpufhndl0+0x81a
01 0206fbdc 61c2602c OraClient10!kpufhndl+0x10
02 0206fbec 0c7e706c OraClient10!OCIHandleFree+0x1a
03 0206fbfc 01e0b4d1 oci!OCIHandleFree+0x18
04 0206fc80 0d314415 0x1e0b4d1
05 0206fca8 79203bd3 0xd314415
06 0206fd34 79203c95 mscorsvr!MethodTable::CallFinalizer+0xee
07 0206fd48 79203395 mscorsvr!CallFinalizer+0x84
08 0206fde0 79203f97 mscorsvr!CallFinalizer+0x255
09 0206fdf4 791cf976 mscorsvr!GCHeap::RestartEE+0x73
0a 0206fe3c 79203f6a mscorsvr!Thread::DoADCallBack+0x5c
0b 0206fedc 792034de mscorsvr!CallFinalizer+0x2fa
0c 0206ff24 7922ec3f mscorsvr!GCHeap::FinalizerThreadStart+0xc2
0d 0206ffb8 77e64829 mscorsvr!Thread::intermediateThreadProc+0x44
0e 0206ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 18
System Thread ID: 3cc
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0238ff40 7c826f4b ntdll!KiFastSystemCallRet
01 0238ff44 77e41ed1 ntdll!NtDelayExecution+0xc
02 0238ffac 791cfb1a kernel32!SleepEx+0x68
03 0238ffb8 77e64829 mscorsvr!ThreadpoolMgr::TimerThreadStart+0x23
04 0238ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 19
System Thread ID: d38
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 023dfe40 7c8277db ntdll!KiFastSystemCallRet
01 023dfe44 77e5bea2 ntdll!ZwRemoveIoCompletion+0xc
02 023dfe70 791fe9b2 kernel32!GetQueuedCompletionStatus+0x29
03 023dfea4 791fe95c mscorsvr!ThreadpoolMgr::CompletionPortThreadStart+0x49
04 023dffb8 77e64829 mscorsvr!ThreadpoolMgr::intermediateThreadProc+0x44
05 023dffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 20
System Thread ID: 46c
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0241feb0 7c826f4b ntdll!KiFastSystemCallRet
01 0241feb4 77e41ed1 ntdll!NtDelayExecution+0xc
02 0241ff1c 77e424ed kernel32!SleepEx+0x68
03 0241ff2c 791bc166 kernel32!Sleep+0xf
04 0241ffb8 77e64829 mscorsvr!ThreadpoolMgr::GateThreadStart+0x54
05 0241ffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 21
System Thread ID: a58
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0d4cff70 7c8277db ntdll!KiFastSystemCallRet
01 0d4cff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 0d4cffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 0d4cffec 00000000 kernel32!BaseThreadStart+0x34




Thread ID: 22
System Thread ID: fd8
Kernel Time: 0:0:0.78
User Time: 0:0:0.343
*** ERROR: Symbol file could not be found. Defaulted to export symbols for
E:\App\Oracle\Product\1020\bin\oracore10.dll -
Thread Status: Thread is in a WAIT state.
Other information: Thread is waiting for a lock to be released. Looking for
lock owner.
Owning thread System ID: b00
Thread Type: Other
# ChildEBP RetAddr
00 0d71f138 7c827d0b ntdll!KiFastSystemCallRet
01 0d71f13c 7c83d236 ntdll!NtWaitForSingleObject+0xc
02 0d71f178 7c83d281 ntdll!RtlpWaitOnCriticalSection+0x1a3
03 0d71f198 0c91a5af ntdll!RtlEnterCriticalSection+0xa8
WARNING: Stack unwind information not available. Following frames may be
wrong.
04 0d71f1a4 61dda82d oracore10!sltsmna+0xf
05 0d71f400 61ccb998 OraClient10!OCIPHeapFree+0x85bc1
06 0d71f414 61c2602c OraClient10!kpufhndl+0x10
07 0d71f424 0c7e706c OraClient10!OCIHandleFree+0x1a
08 0d71f434 01e0b4d1 oci!OCIHandleFree+0x18
09 0d71f4b8 0d323562 0x1e0b4d1
0a 0d71f500 0d3232a9 0xd323562
0b 0d71f530 0d32322d 0xd3232a9
0c 0d71f548 0c795cba 0xd32322d
0d 0d71f620 0c794b1e 0xc795cba
0e 0d71f71c 0c7a53cc 0xc794b1e
0f 0d71f788 0c7a0a0f 0xc7a53cc
10 0d71f7c4 0c7a047b 0xc7a0a0f
11 0d71f820 02459eba 0xc7a047b
12 0d71f86c 02459d8b 0x2459eba
13 0d71f8c4 02116468 0x2459d8b
14 0d71f954 792403e6 0x2116468
15 0d71f994 791cf976 mscorsvr!ComCallMLStubCache::CompileMLStub+0x1af
16 0d71f9dc 7923ff03 mscorsvr!Thread::DoADCallBack+0x5c
17 0d71fa44 01e0a0bd mscorsvr!ComCallMLStubCache::CompileMLStub+0x2ba
18 0d71fb28 0d4e0768 0x1e0a0bd
19 0d4e0768 00000000 0xd4e0768




Thread ID: 23
System Thread ID: 370
Kernel Time: 0:0:0.0
User Time: 0:0:0.0
Thread Type: Other
# ChildEBP RetAddr
00 0d76ff70 7c8277db ntdll!KiFastSystemCallRet
01 0d76ff74 7c839f38 ntdll!ZwRemoveIoCompletion+0xc
02 0d76ffb8 77e64829 ntdll!RtlpWorkerThread+0x3d
03 0d76ffec 00000000 kernel32!BaseThreadStart+0x34

*****

Dump name is formatted as: PID-Timestamp.dmp

Creating E:\Temp\__Richard\IISState2\output\3620-1231507264.dmp - mini user
dump

*****

Closing open log file E:\Temp\__Richard\IISState2\output\IISState-3620.log
---------------------------------------------------------------------------------

 | 
Pages: 1
Prev: DSOFile.dll
Next: IIS 7 - Cannot launch out of process component?