IIS 5 Hang
in [IIS]
|
From: Juan on 16 Aug 2006 17:43 Hi, I wonder if you guys can give us a hand with our analysis of this IIS hangs. We are running 6 in-process ASP websites (only 2 of them have 20+ virtual dirs / logical apps) the other four are pretty light. The 2 main websites have several hundreds of ASP pages. A little bit more than a thousand unique visitors per day. Connections to SQL 2000, DB2, and MQ. The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and 2GB RAM. The inetinfo hangs randomly (no-load related) with an average of 2/3 times a week. Recycling IIS the problem goes away. We are using IIS 5 Process Recycle on daily basis. No too much help. We are using IIS Diagnostic Tools to obtain and analyze memory dumps. Based on your incredible IIS Diagnostic Tools reports, we have the theory that the root cause of our problems is the large number of big pages and the extensive number of include files (more than 100 includes in some pages). We think that they produce heap fragmentation in the ASP template cache that materializes in memory lacks. We have more than 2 months analyzing the symptoms, and we obtained some interesting stuff, but the advise of the experts will help us a lot confirming our theories or giving us new leads. Can I paste in a next message either dump files or IIS diagnostics HTML reports for your opinion? Your help will be really appreciated. Thanks a lot, -- Juan
From: Bernard Cheah [MVP] on 16 Aug 2006 23:38 You can start with the html and provide a link to the dump file if needed. -- Regards, Bernard Cheah http://www.iis.net/ http://www.iis-resources.com/ http://msmvps.com/blogs/bernard/ "Juan" <Juan(a)discussions.microsoft.com> wrote in message news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com... > Hi, > > I wonder if you guys can give us a hand with our analysis of this IIS > hangs. > > We are running 6 in-process ASP websites (only 2 of them have 20+ virtual > dirs / logical apps) the other four are pretty light. The 2 main websites > have several hundreds of ASP pages. > > A little bit more than a thousand unique visitors per day. > > Connections to SQL 2000, DB2, and MQ. > > The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and 2GB > RAM. > > The inetinfo hangs randomly (no-load related) with an average of 2/3 times > a > week. Recycling IIS the problem goes away. > > We are using IIS 5 Process Recycle on daily basis. No too much help. > > We are using IIS Diagnostic Tools to obtain and analyze memory dumps. > > Based on your incredible IIS Diagnostic Tools reports, we have the theory > that the root cause of our problems is the large number of big pages and > the > extensive number of include files (more than 100 includes in some pages). > We > think that they produce heap fragmentation in the ASP template cache that > materializes in memory lacks. > > We have more than 2 months analyzing the symptoms, and we obtained some > interesting stuff, but the advise of the experts will help us a lot > confirming our theories or giving us new leads. > > Can I paste in a next message either dump files or IIS diagnostics HTML > reports for your opinion? > > Your help will be really appreciated. > > Thanks a lot, > > -- > Juan
From: Juan on 17 Aug 2006 09:05 Hi Bernard, Thanks for your quick answer! Below you'll see the Crash/Hanhg analysis report. I will send the Memory analysis in a separated message. A little bit more of background: Using MS Web Application Stress Tool in a paralel environment we can stress the the apps getting hang rule dumps but we cannot really hang the service as in the production incidents. The load generated in our WAS scripts is not as wide and realistic as the prod load. Thanks a lot -- Juan Analysis Summary Type Description Recommendation Warning Detected possible blocking or leaked critical section at NTDLL!LoaderLock owned by thread 112 in inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang Dump.dmp Impact of this lock 71.88% of executing ASP Requests blocked 19.71% of threads blocked (Threads 30 52 55 56 59 66 68 74 75 76 77 79 87 101 103 104 106 107 115 116 117 122 124 128 130 132 136) The following functions are trying to enter this critical section NTDLL!LdrpGetProcedureAddress+122 NTDLL!LdrUnloadDll+5f KERNEL32!GetModuleFileNameW+ad NTDLL!LdrShutdownThread+38 The following module(s) are involved with this critical section C:\WINNT\system32\NTDLL.DLL from Microsoft Corporation C:\WINNT\system32\KERNEL32.DLL from Microsoft Corporation The following vendors were identified for follow up based on root cause analysis .... Warning The following threads in inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang Dump.dmp are processing a client request and is/are not fully resolved. Further analysis of these threads is recommended in order to determine what may be blocking the request(s). ( 0 32 78 81 105 109 113 118 123 127 ) 28.13% of executing ASP Requests blocked 6.25% of IIS worker threads blocked 7.30% of threads blocked .... Warning The COM+ STA ThreadPool may have been depleted prior to the time of the dump in inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang Dump.dmp. There is more than one activity bound to the following COM+ STA ThreadPool threads: 52 68 56 76 32 75 105 81 30 31 See the COM+ STA ThreadPool Report for more detail. 52% of the STA ThreadPool threads are idle, however some threads still have more than one activity bound to them. This can happen for various reasons: If COM+ objects are being leaked. If there is a memory leak, you should use DebugDiag to monitor the leak for root cause. See the "Troubleshooting Memory Leaks with DebugDiag" topic in the DebugDiag.chm file for more information. .... Warning 3 client connection(s) in inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang Dump.dmp have been executing a request for more than 90 seconds. Please see the Client Connections section of this report for more detailed information about the connection(s). Information DebugDiag did not detect any known problems in H:\Logs\IIS Diag analysis\Prod Memory Dumps 08_16_2006_Outage\aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang Dump.dmp using the current set of scripts. Analysis Details Table Of Contents dllhost.exe__System Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang Dump.dmp Top 5 threads by CPU time Thread report Well-Known COM STA Threads Report inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang Dump.dmp Top 5 threads by CPU time Locked critical section report Thread report Well-Known COM STA Threads Report HTTP Report ASP Report aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang Dump.dmp Top 5 threads by CPU time Thread report Well-Known COM STA Threads Report Report for dllhost.exe__System Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang Dump.dmp Type of Analysis Performed Hang Analysis Machine Name NLGPWEBIIS05 Operating System Windows 2000 Service Pack 4 Number Of Processors 4 Process ID 3020 Process Image C:\WINNT\system32\dllhost.exe System Up-Time 1 day(s) 05:05:30 Process Up-Time 1 day(s) 05:04:28 Top 5 Threads by CPU time Note - Times include both user mode and kernel mode for each thread Thread ID: 5 Total CPU Time: 0 day(s) 00:00:05.562 Entry Point for Thread: msvcrt!_endthreadex+32 Thread ID: 6 Total CPU Time: 0 day(s) 00:00:00.750 Entry Point for Thread: rpcrt4!ThreadStartRoutine Thread ID: 4 Total CPU Time: 0 day(s) 00:00:00.31 Entry Point for Thread: msvcrt!_endthread+3f Thread ID: 7 Total CPU Time: 0 day(s) 00:00:00.31 Entry Point for Thread: comsvcs!CEventServer::DispatchEvents Thread ID: 3 Total CPU Time: 0 day(s) 00:00:00.15 Entry Point for Thread: txfaux!WORK_QUEUE::ThreadLoop Thread report Thread 0 - System ID 3016 Entry point dllhost!WinMainCRTStartup Create time 08/15/2006 6:08:50 AM Time spent in user mode 0 Days 0:0:0.0 Time spent in kernel mode 0 Days 0:0:0.15 This thread is not fully resolved and may or may not be a problem. Further analysis of these threads may be required. Function Source NTDLL!ZwWaitForSingleObject+b KERNEL32!WaitForSingleObjectEx+71 KERNEL32!WaitForSingleObject+f .... .... Thread 7 - System ID 3060 Entry point comsvcs!CEventServer::DispatchEvents Create time 08/15/2006 6:08:50 AM Time spent in user mode 0 Days 0:0:0.0 Time spent in kernel mode 0 Days 0:0:0.31 This thread is making a COM call to neutral apartment (NA) within the same process Function Source NTDLL!ZwWaitForMultipleObjects+b KERNEL32!WaitForMultipleObjectsEx+ea KERNEL32!WaitForMultipleObjects+17 rpcrt4!Invoke+30 rpcrt4!NdrStubCall2+664 rpcrt4!CStdStubBuffer_Invoke+c8 OLE32!SyncStubInvoke+61 OLE32!StubInvoke+a8 OLE32!CCtxComChnl::ContextInvoke+bb OLE32!MTAInvoke+18 OLE32!AppInvoke+b5 OLE32!ComInvokeWithLockAndIPID+297 OLE32!ComInvoke+41 OLE32!ThreadDispatch+21 OLE32!DispatchCall+24 OLE32!CRpcChannelBuffer::SwitchAptAndDispatchCall+34 OLE32!CRpcChannelBuffer::SendReceive2+96 OLE32!CRpcChannelBuffer::SendReceive+11 OLE32!CAptRpcChnl::SendReceive+a9 OLE32!CCtxComChnl::SendReceive+124
From: Juan on 17 Aug 2006 10:45 Hi Bernard, Now the main fragments of the memory analysis report. Thanks again. Type Description Recommendation Warning rpcrt4.dll is responsible for 6.90 KBytes worth of outstanding allocations. The following are the top 2 memory consuming functions: rpcrt4!operator new+12: 6.81 KBytes worth of outstanding allocations. rpcrt4!NdrpCreateNonDelegatedAsyncStub+71: 60 Bytes worth of outstanding allocations. This was detected in dllhost.exe__System Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang Dump.dmp If this is unexpected, please contact the vendor of this module, Microsoft Corporation, for further assistance with this issue. Warning clbcatq.dll is responsible for 3.17 KBytes worth of outstanding allocations. The following are the top 2 memory consuming functions: clbcatq!CComCLBCatalog::GetClassInfoW+6b: 1,016 Bytes worth of outstanding allocations. clbcatq!CComClass::Init+8b5: 488 Bytes worth of outstanding allocations. This was detected in dllhost.exe__System Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang Dump.dmp If this is unexpected, please contact the vendor of this module, Microsoft Corporation, for further assistance with this issue. Warning Detected symptoms of high fragmentation in the following heaps in inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang Dump.dmp 0x00070000 (Default process heap - 75.86% Fragmented) 0x02da0000 (ASP!CTemplate::sm_hLargeHeap - 93.68% Fragmented) Heap fragmentation is often caused by one of the following two reasons 1. Small heap memory blocks that are leaked (allocated but never freed) over time 2. Mixing long lived small allocations with short lived long allocations Both of these reasons can prevent the NT heap manager from using free memory effeciently since they are spread as small fragments that cannot be used as a single large allocation Information DebugDiag did not detect LeakTrack.dll loaded in inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang Dump.dmp, so no leak analysis was performed on this file. If you are troubleshooting a memory leak, please ensure LeakTrack.dll is injected into the target process using the DebugDiag tool before or generating new dumps. For information regarding installation and usage of the IISDiag tool, please see the included help file. Information DebugDiag did not detect any known native heap(unmanaged) problems in aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang Dump.dmp using the current set of scripts. Information DebugDiag did not detect LeakTrack.dll loaded in aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang Dump.dmp, so no leak analysis was performed on this file. If you are troubleshooting a memory leak, please ensure LeakTrack.dll is injected into the target process using the DebugDiag tool before or generating new dumps. For information regarding installation and usage of the IISDiag tool, please see the included help file. Information DebugDiag did not detect any known native heap(unmanaged) problems in dllhost.exe__System Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang Dump.dmp using the current set of scripts. Analysis Details Table Of Contents dllhost.exe__System Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang Dump.dmp Virtual Memory Analysis Report Heap Analysis Report Leak Analysis Report Outstanding allocation summary Detailed module report (Memory) Detailed module report (Handles) inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang Dump.dmp Virtual Memory Analysis Report Heap Analysis Report aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang Dump.dmp Virtual Memory Analysis Report Heap Analysis Report Report for dllhost.exe__System Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang Dump.dmp Type of Analysis Performed Memory Pressure Analysis Machine Name NLGPWEBIIS05 Operating System Windows 2000 Service Pack 4 Number Of Processors 4 Process ID 3020 Process Image C:\WINNT\system32\dllhost.exe System Up-Time 1 day(s) 05:05:30 Process Up-Time 1 day(s) 05:04:28 Virtual Memory Analysis Virtual Memory Summary Size of largest free VM block 1.17 GBytes Free memory fragmentation 40.32% Free Memory 1.96 GBytes (98.16% of Total Memory) Reserved Memory 12.79 MBytes (0.62% of Total Memory) Committed Memory 24.84 MBytes (1.21% of Total Memory) Total Memory 2.00 GBytes Largest free block at 0x00000000`1a47d000 Virtual Memory Details Virtual Allocations Loaded Modules Threads System Page Heaps Native Heaps 7.44 MBytes 19.67 MBytes 4.32 MBytes 12.00 KBytes 0 Bytes 6.19 MBytes Virtual Allocation Summary Reserved memory 6.00 MBytes Committed memory 1.44 MBytes Mapped memory 3.32 MBytes Reserved block count 6 blocks Committed block count 30 blocks Mapped block count 17 blocks Loaded Module Summary Number of Modules 64 Modules Total reserved memory 0 Bytes Total committed memory 19.67 MBytes .... Report for dllhost.exe__System Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang Dump.dmp Type of Analysis Performed Memory Pressure Analysis Machine Name NLGPWEBIIS05 Operating System Windows 2000 Service Pack 4 Number Of Processors 4 Process ID 3020 Process Image C:\WINNT\system32\dllhost.exe System Up-Time 1 day(s) 05:05:30 Process Up-Time 1 day(s) 05:04:28 Virtual Memory Analysis Virtual Memory Summary Size of largest free VM block 1.17 GBytes Free memory fragmentation 40.32% Free Memory 1.96 GBytes (98.16% of Total Memory) Reserved Memory 12.79 MBytes (0.62% of Total Memory) Committed Memory 24.84 MBytes (1.21% of Total Memory) Total Memory 2.00 GBytes Largest free block at 0x00000000`1a47d000 .... Virtual Allocation Summary Reserved memory 6.00 MBytes Committed memory 1.44 MBytes Mapped memory
From: Juan on 17 Aug 2006 12:57
Hi Bernard, Here are the links to the full IIS Diagnostics tool reports. The actual dumps are more than 100 MB large, so I cannot upload them by the time being, let me see what I can do. http://www.compilar.com/garage/IIS_Report__Date_08_17_2006__Time_08_18_50AM__117.htm http://www.compilar.com/garage/Memory_Report__Date_08_17_2006__Time_09_04_50AM__215.htm Comment: In other outages and high stress dumps, I got much more leak probability occurencies and higher percentages than in this memory report sample. Thanks again. -- Juan "Bernard Cheah [MVP]" wrote: > You can start with the html and provide a link to the dump file if needed. > > -- > Regards, > Bernard Cheah > http://www.iis.net/ > http://www.iis-resources.com/ > http://msmvps.com/blogs/bernard/ > > > "Juan" <Juan(a)discussions.microsoft.com> wrote in message > news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com... > > Hi, > > > > I wonder if you guys can give us a hand with our analysis of this IIS > > hangs. > > > > We are running 6 in-process ASP websites (only 2 of them have 20+ virtual > > dirs / logical apps) the other four are pretty light. The 2 main websites > > have several hundreds of ASP pages. > > > > A little bit more than a thousand unique visitors per day. > > > > Connections to SQL 2000, DB2, and MQ. > > > > The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and 2GB > > RAM. > > > > The inetinfo hangs randomly (no-load related) with an average of 2/3 times > > a > > week. Recycling IIS the problem goes away. > > > > We are using IIS 5 Process Recycle on daily basis. No too much help. > > > > We are using IIS Diagnostic Tools to obtain and analyze memory dumps. > > > > Based on your incredible IIS Diagnostic Tools reports, we have the theory > > that the root cause of our problems is the large number of big pages and > > the > > extensive number of include files (more than 100 includes in some pages). > > We > > think that they produce heap fragmentation in the ASP template cache that > > materializes in memory lacks. > > > > We have more than 2 months analyzing the symptoms, and we obtained some > > interesting stuff, but the advise of the experts will help us a lot > > confirming our theories or giving us new leads. > > > > Can I paste in a next message either dump files or IIS diagnostics HTML > > reports for your opinion? > > > > Your help will be really appreciated. > > > > Thanks a lot, > > > > -- > > Juan > > > |