From: Juan on
Bernard,

Thanks for your tips and your time, I will try to contact Pat to get his
opinion.

Thanks.
--
Juan


"Bernard Cheah [MVP]" wrote:

> I'm not the person that you should send the dump file :)
> If this is urgent, you can open a support case with Microsoft PSS.
> if not, you can post a new thread with detail link to the log and dump file,
> Attention it to 'Pat', this guy is an expert, me is not :( Anyway, in your
> case the hang looks like from long running request that holding all
> subsequence request, deadlock could occur as well in this case. Sorry that
> I can't be helpful.
>
> --
> Regards,
> Bernard Cheah
> http://www.iis.net/
> http://www.iis-resources.com/
> http://msmvps.com/blogs/bernard/
>
>
> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> news:18AF64DE-D93C-4A18-87D3-E93F694F7855(a)microsoft.com...
> > Hi Bernard,
> >
> > Thanks for your answer.
> >
> > Yes, we are implementing some of the recommendations. However we wanted to
> > be sure that we undersatand the root cause for the hangs. Based on the
> > report
> > the things going wrong seem to be:
> >
> > 1) The high heap fragmentations --> causing memory leaks <-- produced by
> > too
> > many includes
> > 2) Long waits and/or blocking in several threads <-- caused by bottle
> > necks?
> > (db conns, etc.?)
> >
> > Is the first reason alone enough to hang the Inetinfo process? If that is
> > the case how would it be the sequence of actions towards the hang point?
> > Is
> > the case that IIS/ASP "chocks" after too much i/o traffic of huge ASPs
> > with
> > 100s of includes in the ASP Template Cache?
> >
> > Regarding the second point, Do you see a single failure point hanging the
> > server? Or is more that the sum of several waiting points and blocked
> > thread
> > ends up (again) "chocking" the Inetinfo process. Can a sum of waiting
> > points
> > and blocked threads hang the process? Or a hang has to be caused by a
> > single
> > point of failure (deadlock, spin, infinite recursion, etc.) ?
> >
> > I will try to find an FTP place in which I can put the 100+ MB memory dump
> > files for you.
> > Impressive! If you can read the dumps better than the reports that is
> > amazing, it speaks about the level of your expertise in the subject!
> >
> > While I find an FTP place (which can take me a while) would it be possible
> > in parallel to have any of the guys there accustomed to read these reports
> > taking a look to them?
> >
> > I will appreciate your answers and I will send you the dumps ASAP.
> >
> > Thanks a lot for your time!
> >
> > --
> > Juan
> >
> >
> > "Bernard Cheah [MVP]" wrote:
> >
> >> I'm no expert in analyzing the log. from the summary - it looks like lot
> >> of
> >> blocking among process threads.
> >> Have you start looking at some of the 'recommendation' steps?
> >>
> >> --
> >> Regards,
> >> Bernard Cheah
> >> http://www.iis.net/
> >> http://www.iis-resources.com/
> >> http://msmvps.com/blogs/bernard/
> >>
> >>
> >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> >> news:02E6B5CD-F3F2-4A25-8DF7-1138FC7D4247(a)microsoft.com...
> >> > Hi Bernard,
> >> >
> >> > Here are the links to the full IIS Diagnostics tool reports.
> >> >
> >> > The actual dumps are more than 100 MB large, so I cannot upload them by
> >> > the
> >> > time being, let me see what I can do.
> >> >
> >> > http://www.compilar.com/garage/IIS_Report__Date_08_17_2006__Time_08_18_50AM__117.htm
> >> >
> >> > http://www.compilar.com/garage/Memory_Report__Date_08_17_2006__Time_09_04_50AM__215.htm
> >> >
> >> > Comment: In other outages and high stress dumps, I got much more leak
> >> > probability occurencies and higher percentages than in this memory
> >> > report
> >> > sample.
> >> >
> >> > Thanks again.
> >> >
> >> > --
> >> > Juan
> >> >
> >> >
> >> > "Bernard Cheah [MVP]" wrote:
> >> >
> >> >> You can start with the html and provide a link to the dump file if
> >> >> needed.
> >> >>
> >> >> --
> >> >> Regards,
> >> >> Bernard Cheah
> >> >> http://www.iis.net/
> >> >> http://www.iis-resources.com/
> >> >> http://msmvps.com/blogs/bernard/
> >> >>
> >> >>
> >> >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> >> >> news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com...
> >> >> > Hi,
> >> >> >
> >> >> > I wonder if you guys can give us a hand with our analysis of this
> >> >> > IIS
> >> >> > hangs.
> >> >> >
> >> >> > We are running 6 in-process ASP websites (only 2 of them have 20+
> >> >> > virtual
> >> >> > dirs / logical apps) the other four are pretty light. The 2 main
> >> >> > websites
> >> >> > have several hundreds of ASP pages.
> >> >> >
> >> >> > A little bit more than a thousand unique visitors per day.
> >> >> >
> >> >> > Connections to SQL 2000, DB2, and MQ.
> >> >> >
> >> >> > The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and
> >> >> > 2GB
> >> >> > RAM.
> >> >> >
> >> >> > The inetinfo hangs randomly (no-load related) with an average of 2/3
> >> >> > times
> >> >> > a
> >> >> > week. Recycling IIS the problem goes away.
> >> >> >
> >> >> > We are using IIS 5 Process Recycle on daily basis. No too much help.
> >> >> >
> >> >> > We are using IIS Diagnostic Tools to obtain and analyze memory
> >> >> > dumps.
> >> >> >
> >> >> > Based on your incredible IIS Diagnostic Tools reports, we have the
> >> >> > theory
> >> >> > that the root cause of our problems is the large number of big pages
> >> >> > and
> >> >> > the
> >> >> > extensive number of include files (more than 100 includes in some
> >> >> > pages).
> >> >> > We
> >> >> > think that they produce heap fragmentation in the ASP template cache
> >> >> > that
> >> >> > materializes in memory lacks.
> >> >> >
> >> >> > We have more than 2 months analyzing the symptoms, and we obtained
> >> >> > some
> >> >> > interesting stuff, but the advise of the experts will help us a lot
> >> >> > confirming our theories or giving us new leads.
> >> >> >
> >> >> > Can I paste in a next message either dump files or IIS diagnostics
> >> >> > HTML
> >> >> > reports for your opinion?
> >> >> >
> >> >> > Your help will be really appreciated.
> >> >> >
> >> >> > Thanks a lot,
> >> >> >
> >> >> > --
> >> >> > Juan
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
>
>
>
From: Juan on
Bernard,

Thanks for your tips and your time, I will try to contact Pat to get his
opinion.

Thanks.
--
Juan


"Bernard Cheah [MVP]" wrote:

> I'm not the person that you should send the dump file :)
> If this is urgent, you can open a support case with Microsoft PSS.
> if not, you can post a new thread with detail link to the log and dump file,
> Attention it to 'Pat', this guy is an expert, me is not :( Anyway, in your
> case the hang looks like from long running request that holding all
> subsequence request, deadlock could occur as well in this case. Sorry that
> I can't be helpful.
>
> --
> Regards,
> Bernard Cheah
> http://www.iis.net/
> http://www.iis-resources.com/
> http://msmvps.com/blogs/bernard/
>
>
> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> news:18AF64DE-D93C-4A18-87D3-E93F694F7855(a)microsoft.com...
> > Hi Bernard,
> >
> > Thanks for your answer.
> >
> > Yes, we are implementing some of the recommendations. However we wanted to
> > be sure that we undersatand the root cause for the hangs. Based on the
> > report
> > the things going wrong seem to be:
> >
> > 1) The high heap fragmentations --> causing memory leaks <-- produced by
> > too
> > many includes
> > 2) Long waits and/or blocking in several threads <-- caused by bottle
> > necks?
> > (db conns, etc.?)
> >
> > Is the first reason alone enough to hang the Inetinfo process? If that is
> > the case how would it be the sequence of actions towards the hang point?
> > Is
> > the case that IIS/ASP "chocks" after too much i/o traffic of huge ASPs
> > with
> > 100s of includes in the ASP Template Cache?
> >
> > Regarding the second point, Do you see a single failure point hanging the
> > server? Or is more that the sum of several waiting points and blocked
> > thread
> > ends up (again) "chocking" the Inetinfo process. Can a sum of waiting
> > points
> > and blocked threads hang the process? Or a hang has to be caused by a
> > single
> > point of failure (deadlock, spin, infinite recursion, etc.) ?
> >
> > I will try to find an FTP place in which I can put the 100+ MB memory dump
> > files for you.
> > Impressive! If you can read the dumps better than the reports that is
> > amazing, it speaks about the level of your expertise in the subject!
> >
> > While I find an FTP place (which can take me a while) would it be possible
> > in parallel to have any of the guys there accustomed to read these reports
> > taking a look to them?
> >
> > I will appreciate your answers and I will send you the dumps ASAP.
> >
> > Thanks a lot for your time!
> >
> > --
> > Juan
> >
> >
> > "Bernard Cheah [MVP]" wrote:
> >
> >> I'm no expert in analyzing the log. from the summary - it looks like lot
> >> of
> >> blocking among process threads.
> >> Have you start looking at some of the 'recommendation' steps?
> >>
> >> --
> >> Regards,
> >> Bernard Cheah
> >> http://www.iis.net/
> >> http://www.iis-resources.com/
> >> http://msmvps.com/blogs/bernard/
> >>
> >>
> >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> >> news:02E6B5CD-F3F2-4A25-8DF7-1138FC7D4247(a)microsoft.com...
> >> > Hi Bernard,
> >> >
> >> > Here are the links to the full IIS Diagnostics tool reports.
> >> >
> >> > The actual dumps are more than 100 MB large, so I cannot upload them by
> >> > the
> >> > time being, let me see what I can do.
> >> >
> >> > http://www.compilar.com/garage/IIS_Report__Date_08_17_2006__Time_08_18_50AM__117.htm
> >> >
> >> > http://www.compilar.com/garage/Memory_Report__Date_08_17_2006__Time_09_04_50AM__215.htm
> >> >
> >> > Comment: In other outages and high stress dumps, I got much more leak
> >> > probability occurencies and higher percentages than in this memory
> >> > report
> >> > sample.
> >> >
> >> > Thanks again.
> >> >
> >> > --
> >> > Juan
> >> >
> >> >
> >> > "Bernard Cheah [MVP]" wrote:
> >> >
> >> >> You can start with the html and provide a link to the dump file if
> >> >> needed.
> >> >>
> >> >> --
> >> >> Regards,
> >> >> Bernard Cheah
> >> >> http://www.iis.net/
> >> >> http://www.iis-resources.com/
> >> >> http://msmvps.com/blogs/bernard/
> >> >>
> >> >>
> >> >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> >> >> news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com...
> >> >> > Hi,
> >> >> >
> >> >> > I wonder if you guys can give us a hand with our analysis of this
> >> >> > IIS
> >> >> > hangs.
> >> >> >
> >> >> > We are running 6 in-process ASP websites (only 2 of them have 20+
> >> >> > virtual
> >> >> > dirs / logical apps) the other four are pretty light. The 2 main
> >> >> > websites
> >> >> > have several hundreds of ASP pages.
> >> >> >
> >> >> > A little bit more than a thousand unique visitors per day.
> >> >> >
> >> >> > Connections to SQL 2000, DB2, and MQ.
> >> >> >
> >> >> > The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and
> >> >> > 2GB
> >> >> > RAM.
> >> >> >
> >> >> > The inetinfo hangs randomly (no-load related) with an average of 2/3
> >> >> > times
> >> >> > a
> >> >> > week. Recycling IIS the problem goes away.
> >> >> >
> >> >> > We are using IIS 5 Process Recycle on daily basis. No too much help.
> >> >> >
> >> >> > We are using IIS Diagnostic Tools to obtain and analyze memory
> >> >> > dumps.
> >> >> >
> >> >> > Based on your incredible IIS Diagnostic Tools reports, we have the
> >> >> > theory
> >> >> > that the root cause of our problems is the large number of big pages
> >> >> > and
> >> >> > the
> >> >> > extensive number of include files (more than 100 includes in some
> >> >> > pages).
> >> >> > We
> >> >> > think that they produce heap fragmentation in the ASP template cache
> >> >> > that
> >> >> > materializes in memory lacks.
> >> >> >
> >> >> > We have more than 2 months analyzing the symptoms, and we obtained
> >> >> > some
> >> >> > interesting stuff, but the advise of the experts will help us a lot
> >> >> > confirming our theories or giving us new leads.
> >> >> >
> >> >> > Can I paste in a next message either dump files or IIS diagnostics
> >> >> > HTML
> >> >> > reports for your opinion?
> >> >> >
> >> >> > Your help will be really appreciated.
> >> >> >
> >> >> > Thanks a lot,
> >> >> >
> >> >> > --
> >> >> > Juan
> >> >>
> >> >>
> >> >>
> >>
> >>
> >>
>
>
>
From: Pat [MSFT} on
1) There is _no_ indication whatsoever that page size/include count is a
factor.
2) You have at least one thread blocked trying to create an object - there
could be valid reasons for this; consider it a symptom not a cause.
3) You have a bunch of threads blocked on thread 112 - has most likely
thrown an exception & leaked a lock which is leading to a hang. I would say
(based on the blocked threads) that the most likely culprit is MDAC. I
would update that first and see if it helps. The #2 culprit is COM+ (you
would need to contact MS-Support for the latest COM+ rollup).


Pat



"Juan" <Juan(a)discussions.microsoft.com> wrote in message
news:E97D51CF-9FB6-4F16-B7A1-BCCA3A28FA05(a)microsoft.com...
> Hi Bernard,
>
> Thanks for your quick answer!
>
> Below you'll see the Crash/Hanhg analysis report. I will send the Memory
> analysis in a separated message.
>
> A little bit more of background:
>
> Using MS Web Application Stress Tool in a paralel environment we can
> stress
> the the apps getting hang rule dumps but we cannot really hang the service
> as
> in the production incidents. The load generated in our WAS scripts is not
> as
> wide and realistic as the prod load.
>
> Thanks a lot
> --
> Juan
>
>
> Analysis Summary
> Type Description Recommendation
> Warning Detected possible blocking or leaked critical section at
> NTDLL!LoaderLock owned by thread 112 in
>
> inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
> Dump.dmp
>
> Impact of this lock
>
> 71.88% of executing ASP Requests blocked
>
> 19.71% of threads blocked
>
> (Threads 30 52 55 56 59 66 68 74 75 76 77 79 87 101 103 104 106 107 115
> 116
> 117 122 124 128 130 132 136)
>
> The following functions are trying to enter this critical section
> NTDLL!LdrpGetProcedureAddress+122
> NTDLL!LdrUnloadDll+5f
> KERNEL32!GetModuleFileNameW+ad
> NTDLL!LdrShutdownThread+38
>
> The following module(s) are involved with this critical section
> C:\WINNT\system32\NTDLL.DLL from Microsoft Corporation
> C:\WINNT\system32\KERNEL32.DLL from Microsoft Corporation
> The following vendors were identified for follow up based on root cause
> analysis
>
> ...
>
> Warning The following threads in
> inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
>
> Dump.dmp are processing a client request and is/are not fully resolved.
> Further analysis of these threads is
>
> recommended in order to determine what may be blocking the request(s).
>
> ( 0 32 78 81 105 109 113 118 123 127 )
>
> 28.13% of executing ASP Requests blocked
>
> 6.25% of IIS worker threads blocked
>
> 7.30% of threads blocked
>
>
> ...
>
> Warning The COM+ STA ThreadPool may have been depleted prior to the time
> of the dump in
>
> inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
> Dump.dmp. There is more than one
>
> activity bound to the following COM+ STA ThreadPool threads:
>
> 52 68 56 76 32 75 105 81 30 31
>
> See the COM+ STA ThreadPool Report for more detail. 52% of the STA
> ThreadPool threads are idle, however some
>
> threads still have more than one activity bound to them. This can happen
> for
> various reasons:
> If COM+ objects are being leaked. If there is a memory leak, you should
> use
> DebugDiag to monitor the leak for
>
> root cause. See the "Troubleshooting Memory Leaks with DebugDiag" topic in
> the DebugDiag.chm file for more
>
> information.
> ...
>
>
>
> Warning 3 client connection(s) in
> inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
>
> Dump.dmp have been executing a request for more than 90 seconds. Please
> see
> the Client Connections section of
>
> this report for more detailed information about the connection(s).
> Information
> DebugDiag did not detect any known problems in H:\Logs\IIS Diag
> analysis\Prod Memory Dumps
>
> 08_16_2006_Outage\aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS
> Hang Dump.dmp using the
>
> current set of scripts.
>
>
>
>
>
> Analysis Details
> Table Of Contents
> dllhost.exe__System
> Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
> Dump.dmp
> Top 5 threads by CPU time
> Thread report
> Well-Known COM STA Threads Report
>
> inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
> Dump.dmp
> Top 5 threads by CPU time
> Locked critical section report
> Thread report
> Well-Known COM STA Threads Report
> HTTP Report
> ASP Report
>
> aspnet_wp.exe__PID__1040__Date__08_16_2006__Time_11_13_16AM__921__IIS Hang
> Dump.dmp
> Top 5 threads by CPU time
> Thread report
> Well-Known COM STA Threads Report
>
>
> Report for dllhost.exe__System
> Application__PID__3020__Date__08_16_2006__Time_11_13_17AM__984__IIS Hang
> Dump.dmp
> Type of Analysis Performed Hang Analysis
> Machine Name NLGPWEBIIS05
> Operating System Windows 2000 Service Pack 4
> Number Of Processors 4
> Process ID 3020
> Process Image C:\WINNT\system32\dllhost.exe
> System Up-Time 1 day(s) 05:05:30
> Process Up-Time 1 day(s) 05:04:28
>
>
> Top 5 Threads by CPU time
> Note - Times include both user mode and kernel mode for each thread Thread
> ID: 5 Total CPU Time: 0 day(s)
>
> 00:00:05.562 Entry Point for Thread: msvcrt!_endthreadex+32
> Thread ID: 6 Total CPU Time: 0 day(s) 00:00:00.750 Entry Point for
> Thread: rpcrt4!ThreadStartRoutine
> Thread ID: 4 Total CPU Time: 0 day(s) 00:00:00.31 Entry Point for
> Thread: msvcrt!_endthread+3f
> Thread ID: 7 Total CPU Time: 0 day(s) 00:00:00.31 Entry Point for
> Thread:
>
> comsvcs!CEventServer::DispatchEvents
> Thread ID: 3 Total CPU Time: 0 day(s) 00:00:00.15 Entry Point for
> Thread: txfaux!WORK_QUEUE::ThreadLoop
>
>
>
> Thread report
>
> Thread 0 - System ID 3016
> Entry point dllhost!WinMainCRTStartup
> Create time 08/15/2006 6:08:50 AM
> Time spent in user mode 0 Days 0:0:0.0
> Time spent in kernel mode 0 Days 0:0:0.15
>
>
> This thread is not fully resolved and may or may not be a problem. Further
> analysis of these threads may be
>
> required.
>
> Function Source
> NTDLL!ZwWaitForSingleObject+b
> KERNEL32!WaitForSingleObjectEx+71
> KERNEL32!WaitForSingleObject+f
>
>
>
> ...
>
>
> ...
>
> Thread 7 - System ID 3060
> Entry point
From: Juan on
Hi Pat,

Thanks a lot for your points of view. That was the type of expert opinnion
that we were missing.
I have not enough words to thank you for your help on this!

We had the MDAC upgrade in our list but now we are going to prioritize it.

I wasn't awre at all about the COM+ fix/rollup, we are going to implement
it. Is it this one? http://support.microsoft.com/?kbid=912816

We are going to work with the developers tracing down those areas of the
code involved in the blocked threads.

Regarding your point 1 (no attribution to long ASPs/many includes), the only
reason why we believe that it might be a contributor is because in the Crash
and Memory reports that we sent you and in many others there are a lot of ASP
Cache related functions and modules ranked high in memory leak probability,
examples:

We find this in many dump reports:

....
Memory allocations by the asp.dll template cache manager are responsible for
297.92 MBytes worth of outstanding allocations.

Large amounts of memory allocated for the ASP Template Cache are usually
caused by the usage of too much VBScript or JScript within a page (or it's
associated include files), and may result in degraded ASP performance due to
heap fragmentation or memory consumption by the host process.

ASP Template Cache Size: 236.70 MBytes

This was detected in
inetinfo__PID__2192__Date__08_14_2006__Time_03_32_04PM__15__Second_Chance_Exception_C0000005.dmp
....
....
inetinfo.exe__PID__3428__Date__08_11_2006__Time_07_47_45AM
The heap 0x02bb0000 contains 64 segments, with very little committed memory
in the heap. Any allocation request for this heap greater than 32.00 MBytes
will fail.

Memory statistics for this heap:

Reserved Memory: 434.23 MBytes
Committed Memory: 10.30 MBytes (2.37% of reserved)
Heap Name: ALLOC_CACHE_HANDLER heap for ASP:ResponseBuffer

Function asp!CTemplate::LargeReAlloc+1c
Allocation type Heap allocation(s)
Heap handle 0x02c50000
Allocation Count 248 allocation(s)
Allocation Size 314.80 MBytes
Leak Probability 93%

Function vbscript!DllCanUnloadNow+1c537
Allocation type C/C++ runtime allocation(s)
Allocation Count 47 allocation(s)
Allocation Size 38.37 MBytes
Leak Probability 95%
....

or

....
Function asp!CTemplate::LargeReAlloc+1c
Allocation type Heap allocation(s)
Heap handle 0x02e40000
Allocation Count 48 allocation(s)
Allocation Size 52.49 MBytes
Leak Probability 98%
....

Additionally, I see the performance counter ASP Template Cache Misses
growing exponentially minutes or hours before the hangs. While in normal
scenarios it grows smoothly towards the recycling points.

Please see the following doc for additional evidence:
http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc

Also I read in the IIS 5.0 Resource documentation that the delta between the
two following performance counters should be small or inexistent, but in our
case is pretty big:

Maximum File Cache Memory Usage and Current File Cache Memory Usage.

Please see the following doc for additional evidence:
http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc

Thanks a lot again!

--
Juan


"Pat [MSFT}" wrote:

> 1) There is _no_ indication whatsoever that page size/include count is a
> factor.
> 2) You have at least one thread blocked trying to create an object - there
> could be valid reasons for this; consider it a symptom not a cause.
> 3) You have a bunch of threads blocked on thread 112 - has most likely
> thrown an exception & leaked a lock which is leading to a hang. I would say
> (based on the blocked threads) that the most likely culprit is MDAC. I
> would update that first and see if it helps. The #2 culprit is COM+ (you
> would need to contact MS-Support for the latest COM+ rollup).
>
>
> Pat
>
>
>
> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> news:E97D51CF-9FB6-4F16-B7A1-BCCA3A28FA05(a)microsoft.com...
> > Hi Bernard,
> >
> > Thanks for your quick answer!
> >
> > Below you'll see the Crash/Hanhg analysis report. I will send the Memory
> > analysis in a separated message.
> >
> > A little bit more of background:
> >
> > Using MS Web Application Stress Tool in a paralel environment we can
> > stress
> > the the apps getting hang rule dumps but we cannot really hang the service
> > as
> > in the production incidents. The load generated in our WAS scripts is not
> > as
> > wide and realistic as the prod load.
> >
> > Thanks a lot
> > --
> > Juan
> >
> >
> > Analysis Summary
> > Type Description Recommendation
> > Warning Detected possible blocking or leaked critical section at
> > NTDLL!LoaderLock owned by thread 112 in
> >
> > inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
> > Dump.dmp
> >
> > Impact of this lock
> >
> > 71.88% of executing ASP Requests blocked
> >
> > 19.71% of threads blocked
> >
> > (Threads 30 52 55 56 59 66 68 74 75 76 77 79 87 101 103 104 106 107 115
> > 116
> > 117 122 124 128 130 132 136)
> >
> > The following functions are trying to enter this critical section
> > NTDLL!LdrpGetProcedureAddress+122
> > NTDLL!LdrUnloadDll+5f
> > KERNEL32!GetModuleFileNameW+ad
> > NTDLL!LdrShutdownThread+38
> >
> > The following module(s) are involved with this critical section
> > C:\WINNT\system32\NTDLL.DLL from Microsoft Corporation
> > C:\WINNT\system32\KERNEL32.DLL from Microsoft Corporation
> > The following vendors were identified for follow up based on root cause
> > analysis
> >
> > ...
> >
> > Warning The following threads in
> > inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
> >
> > Dump.dmp are processing a client request and is/are not fully resolved.
> > Further analysis of these threads is
> >
> > recommended in order to determine what may be blocking the request(s).
> >
> > ( 0 32 78 81 105 109 113 118 123 127 )
> >
> > 28.13% of executing ASP Requests blocked
> >
> > 6.25% of IIS worker threads blocked
> >
> > 7.30% of threads blocked
> >
> >
> > ...
> >
> > Warning The COM+ STA ThreadPool may have been depleted prior to the time
> > of the dump in
> >
> > inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS Hang
> > Dump.dmp. There is more than one
> >
> > activity bound to the following COM+ STA ThreadPool threads:
> >
> > 5
From: Pat [MSFT} on
Having a large template cache can cause performance problems - basically
there is less memory available for the app to run. So it isn't optimal but
depending on how you are running the code it may not be that big a deal. I
would take it under advisement, but not worry too much about it.

Pat


"Juan" <Juan(a)discussions.microsoft.com> wrote in message
news:85DBE238-BA1F-4BC8-BCA8-5981CEEF5EBC(a)microsoft.com...
> Hi Pat,
>
> Thanks a lot for your points of view. That was the type of expert opinnion
> that we were missing.
> I have not enough words to thank you for your help on this!
>
> We had the MDAC upgrade in our list but now we are going to prioritize it.
>
> I wasn't awre at all about the COM+ fix/rollup, we are going to implement
> it. Is it this one? http://support.microsoft.com/?kbid=912816
>
> We are going to work with the developers tracing down those areas of the
> code involved in the blocked threads.
>
> Regarding your point 1 (no attribution to long ASPs/many includes), the
> only
> reason why we believe that it might be a contributor is because in the
> Crash
> and Memory reports that we sent you and in many others there are a lot of
> ASP
> Cache related functions and modules ranked high in memory leak
> probability,
> examples:
>
> We find this in many dump reports:
>
> ...
> Memory allocations by the asp.dll template cache manager are responsible
> for
> 297.92 MBytes worth of outstanding allocations.
>
> Large amounts of memory allocated for the ASP Template Cache are usually
> caused by the usage of too much VBScript or JScript within a page (or it's
> associated include files), and may result in degraded ASP performance due
> to
> heap fragmentation or memory consumption by the host process.
>
> ASP Template Cache Size: 236.70 MBytes
>
> This was detected in
> inetinfo__PID__2192__Date__08_14_2006__Time_03_32_04PM__15__Second_Chance_Exception_C0000005.dmp
> ...
> ...
> inetinfo.exe__PID__3428__Date__08_11_2006__Time_07_47_45AM
> The heap 0x02bb0000 contains 64 segments, with very little committed
> memory
> in the heap. Any allocation request for this heap greater than 32.00
> MBytes
> will fail.
>
> Memory statistics for this heap:
>
> Reserved Memory: 434.23 MBytes
> Committed Memory: 10.30 MBytes (2.37% of reserved)
> Heap Name: ALLOC_CACHE_HANDLER heap for ASP:ResponseBuffer
>
> Function asp!CTemplate::LargeReAlloc+1c
> Allocation type Heap allocation(s)
> Heap handle 0x02c50000
> Allocation Count 248 allocation(s)
> Allocation Size 314.80 MBytes
> Leak Probability 93%
>
> Function vbscript!DllCanUnloadNow+1c537
> Allocation type C/C++ runtime allocation(s)
> Allocation Count 47 allocation(s)
> Allocation Size 38.37 MBytes
> Leak Probability 95%
> ...
>
> or
>
> ...
> Function asp!CTemplate::LargeReAlloc+1c
> Allocation type Heap allocation(s)
> Heap handle 0x02e40000
> Allocation Count 48 allocation(s)
> Allocation Size 52.49 MBytes
> Leak Probability 98%
> ...
>
> Additionally, I see the performance counter ASP Template Cache Misses
> growing exponentially minutes or hours before the hangs. While in normal
> scenarios it grows smoothly towards the recycling points.
>
> Please see the following doc for additional evidence:
> http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc
>
> Also I read in the IIS 5.0 Resource documentation that the delta between
> the
> two following performance counters should be small or inexistent, but in
> our
> case is pretty big:
>
> Maximum File Cache Memory Usage and Current File Cache Memory Usage.
>
> Please see the following doc for additional evidence:
> http://www.compilar.com/garage/ASPTemplateCacheEvidence.doc
>
> Thanks a lot again!
>
> --
> Juan
>
>
> "Pat [MSFT}" wrote:
>
>> 1) There is _no_ indication whatsoever that page size/include count is a
>> factor.
>> 2) You have at least one thread blocked trying to create an object -
>> there
>> could be valid reasons for this; consider it a symptom not a cause.
>> 3) You have a bunch of threads blocked on thread 112 - has most likely
>> thrown an exception & leaked a lock which is leading to a hang. I would
>> say
>> (based on the blocked threads) that the most likely culprit is MDAC. I
>> would update that first and see if it helps. The #2 culprit is COM+ (you
>> would need to contact MS-Support for the latest COM+ rollup).
>>
>>
>> Pat
>>
>>
>>
>> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
>> news:E97D51CF-9FB6-4F16-B7A1-BCCA3A28FA05(a)microsoft.com...
>> > Hi Bernard,
>> >
>> > Thanks for your quick answer!
>> >
>> > Below you'll see the Crash/Hanhg analysis report. I will send the
>> > Memory
>> > analysis in a separated message.
>> >
>> > A little bit more of background:
>> >
>> > Using MS Web Application Stress Tool in a paralel environment we can
>> > stress
>> > the the apps getting hang rule dumps but we cannot really hang the
>> > service
>> > as
>> > in the production incidents. The load generated in our WAS scripts is
>> > not
>> > as
>> > wide and realistic as the prod load.
>> >
>> > Thanks a lot
>> > --
>> > Juan
>> >
>> >
>> > Analysis Summary
>> > Type Description Recommendation
>> > Warning Detected possible blocking or leaked critical section at
>> > NTDLL!LoaderLock owned by thread 112 in
>> >
>> > inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS
>> > Hang
>> > Dump.dmp
>> >
>> > Impact of this lock
>> >
>> > 71.88% of executing ASP Requests blocked
>> >
>> > 19.71% of threads blocked
>> >
>> > (Threads 30 52 55 56 59 66 68 74 75 76 77 79 87 101 103 104 106 107 115
>> > 116
>> > 117 122 124 128 130 132 136)
>> >
>> > The following functions are trying to enter this critical section
>> > NTDLL!LdrpGetProcedureAddress+122
>> > NTDLL!LdrUnloadDll+5f
>> > KERNEL32!GetModuleFileNameW+ad
>> > NTDLL!LdrShutdownThread+38
>> >
>> > The following module(s) are involved with this critical section
>> > C:\WINNT\system32\NTDLL.DLL from Microsoft Corporation
>> > C:\WINNT\system32\KERNEL32.DLL from Microsoft Corporation
>> > The following vendors were identified for follow up based on root cause
>> > analysis
>> >
>> > ...
>> >
>> > Warning The following threads in
>> > inetinfo.exe__PID__5620__Date__08_16_2006__Time_11_13_18AM__296__IIS
>> > Hang
>> >
>> > Dump.dmp