IIS 5 Hang [IIS]

Prev: W3SVC and .NET runtime crash help
Next: ASP does not work on IIS

From: Juan on 17 Aug 2006 16:56

Hi Benard,

More data:

On production, during the hangs, Inetinfo process is in 0% CPU utilization.
On the testing environment when we stress the apps (we cannot hang it) the
CPU utilization for Inetinfo goes up to 80 - 90 percent.

The server memory utilization oscilates between 400 to 600 MB out of the 2
GB of RAM.

We have spikes of ~1000 sessions and the original developers decided to
store more than a simple id there. I do not know the average bytes that each
session may take but they are kind of fat sessions (all string data).

Internet Services Global - File Cache Misses counter spikes before the
incidents (this indicator increases quickly and exponentially minutes before
the hangs).

The file chache hits % counter oscilates between 55 and 85 %.

The Curent File Cache Memory Usage counter normally moves from 400,000 to
800,000 bytes (are they bytes? Kbytes?) but time to time it grows up to
1,800,00 bytes (bytes? kbytes?). Are these bytes out of the physcal RAM?
Virtual Memory? I am confused about this indicator.

The MAXIMUM File Cache Memory Usage counter, a sort of correlated indicator
to the CURRENT is always much higher (between 3,000,000 and 5,000,000). Based
on the following article that should be really bad, a direct recipy for leaks:

"MAXIMUM File Cache Memory Usage: The value of this counter and the value of
Web Service Cache\Current File Cache Memory Usage should be the same. If the
numbers are different, the cache was flushed. If this counter is
significantly higher than Web Service Cache\Current File Cache Memory Usage,
you might consider recycling the worker process because the application might
have a memory leak."

Source:
http://www.microsoft.com/technet/prodtechnol/WindowsServer2003/Library/IIS/a0483502-c6da-486a-917a-586c463b7ed6.mspx?mfr=true

The number of 500 Internal server errors also grows steady towards the hang
points. It drops to 0 after recycling, and it immediately starts growing
steady again.

Thanks, once again..
--
Juan

"Bernard Cheah [MVP]" wrote:

> You can start with the html and provide a link to the dump file if needed.
>
> --
> Regards,
> Bernard Cheah
> http://www.iis.net/
> http://www.iis-resources.com/
> http://msmvps.com/blogs/bernard/
>
>
> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com...
> > Hi,
> >
> > I wonder if you guys can give us a hand with our analysis of this IIS
> > hangs.
> >
> > We are running 6 in-process ASP websites (only 2 of them have 20+ virtual
> > dirs / logical apps) the other four are pretty light. The 2 main websites
> > have several hundreds of ASP pages.
> >
> > A little bit more than a thousand unique visitors per day.
> >
> > Connections to SQL 2000, DB2, and MQ.
> >
> > The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and 2GB
> > RAM.
> >
> > The inetinfo hangs randomly (no-load related) with an average of 2/3 times
> > a
> > week. Recycling IIS the problem goes away.
> >
> > We are using IIS 5 Process Recycle on daily basis. No too much help.
> >
> > We are using IIS Diagnostic Tools to obtain and analyze memory dumps.
> >
> > Based on your incredible IIS Diagnostic Tools reports, we have the theory
> > that the root cause of our problems is the large number of big pages and
> > the
> > extensive number of include files (more than 100 includes in some pages).
> > We
> > think that they produce heap fragmentation in the ASP template cache that
> > materializes in memory lacks.
> >
> > We have more than 2 months analyzing the symptoms, and we obtained some
> > interesting stuff, but the advise of the experts will help us a lot
> > confirming our theories or giving us new leads.
> >
> > Can I paste in a next message either dump files or IIS diagnostics HTML
> > reports for your opinion?
> >
> > Your help will be really appreciated.
> >
> > Thanks a lot,
> >
> > --
> > Juan
>
>
>

From: Bernard Cheah [MVP] on 18 Aug 2006 04:21

I'm no expert in analyzing the log. from the summary - it looks like lot of
blocking among process threads.
Have you start looking at some of the 'recommendation' steps?

--
Regards,
Bernard Cheah
http://www.iis.net/
http://www.iis-resources.com/
http://msmvps.com/blogs/bernard/

"Juan" <Juan(a)discussions.microsoft.com> wrote in message
news:02E6B5CD-F3F2-4A25-8DF7-1138FC7D4247(a)microsoft.com...
> Hi Bernard,
>
> Here are the links to the full IIS Diagnostics tool reports.
>
> The actual dumps are more than 100 MB large, so I cannot upload them by
> the
> time being, let me see what I can do.
>
> http://www.compilar.com/garage/IIS_Report__Date_08_17_2006__Time_08_18_50AM__117.htm
>
> http://www.compilar.com/garage/Memory_Report__Date_08_17_2006__Time_09_04_50AM__215.htm
>
> Comment: In other outages and high stress dumps, I got much more leak
> probability occurencies and higher percentages than in this memory report
> sample.
>
> Thanks again.
>
> --
> Juan
>
>
> "Bernard Cheah [MVP]" wrote:
>
>> You can start with the html and provide a link to the dump file if
>> needed.
>>
>> --
>> Regards,
>> Bernard Cheah
>> http://www.iis.net/
>> http://www.iis-resources.com/
>> http://msmvps.com/blogs/bernard/
>>
>>
>> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
>> news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com...
>> > Hi,
>> >
>> > I wonder if you guys can give us a hand with our analysis of this IIS
>> > hangs.
>> >
>> > We are running 6 in-process ASP websites (only 2 of them have 20+
>> > virtual
>> > dirs / logical apps) the other four are pretty light. The 2 main
>> > websites
>> > have several hundreds of ASP pages.
>> >
>> > A little bit more than a thousand unique visitors per day.
>> >
>> > Connections to SQL 2000, DB2, and MQ.
>> >
>> > The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and 2GB
>> > RAM.
>> >
>> > The inetinfo hangs randomly (no-load related) with an average of 2/3
>> > times
>> > a
>> > week. Recycling IIS the problem goes away.
>> >
>> > We are using IIS 5 Process Recycle on daily basis. No too much help.
>> >
>> > We are using IIS Diagnostic Tools to obtain and analyze memory dumps.
>> >
>> > Based on your incredible IIS Diagnostic Tools reports, we have the
>> > theory
>> > that the root cause of our problems is the large number of big pages
>> > and
>> > the
>> > extensive number of include files (more than 100 includes in some
>> > pages).
>> > We
>> > think that they produce heap fragmentation in the ASP template cache
>> > that
>> > materializes in memory lacks.
>> >
>> > We have more than 2 months analyzing the symptoms, and we obtained some
>> > interesting stuff, but the advise of the experts will help us a lot
>> > confirming our theories or giving us new leads.
>> >
>> > Can I paste in a next message either dump files or IIS diagnostics HTML
>> > reports for your opinion?
>> >
>> > Your help will be really appreciated.
>> >
>> > Thanks a lot,
>> >
>> > --
>> > Juan
>>
>>
>>

From: Bernard Cheah [MVP] on 18 Aug 2006 04:21

From: Juan on 18 Aug 2006 09:34

Hi Bernard,

Thanks for your answer.

Yes, we are implementing some of the recommendations. However we wanted to
be sure that we undersatand the root cause for the hangs. Based on the report
the things going wrong seem to be:

1) The high heap fragmentations --> causing memory leaks <-- produced by too
many includes
2) Long waits and/or blocking in several threads <-- caused by bottle necks?
(db conns, etc.?)

Is the first reason alone enough to hang the Inetinfo process? If that is
the case how would it be the sequence of actions towards the hang point? Is
the case that IIS/ASP "chocks" after too much i/o traffic of huge ASPs with
100s of includes in the ASP Template Cache?

Regarding the second point, Do you see a single failure point hanging the
server? Or is more that the sum of several waiting points and blocked thread
ends up (again) "chocking" the Inetinfo process. Can a sum of waiting points
and blocked threads hang the process? Or a hang has to be caused by a single
point of failure (deadlock, spin, infinite recursion, etc.) ?

I will try to find an FTP place in which I can put the 100+ MB memory dump
files for you.
Impressive! If you can read the dumps better than the reports that is
amazing, it speaks about the level of your expertise in the subject!

While I find an FTP place (which can take me a while) would it be possible
in parallel to have any of the guys there accustomed to read these reports
taking a look to them?

I will appreciate your answers and I will send you the dumps ASAP.

Thanks a lot for your time!

--
Juan

"Bernard Cheah [MVP]" wrote:

> I'm no expert in analyzing the log. from the summary - it looks like lot of
> blocking among process threads.
> Have you start looking at some of the 'recommendation' steps?
>
> --
> Regards,
> Bernard Cheah
> http://www.iis.net/
> http://www.iis-resources.com/
> http://msmvps.com/blogs/bernard/
>
>
> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> news:02E6B5CD-F3F2-4A25-8DF7-1138FC7D4247(a)microsoft.com...
> > Hi Bernard,
> >
> > Here are the links to the full IIS Diagnostics tool reports.
> >
> > The actual dumps are more than 100 MB large, so I cannot upload them by
> > the
> > time being, let me see what I can do.
> >
> > http://www.compilar.com/garage/IIS_Report__Date_08_17_2006__Time_08_18_50AM__117.htm
> >
> > http://www.compilar.com/garage/Memory_Report__Date_08_17_2006__Time_09_04_50AM__215.htm
> >
> > Comment: In other outages and high stress dumps, I got much more leak
> > probability occurencies and higher percentages than in this memory report
> > sample.
> >
> > Thanks again.
> >
> > --
> > Juan
> >
> >
> > "Bernard Cheah [MVP]" wrote:
> >
> >> You can start with the html and provide a link to the dump file if
> >> needed.
> >>
> >> --
> >> Regards,
> >> Bernard Cheah
> >> http://www.iis.net/
> >> http://www.iis-resources.com/
> >> http://msmvps.com/blogs/bernard/
> >>
> >>
> >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
> >> news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com...
> >> > Hi,
> >> >
> >> > I wonder if you guys can give us a hand with our analysis of this IIS
> >> > hangs.
> >> >
> >> > We are running 6 in-process ASP websites (only 2 of them have 20+
> >> > virtual
> >> > dirs / logical apps) the other four are pretty light. The 2 main
> >> > websites
> >> > have several hundreds of ASP pages.
> >> >
> >> > A little bit more than a thousand unique visitors per day.
> >> >
> >> > Connections to SQL 2000, DB2, and MQ.
> >> >
> >> > The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and 2GB
> >> > RAM.
> >> >
> >> > The inetinfo hangs randomly (no-load related) with an average of 2/3
> >> > times
> >> > a
> >> > week. Recycling IIS the problem goes away.
> >> >
> >> > We are using IIS 5 Process Recycle on daily basis. No too much help.
> >> >
> >> > We are using IIS Diagnostic Tools to obtain and analyze memory dumps.
> >> >
> >> > Based on your incredible IIS Diagnostic Tools reports, we have the
> >> > theory
> >> > that the root cause of our problems is the large number of big pages
> >> > and
> >> > the
> >> > extensive number of include files (more than 100 includes in some
> >> > pages).
> >> > We
> >> > think that they produce heap fragmentation in the ASP template cache
> >> > that
> >> > materializes in memory lacks.
> >> >
> >> > We have more than 2 months analyzing the symptoms, and we obtained some
> >> > interesting stuff, but the advise of the experts will help us a lot
> >> > confirming our theories or giving us new leads.
> >> >
> >> > Can I paste in a next message either dump files or IIS diagnostics HTML
> >> > reports for your opinion?
> >> >
> >> > Your help will be really appreciated.
> >> >
> >> > Thanks a lot,
> >> >
> >> > --
> >> > Juan
> >>
> >>
> >>
>
>
>

From: Bernard Cheah [MVP] on 19 Aug 2006 01:52

I'm not the person that you should send the dump file :)
If this is urgent, you can open a support case with Microsoft PSS.
if not, you can post a new thread with detail link to the log and dump file,
Attention it to 'Pat', this guy is an expert, me is not :( Anyway, in your
case the hang looks like from long running request that holding all
subsequence request, deadlock could occur as well in this case. Sorry that
I can't be helpful.

--
Regards,
Bernard Cheah
http://www.iis.net/
http://www.iis-resources.com/
http://msmvps.com/blogs/bernard/

"Juan" <Juan(a)discussions.microsoft.com> wrote in message
news:18AF64DE-D93C-4A18-87D3-E93F694F7855(a)microsoft.com...
> Hi Bernard,
>
> Thanks for your answer.
>
> Yes, we are implementing some of the recommendations. However we wanted to
> be sure that we undersatand the root cause for the hangs. Based on the
> report
> the things going wrong seem to be:
>
> 1) The high heap fragmentations --> causing memory leaks <-- produced by
> too
> many includes
> 2) Long waits and/or blocking in several threads <-- caused by bottle
> necks?
> (db conns, etc.?)
>
> Is the first reason alone enough to hang the Inetinfo process? If that is
> the case how would it be the sequence of actions towards the hang point?
> Is
> the case that IIS/ASP "chocks" after too much i/o traffic of huge ASPs
> with
> 100s of includes in the ASP Template Cache?
>
> Regarding the second point, Do you see a single failure point hanging the
> server? Or is more that the sum of several waiting points and blocked
> thread
> ends up (again) "chocking" the Inetinfo process. Can a sum of waiting
> points
> and blocked threads hang the process? Or a hang has to be caused by a
> single
> point of failure (deadlock, spin, infinite recursion, etc.) ?
>
> I will try to find an FTP place in which I can put the 100+ MB memory dump
> files for you.
> Impressive! If you can read the dumps better than the reports that is
> amazing, it speaks about the level of your expertise in the subject!
>
> While I find an FTP place (which can take me a while) would it be possible
> in parallel to have any of the guys there accustomed to read these reports
> taking a look to them?
>
> I will appreciate your answers and I will send you the dumps ASAP.
>
> Thanks a lot for your time!
>
> --
> Juan
>
>
> "Bernard Cheah [MVP]" wrote:
>
>> I'm no expert in analyzing the log. from the summary - it looks like lot
>> of
>> blocking among process threads.
>> Have you start looking at some of the 'recommendation' steps?
>>
>> --
>> Regards,
>> Bernard Cheah
>> http://www.iis.net/
>> http://www.iis-resources.com/
>> http://msmvps.com/blogs/bernard/
>>
>>
>> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
>> news:02E6B5CD-F3F2-4A25-8DF7-1138FC7D4247(a)microsoft.com...
>> > Hi Bernard,
>> >
>> > Here are the links to the full IIS Diagnostics tool reports.
>> >
>> > The actual dumps are more than 100 MB large, so I cannot upload them by
>> > the
>> > time being, let me see what I can do.
>> >
>> > http://www.compilar.com/garage/IIS_Report__Date_08_17_2006__Time_08_18_50AM__117.htm
>> >
>> > http://www.compilar.com/garage/Memory_Report__Date_08_17_2006__Time_09_04_50AM__215.htm
>> >
>> > Comment: In other outages and high stress dumps, I got much more leak
>> > probability occurencies and higher percentages than in this memory
>> > report
>> > sample.
>> >
>> > Thanks again.
>> >
>> > --
>> > Juan
>> >
>> >
>> > "Bernard Cheah [MVP]" wrote:
>> >
>> >> You can start with the html and provide a link to the dump file if
>> >> needed.
>> >>
>> >> --
>> >> Regards,
>> >> Bernard Cheah
>> >> http://www.iis.net/
>> >> http://www.iis-resources.com/
>> >> http://msmvps.com/blogs/bernard/
>> >>
>> >>
>> >> "Juan" <Juan(a)discussions.microsoft.com> wrote in message
>> >> news:DD07C69E-7F44-4427-B9D8-A3749F988471(a)microsoft.com...
>> >> > Hi,
>> >> >
>> >> > I wonder if you guys can give us a hand with our analysis of this
>> >> > IIS
>> >> > hangs.
>> >> >
>> >> > We are running 6 in-process ASP websites (only 2 of them have 20+
>> >> > virtual
>> >> > dirs / logical apps) the other four are pretty light. The 2 main
>> >> > websites
>> >> > have several hundreds of ASP pages.
>> >> >
>> >> > A little bit more than a thousand unique visitors per day.
>> >> >
>> >> > Connections to SQL 2000, DB2, and MQ.
>> >> >
>> >> > The server: W2K SP4 + updates, IIS 5.0, ADO 2.6, 4 Processors, and
>> >> > 2GB
>> >> > RAM.
>> >> >
>> >> > The inetinfo hangs randomly (no-load related) with an average of 2/3
>> >> > times
>> >> > a
>> >> > week. Recycling IIS the problem goes away.
>> >> >
>> >> > We are using IIS 5 Process Recycle on daily basis. No too much help.
>> >> >
>> >> > We are using IIS Diagnostic Tools to obtain and analyze memory
>> >> > dumps.
>> >> >
>> >> > Based on your incredible IIS Diagnostic Tools reports, we have the
>> >> > theory
>> >> > that the root cause of our problems is the large number of big pages
>> >> > and
>> >> > the
>> >> > extensive number of include files (more than 100 includes in some
>> >> > pages).
>> >> > We
>> >> > think that they produce heap fragmentation in the ASP template cache
>> >> > that
>> >> > materializes in memory lacks.
>> >> >
>> >> > We have more than 2 months analyzing the symptoms, and we obtained
>> >> > some
>> >> > interesting stuff, but the advise of the experts will help us a lot
>> >> > confirming our theories or giving us new leads.
>> >> >
>> >> > Can I paste in a next message either dump files or IIS diagnostics
>> >> > HTML
>> >> > reports for your opinion?
>> >> >
>> >> > Your help will be really appreciated.
>> >> >
>> >> > Thanks a lot,
>> >> >
>> >> > --
>> >> > Juan
>> >>
>> >>
>> >>
>>
>>
>>

First | Prev | Next | Last
Pages: 1 2 3 4
Prev: W3SVC and .NET runtime crash help
Next: ASP does not work on IIS