Troubleshooting an IIS Crash/Hang

This is an experience of mine at a client. It was a project about integrating a third party application with the client’s web application, so that the client could provide better service to its customers.

I have designed and implemented this ASP integration application and the project was deployed successfully. Everyone was very impressed with my work and I was happy.

I was thinking how well my design was and that all implementations should be as good as mine. how nice would it be if all the systems that are designed and implemented ran smoothly after a successful QA, with out ever crashing or needing any monitoring.

Phone rings !!!!

Wake up Caveman!!! Wake up !!! Ohhhhhhhh Shoot!! it was only a dreammmm….. damn-it!!

I woke up from my dream with the annoying ring of the phone. I got this call from the support groupMe: Hello, Me here

Support Guy(SG): Hi this is SG from the Support group, how are you?

Me: Good, How about you?

SG: alrite

Me: What’s up?

SG: The website is hanging intermittently and we are having to do an IISReset to bring back the site.

Me: hmm… Is there a pattern to this issue?

SG: The CSR Manager said that this happens randomly and that the business is getting effected because of the outage.

Me: Okay I will take a look at this and get back to you.

SG: This priority 1 issue.

Me: (I thought of shouting at him: so what!!! hold on !!! I will get to it when I can get to it) Thanks for letting me know this…. and the call ends.

This is the time when I was scratching my head about what could possibly have gone wrong, that caused the website to hang. My first instinct told me that the third party component might be the culprit (as later turned out to be) as I did not design/code it, heheheheeeee.

There could be several reasons like some of the following that can cause an application failure.

1. Network Issues.

2. Too many database connections.

3. Unreasonable CPU utilization.

4. Disk access errors.

5. Web Service failures.

6. Erroneous third party components.

7. Memory Leaks.

8. Threading issues, etc…

I would usually start with checking the health of the Web application that includes (but not limited to) checking the following:

• No. of Database Connections.
• CPU on Web Server and Database Servers..
• Event viewers on all servers.
• Memory consumed by Dllhosts or Worker (w3wp.exe) processes.
• Web Service call durations.

At times this might also not help and all you would notice is a blip on the radar that does not tell you much, like an iisreset has been automated and that a crash has occurred.

How would you know the cause of the crash/hang?

IIS State is a command line utility, that is a part of the IIS 6.0 Resource toolkit, that is a very handy to diagnoize IIS related issues.To attach IISState to a particular w3wp.exe process execute the following command (where <PID> is the Process ID). This will do an immediate dump of the current process.

iisstate -p <PID>
IISState also supports the following optional switches:

• -sc(waits for a “soft crash” such as an ASP 0115 Trappable Error Occured in an External Object)
• -hc (waits for a “hard crash” where the process terminates unexpectedly)
• -d(write out a dump file, which can be used for further analysis, e.g. by WinDBG)

IISState outputs a logfile containing the stacks of all the threads in the process. I used the IISState utility to get a dump file and a logfile by hooking it to the only w3wp.exe process. I got lucky and the crash happened in a little time. Upon examining the dump I have noticed that the one of the threads was waiting on another thread to get its job done and that, that thread was waiting on another thread. I traversed through a bunch of threads to finally derive at a thread that was the culprit. I was able to figure out that this thread belonged to a third part dll as mentioned in the thread info.

I have checked with the company that owned the dll and found out that they had released a newer version that took care of the thread issue..

Another way of diagnosing the issue would be by further analyzing the dump file with utility like DebugDiag, WinDBG. For this method of diagnosis you will need the .pdb files of the application and the necessary symbols.

Useful tools:
1. IIS 6.0 Resource Kit Tools
2. Debug Diagnostic Tool (DebugDiag)
3. WinDBG

Singleton Design Pattern

The singleton pattern is implemented by creating a class with a method that creates a new instance of the class if one does not exist already. If an instance already exists, it simply returns a reference to that object. To make sure that the object cannot be instantiated in any other way, the constructor is made either private or protected. Note the distinction between a simple static instance of a class and a singleton: although a singleton can be implemented as a static instance, it can also be lazily constructed, requiring no memory or resources until needed. Another notable difference is that static member classes cannot implement an interface, unless that interface is simply a marker. So if the class has to realize a contract expressed by an interface, you really have to make it a singleton. [1]

The singleton pattern must be carefully constructed in a multi-threaded applications. If two threads are to execute the creation method at the same time when a singleton does not yet exist, they both must check for an instance of the singleton and then only one should create the new one. If the programming language has concurrent processing capabilities the method should be constructed to execute as a mutually exclusive operation.[1]

This is definitely not a very good pattern unless used properly. The reason being, the object generated from this class in essence a global object/variable and this is not a good thing in programming world.

Following is a sample implementation of this pattern in C#

Code Explained
The CSingleton class has a private constructor so that a new object of this class cannot be created outside of this class. Rather we create a static instance variable of CSingleton type so that the public CreateInstance method can return (or create and return, if not exists) an instance of this class.

What happens when two threads try to invoke the CreateInstance method at the same time?
In a multi threaded environment, there is a chance that a dead lock situation could occur and that the thread safety is not assured. In order to prevent this situation, we use the syncLock approach to lock on the instance creation mechanism, to avoid dead locks.

You can also check out the webcast on design patterns from MSDN [3]. This webcast looks at examples of the Observer, Factory, Composite, and Singleton patterns, and explains how these patterns work and where you would use them in your .NET applications. Following is a another video tutorial I found on youtube about Singleton Design pattern [2]

References:
1. Wikipedia - Singleton Pattern
2. Youtube - The Singleton Design Pattern - codingkriggs
3. MSDN - Webcast: Design Patterns in .NET (Level 200)

Version Tolerant Serialization

Recently we had a situation where the datatypes of a few properties in a extended commerce server 2007 class had to be changed. This class is serializable and a change to the datatypes would break compatibility with the data from the past. The data from the past has to be supported by the application for a period of about 3 months.

The challenge on hand is to make the changes to the class in such a way that changes would not make the application to break compatibility with the data form the past.

One solution that we have implemented was to change the type Property, add a new version of the private data member and retain the older data member. This gives us the flexibility of casting the old data into the new type and returning it via the property.

Version Tolerant Serialization (VTS) is a set of features introduced in .NET Framework 2.0 that makes it easier, over time, to modify serializable types. Specifically, the VTS features are enabled for classes to which the SerializableAttribute attribute has been applied. VTS makes it possible to add new fields to those classes without breaking compatibility with other versions of the type. [1]

The VTS features are enabled when using the BinaryFormatter. Additionally, all features except extraneous data tolerance are also enabled when using the SoapFormatter. For more information about using these classes for serialization, see Binary Serialization. [1]

The set of features includes the following [1]:

Tolerance of Extraneous or Unexpected Data

In the past, during deserialization, any extraneous or unexpected data caused exceptions to be thrown. With VTS, in the same situation, any extraneous or unexpected data is ignored instead of causing exceptions to be thrown. This enables applications that use newer versions of a type (that is, a version that includes more fields) to send information to applications that expect older versions of the same type.

Tolerance of Missing Data

Fields can be marked as optional by applying the OptionalFieldAttribute attribute to them. During deserialization, if the optional data is missing, the serialization engine ignores the absence and does not throw an exception. Thus, applications that expect older versions of a type can send data to applications that expect newer versions of the same type.

Serialization Callbacks

Serialization callbacks are a mechanism that provides hooks into the serialization/deserialization process at four points.

Using Callbacks

To use callbacks, apply the appropriate attribute to a method that accepts a StreamingContext parameter. Only one method per class can be marked with each of these attributes.

The VersionAdded Property

The OptionalFieldAttribute has the VersionAdded property. In version 2.0 of the .NET Framework, this is not used. However, it is important to set this property correctly to ensure that the type will be compatible with future serialization engines.

The property indicates which version of a type a given field has been added. It should be incremented by exactly one (starting at 2) every time the type is modified.

Best Practices [1]

1. To ensure proper versioning behavior, follow these rules when modifying a type from version to version:

2. Never remove a serialized field.

3. Never apply the NonSerializedAttribute attribute to a field if the attribute was not applied to the field in the previous version.

4. Never change the name or the type of a serialized field.

5. When adding a new serialized field, apply the OptionalFieldAttribute attribute.

6. When removing a NonSerializedAttribute attribute from a field (that was not serializable in a previous version), apply the OptionalFieldAttribute attribute.

7. For all optional fields, set meaningful defaults using the serialization callbacks unless 0 or null as defaults are acceptable.

To ensure that a type will be compatible with future serialization engines, follow these guidelines:

1. Always set the VersionAdded property on the OptionalFieldAttribute attribute correctly.

2. Avoid branched versioning.

References:
1. MSDN Online