Resolve an application hang using a LeanSentry hang diagnostic report

In this article, we will be using the information that we gathered from LeanSentry's hang diagnostic report to resolve a sample application hang.

You can learn about finding and using the LeanSentry hang diagnostic reports in Find Application Hangs using LeanSentry and Determine the cause of a hang using LeanSentry's hang diagnostics.

Cause of the hang

To recap, in part 2: Determine the cause of a hang using LeanSentry's hang diagnostics we had determined the following cause for the hang:

• The DownloadAllArticles function used Parallel.ForEach to call the DownloadArticle function parallely using several threads from the CLR thread pool.
• The DownloadArticle function called WebClient.DownloadString and blocked the thread while performing an IO operation (making an HTTP request) synchronously.
• Since Parallel.ForEach does not return until all operations have been completed, the requests hang. Every new request that calls the DowloadAllArticles function causes more and more of these threads to be blocked (by making an HTTP call) ultimately causing the thread pool exhaustion and thereby causing the application to hang.
• The code causing the hang is at Line 26 (part of DownloadAllArticles function) and Line 34 (part of DownloadArticle function) of the DownloadArticlesController.cs file 

Locating the issue in the source code

LeanSentry has already identified that the code causing the hang is at line 26 (DownloadAllArticles function):



and line 34 (DownloadArticle function) of the DownloadArticlesController.cs.

This validates our findings in the report. Here is how each request contributes to thread pool exhaustion.

Making the fix

The last step in the process is to make the fix. We know that the threads inside DownloadArticle function are making a blocking HTTP call. The correct long term fix here is to make the HTTP calls asynchronous. This will allow the threads to be returned to the pool while the HTTP calls are being completed.    

Caching: This fixes the issue of thread pool exhaustion. We can also implement a cache to further improve performance. The cache is implemented using a simple Dictionary with the url as the key. If we find a url that has been downloaded recently, we return it from the cache else we download and save it in the cache before returning.

The final step in the process is to publish the fix to the servers being monitored by LeanSentry. We should not see the same hangs come up and the performance of the application should be much better especially during high load times.

Conclusion

In this series, we've explored how to succesfully use LeanSentry Hang diagnostics to discover, understand, and resolve an application hang:

1. Find Application Hangs using LeanSentry.

2. Determine the cause of a hang using LeanSentry's hang diagnostics.

3. Resolve an application hang using a LeanSentry hang diagnostic report.

While all applications are different, and some hangs are more complex than others, you can normally follow this process to diagnose and resolve most website hangs.

If you have a more complex hang that requires additional analysis, contact our support team for additional assistance options.