I was checking my office email as always and saw a flood of email related to our production application. As I one by one went through them I realized that our application was constantly going down at around 12:00 AM for two days. Some tables were getting locked and the whole application was stuck.
When things like these happen the first thing to take a look at is the database connections and transaction managment code so we did that. Upon investigation it came out that we were making fresh database connections on every request, our business logic was broken down into several methods which were again getting fresh database connections which were not returned back to the database. To top it up we were also not utilizing any connection pool here because our CEO commented it out 😮
There were no try/catch/finally blocks to safeguard against any exceptional conditions so in case of exceptions the database connection was getting lost where we managed to return the connection. So all in all our transaction/database mgmt code was a mess.
Now how do we fix it up quickly without having any major impact on the performance? We came up with two tasks:
- To fix up the current code quickly.
- Make a proper fix for the next release.
For the quick fix we utilized the ThreadLocal concept. This is like a special singleton for a Thread. Lets say you want each thread to have its own copy of some state. One way to do it is to utilized some kind of a HashMap having the Thread-Id as the key and that state as the value. Infact the exact same concept was utilized in earlier versions of java for implementing ThreadLocal.
We can use ThreadLocal as that state. There is nothing special that we need to do. ThreadLocal is a special class in JVM and JVm takes care of all the specifics of making a new instance for each thread.
We utilized the ThreadLocal to store our database connection which we were getting from a C3P0 connection pool. Now whenever a method was asking for connection we returned the same connection until we were in the same thread and at the end the connection was closed explicitly. So we had to make some minor changes and all the instances were fixed.
I am proposing Spring as the solution for all this mess that we are in right now. I specially like its declarative transaction management and cahcing which really frees up the programmer from these mundane tasks so that the programmer only focuses on the core problem.
Let me know if anyone else has used ThreadLocal in some different way?