If there's not any endgame, we're in quicksand. We take one more step, and we're still there, and there's no way out. - Richard ShelbyWe've had yet another outage this week. We were exceeding connection idle time limit in Oracle. with almost 270 connections blocked by the application just waiting idly. There was something that was holding up the connections, so we rebounded the server. We thought that was an off shoot. However this repeated on another instance. Around 90 minutes into this blocked state, we got this stack trace.
org.springframework.jdbc.UncategorizedSQLException: Hibernate operation: could not inspect JDBC autocommit mode; uncategorized SQLException for SQL [???]; SQL state [null]; error code ; Connection handle is not currently associated with a ManagedConnection; nested exception is java.sql.SQLException: Connection handle is not currently associated with a ManagedConnection
[org.hibernate.util.JDBCExceptionReporter] ORA-02396: exceeded maximum idle time, please connect again
The first thing that came to our *brilliant* minds was - In Jboss, there is an idle-timeout-minutes setting for local-tx-datasource configuration - Why is the max idle time out coming from oracle (which in our case is way larger than idle-timeout-minutes specified in oracle-ds.xml).
A little research showed that the idle-timeout-minutes, is for connections that have been used by the app and returned to the pool. So, there are connections held up in the application. We started looking at where we have long standing transactions. To help us, The admin took a thread dump. all of them were held in a Mainframe call. The call was improperly configured to 90 minutes of time out. Fixed the time out. So far, we are good :)