Wednesday, March 2, 2011

ESB Connection Leak in ESBPool

If I have to come up the list for top 5 things I hate most, Oracle ESB (only 10g) would definitely make it to that list. I have been working on it for almost 5 years and pretty much every single project, including project involving complex business processing in ESB to millions of transaction per hour just using ESB. Each time ESB never fails to disappoint me further. Every new release since 10.1.3.1 fixes hundreds of bugs and introduced another new hundreds. Just an instance "Unable to build the instance relationship", one should get nobel price for solving it as it has been unsolved mystery for 5 years and seems to be more complex than E=MC2. I personally have heard statement - "ESB is unreliable product" from internal Oracle SOA Gurus, and I completely disagree with it; I think that calling ESB a "product" itself is fundamentally breaking the laws of software ethics.

Anyways, recently did 10.1.3.5.2 upgrade and we saw that ESB connection pool (ESBPool) was growing 100 connections per hour. After deprived sleep multiple nights, we found that it was another ESB bug. There were constant error messages like "Unclosed connection detected : 'oracle.oc4j.sql.spi.ConnectionFinalizer@" in log.xml. Upon creating multiple different test cases, I found that it was with a very specific pattern:

AQ -> ESB Consumer -> ESB Async Routing Rule -> Asyn BPEL

The Async Routing Rule was the culprit and such flows were found in OOTB AIA code. Once we fixed it from Async to Sync, connections never went up more than 2, and life was back to normal again.

1 comment:

Chintan Shah said...

BTW, Oracle provided us patch overnight for this issue. You can download the patch 10164308 from Oracle support.