AQ - queue is stopped

We started getting this error message in the log file all of sudden, and not sure how the dequeue for the queue got disabled.

[Linked-exception]
java.sql.SQLException: ORA-25226: dequeue failed, queue JMSUSER.AIA_CUSTOMERJMSQUEUE is not enabled for dequeue
ORA-06512: at "SYS.DBMS_AQIN", line 571
ORA-06512: at line 1

        at oracle.jdbc.driver.DatabaseError.throwSqlException(DatabaseError.java:138)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:316)
        at oracle.jdbc.driver.T4CTTIoer.processError(T4CTTIoer.java:282)
        at oracle.jdbc.driver.T4C8Oall.receive(T4C8Oall.java:639)
        at oracle.jdbc.driver.T4CCallableStatement.doOall8(T4CCallableStatement.java:184)
        at oracle.jdbc.driver.T4CCallableStatement.execute_for_rows(T4CCallableStatement.java:873)
        at oracle.jdbc.driver.OracleStatement.doExecuteWithTimeout(OracleStatement.java:1161)
        at oracle.jdbc.driver.OraclePreparedStatement.executeInternal(OraclePreparedStatement.java:3001)
        at oracle.jdbc.driver.OraclePreparedStatement.executeQuery(OraclePreparedStatement.java:3043)
        at oracle.jms.AQjmsConsumer.dequeue(AQjmsConsumer.java:1601)
        at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:916)
        at oracle.jms.AQjmsConsumer.receiveFromAQ(AQjmsConsumer.java:835)
        at oracle.jms.AQjmsConsumer.receive(AQjmsConsumer.java:776)
        at oracle.tip.adapter.jms.JMS.JMSMessageConsumer.consumeBlockingWithTimeout(JMSMessageConsumer.java:405)
        at oracle.tip.adapter.jms.inbound.JmsConsumer.run(JmsConsumer.java:330)
        at oracle.j2ee.connector.work.WorkWrapper.runTargetWork(WorkWrapper.java:242)
        at oracle.j2ee.connector.work.WorkWrapper.doWork(WorkWrapper.java:215)
        at oracle.j2ee.connector.work.WorkWrapper.run(WorkWrapper.java:190)
        at EDU.oswego.cs.dl.util.concurrent.PooledExecutor$Worker.run(PooledExecutor.java:830)
        at java.lang.Thread.run(Thread.java:595)

It was quite easy to fix it. Following query shows the queues which has dequeue or enqueue disabled:

select * from all_queues where trim(enqueue_enabled) = 'NO'

Once idetified which queue has issue, we can run following query to enabled the dequeue and enqueue for particular queue:

execute DBMS_AQADM.START_QUEUE(queue_name=> 'JMSUSER.AIA_CUSTOMERJMSQUEUE');

ESB 10g version agnostic routing rules - [default version]

In BPEL, it is quite easy that if we don't specify the version during partnerlink invocation, it will use the default one. But in ESB, especially when ESB is calling BPEL via native binding the version of the BPEL process is hard coded inside the routing rule. This was quite annoying because this pattern is heavily used in AIA and redeploying newer version of BPEL requires change in routing rule in ESB.

In 10.1.3.5, Oracle released a new feature which solves this problem. After 10.1.3.5 upgrade, if you deploy any BPEL process, it creates a separate version in BPELSystem called "default" which is visible in ESB Console as well as JDeveloper.

If we select default during routing rule, then the default version of the BPEL process would be called from ESB during native invocation. Therefore if we change the BPEL default version or redeploy BPEL process with higher version, it won't require change in ESB routing rules.

BPELConsole promiscuous mode

Nothing really new, but been having my eyes on this promiscuous mode, but never had time to sit down and put it under the test. It is at the server level accessible via collax-config.xml or BPELAdmin console.

So in 10.1.3.3, we had simple feature called:
productionServer = true/false (false by default)
true: same version of BPEL process is not allowed to be deployed. You get following error message:

is being re-deployed to a Production Server with same revision number.Please modify the revision for the process.

In 10.1.3.5, productionServer is deprecated and it is replaced with serverMode property. As per documentation:

serverMode = production/developement/promiscuous  
Identifies the server mode. Currently supported server modes are:
    * production - re-deployment of process with same revision is not allowed.
    * development - re-deployment of process with same revision marks the existing instance as stale.
    * promiscuous - re-deployment of process with same revision will not stale instances, work items will be migrated. 
The default value is "development". 

I thought promiscuous is something everybody would want, so I thought experimenting it with. First changed the serverMode to promiscuous, and restarted.

Test case 1 (Synchronous) :

I deployed synchronous process A with couple of activities. Fired some instances of process A. Now I completely changed all the activity names in process A, added and deleted some more activities and redeployed process A with same version. Fired some more instances of process A.

Results: It was pretty good and as expected. None of the instances went stale. Old instances were shown based on old code and new instances were shown based on new BPEL code.

Now, as synchronous processes held the ground firm, it was time to experiment asynchronous flow.

Test case 2 (asynchronous ) :

I deployed asynchronous process B with a few activities. I had process B calling another asynchronous process C and then do receive activity. This was to make sure B goes to dehydration and I can get some inflight instances. I ran some transactions so that process B has a few instances inflight mode.

Now, I drastically change the process B, added/deleted new activities (but did keep the call to C and receive from C). I deployed the new code of process B with same version. I created some new transaction and tried to complete some old transactions.

Results: Quite fantastic. Nothing went stale. Process B in-flight instances with older code got migrated to newer code process B after dehydration point, and completed successfully with newer version of code. This is seems to be very powerful but delicate feature and person doing deployment should be very cautious about using promiscuous mode with in flight workflows.


Test case 3 (asynchronous human workflow) :

Created a simple process D, with Human Task, and created some inflight workflows for process D. I made some minor changes in process D, and redeployed the new code with same version.

Result: All bpel instances were inflight, but all human tasks went stale, at least didn't show up in WLA. This definitely didn't pass the expected results.



Conclusion
It is definitely very strong feature and avoid some hassles with stale activities. But in general it would be great to have this feature at the process level instead of server level. That way, we can do promiscuous only for sync and not just version the async one. I could not find such feature in 11g, guess I need to look around if composite have similar feature.

ESB Connection Leak in ESBPool

If I have to come up the list for top 5 things I hate most, Oracle ESB (only 10g) would definitely make it to that list. I have been working on it for almost 5 years and pretty much every single project, including project involving complex business processing in ESB to millions of transaction per hour just using ESB. Each time ESB never fails to disappoint me further. Every new release since 10.1.3.1 fixes hundreds of bugs and introduced another new hundreds. Just an instance "Unable to build the instance relationship", one should get nobel price for solving it as it has been unsolved mystery for 5 years and seems to be more complex than E=MC². I personally have heard statement - "ESB is unreliable product" from internal Oracle SOA Gurus, and I completely disagree with it; I think that calling ESB a "product" itself is fundamentally breaking the laws of software ethics.

Anyways, recently did 10.1.3.5.2 upgrade and we saw that ESB connection pool (ESBPool) was growing 100 connections per hour. After deprived sleep multiple nights, we found that it was another ESB bug. There were constant error messages like "Unclosed connection detected : 'oracle.oc4j.sql.spi.ConnectionFinalizer@" in log.xml. Upon creating multiple different test cases, I found that it was with a very specific pattern:

AQ -> ESB Consumer -> ESB Async Routing Rule -> Asyn BPEL

The Async Routing Rule was the culprit and such flows were found in OOTB AIA code. Once we fixed it from Async to Sync, connections never went up more than 2, and life was back to normal again.

Undeploying BPEL process

Well, not any cutting edge stuff, but did cause us lot of problem for us during 10.1.3.5 upgrade. It seems like 10.1.3.5 and especially 10.1.3.5.2 has lot of XSD validations and it can cause BPEL process not to load if some of the validation fails.

It doesn't allow user to click on this process in BPELConsole. Appropriate action would be to undeploy or redeploy those processes but without clicking on that process you can not undeploy it.

1) One way to undeploy such process would be to clean up database. We tried following:
a) clean up the database tables
PROCESS
PROCESS_DEFAULT
PROCESS_DESCRIPTOR
PROCESS_LOG
SUITCASE_BIN
b) clean up the BPEL process under tmp directory (bpel/domains/domain-name/tmp)
c) clean up ESB in BPELSystem

This was quite cumbersome process.

2) The other way we found was to use following URL:

http://host:port/BPELConsole/<<domain>>/undeployProcess.jsp?processId=<<process-id>>&revisionTag=<<revision>>

It was much cleaner to undeploy this way.

handleTopLevelFault - SOA 10.1.3.5

I think all of us know that transaction=participate in 10.1.3.3 was replaced by handleTopLevelFault=true in 10.1.3.5. I never realized the impact it had in our processes and how Oracle keep changing their mind on what this value is and how the server behaves.

First of all, in 10.1.3.3, all BPEL processes DON'T participate in transaction by default. Which means if you have, process A -> Process B -> Process C, and

[C] is disabled

try {
    insert table 1
    invoke C
} catch {
    insert table 2
    throw
}

In this case, upon the failure of invoke C, entry in table 1 or 2 will never roll back, even if data sources are configured for XA transaction. If you do put transaction=partipate in your process at configuration level, then this process and their operation would be in Global transaction and throw activity would roll back insert in table 1 and 2.

[Note if you don't have throw, it would act like there is no error and nothing would ever roll back in any scenario, which is quite expected.]

Now, in 10.1.3.4, they replaced transaction=participate with handleTopLevelFault=false. The default value in the code [com.collaxa.cube.engine.core.map.BPELProcessBlock] was true. That's why behavior would be exactly same as 10.1.3.3. Nothing would participate in transaction unless you explicitly specify it to be. You can not define this property at domain level, it is only supported at process level.

In 10.1.3.5, the default value in [com.collaxa.cube.engine.core.map.BPELProcessBlock] got changed to "false". It caused numerous problems to us. Based on AIA best practice, we always have our own try/catch block, we do some processing in catch block (e.g. update db, queue, etc) and then we rethrow the error so the caller can get details of the error and instance can show as faulted. We had the pattern mentioned earlier all over the places. In 10.1.3.3 we could see the data which were inserted from catch block, but in 10.1.3.5 everything just rolled back because it started participating in transaction by default.

I agree, not a very good coding practice but it worked in 10.1.3.3 and nobody realized it can cause transaction issues during upgrade. The ideal solution should be to be catch block in local transaction so during roll back it doesn't rollback your work in catch activity.

[C] is disabled
try {
    insert table A
    invoke C
} catch {
    insert table B [make sure DB adapter is using tx-level='local' in data source]
    throw
}

Anyways, we had more than 200+ processes and we couldn't afford changing the code for each one of them, so I came up with neat solution which worked out without any issues. I wrote BPELClient API to insert "handleTopLevelFault=true" in majority of the BPEL processes, and that's it. No restart and no code change and life was back to normal. The only flow would be that if somebody redeploys the code and overwrites descriptor, but we can just run the util again in that case.

Here is the how code looked like:

ProcessDescriptor processDescriptor = bpelProcessHandle.getDescriptor();
ConfigurationsDescriptor configurationsDescriptor = processDescriptor.getConfigurations();
configurationsDescriptor.setPropertyValue("handleTopLevelFault",appropriateValue);  // set or add
bpelProcessHandle.updateDescriptor(processDescriptor);

Another side effect of handleTopLevelFault



If your process has handleTopLevelFault = true, it will show the Faulted Instance in Flow view and Audit view in BPELConsole

if your process has handleTopLevelFault = false (or don't have this property in 10.1.3.5), it will not show the instance in flow view and will only show in Audit view. That was some new enhancement added in 10.1.3.5.

I fixed the JavaScript issue so it shows up the faulted instance in Flow view even if you have handleTopLevelFault=false. Details can be found at http://chintanblog.blogspot.com/2011/01/bpelconsole-faulted-instance-in-10135.html

BAM Filtering / Governance 11g

In one of the project, we were trying to push lot of data to BAM (some of the trace data along with data required for reports). We needed a solid approach were we can control the amount of data going to BAM at run time, just like logging level (info/debug). I agree that for infrastructure and instance tracing, CAMM, Amberpoint or Grid Control would be a better solution, but not all clients have luxury to invest in those tools. Sometime EM and BAM can help with basic troubleshooting.

The only out of the box approach I found was to use EM to disable monitoring at composite level. Pretty much nothing if you are using BAM adapter (e.g. in Mediator). We can also use following at composite level, but it requires redeployment:

   <property name="disableProcessSensors" many="false">true</property>

Following was the pretty intuitive approach we found to control the data going to BAM and we can stop/start/control BAM publishing at runtime. In BAM Sensor (Sensor action) there is feature called Filter. From there we can call any XSL expression, so we wrote custom XSL function which reads some configuration from JVM. We can control the BAM sensor publish event by returning true/false from this function.

We used the similar thing in Mediator, but it was quite simple as it also provides filter expression.

This way we had complete centralized control over BAM publish.