BPELConsole promiscuous mode

Nothing really new, but been having my eyes on this promiscuous mode, but never had time to sit down and put it under the test. It is at the server level accessible via collax-config.xml or BPELAdmin console.

So in 10.1.3.3, we had simple feature called:
productionServer = true/false (false by default)
true: same version of BPEL process is not allowed to be deployed. You get following error message:

is being re-deployed to a Production Server with same revision number.Please modify the revision for the process.

In 10.1.3.5, productionServer is deprecated and it is replaced with serverMode property. As per documentation:

serverMode = production/developement/promiscuous  
Identifies the server mode. Currently supported server modes are:
    * production - re-deployment of process with same revision is not allowed.
    * development - re-deployment of process with same revision marks the existing instance as stale.
    * promiscuous - re-deployment of process with same revision will not stale instances, work items will be migrated. 
The default value is "development". 

I thought promiscuous is something everybody would want, so I thought experimenting it with. First changed the serverMode to promiscuous, and restarted.

Test case 1 (Synchronous) :

I deployed synchronous process A with couple of activities. Fired some instances of process A. Now I completely changed all the activity names in process A, added and deleted some more activities and redeployed process A with same version. Fired some more instances of process A.

Results: It was pretty good and as expected. None of the instances went stale. Old instances were shown based on old code and new instances were shown based on new BPEL code.

Now, as synchronous processes held the ground firm, it was time to experiment asynchronous flow.

Test case 2 (asynchronous ) :

I deployed asynchronous process B with a few activities. I had process B calling another asynchronous process C and then do receive activity. This was to make sure B goes to dehydration and I can get some inflight instances. I ran some transactions so that process B has a few instances inflight mode.

Now, I drastically change the process B, added/deleted new activities (but did keep the call to C and receive from C). I deployed the new code of process B with same version. I created some new transaction and tried to complete some old transactions.

Results: Quite fantastic. Nothing went stale. Process B in-flight instances with older code got migrated to newer code process B after dehydration point, and completed successfully with newer version of code. This is seems to be very powerful but delicate feature and person doing deployment should be very cautious about using promiscuous mode with in flight workflows.


Test case 3 (asynchronous human workflow) :

Created a simple process D, with Human Task, and created some inflight workflows for process D. I made some minor changes in process D, and redeployed the new code with same version.

Result: All bpel instances were inflight, but all human tasks went stale, at least didn't show up in WLA. This definitely didn't pass the expected results.



Conclusion
It is definitely very strong feature and avoid some hassles with stale activities. But in general it would be great to have this feature at the process level instead of server level. That way, we can do promiscuous only for sync and not just version the async one. I could not find such feature in 11g, guess I need to look around if composite have similar feature.

ESB Connection Leak in ESBPool

If I have to come up the list for top 5 things I hate most, Oracle ESB (only 10g) would definitely make it to that list. I have been working on it for almost 5 years and pretty much every single project, including project involving complex business processing in ESB to millions of transaction per hour just using ESB. Each time ESB never fails to disappoint me further. Every new release since 10.1.3.1 fixes hundreds of bugs and introduced another new hundreds. Just an instance "Unable to build the instance relationship", one should get nobel price for solving it as it has been unsolved mystery for 5 years and seems to be more complex than E=MC². I personally have heard statement - "ESB is unreliable product" from internal Oracle SOA Gurus, and I completely disagree with it; I think that calling ESB a "product" itself is fundamentally breaking the laws of software ethics.

Anyways, recently did 10.1.3.5.2 upgrade and we saw that ESB connection pool (ESBPool) was growing 100 connections per hour. After deprived sleep multiple nights, we found that it was another ESB bug. There were constant error messages like "Unclosed connection detected : 'oracle.oc4j.sql.spi.ConnectionFinalizer@" in log.xml. Upon creating multiple different test cases, I found that it was with a very specific pattern:

AQ -> ESB Consumer -> ESB Async Routing Rule -> Asyn BPEL

The Async Routing Rule was the culprit and such flows were found in OOTB AIA code. Once we fixed it from Async to Sync, connections never went up more than 2, and life was back to normal again.

Undeploying BPEL process

Well, not any cutting edge stuff, but did cause us lot of problem for us during 10.1.3.5 upgrade. It seems like 10.1.3.5 and especially 10.1.3.5.2 has lot of XSD validations and it can cause BPEL process not to load if some of the validation fails.

It doesn't allow user to click on this process in BPELConsole. Appropriate action would be to undeploy or redeploy those processes but without clicking on that process you can not undeploy it.

1) One way to undeploy such process would be to clean up database. We tried following:
a) clean up the database tables
PROCESS
PROCESS_DEFAULT
PROCESS_DESCRIPTOR
PROCESS_LOG
SUITCASE_BIN
b) clean up the BPEL process under tmp directory (bpel/domains/domain-name/tmp)
c) clean up ESB in BPELSystem

This was quite cumbersome process.

2) The other way we found was to use following URL:

http://host:port/BPELConsole/<<domain>>/undeployProcess.jsp?processId=<<process-id>>&revisionTag=<<revision>>

It was much cleaner to undeploy this way.

handleTopLevelFault - SOA 10.1.3.5

I think all of us know that transaction=participate in 10.1.3.3 was replaced by handleTopLevelFault=true in 10.1.3.5. I never realized the impact it had in our processes and how Oracle keep changing their mind on what this value is and how the server behaves.

First of all, in 10.1.3.3, all BPEL processes DON'T participate in transaction by default. Which means if you have, process A -> Process B -> Process C, and

[C] is disabled

try {
    insert table 1
    invoke C
} catch {
    insert table 2
    throw
}

In this case, upon the failure of invoke C, entry in table 1 or 2 will never roll back, even if data sources are configured for XA transaction. If you do put transaction=partipate in your process at configuration level, then this process and their operation would be in Global transaction and throw activity would roll back insert in table 1 and 2.

[Note if you don't have throw, it would act like there is no error and nothing would ever roll back in any scenario, which is quite expected.]

Now, in 10.1.3.4, they replaced transaction=participate with handleTopLevelFault=false. The default value in the code [com.collaxa.cube.engine.core.map.BPELProcessBlock] was true. That's why behavior would be exactly same as 10.1.3.3. Nothing would participate in transaction unless you explicitly specify it to be. You can not define this property at domain level, it is only supported at process level.

In 10.1.3.5, the default value in [com.collaxa.cube.engine.core.map.BPELProcessBlock] got changed to "false". It caused numerous problems to us. Based on AIA best practice, we always have our own try/catch block, we do some processing in catch block (e.g. update db, queue, etc) and then we rethrow the error so the caller can get details of the error and instance can show as faulted. We had the pattern mentioned earlier all over the places. In 10.1.3.3 we could see the data which were inserted from catch block, but in 10.1.3.5 everything just rolled back because it started participating in transaction by default.

I agree, not a very good coding practice but it worked in 10.1.3.3 and nobody realized it can cause transaction issues during upgrade. The ideal solution should be to be catch block in local transaction so during roll back it doesn't rollback your work in catch activity.

[C] is disabled
try {
    insert table A
    invoke C
} catch {
    insert table B [make sure DB adapter is using tx-level='local' in data source]
    throw
}

Anyways, we had more than 200+ processes and we couldn't afford changing the code for each one of them, so I came up with neat solution which worked out without any issues. I wrote BPELClient API to insert "handleTopLevelFault=true" in majority of the BPEL processes, and that's it. No restart and no code change and life was back to normal. The only flow would be that if somebody redeploys the code and overwrites descriptor, but we can just run the util again in that case.

Here is the how code looked like:

ProcessDescriptor processDescriptor = bpelProcessHandle.getDescriptor();
ConfigurationsDescriptor configurationsDescriptor = processDescriptor.getConfigurations();
configurationsDescriptor.setPropertyValue("handleTopLevelFault",appropriateValue);  // set or add
bpelProcessHandle.updateDescriptor(processDescriptor);

Another side effect of handleTopLevelFault



If your process has handleTopLevelFault = true, it will show the Faulted Instance in Flow view and Audit view in BPELConsole

if your process has handleTopLevelFault = false (or don't have this property in 10.1.3.5), it will not show the instance in flow view and will only show in Audit view. That was some new enhancement added in 10.1.3.5.

I fixed the JavaScript issue so it shows up the faulted instance in Flow view even if you have handleTopLevelFault=false. Details can be found at http://chintanblog.blogspot.com/2011/01/bpelconsole-faulted-instance-in-10135.html

BAM Filtering / Governance 11g

In one of the project, we were trying to push lot of data to BAM (some of the trace data along with data required for reports). We needed a solid approach were we can control the amount of data going to BAM at run time, just like logging level (info/debug). I agree that for infrastructure and instance tracing, CAMM, Amberpoint or Grid Control would be a better solution, but not all clients have luxury to invest in those tools. Sometime EM and BAM can help with basic troubleshooting.

The only out of the box approach I found was to use EM to disable monitoring at composite level. Pretty much nothing if you are using BAM adapter (e.g. in Mediator). We can also use following at composite level, but it requires redeployment:

   <property name="disableProcessSensors" many="false">true</property>

Following was the pretty intuitive approach we found to control the data going to BAM and we can stop/start/control BAM publishing at runtime. In BAM Sensor (Sensor action) there is feature called Filter. From there we can call any XSL expression, so we wrote custom XSL function which reads some configuration from JVM. We can control the BAM sensor publish event by returning true/false from this function.

We used the similar thing in Mediator, but it was quite simple as it also provides filter expression.

This way we had complete centralized control over BAM publish.

BPELConsole Faulted Instance in 10.1.3.5

Recently we did upgrade of 10.1.3.5 and found that if you have faulted instance in BPELConsole, it doesn't show up the flow view. You can still see the details from Audit tab but flow tab shows following error (Javascript):

Line: 57
Char: 48
Error: 'this.wi.state' is null or not an object
Code: 0
URI: http://localhost:7777/BPELConsole/default/lib/flowviewer.js?gvpn0m

- First of all, faulted instances are the one which you want to look into BPELConsole first
- Audit tab is very busy, I believe flow view is graphical and very compact

Upon following up with Oracle, we found that it is expected behavior! (http://download.oracle.com/docs/cd/E14101_01/doc.1013/e15342/bpelrn.htm#BABCGIBB). Since when, the javascript error on the webpage become expected behavior??

Anyways, it was time back to get into dirty details of the JSP and Javascripts but I finally was able to fix the issue by just changing flowviewer.js file. Here are the steps, which should be applicable to any version:

1) open $ORACLE_HOME/j2ee/oc4j_soa/applications/orabpel/console/lib/flowviewr.js
2) You can use online beatifier as this JS is very compact, I used :
http://jsbeautifier.org/
3) Search for open.faulted, you should see something like

                } else {
                    if (this.wi.state == "4" || this.wi.state == "open.faulted") {
                        this.checkforESB(e);
                        this.templateId = "activity.faulted";
                        return
                    } else {
                        if (this.wi.state == "5" || this.wi.state == "6" || this.wi.state == "7" || this.wi.state == "8" || this.wi.state == "11" || this.wi.state == "12" || this.wi.state.substring(0, 6) == "closed") {
                            this.checkforESB(e);
                            this.templateId = "activity.closed";
                            return
                        } else {
                            this.checkforESB(e);
                            this.templateId = "activity.pending";
                            return
                        }
                    }
                }

4) Make changes as below:

                } else {
                    if (    (  this.wi.state == null &&   this.type == "4") ||    this.wi.state == "4" || this.wi.state == "open.faulted") {
                        this.checkforESB(e);
                        this.templateId = "activity.faulted";
                        return
                    } else {
                        if (   this.wi.state == null ||   this.wi.state == "5" || this.wi.state == "6" || this.wi.state == "7" || this.wi.state == "8" || this.wi.state == "11" || this.wi.state == "12" || this.wi.state.substring(0, 6) == "closed") {
                            this.checkforESB(e);
                            this.templateId = "activity.closed";
                            return
                        } else {
                            this.checkforESB(e);
                            this.templateId = "activity.pending";
                            return
                        }
                    }
                }

5) On IE or Firefox, just do ctrl+F5 to refresh your local copy of JS, and you should be able to see faulted instance just like before.


If you are using 10.1.3.5 or 10.1.3.5.2, you can just download flowviewer.js from here.

OWSM gateway clean up

In OWSM fresh install, usually you get following error message in gateway.log file:

2011-01-15 18:39:37,194 FINEST [Thread-77] configuration.PolicySetWatchdog - Failed to retrieve policy set from policy manager with url http://host:7777/policymanager/services/RegistrationService
com.cfluent.policymanager.sdk.base.exception.ServerException: java.lang.Exception: Invalid component ID - C0003001
        at com.cfluent.policymanager.sdk.client.soap.SoapComponentConfigurator.getPolicies(SoapComponentConfigurator.java:185)
        at com.cfluent.agent.configuration.PolicySetWatchdog.getPolicySetFromPolicyManager(PolicySetWatchdog.java:168)
        at com.cfluent.agent.configuration.PolicySetWatchdog.pollFromPolicyManager(PolicySetWatchdog.java:207)
        at com.cfluent.agent.configuration.PolicySetWatchdog.run(PolicySetWatchdog.java:91)
2011-01-15 18:39:49,484 FINEST [Thread-77] configuration.PolicySetWatchdog - Checking Policy Manager
2011-01-15 18:39:49,530 WARNING [Thread-77] configuration.PolicySetWatchdog - Failed to retrieve policy set from policy manager with url http://host:7777/policymanager/services/RegistrationService: com.cfluent.policymanager.sdk.base.exception.ServerException: java.lang.Exception: Invalid component ID - C0003001
2011-01-15 18:39:49,530 FINEST [Thread-77] configuration.PolicySetWatchdog - Failed to retrieve policy set from policy manager with url http://host:7777/policymanager/services/RegistrationService

This is known issue, because during fresh install there is no gateway and there is no component called C0003001. So if you create gateway component then it will get the id of C0003001 and this error should go away.

In our environment, we had C0003001 component as gateway, but we decided to deactivate it and tried to create new one, and this error shows up again. Now deleting any component or recreating gateway doesn't ensure C0003001 id. On gateway, we started getting error: Gateway is not ready to process requests.

To resolve the issue, we cleaned up the entire OWSM repository and reset the sequence as below:

delete from SERVICES;
delete from COMPONENT_POLICY_MAPPINGS;
delete from COMPONENTS;
drop SEQUENCE COMPONENT_ID_SEQ;
CREATE SEQUENCE "ORAWSM"."COMPONENT_ID_SEQ" MINVALUE 1 MAXVALUE 999999999999 INCREMENT BY 1 START WITH 3001 NOCACHE NOORDER CYCLE 

After that if we recreate gateway as first component, it will have id of C0003001 and issue will be resolved right away.