BPEL Fault Generation - adapter framework

Recently we had to dig into how BPEL was generating Runtime or Binding Fault. E.g. we were calling BRM adaper and based on BRM logs we could see, it was returning and error code. On BPEL console, we saw that error code was prefixed with BRM-. We had no idea what part was doing this conversion and that brought us to the point to analyze BPEL and Adapter code to figure out how it works with EIS system and how EIS fault gets converted to BPEL fault. It should be generic across all the adapter, as BPEL PM would be just calling the adapter and getting generic exception.

It was quite obvious that BRM- would be somewhere in the BRM adapter and that exception would be converted to Fault XML, but we wanted to find exact place where it would happen. Upon analyzing multiple files, we ended up with following findings.

Summary

• oracle.tip.adapter.brm.BRMInteraction had hard coded BRM- when it was throwing PCResourceException
• com.collaxa.cube.ws.WSIFInvocationHandler.WSIFInvocationHandler is getting this exception and converting this in to Fault XML message which is seen in the BPELConsole

Details

• Each adapter entry point would be Interaction, and all adapter will implement that interface
• com.collaxa.cube.ws.WSIFInvocationHandler.WSIFInvocationHandler.invoke seems to be the master piece which calls all the adapters via WSIF - especially you can see invokeRequestHeaderHandlers and invokeLocalService calls here
• invokeLocalService converts adapter level exception (PCResourceException) to WSIFException
• invoke constructs SOAP fault XML message based on WSIFException

AIA deployment error: Please set the ORACLE_HOME environment variable

When you deploy AIA project from Jdeveloper (10.1.3.x release), you usually get an error: Please set the ORACLE_HOME environment variable. Easiest way to solve is to setup dummy Oracle and AIA home on your machine. You don't need to install entire SOA or AIA suite but need to setup the directory structure as below and copy the appropriate directory content from actual AIA installation. It comes to be about 40 to 50 MB.



Once it is setup, you can setup following environment variables:

1. AIA_HOME D:\Profiles\DummyHomes\DummyAIAHome
2. ORACLE_HOME D:\Profiles\DummyHomes\DummyOracleHome
3. BPEL_HOME D:\Profiles\DummyHomes\DummyOracleHome\bpel

Once Jdeveloper is restarted, this error should go away.

Note: it was quite obvious that something else will fail as soon as I hard code my ORACLE_HOME. It didn't take long me to find, my Oracle XE stopped working after my work station restart. I think it is good practice to not set ORACLE_HOME and set only when you need to do critical AIA deployment.

As another work around, you can only try to set AIA_HOME and BPEL_HOME, and based on build.xml file, it might just work fine, that way it doesn't cause problem with existing SOA or ORACLE DB installation on your machine.

How to install & get list of AIA patches

To get list of all the patches installed in Oracle Application Server (10g release), you can run following command to get all patches:

To list patches installed in Oracle Home
set ORACLE_HOME=
cd $ORACLE_HOME/OPatch
opatch lsinventory

For AIA, there is separate OPATCH to install and to get list of patches. Here are the steps:

• Download latest version of AIA Opatch from metalink (p5912518_111000_GENERIC.zip)
• Set following homes

export ORACLE_HOME= (e.g. /home/oracle/product/10.1.3.1/OracleAS_1)
export AIA_HOME= (e.g. /home/oracle/aiahome )
export SOA_HOME= (e.g. /home/oracle/product/10.1.3.1/OracleAS_1)
export PATH= (e.g. /home/oracle/5912518/OPatch:$PATH)

Note: Make sure AIA Opatch is in the path and it is the first entry or prior to any other OPATCH entry.

Now you can run following command to get list of AIA patch or install AIA patch

List of AIA patch

• opatch lsinventory -invPtrLoc $AIA_HOME/oraInst.loc -oh $AIA_HOME -jre $SOA_HOME/jdk/jre

Install AIA patch

• Go to the directory where patch is unzipped and run:
• opatch apply -invPtrLoc $AIA_HOME/oraInst.loc -oh $AIA_HOME -jre $SOA_HOME/jdk/jre

Removing WSSE Headers in OWSM

Recently had chance to work with WSSE headers in OWSM. WSSE information is stored in SOAP Header. When we were getting WSSE security information in SOAP envelope (header), and we were using "Verify Signature" to verify signature. Verify Signature has step to remove the WSSE header so that this signature doesn't get propagated to BPEL, otherwise BPEL will throw exception. E.g.



If you install entire SOASuite in one container (oc4j_soa) and use above option to remove Security headers, it works great. When SOA suite is installed as per Enterprise Deployment Guide where gateway and policy manager are in separate container than oc4j_soa, OWSM doesn't remove the header even with this option.

Quick resolution to this issue is to use XSL and remove the header completely. Earlier we used following XSL to remove entire SOAP header.


We faced the problem with XSL as it removes entire SOAP header. We can use following XSL which can just remove WSSE header from entire SOAP envelope, so if there are custom element in SOAP header, it doesn't get removed.

Non intrusive set title for BPEL, AIA

If you check my blog http://chintanblog.blogspot.com/2010/03/bpel-set-title.html you can see how to configure set title for BPEL process and why it is almost absolutely necessary for enterprise implementation. It also makes search in BPEL console quite useful http://chintanblog.blogspot.com/2010/03/bpel-title-wildcard-search_01.html.

We did encounter multiple issues which forced us to think non intrusive approach of setting title of BPEL process

• We had to change title of more than 80 processes and writing logic and changing each process was very cumbersome (it had to go through un-necessary QA cycles)
• Lot of process were already deployed and had lot of instances, we couldn't go back in time and set the title for them
• Due to business requirement, if we plan to change the title, we have to go through this cycle again
• We didn't want to touch OOTB process which comes with PIP, but we needed our custom title for those

Here is what I came up with:



Implementation Steps

1. EJB 3.0 timer which polls for new instances in BPEL DB
2. Via JDBC, extract Audit Information and Audit Trail (it would be configurable how much data to extract)
3. Once payload is available, apply XPath (based on configuration) to extract the title for a particular process
4. Update the title of the BPEL process

It sounded pretty complicated and resource exhaustive, but when I implemented and ran on the huge amount of data. I could see that we could set title of individual process in just 0.03 seconds, which was quite acceptable. It runs in separate JVM and doesn't talk with BPEL Runtime Engine so it doesn't increase any load on BPEL. It does increase some load on database but it is just select and update, there is no heavy processing going inside the database.

With this approach, we could achieve following

• Centralized approach to configure and set the titles
• We could change title of OOTB processes without touching the code
• We could set the title of old instances (even stale instances)
• Changing title was just matter of configuration and pure XPath. I also increased the size to title from 50 to 256, so you could put more data in title (e.g. order number, organization id, etc...)

Oracle AIA SOA - Multi JVM - Vertical Scaling

It is pretty old concept and I tried earlier couple of times but this time I had much more time to do through analysis of Oracle Application Server with multi JVM container and see how it works out with SOA and AIA in general.

Configuration of multi JVM is fairly straight forward. All you need to do is change number of JVM for container from EM or change numprocs from opmn.xml for a particular container. You can get more information from http://mike-lehmann.blogspot.com/2006/09/scaling-oracleas-with-multiple-jvms.html

(Note: Although you can not change number of JVM for the container where EM is running, so if you install basic SOA suite, adding JVM would cause problem with EM.)

So we added 2 JVM to OC4J_SOA container and tried couple of test cases:

1) Test case: Make sure load is distributed evenly
I believe it is due to mod_oc4j config or default option, but load was distributed in exactly round robin fashion as per log file.

2) Functionality testing
Well, I was not involved much into functional stuff, but seems like functional stuff worked like a charm. We didn't see any issue with BPEL or ESB communication.

At this point we were pretty satisfied with results and about to move forward and then we thought of checking couple of configuration and deployment stuff for AIA and things started looking pretty disappointing:

3) AIA Configuration change
Test case: Changing AIA configuration file and using AIA console to reload the configuration. Expected output would be to have new configuration in both JVM.
Result: Only one JVM loads the AIA Configuration. I tried loading multiple times using UI, it still loads only on one JVM. May be http connection had some static stuff with OC4J. Removing caching or trying from different machine might fix this issue but I would still consider it as failure scenario.
Note: Restarting server certainly forces both JVM to load new AIA configuration file.

4) BPEL Configuration change
Test case: Changing BPEL configuration (e.g. audit level or logging level) from BPEL console. Expected output would be to have new configuration in both JVM.
Result: Only one JVM loads the BPEL configuration.
Note: Restarting server certainly forces both JVM to load new BPEL configuration

5) BPEL deployment
Test case: Redeploying BPEL process with same version with minor code change. Expected output would be to have new BPEL code in both JVM.
Result: Only one JVM picks up new BPEL deployed suitcase. Second JVM was still running with old BPEL code.
Note: Restarting server forces both JVM to load latest version of BPEL suitcase.

6) ESB Configuration and Deployment testing.
I was really hoping this test to work smooth due to ESB-DT and ESB-RT concept. So as long as ESB-DT is in separate container with only one JVM and ESB RT (may be in OC4J_SOA) can have multiple JVM and deployment and configuration change would still work fine. Unfortunately I didn't have access to correct environment where ESBDT and ESBRT were database persisted, and ESBDT in separate container.

With ESB-RT and ESB-DT in same container (oc4j_soa) and having them in-memory topics caused the similar effect as BPEL during configuration chagne and deployment.



In future, I might publish results on ESB with DB persisted topics and some interesting stuff I saw on Java based application (webapp and webservices).

CAVS Configuration

I have been using CAVS for quite a while but didn't see lot of documentation on how to configure it so just thought blogging about it. I have also been using iTKO Lisa for a long time which is much richer in functionality than CAVS, and maybe I will write an article about it someday.

Brief Summary

CAVS (Composite Application Validation System) is mainly used for simulating your integration points for composite testing. It can basically mimic the end system and validate your request and send valid response upon successful validation.

Technical details

• It is a servlet. URL (http://host:port/AIAValidationSystemServlet/syncresponsesimulator). You can configure multiple simulator instances and it get recognized by simid parameter.
  
• It is part of AIAApplication ($CONTAINER_HOME/applications/AIAApplication/ValidationSystemServlet.war)
  
• Execution logs can be found at ($CONTAINER_HOME/application-deployments/AIAApplication/oc4j_soa_default_group_1/application.log)
  

How to use:

Scenario:
Let's say we have simple Echo service, which returns upper case data back. We want to validate the request that, it has input and return the static response back.




Configuration steps:

• Login to AIA console (http://host:port/AIA)
  
• Click on Validation System (GO)
  
• Click on Create Simulator
  
  
• Enter the request and response as below (Make sure to include namespace for both request and response and don't remove CAVS namespace)
  
• Click Next
  
  
  
• Click on Request and Generate XPath (this will generate xpath for all possible combinations)
  
  
• Make changes to validation as necessary. E.g. I made one nodekey (which is required) and just changed "equal to" to "is valid" operation for input.
  
  
• Click on Save and record simulator id number
  
• Now it is ready to tested with SOAP UI and end point would be http://host:port/AIAValidationSystemServlet/syncresponsesimulator?simid=<<>> or http://host:port/AIAValidationSystemServlet/syncresponsesimulator (generic). If you use generic one, it will compare against all simulator to get exact match
  

AIA Configuration

• Out of the box always have support for real or CAVS endpoint for any partnerlink. If you are implementing custom code make sure you use dynamic parterlink for AIA so that it has CAVS support (more information at: http://chintanblog.blogspot.com/2010/03/aia-bpel-dynamic-partnerlink-lookup.html)
  
• In AIAHome/config/AIAConfigurationProperties.xml file, make following changes: