How to start Enterprise Manager

If you shutdown Enterprise Manager from Console or any other way here is the trick to start the Enterprise Manager:

soasuite/j2ee/home/config/default-web-site.xml
<application name="ascontrol" path="../../home/applications/ascontrol.ear" parent="system" start="true" />
<application name="javasso" path="../../home/applications/javasso.ear" parent="default" start="true" /> (not sure if it is required all the time)

soasuite/j2ee/home/config/server.xml
<web-app application="ascontrol" name="ascontrol" load-on-startup="true" root="/em" ohs-routing="true" />

Bounce the home container using following command

opmnctl restartproc process-type=home

Custom Decision Service

I am not very big fan of Decision Service which comes out of the box with Oracle BPEL. There are couple of issues I found

- It creates whole bunch JAXB classes in my bpel process, which are .java and .class files (I have to keep both of them, .class files are not generated during compilation of bpel process!)

- I can NOT change the created Decision Service, it is not like I can click on DB adapter and reconfigure it. It make my life extremely difficult when have to accommodate changes from client

- I can not call the service from ESB or other WS consumer, the decision service exposed takes millions of parameters which are not relevant to web service consumer (e.g. bpelInstanceId? why in the hell rule engine needs to know bpel instance id?)

- There is a decision service for each partner link! We have 20 BPEL processes making almost 30 different type of calls, that ends up in 30 decision services. That make my life impossible to manage.

- It is way too complicated to implement WSIF with the decision service.

I love the idea of BAM, where all Active Data cache operations are exposed as one service. I had very naive and romantic idea of implementing my custom decision service and talk to any type of rule set, rule function, do the connection pooling of rules via one web service interface. It can take any type of parameter and using Java reflection I can convert to right type and that's all I need. My client did buy this idea and I implemented very nice and generic Custom Decision Service.

1. Implemented POJO

- I implemented a small Java POJO service, which does takes configuration

e.g. RuleRepositoryType (webdav, files), File Repository Connection Information, WebDav Repository Connection Information, Whole bunch of debug flags which are supported by Rule Engine, Dictionary Name, Rule Function or Rule Set, Input types and output types.

2. Implemented EJB 3.0 and exposed as Web service via annotations

- Just created EJB 3.0 session facade on top of POJO so that it is container managed, I can easily expose as Web Service via Annotations, and for implementing Connection pool.

3. Implemented Connection Pooling in EJB

- I implemented custom connection pool of Rule Repository and Rule Dictionary. I initiate new Rule Session each time. I also noticed that pooling/caching Rule repository was not just good enough so I had to cache Rule Dictionary as well. Yeah, I implemented

4. WSIF Interface via annotations

- I just followed http://chintanblog.blogspot.com/2007/12/although-wsif-seems-to-be-hack-to-plug.html to create WSIF on top of my EJB 3.0 session facade.

5. Performance Matrix

Test case: Used WebDav repository, Rule Function interface which executes more than 10 rule sets and creates whole bunch of external connection. It usually takes about 2 seconds to complete the Rule Session itself. I used about 10 threads and loop of 50 with 1 second delay in each (so total number of instances will be 500).

Default Decision Service over SOAP: I created sample BPEL process and created decision service with decide activity (which comes with jaxb java and classes, jar, ear, war, and so on...). Here were the results:

Average execution time: 3.1 second

Number of failures: 15

Custom Decision Service over SOAP: To be fair I didn't use WSIF interface. Calling my custom decision service over soap protocol. It is just like normal partner link.

Average execution time: 3.5 second

Number of failures: 10

Notes:

- Reason behind slowness is because of I am doing more reflection which is a bit expensive.

- Because of EJB and container managed stuff, I am getting less number of failures.

- Out-of-box decision service caches the Rule Connection where the custom decision service manages the entire pool which is shared across the enterprise. I believe if I test with 20 different BPEL processes with different pattern of calls, I will be far ahead in comparison with default decision service.

I am more than willing to get your feedback on this. I am not sure if I did right thing, but I believe I reduced (rather completely eliminated) overhead for managing far complex default decision service.

ESB Display Older Instances

I used to have my own interceptor logging all required thing into the database, so never had to look at ESB console.

Issue 1 :

Recently when I was working on ESB console I realized that it displays only a day old instances! While looking into the database I saw all instances where there.

Yeah, of course the ESB client API was causing the issues. As mentioned in my earlier blogs, to debug ESB Console related issues, the best things is to use obtunnel to intercept all your request and then simple wingrep/grep to search for what you interested in.

The reason behind day old instance was easy to find:

File Name: %soasuite%/j2ee/oc4j_soa/applications/esb-dt/esb_console/esb/model/model.InstancesSearch.js

Where: Search for key word "InstanceSearchModel.prototype.init" about the line number: 150 in 10.1.3.1

Content:

Original:

InstanceSearchModel.prototype.init = function(serviceConfigROTID)
{
this._trackingSC = new TrackingSearchCriterion(serviceConfigROTID);
this._serviceEntityInfo = null; // ESBEntityInfo
this._status = "Any";
this._flowID = "";
this._timePeriod = 1;
this._timeUnit = "days";
this._timeZoneString = InstanceSearchModel.getTimezoneString(new Date(), false);
}

Basically InstanceSearchModel creates Rest Webservice request to ESB server, and timePeriod is configurable parameter to this object. Based on my observation, other javascript which are using InstanceSearchModel object are not passing this parameter, so default parameter will be used which is 1 day. I changed the default to 20 days and it showed 20 days worth of instances...

Modified:

InstanceSearchModel.prototype.init = function(serviceConfigROTID)
{
this._trackingSC = new TrackingSearchCriterion(serviceConfigROTID);
this._serviceEntityInfo = null; // ESBEntityInfo
this._status = "Any";
this._flowID = "";
this._timePeriod = 20;
this._timeUnit = "days";
this._timeZoneString = InstanceSearchModel.getTimezoneString(new Date(), false);
}

Issue 2 :

ESB has limitation on returning only 100 results. It was an amanzing experience to rip off the entire oraesb.jar and go through the hierachy of java files,

soasuite/j2ee/oc4j_soa/applications/esb-dt/esb_console/WEB-INF/web.xml
oraesb/oracle/tip/esb/configuration/servlet/CommandServlet.jad
oraesb/oracle/tip/esb/configuration/servlet/command/GetInstancesCommand.jad (xmlInstanceManager.getInstances)
oraesb/oracle/tip/esb/console/XMLInstanceManager.jad
oraesb/oracle/tip/esb/console/XMLInstanceManagerImpl.jad
oracle/tip/esb/monitor/manager/ActivityMessageStore.jad
oraesb/oracle/tip/esb/monitor/manager/database/DBActivityMessageStore.jad (InstanceListXMLBuilder)
oraesb/oracle/tip/esb/monitor/manager/database/InstanceListXMLBuilder.jad
oraesb/oracle/tip/esb/monitor/manager/database/FilterParser.jad (String s = RepositoryFactory.getRepository("RUNTIME").getESBParameter("MaxInstanceCount"))

Bingo! Finally I found the last file which was reading the parameter from ESB parameter table, now it was piece of cake!

I added, insert into esb_parameter (PARAM_NAME, PARAM_VALUE) values ( 'MaxInstanceCount', '1000' ); in ORAESB schema, and bounced the server and it started working like magic!

MQ Adapter Anatomy

Recently I had chance to work with MQ Adapter, and it was pretty nice experience and I pretty much tried all different types of settings supported by MQAdapter. Here I would like to share something.

If you religiously follow the document: http://download.oracle.com/docs/cd/B32110_01/web.1013/b28956.pdf then there is not much value I will be adding, but still you might find something useful and accumulated information.

1) Different Settings

Now let's take a look at each parameter:

• Scheme: Three types of scheme

  1. dynamic : use max connections, if all are in use, then give new one and don't worry about maxConnections limit

  2. fixed: use max connections, if all are in use, then throw exception for new request

  3. fixed_wait: use max connections, if all are in use, then do fixed amount of wait for newly requested connection

• Best practice is to use fixed_wait with higher timeout so that we can restrict the number of connection

• minConnections/maxConnections/initial-capacity: Decided number of connections and initial connections during start up.

• waitTimeout : This is the timeout used during fixed_wait scheme for new requests

• inactivity-timeout-check: When to check for expired or inactive connections. Supported values are "never","periodic","piggyback" (when a new connection is fetched), and "all" (periodically and when a new connection is fetched).

• We kept it “all” as we don’t want to keep connections idle longer than certain time period.

• inactivity-timeout: The desired connection timeout, in seconds, as a positive integer, or 0 for connections to never expire. Negative values are disallowed.

• We have kept it 60 seconds.

2) "Garbage Collector" - clean up routine

Basically three parameters are used to configure for releasing the idle connection held by OC4J connection pool.

• inactivity-timeout-check: has to be "all" or "piggyback", all is desired if you want to periodically run clean up routine.

• inactivity-timeout: The desired connection timeout, in seconds, as a positive integer, or 0 for connections to never expire. Negative values are disallowed.

• Server.xml (taskmanager-granularity): When inactivity-timeout-check is configured to be periodic, this value specify how specified how frequently we want to run the periodic recycling routine.

Here is sample server.xml entry:

3) Different Level of Caching and Related Errors

Actually, after setting 1 and 2, connections were still not getting cleared, and we were able to see on MQ server side (using Tivoli) that connections are not getting closed. We found from Oracle product managers, that Adapter itself manages cache for MQ (or any type of) connections.

We need to provide, cacheConnections=false property in BPEL Partnerlink or ESB end point property. After that ESB and BPEL releases connection nicely and clean up routine described in No 2, works great and cleans up all idle connections.

4) Inbound and Outbound MQ adapter

Another finding I found was, difference between inbound and outbound adapters. If you have MQ outbound adapter with "cacheConnections=false", it releases connection right after completing the unit of work, and connection is available to OC4J connection pool. But if you have MQ inbound adapter with "cacheConnections=false", it still holds the connection. Probably it is because of continuous polling.

I have not tried it with changing the polling frequency to see if connection get releasesd.

Big News - Resigned Oracle

Joined on Nov 29, 2005 and finally completed my last day at Oracle today. Feel pretty sad and same time very exciting to face new challenges by my self. Now I am becoming independent consultant in Oracle SOA space and without any expert's support.

I am may be putting my career at risk but as we all know: The biggest risk in life is not to take one!

ESB (and BPEL) purge instances

There are multiple ways we can use to purge ESB instances.

Database Scripts
For 10.1.3.3.1 SOASuite ESB purge instances scripts are provided as part of installation (%soasuite%/integration/esb/sql/other):
- purge_by_date.sql
- purge_by_instance_id.sql
- purge_by_id.sql


ESB Client API
Database scripts are always good, but it should be best practice to rely on API provided by product. Same as BPEL Client API, whatever you can do in ESB console you can do via ESB client API. I used %soasuite%/bpel/bin/obtunnel.bat to check what XML based payload it is using. For cleaning the ESB instances, here is the sample code:

purgeInstance.consoleClient = purgeInstance.getESBClient();
String purge = "<instanceManage enable='true' userPurgeTimePeriod='14400000'/>";
HashMap<String, String> requestProps = new HashMap<String, String>();
requestProps.put("root", purge);
purgeInstance.consoleClient.perform("UpdateTrackingConfig", requestProps);

Here time period is in milliseconds and all instances older than timestamp will be purged. If we need to purge all instances, we need to change userPurgeTimePeriod='0'.


Custom ANT Task
I saw the custom ANT tasks for managing BPEL domain, purge instances, etc.. It is an amazing job done by Ramkumar Menon (http://blogs.oracle.com/rammenon/2007/11/26#a74 ). I used his framework and added some tasks for managing ESB.

It is really easy to use this tasks, e.g. for managing BPEL domain you can specify following in build.xml:

<BPELServerAdmin providerurl="${bpelProviderURL}" username="${bpelUsername}" password="${bpelPassword}">
<manageDomain domain="${bpelDomainName}">
<purgeInstances select="all" fromdatetime="${bpelPurgeInstanceFromDateTime}" todatetime="${bpelPurgeInstanceToDateTime}"/>
<undeployProcesses select="BPELProcess.*" type="all" revision="all"/>
</manageDomain>
</BPELServerAdmin>

This will purge instances based on timestamp and undeploy the processes.

For ESB, this is what I added:

<ESBServerAdmin username="${esbUsername}" password="${esbPassword}" hostname="${esbHostName}" port="${esbPort}">
<purgeESBInstances todatetime="${esbPurgeInstanceToDateTime}"/>
</ESBServerAdmin>

You can purge ESB instances based on timestamp, from and to is not supported. Probably it can be enhanced if there is requirement.

Here is the link for downloading Ant task with readme file, sample build.xml and sample build.properties file.

Performance with ESB and BPEL UDDI runtime lookups

Well, nothing new with it, as long as you are using SOASuite 10.1.3.3.1 you should have both of them out of the box. Just to recap:

BPEL run-time look-ups:

1) You have to define/publish wsdl in service registry. If you publish wsdl through User Interface, it creates unique sequence id for registry key which you can change using registry user interface. You can also use registry API to publish wsdl of the service.

2) Define registryServiceKey as property of partnerlink

You can either edit bpel.xml or edit the partnerlink to achieve it:

3) Define registry URLs in domain.xml

URL for uddiLocation should be http://host:port/registry/uddi/inquiry.

ESB run-time look-ups:

1) You have to define/publish wsdl in service registry. If you publish wsdl through User Interface, it creates unique sequence id for registry key which you can change using registry user interface. You can also use registry API to publish wsdl of the service.

2) Define registryServiceKey as end point property for service

3) Define registry URLs in integration/esb/config/esb_config.ini file

I have tried both ESB and BPEL run-time look-ups and they work great. Although there are some really bad performance issues I saw:

After enabling run-time look-ups ESB or BPEL calling any service via UDDI was taking longer than 1 minutes to invoke (it was couple of milliseconds earlier). Upon analysis we find out the actual invocation was not the issue but preparing the call (get-wsdl, parse wsdl and store in specific cache) was taking the longest time.

Also we saw that if load on BPEL and ESB if not very high, I don't see many performance issues but as soon as load increases (with number of threads) things get really slow. We did look at service registry logs and it's response time, service registry was extremely fast.

Here is what we found the reason behind slowness: ESB and BPEL both caches WSDL into their cache in specific format. If we do run-time look-ups BPEL and ESB gets the WSDL using service registry (which is pretty fast), but analyzing those wsdl and storing them in structure which BPEL and ESB run-time engine can understand takes a long time.

In conclusion, all this stuff BPEL, ESB, OSR - run-time look-ups looked like an ideal approach but had some performance drawbacks. All that glitters is not gold :)