It is pretty old concept and I tried earlier couple of times but this time I had much more time to do through analysis of Oracle Application Server with multi JVM container and see how it works out with SOA and AIA in general.
Configuration of multi JVM is fairly straight forward. All you need to do is change number of JVM for container from EM or change numprocs from opmn.xml for a particular container. You can get more information from http://mike-lehmann.blogspot.com/2006/09/scaling-oracleas-with-multiple-jvms.html
(Note: Although you can not change number of JVM for the container where EM is running, so if you install basic SOA suite, adding JVM would cause problem with EM.)
So we added 2 JVM to OC4J_SOA container and tried couple of test cases:
1) Test case: Make sure load is distributed evenly
I believe it is due to mod_oc4j config or default option, but load was distributed in exactly round robin fashion as per log file.
2) Functionality testing
Well, I was not involved much into functional stuff, but seems like functional stuff worked like a charm. We didn't see any issue with BPEL or ESB communication.
At this point we were pretty satisfied with results and about to move forward and then we thought of checking couple of configuration and deployment stuff for AIA and things started looking pretty disappointing:
3) AIA Configuration change
Test case: Changing AIA configuration file and using AIA console to reload the configuration. Expected output would be to have new configuration in both JVM.
Result: Only one JVM loads the AIA Configuration. I tried loading multiple times using UI, it still loads only on one JVM. May be http connection had some static stuff with OC4J. Removing caching or trying from different machine might fix this issue but I would still consider it as failure scenario.
Note: Restarting server certainly forces both JVM to load new AIA configuration file.
4) BPEL Configuration change
Test case: Changing BPEL configuration (e.g. audit level or logging level) from BPEL console. Expected output would be to have new configuration in both JVM.
Result: Only one JVM loads the BPEL configuration.
Note: Restarting server certainly forces both JVM to load new BPEL configuration
5) BPEL deployment
Test case: Redeploying BPEL process with same version with minor code change. Expected output would be to have new BPEL code in both JVM.
Result: Only one JVM picks up new BPEL deployed suitcase. Second JVM was still running with old BPEL code.
Note: Restarting server forces both JVM to load latest version of BPEL suitcase.
6) ESB Configuration and Deployment testing.
I was really hoping this test to work smooth due to ESB-DT and ESB-RT concept. So as long as ESB-DT is in separate container with only one JVM and ESB RT (may be in OC4J_SOA) can have multiple JVM and deployment and configuration change would still work fine. Unfortunately I didn't have access to correct environment where ESBDT and ESBRT were database persisted, and ESBDT in separate container.
With ESB-RT and ESB-DT in same container (oc4j_soa) and having them in-memory topics caused the similar effect as BPEL during configuration chagne and deployment.
In future, I might publish results on ESB with DB persisted topics and some interesting stuff I saw on Java based application (webapp and webservices).