Node Manger Configuration 11g/12c

Node Manager Configuration and Start up in 11g (10.3.6)

Generate Node Manager Properties (nodemanager.properties):

 cd %WLS_HOME%\server\bin  
 startNodeManager.cmd  

This will generate the %WLS_HOME%\common\nodemanager\nodemanager.properties file. We can verify the nodemanager.properties file for all properties. I usually set up SecureListener=false, as it requires more configuration if we are using Custom Identity/Trust.

Now we can start again, and new properties will take in effect.

 cd %WLS_HOME%\server\bin  
 startNodeManager.cmd  

Node Manager Configuration and Start up in 12c

Node manager properties file %DOMAIN_HOME%\nodemanager\nodemanager.properties is generated as part of the installation process or domain creation. Once it is verified, we can start the node manager with following command

 cd %DOMAIN_HOME%\bin  
 setDomainEnv.cmd  
 cd %DOMAIN_HOME%\bin  
 startNodeManager.cmd  

Admin Console (11g/12c)
Admin console part remains the same in both version. where you can verify the status of Node Manager, configure a few properties and assign machine to node manager if needed.

WebLogic/OSB one way and two way SSL - outbound calls

I found portecle quite useful when importing and exporting keys and it provides neat user interface. In standalone JVM, we can specify keystore using command line parameters as below

 -Djavax.net.ssl.keyStore  
 -Djavax.net.ssl.trustStore  
 -Djavax.net.ssl.trustStorePassword  
 -Djavax.net.ssl.keyStorePassword  
 -Djavax.net.ssl.keyStoreProvider  

In WebLogic, it is done via Admin Console.

One Way SSL
It is quite straight forward to configure one way SSL. I believe preferred way is to use Custom Identity and Custom Trust store, which is much better to manage than Demo trust and identity.

Using DemoTrust and DemoIdentity
The location of DemoTrust and DemoIdentity can be found from Admin Console:

The default passwords are:

 DemoIdentity.jks -> DemoIdentityKeyStorePassPhrase  
 DemoTrust.jks -> DemoTrustKeyStorePassPhrase  

We can open them in Portecle and import the SSL cert.

Using Custom Identity and Custom Trust
I prefer this option better than Demo. I usually use jre/lib/cacerts as starting point and import all necessary certs in cacerts. Once all certs are imported in cacerts, then weblogic configuration as below

Under KeyStores tab:

Under SSL tab:
It needs to be done on all servers which would be affected, and server restart would be necessary after this.

Two Way SSL
Using Custom Identity and Custom Trust
I imported both private key and root cert in cacerts file. The password of the cacerts file was same as password of private key, I was not sure if it was absolutely necessary but that is how it worked in standalone JVM. Once private key is imported, below is the WebLogic configuration

Enable SSL Debug

 -Dssl.debug=true  
 -Dweblogic.security.SSL.verbose=true  
 -Djavax.net.debug=all  

Loosen up Security Constraint

 -Dweblogic.webservice.client.ssl.strictcertchecking=false  
 -Dweblogic.security.SSL.allowSmallRSAExponent=true  
 -Dweblogic.security.SSL.enforceConstraints=off  
 -Dweblogic.security.SSL.enable.renegotiation=true  
 -Dsun.security.ssl.allowUnsafeRenegotiation=true  

webMethods NSRuntimeException

In webMethods, I was facing "com.wm.app.b2b.client.ns.NSRuntimeException". It is quite generic exception caused in many different situations. The solution in each case quite same, drop the variables in pipeline, especially some of the variables which are sensitive to Java and Notification service, e.g. document, properties were causing the problems.

Consuming EBS events (WF_BPEL_Q) using Java/PLSQL AQ API

When we raise event in EBS, it is enqueued in durable subscriber queue (WF_BPEL_Q). We could use EBS API or pure AQ (java/plsql) API to dequeue them. The message type stored in this queue is object type (WF_EVENT_T)

Use below query to check list of subscribers and subscriber number. Both queries shows subscriber list, but second query shows subscriber number, which is very helpful to troubleshoot later.

 SELECT * FROM ALL_QUEUE_SUBSCRIBERS WHERE QUEUE_NAME = 'WF_BPEL_Q';  
 SELECT * FROM AQ$_WF_BPEL_QTAB_S WHERE NAME IS NOT NULL;  

In WF_BPEL_QTAB, we can see all messages, and AQ$_WF_BPEL_QTAB_I would store copy of the message for each subscriber. The durable subscription is explained here in detail.

 SELECT * FROM AQ$_WF_BPEL_QTAB_I;  

Enqueue using PLSQL (instead of raise event)

 DECLARE  
   enqueue_options   dbms_aq.enqueue_options_t;  
   message_properties dbms_aq.message_properties_t;  
   message_handle   RAW(16);  
   message       APPS.WF_EVENT_T;  
 BEGIN  
   message := APPS.WF_EVENT_T(50,SYSDATE,SYSDATE,NULL,APPS.WF_PARAMETER_LIST_T(APPS.WF_PARAMETER_T('USER_ID','20421'),APPS.WF_PARAMETER_T('RESP_ID','51378'),APPS.WF_PARAMETER_T('RESP_APPL_ID','800'),APPS.WF_PARAMETER_T('SECURITY_GROUP_ID','0'),APPS.WF_PARAMETER_T('ORG_ID','122'),APPS.WF_PARAMETER_T('PARTY_ID','3667135'),APPS.WF_PARAMETER_T('#CURRENT_PHASE','101')),'oracle.apps.ar.hz.Person.create','oracle.apps.ar.hz.Person.create659162',NULL,APPS.WF_AGENT_T('WF_BPEL_QAGENT','EBIZPROD.SPRINGSOA.COM'),NULL,NULL,NULL,NULL);  
   dbms_aq.enqueue(queue_name => 'WF_BPEL_Q',        
      enqueue_options   => enqueue_options,      
      message_properties  => message_properties,     
      payload       => message,          
      msgid        => message_handle);  
   COMMIT;  
 END;  

Register Custom Subscriber
Below code can be used to register custom durable subscriber to the queue.

 DECLARE  
   SUBSCRIBER SYS.AQ$_AGENT;  
 BEGIN  
   SUBSCRIBER := SYS.AQ$_AGENT('CUSTOM', NULL, NULL);  
   DBMS_AQADM.ADD_SUBSCRIBER(QUEUE_NAME => 'WF_BPEL_Q',SUBSCRIBER => SUBSCRIBER);  
 END;       

Dequeue using PLSQL
To Dequeue this using PLSQL, was a lot easier than Java, as WF_EVENT_T object is available at database level under APPS schema. PLSQL code below:

 DECLARE  
   DEQUEUE_OPTIONS   dbms_aq.dequeue_options_t;  
   MESSAGE_PROPERTIES dbms_aq.message_properties_t;  
   MESSAGE_HANDLE   RAW(16);  
   MESSAGE       APPS.WF_EVENT_T;  
 BEGIN  
      DEQUEUE_OPTIONS.WAIT := DBMS_AQ.NO_WAIT;  
      DEQUEUE_OPTIONS.CONSUMER_NAME := 'CUSTOM';  
      DEQUEUE_OPTIONS.NAVIGATION := DBMS_AQ.FIRST_MESSAGE;  
      DBMS_AQ.DEQUEUE(QUEUE_NAME => 'WF_BPEL_Q',  
       DEQUEUE_OPTIONS  => DEQUEUE_OPTIONS,  
       MESSAGE_PROPERTIES => MESSAGE_PROPERTIES,  
       PAYLOAD      => MESSAGE,  
       MSGID       => MESSAGE_HANDLE);  
      DBMS_OUTPUT.PUT_LINE ('Message: ' || MESSAGE );  
      DEQUEUE_OPTIONS.NAVIGATION := DBMS_AQ.NEXT_MESSAGE;  
      COMMIT;  
 END;  

Dequeue using Java API
To dequeue using Java using pure AQ API, I used JPublisher to convert WF_EVENT_T data type to WF_EVENT_T java class as below:

1) Running JPublisher on DOS prompt to generate Java object for WF_EVENT_T

 set DATABASE_HOME=C:\oracledb\product\11.2.0\dbhome_1  
 set JAVA_HOME=C:\Oracle\Java\hotspot\jdk  
 set CLASSPATH=%DATABASE_HOME%\jdbc\lib\ojdbc5.jar;%DATABASE_HOME%\sqlj\lib\translator.jar;%DATABASE_HOME%\sqlj\lib\runtime12.jar  
 %JAVA_HOME%\bin\java -classpath %CLASSPATH% oracle.jpub.Doit -url=jdbc:oracle:thin:@ebiz.springsoa.com:1521:ebiz -user=apps/***** -sql=WF_EVENT_T -usertypes=oracle -methods=false -package=com.springsoa.jpub -usertypes=jdbc  

This generate WF_EVENT_T.java, which you can copy to your project under right package folder (e.g. com.springsoa.jpub) in this case.

2) Writing Java code for Dequeue

 package com.springsoa.util;  
 import com.springsoa.jpub.WF_EVENT_T;  
 import java.sql.Connection;  
 import java.sql.DriverManager;  
 import java.util.HashMap;  
 import java.util.Map;  
 import oracle.AQ.AQAgent;  
 import oracle.AQ.AQDequeueOption;  
 import oracle.AQ.AQDriverManager;  
 import oracle.AQ.AQMessage;  
 import oracle.AQ.AQQueue;  
 import oracle.AQ.AQSession;  
 import oracle.sql.STRUCT;  
 public class AQQueueConsumer {  
   public AQSession createSession() {  
     Connection connection;  
     AQSession aqSession = null;  
     try {  
       Class.forName("oracle.jdbc.driver.OracleDriver");  
       connection = DriverManager.getConnection("jdbc:oracle:thin:@ebiz.springsoa.com:1521:ebiz","apps", "******");  
       connection.setAutoCommit(true);  
       Class.forName("oracle.AQ.AQOracleDriver");  
       aqSession = AQDriverManager.createAQSession(connection);  
     } catch (Exception ex) {  
       ex.printStackTrace();  
     }   
     return aqSession;  
   }  
   public void listDurableSubscribers(AQSession aqSession, String queueOwner, String queueName) throws Exception {  
     AQQueue queue = aqSession.getQueue(queueOwner, queueName);        // select * from all_queues  
     AQAgent[] agents = queue.getSubscribers();  
     for(int i=0; i<agents.length; i++) {  
       System.out.println(" Consumers : " + agents[i].getName() );  
     }  
   }  
   public Map<String,String> dequeue(AQSession aqSession, String queueOwner, String queueName) throws Exception {  
     Map<String,String> messageMap = new HashMap<String,String>();  
     AQQueue queue = aqSession.getQueue(queueOwner, queueName);        // select * from all_queues  
     AQDequeueOption aqDequeueOption = new AQDequeueOption();        // https://docs.oracle.com/cd/B19306_01/server.102/b14257/aq_views.htm  
     aqDequeueOption.setConsumerName("CUSTOM");   // ORA_6H0JECQ66OO48D1N6SRJIC1N61 ORA_70skccq26kok8cq6752j0da665      
     aqDequeueOption.setNavigationMode(AQDequeueOption.NAVIGATION_FIRST_MESSAGE);  
     //aqDequeueOption.setCondition("tab.user_data.event_name = 'oracle.apps.xxbmc.employeePublish.updateEvent'");  
     aqDequeueOption.setWaitTime(AQDequeueOption.WAIT_NONE);  
     AQMessage message = queue.dequeue(aqDequeueOption, WF_EVENT_T.class);  
     if( message != null && message.getObjectPayload() != null && message.getObjectPayload().getPayloadData() != null ) {  
       WF_EVENT_T wfEventT = (WF_EVENT_T) message.getObjectPayload().getPayloadData();  
       messageMap.put("eventName",wfEventT.getEventName());  
       messageMap.put("eventKey",wfEventT.getEventKey());  
       if( wfEventT.getParameterList() != null && wfEventT.getParameterList().getArray() != null ) {  
         Object[] parameters = (Object[]) wfEventT.getParameterList().getArray();  
         for(int i=0;i<parameters.length;i++) {  
           //System.out.println(" parameter[" + i + "] : " + ( (oracle.sql.STRUCT)parameters[i] ).dump() );  
           STRUCT parameter = (oracle.sql.STRUCT)parameters[i];  
           Object[] attributes = parameter.getAttributes();  
           if(attributes.length == 2) {  
             messageMap.put((String)attributes[0],(String)attributes[1]);  
           }  
         }  
       }  
     }  
     return messageMap;  
   }  
   public static void main(String[] args) {  
     AQSession aq_sess = null;  
     try {  
       AQQueueConsumer aqQueueConsumer = new AQQueueConsumer();  
       AQSession aqSession = aqQueueConsumer.createSession();  
       aqQueueConsumer.listDurableSubscribers(aqSession,"APPS", "WF_BPEL_Q");  
       Map map = aqQueueConsumer.dequeue(aqSession,"APPS", "WF_BPEL_Q");  
       System.out.println(" map " + map );  
     } catch (Exception ex) {  
       System.out.println("Exception: " + ex);  
       ex.printStackTrace();  
     }  
   }  
 }  

In above code, dequeue method does the dequeue, code is quite similar to what we have in PLSQL, however some extra work needs to be done to get all parameters from WF_EVENT_T as WF_PARAMETER is not given as SQL object.

Please note that message in WF_BPEL_QTAB will not be removed or marked as status 2 until all durable subscribers consumes the message. However, message will be deleted for a specific subscriber ("CUSTOM" in above case) in AQ$_WF_BPEL_QTAB_I. 

Purge records

 EXECUTE DBMS_AQADM.PURGE_QUEUE_TABLE('APPS.WF_BPEL_QTAB', NULL, NULL);  

AQ Durable Subscripton

AQ Durable subscription works like durable topic subscription, and below are some tweaks we can use to check how it works internally and might help troubleshoot further.

 EXECUTE DBMS_AQADM.STOP_QUEUE ( QUEUE_NAME => 'CS_MULTI_Q');  
 EXECUTE DBMS_AQADM.DROP_QUEUE ( QUEUE_NAME => 'CS_MULTI_Q');  
 EXECUTE DBMS_AQADM.DROP_QUEUE_TABLE ( QUEUE_TABLE => 'CS_MULTI_QTAB');  
 EXECUTE DBMS_AQADM.CREATE_QUEUE_TABLE ( QUEUE_TABLE     => 'CS_MULTI_QTAB', MULTIPLE_CONSUMERS => TRUE, QUEUE_PAYLOAD_TYPE => 'RAW' );   
 EXECUTE DBMS_AQADM.CREATE_QUEUE ( QUEUE_NAME => 'CS_MULTI_Q', QUEUE_TABLE => 'CS_MULTI_QTAB', RETENTION_TIME => DBMS_AQADM.INFINITE );   
 EXECUTE DBMS_AQADM.START_QUEUE ( QUEUE_NAME     => 'CS_MULTI_Q');   

Create durable subscriber

 DECLARE  
   subscriber sys.aq$_agent;  
 BEGIN  
   subscriber := sys.aq$_agent('ONE', NULL, NULL);  
   DBMS_AQADM.ADD_SUBSCRIBER(queue_name => 'CS_MULTI_Q',subscriber => subscriber);  
   subscriber := sys.aq$_agent('TWO', NULL, NULL);  
   DBMS_AQADM.ADD_SUBSCRIBER(queue_name => 'CS_MULTI_Q',subscriber => subscriber);   
 END;  

En-queue Messages:

 DECLARE  
   ENQUEUE_OPTIONS   DBMS_AQ.ENQUEUE_OPTIONS_T;  
   MESSAGE_PROPERTIES DBMS_AQ.MESSAGE_PROPERTIES_T;  
   MESSAGE_HANDLE   RAW(16);  
   MESSAGE       RAW(4096);  
 BEGIN  
   MESSAGE := HEXTORAW(RPAD('FF',4095,'FF'));   
   DBMS_AQ.ENQUEUE(QUEUE_NAME => 'CS_MULTI_Q',  
      ENQUEUE_OPTIONS   => ENQUEUE_OPTIONS,  
      MESSAGE_PROPERTIES  => MESSAGE_PROPERTIES,     
      PAYLOAD       => MESSAGE,          
      MSGID        => MESSAGE_HANDLE);  
   COMMIT;  
 END;  

Now if we see, below queries shows the all the subscribers to this queue (ONE and TWO), and their numbers. (This query shows different results than select * from all_subscribers)

 SELECT * FROM AQ$_CS_MULTI_QTAB_S WHERE NAME IS NOT NULL;  

Below query shows the multiple copy of the messages - one per each subscriber.

 SELECT * FROM AQ$_CS_MULTI_QTAB_I;  

Dequeue the message as subscriber "TWO" using below PLSQL block:

 DECLARE  
   DEQUEUE_OPTIONS   DBMS_AQ.DEQUEUE_OPTIONS_T;  
   MESSAGE_PROPERTIES DBMS_AQ.MESSAGE_PROPERTIES_T;  
   MESSAGE_HANDLE   RAW(16);  
   MESSAGE       RAW(4096);  
 BEGIN  
      DEQUEUE_OPTIONS.WAIT := DBMS_AQ.NO_WAIT;  
      DEQUEUE_OPTIONS.CONSUMER_NAME := 'TWO';  
      DEQUEUE_OPTIONS.NAVIGATION := DBMS_AQ.FIRST_MESSAGE;  
      DBMS_AQ.DEQUEUE(QUEUE_NAME => 'CS_MULTI_Q',  
       DEQUEUE_OPTIONS  => DEQUEUE_OPTIONS,  
       MESSAGE_PROPERTIES => MESSAGE_PROPERTIES,  
       PAYLOAD      => MESSAGE,  
       MSGID       => MESSAGE_HANDLE);  
      DBMS_OUTPUT.PUT_LINE ('Message: ' || message );  
      DEQUEUE_OPTIONS.NAVIGATION := DBMS_AQ.NEXT_MESSAGE;  
      COMMIT;  
 END;    

After we dequeue this message, we can see that in queue table (CS_MULTI_QTAB), status field is still set to 0. This is because other subscribers (e.g. ONE) has not dequeued the message. Also we can see that in table AQ$_CS_MULTI_QTAB_I, the message for subscriber "TWO" is deleted.

Once all subscribers dequeues the message, the status field in Queue table (CS_MULTI_QTAB) will be set to 2 (if retention time is set), or message from queue table will be deleted.


Purge Records

 EXECUTE DBMS_AQADM.PURGE_QUEUE_TABLE('CS_MULTI_QTAB', NULL, NULL);