Restarting Weblogic when DataSource is down

We observed that if one/more of the data source is down (if database is down or expired password), when we restart the managed server it doesn't start and remains in Admin state. The reason behind that is the Initial and Minimum capacity specified in Data Source. E.g. for majority of the cases, when we create data-source, following values are set to 1 by default, which was causing problem.

Changing them to 0, resolves the restart issue. There are still tons of constant error in logs related to the data source which is down, but at least server comes up fine.

Change hostname in weblogic datasource / connection pool without restarting entire weblogic

A while ago, I posted to change password without restart weblogic, we saw similar issue when we had to change the hostname of the datasource. Changing the hostname in datasource didn't show as effective right away, however after following below steps, it worked like a charm:

• Change the hostname in connection pool on WebLogic Console for a specific datasource 

• Untarget the datasource from all servers

 

• Retarget the datasource to all servers where it was configured before

• Shutdown Datasource
• Start Datasource

Shutdown composite

Just a sample code to shutdown your composite in your Java Embed activity:

try {       
   oracle.soa.management.facade.Locator locator = oracle.soa.management.facade.LocatorFactory.createLocator();         
   oracle.soa.management.util.CompositeFilter compositeFilter = new oracle.soa.management.util.CompositeFilter();         
   compositeFilter.setCompositeDN("default/CompositeShutdown!1.0");         
   java.util.List<oracle.soa.management.facade.Composite> composites = locator.getComposites(compositeFilter);         
   for(oracle.soa.management.facade.Composite composite: composites) {         
      composite.stop();         
   }       
} catch (Exception e) {         
   e.printStackTrace();        
}

Force local optimization in cluster

It is done by default in single node installation, but some of configuration needs to be done specially for cluster.

•  (optional) : Make sure composite.xml have oracle.webservices.local.optimization for this as true (which is default), or doesn't have this property at all so true would be default value.

•  Configure serverURL as Load Balancer URL

• Configure Frontend HTTP Host and Frontend HTTP Port as Load Balancer Server and Port.

• Make sure in EM console, the invocation shows as "local" as below

WLST in Cluster - Try to use server name with application name. Multiple server targets deployed for application

While running WLST for policy related commands in single server domain, it worked fine.

listWebServicePolicies(None, 'default/AIASessionPoolManager[1.0]','soa','client','AIASessionPoolManager')
listWebServicePolicies(None, 'default/AIAErrorTaskAdministrationProcess[1.0]', 'soa', 'aiaerrortaskadministrationprocess_client_ep', 'AIAErrorTaskAdministrationProcess_pt')
listWebServicePolicies(None, 'default/AIAAsyncErrorHandlingBPELProcess[1.0]', 'soa', 'client', 'AIAAsyncErrorHandlingBPELProcess')
enableWebServicePolicy(None, 'default/AIASessionPoolManager[1.0]', 'soa', 'client', 'AIASessionPoolManager', 'oracle/aia_wss_saml_or_username_token_service_policy_OPT_ON', false, None )
enableWebServicePolicy(None, 'default/AIAErrorTaskAdministrationProcess[1.0]', 'soa', 'aiaerrortaskadministrationprocess_client_ep', 'AIAErrorTaskAdministrationProcess_pt', 'oracle/aia_wss_saml_or_username_token_service_policy_OPT_ON', false, None )
enableWebServicePolicy(None, 'default/AIAAsyncErrorHandlingBPELProcess[1.0]', 'soa', 'client', 'AIAAsyncErrorHandlingBPELProcess', 'oracle/aia_wss_saml_or_username_or_http_token_service_policy_OPT_ON', false, None )

When running in domain with multiple server, it failed with error message:
Try to use server name with application name. Multiple server targets deployed for application

Resolution is quite simple, run it against one node or each node:

listWebServicePolicies('/soa_server1/soa-infra', 'default/AIASessionPoolManager[1.0]','soa','client','AIASessionPoolManager')
listWebServicePolicies('/soa_server1/soa-infra', 'default/AIAErrorTaskAdministrationProcess[1.0]', 'soa', 'aiaerrortaskadministrationprocess_client_ep', 'AIAErrorTaskAdministrationProcess_pt')
listWebServicePolicies('/soa_server1/soa-infra', 'default/AIAAsyncErrorHandlingBPELProcess[1.0]', 'soa', 'client', 'AIAAsyncErrorHandlingBPELProcess')
enableWebServicePolicy('/soa_server1/soa-infra', 'default/AIASessionPoolManager[1.0]', 'soa', 'client', 'AIASessionPoolManager', 'oracle/aia_wss_saml_or_username_token_service_policy_OPT_ON', false, None )
enableWebServicePolicy('/soa_server1/soa-infra', 'default/AIAErrorTaskAdministrationProcess[1.0]', 'soa', 'aiaerrortaskadministrationprocess_client_ep', 'AIAErrorTaskAdministrationProcess_pt', 'oracle/aia_wss_saml_or_username_token_service_policy_OPT_ON', false, None )
enableWebServicePolicy('/soa_server1/soa-infra', 'default/AIAAsyncErrorHandlingBPELProcess[1.0]', 'soa', 'client', 'AIAAsyncErrorHandlingBPELProcess', 'oracle/aia_wss_saml_or_username_or_http_token_service_policy_OPT_ON', false, None )

listWebServicePolicies('/soa_server2/soa-infra', 'default/AIASessionPoolManager[1.0]','soa','client','AIASessionPoolManager')
listWebServicePolicies('/soa_server2/soa-infra', 'default/AIAErrorTaskAdministrationProcess[1.0]', 'soa', 'aiaerrortaskadministrationprocess_client_ep', 'AIAErrorTaskAdministrationProcess_pt')
listWebServicePolicies('/soa_server2/soa-infra', 'default/AIAAsyncErrorHandlingBPELProcess[1.0]', 'soa', 'client', 'AIAAsyncErrorHandlingBPELProcess')
enableWebServicePolicy('/soa_server2/soa-infra', 'default/AIASessionPoolManager[1.0]', 'soa', 'client', 'AIASessionPoolManager', 'oracle/aia_wss_saml_or_username_token_service_policy_OPT_ON', false, None )
enableWebServicePolicy('/soa_server2/soa-infra', 'default/AIAErrorTaskAdministrationProcess[1.0]', 'soa', 'aiaerrortaskadministrationprocess_client_ep', 'AIAErrorTaskAdministrationProcess_pt', 'oracle/aia_wss_saml_or_username_token_service_policy_OPT_ON', false, None )
enableWebServicePolicy('/soa_server2/soa-infra', 'default/AIAAsyncErrorHandlingBPELProcess[1.0]', 'soa', 'client', 'AIAAsyncErrorHandlingBPELProcess', 'oracle/aia_wss_saml_or_username_or_http_token_service_policy_OPT_ON', false, None )

Connection timeout and recovery - usecase

Connection timeout and instance retries are quite related and easy to configure from Oracle SOA 11g console. Here is one of the use case we wanted go over to see how it works out in realistic production scenario.

Scenario for our environment:
Multiple processes deployed with interaction to mostly external web services and database (local and XA). There are some services or db calls which can take 10/20 minutes (entire call or even from one dehydration point to next dehydration point) and some should take no longer than 10 seconds (SLA). If takes longer than configured time, we certainly don't want to wait, and error out and do another process (e.g. send error notification) or send back error message, etc..


Configuration Screens: Below provides information on how to configure connection retries or various timeouts in SOA 11g.
 


Connection Retries

• EM -> SOA-INFRA -> SOA Administration -> Common

• EM -> SOA-INFRA -> SOA Administration -> BPEL

• Retries related with global transaction rollback

• Scheduled Retries 

Connection Timeouts

• BPEL engine (EJB Timeout)

• console -> deployments -> soa-infra -> control -> [ SELECT BEAN ] -> Configuration -> X seconds

• BPELEngineBean
• BPELDeliveryBean
• BPELActivityManagerBean
• BPELServerManagerBean
• BPELProcessManagerBean
• BPELInstanceManagerBean
• BPELFinderBean

• Sync Max

• Query Timeout : Query timeout can be specified either from BPEL process or at data source level. Value in BPEL takes precedence over data source. 

• Composite Level QueryTimeout (JCA file or Adapter Wizard)

  

• Data Source Level - QueryTimeOut (Statement Timeout)

• XA Timeout : if data source is XA, we can specify XA timeout, which takes precedence over JTA timeout

• HTTP Connection Timeout (WS partnerlink or HTTP binding calls) : we can specify that in composite.xml file

  <reference name="PLSQL_WS" ui:wsdlLocation="http://mycomputer:8001/soa-infra/services/default/TimeoutChildComposite/PLSQL_JCA.wsdl">
    <interface.wsdl interface="http://xmlns.oracle.com/TokenTesting/TimeoutChildComposite/PLSQL_JCA#wsdl.interface(PLSQL_JCA)"/>
    <binding.ws port="http://xmlns.oracle.com/TokenTesting/TimeoutChildComposite/PLSQL_JCA#wsdl.endpoint(plsql_jca_client_ep/PLSQL_JCA_pt)" location="http://mycomputer:8001/soa-infra/services/default/TimeoutChildComposite/plsql_jca_client_ep?WSDL" soapVersion="1.1">
        <property name="oracle.webservices.httpConnTimeout" type="xs:integer" many="false" override="may">10000</property>
        <property name="oracle.webservices.httpReadTimeout" type="xs:integer" many="false" override="may">10000</property>
        <property name="weblogic.wsee.wsat.transaction.flowOption" type="xs:string" many="false">WSDLDriven</property>
    </binding.ws>
  </reference>

Target Environment Settings
Below are the settings we used to achieve our scenario mentioned below.


Scenario for our environment:
Multiple processes deployed with interaction to mostly external web services and database (local and XA). There are some services or db calls which can take 10/20 minutes (entire call or even from one dehydration point to next dehydration point) and some should take no longer than 10 seconds (SLA). If takes longer than configured time, we certainly don't want to wait, and error out and do another process (e.g. send error notification) or send back error message, etc..


1. Disable All Retries
Disable All automated retries : It makes no sense for BPEL to automated retry upon JTA timeout or failed instances for any other reason. As it could be data issue or environment issue, and blind retries usually doesn't solve any problem. Retries can be orchestrated via Fault Policy or manually for a specific instance.

• EM -> SOA-INFRA -> SOA Administration -> Common -> more SOA Infra advanced configuration properties

• GlobalTxMaxRetry : 0
• GlobalTxRetryInterval : 0

• EM -> SOA-INFRA -> SOA Administration -> BPEL -> more BPEL Configuration Properties

• ExpirationMaxRetry : 0
• MaxRecoveryAttempt : 0
• RecoveryConfig -> RecurringScheduleConfig

• maxMessageRaiseSize : 0
• startWindowTime: 00:00
• stopWindowTime: 00:00

• RecoveryConfig -> StartupScheduleConfig

• maxMessageRaiseSize : 0
• startupRecoveryDuration : 0
• subsequentTriggerDelay : 0

2. Increase Global Timeouts
Increase all global timeout to support the slowest process in the system. Unfortunately there is no exception to this. You can not assign global timeout to specific set of processes. (I believe in 12c they might come up with feature to assign at partition level). We must increase all global timeout so that legitimate slowest process can be completed. After that we have to work backward so that process which needs to be finished faster doesn't get punished. We set all global timeout to 1200 seconds.

• BPEL engine (EJB Timeout)

• console -> deployments -> soa-infra -> control -> [ SELECT BEAN ] -> Configuration -> 1200 seconds

• BPELEngineBean
• BPELDeliveryBean
• BPELActivityManagerBean
• BPELServerManagerBean
• BPELProcessManagerBean
• BPELInstanceManagerBean
• BPELFinderBean

• SyncMaxWaitTime

• EM -> SOA-INFRA -> SOA Administration -> BPEL -> more BPEL Configuration Properties -> SyncMaxWaitTime : 1200 seconds

3. Reset XA and Query Timeout
XA and Query Timeout doesn't work, at least not the way it is expected. Query and DB connection is going to get stuck as long as query runs (as explained in blog), unless it takes longer than BPEL/JTA timeout.

• QueryTimeout

• Composite JCA file (or Wizard)

 

• Data source Level

• Transaction Timeout

• Enable (Set XA Transaction Timeout)
• XA Transaction Timeout : 0
• XA Retry Duration : 0
• XA Retry Interval : 0

4. Process Configuration with SLA

Now as we have global timeout set at 20 minutes, but there are process which we don't want to wait for 20 minutes, but want to finish in matter or seconds (e.g. 10 seconds), and if not completed we want to do special processing (e.g. send email or return with error), etc.

Process is calling SOAP WS or HTTP Binding

We can specify httpConnTimeout and httpReadTimeout.
Note: If we don't specify those values, and if call takes longer than BPEL engine timeout/JTA timeout, entire transaction will be rolled back and process won't be able to do any error handling.

  <reference name="PLSQL_WS" ui:wsdlLocation="http://mycomputer:8001/soa-infra/services/default/TimeoutChildComposite/PLSQL_JCA.wsdl">
    <interface.wsdl interface="http://xmlns.oracle.com/TokenTesting/TimeoutChildComposite/PLSQL_JCA#wsdl.interface(PLSQL_JCA)"/>
    <binding.ws port="http://xmlns.oracle.com/TokenTesting/TimeoutChildComposite/PLSQL_JCA#wsdl.endpoint(plsql_jca_client_ep/PLSQL_JCA_pt)" location="http://mycomputer:8001/soa-infra/services/default/TimeoutChildComposite/plsql_jca_client_ep?WSDL" soapVersion="1.1">
        <property name="oracle.webservices.httpConnTimeout" type="xs:integer" many="false" override="may">5000</property>
        <property name="oracle.webservices.httpReadTimeout" type="xs:integer" many="false" override="may">5000</property>
        <property name="weblogic.wsee.wsat.transaction.flowOption" type="xs:string" many="false">WSDLDriven</property>
    </binding.ws>
  </reference>

Process is calling DB Adapter
If you are calling JDBC, it is quite hard to provide connection timeout. Query timeout and XA timeout (for XA data sources) doesn't work they way expected as mentioned in blog. There are some work around we can use:

We can put DB adapter in a separate composite, or java WS, and then call from original composite with HTTP timeout, as below:

JDBC Connection Timeout

JDBC connection timeout works quite differently. E.g. for HTTP connection if you specify timeout, it will timeout and thread control will be given back to the caller. In JDBC connection timeout, it doesn't work that way, at least with JDBC driver. E.g. if we are making call to a stored procedure with timeout of 10 seconds, if stored procedure takes 5 minutes, it will throw Timeout Exception after 5 minutes.

E.g. for below simple stored procedure with sleep:

CREATE OR REPLACE PACKAGE SLEEP_PACKAGE AS
    PROCEDURE SLEEP_PROC (
        P_SECONDS IN NUMBER, 
        P_MESSAGE OUT VARCHAR2 
    );
    PROCEDURE HELLO_WORLD(
        P_INPUT IN VARCHAR2, 
        P_OUTPUT OUT VARCHAR2 
    );
END SLEEP_PACKAGE;
/

CREATE OR REPLACE PACKAGE BODY SLEEP_PACKAGE AS

    PROCEDURE SLEEP_PROC (
        P_SECONDS IN NUMBER, 
        P_MESSAGE OUT VARCHAR2 
        ) IS
        MESSAGE       VARCHAR2(1000);
        START_TIME    TIMESTAMP;
        END_TIME      TIMESTAMP;
    BEGIN
        SELECT SYSTIMESTAMP INTO START_TIME FROM DUAL;
        MESSAGE := 'START: ' || START_TIME;
        DBMS_OUTPUT.PUT_LINE('SLEEP PROCEDURE START : ' || START_TIME );
        DBMS_LOCK.SLEEP( P_SECONDS );
        SELECT SYSTIMESTAMP INTO END_TIME FROM DUAL;
        MESSAGE := MESSAGE || ' END: ' || END_TIME || ' TOTAL_TIME: ' || P_SECONDS || ' SECONDS';
        DBMS_OUTPUT.PUT_LINE('SLEEP PROCEDURE END   : ' || END_TIME );
        DBMS_OUTPUT.PUT_LINE(MESSAGE);
        P_MESSAGE := MESSAGE;
    END SLEEP_PROC;

    PROCEDURE HELLO_WORLD (
        P_INPUT IN VARCHAR2, 
        P_OUTPUT OUT VARCHAR2 
        ) IS
    BEGIN
        P_OUTPUT := UPPER(P_INPUT);
    END HELLO_WORLD;
    

END SLEEP_PACKAGE;
/

Below is java code which calls this stored procedure with 5 second timeout and 120 seconds sleep in database stored procedure.

    public String callAdvanceSleepService(String dataSourceName, String input) throws Exception {
        try {
            Connection connection = getConnection(dataSourceName);
            CallableStatement statement = connection.prepareCall("{call SPRINGJMS.SLEEP_PACKAGE.SLEEP_PROC(?,?)}");
            statement.setInt(1, Integer.parseInt(input));
            statement.registerOutParameter(2, Types.VARCHAR);
            statement.setQueryTimeout(5);
            System.out.println(new Timestamp(System.currentTimeMillis()) + " Executing SPRINGJMS.SLEEP_PACKAGE.SLEEP_PROC ");
            int execution = statement.executeUpdate();
            String output = statement.getString(2);
            System.out.println(" execution : " + execution + " output " + output );
            return output;
        } catch(Exception e) {
            System.out.println(new Timestamp(System.currentTimeMillis()) + " Exception ");
            e.printStackTrace();
            throw e;
        }
    }        
    
    public static void main(String args[]) throws Exception {
        WaitService waitService = new WaitService();
        waitService.callAdvanceSleepService("local", "120");
    }

In above example, the code will throw an exception after 120 seconds instead of timeout of 5 seconds.

Weblogic Data Source

Now if we configure the datasource at weblogic level and execute the code as deployed service, the same thing remains true. The data source does have Statement Query Timout, but as I have it in my code, it will overwrite the connection level query timeout.

SOA DB Adapter

For SOA DB adapter, the behavior remains the same, and Query Timeout can be specified either at data source level or in JCA file as below.

<adapter-config name="PLSQL_WAIT" adapter="Database Adapter" wsdlLocation="PLSQL_WAIT.wsdl" xmlns="http://platform.integration.oracle/blocks/adapter/fw/metadata">
  
  <connection-factory location="eis/DB/springjms" UIConnectionName="springjms" adapterRef=""/>
  <endpoint-interaction portType="PLSQL_WAIT_ptt" operation="PLSQL_WAIT">
    <interaction-spec className="oracle.tip.adapter.db.DBStoredProcedureInteractionSpec">
      <property name="SchemaName" value="SPRINGJMS"/>
      <property name="PackageName" value="SLEEP_PACKAGE"/>
      <property name="ProcedureName" value="SLEEP_PROC"/>
      <property name="QueryTimeout" value="60"/>
      <property name="GetActiveUnitOfWork" value="false"/>
    </interaction-spec>
    <output/>
  </endpoint-interaction>

</adapter-config>

However, in DB Adapter, BPEL EJB timeout and JTA would also play critical role. JTA/BPEL timeout would take precedence over any other.

Query Timeout	Actual Query Execution Time	JTA and BPEL engine timeout	Actal Output
30	120	60	Get JTA timeout exception after 60 seconds - Exception is not catched in BPEL
30	120	300	Get SQLTimeoutException exception after 120 seconds - Exception in catched in BPEL
None	120	60	Get JTA timeout after 60 seconds - Exception is not catched in BPEL
600	120	60	Get JTA timeout after 60 seconds - Exception no catched in BPEL

For a specific use case, if we want to achieve similar effect as HTTP connection timeout, we can put the DB adapter service in separate composite, that way caller process (Composite A) will continue its work after timeout and Composite B can be stuck if procedure is slow.

Composite A (BPEL) -> Composite B (BPEL + DB Adapter)