Friday, September 6, 2013

SOA 11g purge specific instances

You can delete specific instances via EM, but we wanted to delete it from database as we were facing issue on EM console batch delete mentioned here.

The OOTB purge process seems to be having some challenges:

DECLARE
  MIN_CREATION_DATE TIMESTAMP;
  MAX_CREATION_DATE TIMESTAMP;
  BATCH_SIZE NUMBER;
  MAX_RUNTIME NUMBER;
  RETENTION_PERIOD TIMESTAMP;
  PURGE_PARTITIONED_COMPONENT BOOLEAN;
  COMPOSITE_NAME VARCHAR2(200);
  COMPOSITE_REVISION VARCHAR2(200);
  SOA_PARTITION_NAME VARCHAR2(200);
BEGIN
  MIN_CREATION_DATE := systimestamp - 10000;
  MAX_CREATION_DATE := systimestamp;
  BATCH_SIZE := 20000;
  MAX_RUNTIME := 60;
  RETENTION_PERIOD := null;
  PURGE_PARTITIONED_COMPONENT := true;
  COMPOSITE_NAME := 'MyProcess';
  COMPOSITE_REVISION := '1.0';
  SOA_PARTITION_NAME := 'MyPartition';

  SOA.DELETE_INSTANCES(
    MIN_CREATION_DATE => MIN_CREATION_DATE,
    MAX_CREATION_DATE => MAX_CREATION_DATE,
    BATCH_SIZE => BATCH_SIZE,
    MAX_RUNTIME => MAX_RUNTIME,
    RETENTION_PERIOD => RETENTION_PERIOD,
    PURGE_PARTITIONED_COMPONENT => PURGE_PARTITIONED_COMPONENT,
    COMPOSITE_NAME => COMPOSITE_NAME,
    COMPOSITE_REVISION => COMPOSITE_REVISION,
    SOA_PARTITION_NAME => SOA_PARTITION_NAME
  );
--rollback; 
END;    

  • Procedure doesn't allow selective purge - e.g. I can not purge specific list of instance ids
  • Procedure runs pruning job which basically prevents purging faulted or running instances
Here is workaround :

  • Insert the ecid in ecid_purge table. Below is sample query, but select statement can be changed as you like.
truncate table ecid_purge;
insert into ecid_purge   select distinct ecid from composite_instance where composite_dn  like '%CustomerParty/SyncCustomerPartyListEBizProvABCSImpl!1.0%';  
commit; 

  • Run the following procedure to clean up the data. It will clean up faulted and open running instances as well
begin
  soa_orabpel.deleteComponentInstances('ecid_purge',true);
  soa_workflow.deleteComponentInstances('ecid_purge');
  soa_mediator.deleteComponentInstances('ecid_purge');
  soa_decision.deleteComponentInstances('ecid_purge');
  soa_fabric.deleteCompositeInstances('ecid_purge',true);
  commit;
end;


Thursday, September 5, 2013

Enable logging in purge process

I guess out of the box 11g SOA instance purge process kinda sucks, and majority of the time we end up customizing it for faster execution or just so that it is easy to read. However, time to time, I end up using OOTB purge process. We were trying to delete it from EM console and somehow just got hung and faced below error in logs.


Caused By: java.lang.NullPointerException
        at sym.productext.ui.bean.CreateDerivationHeaderBean.doDeleteRow(CreateDerivationHeaderBean.java:148)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at com.sun.el.parser.AstValue.invoke(AstValue.java:187)
        at com.sun.el.MethodExpressionImpl.invoke(MethodExpressionImpl.java:297)
        at org.apache.myfaces.trinidadinternal.taglib.util.MethodExpressionMethodBinding.invoke(MethodExpressionMethodBinding.java:53)
        at org.apache.myfaces.trinidad.component.UIXComponentBase.broadcastToMethodBinding(UIXComponentBase.java:1256)

So I had to try it from database. We ran following standard routine with standard parameters to delete all instances of specific composite, and the process completed in a few seconds - nothing got deleted and no error.

DECLARE
  MIN_CREATION_DATE TIMESTAMP;
  MAX_CREATION_DATE TIMESTAMP;
  BATCH_SIZE NUMBER;
  MAX_RUNTIME NUMBER;
  RETENTION_PERIOD TIMESTAMP;
  PURGE_PARTITIONED_COMPONENT BOOLEAN;
  COMPOSITE_NAME VARCHAR2(200);
  COMPOSITE_REVISION VARCHAR2(200);
  SOA_PARTITION_NAME VARCHAR2(200);
BEGIN
  MIN_CREATION_DATE := systimestamp - 10000;
  MAX_CREATION_DATE := systimestamp;
  BATCH_SIZE := 20000;
  MAX_RUNTIME := 60;
  RETENTION_PERIOD := null;
  PURGE_PARTITIONED_COMPONENT := true;
  COMPOSITE_NAME := 'MyProcess';
  COMPOSITE_REVISION := '1.0';
  SOA_PARTITION_NAME := 'MyPartition';

  SOA.DELETE_INSTANCES(
    MIN_CREATION_DATE => MIN_CREATION_DATE,
    MAX_CREATION_DATE => MAX_CREATION_DATE,
    BATCH_SIZE => BATCH_SIZE,
    MAX_RUNTIME => MAX_RUNTIME,
    RETENTION_PERIOD => RETENTION_PERIOD,
    PURGE_PARTITIONED_COMPONENT => PURGE_PARTITIONED_COMPONENT,
    COMPOSITE_NAME => COMPOSITE_NAME,
    COMPOSITE_REVISION => COMPOSITE_REVISION,
    SOA_PARTITION_NAME => SOA_PARTITION_NAME
  );
--rollback; 
END;                   



Upon looking further in PLSQL, found that it is using log_info and debug_purge to log the messages, but it will only be enabled if $$debug_on is there.

How to enable purge logging
ALTER PROCEDURE debug_purge  COMPILE PLSQL_CCFLAGS = 'debug_on:TRUE' REUSE SETTINGS;

ALTER PROCEDURE log_info COMPILE PLSQL_CCFLAGS = 'debug_on:TRUE' REUSE SETTINGS;


How to disable purge logging
ALTER PROCEDURE debug_purge COMPILE PLSQL_CCFLAGS = 'debug_on:false' REUSE SETTINGS;

ALTER PROCEDURE log_info COMPILE PLSQL_CCFLAGS = 'debug_on:false' REUSE SETTINGS;

The same can be found under $SOA_HOME/rcu/integration/soainfra/oracle/soa_purge/common/debug_on.sql and debug_off.sql.

It gives quite a bit of information on purge routine once it is enabled.

Saturday, July 13, 2013

Remote JMS Server

Using JMS server (topic/queues) is quite common for pub/sub or any other decoupling design patterns, and we tried putting it in completely different dedicated cluster. Having it in dedicated cluster helps decouple SOA restart from JMS unavailability and also JMS can be scaled much easily. We faced some issue while connecting to remote JMS server, and just wanted to provide two solution which worked out well.

Basic layout: Below diagram describes separate SOA and JMS servers. Topic and Connection Factories are created on JMS server. Composite and JMS adapters are running on SOA servers.





Solution 1:Configure JMS adapter (eis/jms/DemoJMSAdapter) with remote server settings as below (Note: Exactly the same settings will be applied if you would be using stand alone JMS java client).


e.g. java.naming.factory.initial=weblogic.jndi.WLInitialContextFactory;java.naming.provider.url=t3://host1:port1,host2:port2;java.naming.security.principal=weblogic;java.naming.security.credentials=welcome1
  • Factory properties, username and password remains blank if connection factory and topic are locally available to SOA cluster, but in our case, we have to provide Factory Properties so that it can lookup jms/DemoTopicCF in remote JMS server (jms_server1)  
  • JCA file inside the composite will have no extra settings. It will point to eis/jms/DemoJMSAdapter and jms/DemoTopic


Solution 2:
Configure Foreign JNDI provider for soa_server1, so topic and connection factory can be used just like local resources inside soa_server1.

a) configure Foreign JNDI provider (target it to soa_server1) and provide the connection detail for jms_server1 as below.



b) configure link for jms/DemoTopic and jms/DemoTopicCF
Note: in my case, jms_server1 and soa_server1 are still in same domain so somehow it didn't let me reuse JNDI name, but if they are in different domain same name can be used.



c) Configure JMS Adapter and Composite JCA as local resource


JCA file will point to eis/jms/DemoJMSAdapter and jms/local/DemoTopic.


Comparison:
  • Solution 1 is easier and straight forward to configure. Solution 2 requires extra set of configuration
  • In long term, if you are using multiple topic and queues, solution 2 might provide better solution with centralized authentication details
  • If you are in same domain, solution 2 has another problem of duplicate JNDI, so extra set of JNDI is required

Tuesday, March 12, 2013

Service Throttling

We had to throttle the end service, and following are the two approaches which came quite handy:  

1. Out of the Box OSB
For the Business Service in OSB, we can configure the throttling as below. It allows us to configure concurrent threads, thread queue size, message expiration

  
2. HTTP Proxy Service
There could be a scenario where we don't have OSB or we don't want to use OSB to avoid piling up the request affect other important integration. The other standard approach to achieve similar thing via HTTP Proxy servlet and throttle down the number of threads to desired value and queue rest of the requests. This is involves some coding but it can be deployed on any weblogic managed server.

Create a Proxy Servlet
Instead of writing code from scratch, used com.jsos.httpproxy.HttpProxyServlet as below:

<?xml version = '1.0' encoding = 'windows-1252'?>
<web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://java.sun.com/xml/ns/javaee http://java.sun.com/xml/ns/javaee/web-app_2_5.xsd"
         version="2.5" xmlns="http://java.sun.com/xml/ns/javaee">
  <servlet>
    <servlet-name>TestServlet</servlet-name>
    <servlet-class>com.spring.service.proxy.TestServlet</servlet-class>
  </servlet>
  
  <servlet>
    <servlet-name>ProxyServlet</servlet-name>
    <servlet-class>com.jsos.httpproxy.HttpProxyServlet</servlet-class>
    <init-param>
      <param-name>host</param-name>
      <param-value>http://localhost:7001/NotificationService/NotificationPort</param-value>
    </init-param>
  </servlet>
  
  <servlet-mapping>
    <servlet-name>TestServlet</servlet-name>
    <url-pattern>/testservlet</url-pattern>
  </servlet-mapping>
  
  <servlet-mapping>
    <servlet-name>ProxyServlet</servlet-name>
    <url-pattern>/proxyservlet</url-pattern>
  </servlet-mapping>
  
</web-app>


Configure thread throttling
We can do that using following two different ways:

a) Local Work Manager in weblogic.xml
Configure Work Manager and Servlet dispatch policy in weblogic.xml

weblogic.xml
<?xml version = '1.0' encoding = 'windows-1252'?>
<weblogic-web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bea.com/ns/weblogic/weblogic-web-app http://www.bea.com/ns/weblogic/weblogic-web-app/1.0/weblogic-web-app.xsd" xmlns="http://www.bea.com/ns/weblogic/weblogic-web-app">
  
  <wl-dispatch-policy>LocalProxyWorkManager</wl-dispatch-policy>                  
  
  <servlet-descriptor>
    <servlet-name>ProxyServlet</servlet-name>
    <dispatch-policy>LocalProxyWorkManager</dispatch-policy>
  </servlet-descriptor>
  
  <work-manager>
    <name>LocalProxyWorkManager</name>
    <max-threads-constraint>
      <name>LocalProxyWorkManagerMaximumThreadConstraint</name>
      <count>1</count>
    </max-threads-constraint>
    <capacity>
      <name>LocalProxyWorkManagerCapacityConstraint</name>
      <count>1000</count>
    </capacity>
    <ignore-stuck-threads>true</ignore-stuck-threads>
  </work-manager>
  
</weblogic-web-app>


b) Global Work Manager in weblogic.xml and Weblogic Console 
Configure Work Manager in Weblogic Console and Servlet dispatch policy in weblogic.xml

weblogic.xml
<?xml version = '1.0' encoding = 'windows-1252'?>
<weblogic-web-app xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="http://www.bea.com/ns/weblogic/weblogic-web-app http://www.bea.com/ns/weblogic/weblogic-web-app/1.0/weblogic-web-app.xsd" xmlns="http://www.bea.com/ns/weblogic/weblogic-web-app">
  
  <wl-dispatch-policy>GlobalWorkManager</wl-dispatch-policy>                  
  
  <servlet-descriptor>
    <servlet-name>ProxyServlet</servlet-name>
    <dispatch-policy>GlobalWorkManager</dispatch-policy>
  </servlet-descriptor>
  
</weblogic-web-app>


Weblogic Console:




Source code can be downloaded from here.

Saturday, January 19, 2013

AIA 11g PIP security policy with SOAP UI

With AIA PIP installation, you see basically three policy installed out of the box and there are lot of global policy set configured using these policies and attached to Provider, Requester or Adapter services.

Server side policies
1. oracle/aia_wss_saml_or_username_or_http_token_service_policy_OPT_ON 

If service is configured with this policy, then client needs to provide one of three security measures:
  • SAML
  • WSSE Username Token
  • HTTP basic authentication

2. oracle/aia_wss_saml_or_username_token_service_policy_OPT_ON

If service is configured with this policy, then client needs to provide one of the two security measures:
  • SAML
  • WSSE Username Token



Client Side Policies
oracle/aia_wss10_saml_token_client_policy_OPT_ON

This is client side policy and it can be configured for any web service or composite which is protected via AIA server side policies.



Testing Service Side Policies using SOAP UI (or any other WS testing client)

1. oracle/aia_wss_saml_or_username_or_http_token_service_policy_OPT_ON 

  • WSSE Username Token
  • <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
                   xmlns:sam="http://xmlns.oracle.com/SAMLProject/SAMLProcess2/SAMLBPELProcess2">
       <soapenv:Header>
          <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
             <wsse:UsernameToken>
                <wsse:Username>weblogic</wsse:Username>
                <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">*******</wsse:Password>
             </wsse:UsernameToken>
          </wsse:Security>
       </soapenv:Header>
       <soapenv:Body>
          <sam:process>
             <sam:input>asdf</sam:input>
          </sam:process>
       </soapenv:Body>
    </soapenv:Envelope>
    

  • HTTP basic authentication




2. oracle/aia_wss_saml_or_username_token_service_policy_OPT_ON

  • WSSE Username Token
  • <soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" 
                   xmlns:sam="http://xmlns.oracle.com/SAMLProject/SAMLProcess2/SAMLBPELProcess2">
       <soapenv:Header>
          <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd">
             <wsse:UsernameToken>
                <wsse:Username>weblogic</wsse:Username>
                <wsse:Password Type="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-username-token-profile-1.0#PasswordText">*******</wsse:Password>
             </wsse:UsernameToken>
          </wsse:Security>
       </soapenv:Header>
       <soapenv:Body>
          <sam:process>
             <sam:input>asdf</sam:input>
          </sam:process>
       </soapenv:Body>
    </soapenv:Envelope>
    

Thursday, January 3, 2013

Weblogic Custom Authentication Provider #2

I wrote in previous blog entry about how to configure custom authentication provider with Weblogic server. However, there are quite a few concerns associated with this approach, so I had to write generic custom authentication provider, and then I can plugin any module I like.

Some of the concerns with out of the box approches
  • Need to implement both WLS and JPS identity 
  •  WLS security is used for all basic Weblogic Modules (e.g. Console, EM, etc.)
  • JPS security provider is used for SOA modules (e.g. Worklist application)
  • WLS custom security provider is relatively easy to write - details
  • JPS custom security provider is really lot of work as it requires you to implement multiple interfaces (similar to 10g custom identity service) and multiple methods details
  • Need to register different security provider at different places


Implemented Solution

I believe at the end if would be much easier to if all the complex details can be hidden regarding WebLogic and SOA security provider and if client has to just implement a simple interface and provide that in the class path that would be ideal way to go. So here it goes:

Install Java Custom Security Provider
  1. download CustomSecurityProvider.jar
  2. Copy this jar file to $wls_server_home/server/lib/mbeantypes and $domain_home/lib directories
  3. Modify file : $domain_home/config/fmwconfig/jps-config.xml
    • Add Following  
    • <serviceProvider type="IDENTITY_STORE" name="custom.provider" class="oracle.security.jps.internal.idstore.generic.GenericIdentityStoreProvider">
          <description>Custom IdStore Provider</description>
      </serviceProvider>
      
      <serviceInstance name="idstore.custom" provider="custom.provider"  location="./">
          <description>Custom Identity Store Service Instance</description>
          <property name="idstore.type" value="CUSTOM"/>
          <property name="ADF_IM_FACTORY_CLASS" value="com.spring.security.jps.identity.CustomIdentityStoreFactory"/>
          <property name="CustomSecurityProviderPlugIn" value="com.spring.security.plugin.CustomSecurityProviderPlugin"/>
      </serviceInstance>
      
    • Replace Following
    • <jpsContext name="default">
          <serviceInstanceRef ref="credstore"/>
          <serviceInstanceRef ref="keystore"/> 
          <serviceInstanceRef ref="policystore.xml"/>
          <serviceInstanceRef ref="audit"/>
          <!--
              <serviceInstanceRef ref="idstore.ldap"/>
              <serviceInstanceRef ref="trust"/>
              <serviceInstanceRef ref="pdp.service"/>
              <serviceInstanceRef ref="attribute"/>
          -->
          <serviceInstanceRef ref="idstore.custom"/>
      </jpsContext>
      
  4. Restart Admin and Managed servers


Configure Java Custom Security Provider
  • Implement the custom java security provider interface: com.spring.security.plugin.ICustomSecurityProviderPlugIn 
    • Note that for given custom repository we only need to implement following methods
    • package com.spring.security.plugin;
      
      import java.util.List;
      import java.util.Map;
      import java.util.Properties;
      
      public interface ICustomSecurityProviderPlugIn {
          
           /* WLS */ 
          void initialize(Properties properties);
          boolean login(String userName, java.lang.String password);
          List<String> getUserRoles(java.lang.String userName);
      
           /* JPS */ 
          List<Map> searchUsers(String userNamePattern);
          List<Map> searchRoles(String roleNamePattern);
          Map getUserDetail(String userName);
          Map getRoleDetail(String roleName);
      }
      
      
    • If you opt to implement WLS, you can ignore to implement JPS related methods
    • A sample implementation is provided with jar file (com.spring.security.plugin.CustomSecurityProviderPlugIn)


  • Make your implemented java or jar class available to weblogic classpath ($domain_home/lib)
  • Custom Security Provider should be available in drop down as below


  • Modify file : $domain_home/config/fmwconfig/jps-config.xml with your implementation

  • <serviceInstance name="idstore.custom" provider="custom.provider"  location="./">
        <description>Custom Identity Store Service Instance</description>
        <property name="idstore.type" value="CUSTOM"/>
        <property name="ADF_IM_FACTORY_CLASS" value="com.spring.security.jps.identity.CustomIdentityStoreFactory"/>
        <property name="CustomSecurityProviderPlugIn" value="com.spring.security.plugin.CustomSecurityProviderPlugin"/>
    </serviceInstance>
    


  • Restart the server
  • Wednesday, January 2, 2013

    Weblogic Custom Authentication #1

    We can configure multiple WLS authentication provider (e.g. ActiveDirectory, Sun LDAP) as shown below.



    If authentication and authorization information is stored custom repository not supported by above list, we can use following option.



    CustomDBMSAuthenticator : Once it is configured as below, you can plugin in any Java class as long as it implements weblogic.security.providers.authentication.CustomDBMSAuthenticatorPlugin interface.



    We can see the method "lookupPassword" which is called during authentication.


    package weblogic.security.providers.authentication;
    public interface CustomDBMSAuthenticatorPlugin {
        void initialize(weblogic.management.security.ProviderMBean providerMBean) { }
        void shutdown() { }
        java.lang.String lookupPassword(java.sql.Connection connection, java.lang.String userName) { }
        boolean userExists(java.sql.Connection connection, java.lang.String userName) { }
        java.lang.String[] lookupUserGroups(java.sql.Connection connection, java.lang.String userName) { }
    }
    
    

    We can completely ignore connection information and write custom java code to reach out to any custom repository and return the password. In that way it can be used for any custom repository instead of just custom database repository.

    The major concern with this interface is that it requires you to return the password in lookupPassword method. Majority of the time enterprise level identity repository is not going to give you the password. Enterprise custom repository usually have their own authenticate method but above interface doesn't provide the password.

    Another concern is that it only support WLS authentication and authorization. It doesn't provide JPS authentication and authorization.

    • WLS authentication is used for all basic WLS modules (e.g. Console, EM, etc.)
    • JPS authentication is used for SOA specific component, especially Worklist Application. 
    If we implement custom authentication provider using above approach, it only covers WLS authentication and authorization, it would not be called during SOA module login (e.g. Worklist App).