Issue Soa Suite 12.1.3 Context of the same RID xxxxxx already present in this family

Lately in our production environment we saw that when some of our services where called in rapid succession, some of them failed. Under normal load we never had issues but under bigger load we saw the following error in our BPEL instances:

Failure in Oracle WSM Agent, category= security, function=agent.function.client, stage=fault due to RuntimeException.
java.lang.IllegalArgumentException: Context of the same RID, 0:1:4 already present in this family, f42da30f-5c89-49d9-a4fd-bd2b7e80644e-00154a3f.

Bug

It also seemed that those faults where creating stuck threads and after a while making our managed servers unstable and unreachable. After some digging around we found that there was a patch for it. See Sporadic error during invocation of a SOA composite, ‘Context of the same RID, xxxxxx already present in this family’ (Doc ID 2150992.1). It was released since june 2016 but it seems it was never put into a bundle patch somewhere. After applying the patch the errors where replaced by warnings:

Sep 13, 2017 11:48:22 AM CEST Warning oracle.dms.context BEA-000000 Duplicate RID detected. Multiple tasks are attempting to execute with the same exec-ctx identifier 9f31c77a-4bee-4f57-a970-9275a6577bcb-00030ae5,0:1:4. Will use the unrooted identifier 9f31c77a-4bee-4f57-a970-9275a6577bcb-00030ae5,1:32171 for this task.

and we had no more stuck threads and all our instances where running fine again.

Installing SOA BP 12.1.3.0.5 makes the flowinstance title disappear. Incoming workaround!

After installing bundle patch 12.1.3.0.5 we noticed that the flow instance title of our instances in SOA where flakey. Meaning sometimes they appeared, but sometimes they didn’t. After some testing and going back and forth with Oracle support, we where not able to steadily reproduce the bug. We did come to a work-around though and that is forcing a dehydrate. Not the best option is my opinion but a possible workaround. So if you have this issue, just add a dehydrate to the BPEL and magically see your flow instance titles come back to live again.

dehydrate

When this bug is going to be resolved is still unknown.

OFM 12C: Slow SB deployment issue

Working with 12C a while, we started noticing that the deployment of our SharedResources SB project started to slow down. This project contains re-usable resources and contracts. The strange thing was that it was only this specific SB project. All the other projects where fine. Both deploying in JDeveloper and deploying using Maven started at around 40 seconds but after a while, it went up to 10 minutes even which is of course not workable.

After some proper investigation and contact with Oracle Support we came to the conclusion that the Maven deployments where causing this. See bug 22051706: Maven OSB deploy causes open activation sessions on OSB in case of failure, which if unresolved cause slow deployment performance. So basically when your maven deployment fails, it causes an open session. If the session contains a lot of items, which our SharedResources project has, then even 1 or 2 open sessions will drastic slow your deployments down.

sessions

The solutions is either to use the servicebus console to navigate to the sessions tab at the bottom, select a session, take control of it and discard it. I think that also the $DOMAIN_HOME/osb folder holds the sessions which can be deleted on the server.

Another way is to remove them using a piece of code:

/* To run this, make sure the setDomainEnv.sh is sourced:
    cd $DOMAIN_HOME/bin
    . setDomainEnv.sh
*/

import weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean;
import weblogic.management.jmx.MBeanServerInvocationHandler;
import javax.management.MBeanServerConnection;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXServiceURL;
import javax.management.remote.JMXConnectorFactory;
import javax.management.ObjectName;
import com.bea.wli.sb.management.configuration.SessionManagementMBean;
import java.util.Set;
import java.util.Hashtable;
import java.util.HashSet;
import java.net.MalformedURLException;
import javax.naming.Context;
import java.io.IOException;

public class DiscardOSBSessions {
    public static final String hostname = "MY_HOSTNAME";
    public static final int port = 7001;
    public static final String username = "MY_USERNAME";
    public static final String password = "MY_PASSWORD";

    static public void main(String[] args)
    {
        JMXConnector conn = null;
        
        try
        {
            // get the jmx connector
            conn = initConnection(hostname, port, username, password);

            // get mbean connection
            MBeanServerConnection mbconn = conn.getMBeanServerConnection();

            // get the Session names:
            //com.bea:Name=ALSBConfiguration.weblogic,Location=AdminServer,Type=com.bea.wli.sb.management.configuration.ALSBConfigurationMBean
            ObjectName mbeanQuery = new ObjectName("*:Name=ALSBConfiguration.*,Type=com.bea.wli.sb.management.configuration.ALSBConfigurationMBean,Location=AdminServer");
            Set<ObjectName> mbeans = mbconn.queryNames(mbeanQuery, null);
            Set<String> sessionNames = new HashSet<String>();
            for (ObjectName mbeanName : mbeans) {
                sessionNames.add(mbeanName.getKeyPropertyList().get("Name").replace("ALSBConfiguration.",""));
            }
            
            System.out.println(sessionNames.size()+" open sessions:");
            if (!sessionNames.isEmpty()){
                for (String sessionName : sessionNames) {
                    System.out.println(" - "+sessionName);
                }

                System.out.println();
                System.out.println("Destroying the sessions:");
                
                // get domain service mbean. This is the topmost mbean
                DomainRuntimeServiceMBean domainService = (DomainRuntimeServiceMBean) MBeanServerInvocationHandler.newProxyInstance(mbconn, new ObjectName(DomainRuntimeServiceMBean.OBJECT_NAME));

                // obtain session management mbean to destroy a session.
                // This mbean instance can be used more than once to create/discard/commit many sessions
                SessionManagementMBean sm = (SessionManagementMBean) domainService.findService(SessionManagementMBean.NAME, SessionManagementMBean.TYPE, null);

                for (String sessionName : sessionNames) {
                    System.out.println(" - "+sessionName);
                    sm.discardSession(sessionName);
                }
            }
            System.out.println();
            System.out.println("Successful completion");
        }
        catch (Exception e) {
            e.printStackTrace();
        } finally {
            if (conn != null)
                try {
                    conn.close();
                } catch (Exception e) {                
                    e.printStackTrace();
                }
        }
    }

    private static JMXConnector initConnection(String hostname, int port, String username, String password) throws IOException,MalformedURLException
    {
        JMXServiceURL serviceURL = new JMXServiceURL("t3", hostname, port, "/jndi/" + DomainRuntimeServiceMBean.MBEANSERVER_JNDI_NAME);
        Hashtable<String, String> h = new Hashtable<String, String>();
        h.put(Context.SECURITY_PRINCIPAL, username);
        h.put(Context.SECURITY_CREDENTIALS, password);
        h.put(JMXConnectorFactory.PROTOCOL_PROVIDER_PACKAGES, "weblogic.management.remote");
        return JMXConnectorFactory.connect(serviceURL, h);
    }
}

As you can see, you can only run this on the appropriate server with the right classes on the classpath. I tried to Mavenize it into a maven project with the proper dependencies but I wasn’t able to find the right jar files for:

  • weblogic.management.mbeanservers.domainruntime.DomainRuntimeServiceMBean
  • weblogic.management.jmx.MBeanServerInvocationHandler
  • com.bea.wli.sb.management.configuration.SessionManagementMBean

If anyone can help me out here….that would be great!

A feature request has been made to add the ‘discard session’ parameter to the Maven properties so hopefully this will come in future releases.

OFM 12C from the trenches: Business flow instance name bug and solution

A new feature in OFM 12C is that you can track Business flow instances. I won’t go into the details as you can read all about it here. As this is a nice feature I was going to use this to set a good Flow instance name as descripted here in chapter 47.5.4 How to Set the Business Flow Instance Name or Composite Instance Name at Design Time. I created a simple service and added an assign in my mediator component as written in the documentation.

<assign>
   <copy
      target="$out.property.tracking.setFlowInstanceTitle"
      expression="oraext:setFlowInstanceTitle("Test")"
      xmlns:med="http://schemas.oracle.com/mediator/xpath"/>
</assign>

As I added it, I noticed that the namespace declared here for the setFlowInstanceTitle wasn’t declared. I thought I would have have been declared somewhere else so I just tried it. As I compiled and deployed my service and ran the test I got the following error though:

ORAMED-01001:[Error in assign operation]Error occurred while assigning to target "$out.property.tracking.setFlowInstanceTitle" using expression "oraext:setFlowInstanceTitle('Test')".Possible Fix:Check if source message has right values or source expression is valid. Cause:Extension function error: Method not found 'setFlowInstanceTitle' 

After some digging around and a chat with Oracle support it seems that

a) You have to add the namespace. It seems they forgot it in the online documentation. So the assign should look like this:

<copy
target="$out.property.tracking.setFlowInstanceTitle"
expression="oraext:setFlowInstanceTitle('Test')" xmlns:oraext="http://www.oracle.com/XSL/Transform/java/oracle.tip.pc.services.functions.ExtFunc"
xmlns:med="http://schemas.oracle.com/mediator/xpath"/> 

b) You have to apply patch 18310693 to the server. This means downloading it, unzipping and running the patch.

C:\Oracle_patch\18310693>c:\Oracle\Middleware\Oracle_Home\OPatch\opatch apply
Oracle Interim Patch Installer version 13.2.0.0.0
Copyright (c) 2014, Oracle Corporation.  All rights reserved.


Oracle Home       : C:\Oracle\Middleware\Oracle_Home
Central Inventory : C:\Program Files\Oracle\Inventory
   from           : n/a
OPatch version    : 13.2.0.0.0
OUI version       : 13.2.0.0.0
Log file location : C:\Oracle\Middleware\Oracle_Home\cfgtoollogs\opatch\18310693_Jan_18_2015_16_25_03\apply2015-01-18_16-24-53PM_1.log


OPatch detects the Middleware Home as "C:\Oracle\Middleware\Oracle_Home"

Jan 18, 2015 4:25:03 PM oracle.sysman.oii.oiii.OiiiInstallAreaControl initAreaControl
INFO: Install area Control created with access level  0
Applying interim patch '18310693' to OH 'C:\Oracle\Middleware\Oracle_Home'
Verifying environment and performing prerequisite checks...
All checks passed.

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = 'C:\Oracle\Middleware\Oracle_Home')


Is the local system ready for patching? [y|n]
y
User Responded with: Y
Backing up files...

Patching component oracle.integration.soainfra, 12.1.3.0.0...

Patching component oracle.integration.soainfra, 12.1.3.0.0...

Patching component oracle.soacommon.plugins, 12.1.3.0.0...

Patching component oracle.soacommon.plugins, 12.1.3.0.0...

Verifying the update...
Patch 18310693 successfully applied
Log file location: C:\Oracle\Middleware\Oracle_Home\cfgtoollogs\opatch\18310693_Jan_18_2015_16_25_03\apply2015-01-18_16-24-53PM_1.log

OPatch succeeded.

After this….start your server again and try your composite again. The result should be:

FlowInstanceTitle