Correlation in Oracle BPEL

Correlation is one of the important & tricky technique available within Oracle BPEL PM.

Synchronous & Asynchronous web services:

Whenever a synchronous web service is invoked from Oracle BPEL via a partnerlink, only one port is established for communication with the exposed web service which is used by both request & response messages. However, when an asynchronous web service is invoked, two ports are opened for communcation: one for request & one for response messages.

How does Oracle BPEL identify asynchronous responses?

As response from an asynchronous web service is not guaranteed to be received within a specified time frame, and many instances of the same service might be invoked before even a response can be obtained, how does Oracle BPEL identify and relate the responses to the appropriate requests and proceed for completion of further activities that may be scheduled? The answer is "WS-Addressing".

What is WS-Addressing?

WS-Addressing is a transport-neutral mechanism by which web services communicate addressing information. SOAP envelopes & headers used within web services for transporting data throught transport layers like HTTP does not possess intelligence to specify unique addressing information. Hence, WS-Addressing evolved which contained endpoint references (EPR) and message information headers for identification. This information is processed independently of the transport mechanism or application.

By default, Oracle BPEL PM implements WS-Addressing for all asynchronous web service calls, hence we don't have to explicitly implement identification relationship between the incoming & outgoing messages.

Correlation:

In short, Correlation is a BPEL technique which provides correlation of asynchronous messages based on the content of the message.

Why & Where Correlation?

The following are few scenarios in which we would be required to implement Correlation to identify & relate message communication; 

1. When the external web service called doesn't have WS-Addressing capability
2. When the message travels through several services and response is solicited by the initial service from the last service directly

For instance, request flow process1 -> process2 -> process3 and response is received from process3 ->process1

Implementing Correlation:

a) Creating Correlation Sets:

In Structure Window of the JDeveloper IDE, right click on the Correlation Sets and choose 'Create Correlation Set' Provide a name for your correlation set being created.
In the properties section, select 'Add' to display the property chooser window Choose 'Create' to create a property on which the correlation has to be initiated. Provide a name and type for the property.

b) Associating the Correlation set on receive/invoke, pick activities

Go the the correlations tab on the activity (invoke/receive/pick) on which you need to set & validate the correlation Add the created correlation set to the activity

Initiate Attribute: (Value Set - yes, no) 
When set to yes, correlation set is initiated with the values of the properties available in the message being transferred When set to no, correlation set validates the value of the property available in the message

Pattern Attribute: (Value Set - in, out, in-out) 
When the value is 'in', it means that the correlation property is set/validated on the incoming message When the value is 'out', it means that the correlation property is set/validated on the message going out of BPEL In case of 'in-out', the property will be set/validated on both incoming & outgoing messages

c) Creating Propery Alias:

In the Structure Window of the JDeveloper, right click on the 'Property Aliases' and select 'Create Propery Alias' Select the message type that you want to set to the correlation propery (already created)

Now, correlation design is complete. However, correlation will not be established unless we reference the correlations on the WSDL file of the BPEL process. To do this, import the correlation WSDL file (created under the project) in the BPEL process main WSDL.

Friday, May 22, 2015

Understand the Dehydration Process in Oracle SOA BPEL

Lets discuss how the dehydration process works in BPEL to provided reliability (non message lost) amongst others benefits. The dehydration point affect the transaction boundaries of your SOA composite. 

What is a BPEL dehydration process ?

BPEL dehydration process is the action from the BPEL engine to compress and store the BPEL instances into the database. The incoming messages are stored in the database for processing later by worker threads.

• Storing the current status of the BPEL process(i.e. long running process, asynchronous process) into the database tables is known as dehydration
• SOA_INFRA schema is the dehydration store which contains tables to hold the meta data of the process.

Why do we need dehydration point in BPEL process ?

• Long running process or processes waiting for response consumes memory and CPU
• While waiting for the response the bpel engine can store the process, thus freeing up server resources.
• To increase both Reliability and Scalability of the BPEL process.
• You can also use it to support clustering and failover.
• Over the life cycle of the BPEL instance, the instance with the current state of execution may be saved in database.

Note: BPEL Activity is dehydrated immediately after execution and recorded in the dehydration store. It provides better failover protection, but may impact performance if the BPEL process accesses the dehydration store frequently.

Synchronous Process 

• Process gets dehydrated only at the end of the process.
• Using Dehydrate activity we can explicitly dehydrate process state if required.

Asynchronous Process

• Automatically dehydrated the process based on the activities used.

When dehydration occurs ?

The dehydration points occurs:

1. When the activities are used : (not just after, depends on BPEL Engine)
   • a mid-receive activities(without create instance). Please not that the first receive activity will not automatically dehydrate the BPEL process. This depends on the pattern used and on the specific BPEL properties optimization.
   • Pick (onMessage & onAlarm) activity
   • just before wait activity (depends on the duration)
   • Dehydrate(11g) / Checkpoint(10g)
   • Commit(Java embedded)
   • a (implicit) transaction is committed
   That is where an existing BPEL instance must wait for an event, which can be either a timer expiration or message arrival. When the event occurs (the alarm expires or the message arrives), the instance is loaded from the dehydration store and execution is resumed. This type of dehydration occurs only in durable processes, which have mid-process breakpoint activities. A transient process does not have any mid process breakpoint activities.
2. When calling a non-idempotent idempotent=false service or Partner Link
   When Oracle BPEL Server recovers after a crash, it retries the activities in the process instance. However, it should only retry the idempotent activities. Therefore, when Oracle BPEL Server encounters a non-idempotent activity, it dehydrates it. This enables Oracle BPEL Server to memorize that this activity was performed once and is not performed again when Oracle BPEL Server recovers from a crash.
   Idempotent activities are those activities where the result is the same irrespective of no. of times you execute the process.
   Repeated invocations have the same effect as one invocation. 
   E.g. Read only services, Receive activity
3. When RequiresNew BPEL transaction is completed or When the BPEL instance finishes
   At the end of the BPEL process, Oracle BPEL Server saves the process instance to the dehydration store, unless you explicitly configure it not to do so. This happens to both durable and transient processes.
   Note: A BPEL invoke activity is by default (true) an idempotent activity, meaning that the BPEL process does not dehydrate instances immediately after invoke activities. Therefore, if idempotent is set to true and Oracle BPEL Server fails right after an invoke activity executes, Oracle BPEL Server performs the invoke again after restarting. This is because no record exists that the invoke activity has executed. This property is applicable to both durable and transient processes. If idempotent is set to false, the invoke activity is dehydrated immediately after execution and recorded in the dehydration store. If Oracle BPEL Server then fails and is restarted, the invoke activity is not repeated, because Oracle BPEL Process Manager sees that the invoke already executed.

Types of BPEL Process

When and how dehydration occurs differ based on process types. Based on the dehydration we can categorize process in to 2 categories

1. Transient Process: Oracle BPEL server dehydrates the process instance only at the end of the process. If server crashes in middle of the running process instance, the instances will not be visible in EM
2. Durable Process: Oracle BPEL Server dehydrates the process instance in-flight at all midprocess breakpoint and non-idempotent activities, plus the end of the process. When the server crashes, this process instance appears in Oracle BPEL Control up to the last dehydration point (breakpoint activity) once the server restarts. If the server crashes before the process instance reaches the first midprocess breakpoint activity, the instance is not visible in Oracle BPEL Control after the server restarts.

Important Notes:

1. Remember that theses dehydration statement must be avoided in synchronous BPEL process.
2. Idempotence is a mathematical term that basically means that calling a function multiple times doesn’t change the result for example f(f(x))=f(x).
3. What does it mean for messaging service system ? If a service receives the same message again, it should be able to handle it without changing the state of the system. For example; in a bank scenario you wouldn’t want that withdraw message to be processed more than once !
4. An idempotent activity in BPEL (for example, an assign activity or an invoke activity) is an activity that can be retried
5. To ensure data consistency, the dehydration database is done using the same transaction context in which the BPEL process is executed. However, BPEL engine use a separate transaction context for the audit logging information, which aids (a lot) for debugging failed instance.

Skipping dehydration

• Skipping dehydration will increase performance with a cost of auditing.
• By Default, all BPEL processes are dehydrated regardless of whether they are Synchronous or Asynchronous process.
• For Synchronous processes that do not need to be durable, you can turn off the dehydration mechanism by setting the following properties at process or engine level:
  • Set inMemoryOptimization to true.
  • Set completionPersistPolicy property to faulted or Off.
• Asynchronous process invocation messages are always saved to the dehydration store.
• Durable process are always dehydrated regardless of the settings of the persistence properties.

Dehydration Tables

1. Cube_Instance: Stores the information about the composite instance that gets created.
2. Cube_scope: Stores information about the scopes and variables declared and used.
3. DLV_Message: All the incoming payload details are stored.
4. Invoke_Message: All the outgoing message payload details are stored.
5. Audit_Trail: Used to store information of the xml that is rendered in EM console.

Performance Recommendations:

Engine dispatch messages are generated whenever an activity must be processed asynchronously by the BPEL engine. If the majority of processed deployed on the BPEL server are durable with a large number of dehydration points (mid-process receive, onMessage, onAlarm, wait), greater performance may be achieved by increasing the number of engine threads.

Thursday, May 21, 2015

Transient vs. Durable Processes in Oracle BPEL

As a general practice, it is better to design your BPEL processes as transient instead of durable if performance is a concern. Note that this may not always be possible due to the nature of your process, but keep the following points in mind. 

The dehydration store is used to maintain long-running asynchronous BPEL instances state information as they wait for asynchronous callbacks. This ensures the reliability of these processes in the event of server or network loss.

Oracle BPEL Process Manager supports two types of processes; Transient and Durable.

Transient Processes

Transient processes do not incur dehydration during their process execution. If an executing process experiences an unhandled fault or the server crashes, instances of a transient process do not leave a trace in the system. Thus, these instances cannot be saved in-flight regardless if they complete normally or abnormally. Transient processes are typically short-lived, request-response style processes. Synchronous processes are examples of transient processes.

Durable Processes

Durable processes incur one or more dehydration points in the database during execution. Dehydration is triggered by one of the following activities:

• Receive activity
• OnMessage branch in a pick activity
• OnAlarm branch in a pick activity
• Wait activity
• Reply activity
• checkPoint() within a bpelx:exec activity

Durable processes are typically long-living and initiated through a one-way invocation. Because of out-of-memory and system downtime issues, durable processes cannot be memory-optimized.

What should you do?

• If the design of the process allows it, design your BPEL processes as short-lived, synchronous transactions.
• If the design of the process allows it, avoid the activities listed above.

Any time your process is dehydrated to the dehydration store, this naturally impacts the performance of the process, and becomes a concern particularly in high volume environments.