Business Process Analytics using BPAF

Over last 6 month I've been working on several components that will be assembled to a bigger business activity monitoring solution. I started off with a CEP based activity monitor (SAM) that allows for advanced correlation across heterogeneous event sources. After that I've spent some time integrating the BIRT reporting runtime with the BPM console, which allows you to run complex reports on process-related audit data. Only to realize that I've started on the wrong foot.

Process analytics bottom to top

I made a mistake to begin with the most complex components first, without realizing that the key to a successful BAM solution is underlying data model. Without a solid baseline the creation of report templates becomes a daunting task because the missing unification of audit data basically requires you to customize the report algorithms for any instrumented subsystem. Creating rules for the CEP component that fire when certain SLA's are breached doesn't get more simple either, when working on a heterogeneous event model.

I finally spend some time prototyping around an idea that has been in the wild for quiet some time: Baseline process-related audit data on BPAF: A WFMC standard that describes a data structure that enables the aggregation and correlation of audit data across multiple platforms.

In the following sections I would like to walk through a simple protoype, that already shows the benefits of this approach. It is really impressive how little code it requires to deliver a solution that already matches an important number of the use cases.

Data model, storage and retrieval

To begin with an implementation for storing and querying BPAF event data was needed. Fortunately JAXB and Hibernate made this a trivial task:

@XmlAccessorType(XmlAccessType.FIELD)
@XmlType(name = "", propOrder = {
    "eventDetails",
    "dataElement"
})
@XmlRootElement(name = "Event")
@Entity
@Table(name="BPAF_EVENT")
public class Event {

  @XmlElement(name = "EventDetails", required = true)
  protected Event.EventDetails eventDetails;

  @XmlElement(name = "DataElement")
  protected List dataElement;

  /**
   * A globally unique identifier for the individual event
   */
  @XmlAttribute(name = "EventID", required = true)  
  protected long eventID;

[...]

}

The current implementation allows two approaches for querying the BPAF data: Either leverage the default model and it's persistence implementation, or adopt a proprietary audit data structure to the BPAF model when needed. For later use case the API fo querying is split from the default data model:

/**
 * @author: Heiko Braun 
 * @date: Mar 10, 2010
 */
public interface BPAFDataSource
{
  /**
   * Get a list of distinct process definition ID's that are known to the system.
   * @return a collection of process definition ID's
   */
  List getProcessDefinitions();

  /**
   * Get a list of distinct process instance ID's that are known to the system.
   * @return a collection of process instance ID's
   */

  List getProcessInstances(String processDefinition);

  /**
   * Get a list of distinct activity definition ID's that are known to the system.
   * @return a collection of activity definition ID's
   */

  List getActivityDefinitions(String processInstance);

[...]
}

Now that we have components in place for storing and querying raw BPAF data, let's move on to a more interesting topic: How to make sense of it.

Turning data into information

The whole purpose of this exercise is to provide a way to gain insights about the execution characteristics of the processes that drive your business. While BPAF itself is a compact but still very expressive format (BPAF State Model) to store process-related audit data, it requires additional steps to turn it into consumable information that participants and decision makers can make sense of.

For the sake of this example we've added some algorithms that allow grouping and sorting of the underlying data, which will be used by the presentation layer.

Let's take a look at the prototype UI (yet another BPM Console plugin) and see how much information can already be derived without going through hoops and loops.

Presenting audit data to decision makers

The prototype UI demos a simple use case: "How did the processes execute in the past?" Although a simple question, it already implies a lot of fine grained functionality a user might expect:

• Selecting a relevant process
  
• Restricting the query to a certain time period
  
• Scrolling through business related events that occurred over a period of time
  
• Zooming in and out on a time scale (hourly, daily, weekly, etc)
  

Concepts represented by the prototype UI

For the sake of simplicity the current implementation (at the time of this writing) is limited in functionality. Nevertheless I would like to walk you through the current features and explain some of the key concepts.

To begin with a user selects the Process Defintion and the relevant time period:




The time period offers a set of commonly used intervals, like "Last Week", "Last Quarter" or "Last 24 Hours".

The BPAF data is loaded and grouped according to the time period and presented in an interactive chart:




The charts allow you to zoom in on the time scale (i.e. "1 day, 5 day, max") and scroll through the event data on each level.
(Shout-Outs to the chronoscope project)

You can comment on the information and create markers on business relevant points that either require clarification or otherwise mark notable changes in the execution of a process:




Conclusion and next steps

As you see, this simple prototype already covers important features that you would expect from such an implementation.
And we didn't yet touch sophisticated topics like comparison of process executions or creation of key performance indicators that trigger notifications (do the markers ring a bell?).

In the next few month you can expect this approach to be married to SAM, our CEP based BAM component that allows correlation and execution on process activities close to real time.

Stay tuned, now that we have a common baseline for representing and querying process-related audit data we'll look into more advanced topics.

BPM Console 2.0.1 released

I am glad to announce the BPM Console 2.0.1. Although it doesn't include many new functional enhancements, architecturally it's still a major step forward. Let's take a look at the most prominent changes first:

Workspace API consolidation

This is probably one of the most important steps. Something that we had on the roadmap for quiet some time. Before 2.0.x the BPM build on it's own workspace API, which is fully merged with errai-workspaces. Not only does this consolidate the web tooling efforts within JBoss, but it also allows building of true federated toolsets using the errai bus and workspaces modules as a foundation.

This allows us to easily integrate tools from other JBoss projects into a custom workspace build. I.e. something that we are currently thinking about is integrating the Apache Juddi web console as part of the Riftsaw console offering.

Improved performance and development tools

We've been running the console on GWT 1.5.3 for quiet some time but now it leverages GWT 2.0. Probably the most important changes for the BPM console are support for code splitting and declarative UI's.


JBoss system theme

We started working on a custom theme, that should apply to JBoss web tooling in general. It's still in the early stages, but already included in the recent release. We are aiming for a design that's clean, concise and doesn't get in your way:



Role based authorization

Since we migrated to errai-workspaces the BPM console now has support for role based authorization. This means you can easily restrict access to certain tools (i.e. admin vs. user) or use this mechanism to create customized profiles according to a users role (i.e. sales vs. marketing)

@LoadTool(name = "Deployments", group = "Runtime")
@RequireRoles({"administrator"})
public class DeploymentModule implements WidgetProvider
{
  [...]
}

History and perma links

One feature that has been requested a lot, was support for linking to tools from an external application. I.e. when a email notification is send for an outstanding task. This is now built into errai-workspaces. The same mechanism that allows you to use the default browser history navigation, can now be used to reference tools from external applications.

Report parameter preferences

The console supports parametrized report templates since version 1.3. What has been added is the ability to store report parameters for each template as part of the console preferences. Imagine a report that requires you to enter a certain time period for rendering. This data is now store within as part of console preferences, so can easily switch between report templates without having to provide that data over and over again.





Roadmap

In the next few weeks we'll focus on the toolset for the Riftsaw BPEL Engine. Areas that need improvement are activity monitoring tools and tools for maintaining WS endpoint references, like the UDDI browser.

Contributions

As I said earlier, 2.0.x was an important step regarding maintainability and API consolidation. We now have everything in place for you to get your hands on the errai-workspaces module and start developing plugins for the BPM console in particular or the JBoss SOA platform in general. In case you are interested, don't hesitate to get in touch with us:

IRC: errai@irc.freenode.net
Mail: http://jboss.org/errai/MailingLists.html

Drools 5.1 M1 ships the BPM console

Drools 5.1 M1 includes the recent BPM console. Take a look, give it a ride: http://blog.athico.com/2009/10/drools-51-m1-release-notes.html

Activity Monitoring, Part 1: A twitter example

SAM has been around for quiet some time, but still didn’t get much traction. On the one hand I was involved with other projects and on the other SAM did lack good examples that reveal the underlying concepts and put them into context.
In the past few days I’ve put together an activity monitoring demo that will be used to explain the core concepts and give you an idea how SAM can be used to monitor activities of a particular domain.

Event streams

First of all, a good example needs constant event feeds. I was thinking about monitoring TCP packages on my box, but then came up with much more comprehensive example: Twitter feeds. Twitter has such a high volume of messages that it’s easy to derive event streams from it. In order for correlation to make sense we need events that share some kind of context. Using the twitter search API we can query messages that contain particular keywords, which is just enough contextual information for this example to make sense. Because I was tired of the king of pop, I decided on another high volume discussion, in this case iran.

You may ask: “Why event streams to begin with?” Because SAM is intended to monitor activities in a system close to real time. System activity is expressed a number of events that happen in no particular order but constantly over time. Event’s don’t even need to share the same properties, even though in our example they do.
And last but not least: It’s no fun to monitor a system without activity. You’d just get blank graphs and empty tables, which is hard to explain to the audience.



Correlation

Events come in all kinds of flavors. Most likely events share common properties if they derive from the same domain, i.e the twitter message cloud. When instrumenting a target domain you create event representations and push them to SAM using a specific stream input. A stream input is bound to a transport layer (i.e. JMS) and has a set of processing rules associated with it. These rules are used to monitor and act upon event streams if certain conditions kick in:

rule "all iran tweets"
when
    $tweets: List() from accumulate(
      $t : Tweet(id>0) over window:time(15s) from entry-point "iran tweets",
      collectList( $t )
   )
then      
   SAM.getListener("iran results").update($tweets);

end

In this example a simple rule is associated with the event stream input “iran tweets” that collects messages over a 15sec time window and pushes them to a stream output “iran results”.

While stream inputs collect events, stream outputs are used to forward events,
i.e. for notifying SLA watchdogs. The number of processing steps between input and output is not limited. You may use rules that trigger other rules. I.e. first filter certain events and then correlate them:



But for the sake of simplicity we simply collect events over time and push them to the monitoring UI. We could add more complex processing rules, but that would extend the scope of this blog post and will be explained in Part 2 of this series. But for those of you who cannot wait, I suggest to take a look at Drools Fusion, the CEP engine that drives the complex correlation capabilities within SAM.

Monitoring

The whole purpose of SAM is monitoring and management of a system in place. SAM has a close relation to BPM console, as it will be used to realize the BAM capabilities (Business Activity Monitoring) for JBoss BPM projects. For this reason I’ve chosen to implement the monitoring UI right atop of the BPM console framework. The user interface you see in the next sections is actually an editor component that can be plugged into the BPM console.

But before we dive into the details, we need explain how SAM interfaces with a monitoring console.

The monitoring UI and SAM are decoupled through the use of current value tables (CVT). A CVT holds the correlation results at a particular point in time. Any processing result that is forwarded to an event stream output will update a specific CVT that is associated that output. The monitoring UI doesn’t access the stream output directly, but instead reads the data from a CVT:



So, whats happening in our twitter example? Every 15 sec we collect the events from our stream input and simply forward to a stream output, that in our case is writing to a current value table. The monitoring UI runs in a another thread and uses a different update interval. It assembles a graphical representation and displays it to the user. In this case a line chart that depicts that number of messages at given point in time:



Et voila! We’ve reached our first milestone: Monitoring twitter participation on a certain topic close to real-time.

Analysis

As the name suggests, a current value table only holds most recent data. In our example it means the last sample point on the x-axis of the chart above. Here’s the corresponding value table:



In this case four people have been tweeting about “iran” at that given point. But what if we want to go back in time? I.e. inspecting a previous peak or any other kind of derivation? Fortunately our CVT implementation doesn’t simply purge all records when an update occurs, but instead “swaps” all data to a Handler that will maintain a CVT’s history. It stores all snapshots so they can be retrieved later on.



Second milestone: We are not restricted to the most recent data, but can also go back in time and compare current and past event streams by simply navigating through the chart:



Drill down

That leaves us with one remaining question. The event properties available are pretty limited. Why is that? And how can we get to the details?

In order to answer these questions we need to step back a little. SAM tries to provide event stream analysis close to real time. That means we aim for minimum footprint whenever possible. Event representations travel back and forth through the whole monitoring infrastructure, beginning with the target domain being monitoring, then entering the event processor itself and finally being represented visually in a monitoring UI. It’s important to restrict the payload without sacrificing the analysis possibilities. The solution is simple: SAM works with references to both the target domain and the event entity. Basically everything that is required are an event context (i.e. twitter user name), a timestamp and a reference id (i.e. message id). Other properties that go beyond that depend on the correlation rules you want to apply. The challenge when creating a monitoring solution is finding the right balance between target domain instrumentation (initial event representation) and correlation capabilities.

If we stick to our example, both message ID and username are sufficient to drill down later on. We can always go back to twitter and retrieve the message contents for a particular message ID. It leverages one of the fundamental ideas behind SAM CVT implementation: On-demand content retrieval:



Putting it all together

Let us quickly recapture what has been explained here.
We capture events of a target domain and push them to SAM through event stream inputs. Each input is associated with a set of processing rules that monitor the event stream and kick in on certain conditions. When rules kick in we can forward events to stream outputs or further feed them into the processor.

One of the use cases for stream outputs are monitoring tools that display the information to a user. Users watch for exceptional situations and need to know what’s causing them. This also implies access to events that did happn in the past.

Thorough analysis requires you to drill down to the event details and the source of information. Which is not necessary for all events, but only those of interest hence the need for on-demand content retrieval.

Comments? Thoughts? Please let us know at the Overlord forums.

In the meantime enjoy the screencast of the twitter monitoring demo in action.

BPM Console screencasts

Several new screencast have popped up recently that show the BPM console in Action.

- Joram shows how to customize the reporting.
- Kris did an excellent overview of the console as a whole, including task forms.