Geeks With Blogs

Charles Young

Following on from the news about Microsoft's CEP engine, Richard Seroter attended yesterday's TechEd session on the new engine and has blogged very detailed notes.   See    I'm delighted to see that Microsoft is disclosing much the same level of information publically that they made available internally a couple of months ago.  Now all we need is CTP 1.

Richard notes that the session was sparsely attended.   CEP has some way to go before it is widely acknowledged or understood in the mainstream, and the potential impact of engines like this will be poorly understood at present.   I intend to put together an article or two in which I'll attempt to explain why I think we should all take note.

I responded to a comment from Tim Bass last night.   Tim is a vocal critic of the CEP marketplace, and one of his common themes is the belief that most/all CEP vendors currently attach the term 'CEP' incorrectly to what are 'mere' event stream processing engines.   I responded by pointing out the CEDR technology, on which I believe Microsoft's engine in based, directly tackles one of the chief characteristics of the event 'cloud' (a core concept in CEP) which is that events may be detected out-of-order in terms of time and causality, and that CEP engines must be able to handle this efficiently.   I was interested to see that Richard recorded an emphasis on this capability of Microsoft’s new engine.

Also, one other point.  I had a Twitter message yesterday from another well-known person on the CEP circuit who said that he hadn't seen any sign of anyone using the new Microsoft engine yet.  For the avoidance of confusion, Microsoft only announced the engine yesterday.   There is a very early private preview of the code doing the rounds internally within Microsoft.   It looks like the first public CTP will be available later this year and that launch is expected next year.   So no one is yet using the engine, apart from Microsoft themselves (apparently, if I understood comments made during the keynote correctly, they are currently handling 500 million events a day through the engine in association with their web site).

Posted on Wednesday, May 13, 2009 6:56 AM | Back to top

Comments on this post: CEP: Microsoft reveals more at TechEd

# re: CEP: Microsoft reveals more at TechEd
Requesting Gravatar...
Hi Charles,

Actually, the notion you mention that "a core concept in CEP" is that events can be processed out-of-order is a red-herring discussion that was created by the stream processing vendors a number of years ago. You are accidentally repeating an old red-herring argument created by "the streaming crowd", I am sorry to inform. It is important that we be technically accurate:

A core concept in CEP is that CEP has the capability to process POSETS, and this is more complex that processing a simple linear ordering or sequencing problem. TCP, the transmission control protocol used in the Internet, processes (and reorders) out-of-sequence packets upon arrival, but we don't call TCP/IP and other network-layer protocols, "CEP".

The notion of the "event cloud" is defined by (event) POSETs, which are events that are not "completely ordered" based on relationship, taxonomy, causality, etc. In other words, the notion of processing out-of-order linear sequences of events is not a core CEP concept, it is a core stream processing concept. This is the core concept, the big elephant in the room so to speak, which the stream processing community fails to acknowledge. In fact, many go so far as to discount POSET processing altogether. In most of the historical discussions, is has been stream processing folks who are processing linearly ordered market data that POSET processing does not define the "event cloud". This is because processing linearly ordered market data (ordered because the data has time stamps and we know that quote A came before quote B, etc.) is fundamentally "simple event processing" unless you are processing to discover or reveal POSET causality, for example.

In your recent posts, you are repeating historical misinformation published a few years ago by stream processing vendors who were arguing, incorrectly, that they were processing "event clouds" (POSETs) because they could process "out-of-order" events in a linear sequence. This is not accurate.

As I have pointed out a number of times in CEP/EP discussion groups and on my blog, an out-of-order linear sequence of events typically remains a totally ordered set (TOSET) because we know the total order, a priori (and can easily reorder, just like TCP). The property of total order of known sequences do not transform to a POSET just because the events arrive out-of-sequence when they are received by an event processing node.

Yours, Tim

Left by Tim Bass on May 19, 2009 1:02 PM

# re: CEP: Microsoft reveals more at TechEd
Requesting Gravatar...
Thanks Tim.

The point you make is valid. This particular post was just a brief report in light of the Tuesday session at TechEd and your previous comments. Hopefully I did a better job of explaining the true complexities of processing posets in the much longer article I posted yesterday. The speculative processing features of CEDR, as I understand them, are there primarily to handle processing of posets containing events that exhibit causal or other types of relationships rather than fixing issues of linear sequencing within individual streams.

Of course, handling problems within individual streams can also be an aspect of the totality of what a CEP engine is asked to handle, but I absolutely agree that it isn't specific to the processing of complex events. My apologies if this post in any way gives the wrong impression.
Left by Charles Young on May 19, 2009 11:03 PM

Your comment:
 (will show your gravatar)

Copyright © Charles Young | Powered by: