Colin Clark on Event Processing

My new blog, and Coral8’s new web site, is up.  You can check it out here.

Marc Adler wrote about my new job - you can find the details here.

After settling in over at Coral8, I’ll continue my blog there.  

I’ll announce the winner of the MacBook soon.

OPPORTUNITIES

I recently realized that there was no way I was going to get OMICRON, or the surveillance system to market any time soon working on my own.  If I wanted to take advantage of all the opportunities that recent market events have opened up, I needed to be a part of something larger.

AFTER TODAY

I won’t be posting much on this site anymore, because next week I will be starting a new job and taking some of the Kaskad goodies with me.  Details forthcoming.

STILL GIVING AWAY THE MACBOOK!

The deadline for CEP case studies is today, and I’ll be making a decision about the winner soon.  Stay tuned for that announcement.

HAPPY HALLOWEEN EVERYONE!

Halloween is one of my favorite events, and this Halloween is my son’s first - so we’re all dressing up for the Costume Parade at our day care provider.

Halloween is also the last day to enter your CEP project for the MacBook prize (details here).   I’ve received a couple of interesting submissions thus far and I will be choosing the winner soon.

EVENTS UPDATE

I will not be attending the FPL (NYC) and SIFMA (London) event.  But I will still be at STAC - one of my favorite shows.  And I’ll be attending the first Coral8 financial services users forum this coming December 2nd in NYC.

We’ve been performing CPR on an elephant - sometimes that’s what this feels like digging through code as the team re-familiarizes itself with the project.

Here’s a screen shot of the analyst application up and running:

We’ve set up a number of machines running Red Hat and the process of getting the feed handlers up and running is progressing.

I love digging through old code.  It’s amazing to me that you can distill the efforts of many. many person-years of work onto a small, flat, circular device that uses light to communicate information to a computer.  Of course I’m talking about DVD’s.  And what’s on this DVD is all of the Kaskad source code for a Reg NMS compliant, Complex Event Processing based Surveillance System.

GETTING THE SYSTEM UP AND RUNNING

There are a few components required for an end to end test of the surveillance system:

• Market Data Line Handler (ours)
• Bus (Coral8 API)
• CEP Engine (Coral8)
• Database (PostGreSQL - only for testing)
• GUI (ours - .NET C#)

FAST LINE HANDLERS

When we built the Surveillance System at Kaskad, one of the first things we did was create a team to focus only on line handlers - we wanted very high performance, low overhead line handlers that could run on a pizza bus, connect to market data, decode it and push it out on a bus.  We built handlers for listed and over the counter feeds (SIP & SIAC) and other feeds via MMTP (AEMS), RV (TIBCO) and LBM (29West).

ABSTRACTION - BUT NOT TOO MUCH

When we built the line handlers (ON RAMPS in Kaskad-speak), we abstracted the transport so that we could use either a direct socket connection, RV, or LBM.  The line handler wrote to our remote stream API, which in turn utilized whichever transport the customer wanted.  For this particular effort, we’re using Coral8’s stream API.

FIRE IT UP

We’re getting the test harness up and running now and should be pumping market data into the engine very soon.  More on the specific configurations later.

MARKET & SPECIALIST SURVEILLANCE

The genesis of Kaskad was to create something that could be used by multiple groups within the Boston Stock Exchange.  One of the natural choices was to create a Surveillance System.  And what better to run a Surveillance System on than an ESP/CEP engine.

PAPER FREE - VERY GREEN

If you’ve ever stepped into the back office at an exchange, you would see towers of paper on the surveillance analysts’ desk - programs that look for potentially illegal activity or unusual behavior typically run over night in batch mode and then spit out reams of paper.  The sophisticated surveillance systems *might* interface with some type of case management tool, then again, they might not.

Kaskad’s Surveillance System eliminated all the paper, instead providing an electronic Analyst Workstation that was used to research and resolve exceptions found in the Event Engine.

WHAT DID THE REG-NMS COMPLIANT SURVEILLANCE SYSTEM DO?

The surveillance system provided:

• Orders and Trades Audit Trail
• Clean and Dirty NBBO Audit Trail
• Quotes and Prints Audit Trail
• Market Action Notification and Audit Trail
• Market Maker Routing Performance
• Wash Sales Exceptions
• Front-Running on Block Trade Exceptions
• Trade Through Exceptions
• Short Sale Exceptions
• Locked/Crossed Market Exceptions
• Post Close Trade Exceptions
• Trading Ahead Exceptions
• Unusual Activity Exceptions - Historical and Real -Time
• Marking the Close Exceptions
• Complete Analyst Workstation - Analysis, Workflow, and Reporting

We’ve got a contender for the MacBook. A firm is using a CEP engine (many of them) to look at the entire US equities market and forecast short term price movement.

KEEP YOUR OPTIONS OPEN

The firm is looking at where options are trading relative to option bid/ask spreads for at or near the money strikes and then looking at the underlying stock activity.

IT’S OCTOBER, AND THE DEADLINE DRAWS NIGH

If you’ve got an innovative CEP story, tell me about it (details here) and if I think it’s really cool, you’ll win a MacBook.

I stumbled upon this while swimming through the CEP blogosphere this weekend,

F# and CEP

At the last New York .Net Meetup, Luke Hoban presented an overview of F#. Like everyone else who’s catching the F# bug I was quite impressed with it’s a succinctness, sequences, forward pipes, and support for asynchronous programming (called workflows).

Could a reasonable amount of F# code replace the big expensive CEP engines we use?

So I asked how to compute over a stream of data, using standard deviation is a simple example. Luke has a great write up on this topic.

http://blogs.msdn.com/lukeh/archive/2008/10/10/standard-deviation-and-event-based-programming.aspx

I like the direction that F# is going in, but writing a CEP engine in a language that’s not ready for prime time yet?  I think that…

In 10 years time some applications will be large event type space applications. This will happen when a system gets to the stage where it is unifying a number of smaller event processing applications that deal with problems in different type spaces, and it is attempting to leverage the interrelationships between them.  I call such a system aholistic event processing application.

Chicago - The Shoulders of America

I will be attending the FIA Expo show in Chicago this coming November 10-12.  There are a couple of vendors there that I’ve been working with and this show, in addition to the STAC show in Janurary, are simply not to be missed.

What is Expo?
Expo showcases products, services and information for market professionals and participants. Hundreds of risk management products, trading tools, books and exchange and technology products are presented by more than 100 companies from around the world.

Expo also offers a variety of sessions that allow participants to discuss industry trends, hear expert views on key issues, improve trading skills and learn about new products, systems and practices.  The program includes sessions for business, operations and technology professionals and traders.

Who Attends?
More than 4,500 people from more than 30 countries will attend Expo. Attendees range from senior staff at brokerage firms and exchanges to floor traders, pension fund managers, corporate treasurers, CTAs and CPOs, and individual investors.