Published on May 22, 2015 by Manoj Fernando
Real-time analytics is generally not a straightforward match for financial research. Historical data typically takes on a rather static formation, and the general view is that you can apply certain ‘filters’ on a subset of fields to get what you need. This is one reason for Microsoft Excel’s popularity among analysts when it comes to using an application for data collection and filtering.
Financial researchers tend to observe historical data, and the concept of real-time analytics playing a role therein is difficult to comprehend.
But what if you have to deal with a very large volume of indirectly related data collected over a long period of time, and the usual filtering you have been using does not scale up to support it?
At a glance, it may appear that you could apply multiple levels of filtering until you arrive at the desired output, but this could take much time and effort if done manually. Moreover, if you are dealing with large volumes of unstructured and indirectly related data, it would require a technical team to build complex correlation logic for each filter criteria.
On the other hand, technology solutions built by leveraging the power of real-time analytics already provide capabilities to process large data within the shortest possible time. For example, emerging industry trends such as Complex Event Processing (CEP) frameworks have been designed to combine simplified information-processing logic with high volumes of fast-moving data. For financial research, we could probably give less prominence to the ultra-high speeds supported by these frameworks, but processing logic simplification, combined with high scalability, could certainly play a significant role in financial data analysis.
Let us take an example of an analyst wanting to obtain seasonal variances of sales data of a particular product across several geographies, overlapped with some news feeds that may indicate corresponding market demand. This could be a very large volume of data demanding great deal of time to filter manually. What if we could create a procedure to feed this data in batches to a CEP engine in which multiple filter rules are applied immediately to a ‘free-flowing’ stream of data?
As depicted in the diagram, event definitions can be applied in real time to a stream of data loaded either by a batch runner or data feeds. The CEP engine will use these rule definitions to match multiple data streams; the output can be fed straight into the analyst model (possibly through an interim data store) or even be used as input to another CEP cycle.
Where the processing component of the solution is concerned, CEP frameworks generally provide an Event Processing Language (EPL) – a declarative language similar to SQL for dealing with high-frequency, time-based event data. Esper and WSO2 CEP are two of the leading frameworks in the market that can be used when building such solutions related to the data capturing and processing needs of financial research.
What's your view?
Thank you for sharing your Comments
About the Author
Manoj currently provides technical leadership across software development engagements. In addition, he provides solutions and architecture support to clients, trains and mentors employees, and sets up standards and best practices. He has over 15 years of experience in delivering enterprise scale system integrations and introducing IT strategy transformation initiatives for global clients across multiple domains.
Prior to joining Acuity Knowledge Partners, he worked as Director at WSO2 Pvt. Ltd., leading a number of client-facing assignments, architectural reviews, and product presentations. Manoj holds a bachelor’s degree in Science (Information Systems).
Community banking’s role in serving small busi....
Community banking trends Amid the pandemic-related uncertainties and shifts in market con....Read More
What led to the global supply chain crisis, and ....
The global economy has faced severe supply chain disruptions since the start of the COVID-....Read More
Technology impact on trading and compliance....
With companies forced to adopt remote working options amid the pandemic, demand for digita....Read More
Like the way we think?
Next time we post something new, we'll send it to your inbox