Realtime stream processing using Apache Storm - Part 1

Choosing a tool for stream processing depends on whether we want the data processed in "realtime" or "near realtime", deciding on fault tolerance between "atleast once semantics" and "exactly once semantics" and between "sub-second" latency and "few seconds" latency should help us decide between Storm and Spark. 

In this post we will look at the innards of storm as applicable to stream processing in continuation to the streaming analysis using Spark that we discussed previously here, here and here.

Here's Storm and Spark in conversation about what they are capable of:

Storm: I process one incoming event at a time

Spark:  I collect data over small time window as a micro batch and process them

Storm: I can provide sub-second latency

Spark: I only provide few seconds latency. But hey, I ensure that data is read exactly once and guarantee better fault tolerance.

Storm:  I guarantee that I read data at least once. 

Spark: But that may allow duplicate data once you recover from faults.

Storm: Ya. But that's a compromise you have to make if you need faster processing. Never mind. Let me show you how I do my stuff.....

Core concepts:

Topology: Its the container of the business logic in the real time application. 

Streams: Core abstraction in Storm.Unbounded sequence of tuples that is processed and created in parallel.

Spouts: Source of streams in a topology. Reads tuples from external source and emits them into topology.

Bolts: Consumes any number of input streams, applies business logic and may emit new streams. It is where actual processing takes place.

• Network of spouts and bolts constitute a "topology" which is submitted to a storm cluster for execution.
• A topology is a graph of stream transformations where each node is a spout or a bolt and edges represent which bolt subscribes to which stream.

Lets create a Twitter wordcount topology that counts the total occurrences of words and prints the top words every few seconds. 
You can follow the steps in this link to create your twitter oAuth tokens.

Create a Java maven project and update the pom.xml as given below:
Topology:
Now create a TwitterWordCountTopology.java class to submit the "TwitterWordCountStorm" topology.

A look at how the topology is configured:

    TopologyBuilder builder = new TopologyBuilder();  
    builder.setSpout("twitter-spout", new TwitterSampleSpout(consumerKey,  
      consumerSecret, accessToken, accessTokenSecret, keyWords));  
    builder.setBolt("WordSplitterBolt", new WordSplitterBolt(5)).shuffleGrouping("twitter-spout");  
    builder.setBolt("IgnoreWordsBolt", new IgnoreWordsBolt()).shuffleGrouping("WordSplitterBolt");  
    builder.setBolt("WordCounterBolt", new WordCounterBolt(5, 5 * 60, 50)).shuffleGrouping("IgnoreWordsBolt");  

The topology contains a spout and 3 bolts.

• We have a spout class TwitterSampleSpout named twitter-spout that takes the twitter oAuth tokens as arguements and generates a stream of tweets based on keywords using the twitter4j library.
• A WordSplitterBolt class named WordSplitterBolt that consumes from twitter-spout. This class splits the incoming streams into words. It takes the minimum word length as argument. 
• A IgnoreWordsBolt class named IgnoreWordsBolt that consumes from WordSplitterBolt bolt. This class ignores a set of words from the stream and passes the rest of the words further downstream. 
• Words from IgnoreWordsBolt stream are consumed by WordCounterBolt class named WordCounterBolt. This class takes the time intervals for the wordcount as arguments and displays the wordcount and top words over the given time interval.

Grouping:

How the streams are distributed to the bolts is decided by 4 types of grouping that can be applied to the setBolt method: