How does the Storm handle nextTuple in the Bolt

apache-storm, java

Solution

how does the Storm framework internally calls the nextTuple() method continuously.

I believe this actually involves a very detail discussion about the entire life cycle of a storm topology as well as a clear concepts of different entities like workers, executors, tasks etc. The actual processing of a topology is carried out by the `StormSubmitter` class with its `submitTopology` method.

The very first thing it does is start uploading the jar using Nimbus's Thrift interface and then calls the submitTopology which eventually submit the topology to Nimbus. The Nimbus then start by normalizing the topology (from doc: The main purpose of normalization is to ensure that every single task will have the same serialization registrations, which is critical for getting serialization working correctly) followed by serialization, zookeeper hand shaking , supervisor and worker process startup and so on. Its too broad to discuss but If you really want to dig more you can go through the life cycle of storm topology where it explain nicely the step by step actions performs during the entire time. ( quick note from the documentation)

First a couple of important notes about topologies:

The actual topology that runs is different than the topology the user specifies. The actual topology has implicit streams and an implicit "acker" bolt added to manage the acking framework (used to guarantee data processing).

The implicit topology is created via the system-topology! function. system-topology! is used in two places: - - when Nimbus is creating tasks for the topology code - - in the worker so it knows where it needs to route messages to code

Now here's few clue I could try to share ... Spouts or Bolts are actually the components which does the real processing (the logic). In storm terminology they executes as many tasks across the structure. From the doc page : Each task corresponds to one thread of execution

Now, among many others, one typical responsibility of a `worker process` (read here) in storm is to monitor weather a topology is active or not and stored that particular state in a variable named `storm-active-atom`. This variable is used by the tasks to determine whether or not to call the `nextTuple` method.. So as long as your topology is live (you haven't put your spout code but assuming) till the time your timer is active (as you said for certain time) it will keep calling the nextTuple method. You can dig even further to understand the storm's Acking framework implementation to understand how it understand and acknowledge once a tuple is successfully processed and Guarantee-message-processing

I am sure that my understanding is missing something here and due to this gap I am unable to connect to the internal logic of this framework

Having said this I think its more important to get a clear understanding of how to work with storm rather than how to understand storm in the early stage. e.g instead of learning the internal mechanism of storm its important to realize that if we set a spout to read a file line by line then it keep on emitting each lines using the `_collector.emit` method till it reaches EOF. And the bolt connected to it receive the same in its `execute(tuple input)` method

Hope this help you share more with us in future

Problem

Original source