Adaptive Online Scheduling in Storm: Enhancing Data Processing Efficiency

Paper presented by Keshav Santhanam on adaptive online scheduling in Storm, focusing on efficient data processing in the era of big data. The presentation covers the motivation behind the need for complex event processing, overview of Storm as a high throughput data processing engine, key concepts such as topologies, tuples, and bolts, challenges with existing scheduling strategies, and the proposed adaptive scheduling approach. The strategy involves offline and online schedulers that aim to reduce network traffic by placing frequently communicating executors together in the same slot. Offline scheduling is based on topology analysis before deployment, while online scheduling adjusts schedules based on real-time network traffic analysis. The presentation highlights the heuristic used for executor placement, assumptions made, and the topology-based scheduling process.

lamallaoui + Follow