DATA3404 Lecture Notes - Lecture 11: Wireless Sensor Network, Network Monitoring, Stream Processing
22 Jun 2020
School
Department
Course
Professor
Document Summary
Many applications with large streams of live data that needs to be processed immediately ( real- time": traffic management applications, stock markets, fraud detection in payment systems, sensor networks, environmental monitoring systems. Why not a dbms: not agile enough; needs persistent storage and indexing before processing, data at rest ; favours infrequent updates. Stream processing systems: storage is less important than reacting to data changes in real time, data in motion : processing data as it flows without storing persistently, but may buffer temporarily to provide windows. Inserts are typically batched, high overhead: expressiveness: simple conditions (triggers), no built-in notion of sequence (views, no notion of approximation, current systems don"t scale to large # of triggers, views don"t provide streamed results. As seen in this diagram, input streams are directly sent to stream query processor. Selections and (duplicate preserving) projections are straight forward. For data stream query: local, per element operators, select sourceip, time.
