Event time versus processing time, watched live

flume

Every event here carries two times: when it happened, and when it arrived. Windows close on a watermark over the first, never on a clock over the second.

Whose semantics these are, and what this is not

The watermark, the trigger, the allowed-lateness bound and the retracted pane are the Dataflow model’s: Akidau et al., The Dataflow Model, VLDB 2015. flume implements a small, single-process reading of those semantics so they can be watched and checked. It is not Flink, not Beam, and not Kafka Streams.

Loading the engine…

How to drive it, and where the planted bug hides

Every control is in the URL, so any view you reach is a link you can send. The demo log pins its window size and allowed lateness - those two are what make nineteen rows tell the story - but the policy, the termination and the watermark strategy apply to every stream. processing-time is one of the planted bugs, and it needs a stream whose arrival rate diverges from its event rate before it does any damage: switch to generated, seeded to watch it overshoot.

What the picture says

The blue line is the watermark: the engine’s standing claim that no event with an event time at or below it will arrive from here on. It only ever moves right as you go down, because a watermark that moved back would retract a claim the engine had already acted on.

Every dot to the left of that line arrived late. That is not a definition to memorise; it is a region of the plane. A dot inside the allowed lateness makes its window emit a corrected pane - the window is revised, and the row shows the old value it retracted. A dot past the bound cannot: its window’s state is gone, so the event goes to the side output carrying its reason, and the window becomes unverifiable.

Three outcomes, never two

complete - the watermark closed it, and the emitted value equals what a batch recomputation over the whole log says. Nothing arriving later changed it.
revised - late data inside the bound corrected an already-emitted pane. The final value is right; it was not right the first time, and the report refuses to blur those two.
unverifiable - the emitted panes cannot be shown to be right, for one of two stated reasons: the watermark never reached the window (the log ended first), or data arrived past the allowed lateness. Both are printed; neither is rounded to “fine”.

Switch the stream to USGS earthquakes to watch a bound that is fine for one stream fail on another. That catalogue revises records for weeks, so a watermark bound measured in seconds turns almost every window unverifiable - with a receipt for every event, and no accusation against the engine, because the engine did exactly what it was configured to do.

What this is not

One process, in-memory state, no checkpointing, no exactly-once across a restart, tumbling windows only, and one aggregate (a count) chosen because its ground truth is one line of code. It is a demonstration of semantics, not a stream processor to run anything on.

Source, tests, the planted fixtures and the three vendored real streams: github.com/jamessuuu/flume.