The Reading Happened Before the Window Closed

Streaming Systems Cannot Wait for an End of File

BOTIE

A file eventually ends. A stream of sensor readings may keep arriving, with yesterday's disconnected sensor sending a reading after today's records. Waiting for every possible late record would prevent us from publishing an answer at all.

The 2015 Dataflow Model paper addressed this tension between timely results and out-of-order data. It separated the time an event happened from the time a system processed it. Windows grouped events into intervals; watermarks estimated how far event time had progressed. Triggers made the decision about when to emit a result explicit. These ideas became part of Apache Beam's processing model.

A watermark is an estimate, so a late reading still needs a policy. We will inspect membership in one event-time window and watch what a delayed record changes. The local exercise exposes that decision; it does not implement Beam's distributed state or recovery.

Sources for this history

A Late Packet Changes Yesterday’s Hour

Botie’s agent uses a recent sensor average. Most packets arrive quickly, but a disconnected device sends its backlog later. Grouping by arrival time makes an old event look like a current condition.

Window membership should follow the time the event describes. Deciding when a window is complete is another problem, because we cannot wait forever for missing packets.

Arrival handling, watermarks and persistent streaming state are explained but not simulated by this finite membership circuit.

Step 1 of 1

A window uses the event timestamp

The interval includes its start and excludes its end.

A watermark is a progress estimate, not proof that every earlier event arrived.
Event	Arrival	Window [10,20)
10	11	Inside
19	25	Inside, late arrival
20	20	Next window

10
Arrival
11
Window [10,20)
Inside
19
Arrival
25
Window [10,20)
Inside, late arrival
20
Arrival
20
Window [10,20)
Next window

A watermark is a progress estimate, not proof that every earlier event arrived.

A half-open interval avoids double counting

The upper comparison is strict so adjacent windows remain disjoint.

Place events in the right feature window

Botie must keep an old delayed reading out of the current window.

BOTIE

Botie must keep an old delayed reading out of the current window.

Choose an example

Event timestamps

1

2

Sensor values

1

2

Included start

Excluded end

Values belonging to this event-time window

04
18

Follow the operations

1. Sensor values
1. 04
2. 18
2. Event timestamps
1. 011
2. 115
3. Included start
```
10
```
4. Threshold gate
1. 01
2. 11
5. Excluded end
```
20
```
6. Strict threshold
1. 01
2. 11
7. Multiply
1. 01
2. 11
8. Select real entries
1. 04
2. 18
9. Result
1. 04
2. 18

Arrival handling, watermarks and persistent streaming state are explained but not simulated by this finite membership circuit.

1. A half-open interval avoids double counting

Choose a start that is included and an end that is excluded. An event exactly on a shared boundary belongs to the next window, so adjacent windows do not count it twice.

Keep event time with the raw record even when ingestion adds a separate arrival timestamp.

Window width should follow the feature’s meaning and the response time the application needs.

2. Build membership without using arrival order

Compare each event time against the start and end. Use the resulting mask on the aligned readings.

events = [10,19,20]
window = [10,20)
keep = [1,1,0]
values = [4,8,100]
window mean = (4 + 8) / 2 = 6

The record at event 19 still belongs here when it arrives at 25. Whether to revise an emitted result depends on the lateness policy.

The shared boundary appears only once

Compare one event with two neighbouring intervals.

Python

event = 20
print(10 <= event < 20)
print(20 <= event < 30)

Output

False
True

The same timestamp belongs to the later window because the earlier upper endpoint is excluded.

Check yourself

Membership and arrival are different questions

Question 1 of 2

Where does it belong?

Use the information below.

Window [10,20), event at 20.

3. An emitted feature may have versions

A service can emit a provisional result, then revise it when permitted late data arrive. Preserve the revision and its availability time. A training replay must use the version available at its decision, not the final corrected average.

Alternatively, a policy may drop overly late events from online state while retaining them for investigation. Make that exclusion measurable.

The historical retrieval lesson applies again: a correction’s event window and availability are different facts.

4. Recovery must preserve contributions

If a stream restarts from an earlier offset, records may replay. Deduplicate by a stable event identity where the delivery semantics require it. A window mean needs a sum and count with consistent update semantics.

The coding exercise processes a finite fixture and returns None for an empty window. A running system also needs bounded state retention and checkpoint recovery.

Tests should include boundary events and out-of-order arrivals, not only an already sorted stream.