Why Does the Dashboard Still Count a Job That Was Closed?

Botie closes J1 in the job store. The asset dashboard is a separate read view, so its open-job count still includes J1 until that change reaches it. Updating both places with two unrelated writes creates a gap if one succeeds and the other fails.

A change stream can carry source changes to a projection, a derived representation built for a particular read. We will retain the latest image of each job and derive the asset counts from that state.

Merge job versions first, count only current open jobs by asset, and retain the images alongside the derived result.

{
  "state": [
    {
      "id": "J1",
      "asset": "P7",
      "version": 2,
      "status": "done"
    },
    {
      "id": "J2",
      "asset": "P8",
      "version": 1,
      "status": "open"
    }
  ],
  "open_by_asset": {
    "P8": 1
  }
}

The job store owns job state. The dashboard owns a derived count that can be rebuilt from it. Users should not independently edit the same job status through both stores without a conflict policy.

A projection can be a document, an index, a search view or another table. Its useful difference is a read shape suited to the dashboard. That convenience comes with a freshness lag and a repair responsibility.

The dashboard should expose enough freshness information to avoid presenting old counts as a new source decision.

A change may contain a full document image, changed fields, a deletion key, or before and after images depending on the system and configuration. Applying a patch requires the state it expects. A delete needs enough identity to remove the previous contribution.

Our exercise deliberately uses full images with id, asset, version and status. A deleted status is a retained tombstone in this toy protocol. Actual stream event formats and image options must be read from their source documentation.

The explicit image contract lets us reason about replay without inventing a database event format.

A dictionary keyed by job ID keeps one current image per identity. A greater version replaces the old image, a smaller version is ignored, and an equal version with an identical image is a replay. Conflicting images for the same identity and version anywhere in the supplied state and batch violate the input contract and raise an error, including older images already superseded by a newer version. The model cannot inspect historical images that were not supplied.

We then count open images by asset. Because we recompute from retained state, a moved job stops contributing to its old asset. This is a deliberately small rebuild, not a claim that every production dashboard should rescan its full state on each event.

The stored images make the count’s explanation available.

images = [{"id":"J1","asset":"P7","status":"open"},
          {"id":"J1","asset":"P7","status":"open"}]
by_id = {row["id"]:row for row in images}
print(len(images))
print(len(by_id))

2
1

Suppose J2 was assigned to P7 and is corrected to P8. An incremental count needs to retract the old contribution before adding the new one. Without the old image or equivalent maintained state, the consumer cannot know which group to decrement.

Our rebuilt view handles the correction by counting only J2’s latest retained image. In a larger incremental projection, test retractions, deletes and reclassification explicitly. A system tested only on inserts can produce plausible counts long after its first correction.

The changed workshop case moves a job rather than merely adding another arrival.

A consumer can receive a message, update its view, then lose its response before saving progress. Resuming can repeat application. Saving the source position before the view update risks skipping an unapplied effect.

Use an atomic sink/checkpoint operation where the architecture supports it, or a replay-safe consumer and a documented recovery protocol. The versioned full-image merge here makes repetition harmless to this derived state. It does not make an unrelated email or stock decrement replay-safe.

A stream’s delivery wording does not establish exactly-once effects across arbitrary downstream services.

MongoDB change streams provide resume tokens and require enough retained history to resume. Keep the token with its stream configuration; do not parse it into an application timestamp or reuse it with a different pipeline casually. DynamoDB Streams uses a different shard-based interface and retention contract.

A consumer falling behind retention needs a snapshot-and-stream recovery procedure with a consistent handoff. A success message from the last process run cannot prove that every required change remains available.

The browser input is a finite supplied batch. It does not emulate a resumable network cursor.

Rebuild a damaged projection into a separate destination when appropriate, compare identities and counts against the source, then switch readers deliberately. Keep a version for the projection logic so a transformation change is distinguishable from new business data.

Bound retries and watch processing lag, not only worker uptime. A worker can stay alive while a malformed event prevents later events from applying. Preserve that event for diagnosis instead of silently advancing its checkpoint.

The next lesson makes a representation change explicit so old and new documents can be read without guessing their fields.