One Training Dataset, Several Unequal Files

Move Summaries Instead of Every Row

BOTIE

Suppose a training dataset is spread across more files than one worker can process quickly. We can give each worker a portion, but then we need a way to combine its answer. Sending every original row back to one machine can recreate the bottleneck we were trying to remove.

Dean and Ghemawat's 2004 MapReduce paper described a programming model that separated work on individual records from combining results with the same key. The system handled distributing that work and retrying failed tasks. It gave programmers a way to describe the calculation without writing a separate scheduling system for every job.

For our mean, each worker can send a sum and a count. Those quantities still tell us how much data its answer represents, even when file sizes differ. A local mean alone has discarded that information. We will follow these small summaries through the same kind of merge, using ordinary Python before needing a distributed service.

Sources for this history

Dean and Ghemawat: MapReduce, OSDI 2004

Why Did Splitting the File Change the Mean?

Botie’s training records no longer fit comfortably in one read. We process each file separately and average the file means. That sounds reasonable until one file has a hundred readings and another has two. The tiny file now has as much influence as the large one.

We need a summary that can be combined without forgetting how many observations it represents. For a mean, the sum and count are enough.

Counts are nonnegative and the workshop has at least one observation. Python handles the all-empty case.

Step 1 of 1

The partitions have different sizes

Both summaries describe the same temperature feature.

The combined mean is 30; the mean of the two partition means is 50.
Partition	Sum	Count
Morning	30	3
Night	90	1

Morning
Sum
30
Count
3
Night
Sum
90
Count
1

The combined mean is 30; the mean of the two partition means is 50.

A partition boundary is not a sampling weight

The denominator preserves the number of contributing readings.

Merge the worker summaries

Unequal sensor files must still give each reading one contribution.

BOTIE

Unequal sensor files must still give each reading one contribution.

Choose an example

Partition sums

1

2

Partition counts

1

2

Mean across all records

Follow the operations

1. Partition sums
1. 020
2. 140
2. Add a list
```
60
```
3. Partition counts
1. 02
2. 12
4. Add a list
```
4
```
5. Divide
```
15
```
6. Result
```
15
```

Counts are nonnegative and the workshop has at least one observation. Python handles the all-empty case.

1. A partition boundary is not a sampling weight

A partition is a storage or processing division. Its size may follow a date, a machine, or an upload batch. It need not represent an equally important population.

Keep a pair of numbers for each partition: the sum of accepted values and their count. Missing values must follow the same inclusion policy in both numbers.

Selecting one feature column avoids reading irrelevant fields, but does not change the meaning of the observations.

2. Merge the quantities that add

Sums add across disjoint partitions. Counts do too. Divide only after those two merges.

total sum = 30 + 90 = 120
total count = 3 + 1 = 4
combined mean = 120 / 4 = 30

An empty partition contributes zero to both totals. An entirely empty dataset has no mean; return an explicit missing result.

Keep the count with the subtotal

One tuple describes each file without pretending it represents an equal-sized sample.

Python

parts = [[10,10,10],[90]]
print([(sum(p),len(p)) for p in parts])

Output

[(30, 3), (90, 1)]

The first tuple summarises three observations; the second summarises one.

Check yourself

What does each partition represent?

Question 1 of 2

What is the combined mean?

Use the information below.

Partition A: sum 30, count 3. B: sum 90, count 1.

3. A retry can count the same file twice

A merge assumes each observation contributes once. Retrying a successful task and appending its summary again violates that assumption. Store summaries by partition identity and input version, replacing the result for that logical partition.

Use immutable input snapshots during a training run. If a partition changes halfway through processing, even a correct reducer can combine incompatible versions.

This is where data engineering meets the workflow graph: completion must refer to one identified input.

4. Precision and other summaries

Floating-point sums can differ slightly with merge order. Use suitable precision and numerical tolerances. Variance needs more information than a sum and count; quantiles generally cannot be recovered by averaging partition quantiles.

Choose a mergeable summary for the statistic you actually need, and compare it against an unpartitioned calculation on a small fixture.

The workshop uses supplied sums and counts. The Python challenge constructs them from records so we can test changing partition boundaries.

5. A sum can lose small contributions

Floating-point arithmetic has finite precision. Adding many small values to a much larger total can lose information, and a different partitioning can change the reduction order. Pairwise or compensated summation can reduce the error.

Use a high-precision reference on a small difficult fixture and compare with a justified tolerance. Do not fix a mismatch by rounding every intermediate value, which may introduce still more error.

The merge identity is exact over real arithmetic. Its floating-point implementation needs numerical testing as well as correct weights.