Should the Prediction Happen Here or on a Server?

Sending an image away adds work before inference can begin. Compare communication and computation, then examine what happens when the connection disappears.

13 sections · 4 questions · 1 coding challenge

1 / 13

The arrow keys move between sections too.

Communication Belongs in the Design

BOTIE

Satyanarayanan’s discussion of edge computing describes the role of computation close to data sources when latency and bandwidth make distant processing difficult. The location of computation changes how much information has to travel and how long a response takes.

Our inspection example compares a local estimate with a simplified remote path. It does not claim that local computation always wins. A larger remote model may be more useful, and many systems divide work between the device and a server.

Sources for this history

Satyanarayanan: The Emergence of Edge Computing (2017)

A Faster Server Still Has to Receive the Image

A camera watches parts moving through an inspection station. Botie can run a small model beside the camera or send each picture to a larger server. The server computes faster, but the picture has to get there and the result has to return before the part moves on.

There is no useful answer in “the cloud is faster” by itself. Let’s describe the path in units of time, then look at the assumptions a local-versus-remote comparison leaves out.

The estimates exclude queueing and assume negligible response payload, with return delay included in fixed overhead.

Step 1 of 2

Follow the remote route

The server cannot classify an image it has not received.

1
Upload
Image bytes / effective bytes per second
2
Network overhead
Connection and round-trip delay allowance
3
Compute
Remote inference time

For this simplified path, the times are sequential and can be added.

Step 2 of 2

Compare a local and a remote proposal

Both proposals must be evaluated on the same inspection task. The times here are invented.

A faster remote model still waits for communication.
Quantity	Estimate
Image size	600,000 bytes
Effective upload rate	3,000,000 bytes/s
Fixed network overhead	0.04 s
Remote inference	0.03 s
Local inference	0.12 s

Image size
Estimate
600,000 bytes
Effective upload rate
Estimate
3,000,000 bytes/s
Fixed network overhead
Estimate
0.04 s
Remote inference
Estimate
0.03 s
Local inference
Estimate
0.12 s

A faster remote model still waits for communication.

Count the journey before choosing where to compute

The sign tells us which estimated path is faster. Changing the payload or link can reverse that comparison.

Does faster inference make up for the upload?

Change image size, effective bandwidth, network overhead, or compute time. The result is remote time minus local time.

BOTIE

Botie needs a prediction before an inspected part moves on. The server is faster at inference but requires communication.

Choose an example

Image payload in bytes

Effective upload bytes per second

Fixed network overhead in seconds

Remote computation seconds

Local computation seconds

Remote minus local response time (seconds)

0.15

Follow the operations

1. Image payload in bytes
```
600000
```
2. Effective upload bytes per second
```
3000000
```
3. Divide
```
0.2
```
4. Fixed network overhead in seconds
```
0.04
```
5. Add
```
0.24
```
6. Remote computation seconds
```
0.03
```
7. Add
```
0.27
```
8. Local computation seconds
```
0.12
```
9. Subtract
```
0.15
```
10. Result
```
0.15
```

Shared preparation is excluded equally. Server queueing, response payload, outages, energy, and prediction quality need further evidence.

1. Separate the data size from the link rate

Image size is an amount of data. Bandwidth is an amount per unit time. Dividing bytes by bytes per second gives seconds. Network links are often advertised in bits per second, so convert units before using that number.

Use an effective rate measured for the workload, including the practical effects that reduce payload throughput. A maximum link specification may be a poor estimate of an upload during a busy shift.

Upload time = 600,000 bytes / 3,000,000 bytes per second = 0.20 s

Compression changes the payload size. If it also changes image details, evaluate whether the inspection model loses useful evidence.

2. Add the time between sending and acting

Remote response time includes upload, network overhead, server computation, and any response payload or queueing. Our first estimate treats the response payload as negligible and includes its travel delay in a fixed overhead term.

For local inference, preparation and actuation also take time. In this toy comparison, shared preparation and actuation are excluded from both alternatives, so we compare the part of the path that differs.

Remote estimate = 0.20 + 0.04 + 0.03 = 0.27 s
Local estimate = 0.12 s
Remote minus local = 0.27 − 0.12 = 0.15 s

A positive difference means the remote path is slower under these estimates. It says nothing yet about which model is more accurate.

3. Find the network conditions that change the choice

A smaller payload or a faster effective link can make the remote path competitive. A slower connection can reverse that result. Sweep several rates instead of treating one measurement as permanent.

NumPy can calculate a time for each candidate rate. Keep the array positions aligned so each result belongs to the intended link condition.

The comparison becomes more useful when we see where its conclusion changes.

Inspect two network conditions

This separate example uses a 200-byte teaching payload and a fixed overhead of 0.1 seconds.

Python

import numpy as np
rates = np.array([1000., 2000.])
seconds = 200 / rates + 0.1
print(np.round(seconds, 2).tolist())

Output

[0.3, 0.2]

Doubling bandwidth reduces the upload term. The fixed overhead remains.

4. Check whether the two predictions solve the same task

A tiny local model and a large remote model may have different errors. Compare them on representative inspection images, including difficult lighting and uncommon faults. If the remote model improves quality, decide whether that improvement is usable within the response deadline.

A hybrid might run locally for routine cases and send selected cases for review. That selection rule needs its own evaluation; a confidence score does not automatically identify all difficult cases.

Compare whole decisions under requirements. A latency-only winner may still be unsuitable.

5. Treat a missing connection as a real condition

A disconnected camera cannot receive a remote answer by waiting for an upload estimate. Decide whether the machine can pause, use a local fallback, or record the part for later inspection. The appropriate choice depends on the consequences of proceeding.

Buffering helps only if a later answer is still useful and storage is sufficient. A stale decision about a part that has already passed the station may have no operational value.

Do not encode a network outage as zero latency. It is an unavailable path requiring another action.

6. Consider energy and data exposure separately

Sending data can consume energy, as can local computation. The balance depends on the hardware, radio, workload, and duty cycle. Measure it when battery life is a requirement rather than assuming one location is always cheaper.

Moving images off a device also changes where data are stored and who can access them. Determine whether the task needs raw images, cropped regions, or a smaller representation, and document retention and access requirements.

The workshop compares response-time estimates. A deployment decision also needs evidence about accuracy, resource use, and the permitted data path.