Section 2 of 4 · 4 lessons

Model the System Before Building It

A prediction service needs more than a fast model. Images must be prepared, requests can arrive together, and devices have limited memory. We will calculate where the work goes, then compare local and remote inference using the same task.

0 of 4 finished

Throughput and Waiting

Requests pass through several stages, and unfinished work carries forward. Follow that work to find the limiting stage and see what a burst does to the queue.

  1. 01Where Does the Prediction Pipeline Run Out of Room?A fast model can sit inside a slow service. Follow requests through preparation, inference, and storage to find the stage that limits the whole path.
    Build with Botie →Read the lesson
  2. 02What Happens When Requests Arrive Together?Average demand can fit while a burst still leaves people waiting. Follow unfinished requests from one interval to the next.
    Build with Botie →Read the lesson

Memory and Communication

A model needs space to run, and a remote model needs time to receive its inputs. Account for those costs before choosing a device or moving computation to a server.

  1. 03Will the Model Fit on the Device?A model file can fit while its running process runs out of memory. Account for parameters, temporary values, and the rest of the runtime.
    Build with Botie →Read the lesson
  2. 04Should the Prediction Happen Here or on a Server?Sending an image away adds work before inference can begin. Compare communication and computation, then examine what happens when the connection disappears.
    Build with Botie →Read the lesson