← All workshopsRead Morfessor

Natural Language Processing

Look for reusable word parts

BOTIE

The model supplies description costs for reusable pieces and whole words. Choose the least costly complete analysis, then return the pieces' IDs.
Ready to test
0 passed · 0 failed · 1 not run

An empty floor. Start with the information your machine needs.

Received result

Your answer will arrive here.

Examples, experiments and workspace controls
Live values

Connect a tool's inputs to see what it produces. You can try a branch before connecting it to Result.

Try your own inputs

Use a copy of the example to try your own data. Keep the problem's assumptions in mind, and predict the effect of a change before checking the output. You can also add another branch to compare two ways of doing the calculation.

Start at an output arrow, then click or drag to an input. Drag empty space to pan. Delete removes a selection. Ctrl/Cmd+Z undoes a change.
Context for this challenge

The model supplies description costs for reusable pieces and whole words. Choose the least costly complete analysis, then return the pieces' IDs.

Botie's related words share a stem but have different endings. Supplied description costs let us compare reusing those pieces with storing each complete word.

Select the least-cost complete segmentation under the supplied description costs, then retrieve the selected morpheme IDs.

Examples and results
Word
walking
Supplied description costs
{
  "walk": 1,
  "ing": 1,
  "walking": 3
}
Morpheme IDs
{
  "walk": 1,
  "ing": 2,
  "walking": 3
}
Required result
  1. 01
  2. 12
Try a fresh construction

This clears this draft, including its undo history. Earlier successful attempts stay recorded. Build again without opening hints to record an independent attempt.

Stuck? Inspect a worked construction

The tools below form one possible solution. You can assemble it, follow its signals, and undo that change to return to your own construction.

Assumptions and test precision

The costs are supplied to isolate decoding. Full Morfessor learning balances the corpus description against the cost of storing its lexicon. Numerical tests allow a difference of 0.001. Lists and structures must also have the right shape.

The supplied description costs favour a whole word

This check uses the pieces from “Compare whole-word and reusable-part analyses”.

BOTIE

The candidate inventory can cover replayed in several ways. Return the IDs for the lowest-total-cost segmentation. Use the supplied description costs even when a more finely split word looks linguistically plausible.
Word
replayed
Supplied description costs
{
  "re": 0.8,
  "play": 0.7,
  "ed": 0.4,
  "replay": 1.2,
  "replayed": 1.4
}
Morpheme IDs
{
  "re": 1,
  "play": 2,
  "ed": 3,
  "replay": 4,
  "replayed": 5
}

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

Why would always splitting into smaller pieces omit part of Morfessor's description-length tradeoff?

Learning balances representation of the corpus with storage of the lexicon; this construction only decodes supplied costs.

The costs are supplied to isolate decoding. Full Morfessor learning balances the corpus description against the cost of storing its lexicon.

Work through the Python challenges →
Continue the courseFinite-State Morphology →
← Moving a Vocabulary