← All workshopsRead Moving a Vocabulary

Natural Language Processing

Keep the IDs and embeddings together

The same integer now names another token

Botie is changing token order while keeping learned vectors. A row must move with its token or the new vocabulary IDs will retrieve someone else's features.

BOTIE

After the red and blue token order is swapped, why is it unsafe to look up the new IDs in the old row order?
The inputs we are comparing

Old token order

Example 1

  1. 0cat
  2. 1dog

Example 2

  1. 0red
  2. 1blue

Old embedding rows

Example 1

10
02

Example 2

31
14

New token order

Example 1

  1. 0dog
  2. 1cat

Example 2

  1. 0blue
  2. 1red

IDs in the new vocabulary

Example 1

  1. 00
  2. 11
  3. 20

Example 2

  1. 01
  2. 10
Result for example 1
02
10
02
Choose an explanation to test
Three tree tokens change their addresses

This check uses the pieces from “Move embedding rows by token identity”.

BOTIE

Reorder the embedding rows to match the new vocabulary, then retrieve the supplied new IDs. Preserve token identity and every repeated ID rather than keeping the old row addresses.
Old token order
  1. 0oak
  2. 1elm
  3. 2ash
Old embedding rows
10
02
33
New token order
  1. 0ash
  2. 1oak
  3. 2elm
IDs in the new vocabulary
  1. 02
  2. 10
  3. 22

Use a number, text, or an array matching the requested result. For a matrix, put each row in brackets. Saving your prediction runs your current construction on these inputs and checks the earlier examples too.

Build a construction in the workspace above first. This case will test those pieces.

How does the repeated ID in the first example help check that identity survived the move?

Preserve token identity across vocabulary migrations and check special-token conventions separately.

This is a small construction problem. The lesson explains the assumptions and the parts of the full method that this example leaves out.

Work through the Python challenges →
Continue the courseMorfessor →
← Greedy Coverage