How ‘Embeddings’ Encode What Words Mean — Sort Of

A picture may be worth a thousand words, but how many numbers is a word worth? The question may sound silly, but it happens to be the foundation that underlies large language models, or LLMs — and through them, many modern applications of artificial intelligence. Every LLM has its own answer. In Meta’s open-source Llama 3 model, each word contains 4,096 numbers; for GPT-3, it’s 12,288.

Source