Language modeling notes from count-based N-gram models to pretrained Transformer encoders.
Notes
| Area | Note | Status |
|---|---|---|
| Language modeling | N-Gram Language Models | draft |
| Pretrained model | BERT | draft |
Learning Use
- Use N-gram models as the baseline for sequence probability.
- Treat BERT as the bridge from static embeddings to pretrained contextual encoders.
- Prepare for LLM pretraining and alignment notes.