BYNotes

NLP and Large Language Models

2 min read

Natural language processing and large language model notes covering static embeddings, language models, BERT, LLM training, reasoning, uncertainty, and context engineering.

Start Here

For a structured pass through NLP and LLMs, read the modules in this order:

  1. Embeddings
  2. Language Models and Pretrained Models
  3. LLM Training and Alignment
  4. LLM Inference and Analysis

Module Map

ModuleScope
EmbeddingsWord vectors, Word2Vec, negative sampling, and GloVe
Language Models and Pretrained ModelsN-gram models, BERT, and pretrained Transformer encoders
LLM Training and AlignmentPretraining, alignment, post-training, and forgetting
LLM Inference and AnalysisReasoning, uncertainty, context engineering, and analysis

Notes

Embeddings

AreaNoteStatus
OverviewWord Embeddingsdraft
Static embeddingWord2Vecdraft
OptimizationNegative Samplingdraft
Static embeddingGloVedraft

Language Models and Pretrained Models

AreaNoteStatus
Language modelingN-Gram Language Modelsdraft
Pretrained modelBERTdraft

LLM Training and Alignment

AreaNoteStatus
AlignmentPretraining and Alignment for LLMsdraft
Post-trainingPost-Training and Forgettingdraft

LLM Inference and Analysis

AreaNoteStatus
ReasoningDeep Reasoning for LLMsdraft
UncertaintyUncertainty in LLMsdraft
ContextContext Engineeringdraft

Concept Map

  • Embeddings: one-hot encoding, dense word vectors, Word2Vec, negative sampling, GloVe, and contextual embeddings.
  • Language models: N-gram models, Markov assumptions, sparsity, and pretrained Transformer encoders.
  • LLM training: pretraining, SFT, RLHF, alignment, knowledge distillation, post-training, and catastrophic forgetting.
  • LLM inference: chain of thought, test-time compute, reasoning workflows, uncertainty estimation, semantic entropy, and context engineering.

Planned Notes

  • LLM Sampling Methods
  • Latent Space LLM Survey 2024
  • Mitigating LLM Hallucinations via Conformal Abstention
  • Information Theory for NLP
  • Transformer and Attention

4 items under this folder.