LLM Notes

Links:

1. Tokenization (BPE, WordPiece, SentencePiece):

   • karpathy’s video
   • huggingface tokenizers
   • HF tokenizers usage example

2. Attention:

   • self attention
   • multi-head attention
   • cross attention
   • multi-query attention
   • grouped-query attention
   • multi-head latent attention

3. Positional Encoding:

   • sin-cos (traditional) positional embedding
   • rotary positional embedding

4. Normalization:

   • layer normalization
   • RMS normalization

5. Activation Functions & attention inference trick:

   • Swiglu

6. Mixture of Experts (MoE):

   • MoE
   • Vizura’s moe

7. LLM tricks & advanced techniques:

   • KV cache
   • Flash Attention
   • Paged Attention
   • VLLM

8. Finetuning:

   • LoRA
   • QLoRA
   • PEFT
   • SFT
   • RLHF