Big Data and Machine Learning Systems

Date	Topic
Week 1 (1/24)	Introduction, the Model Optional Readings: Deep Learning Book Chap 6.1-6.4 How to read a paper
Week 2 (1/31)	The Frameworks Required Readings: TensorFlow: A System for Large-Scale Machine Learning PyTorch: An Imperative Style, High-Performance Deep Learning Library
Week 3 (2/7)	The Hardware Required Readings: An even easier introduction to CUDA Optional: Cuda C++ Programming Guide(Sec 2 and 3)
Week 4 (2/14)	Compilation and Optimization I Required Readings: TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Ansor: Generating High-Performance Tensor Programs for Deep Learning Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations Optional: Learning to Optimize Tensor Programs TensorIR: An Abstraction for Automatic Tensorized Program Optimization Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programs
Week 5 (2/21)	Compilation and Optimization II Required Readings: DNNFusion: accelerating deep neural networks execution with advanced operator fusion TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions Optional: Pet: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections
Week 6 (2/28)	Training I: data vs. model parallelism Required Readings: PyTorch Distributed: Experiences on Accelerating Data Parallel Training Supporting Very Large Models using Automatic Dataflow Graph Partitioning Optional: One weird trick for parallelizing convolutional neural networks
Week 7 (3/6)	Training II: pipeline parallelism Required Readings: GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism PipeDream: Fast and Efficient Pipeline Parallel DNN Training Optional: Memory-Efficient Pipeline-Parallel DNN Training
Week 8 (3/13)	Training III: automatic parallelization Required Readings: Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization Optional: GSPMD: General and Scalable Parallelization for ML Computation Graphs
Week 9 (3/20)	Spring Break
Week 10 (3/27)	LLM Background Required Readings: Attention Is All You Need FlashAttention-2:Faster Attention with Better Parallelism and Work Partitioning Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Optional: Eight Things to Know about Large Language Models
Week 11 (4/3)	Training LLMs I Required Readings: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Optional: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Reducing Activation Recomputation in Large Transformer Models Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models
Week 12 (4/10)	Inference: Required readings: Orca: A Distributed Serving System for Transformer-Based Generative Models Efficient Memory Management for Large Language Model Serving with PagedAttention Optional: Fast Inference from Transformers via Speculative Decoding Accelerating large language model decoding with speculative sampling
Week 13 (4/17)	Quantization Required Readings: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models QLoRA: Efficient Finetuning of Quantized LLMs Optional: GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
Week 14 (4/24)	Sparsification Required Readings: Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot Optional: Optimal Brain Damage The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Week 15 (5/1)	Project Presentation