Big Data and Machine Learning Systems

Date	Topic
Week 1 (9/3)	Introduction, the Model Optional Readings: Deep Learning Book Chap 6.1-6.4 How to read a paper
Week 2 (9/10)	The Frameworks Required Readings: TensorFlow: A System for Large-Scale Machine Learning PyTorch: An Imperative Style, High-Performance Deep Learning Library
Week 3 (9/17)	The Hardware Required Readings: An even easier introduction to CUDA Optional: Cuda C++ Programming Guide(Sec 2 and 3)
Week 4 (9/24)	Compilation and Optimization I Required Readings: PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations Optional: TensorIR: An Abstraction for Automatic Tensorized Program Optimization Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programs
Week 5 (10/1)	Compilation and Optimization II Required Readings: TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Ansor: Generating High-Performance Tensor Programs for Deep Learning TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions Optional: Learning to Optimize Tensor Programs DNNFusion: accelerating deep neural networks execution with advanced operator fusion Pet: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections
Week 6 (10/8)	Training I: data vs. model parallelism Required Readings: PyTorch Distributed: Experiences on Accelerating Data Parallel Training Supporting Very Large Models using Automatic Dataflow Graph Partitioning Optional: PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel
Week 7 (10/15)	Training II: pipeline parallelism Required Readings: GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism PipeDream: Generalized Pipeline Parallelism for DNN Training Optional: Memory-Efficient Pipeline-Parallel DNN Training
Week 8 (10/22)	Training III: automatic parallelization and optimization Required Readings: Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models Optional: GSPMD: General and Scalable Parallelization for ML Computation Graphs Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization
Week 9 (10/29)	LLM Background Required Readings: Attention Is All You Need FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness FlashAttention-2:Faster Attention with Better Parallelism and Work Partitioning Optional: Flash Attention 3 Eight Things to Know about Large Language Models
Week 10 (11/5)	Training LLMs Required Readings: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Ring Attention with Blockwise Transformers for Near-Infinite Context Optional: Robust LLM Training Infrastructure at ByteDance MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Reducing Activation Recomputation in Large Transformer Models DeepSpeed Ulysses: System Optimizations for enabling training of extreme long sequence transformer
Week 11 (11/12)	Inference: Required readings: Orca: A Distributed Serving System for Transformer-Based Generative Models Efficient Memory Management for Large Language Model Serving with PagedAttention SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verification Optional: Fast Inference from Transformers via Speculative Decoding Accelerating large language model decoding with speculative sampling
Week 12 (11/19)	Quantization Required Readings: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale Pretraining Large Language Models with NVFP4 QLoRA: Efficient Finetuning of Quantized LLMs Optional: Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models
Week 13 (11/26)	No class (Legislative Friday)
Week 14 (12/3)	Sparsification (MoE and Sparse Attention) Required Readings: DeepSeek MoE MegaScale-Infer: Efficient Mixture-of-Experts Model Serving with Disaggregated Expert Parallelism Native Sparse Attention: Hardware-Aligned and Natively Trainable Sparse Attention Optional: Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot Optimal Brain Damage The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Week 15 (12/10)	Project Presentation