Big Data and Machine Learning Systems

Date	Topic
Week 1 (9/3)	Introduction, the Model Optional Readings: Deep Learning Book Chap 6.1-6.4 How to read a paper
Week 2 (9/10)	The Frameworks Required Readings: TensorFlow: A System for Large-Scale Machine Learning PyTorch: An Imperative Style, High-Performance Deep Learning Library
Week 3 (9/17)	The Hardware Required Readings: An even easier introduction to CUDA Optional: Cuda C++ Programming Guide(Sec 2 and 3)
Week 4 (9/24)	Compilation and Optimization I Required Readings: PyTorch 2: Faster Machine Learning Through Dynamic Python Bytecode Transformation and Graph Compilation Triton: An Intermediate Language and Compiler for Tiled Neural Network Computations Optional: TVM: An Automated End-to-End Optimizing Compiler for Deep Learning Ansor: Generating High-Performance Tensor Programs for Deep Learning TensorIR: An Abstraction for Automatic Tensorized Program Optimization Hidet: Task-Mapping Programming Paradigm for Deep Learning Tensor Programs
Week 5 (10/1)	Compilation and Optimization II Required Readings: DNNFusion: accelerating deep neural networks execution with advanced operator fusion TASO: Optimizing Deep Learning Computation with Automatic Generation of Graph Substitutions Optional: Pet: Optimizing Tensor Programs with Partially Equivalent Transformations and Automated Corrections
Week 6 (10/8)	Training I: data vs. model parallelism Required Readings: PyTorch Distributed: Experiences on Accelerating Data Parallel Training Supporting Very Large Models using Automatic Dataflow Graph Partitioning Optional: One weird trick for parallelizing convolutional neural networks
Week 7 (10/15)	Training II: pipeline parallelism Required Readings: GPipe: Efficient Training of Giant Neural Networks using Pipeline Parallelism PipeDream: Fast and Efficient Pipeline Parallel DNN Training Optional: Memory-Efficient Pipeline-Parallel DNN Training
Week 8 (10/22)	Training III: automatic parallelization Required Readings: Alpa: Automating Inter- and Intra-Operator Parallelism for Distributed Deep Learning Unity: Accelerating DNN Training Through Joint Optimization of Algebraic Transformations and Parallelization Optional: GSPMD: General and Scalable Parallelization for ML Computation Graphs
Week 9 (10/29)	LLM Background Required Readings: Attention Is All You Need FlashAttention-2:Faster Attention with Better Parallelism and Work Partitioning Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism Optional: Eight Things to Know about Large Language Models
Week 10 (11/5)	Training LLMs I Required Readings: Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel MegaScale: Scaling Large Language Model Training to More Than 10,000 GPUs Optional: ZeRO: Memory Optimizations Toward Training Trillion Parameter Models Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM Reducing Activation Recomputation in Large Transformer Models Overlap Communication with Dependent Computation via Decomposition in Large Deep Learning Models
Week 11 (11/12)	Inference: Required readings: Orca: A Distributed Serving System for Transformer-Based Generative Models Efficient Memory Management for Large Language Model Serving with PagedAttention Optional: Fast Inference from Transformers via Speculative Decoding Accelerating large language model decoding with speculative sampling
Week 12 (11/19)	Quantization Required Readings: LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Models QLoRA: Efficient Finetuning of Quantized LLMs Optional: Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers Atom: Low-bit Quantization for Efficient and Accurate LLM Serving
Week 13 (11/26)	No class (Legislative Friday)
Week 14 (12/3)	Sparsification Required Readings: Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding SparseGPT: Massive Language Models Can be Accurately Pruned in One-Shot Optional: Optimal Brain Damage The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks
Week 15 (12/10)	Project Presentation