Explore the fundamental principles and practical applications of scaling and distributing deep neural networks in this comprehensive symposium talk from HOTI's Hot Interconnects conference. Delve into both training and inference aspects of distributed neural networks, examining key architectural considerations and implementation strategies. Learn how to effectively scale deep learning systems across distributed computing environments while understanding the underlying theoretical frameworks that enable such scalability. Chaired by Amanda Bienz from UNM, this technical session provides valuable insights into the challenges and solutions for building large-scale neural network systems.