Let's Make Block Coordinate Descent Go Fast
Explore block coordinate descent methods for large-scale optimization problems in this 39-minute lecture by Mark Schmidt from the University of British Columbia. Delve into the advantages of coordinate descent, suitable problem types, and various block selection rules including Gauss-Southwell and greedy approaches. Examine Newton-steps, quadratic norms, and matrix updates, comparing them to traditional methods. Analyze experimental results for multi-class logistic regression and sparse quadratic problems. Investigate optimization with bound constraints, manifold identification properties, and superlinear convergence. Gain insights into message-passing techniques for sparse quadratics and understand the key takeaways for implementing fast iterative methods in optimization.