Gradient Surgery for Multi-Task Learning
Explore PCGrad, a technique addressing gradient conflicts in multi-task learning, enhancing efficiency and performance in supervised and reinforcement learning scenarios.
Longformer - The Long-Document Transformer
Explore Longformer's innovative attention mechanism for processing lengthy documents, combining local windowed and global attention to overcome Transformer limitations in handling long sequences.
Backpropagation and the Brain
Explore a biologically plausible variant of backpropagation and its potential role in brain learning, discussing feedback connections and error signal approximation in cortical networks.
Shortcut Learning in Deep Neural Networks
Explore deep neural networks' shortcut learning, its impact on AI generalization, and strategies to improve model robustness and real-world applicability.
Feature Visualization and the OpenAI Microscope
Explore neural network visualizations using OpenAI Microscope, understanding feature extraction in ImageNet classifiers and techniques for visualizing internal model workings.
Evolving Normalization-Activation Layers
Explore evolutionary search for optimal normalization-activation layers in neural networks, uncovering novel designs that outperform traditional methods across various image-related tasks.
Dream to Control - Learning Behaviors by Latent Imagination
Explore DeepMind's Dreamer, an innovative RL agent that learns complex behaviors through latent imagination, surpassing existing approaches in visual control tasks.
Axial Attention & MetNet - A Neural Weather Model for Precipitation Forecasting
Explore a neural network model for precise precipitation forecasting using axial attention, offering high-resolution predictions up to 8 hours ahead with improved accuracy over traditional methods.
Deep Learning for Symbolic Mathematics
Explore neural networks solving complex mathematical problems like symbolic integration and differential equations, outperforming traditional Computer Algebra Systems.
Turing-NLG, DeepSpeed and the ZeRO Optimizer
Explore Microsoft's 17-billion parameter language model, ZeRO optimizer, and DeepSpeed, enabling efficient model and data parallelism for state-of-the-art natural language processing breakthroughs.
Accelerating Deep Learning by Focusing on the Biggest Losers
Explore a technique to speed up neural network training by prioritizing high-loss examples, reducing backpropagation steps, and achieving faster convergence in deep learning models.
Blockwise Parallel Decoding for Deep Autoregressive Models
Explores a novel blockwise parallel decoding method for deep autoregressive models, enabling faster generation in tasks like machine translation and image super-resolution without sacrificing quality.
Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
Explore how Batch Normalization accelerates deep network training by reducing internal covariate shift, enabling higher learning rates and improved model performance.
BERT- Pre-training of Deep Bidirectional Transformers for Language Understanding
Explore BERT's innovative approach to language representation, its bidirectional pre-training method, and its state-of-the-art performance across various natural language processing tasks.
Attention Is All You Need
Explore the groundbreaking Transformer architecture, revolutionizing sequence transduction with attention mechanisms. Learn its superior performance in machine translation and generalization to other NLP tasks.
Deep Image Reconstruction from Human Brain Activity
Explores innovative technique for reconstructing visual imagery from brain activity using fMRI and deep neural networks, demonstrating potential for "mind-reading" technology and insights into neural representations.
Weight Standardization
Explores Weight Standardization technique for neural networks, improving performance in micro-batch training and outperforming other normalization methods when combined with Group Normalization.
SinGAN - Learning a Generative Model from a Single Natural Image
Unconditional generative model learns from a single image to create diverse, high-quality variations. Uses pyramid of GANs to capture patch distribution at multiple scales, enabling flexible image generation and manipulation.
Processing Megapixel Images with Deep Attention-Sampling Models
Explore a novel deep learning approach for efficiently processing high-resolution images by selectively focusing on informative regions, reducing computational costs while maintaining accuracy.
Typical Decoding for Natural Language Generation - Get More Human-Like Outputs From Language Models
Explore typical sampling, a new decoding method for language models that balances high-probability and high-information outputs, resulting in more diverse and interesting generated text.
One Model for All the Tasks - BLIP - Author Interview
Interview exploring BLIP, a versatile vision-language model that excels in both understanding and generation tasks. Discusses data bootstrapping, architecture evolution, and future directions in multi-modal AI research.