Using Algorithms to Understand Transformers and Using Transformers to Understand Algorithms
Explore algorithmic tools and optimization theory to uncover Transformer mechanisms for computational tasks, and discover how Transformers design data structures for tasks like nearest neighbor search.
Towards Understanding Modern Alchemy - Transformers as a Computational Model
Explore in-context learning of formal languages, comparing Transformers to other models and examining specialized "n-gram heads" for improved performance in language modeling.
Associative Memories as a Building Block in Transformers
Explore associative memories in Transformers, their role in storing knowledge, and how gradients and over-parameterization affect their learning and capacity.
The Emergence of Clusters in Self-Attention Dynamics
Explore the formation of clusters in self-attention dynamics, examining their impact on transformer models and computational processes.
Interpretability Agents: Automating and Scaling Model Interpretation
Explore automated interpretability agents for scaling model interpretation, enabling efficient analysis of AI systems through hypothesis-driven experimentation and modular design.
Understanding and Improving Efficient Language Models
Explore efficient language models, focusing on associative recall's impact on performance. Learn about new hardware-efficient architectures expanding quality-efficiency frontiers in machine learning.
Transformer Expressivity and Formal Logic
Explore the relationship between transformers and formal logic, uncovering their abilities, limitations, and implications for computational models.
Using Recurrence to Achieve Weak to Strong Generalization
Explore recurrent architectures for weak-to-strong generalization in reasoning models, enhancing problem-solving capabilities beyond training data.
Transformers as a Computational Model - What Was Revolutionized by the "Transformer Revolution?"
Explore the paradigm shift in machine learning brought by transformers, examining new research questions and recent findings in AI development.
Exact Solutions to the Geometric Dynamics of Signal Propagation Through Transformers
Explore exact solutions to geometric dynamics of signal propagation in transformers, predicting their behavior as computational models.
Language Generation in the Limit - Transformers as a Computational Model
Explore language generation using basic specifications, contrasting with language learning models. Discover non-trivial guarantees for generating valid new strings from unknown languages.
Computational Benefits and Limitations of Transformers and State-Space Models
Explore mechanisms enabling retrieval, copying, and length generalization in language models. Compare Transformers' excellence with LSTM and state-space models' limitations in sequence modeling tasks.
Learning to Reason with Large Language Models
Explore OpenAI's o1 model: an LLM trained via reinforcement learning for improved reasoning in math, programming, and science tasks.
Iterated Models: Expressive Power, Learning, and Chain of Thought
Explore sequence-to-sequence models, focusing on transformers, their computational power, and learning complexities. Gain insights into iterated functions and chain of thought.
The Parallelism Tradeoff: Understanding Transformer Expressivity Through Circuit Complexity
Explore transformer neural nets' computational power and limitations through circuit complexity analysis, revealing insights into parallelism tradeoffs in large-scale model architectures.
Limitations of Attention Mechanism in Transformers - Implications for Generalization and Optimization
Explore Transformer's reasoning capabilities and limitations in sequential tasks, examining optimization challenges and out-of-distribution generalization issues compared to RNNs.
Transformers, Parallel Computation, and Logarithmic Depth
Explore the computational power of transformers, their connection to parallel processing, and efficiency in solving complex tasks with logarithmic depth.
Transformers as a Computational Model
Explore Transformers as a computational model with Misha Belkin, delving into their architecture, capabilities, and implications for AI research and applications.
Accelerating Transformers with Hugging Face Optimum and Infinity
Explore techniques to accelerate Transformer models using Hugging Face's Optimum library and Infinity solution for millisecond-scale latencies in production environments.
TransGNN: Harnessing the Collaborative Power of Transformers and Graph Neural Networks for Recommender Systems
Explore TransGNN's fusion of Transformers and Graph Neural Networks for enhanced recommender systems, improving collaborative filtering and user-item interactions.
Geometric Representations of Far Future Predictions in Deep Neural Networks - Next-Token Training
Explore universal geometric representations of far-future beliefs in deep neural networks trained on next-token prediction, uncovering fractal and quantum-like structures embedded in network activations.