Главная
Study mode:
on
1
Intro
2
Background and previous work
3
Self-attention
4
Absolute Position Encoding and Relative Position Encoding (RPE)
5
RPE in Transformer-XL
6
Bias and Contextual Mode
7
A Piecewise Index Function
8
2D Relative Position Calculation
9
Experiments
10
Implementation details
11
Directed vs. Undirected Bias vs. Contextual
12
Shared v.s. Unshared
13
Piecewise v.s. Clip
14
Number of buckets
15
Component-wise analysis
16
Complexity Analysis
17
Visualization
18
Conclusion
Description:
Explore the advancements in Relative Position Encoding (RPE) for Vision Transformers in this 32-minute lecture from the University of Central Florida's CAP6412 2022 series. Delve into the background of self-attention mechanisms and position encoding techniques, focusing on the evolution from absolute to relative position encoding. Examine the improvements made to RPE, including the introduction of bias and contextual modes, a piecewise index function, and 2D relative position calculation. Analyze experimental results comparing directed vs. undirected approaches, shared vs. unshared implementations, and the impact of different numbers of buckets. Gain insights into component-wise analysis, complexity considerations, and visualizations that demonstrate the effectiveness of these enhancements. Conclude with a comprehensive understanding of how these innovations contribute to the performance of Vision Transformers in various computer vision tasks.

Rethinking and Improving Relative Position Encoding for Vision Transformer - Lecture 23

University of Central Florida
Add to list
0:00 / 0:00