Dynamic Resource Allocation in Vision

At the embedding level
- 1D tokens (truncating patch suffix (at the sequence dim))
- Principal Components Enable A New Language of Images
- FlexTok - Resampling Images into 1D Token Sequences of Flexible Length
- 2D tokens (truncating patch suffixes (at the embed dim))
- Matryoshka Representation Learning
- Franca - Nested Matryoshka Clustering for Scalable Visual Representation Learning

At the network level (embed_dim, depth, etc)
- EA-ViT - Efficient Adaptation for Elastic Vision Transformer
- MatFormer - Nested Transformer for Elastic Inference
- HydraViT - Stacking Heads for a Scalable ViT
- Slicing Vision Transformer for Flexible Inference

At the network level we also have patch pruning and merging methods