Second Brain
PyTorch internals
Initializing search
    • Home
      • 2D Convolutions
      • Ahead of Time (AOT) Compilation
      • Are less inductive biases better or worse?
      • Bit Palettization
      • Block Expansion
      • Convergence rate and Hessian spectra
      • Depthwise separable convolutions
      • Do Vision Foundation models exist?
      • Dynamic Resource Allocation in Vision
      • Effect of weight symmetries on training dynamics
      • Equivariance Initialization
      • Global Precedence Effect
      • Grokking
      • Group Axioms
      • Group direct product
      • Hardware specific structured pruning
      • Input dependent convolutions
      • K Means based Quantization
      • KV Cache
      • Linear Quantization
      • LoRa Adapter
      • Masked Image Modelling
      • Maximal pruning and functional recovery
      • Mean Attention Distance
      • Multiple global minima
      • Neural Network Quantization
      • Non translationally equivariant convolutions
      • Positive Logic Programs
      • Priors over Neural Network weights
      • PyTorch Functionalization
      • PyTorch Quantization for TensorRT
      • Rate Distortion and Spectral Analysis on Representations
      • Representation (Group Theory)
      • Residual stream
      • SK Centering
      • Should we separate semantics and pose?
        • 4M Massively Multimodal Masked Modeling
        • A Brief Review of Hypernetworks in Deep Learning
        • A ConvNet for the 2020s
        • A Cookbook of Self Supervised Learning
        • A Hierarchy of Graph Neural Networks Based on Learnable Local Features
        • A Mathematical Framework for Transformer Circuits
        • A general theory of correct, incorrect, and extrinsic equivariance
        • A survey of quantization methods for efficient neural network inference
        • AWQ Activation aware Weight Quantization for LLM Compression and Acceleration
        • AdaGlimpse Active Visual Exploration with Arbitrary Glimpse Position and Scale
        • Adapting Vision Foundation Models for Plant Phenotyping
        • An Image is Worth More Than 16x16 Patches Exploring Transformers on Individual Pixels
        • An Investigation into Neural Net Optimization via Hessian Eigenvalue Density
        • An image is worth 16x16 words Transformers for image recognition at scale
        • Apple Intelligence Foundation Language Models
        • Approximately equivariant networks for imperfectly symmetric dynamics
        • Approximation Generalization Trade offs under (Approximate) Group Equivariance
        • Are Hierarchical Visual Representations Hierarchical
        • Autoequivariant Network Search via Group Decomposition
        • Average entropy of Gaussian mixtures
        • Battle of the Backbones A Large Scale Comparison of Pretrained Models across Computer Vision Tasks
        • Beyond cls Exploring the true potential of Masked Image Modeling representations
        • Block Transformer Global to Local Language Modeling for Fast Inference
        • BoxeR Box Attention for 2D and 3D Transformers
        • Building on Efficient Foundations Effectively Training LLMs with Structured Feedforward Layers
        • Byte Latent Transformer Patches Scale Better Than Tokens
        • CKConv Continuous Kernel Convolution For Sequential Data
        • Certifying Adapters Enabling and Enhancing the Certification of Classifier Adversarial Robustness
        • Color Equivariant Convolutional Networks
        • Color Space Transformation Network
        • ConViT Improving Vision Transformers with Soft Convolutional Inductive Biases
        • Correlational Image Modeling for Self Supervised Visual Pre Training
        • Curiosity driven Exploration by Self supervised Prediction
        • DETRs Beat YOLOs on Real time Object Detection
        • DETRs with Collaborative Hybrid Assignments Training
        • DINOv2 Learning Robust Visual Features without Supervision
        • Deep Learning Book
        • Deep Learning is Not So Mysterious or Different
        • Deformable Convolutional Networks
        • Deformable DETR Deformable Transformers for End to End Object Detection
        • DeiT III Revenge of the ViT
        • Dense Contrastive Learning for Self Supervised Visual Pre Training
        • DenseNets Reloaded Paradigm Shift Beyond ResNets and ViTs
        • Discovering Symmetry Breaking in Physical Systems with Relaxed Group Convolution
        • DropPos Pre Training Vision Transformers by Reconstructing Dropped Positions
        • EA ViT Efficient Adaptation for Elastic Vision Transformer
        • EVA 02 A Visual Representation for Neon Genesis
        • Early Convolutions Help Transformers See Better
        • Efficient Equivariant Transfer Learning from Pretrained Models
        • Efficient Modulation for Vision Networks
        • EfficientViT SAM Accelerated Segment Anything Model Without Accuracy Loss
        • Emergent Equivariance in Deep Ensembles
        • Emerging Properties in Self Supervised Vision Transformers
        • End to End Object Detection with Transformers
        • Equi Tuning Group Equivariant Fine Tuning of Pretrained Models
        • Equivariance with Learned Canonicalization Functions
        • Equivariance aware architectural optimization of neural networks
        • Equivariant Representation Learning via Class Pose Decomposition
        • Exact Conversion of In Context Learning to Model Weights in Linearized Attention Transformers
        • Exploiting Redundancy Separable Group Convolutional Networks on Lie Groups
        • Exploring Plain Vision Transformer Backbones for Object Detection
        • Exploring Simple Siamese Representation Learning
        • FLSL Feature level Self supervised Learning
        • FNet Mixing Tokens with Fourier Transforms
        • Fast, Expressive SE(n) Equivariant Networks through Weight Sharing in Position Orientation Space
        • Fixing the train test resolution discrepancy
        • FlexTok Resampling Images into 1D Token Sequences of Flexible Length
        • FlexiViT One Model for All Patch Sizes
        • Fourier Transformer Fast Long Range Modeling by Removing Sequence Redundancy with FFT Operator
        • Franca Nested Matryoshka Clustering for Scalable Visual Representation Learning
        • From Image to Video An Empirical Study of Diffusion Representations
        • From Pixels to Components Eigenvector Masking for Visual Representation Learning
        • G SGD Optimizing ReLU Neural Networks in its Positively Scale Invariant Space
        • Grokked Transformers are Implicit Reasoners A Mechanistic Journey to the Edge of Generalization
        • Group Crosscoders for Mechanistic Analysis of Symmetry
        • Guillotine Regularization Why removing layers is needed to improve generalization in Self Supervised Learning
        • Harmonics of Learning Universal Fourier Features Emerge in Invariant Networks
        • HoPE A Novel Positional Encoding Without Long Term Decay for Enhanced Context Awareness and Extrapolation
        • How Does SimSiam Avoid Collapse Without Negative Samples A Unified Understanding with Self supervised Contrastive Learning
        • How JEPA Avoids Noisy Features The Implicit Bias of DeepLinear Self Distillation Networks
        • How do vision transformers work?
        • How to represent part whole hierarchies in a neural network
        • Hydra Bidirectional State Space Models Through Generalized Matrix Mixers
        • HydraViT Stacking Heads for a Scalable ViT
        • Hyperspherical Variational Auto Encoders
        • Improving Convergence and Generalization Using Parameter Symmetries
        • Improving Self Consistency in LLMs through Probabilistic Tokenization
        • In Search of Projectively Equivariant Networks
        • Interaction Asymmetry A General Principle for Learning Composable Abstractions
        • Into the Rabbit Hull From Task Relevant Concepts in DINO to Minkowski Geometry
        • Intuitive physics understanding emerges from self supervised pretraining on natural videos
        • Is ImageNet worth 1 video Learning strong image encoders from 1 long unlabelled video
        • KV Cache Steering for Inducing Reasoning in Small Language Models
        • Knowledge Transfer from Vision Foundation Models for Efficient Training of Small Task specific Models
        • LRP QViT Mixed Precision Vision Transformer Quantization via Layer wise Relevance Propagation
        • Layer by Layer Uncovering Hidden Representations in Language Models
        • Learned Gridification for Efficient Point Cloud Processing
        • Learning Continually by Spectral Regularization
        • Learning Partial Equivariances from Data
        • Learning Representations on the Unit Sphere Investigating Angular Gaussian and von Mises Fisher Distributions for Online Continual Learning
        • Learning Unseen Modality Interaction
        • Learning both Weights and Connections for Efficient Neural Networks
        • Learning from Streaming Video with Orthogonal Gradients
        • Learning to Learn without Forgetting using Attention
        • Learning with Unmasked Tokens Drives Stronger Vision Learners
        • LieRE Generalizing Rotary Position Encodings
        • Llama 2 Open Foundation and Fine Tuned Chat Models
        • LoRA Low Rank Adaptation of Large Language Models
        • LoRA vs Full Fine tuning An Illusion of Equivalence
        • Location Aware Self Supervised Transformers for Semantic Segmentation
        • Lost in Time A New Temporal Benchmark for VideoLLMs
        • Low Resource Vision Challenges for Foundation Models
        • Mamba Linear Time Sequence Modeling with Selective State Spaces
        • Masked Autoencoders Are Scalable Vision Learners
        • MatFormer Nested Transformer for Elastic Inference
        • Matryoshka Representation Learning
        • Memorization Through the Lens of Curvature of Loss Function Around Samples
        • Mixture of LoRa Experts
        • MoSiC Optimal Transport Motion Trajectory for Dense Self Supervised Learning
        • MobileCLIP Fast Image Text Models through Multi Modal Reinforced Training
        • MobileViT light weight, general purpose, and mobile friendly vision transformer
        • Model Compression in Practice Lessons Learned from Practitioners Creating On device Machine Learning Experiences
        • Near, far Patch ordering enhances vision foundation models' scene understanding
        • NeoBabel A Multilingual Open Tower for Visual Generation
        • Neural Mechanics Symmetry and Broken Conservation Laws in Deep Learning Dynamics
        • No Train, all Gain Self Supervised Gradients Improve Deep Frozen Representations Yuki M Asano
        • Occam's model Selecting simpler representations for better transferability estimation
        • On Good Practices for Task Specific Distillation of Large Pretrained Visual Models
        • On the Relationship between Self Attention and Convolutional Layers
        • On the Symmetries of Deep Learning Models and their Internal Representations
        • On the duality between contrastive and non contrastive self supervised learning
        • OpenELM An Efficient Language Model Family with Open source Training and Inference Framework
        • Optimal Brain Damage
        • Optimization Dynamics of Equivariant and Augmented Neural Networks
        • PART Self supervised Pretraining with Pairwise Relative Translations
        • PIN Positional Insert Unlocks Object Localisation Abilities in VLMs
        • Parameter Efficient Fine tuning of Self supervised ViTs without Catastrophic Forgetting
        • Parameter Efficient Fine Tuning for Pre Trained Vision Models A Survey
        • Patch Wise Self Supervised Visual Representation Learning A Fine Grained Approach
        • PatchRot A Self Supervised Technique for Training Vision Transformers
        • Position Prediction as an Effective Pretraining Strategy
        • Principal Components Enable A New Language of Images
        • Progress measures for grokking via mechanistic interpretability
        • Provably Strict Generalisation Benefit for Equivariant Models
        • ProxylessNAS Direct Neural Architecture Search on Target Task and Hardware
        • R MAE Regions Meet Masked Autoencoders
        • Rate–Distortion–Perception Trade Off in Information Theory, Generative Models, and Intelligent Communications
        • Refusal in Language Models Is Mediated by a Single Direction
        • Relaxed Octahedral Group Convolution for Learning Symmetry Breaking in 3D Physical Systems
        • Relaxing Equivariance Constraints with Non stationary Continuous Filters
        • Rethinking Lossy Compression The Rate Distortion Perception Tradeoff
        • Retrospective EIE Efficient Inference Engine onSparse and Compressed Neural Network
        • Revealing the Utilized Rank of Subspaces of Learning in Neural Networks
        • Revisiting Feature Prediction for Learning Visual Representations from Video
        • Rewrite the Stars
        • Robust Self Supervised Learning with Lie Groups
        • Rotary Position Embedding for Vision Transformer
        • Round and Round We Go! What makes Rotary Positional Encodings useful?
        • SAM CLIP Merging Vision Foundation Models towards Semantic and Spatial Understanding
        • Scaling (Down) CLIP A Comprehensive Analysis of Data, Architecture, and Training Strategies
        • Scaling 4D Representations
        • Scaling and Benchmarking Self Supervised Visual Representation Learning
        • Searching for Efficient Linear Layers over a Continuous Space of Structured Matrices
        • Segment Anything
        • Self Guided Diffusion Models
        • Self Supervised Detection of Perfect and Partial Input Dependent Symmetries
        • Self Supervised Learning from Images with a Joint Embedding Predictive Architecture
        • Self Supervised Learning of Object Parts for Semantic Segmentation
        • Self supervised learning of Split Invariant Equivariant representations
        • Self supervised learning of intertwined content and positional features for object detection
        • SimPLR A Simple and Plain Transformer for Scaling Efficient Object Detection and Segmentation
        • Simplifying DINO via Coding Rate Regularization
        • Simultaneous linear connectivity of neural networks modulo permutation
        • Slicing Vision Transformer for Flexible Inference
        • Stand Alone Self Attention in Vision Models
        • Stochastic positional embeddings improve masked image modeling
        • Structuring Representation Geometry with Rotationally Equivariant Contrastive Learning
        • Surgical Fine Tuning Improves Adaptation to Distribution Shifts
        • Surgical DINO Adapter Learning of Foundation Models for Depth Estimation in Endoscopic Surgery
        • Symmetries in Overparametrized Neural Networks A Mean Field View
        • TRecViT A Recurrent Video Transformer
        • Talaria Interactively Optimizing Machine Learning Models for Efficient Inference
        • The Empirical Impact of Neural Parameter Symmetries, or Lack Thereof
        • The Forward Forward Algorithm Some Preliminary Investigations
        • The Geometry of Categorical and Hierarchical Concepts in Large Language Models
        • The Hidden Uniform Cluster Prior in Self Supervised Learning
        • The Lie derivative for measuring learned equivariance
        • The Unreasonable Ineffectiveness of the Deeper Layers
        • Three things everyone should know about Vision Transformers
        • TiC CLIP Continual Training of CLIP models
        • To Compress or Not to Compress Self Supervised Learning and Information Theory A Review
        • Toward a Geometrical Understanding of Self supervised Contrastive Learning
        • Towards Understanding the Spectral Bias of Deep Learning
        • Towards a Definition of Disentangled Representations
        • Training quantized nets A deeper understanding
        • Understanding Deep Learning Chapter 10
        • Understanding Deep Learning Chapter 20
        • Understanding symmetries in deep networks
        • Unlocking Slot Attention by Changing Optimal Transport Costs
        • Unsupervised Learning of Visual Features by Contrasting Cluster Assignments
        • Unsupervised Visual Representation Learning by Context Prediction
        • Using Degeneracy in the Loss Landscape for Mechanistic Interpretability
        • VICRegL Self Supervised Learning of Local Visual Features
        • Variance Covariance Regularization Enforces Pairwise Independence in Self Supervised Representations
        • ViDT An Efficient and Effective Fully Transformer based Object Detector
        • ViTAR Vision Transformer with Any Resolution
        • Vision Mamba Efficient Visual Representation Learning with Bidirectional State Space Model
        • Vision Transformer with Deformable Attention
        • Vision Transformers Need Registers
        • What Do Self Supervised Vision Transformers Learn?
        • nGPT Normalized Transformer with Representation Learning on the Hypersphere
        • Adrien Bardes
        • Albert Gu
        • Alex Flinth
        • Alex Vitvitskyi
        • Alexander Kirillov
        • Alexey Dosovitskiy
        • Amir Zamir
        • Ananya Kumar
        • Andreas Loukas
        • Andreas Savakis
        • Andrew Gordon Wilson
        • Andrew Zisserman
        • Angela Fan
        • Annie S. Chen
        • Antonio Orvieto
        • Ardavan Pedram
        • Armand Joulin
        • Attila Lengyel
        • Binxu Wang
        • Boris Ginsburg
        • Boshi Wang
        • Byeongho Heo
        • Caglar Gulcehre
        • Carmen Amo Alonso
        • Cees G. M. Snoek
        • Chelsea Finn
        • Cheng Ping Hsieh
        • Chong Wang
        • Christopher Olah
        • Christos Perivolaropoulos
        • Chuhan Zhang
        • Cordelia Schmid
        • Daniel M. Roy
        • Daniel Ulbricht
        • Daniel Zoran
        • David M. Knigge
        • David W. Romero
        • David W. Zhang
        • Dheeraj Varghese
        • Diane Larlus
        • Dilara Gokay
        • Dima Damen
        • Donghyun Kim
        • Dongyoon Han
        • Duy Kien Nguyen
        • Edward J. Hu
        • Edward Z. Yang
        • Efstratios Gavves
        • Eric Mintun
        • Erik J. Bekkers
        • Eshan Verma
        • Fahim Tajwar
        • Fartash Faghri
        • Federico Barbero
        • Florian Bordes
        • Francisco Massa
        • Fred Hohman
        • Furu Wei
        • Gabriel Synnaeve
        • Geoffrey Hinton
        • Gertjan Burghouts
        • Gertjan J. Burghouts
        • Gintare Karolina Dziugaite
        • Giovanni Chierchia
        • Grégoire Mialon
        • Hadi Pouransari
        • Han Cai
        • Hanzi Mao
        • Haoxiang Wang
        • Hervé Jegou
        • Huaxiu Yao
        • Hugo Touvron
        • Huizi Mao
        • Ilya Loshchilov
        • Ioana Simion
        • Isha Garg
        • Ishan Misra
        • Jan E. Gerken
        • Javier Maass Martinez
        • Jean Ponce
        • Jean Baptiste Cordonnier
        • Jean François Bercher
        • Jeff Pool
        • Jesse Cai
        • Jing Pu
        • Joaquin Fontbona
        • Joaquin Vanschoren
        • John Denker
        • John Tran
        • Joseph Heyward
        • João Carreira
        • Julien Mairal
        • Juliette Marrie
        • Kaiming He
        • Kamyar Azizzadenesheli
        • Kaushik Roy
        • Laurent Najman
        • Lawrence Chan
        • Lucas Beyer
        • Lucia Donatelli
        • Lucius Bushnaq
        • Maciej Wołczyk
        • Mahdi Soltanolkotabi
        • Mahmoud Assran
        • Marc Finzi
        • Mark A. Horowitz
        • Mark Ibrahim
        • Martin Jaggi
        • Martin R. Oswald
        • Marzieh Fadaee
        • Mathilde Caron
        • Max Belistky
        • Maxime Oquab
        • Mehrdad Farajtabar
        • Micah Goldblum
        • Michael Arbel
        • Michael Dorkenwald
        • Mohammad Mahdi Derakhshani
        • Mohammad Rastegari
        • Mohammadreza Salehi
        • Namuk Park
        • Navin Ranjan
        • Neel Nanda
        • Neil Houlsby
        • Nicolas Carion
        • Nicolas Michel
        • Nicolas Usunier
        • Olivier J. Hénaff
        • Oncel Tuzel
        • Pascal Vincent
        • Patrick Forré
        • Pavan Kumar Anasosalu Vasu
        • Percy Liang
        • Petar Veličković
        • Phillip Isola
        • Piotr Bojanowski
        • Piotr Dollár
        • Quentin Garrido
        • Randall Balestriero
        • Ravid Shwartz Ziv
        • Raviteja Vemulapalli
        • Razvan Pascanu
        • Robin Walters
        • Romain Cosentino
        • Romain Negrel
        • Rose Yu
        • Ross Girshick
        • Rui Wang
        • Ruoming Pang
        • Sachin Mehta
        • Saining Xie
        • Sangdoo Yun
        • Sanghyuk Chun
        • Sara Solla
        • Sergey Zagoruyko
        • Shaohan Huang
        • Shashanka Venkataramanan
        • Sibylle Hess
        • Simeng Sun
        • Simon J.D. Prince
        • Skander Moalla
        • Soham De
        • Song Han
        • Song Park
        • Songkuk Kim
        • Sonia Joseph
        • Sourya Basu
        • Stéphane d'Ascoli
        • Sukjun Hwang
        • Taekyung Kim
        • Tengda Han
        • Tete Xiao
        • Thomas Fel
        • Thomas Kipf
        • Tim R. Davidson
        • Tom Goldstein
        • Tom Gunter
        • Tom Lieberum
        • Tongzhou Wang
        • Vaibhav Aggarwal
        • Valentinos Pariza
        • Viorica Pătrăucean
        • Walter Simoncini
        • Wei Lu
        • Wieland Brendel
        • William J. Dally
        • Wonjae Kim
        • Xiang Yue
        • Xingyu Liu
        • Xinlei Chen
        • Xiuying Wei
        • Xu Ma
        • Xun Wu
        • Yanghao Li
        • Yann LeCun
        • Yelong Shen
        • Yi Ma
        • Yoonho Lee
        • Yubei Chen
        • Yuki M. Asano
        • Zeyuan Allen Zhu
        • Zhuoyang Zhang
        • Ziaoyi Zhang
        • Zirui Wang
        • Ziyang Wu
        • Anthropic
        • Apollo Research
        • Apple
        • CLAIRE
        • Carnegie Mellon University
        • Chalmers University of Technology
        • Cohere
        • EPFL
        • FAIR
        • FunAI Lab
        • Google DeepMind
        • Google
        • HKU
        • IBM Research
        • INRIA
        • MIT
        • McGill University
        • Meta
        • Microsoft
        • Mila Quebec AI Institute
        • NVIDIA
        • Naver AI Lab
        • Naver Cloud AI
        • Naver Labs Europe
        • New York University
        • Northeastern University
        • OpenAI
        • Princeton University
        • PyTorch
        • QUVA Lab
        • Rochester Institute of Technology
        • Stanford
        • TNO
        • TU Delft
        • Tsinghua University
        • UC Berkeley
        • UC San Diego
        • UC Santa Barbara
        • UCLA
        • Univ Gustave Eiffel
        • University of Amsterdam
        • University of Chile
        • University of Illinois at Urbana Champaign
        • University of Oxford
        • University of Technology Nuremberg
        • VIS Lab
        • Vector Institute
        • Vrije Universiteit Amsterdam
        • Yonsei University
        • EPFL CS439 Optimization for Machine Learning
        • GPU mode Sparsity
        • Introducing Apple’s On Device and Server Foundation Models
        • Introduction to Quantization on PyTorch
        • Let's talk about the Python Dispatcher
        • MIT 65940 TinyML and Efficient Deep Learning Computing
        • Optimizing Vision Transformer Model for Deployment
        • PyTorch ExecuTorch Export IR Specification
        • PyTorch ExecuTorch How ExecuTorch works?
        • PyTorch ExecuTorch Quantization Overview
        • PyTorch Functionalization in PyTorch Everything you need to know
        • PyTorch PyTorch 2 Export Post Training Quantization
        • PyTorch Quantization
        • PyTorch Compilers What makes PyTorch beloved makes it hard to compile
        • PyTorch Conference 2024 Fast Sparse Vision Transformers with minimal accuracy loss
        • PyTorch Conference 2024 What’s new in torch.export?
        • PyTorch Conference 2024
        • PyTorch Eager Mode Quantization TensorRT Acceleration
        • PyTorch internals
        • Quantized Transfer Learning for Computer Vision Tutorial
        • Reinforcement Learning An Introduction Chapter 10
        • Reinforcement Learning An Introduction Chapter 11
        • Reinforcement Learning An Introduction Chapter 13
        • Reinforcement Learning An Introduction Chapter 16
        • Reinforcement Learning An Introduction Chapter 2
        • Reinforcement Learning An Introduction Chapter 3
        • Reinforcement Learning An Introduction Chapter 4
        • Reinforcement Learning An Introduction Chapter 5
        • Reinforcement Learning An Introduction Chapter 6
        • Reinforcement Learning An Introduction Chapter 7
        • Reinforcement Learning An Introduction Chapter 8
        • Reinforcement Learning An Introduction Chapter 9
        • Reinforcement Learning An Introduction
        • TinyML and Efficient Deep Learning Computing Lecture 12
        • TinyML and Efficient Deep Learning Computing Lecture 3
        • TinyML and Efficient Deep Learning Computing Lecture 5
        • TinyML and Efficient Deep Learning Computing Lecture 6
        • TinyML and Efficient Deep Learning Computing
        • Tweet Stable Diffusion XL on iPhone with Core ML!
    blog

    PyTorch internals

    Properties
    authors Edward Z. Yang
    year 2019
    url http://blog.ezyang.com/2019/05/pytorch-internals/

    Depending on tensor metadata (if it's CUDA, or sparse, etc) it's dispatched to different implementations ()
    Pasted image 20240928131008.png{ width="500" }

    2024-10-01
    Made with Material for MkDocs