Efficient Equivariant Transfer Learning from Pretrained Models

Properties
authors	Sourya Basu

Builds on top of Equi-Tuning - Group Equivariant Fine-Tuning of Pretrained Models and Equivariance with Learned Canonicalization Functions

Hypothesis

Pretrained models provide better quality features for certain transformations than others and simply averaging them is bad.

Main idea

Lambda-Equitune: Weighted average with learned weights, \(\lambda\).

\[ M_G^\lambda(x) = \frac{1}{\sum_{g \in G} \lambda(gx)} \sum_{g \in G} \lambda(gx) g^{-1} M(gx) \]