ProbXiv
Explorersign in
Problem archiveProblem record

Statement

Are self-supervised models doing kernel PCA? I have introduced this framework and the foundations of the most recent algorith- mic developments, as well as discussed the potential for representation learning in more detail. This talk will have two parts. In the first part, I will briefly introduce self- supervised pretraining commonly used in foundation models for vision and tabular data. I will also introduce the key question related to statistical generalisation in foundation models: How do we guarantee statistical generalisation for different downstream prediction tasks given that the model is pre-trained with large amount of unlabelled (augmented) data? I will conclude this part with some initial ideas and results assuming that the foundation model uses a kernel-based encoder to learn representations via self-supervised pre-training. In the second part part, I will focus on the equivalence between self-supervised neural networks and kernel principal component analysis (PCA). This equivalence is based on two ideas: (i) optimal solution of self-supervised kernel models can be computed as a spectral embedding [1], and (ii) infinitely wide neural networks are equivalent to kernel models, characterised by the neural tangent kernel (NTK). I conclude with our recent results on the convergence to the NTK under self- supervised training [2, 3].

Record

Comments

Nothing has been published against this problem yet, and nothing has been said about it.

Sign in with an institutional address to take part in the discussion. Reading every thread stays open to everyone.

Sign in

Solve with an agent

Open the statement in a chat, with the problem and the ground rules already written into the prompt.

This opens a third-party site. Nothing is posted back to ProbXiv and nothing you write there is recorded here — what a model gives you is an attempt, which a person still has to check.