Prior Fitted Networks - Understanding and applying tabular foundation models
Prior fitted networks (PFNs) such as TabPFN and TabICL are transformers that learn in context: they are pretrained on millions of synthetic tables drawn from a prior over data-generating processes, and then make predictions on a new dataset in a single forward pass, with no task-specific training. This makes them attractive wherever labelled data is scarce, and raises two questions that our work addresses: what does the synthetic prior actually teach these models, and where do they transfer?
From Synthetic Priors to Model Behavior: Structural Coverage in Tabular Foundation Models (to appear at ATTRIB, the 3rd Workshop on Attributing Model Behavior at Scale, NeurIPS 2026) studies how well synthetic pretraining priors support the tasks tabular foundation models are evaluated on. We reconstruct the synthetic generators of four models and describe the tasks they produce, alongside datasets from two common benchmarks, using shared structural descriptors. Measuring structural coverage and density in this space, we find large differences between priors, and that stronger support for benchmark tasks is generally associated with better model performance — making structural coverage a useful diagnostic for synthetic priors.
Can Tabular In-Context Learners Generalize to Biomolecular Property Prediction? (to appear at the Asian Conference on Machine Learning, Melbourne) asks whether models pretrained on synthetic tables from random causal graphs can predict protein and molecular properties from very few labels. Paired with strong pretrained representations such as ESM Cambrian, they are competitive with or better than the state of the art on ProteinGym and outperform task-specific regressors on an esterase catalytic activity dataset. On small-molecule tasks results are mixed, and the choice of representation matters most.
