Fine-tuning an ICL Tabular Foundation Model for Multi-Omics Prediction

Yinuo Cheng

Department of Biostatistics, Harvard T.H. Chan School of Public Health

The Ragon Institute of Mass General Brigham, MIT, and Harvard

The Broad Institute of MIT and Harvard

1. Overview

Scientific rationale & importance: TabPFN is a tabular foundation model that leverages in-context learning (ICL) and has shown strong performance and data efficiency on general tabular tasks. Omics data (bulk RNA-seq, scRNA-seq, spatial transcriptomics, proteomics, epigenomics) exhibit distinct statistical structure: extreme feature dimensionality, strong correlation blocks, sparsity/zero-inflation (single-cell), and pervasive batch effects. This project tests whether domain- and task-adaptive fine-tuning of TabPFN on omics distributions yields improved generalization and small-sample robustness for key downstream tasks (disease risk prediction; cell type classification), compared with standard baselines (ElasticNet/GBDT/MLP). The outcome will be an open benchmark and finetuned weights enabling broader omics adoption of ICL tabular models.

Funding/facilities/ethics approvals: Analyses will be conducted using existing institutional HPC/GPU resources and/or cloud GPU instances as needed. Primary analyses will use public, de-identified datasets and do not involve recruiting human participants. For any controlled-access data, we will follow the dataset’s data use agreements and institutional requirements.

Estimated timeline (6 months): Month 1: data curation + standardized preprocessing + baseline models. Month 2: zero-shot/no-finetune TabPFN benchmarking. Months 3–4: fine-tuning (task-adaptive + optional domain-adaptive) + ablations. Month 5: cross-dataset generalization evaluation + error analysis. Month 6: finalize preregistered report, release code/models/derived outputs, prepare preprint.

Data sharing commitment: We will release preprocessing and evaluation pipelines, model configurations and training scripts, finetuned weights (when licensing permits), and derived outputs (embeddings, predictions, splits, metrics) in an open-access repository. Raw data will be shared via original public repositories or via dataset-specific access procedures.

2. Introduction

Foundation models and ICL[1] have reshaped modeling in language and vision, and are increasingly applied to tabular data. TabPFN is a prominent example: learning a prior over tabular problems to enable strong performance with limited data via ICL and/or adaptation [2,3]. Omics prediction has mature baselines (e.g., single-cell omics integration and clustering [4], tree-based models[5,6], and deep learning approaches[7]), yet faces distribution shifts across cohorts/platforms and limited labeled data for many tasks. A principled fine-tuning strategy for tabular foundation models on omics could improve robustness and data efficiency while simplifying downstream workflows.

Research questions

RQ1: Does fine-tuning TabPFN on omics distributions improve cross-dataset generalization for downstream task (e.g. cell type classification, disease biomarker, etc.) relative to (a) no-finetune TabPFN and (b) standard ML baselines?

RQ2: Which strategy is most effective under omics constraints: task-adaptive fine-tuning or domain-adaptive fine-tuning?

RQ3: What data regimes (sample size, feature selection) yield the largest gains?

Hypotheses

H1: Finetuned TabPFN reduces cross-dataset cell type classification and disease prediction error (MAE) vs. no-finetune TabPFN and ElasticNet/GBDT baselines.

H2: Under limited labeled samples (e.g., 50–500), finetuned TabPFN outperforms baselines by a larger margin than in large-sample regimes.

H3: Performance degradation under distribution shift is smaller for finetuned TabPFN than for baselines.

Impact statement

If successful, this project provides a validated and reusable omics fine-tuning recipe for tabular foundation models, enabling faster hypothesis testing and improving robustness in common predictive tasks. If unsuccessful, the project will still produce an open benchmark diagnosing why ICL tabular priors fail on omics (e.g., dimensionality, sparsity, batch effects), guiding next-generation model designs and preprocessing standards.

3. Methods

Participants/Sample

This is a computational reanalysis; “participants” are samples from existing datasets.

Target sample and inclusion/exclusion criteria

Task A: Multi-omics disease risk prediction

Inclusion: Human multi omics data with disease status/risk metadata; adult samples preferred.

Exclusion: Missing/ambiguous disease status or age; low-quality samples per dataset QC; extreme technical artifacts; duplicated samples.

Target size: ~3,000–10,000 samples for each kind of omics data, aggregated across multiple public cohorts (dependent on availability and QC).

Primary evaluation: cross-dataset (or cross-platform) splits.

Task B: Cell type classification

Inclusion: Public scRNA datasets with cell-type annotations (e.g., PBMC and/or atlas-style datasets); cells passing standard QC.

Exclusion: Cells with extreme QC metrics or ambiguous labels; incompatible label taxonomies unless harmonization is feasible.

Target size: ~100,000–500,000 cells total; focus is cross-dataset transfer rather than maximum scale.

Task C: Perturbation classification

Inclusion: Public perturbation omics datasets with perturbation labels, such as CRISPR Perturb-seq, drug or cytokine stimulation experiments, and multi-modal single-cell datasets (e.g., CITE-seq) where perturbation identity is available. Samples/cells must pass standard QC.

Exclusion: Missing/ambiguous perturbation labels; extreme QC outliers; duplicated samples/cells; perturbations with too few examples for evaluation (e.g., <50 cells per class unless merged).

Target size: ~100,000–500,000 cells total across multiple studies.

Sample size justification / power analysis

We will quantify uncertainty in performance differences using bootstrap over test samples to obtain 95% confidence intervals for ΔMAE (continuous labels) and Δmacro-F1 (cell type/perturbation/discrete label prediction).

Materials and Procedures

Data preprocessing

Normalize and QC using standard array pipelines (dataset-dependent).

Feature construction (pre-registered options): (i) top-k variable (k ∈ {1k, 5k, 10k}) or (ii) PCA embedding to fixed dimension d ∈ {128, 256, 512}.

All selection/normalization parameters are fit on training only and applied to test to prevent leakage.

Data splitting strategy

Epigenomics: dataset level split and platform split when possible.

Transcriptomics: dataset-level split with donor-level grouping to avoid leakage.

Proteomics: dataset level split.

Model training procedures

Baselines (minimum set)

ElasticNet (continuous label regression, e.g. disease risk)

Gradient-boosted trees (LightGBM/XGBoost)

MLP (simple feedforward network)

TabPFN conditions

C0: No-finetune TabPFN.

C1: Task-adaptive finetune on target task.

C2: Domain-adaptive finetune followed by task finetune.

Planned Analyses

Primary outcomes and metrics

Label prediction (continuous disease risk): MAE (primary), RMSE, Pearson/Spearman correlation

Cell type classification: macro-F1 (primary), accuracy, per-class F1

Perturbation classification: macro-F1 (primary), accuracy, per-class F1

Hypothesis-specific tests and predictions

H1: Bootstrap CI for ΔMAE (or Δmacro-F1) under cross-dataset tests; improvement if CI excludes 0 in favorable direction.

H2: Learning curves over training size n ∈ {50, 100, 200, 500, 1k, …}; expect larger relative gains at smaller n.

H3: Compare within-dataset vs cross-dataset performance drops; expect smaller drop for finetuned TabPFN.

Contingent decisions

IF TabPFN cannot handle raw feature dimensionality (memory/time), THEN switch to fixed-dimension PCA embeddings.

IF batch effects dominate, THEN enforce stricter splits, add batch-aware normalization, and report leakage diagnostics (e.g., batch prediction accuracy).

IF scRNA label taxonomies differ across datasets, THEN harmonize to a shared coarse label set with a predefined mapping and report coarse results.

IF full fine-tuning overfits or is unstable, THEN prioritize parameter-efficient fine-tuning and regularization and report stability.

Ethics and Data Management

Ethical approval status

Public de-identified datasets do not require new IRB. For controlled-access datasets, approvals and data use agreements will be obtained as required.

Confidentiality

No direct identifiers will be processed. Only minimal metadata necessary for grouping and splitting will be retained.

Data sharing/archiving plan

Code, configs, splits, derived features/embeddings (when allowed), and predictions will be deposited in an open-access repository. Model weights will be shared when permitted; otherwise we will share training recipes and reproducible checkpoints.

4. Pilot Data (Optional)

In Month 1, we will use multi omics subsets to validate preprocessing and compare feature pipelines (e.g. for DNA methylation, top-k CpGs vs PCA). For scRNA, we will use subsets to validate label harmonization and splitting logic.

5. Budget

Compute (GPU/CPU hours, cloud credits, storage): $500

Personnel support (junior researcher time for curation/experiments/reporting): $4,500

6. References

[1] Brown, T. B., Mann, B., Ryder, N. et al. Language models are few-shot learners. Adv. Neural Inf. Process. Syst. 33, 1877–1901 (2020).

[2] Hollmann, N., Müller, S., Purucker, L. et al. Accurate predictions on small data with a tabular foundation model. Nature 637, 319–326 (2025). https://doi.org/10.1038/s41586-024-08328-6

[3] Qu, J., Holzmüller, D., Varoquaux, G. & Le Morvan, M. TabICL: A tabular foundation model for in-context learning on large data. Proc. 42nd Int. Conf. Mach. Learn. (ICML 2025) (2025).

[4] Stuart, T., Butler, A., Hoffman, P. et al. Comprehensive integration of single-cell data. Cell 177, 1888–1902.e21 (2019). https://doi.org/10.1016/j.cell.2019.05.031

[5] Chen, T. & Guestrin, C. XGBoost: A scalable tree boosting system. Proc. 22nd ACM SIGKDD Int. Conf. Knowl. Discov. Data Min. (KDD ’16) 785–794 (2016)https://doi.org/10.1145/2939672.2939785

[6] Ke, G., Meng, Q., Finley, T. et al. LightGBM: A highly efficient gradient boosting decision tree. Adv. Neural Inf. Process. Syst. 30, 3146–3154 (2017).

[7] McElfresh, D. et al. When do neural nets outperform boosted trees on tabular data? In Proc. 37th International Conference on Neural Information Processing Systems 36, 76336–76369 (ACM, 2024).

Fine-tuning an ICL Tabular Foundation Model for Multi-Omics Prediction

Fine-tuning an ICL Tabular Foundation Model for Multi-Omics Prediction

Funding Progress

Peer Reviews

Related Funding

Request for Proposals - Genomics and Proteomics

Nonprofit Recipient

Topics

DOI