Boylan, James, Solovyeva, Elizaveta, Bouiller, Theophile, Liu, Xiong, Hoersch, Sebastian, Ataman, Bulent, Jenkins, Jeremy and Devarakonda, Murthy (2025) Single Cell Foundation Models Evaluation (scFME) for In-Silico Perturbation. bioRxiv. ISSN 2692-8205

Abstract

Foundation models pre-trained on large single-cell RNA atlas present a com�pelling alternative to in-vitro experimentation for understanding gene regulatory networks and conducting gene perturbation analyses, with significant implica�tions to target identification. Numerous foundation models have been developed, expanding upon early efforts such as Geneformer and scGPT, and they are also configurable through hyper-parameterization, resulting in multiple models and instances that require comparative analysis. Current benchmarking focuses on feature-based assessments or employing intuitive biological and statistical tasks that may not align with the models’ training objectives. A recent study proposed a systematic benchmarking; however, its scope was limited to pre-trained (zero�shot) models. To address these limitations, we propose Single-Cell Foundation Model Evaluation (scFME), a systematic method designed for benchmarking fine-tuned foundation models for in-silico perturbation (ISP). scFME ensures comprehensive and robust assessment by requiring sufficient separation between control and perturbed cells to begin with and quantifying ISP accuracy against zero and random perturbations baselines. Furthermore, scFME enables explo�ration of models’ performance across distinct gene categories enabling biological implications and functional relevance. Using this framework, we assessed several commonly used models (and some of their variants) and demonstrated that the methodology allows us to characterize their performance for ISP studies. Our results position scFME as a versatile and rigorous methodology for the evaluation
and comparison of current and future foundation models.

Item Type:	Article
Keywords:	Foundation models scRNA-seq In-silico perturbation Benchmarking
Date Deposited:	12 Nov 2025 00:45
Last Modified:	12 Nov 2025 00:45
URI:	https://oak.novartis.com/id/eprint/58560