Tabular data—structured information stored in rows and columns—is at the heart of most real-world machine learning problems, from healthcare records to financial transactions. Models based on decisions trees such as Random Forest, XGBoost” CatBoostThey have been the first choice in these situations. The strength of these algorithms is their ability to handle mixed data types and capture complex feature interaction. Additionally, they deliver strong performance with minimal preprocessing. Although deep learning is transforming areas such as computer vision, natural language processing and other fields like that, historically it has struggled to outperform tree-based methods on tabular datasets.
The long-held tradition is under scrutiny. This newer method is being questioned. TabPFN, introduces a different way of tackling tabular problems—one that avoids traditional dataset-specific training altogether. In order to avoid learning each time from scratch, the model is pretrained to predict directly. The learning phase of learning has been shifted to inference. This article examines this concept and tests it by comparing TabPFN to established tree models such as Random Forest and CatBoost using a test dataset.
TabPFN tabulated file name. tabular foundation model TabPFN is designed to deal with structured data differently than traditional machine learning. TabPFN does not require you to train a model every time. Pre-trained on thousands of synthetic Tabular Tasks Cause-and-effect processes are generated. It can then learn how to solve supervised learning tasks in general. When you give it your dataset, it doesn’t go through iterative training like tree-based models—instead, it performs predictions directly by leveraging what it has already learned. It is a type of In-context Learning The large-language models of text are applied to table data.
TabPFN 2.5, the latest version of TabPFN, expands on this concept by supporting more data and complex datasets while improving performance. It was shown to XGBoost, CatBoost and other tuned trees-based models are more powerful than the XGBoost. AutoGluon, for example, can match the performance of strong ensemble systems. TabPFN reduces hyperparameter tuning, and the manual effort required. TabPFN has also introduced a practical deployment method. distillation approach, where its predictions can be converted into smaller models like neural networks or tree ensembles—retaining most of the accuracy while enabling much faster inference.
Create dependencies
Install tabpfn scikit learn catboostimport time
Numpy can be imported as np
Make_classification import sklearn.datasets
from sklearn.model_selection import train_test_split
From sklearn.metrics, import accuracy_score
# Models
from sklearn.ensemble import RandomForestClassifier
CatBoostClassifier imports catboost
Import TabPFNClassifier from tabpfn_clientTabPFN API key is required to run the TabPFN model. This can be found at https://ux.priorlabs.ai/home
Import os
From getpass import Getpass
os.environ['TABPFN_TOKEN'] = getpass('Enter TABPFN Token: ')Creating a dataset
In our experiment we create a binary synthetic classification dataset by using the make_classification function from Scikit-Learn. This dataset has 5,000 samples with 20 features. Of these, 10 of them are informative and help to predict the target. The other five are redundant and derived from informative features. The setup simulates a real-life scenario in which not all features contribute equally to predicting the target and that some add noise or correlation.
The data is then divided into testing (80%), training (80%), and other (20%) data sets. This allows us to assess the model’s performance using unseen information. We can control the characteristics of our data by using a synthetic dataset. This allows for a more fair comparison between TabPFN models and tree-based ones.
X, y = make_classification(
n_samples=5000,
n_features=20,
n_informative=10,
n_redundant=5,
random_state=42
)
X_train, X_test, y_train, y_test = train_test_split(
X, y, test_size=0.2, random_state=42
)Testing Random Forest
As a base, we start by using 200 random forest trees. Random Forest builds multiple decision-trees and aggregates predictions. It is an ensemble that can be used to analyze tabular data.
After training the model on a dataset, it achieves an accuracy rate of 95.5%It is impressive given that the data are synthetic. The training period is only a few hours. 9.56 The cost to build hundreds of trees is reflected in seconds. Inferences are relatively quick. 0.0627 This is because predictions are only a matter of passing the data through trees that have already been constructed. This results can be used as a baseline against other methods, such CatBoost or TabPFN.
rf = RandomForestClassifier(n_estimators=200)
"Start" = "time.time()
rf.fit(X_train, y_train)
rf_train_time = time.time() Startseite
"Start" = "time.time()
rf_preds = rf.predict(X_test)
rf_infer_time = time.time() Startseite
rf_acc = accuracy_score(y_test, rf_preds)
print(f"RandomForest → Acc: {rf_acc:.4f}, Train: {rf_train_time:.2f}s, Infer: {rf_infer_time:.4f}s")Testing CatBoost
We then train the CatBoost model, which is a gradient-boosting algorithm specifically developed for tabular data. The trees are built sequentially and each tree is designed to correct the mistakes of previous trees. CatBoost tends to be more accurate than Random Forest because it uses a boosting technique and can model more complex patterns.
CatBoost achieved an accuracy of 95% on our dataset. 96.7%It is a tree-based algorithm that outperforms Random Forest, demonstrating it as the most advanced method. This method also trains faster. 8.15 seconds, despite using 500 boosting iterations. One of its biggest advantages is inference speed—predictions are extremely fast at just 0.0119 This makes it a good choice for scenarios in production where low latencies are critical. CatBoost is a good benchmark to compare with newer solutions like TabPFN.
CatBoostClassifier = cat
iterations=500,
depth=6,
learning_rate=0.1,
verbose=0
)
"Start" = "time.time()
cat.fit(X_train, y_train)
cat_train_time = time.time() Start
"Start" = "time.time()
cat_preds = cat.predict(X_test)
cat_infer_time = time.time() Start
cat_acc = accuracy_score(y_test, cat_preds)
print(f"CatBoost → Acc: {cat_acc:.4f}, Train: {cat_train_time:.2f}s, Infer: {cat_infer_time:.4f}s")Testing TabPFN
TabPFN was evaluated as a final model. It takes an entirely different approach to the traditional models. It uses a pre-trained model to infer data instead of starting from scratch. The.fit() The step is very fast because it involves primarily loading the pre-trained weights.
TabPFN is the most accurate of all our datasets. 98.8%CatBoost and Random Forest are outperformed. Fit time takes just a few minutes 0.47 The tree-based model is significantly faster because no actual training takes place. However, this shift comes with a trade-off—inference takes 2.21 This is because TabPFN processes both the training and test data together during prediction, effectively performing the same function as CatBoost or Random Forest. TabPFN performs the prediction by combining the data from both training and testing. “learning” Inference is a step in the right direction
TabPFN is a powerful tool for calculating tabular data. It offers a significant advantage over traditional models in terms of accuracy, setup time, and computational efficiency.
tabpfn is the same as TabPFNClassifier()
"Start" = "time.time()
tabpfn.fit(X_train, y_train) # loads pretrained model
tabpfn_train_time = time.time() Startseite
"Start" = "time.time()
tabpfn_preds = tabpfn.predict(X_test)
tabpfn_infer_time = time.time() Start
tabpfn_acc = accuracy_score(y_test, tabpfn_preds)
print(f"TabPFN → Acc: {tabpfn_acc:.4f}, Fit: {tabpfn_train_time:.2f}s, Infer: {tabpfn_infer_time:.4f}s")TabPFN is the most accurate and robust solution in our tests.98.8%The training required is minimal (0.47sRandom Forest (9.56sCatBoost (8.15s). Its key benefit is that it eliminates dataset-specific learning and hyperparameter optimization while outperforming established tree-based techniques. However, this benefit comes with a trade-off—The inference delay is much higher than the average (2.21s).The model is able to predict using both test and training data. CatBoost, Random Forest and other models offer faster inferences, which makes them suitable for real time applications.
TabPFN can be highly useful for a variety of purposes. small-to-medium tabular tasksRapid experimentation and scenarios in which minimizing the development time is crucial. In production environments that require low-latency forecasts or handle very large datasets (e.g., those with high data volumes), newer developments such as TabPFN’s distillation engine This gap can be bridged by converting a model into compact tree ensembles or neural networks, which retains most of the accuracy and improves inference speeds dramatically. It is also more viable in enterprise scenarios due to its support of scaling up to millions rows. Overall, TabPFN represents a shift in tabular machine learning—trading traditional training effort for a more flexible, inference-driven approach.
Check out the Full Codes with Notebook here. Also, feel free to follow us on Twitter Join our Facebook group! 130k+ ML SubReddit Subscribe now our Newsletter. Wait! What? now you can join us on telegram as well.
Want to promote your GitHub repo, Hugging Face page, Product release or Webinar?? Connect with us
I’m a Civil Engineering graduate (2022) at Jamia Millia Islamia in New Delhi. I have an interest in Data Science and in particular, Neural Networks, as well as their applications in different areas.
