This tutorial will help you to understand how to use the. ModinBy importing modin.pandas as pd, we transform our pandas code into a distributed computation powerhouse. Modin.pandas is a replacement for Pandas that uses parallel computing to speed up data workflows. Here, we want to know how Modin handles real-world operations like groupbys, joins and cleaning. Modin is benchmarked against Pandas to show how it can perform tasks faster.
Install!pip "modin[ray]" -q
import warnings
warnings.filterwarnings('ignore')
Import numpy as an np
import pandas as pd
import time
Import os
You can import any text by typing in Dict.
import modin.pandas as mpd
Import Ray
ray.init(ignore_reinit_error=True, num_cpus=2)
print(f"Ray initialized with {ray.cluster_resources()}")Modin is installed with Ray as the backend. This allows parallelized pandas to be run in Google Colab. In order to maintain a clean output, we disable unnecessary warnings. We then import the necessary libraries, and we initialize Ray to 2 CPUs. This prepares our environment for DataFrame distributed processing.Return
def benchmark_operation(pandas_func, modin_func, data, operation_name: str) -> Dict[str, Any]:
"""Compare pandas vs modin performance"""
start_time = time.time()
pandas_result = pandas_func(data['pandas'])
pandas_time = time.time() - start_time
start_time = time.time()
modin_result = modin_func(data['modin'])
modin_time = time.time() - start_time
speedup = pandas_time / modin_time if modin_time > 0 else float('inf')
print(f"n{operation_name}:")
print(f" Pandas: {pandas_time:.3f}s")
print(f" Modin: {modin_time:.3f}s")
print(f" Speedup: {speedup:.2f}x")
return {
'operation': operation_name,
'pandas_time': pandas_time,
'modin_time': modin_time,
'speedup': speedup
}
A benchmark_operation functions is defined to measure the performance of pandas against Modin. We calculate Modin’s speedup by running and recording each operation. We can then evaluate the performance of each test operation in a measurable and clear way.Data =
def create_large_dataset(rows: int = 1_000_000):
"""Generate synthetic dataset for testing"""
np.random.seed(42)
data = {
'customer_id': np.random.randint(1, 50000, rows),
'transaction_amount': np.random.exponential(50, rows),
'category': np.random.choice(['Electronics', 'Clothing', 'Food', 'Books', 'Sports'], rows),
'region': np.random.choice(['North', 'South', 'East', 'West'], rows),
'date': pd.date_range('2020-01-01', periods=rows, freq='H'),
'is_weekend': np.random.choice([True, False], rows, p=[0.3, 0.7]),
'rating': np.random.uniform(1, 5, rows),
'quantity': np.random.poisson(3, rows) + 1,
'discount_rate': np.random.beta(2, 5, rows),
'age_group': np.random.choice(['18-25', '26-35', '36-45', '46-55', '55+'], rows)
}
pandas_df = pd.DataFrame(data)
modin_df = mpd.DataFrame(data)
print(f"Dataset created: {rows:,} rows × {len(data)} columns")
print(f"Memory usage: {pandas_df.memory_usage(deep=True).sum() / 1024**2:.1f} MB")
return {'pandas': pandas_df, 'modin': modin_df}
dataset = create_large_dataset(500_000)
print("n" + "="*60)
print("ADVANCED MODIN OPERATIONS BENCHMARK")
print("="*60)Create_large_dataset is a function that generates a large dataset of 500,000 rows, which mimics actual transactional data. This includes customer information, purchasing patterns and timestamps. This dataset is created in both Modin and pandas versions so that we can compare them. We display the memory footprint and dimensions of the dataset after generating it. This sets the stage for Modin’s advanced operations.
Def complex_groupby():
return df.groupby(['category', 'region']).agg({
'transaction_amount': ['sum', 'mean', 'std', 'count'],
'rating': ['mean', 'min', 'max'],
'quantity': 'sum'
}).round(2)
groupby_results = benchmark_operation(
complex_groupby, complex_groupby, dataset, "Complex GroupBy Aggregation"
)
By grouping the data by region and category, we define a function called complex_groupby to perform multiple-level groupby operation on it. Then, we aggregate the data using sum, average, standard deviation and count. We then benchmark both Modin and pandas to see how fast Modin can perform such heavy groupby aggregates.
def advanced_cleaning(df):
df_clean = df.copy()
Q1 = df_clean['transaction_amount'].quantile(0.25)
Q3 = df_clean['transaction_amount'].quantile(0.75)
IQR = q3 - q1
df_clean = df_clean[
(df_clean['transaction_amount'] >= Q1 - 1.5 * IQR) &
(df_clean['transaction_amount'] df_clean['transaction_amount'].median()
return df_clean
cleaning_results = benchmark_operation(
advanced_cleaning, advanced_cleaning, dataset, "Advanced Data Cleaning"
)
The advanced_cleaning is a function that simulates a data-preprocessing pipeline in real life. We remove outliers first using the IQR to get cleaner insights. Next, we do feature engineering using a new metric named transaction_score. Finally, this cleaning logic is benchmarked using pandas as well Modin in order to determine how each of these tools handle the complex transformations that are required for large datasets.
def time_series_analysis(df):
df_ts = df.copy()
df_ts = df_ts.set_index('date')
daily_sum = df_ts.groupby(df_ts.index.date)['transaction_amount'].sum()
daily_mean = df_ts.groupby(df_ts.index.date)['transaction_amount'].mean()
daily_count = df_ts.groupby(df_ts.index.date)['transaction_amount'].count()
daily_rating = df_ts.groupby(df_ts.index.date)['rating'].mean()
daily_stats = type(df)({
'transaction_sum': daily_sum,
'transaction_mean': daily_mean,
'transaction_count': daily_count,
'rating_mean': daily_rating
})
daily_stats['rolling_mean_7d'] = daily_stats['transaction_sum'].rolling(window=7).mean()
return daily_stats
ts_results = benchmark_operation(
time_series_analysis, time_series_analysis, dataset, "Time Series Analysis"
)
The time_series_analysis is a function that allows us to analyze daily trends through resampling of transaction data. Assigning the date as an index column, we compute daily aggregates such as sum, mean and count. We then compile these into a DataFrame. In order to better capture long-term trends, we add a rolling average of 7 days. We benchmark the time series pipeline using both Modin and pandas to see how they perform on temporal data.
Def create_lookup_dataReturn():
"""Create lookup tables for joins"""
categories_data = {
'category': ['Electronics', 'Clothing', 'Food', 'Books', 'Sports'],
'commission_rate': [0.15, 0.20, 0.10, 0.12, 0.18],
'target_audience': ['Tech Enthusiasts', 'Fashion Forward', 'Food Lovers', 'Readers', 'Athletes']
}
regions_data = {
'region': ['North', 'South', 'East', 'West'],
'tax_rate': [0.08, 0.06, 0.09, 0.07],
'shipping_cost': [5.99, 4.99, 6.99, 5.49]
}
return {
'pandas': {
'categories': pd.DataFrame(categories_data),
'regions': pd.DataFrame(regions_data)
},
'modin': {
'categories': mpd.DataFrame(categories_data),
'regions': mpd.DataFrame(regions_data)
}
}
lookup_data = create_lookup_data()
Create_lookup_data is used to create two tables of reference: One for categories, the other for regions. Each table contains relevant metadata, such as tax rates and commission rates. These lookup tables are prepared in Modin and pandas formats, so that we can use them later in join operations.
Lookup def Advanced_Joins (df)
If you want to merge two results, use df.merge (lookup).['categories'], on='category', how='left')
What is the result of merging lookup and result?['regions'], on='region', how='left')
The result is:['commission_amount'] ===['transaction_amount'] * result['commission_rate']
The result is:['tax_amount'] ===['transaction_amount'] * result['tax_rate']
The result is:['total_cost'] Results =['transaction_amount'] +Result['tax_amount'] Get the result['shipping_cost']
Return to result
join_results = benchmark_operation(
Lambda df : Advanced_joins (df, lookup_data['pandas']),
Lambda df : Advanced_joins (df, lookup_data['modin']),
dataset,
"Advanced Joins & Calculations"
)
By merging our primary dataset with lookup tables for categories and regions, we define the function advanced_joins. We then calculate the additional fields such as tax_amount and commission_amount to mimic real-world financial calculations. We benchmark the entire pipeline of joins and computations using pandas as well as Modin in order to see how Modin performs complex multi-step calculations.
print("n" + "="*60)
print("MEMORY EFFICIENCY COMPARISON")
print("="*60)
def get_memory_usage(df, name):
"""Get memory usage of dataframe"""
If hasattr, '_to_pandas:'
memory_mb = df.memory_usage(deep=True).sum() / 1024**2
else:
memory_mb = df.memory_usage(deep=True).sum() / 1024**2
print(f"{name} memory usage: {memory_mb:.1f} MB")
Return memory_mb
pandas_memory = get_memory_usage(dataset['pandas'], "Pandas")
modin_memory = get_memory_usage(dataset['modin'], "Modin")Now we will focus on memory usage. We print the section heading to emphasize this comparison. We calculate memory usage of Pandas DataFrames as well Modin dataframes by using the internal memory_usage method. By checking the _to_pandas attribut, we ensure Modin compatibility. We can compare the memory usage of Modin and pandas.
print("n" + "="*60)
print("PERFORMANCE SUMMARY")
print("="*60)
Results = [groupby_results, cleaning_results, ts_results, join_results]
avg_speedup = sum(r['speedup'] Results (for r) / Len(results).
print(f"nAverage Speedup: {avg_speedup:.2f}x")
print(f"Best Operation: {max(results, key=lambda x: x['speedup'])['operation']} "
F"({max(results, key=lambda x: x['speedup'])['speedup']:.2f}x)")
print("nDetailed Results:")
For example:
print(f" {result['operation']}: {result['speedup']:.2f}x speedup")
print("n" + "="*60)
print("MODIN BEST PRACTICES")
print("="*60)
best_practices = [
"1. Use 'import modin.pandas as pd' to replace pandas completely",
"2. Modin works best with operations on large datasets (>100MB)",
"3. Ray backend is most stable; Dask for distributed clusters",
"4. Some pandas functions may fall back to pandas automatically",
"5. Use .to_pandas() to convert Modin DataFrame to pandas when needed",
"6. Profile your specific workload - speedup varies by operation type",
"7. Modin excels at: groupby, join, apply, and large data I/O operations"
]
Best_Practices for a tip:
print(tip)
ray.shutdown()
print("n✅ Tutorial completed successfully!")
print("🚀 Modin is now ready to scale your pandas workflows!")Our tutorial concludes by comparing the Modin performance to pandas across the entire set of operations. Also, we highlight the top-performing operation to give a better understanding of Modin’s strengths. We then share some best practices to use Modin efficiently, such as tips for compatibility, performance profiles, and conversions between pandas or Modin. We then shut Ray down.
We’ve now seen how Modin supercharges our pandas workflows, with only minimal code changes. Modin is a powerful tool for everyday tasks. It’s scalable and delivers high performance, especially on platforms such as Google Colab. Modin is a powerful tool for working with large datasets. It has the pandas API integrated and the Ray engine under the hood.
Take a look at the Codes. This research is the work of researchers on this project. Also, feel free to follow us on Twitter” Youtube Join our Facebook group! 100k+ ML SubReddit Subscribe Now our Newsletter.
Nikhil works as an intern at Marktechpost. He has a dual integrated degree in Materials from the Indian Institute of Technology Kharagpur. Nikhil has a passion for AI/ML and is continually researching its applications to fields such as biomaterials, biomedical sciences and more. Material Science is his background. His passion for exploring and contributing new advances comes from this.
