We will build a data pipeline for advanced analytics using this tutorial. PolarsThis library is designed for maximum performance and scaleability. We want to show how Polars lazy evaluation, expressions with complex syntax, SQL interface, and window functions can be used to efficiently process financial data. The pipeline begins with the generation of a financial time-series dataset. We then move from rolling statistics and feature engineering to multidimensional analysis and ranking. Polars allows us to perform expressive data transformations while maintaining low memory consumption and fast execution.
Polars can be imported as pl
Numpy can be imported as np
Datetime can be imported as timedelta
Import io
try:
Import polars in the form of pl
Except ImportError
Subprocess import
subprocess.run(["pip", "install", "polars"], check=True)
Polars can be imported as polars
print("🚀 Advanced Polars Analytics Pipeline")
print("=" * 50)Importing the libraries is our first step. This includes Polars, which provides high performance DataFrame operation and NumPy, for creating synthetic data. We add an alternative installation for Polars to ensure compatibility in the event that it’s not already installed. We start our pipeline of advanced analytics once the setup is completed.
np.random.seed(42)
n_records = 100000
Dates = [datetime(2020, 1, 1) + timedelta(days=i//100) for i in range(n_records)]
The tickers are random choices.['AAPL', 'GOOGL', 'MSFT', 'TSLA', 'AMZN'], n_records)
Create complex synthetic datasetData =
data = {
'timestamp': dates,
'ticker': tickers,
'price': np.random.lognormal(4, 0.3, n_records),
'volume': np.random.exponential(1000000, n_records).astype(int),
'bid_ask_spread': np.random.exponential(0.01, n_records),
'market_cap': np.random.lognormal(25, 1, n_records),
'sector': np.random.choice(['Tech', 'Finance', 'Healthcare', 'Energy'], n_records)
}
print(f"📊 Generated {n_records:,} synthetic financial records")NumPy allows us to create a financial dataset consisting of 100 000 records. This data is used for simulating the daily ticker prices on major stock symbols such as AAPL, TSLA, and AAPL. Each entry contains key market characteristics such as volume, price, the bid-ask split, the market capitalization, and sectors. The dataset provides an ideal platform for advanced Polars analysis.
lf = pl.LazyFrame(data)
result =
Lf
.with_columns([
pl.col('timestamp').dt.year().alias('year'),
pl.col('timestamp').dt.month().alias('month'),
pl.col('timestamp').dt.weekday().alias('weekday'),
pl.col('timestamp').dt.quarter().alias('quarter')
])
.with_columns([
pl.col('price').rolling_mean(20).over('ticker').alias('sma_20'),
pl.col('price').rolling_std(20).over('ticker').alias('volatility_20'),
pl.col('price').ewm_mean(span=12).over('ticker').alias('ema_12'),
pl.col('price').diff().alias('price_diff'),
(pl.col('volume') * pl.col('price')).alias('dollar_volume')
])
.with_columns([
pl.col('price_diff').clip(0, None).rolling_mean(14).over('ticker').alias('rsi_up'),
pl.col('price_diff').abs().rolling_mean(14).over('ticker').alias('rsi_down'),
(pl.col('price') - pl.col('sma_20')).alias('bb_position')
])
.with_columns([
(100 - (100 / (1 + pl.col('rsi_up') / pl.col('rsi_down')))).alias('rsi')
])
.filter(
(pl.col('price') > 10) &
(pl.col('volume') > 100000) &
(pl.col('sma_20').is_not_null())
)
.group_by(['ticker', 'year', 'quarter'])
.agg([
pl.col('price').mean().alias('avg_price'),
pl.col('price').std().alias('price_volatility'),
pl.col('price').min().alias('min_price'),
pl.col('price').max().alias('max_price'),
pl.col('price').quantile(0.5).alias('median_price'),
pl.col('volume').sum().alias('total_volume'),
pl.col('dollar_volume').sum().alias('total_dollar_volume'),
pl.col('rsi').filter(pl.col('rsi').is_not_null()).mean().alias('avg_rsi'),
pl.col('volatility_20').mean().alias('avg_volatility'),
pl.col('bb_position').std().alias('bollinger_deviation'),
pl.len().alias('trading_days'),
pl.col('sector').n_unique().alias('sectors_count'),
(pl.col('price') > pl.col('sma_20')).mean().alias('above_sma_ratio'),
((pl.col('price').max() - pl.col('price').min()) / pl.col('price').min())
.alias('price_range_pct')
])
.with_columns([
pl.col('total_dollar_volume').rank(method='ordinal', descending=True).alias('volume_rank'),
pl.col('price_volatility').rank(method='ordinal', descending=True).alias('volatility_rank')
])
.filter(pl.col('trading_days') >= 10)
.sort(['ticker', 'year', 'quarter'])
)
Our synthetic dataset is loaded into a Polars LazyFrame for deferred processing, which allows us to efficiently chain transformations. Using window functions and rolling functions, we then enrich the data by adding time-based elements and advanced technical indicators such as Bollinger bands and RSI. After that, we use grouped aggregates by ticker and year and quarter in order to get key indicators and financial statistics. We then rank the results by volume and volatility. Then we filter out the under-traded segment and sort the data to allow for intuitive exploration.
df= result.collect()
print(f"n📈 Analysis Results: {df.height:,} aggregated records")
print("nTop 10 High-Volume Quarters:")
print(df.sort('total_dollar_volume', descending=True).head(10).to_pandas())
print("n🔍 Advanced Analytics:")
pivot_analysis = (
df.group_by('ticker')
.agg([
pl.col('avg_price').mean().alias('overall_avg_price'),
pl.col('price_volatility').mean().alias('overall_volatility'),
pl.col('total_dollar_volume').sum().alias('lifetime_volume'),
pl.col('above_sma_ratio').mean().alias('momentum_score'),
pl.col('price_range_pct').mean().alias('avg_range_pct')
])
.with_columns([
(pl.col('overall_avg_price') / pl.col('overall_volatility')).alias('risk_adj_score'),
(pl.col('momentum_score') * 0.4 +
pl.col('avg_range_pct') * 0.3 +
(pl.col('lifetime_volume') / pl.col('lifetime_volume').max()) * 0.3)
.alias('composite_score')
])
.sort('composite_score', descending=True)
)
print("n🏆 Ticker Performance Ranking:")
print(pivot_analysis.to_pandas())After our lazy pipeline has been completed, we gather the results in a DataFrame. We then review the 10 top quarters by total dollar volume. This allows us to identify intense periods of trading. Our analysis is then taken a step forward by grouping data according to ticker, allowing us to gain higher level insights such as average price volatility and lifetime trading volumes. The multi-dimensional overview allows us to compare stocks by not only raw volume but also momentum, risk-adjusted performances, and overall ticker behaviour.
print("n🔄 SQL Interface Demo:")
pl.Config.set_tbl_rows(5)
sql_result = pl.sql("""
Select
ticker,
Mean_price = AVG(avg_price).
STDDEV(price_volatility) as volatility_consistency,
SUM(total_dollar_volume) as total_volume,
COUNT"" as quarters_tracked
From df
WHERE year >= 2021
GROUP BY TICKER
ORDER BY total_volume DESC
"n⚡ Performance Metrics:"", eager=True)
print(sql_result)
print(f" • Lazy evaluation optimizations applied")
print(f" • {n_records:,} records processed efficiently")
print(f" • Memory-efficient columnar operations")
print(f" • Zero-copy operations where possible")
print(f"n💾 Export Options:")
print(f" • Parquet (high compression): df.write_parquet('data.parquet')")
print(" • Delta Lake: df.write_delta('delta_table')")
print(" • JSON streaming: df.write_ndjson('data.jsonl')")
print(" • Apache Arrow: df.to_arrow()")
print("n✅ Advanced Polars pipeline completed successfully!")
print("🎯 Demonstrated: Lazy evaluation, complex expressions, window functions,")
print(" SQL interface, advanced aggregations, and high-performance analytics")
print()
The pipeline is completed by running an aggregate SQL query using familiar SQL syntax to analyse ticker performance post-2021. We can seamlessly blend declarative SQL and expressive Polars queries with this hybrid functionality. We print performance metrics to highlight the efficiency of this hybrid capability. These include lazy evaluation, memory-efficiency, and zero copy execution. Finaly, we show you how to easily export your results in different formats like Parquet Arrow or JSONL. We have now completed a high-performance, full-circle analytics workflow with Polars.
We’ve experienced first-hand the benefits of Polars’ lazy API in optimizing complex analytics workflows, which would be slow with traditional tools. From raw data to advanced scoring and grouped aggregates, we’ve built a complete financial analysis pipeline. We also used Polars’ SQL interface, which allows us to execute familiar SQL queries over DataFrames. Polars is a powerful tool because it allows you to use both SQL and functional expressions.Take a look at thePaper .Twitter The researchers are the sole owners of all credit. Also, feel free to follow us on 100k+ ML SubReddit Join our Facebook group! our Newsletter Subscribe now
Sana Hassan is a dual-degree IIT Madras student and consulting intern with Marktechpost. She loves to apply technology and AI in order to solve real-world problems. Sana Hassan, an intern at Marktechpost and dual-degree student at IIT Madras is passionate about applying technology and AI to real-world challenges.(*)
