Table of Contents
- CPU-Bound vs I/O-Bound — The First Question
- The GIL — Why Threading Does Not Speed Up CPU Work
- threading — Concurrent I/O Operations
- ThreadPoolExecutor — The Modern Way
- multiprocessing — True Parallelism for CPU Work
- ProcessPoolExecutor — Clean API for CPU Work
- asyncio — Event-Driven Concurrency
- aiohttp — Parallel API Calls with asyncio
- Choosing the Right Approach
- Real-World Data Engineering Patterns
- Common Mistakes
- Interview Questions
- Wrapping Up
Our ETL Patterns post built pipelines that process data sequentially — extract, then transform, then load, one step at a time. But what happens when you need to call 50 APIs, process 200 CSV files, or transform data across all CPU cores? Sequential execution takes 50 times as long as it needs to. Concurrency and parallelism let you do multiple things at once.
Analogy — A restaurant kitchen. A sequential kitchen has one chef who cooks one dish at a time. Customer 10 waits while dishes 1 through 9 are prepared. A concurrent kitchen (threading) has one chef who starts the soup simmering, puts bread in the oven, and chops vegetables while both cook — one person, multiple tasks in progress. A parallel kitchen (multiprocessing) has five chefs, each cooking a different dish simultaneously — true parallel work. An async kitchen (asyncio) has one chef with a timer system — start the soup, set a timer, start the bread, set a timer, and handle whichever finishes first. This post teaches you all three approaches and when to use each.
CPU-Bound vs I/O-Bound — The First Question
Before choosing a concurrency approach, you must answer one question: is your task CPU-bound or I/O-bound?
CPU-bound tasks spend their time computing — crunching numbers, transforming DataFrames, hashing values, running ML models. The bottleneck is the processor. Adding more I/O speed does not help because the CPU is already at 100%.
I/O-bound tasks spend their time waiting — waiting for an API response, waiting for a file to download, waiting for a database query to return. The CPU is idle most of the time. Adding more CPU does not help because the bottleneck is the network or disk.
| Task Type | Bottleneck | Example | Best Approach |
|---|---|---|---|
| **I/O-bound** | Network, disk, database | API calls, file downloads, SQL queries | threading or asyncio |
| **CPU-bound** | Processor | DataFrame transforms, hashing, parsing, ML training | multiprocessing |
| **Mixed** | Both | Fetch from API (I/O) then transform (CPU) | asyncio + ProcessPoolExecutor |
The GIL — Why Threading Does Not Speed Up CPU Work
Python has a Global Interpreter Lock (GIL) — a mutex that allows only one thread to execute Python bytecode at a time. Even if you create 10 threads on a 10-core machine, only one thread runs Python code at any given moment. The others wait.
Analogy — A single bathroom in a shared apartment. Ten roommates (threads) live in the apartment, but there is only one bathroom (GIL). Only one person can use it at a time. The others queue up. If everyone needs to shower (CPU work), having 10 roommates does not make the morning faster — they still go one at a time. But if most roommates are out at work (I/O wait) and only occasionally need the bathroom, 10 roommates barely notice the single bathroom.
This is why threading works for I/O-bound tasks (threads release the GIL while waiting for network/disk) but not for CPU-bound tasks (threads hold the GIL while computing). For CPU-bound work, you need multiprocessing — separate processes, each with its own GIL.
threading — Concurrent I/O Operations
The threading module runs functions in separate threads within the same process. Threads share memory (same variables, same DataFrames) and are lightweight to create. Use threading when you need to run multiple I/O operations concurrently.
import threading
import requests
import time
results = {} # Shared dictionary (threads share memory)
def fetch_url(name, url):
# Each thread fetches a different URL concurrently.
response = requests.get(url, timeout=30)
results[name] = {
"status": response.status_code,
"size": len(response.content)
}
# Create threads
urls = {
"github": "https://api.github.com",
"httpbin": "https://httpbin.org/get",
"jsonplaceholder": "https://jsonplaceholder.typicode.com/posts/1"
}
start = time.time()
threads = []
for name, url in urls.items():
t = threading.Thread(target=fetch_url, args=(name, url))
threads.append(t)
t.start()
# Wait for all threads to finish
for t in threads:
t.join()
print(f"Fetched {len(results)} URLs in {time.time() - start:.1f}s")
# Sequential: ~3s (1s per request). Threaded: ~1s (all 3 in parallel)ThreadPoolExecutor — The Modern Way
concurrent.futures.ThreadPoolExecutor is the modern, higher-level interface for threading. It manages a pool of worker threads, handles thread lifecycle, and provides a clean API for submitting tasks and collecting results.
Analogy — A taxi dispatch center. Instead of managing individual taxis (threads) yourself, you call the dispatch center (executor). You say “I need 5 rides” (submit 5 tasks), and the center assigns available taxis. When a ride finishes, the taxi returns to the pool for the next customer.
from concurrent.futures import ThreadPoolExecutor, as_completed
import requests
import time
def fetch_url(url):
# Fetch a URL and return the response size.
response = requests.get(url, timeout=30)
return {"url": url, "status": response.status_code, "size": len(response.content)}
urls = [
"https://api.github.com",
"https://httpbin.org/get",
"https://jsonplaceholder.typicode.com/posts/1",
"https://jsonplaceholder.typicode.com/posts/2",
"https://jsonplaceholder.typicode.com/posts/3",
]
start = time.time()
# Using context manager (auto-cleanup)
with ThreadPoolExecutor(max_workers=5) as executor:
# Submit all tasks
futures = {executor.submit(fetch_url, url): url for url in urls}
# Collect results as they complete (not in submission order)
for future in as_completed(futures):
url = futures[future]
try:
result = future.result()
print(f" {result['status']} {result['url']}")
except Exception as e:
print(f" FAILED {url}: {e}")
print(f"Done in {time.time() - start:.1f}s")executor.map() — Simpler for Uniform Tasks
When every task calls the same function with different arguments, map() is cleaner than submit(). Results are returned in submission order (not completion order).
from concurrent.futures import ThreadPoolExecutor
import requests
def fetch_url(url):
response = requests.get(url, timeout=30)
return response.status_code
urls = [f"https://jsonplaceholder.typicode.com/posts/{i}" for i in range(1, 51)]
with ThreadPoolExecutor(max_workers=10) as executor:
status_codes = list(executor.map(fetch_url, urls))
print(f"Fetched {len(status_codes)} URLs")
print(f"All 200s: {all(s == 200 for s in status_codes)}")
# 50 sequential requests at 0.5s each = 25s
# 50 threaded requests with 10 workers = ~2.5smultiprocessing — True Parallelism for CPU Work
The multiprocessing module spawns separate Python processes, each with its own GIL. This gives you true parallel execution on multiple CPU cores — the only way to speed up CPU-bound Python code.
Analogy — Hiring additional chefs. Each chef (process) has their own kitchen station (memory space), their own set of knives (GIL), and works completely independently. They do not share cutting boards (no shared memory by default) — you hand them ingredients (function arguments) and they hand back finished dishes (return values).
import multiprocessing
import time
def cpu_heavy_task(n):
# Simulate CPU-intensive work: compute sum of squares.
return sum(i ** 2 for i in range(n))
if __name__ == "__main__":
numbers = [10_000_000] * 8 # 8 identical CPU-heavy tasks
# Sequential
start = time.time()
results_seq = [cpu_heavy_task(n) for n in numbers]
print(f"Sequential: {time.time() - start:.1f}s")
# Parallel with multiprocessing.Pool
start = time.time()
with multiprocessing.Pool(processes=4) as pool:
results_par = pool.map(cpu_heavy_task, numbers)
print(f"Parallel (4 cores): {time.time() - start:.1f}s")
# On a 4-core machine: Sequential ~8s, Parallel ~2sProcessPoolExecutor — Clean API for CPU Work
ProcessPoolExecutor provides the same clean API as ThreadPoolExecutor but uses processes instead of threads. Swap one for the other depending on whether your task is I/O-bound or CPU-bound.
from concurrent.futures import ProcessPoolExecutor
import time
def transform_chunk(chunk_data):
# CPU-intensive transformation on a data chunk.
import pandas as pd
df = pd.DataFrame(chunk_data)
df["amount_usd"] = df["amount"] * df["exchange_rate"]
df["category"] = df["product"].str.lower().str.strip()
df["hash_id"] = df.apply(lambda r: hash(f"{r['order_id']}_{r['product']}"), axis=1)
return df.to_dict("records")
if __name__ == "__main__":
# Split large dataset into chunks for parallel processing
import pandas as pd
large_df = pd.DataFrame({
"order_id": range(1_000_000),
"product": ["Widget"] * 1_000_000,
"amount": [29.99] * 1_000_000,
"exchange_rate": [1.35] * 1_000_000
})
# Split into chunks
chunk_size = 250_000
chunks = [
large_df.iloc[i:i+chunk_size].to_dict("records")
for i in range(0, len(large_df), chunk_size)
]
start = time.time()
with ProcessPoolExecutor(max_workers=4) as executor:
results = list(executor.map(transform_chunk, chunks))
# Flatten and reassemble
all_records = [rec for chunk in results for rec in chunk]
result_df = pd.DataFrame(all_records)
print(f"Processed {len(result_df)} rows in {time.time() - start:.1f}s")asyncio — Event-Driven Concurrency
asyncio is Python’s built-in framework for writing concurrent code using async/await syntax. It runs a single-threaded event loop that switches between tasks whenever one is waiting for I/O. No threads, no processes — just one thread that is extremely efficient at juggling many I/O operations.
Analogy — A waiter at a busy restaurant. One waiter (event loop) serves 20 tables (tasks). They take an order at table 1, walk to table 2, deliver food to table 3, and circle back to check on table 1. The waiter never sits idle — they always move to the next table that needs attention. This is far more efficient than hiring 20 waiters (threads) who each stand at one table waiting.
async/await Basics
import asyncio
async def fetch_data(name, delay):
# An async function (coroutine) that simulates an I/O wait.
print(f" Starting {name}...")
await asyncio.sleep(delay) # Non-blocking sleep (simulates network wait)
print(f" Finished {name}")
return f"{name}: {delay}s"
async def main():
# Run three tasks concurrently with asyncio.gather()
results = await asyncio.gather(
fetch_data("API-1", 2),
fetch_data("API-2", 1),
fetch_data("API-3", 3)
)
print(f"All done: {results}")
# Total time: ~3s (longest task), not 6s (sum of all tasks)
asyncio.run(main())asyncio.gather() vs asyncio.create_task()
gather() runs multiple coroutines concurrently and returns all results. create_task() schedules a coroutine to run in the background while you do other work.
import asyncio
async def fetch(name, delay):
await asyncio.sleep(delay)
return f"{name} done"
async def main():
# gather: wait for all to complete, get results as a list
results = await asyncio.gather(
fetch("A", 1),
fetch("B", 2),
fetch("C", 1)
)
print(results) # ["A done", "B done", "C done"]
# create_task: fire-and-forget, check results later
task_a = asyncio.create_task(fetch("X", 2))
task_b = asyncio.create_task(fetch("Y", 1))
# Do other work while tasks run in the background
print("Tasks running in background...")
# Await results when needed
result_a = await task_a
result_b = await task_b
print(f"Background results: {result_a}, {result_b}")
asyncio.run(main())aiohttp — Parallel API Calls with asyncio
The requests library is synchronous — it blocks the event loop. For async HTTP calls, use aiohttp. This is the fastest way to call many APIs concurrently from Python.
import asyncio
import aiohttp
import time
async def fetch_url(session, url):
# Async HTTP GET using aiohttp.
async with session.get(url) as response:
data = await response.json()
return {"url": url, "status": response.status, "data": data}
async def fetch_all(urls, max_concurrent=10):
# Fetch multiple URLs concurrently with a concurrency limit.
semaphore = asyncio.Semaphore(max_concurrent) # Limit concurrent requests
async def limited_fetch(session, url):
async with semaphore:
return await fetch_url(session, url)
async with aiohttp.ClientSession() as session:
tasks = [limited_fetch(session, url) for url in urls]
return await asyncio.gather(*tasks, return_exceptions=True)
# Fetch 50 API endpoints concurrently
urls = [f"https://jsonplaceholder.typicode.com/posts/{i}" for i in range(1, 51)]
start = time.time()
results = asyncio.run(fetch_all(urls, max_concurrent=10))
print(f"Fetched {len(results)} URLs in {time.time() - start:.1f}s")
# Sequential with requests: ~25s. Async with aiohttp: ~1sControlling Concurrency with Semaphores
Without a semaphore, asyncio.gather() fires all requests simultaneously. If you have 1,000 URLs, that means 1,000 concurrent connections — most APIs will rate-limit or block you. A Semaphore limits the number of concurrent operations, acting like a bouncer at a club.
import asyncio
import aiohttp
async def fetch_with_rate_limit(urls, max_concurrent=5, delay_between=0.1):
# Fetch URLs with rate limiting: max N concurrent + delay between starts.
semaphore = asyncio.Semaphore(max_concurrent)
results = []
async def fetch_one(session, url):
async with semaphore:
await asyncio.sleep(delay_between) # Polite delay
async with session.get(url) as response:
data = await response.json()
return {"url": url, "status": response.status}
async with aiohttp.ClientSession() as session:
tasks = [fetch_one(session, url) for url in urls]
results = await asyncio.gather(*tasks, return_exceptions=True)
successes = [r for r in results if isinstance(r, dict)]
failures = [r for r in results if isinstance(r, Exception)]
print(f"Success: {len(successes)}, Failed: {len(failures)}")
return successesChoosing the Right Approach
Here is the decision framework. Start with the task type, then choose the simplest tool that fits.
| Scenario | Best Approach | Why |
|---|---|---|
| Fetch 5-50 URLs using `requests` | `ThreadPoolExecutor` | Simple, blocking library, moderate concurrency |
| Fetch 100-10,000 URLs | `asyncio` + `aiohttp` | Non-blocking, handles massive concurrency efficiently |
| Process 200 CSV files (reading from disk) | `ThreadPoolExecutor` | I/O-bound (disk reads), threads release GIL during I/O |
| Transform a large DataFrame across CPU cores | `ProcessPoolExecutor` | CPU-bound, needs true parallelism (bypass GIL) |
| Call 10 database queries concurrently | `ThreadPoolExecutor` | I/O-bound (network), database drivers release GIL |
| Mix: fetch from APIs then transform | `asyncio` + `run_in_executor` | Async I/O for fetching, process pool for CPU transforms |
| Simple script, 3 sequential API calls | Sequential (`requests`) | Overhead of concurrency exceeds benefit for small tasks |
Rule of thumb: If the task takes less than 100ms per item, run it sequentially. If it takes 100ms-1s per item and you have 10+ items, use ThreadPoolExecutor. If you have 100+ I/O tasks, use asyncio. If the task is CPU-intensive, use ProcessPoolExecutor.
Real-World Data Engineering Patterns
Pattern 1: Parallel File Processing
Process 200 CSV files concurrently using ThreadPoolExecutor (I/O-bound: reading files from disk).
from concurrent.futures import ThreadPoolExecutor
from pathlib import Path
import pandas as pd
def process_file(filepath):
# Read, clean, and return summary for one file.
df = pd.read_csv(filepath)
df.columns = df.columns.str.strip().str.lower().str.replace(r"[^a-z0-9]+", "_", regex=True)
df = df.dropna(subset=["order_id"])
return {
"file": filepath.name,
"rows": len(df),
"revenue": df["amount"].sum() if "amount" in df.columns else 0
}
files = list(Path("vendor_drops/").glob("*.csv"))
with ThreadPoolExecutor(max_workers=8) as executor:
summaries = list(executor.map(process_file, files))
summary_df = pd.DataFrame(summaries)
print(summary_df)
print(f"Total revenue: {summary_df['revenue'].sum():,.2f}")Pattern 2: Concurrent API Extraction
Extract data from 50 API endpoints concurrently, then combine into a single DataFrame.
import asyncio
import aiohttp
import pandas as pd
async def extract_endpoint(session, endpoint, semaphore):
async with semaphore:
async with session.get(endpoint) as response:
data = await response.json()
return data
async def extract_all_endpoints(base_url, endpoint_ids, max_concurrent=10):
semaphore = asyncio.Semaphore(max_concurrent)
async with aiohttp.ClientSession() as session:
tasks = [
extract_endpoint(session, f"{base_url}/{eid}", semaphore)
for eid in endpoint_ids
]
results = await asyncio.gather(*tasks, return_exceptions=True)
# Filter out failures
valid = [r for r in results if isinstance(r, dict)]
return pd.DataFrame(valid)
# Usage
df = asyncio.run(extract_all_endpoints(
"https://api.example.com/orders",
endpoint_ids=range(1, 51),
max_concurrent=10
))
print(f"Extracted {len(df)} records from API")Pattern 3: Parallel DataFrame Transform
Split a large DataFrame into chunks, transform each chunk in a separate process, and reassemble.
from concurrent.futures import ProcessPoolExecutor
import pandas as pd
import numpy as np
def transform_chunk(chunk_dict):
# CPU-intensive transform on a DataFrame chunk.
df = pd.DataFrame(chunk_dict)
df["amount_normalized"] = (df["amount"] - df["amount"].mean()) / df["amount"].std()
df["category_clean"] = df["product"].str.lower().str.strip()
df["row_hash"] = df.apply(lambda r: hash(str(r.values)), axis=1)
return df.to_dict("records")
def parallel_transform(df, n_workers=4):
# Split, transform in parallel, reassemble.
chunks = np.array_split(df, n_workers)
chunk_dicts = [chunk.to_dict("records") for chunk in chunks]
with ProcessPoolExecutor(max_workers=n_workers) as executor:
results = list(executor.map(transform_chunk, chunk_dicts))
# Flatten and reassemble
all_records = [rec for chunk in results for rec in chunk]
return pd.DataFrame(all_records)
if __name__ == "__main__":
large_df = pd.DataFrame({
"order_id": range(500_000),
"product": np.random.choice(["Widget", "Gadget", "Doohickey"], 500_000),
"amount": np.random.uniform(5, 500, 500_000)
})
result = parallel_transform(large_df, n_workers=4)
print(f"Transformed {len(result)} rows")Pattern 4: Hybrid — Async I/O + Parallel CPU
The most powerful pattern: use asyncio for I/O (fetching data) and ProcessPoolExecutor for CPU work (transforming data), in the same pipeline.
import asyncio
import aiohttp
from concurrent.futures import ProcessPoolExecutor
import pandas as pd
def heavy_transform(records):
# CPU-intensive work -- runs in a separate process.
df = pd.DataFrame(records)
df["score"] = df.apply(lambda r: sum(ord(c) for c in str(r["title"])), axis=1)
return df.to_dict("records")
async def fetch_and_transform(urls, max_concurrent=10):
# Phase 1: Async I/O -- fetch all URLs concurrently
semaphore = asyncio.Semaphore(max_concurrent)
async def fetch(session, url):
async with semaphore:
async with session.get(url) as resp:
return await resp.json()
async with aiohttp.ClientSession() as session:
tasks = [fetch(session, url) for url in urls]
records = await asyncio.gather(*tasks)
# Phase 2: CPU work -- transform in a process pool
loop = asyncio.get_event_loop()
with ProcessPoolExecutor(max_workers=4) as executor:
result = await loop.run_in_executor(executor, heavy_transform, records)
return pd.DataFrame(result)
if __name__ == "__main__":
urls = [f"https://jsonplaceholder.typicode.com/posts/{i}" for i in range(1, 51)]
df = asyncio.run(fetch_and_transform(urls))
print(f"Fetched and transformed {len(df)} records")Common Mistakes
1. Using threading for CPU-bound work. Due to the GIL, threads cannot execute Python bytecode in parallel. A CPU-intensive loop in 4 threads takes the same time as 1 thread (or longer, due to thread-switching overhead). Use ProcessPoolExecutor for CPU work.
2. Using requests with asyncio. The requests library is synchronous — it blocks the entire event loop while waiting for a response. Use aiohttp for async HTTP calls. If you must use requests, wrap it with loop.run_in_executor() to run it in a thread pool.
3. Creating too many concurrent connections. Firing 10,000 simultaneous HTTP requests will overwhelm the target API (rate limiting, IP ban) and your machine (socket exhaustion). Always use an asyncio.Semaphore or set max_workers to a reasonable number (10-50 for APIs, CPU count for processes).
4. Forgetting if __name__ == "__main__" with multiprocessing. On Windows and macOS, multiprocessing spawns new Python processes that re-import your module. Without the guard, the spawned process re-runs the top-level code (including creating more processes) in an infinite fork bomb.
5. Sharing mutable state between threads without locks. Threads share memory. If two threads modify the same dictionary or list simultaneously, you get race conditions — lost updates, corrupted data. Use threading.Lock() for shared state, or better yet, design your tasks to return results instead of modifying shared variables.
6. Passing large DataFrames to ProcessPoolExecutor. Each argument to a process pool task is serialized (pickled) and sent to the child process. A 1 GB DataFrame gets copied entirely, doubling memory usage. Split into chunks and pass lightweight data (lists of dicts, not DataFrames) to child processes.
7. Not handling exceptions in concurrent tasks. If one task in executor.map() raises an exception, it propagates when you iterate results — but only at that position. Other tasks continue running. With asyncio.gather(), use return_exceptions=True to collect errors without crashing the entire batch.
8. Using asyncio for everything. asyncio adds complexity (async/await, event loop, async-compatible libraries). For simple scripts with a few API calls, ThreadPoolExecutor with requests is simpler, more readable, and almost as fast. Reserve asyncio for high-concurrency scenarios (100+ I/O tasks).
Interview Questions
Q: What is the GIL and how does it affect concurrency in Python? A: The Global Interpreter Lock (GIL) is a mutex that allows only one thread to execute Python bytecode at a time, even on multi-core machines. It exists because CPython’s memory management is not thread-safe. The GIL means threading cannot achieve true parallelism for CPU-bound tasks — 4 threads computing in parallel take the same time as 1 thread. However, the GIL is released during I/O operations (network calls, disk reads), so threading effectively parallelizes I/O-bound tasks. For CPU-bound parallelism, use multiprocessing, which creates separate processes each with their own GIL.
Q: When would you use threading vs multiprocessing vs asyncio? A: Threading for moderate I/O-bound concurrency with blocking libraries (10-100 concurrent database queries using pyodbc). Multiprocessing for CPU-bound work (transforming large DataFrames, hashing millions of rows) where you need true parallelism across CPU cores. Asyncio for high-volume I/O-bound concurrency (100-10,000 concurrent HTTP requests using aiohttp) where the overhead of threads becomes significant. The decision starts with whether the task is I/O-bound or CPU-bound.
Q: What is the difference between ThreadPoolExecutor and ProcessPoolExecutor? A: Both provide the same API (submit, map, as_completed) from concurrent.futures. ThreadPoolExecutor uses threads (shared memory, limited by GIL for CPU work, great for I/O). ProcessPoolExecutor uses separate processes (separate memory spaces, bypasses GIL, true parallelism for CPU work). The trade-off: processes have higher startup cost and require serialization (pickling) of arguments and return values, so they are slower for small tasks and data-heavy transfers. Threads share memory and have near-zero overhead.
Q: What is asyncio.gather() and how does it work? A: asyncio.gather() takes multiple coroutines, schedules them all on the event loop, and waits until every one completes. It returns a list of results in the same order as the input coroutines. It runs them concurrently (not sequentially), so the total time is the duration of the slowest coroutine, not the sum of all. Use return_exceptions=True to collect exceptions as results instead of raising the first one immediately.
Q: How do you limit the number of concurrent operations in asyncio? A: Use asyncio.Semaphore(n) where n is the maximum number of concurrent operations. Wrap each task in async with semaphore: so that only n tasks execute simultaneously. When a task finishes and releases the semaphore, the next waiting task starts. This prevents overwhelming APIs with too many simultaneous requests or exhausting system resources like file descriptors and sockets.
Q: Can you mix asyncio with multiprocessing? A: Yes. Use loop.run_in_executor() to offload CPU-intensive work to a ProcessPoolExecutor from within an async function. The async event loop handles I/O concurrently (fetching data from APIs), and when CPU-intensive transformation is needed, it sends the work to a process pool and awaits the result without blocking the event loop. This hybrid pattern gives you the best of both worlds — async I/O and parallel CPU.
Q: How would you process 1,000 CSV files as fast as possible in Python? A: Reading files is I/O-bound, so use ThreadPoolExecutor with 8-16 workers. Each worker reads one file, cleans it, and returns a summary or the cleaned DataFrame. After all workers finish, combine results with pd.concat(). If the files are small and the transformation is heavy (CPU-bound), split the work: use ThreadPoolExecutor for reading (I/O) and ProcessPoolExecutor for transforming (CPU). For very large files, read in chunks within each worker to limit memory usage.
Wrapping Up
Concurrency is a force multiplier for data engineering. A sequential pipeline that takes 30 minutes to fetch 500 API endpoints can run in under a minute with asyncio. A CPU-intensive transformation on 10 million rows can finish 4x faster with multiprocessing. But concurrency also adds complexity — race conditions, pickling errors, event loop gotchas. Start with ThreadPoolExecutor (the simplest upgrade from sequential), move to asyncio when you need high-concurrency I/O, and reach for ProcessPoolExecutor only when you have measured a CPU bottleneck.
Related posts: — ETL Patterns (extract, transform, load) — Working with APIs & HTTP — File Formats (CSV, JSON, Parquet, Excel) — Decorators & Context Managers
Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.