Python Packaging and CLI Tools for Data Engineers: argparse, click, pyproject.toml, setuptools, Entry Points, Building Distributable Packages, and Reusable Pipeline Tools

Table of Contents

Our previous posts built ETL pipelines, database connectors, and cloud storage tools — but they all run as Python scripts: python3 my_pipeline.py. This post teaches you how to turn those scripts into installable command-line tools that your team can run with simple commands like etl run --source orders --target warehouse or datacheck validate --file orders.csv. We also cover packaging — how to structure, build, and distribute your Python code so others can pip install it.

Analogy — From a recipe card to a boxed meal kit. A Python script is like a recipe card — it works, but the user needs to gather ingredients (install dependencies), find the right kitchen (set up the environment), and follow exact steps. A packaged CLI tool is like a meal kit: everything is included, instructions are on the box, and the user just runs one command. This post teaches you how to box up your recipes.

Why Package Your Code?

Data engineers often have a collection of utility scripts scattered across directories. Without packaging, sharing them with teammates means emailing files, copying folders, or saying “just clone the repo and run python3 scripts/transform.py.” Packaging solves this with one command: pip install your-tool.

Without PackagingWith Packaging
`python3 scripts/run_pipeline.py –source orders``etl run –source orders`
Teammates need to clone the repo and find the right script`pip install etl-tools` and the command is available everywhere
Dependencies managed manually with requirements.txtDependencies installed automatically during `pip install`
No versioning — “which version are you running?”Versioned releases: `pip install etl-tools==1.2.0`
Import paths break when moving filesStable import paths: `from etl_tools.transforms import clean`

argparse — Quick CLI Scripts

argparse is Python’s built-in argument parser — no installation needed. It is the right choice for simple scripts with a few arguments. You define what arguments your script accepts, argparse parses them from the command line, validates them, and generates help text automatically.

Analogy — A form at a doctor’s office. The form (argparse) defines which fields exist (arguments), which are required (positional), which are optional (flags), and what type of input each expects (int, string, file path). The receptionist (parser) checks that you filled it out correctly before sending you in.

Basic argparse Usage

# scripts/run_pipeline.py
import argparse

def main():
    parser = argparse.ArgumentParser(
        description="Run an ETL pipeline for a given source"
    )

    # Positional argument (required)
    parser.add_argument("source", help="Data source name (e.g., orders, customers)")

    # Optional arguments with flags
    parser.add_argument("--target", default="warehouse",
                        help="Destination (default: warehouse)")
    parser.add_argument("--date", help="Processing date (YYYY-MM-DD)")
    parser.add_argument("--dry-run", action="store_true",
                        help="Preview without loading")
    parser.add_argument("--chunk-size", type=int, default=10000,
                        help="Rows per chunk (default: 10000)")
    parser.add_argument("-v", "--verbose", action="store_true",
                        help="Enable verbose logging")

    args = parser.parse_args()

    print(f"Source: {args.source}")
    print(f"Target: {args.target}")
    print(f"Date: {args.date}")
    print(f"Dry run: {args.dry_run}")
    print(f"Chunk size: {args.chunk_size}")

if __name__ == "__main__":
    main()
# Run with positional argument
python3 run_pipeline.py orders

# Run with optional flags
python3 run_pipeline.py orders --target staging --date 2026-07-19 --dry-run

# Auto-generated help text
python3 run_pipeline.py --help
# usage: run_pipeline.py [-h] [--target TARGET] [--date DATE] [--dry-run]
#                        [--chunk-size CHUNK_SIZE] [-v]
#                        source

Subcommands with argparse

For tools with multiple actions (like git commit, git push, git log), use subparsers.

import argparse

def cmd_run(args):
    print(f"Running pipeline: {args.source} -> {args.target}")

def cmd_validate(args):
    print(f"Validating: {args.file}")

def cmd_status(args):
    print("All pipelines healthy")

def main():
    parser = argparse.ArgumentParser(description="ETL pipeline manager")
    subparsers = parser.add_subparsers(dest="command", required=True)

    # etl run --source orders --target warehouse
    run_parser = subparsers.add_parser("run", help="Run a pipeline")
    run_parser.add_argument("--source", required=True)
    run_parser.add_argument("--target", default="warehouse")
    run_parser.set_defaults(func=cmd_run)

    # etl validate --file orders.csv
    val_parser = subparsers.add_parser("validate", help="Validate a data file")
    val_parser.add_argument("--file", required=True)
    val_parser.set_defaults(func=cmd_validate)

    # etl status
    status_parser = subparsers.add_parser("status", help="Check pipeline health")
    status_parser.set_defaults(func=cmd_status)

    args = parser.parse_args()
    args.func(args)  # Call the function associated with the subcommand

if __name__ == "__main__":
    main()
python3 etl.py run --source orders --target staging
python3 etl.py validate --file orders.csv
python3 etl.py status
python3 etl.py --help       # Shows all subcommands
python3 etl.py run --help   # Shows run-specific options

click — Professional CLI Framework

click is a third-party library that makes building complex CLIs much cleaner than argparse. It uses decorators instead of manual parser construction, handles types and validation automatically, and supports features like prompts, colors, progress bars, and command groups out of the box.

When to use click vs argparse: Use argparse for simple scripts with 2-5 arguments (no extra dependency). Use click for tools with subcommands, rich output, or anything your team will use daily.

pip install click

Basic click Usage

# cli.py
import click

@click.command()
@click.argument("source")
@click.option("--target", default="warehouse", help="Destination")
@click.option("--date", help="Processing date (YYYY-MM-DD)")
@click.option("--dry-run", is_flag=True, help="Preview without loading")
@click.option("--chunk-size", type=int, default=10000, help="Rows per chunk")
@click.option("-v", "--verbose", is_flag=True, help="Verbose logging")
def run(source, target, date, dry_run, chunk_size, verbose):
    # Run an ETL pipeline for the given source.
    click.echo(f"Source: {source}")
    click.echo(f"Target: {target}")
    if dry_run:
        click.secho("DRY RUN -- no data will be loaded", fg="yellow")

if __name__ == "__main__":
    run()

click Command Groups (Subcommands)

# cli.py
import click

@click.group()
def cli():
    # ETL pipeline manager.
    pass

@cli.command()
@click.option("--source", required=True, help="Data source name")
@click.option("--target", default="warehouse", help="Destination")
@click.option("--dry-run", is_flag=True)
def run(source, target, dry_run):
    # Run a pipeline.
    click.echo(f"Running: {source} -> {target}")
    if dry_run:
        click.secho("DRY RUN mode", fg="yellow")

@cli.command()
@click.argument("filepath", type=click.Path(exists=True))
@click.option("--schema", help="Expected schema file")
def validate(filepath, schema):
    # Validate a data file.
    click.echo(f"Validating: {filepath}")
    # ... validation logic
    click.secho("Validation passed!", fg="green")

@cli.command()
def status():
    # Check pipeline health.
    click.echo("All pipelines healthy")

if __name__ == "__main__":
    cli()
python3 cli.py run --source orders --target staging
python3 cli.py validate orders.csv
python3 cli.py status
python3 cli.py --help

click Features for Data Engineering

import click
import time

@click.command()
@click.argument("files", nargs=-1, type=click.Path(exists=True))
@click.option("--format", "output_format",
              type=click.Choice(["csv", "parquet", "json"]), default="parquet")
@click.option("--confirm", is_flag=True, help="Skip confirmation prompt")
def convert(files, output_format, confirm):
    # Convert files to a target format.
    if not files:
        click.echo("No files provided")
        return

    click.echo(f"Converting {len(files)} files to {output_format}")

    if not confirm:
        click.confirm("Proceed?", abort=True)  # Asks Y/n, aborts on N

    # Progress bar for long operations
    with click.progressbar(files, label="Processing") as bar:
        for f in bar:
            time.sleep(0.5)  # Simulate processing

    click.secho(f"Done! Converted {len(files)} files.", fg="green", bold=True)

if __name__ == "__main__":
    convert()

pyproject.toml — The Modern Project Configuration

pyproject.toml is the standard configuration file for Python projects in 2026. It replaces setup.py, setup.cfg, and requirements.txt with a single, declarative file. Think of it as the birth certificate of your project — it defines the name, version, dependencies, and how to build it.

Project Structure

The recommended layout uses a src/ directory to separate source code from tests and configuration.

etl-tools/
  src/
    etl_tools/
      __init__.py
      cli.py             # CLI entry point
      extractors.py      # Extract functions
      transforms.py      # Transform functions
      loaders.py         # Load functions
      pipeline.py        # Pipeline orchestration
      config.py          # Configuration loading
  tests/
    conftest.py
    test_transforms.py
    test_loaders.py
    test_cli.py
  pyproject.toml         # Project configuration (THE file)
  README.md
  LICENSE

Complete pyproject.toml Example

[build-system]
requires = ["setuptools>=61.0", "wheel"]
build-backend = "setuptools.build_meta"

[project]
name = "etl-tools"
version = "1.0.0"
description = "ETL pipeline tools for data engineering"
readme = "README.md"
license = {text = "MIT"}
requires-python = ">=3.9"
authors = [
    {name = "Naveen Vuppula", email = "naveen@example.com"}
]

dependencies = [
    "pandas>=2.0",
    "sqlalchemy>=2.0",
    "click>=8.0",
    "pyarrow>=12.0",
    "pyyaml>=6.0",
    "requests>=2.28",
]

[project.optional-dependencies]
dev = [
    "pytest>=7.0",
    "pytest-cov>=4.0",
    "black>=23.0",
    "ruff>=0.1.0",
]
aws = ["boto3>=1.28"]
azure = ["azure-storage-blob>=12.0", "azure-identity>=1.12"]

# THIS is what creates the command-line tool
[project.scripts]
etl = "etl_tools.cli:main"
# After pip install, running "etl" in the terminal calls etl_tools.cli:main()

[tool.setuptools.packages.find]
where = ["src"]

[tool.pytest.ini_options]
testpaths = ["tests"]
addopts = "-v --strict-markers"

[tool.black]
line-length = 100

[tool.ruff]
line-length = 100

Entry Points — Creating Installable Commands

The [project.scripts] section in pyproject.toml is the magic that turns your Python function into a terminal command. When someone runs pip install etl-tools, pip reads this section and creates an executable script that calls your function.

Analogy — Installing an app on your phone. When you install an app, an icon appears on your home screen. Tapping the icon launches the app’s main function. [project.scripts] is the equivalent — it creates a “command icon” in your terminal that launches your Python function.

# pyproject.toml
[project.scripts]
etl = "etl_tools.cli:main"           # etl command -> calls main() in etl_tools/cli.py
datacheck = "etl_tools.validate:run"  # datacheck command -> calls run() in etl_tools/validate.py
# src/etl_tools/cli.py
import click

@click.group()
@click.version_option(version="1.0.0")
def main():
    # ETL Tools -- pipeline management CLI.
    pass

@main.command()
@click.option("--source", required=True)
@click.option("--target", default="warehouse")
@click.option("--dry-run", is_flag=True)
def run(source, target, dry_run):
    # Run an ETL pipeline.
    from etl_tools.pipeline import execute_pipeline
    execute_pipeline(source, target, dry_run=dry_run)

@main.command()
@click.argument("filepath", type=click.Path(exists=True))
def validate(filepath):
    # Validate a data file.
    from etl_tools.validate import check_file
    check_file(filepath)
# After pip install:
etl run --source orders --target staging --dry-run
etl validate orders.csv
etl --version
etl --help

Installing in Development Mode

During development, you do not want to rebuild and reinstall the package every time you change a line of code. pip install -e . installs the package in editable mode — it creates a link to your source directory so changes take effect immediately.

# From the project root (where pyproject.toml lives)
pip install -e .

# Install with optional dependencies (dev tools + AWS support)
pip install -e ".[dev,aws]"

# Now you can:
#   1. Run the CLI: etl run --source orders
#   2. Import in Python: from etl_tools.transforms import clean
#   3. Edit src/etl_tools/transforms.py and changes are instant
#   4. Run tests: pytest

Building and Distributing

When your tool is ready for others, build it into a distributable wheel (.whl) file. A wheel is a zip archive that pip can install directly — no compilation needed.

# Install build tools
pip install build

# Build the package (creates dist/ directory)
python -m build
# Creates:
#   dist/etl_tools-1.0.0-py3-none-any.whl   (wheel -- fast install)
#   dist/etl_tools-1.0.0.tar.gz              (source distribution)

# Install from the wheel file
pip install dist/etl_tools-1.0.0-py3-none-any.whl

# Share with teammates: send the .whl file or publish to a private PyPI
# Install from a URL or shared drive
pip install /shared/packages/etl_tools-1.0.0-py3-none-any.whl

# Publish to PyPI (public)
pip install twine
twine upload dist/*
# Now anyone can: pip install etl-tools

Real-World CLI Tool: Data File Inspector

Here is a complete, production-ready CLI tool that inspects data files — a tool every data engineer wishes they had. It shows schema, row count, null counts, sample data, and basic statistics for CSV, Parquet, and JSON files.

# src/etl_tools/inspector.py
import click
import pandas as pd
from pathlib import Path

@click.command()
@click.argument("filepath", type=click.Path(exists=True))
@click.option("--rows", default=5, help="Number of sample rows to show")
@click.option("--stats", is_flag=True, help="Show column statistics")
@click.option("--nulls", is_flag=True, help="Show null counts per column")
def inspect(filepath, rows, stats, nulls):
    # Inspect a data file: schema, sample rows, nulls, and statistics.
    path = Path(filepath)
    ext = path.suffix.lower()

    # Read based on file type
    if ext == ".csv":
        df = pd.read_csv(filepath)
    elif ext == ".parquet":
        df = pd.read_parquet(filepath)
    elif ext in (".json", ".jsonl"):
        df = pd.read_json(filepath, lines=(ext == ".jsonl"))
    else:
        click.secho(f"Unsupported format: {ext}", fg="red")
        raise SystemExit(1)

    # Header
    click.secho(f"\nFile: {filepath}", fg="cyan", bold=True)
    click.echo(f"Format: {ext[1:].upper()}  |  Rows: {len(df):,}  |  Columns: {len(df.columns)}")
    click.echo(f"Memory: {df.memory_usage(deep=True).sum() / 1e6:.1f} MB")

    # Schema
    click.secho("\nSchema:", fg="yellow", bold=True)
    for col in df.columns:
        click.echo(f"  {col:30s}  {str(df[col].dtype):15s}")

    # Sample rows
    click.secho(f"\nSample ({rows} rows):", fg="yellow", bold=True)
    click.echo(df.head(rows).to_string(index=False))

    # Null counts
    if nulls:
        click.secho("\nNull Counts:", fg="yellow", bold=True)
        null_counts = df.isnull().sum()
        for col, count in null_counts.items():
            pct = count / len(df) * 100
            color = "red" if pct > 10 else "yellow" if pct > 0 else "green"
            click.secho(f"  {col:30s}  {count:8,}  ({pct:.1f}%)", fg=color)

    # Statistics
    if stats:
        click.secho("\nStatistics:", fg="yellow", bold=True)
        click.echo(df.describe().to_string())

if __name__ == "__main__":
    inspect()
# After registering as entry point and pip install -e .
datainspect orders.csv
datainspect sales.parquet --stats --nulls
datainspect events.jsonl --rows 10

Real-World CLI Tool: Pipeline Runner

A CLI tool that reads pipeline configuration from YAML and executes the appropriate pipeline — used to run any pipeline from a single command.

# src/etl_tools/runner.py
import click
import yaml
import logging
from pathlib import Path
from datetime import datetime

@click.group()
@click.option("--config-dir", default="configs/",
              type=click.Path(exists=True), help="Config directory")
@click.pass_context
def cli(ctx, config_dir):
    # Pipeline runner -- execute ETL pipelines from config files.
    ctx.ensure_object(dict)
    ctx.obj["config_dir"] = config_dir

@cli.command()
@click.argument("pipeline_name")
@click.option("--date", default=None, help="Processing date (YYYY-MM-DD)")
@click.option("--dry-run", is_flag=True)
@click.pass_context
def run(ctx, pipeline_name, date, dry_run):
    # Run a pipeline by name.
    config_path = Path(ctx.obj["config_dir"]) / f"{pipeline_name}.yaml"
    if not config_path.exists():
        click.secho(f"Config not found: {config_path}", fg="red")
        raise SystemExit(1)

    with open(config_path) as f:
        config = yaml.safe_load(f)

    process_date = date or datetime.now().strftime("%Y-%m-%d")
    click.echo(f"Running pipeline: {pipeline_name}")
    click.echo(f"Config: {config_path}")
    click.echo(f"Date: {process_date}")

    if dry_run:
        click.secho("DRY RUN -- no data will be loaded", fg="yellow")
        click.echo(f"Source: {config.get('source', {})}")
        click.echo(f"Target: {config.get('destination', {})}")
        return

    # Execute the pipeline
    from etl_tools.pipeline import execute_from_config
    result = execute_from_config(config, process_date)
    click.secho(f"Complete: {result['rows_loaded']} rows in {result['duration']}s", fg="green")

@cli.command(name="list")
@click.pass_context
def list_pipelines(ctx):
    # List all available pipeline configs.
    config_dir = Path(ctx.obj["config_dir"])
    configs = sorted(config_dir.glob("*.yaml"))
    click.echo(f"Available pipelines ({len(configs)}):")
    for c in configs:
        click.echo(f"  {c.stem}")

if __name__ == "__main__":
    cli()
# List available pipelines
pipeline list

# Run a pipeline
pipeline run orders --date 2026-07-19
pipeline run customers --dry-run

# Use a custom config directory
pipeline --config-dir /etc/etl/configs/ run orders

Testing CLI Tools

click provides CliRunner for testing CLI commands without running them in a subprocess. This is fast, isolated, and captures output for assertions.

# tests/test_cli.py
from click.testing import CliRunner
from etl_tools.inspector import inspect

def test_inspect_csv(tmp_path):
    # Create a test CSV
    csv_path = tmp_path / "test.csv"
    csv_path.write_text("id,name,amount\n1,Alice,29.99\n2,Bob,49.99\n")

    runner = CliRunner()
    result = runner.invoke(inspect, [str(csv_path)])

    assert result.exit_code == 0
    assert "Rows: 2" in result.output
    assert "Columns: 3" in result.output
    assert "id" in result.output

def test_inspect_with_nulls(tmp_path):
    csv_path = tmp_path / "test.csv"
    csv_path.write_text("id,name\n1,Alice\n2,\n")

    runner = CliRunner()
    result = runner.invoke(inspect, [str(csv_path), "--nulls"])

    assert result.exit_code == 0
    assert "Null Counts" in result.output

def test_inspect_unsupported_format(tmp_path):
    txt_path = tmp_path / "test.txt"
    txt_path.write_text("hello")

    runner = CliRunner()
    result = runner.invoke(inspect, [str(txt_path)])

    assert result.exit_code == 1
    assert "Unsupported format" in result.output

argparse vs click Comparison

Featureargparseclick
**Installation**Built-in (no dependency)`pip install click`
**Syntax**Imperative (add_argument, parse_args)Declarative (decorators)
**Subcommands**Verbose (add_subparsers, set_defaults)Clean (`@group`, `@command`)
**Type validation**Manual (type=int, choices=[])Automatic (type=click.INT, click.Choice)
**File validation**Manual (os.path.exists)Built-in (click.Path(exists=True))
**Colored output**Not built-in (needs colorama)Built-in (click.secho)
**Progress bars**Not built-in (needs tqdm)Built-in (click.progressbar)
**Testing**Subprocess or manualCliRunner (fast, isolated)
**Best for**Simple scripts, no dependenciesProfessional tools, team usage

Common Mistakes

1. Putting code in setup.py instead of pyproject.toml. setup.py is the old way. In 2026, pyproject.toml with [build-system] and [project] tables is the standard. It is declarative (no Python code to execute), supports all tools (pytest, black, ruff), and is the format every modern tutorial uses.

2. Not using the src/ layout. Without src/, Python can import your package from the project root even without installing it, masking import errors. The src/ layout forces you to install the package (even in editable mode), ensuring your import paths match what users will experience after pip install.

3. Forgetting [project.scripts] entry points. Without entry points, users must run python3 -m etl_tools.cli or python3 src/etl_tools/cli.py. With entry points, they just type etl. This is the difference between a script and a tool.

4. Hardcoding paths inside CLI tools. A CLI tool that reads from /home/naveen/data/orders.csv only works on your machine. Accept paths as arguments or options (click.Path(exists=True)), and use config files for environment-specific values.

5. Not setting requires-python in pyproject.toml. Without it, someone on Python 3.7 can install your package and get cryptic syntax errors from f-strings or walrus operators. Setting requires-python = ">=3.9" prevents installation on incompatible versions.

6. Mixing business logic with CLI code. Your cli.py should only parse arguments and call functions from other modules. The actual pipeline logic belongs in pipeline.py, transforms.py, etc. This keeps business logic testable without needing to invoke the CLI.

7. Not returning exit codes. A CLI tool should return 0 on success and non-zero on failure. Scripts that call your tool (cron jobs, CI/CD) check the exit code to decide next steps. Use raise SystemExit(1) for errors or sys.exit(1) with click.

8. Publishing to PyPI without testing the wheel. Always pip install dist/your_package.whl in a fresh virtual environment and verify the command works before publishing. A broken wheel on PyPI is embarrassing and hard to retract.

Interview Questions

Q: What is pyproject.toml and why is it the modern standard? A: pyproject.toml is a single configuration file that defines a Python project’s metadata (name, version, dependencies), build system (setuptools, hatchling), and tool configurations (pytest, black, ruff). It replaces the scattered approach of setup.py + setup.cfg + requirements.txt + pytest.ini + .flake8. Standardized in PEP 621, it is declarative (no Python code), human-readable (TOML format), and supported by all modern packaging tools.

Q: What is the difference between argparse and click? A: argparse is Python’s built-in argument parser — imperative style (add_argument calls), no external dependency, suitable for simple scripts. click is a third-party library — declarative style (decorators), with built-in features for colors, progress bars, file validation, prompts, and testability (CliRunner). Use argparse for scripts with 2-5 flags and no dependency requirement. Use click for tools with subcommands, team usage, or any CLI that needs professional polish.

Q: What are entry points in Python packaging? A: Entry points are declarations in pyproject.toml (under [project.scripts]) that tell pip to create executable commands when the package is installed. For example, etl = "etl_tools.cli:main" means installing the package creates a command etl that calls the main() function in etl_tools/cli.py. This lets users type etl run --source orders instead of python3 -m etl_tools.cli run --source orders.

Q: What is pip install -e . and when do you use it? A: The -e flag installs a package in editable (development) mode. Instead of copying files into site-packages, pip creates a link to your source directory. Any changes you make to the source files take effect immediately without reinstalling. Use it during development so you can edit code and test CLI commands without rebuilding. The . refers to the current directory (where pyproject.toml lives).

Q: Why use the src/ layout instead of putting code at the project root? A: The src/ layout prevents accidental imports from the project root. Without src/, Python can import your package from the current directory even when it is not installed, hiding import errors that users would encounter after pip install. With src/, you must install the package (at least in editable mode) for imports to work, ensuring your tests use the same import paths as production.

Q: How do you test a CLI tool built with click? A: Use click’s CliRunner, which invokes your CLI command in isolation without running a subprocess. You call runner.invoke(command, [args]) and check result.exit_code and result.output. This is fast (no process overhead), isolated (no side effects on the real filesystem unless you use tmp_path), and deterministic. For argparse-based CLIs, you can test the underlying functions directly or use subprocess.

Q: How would you distribute a Python CLI tool to your team? A: Build a wheel with python -m build, which creates a .whl file in the dist/ directory. Share the wheel via a shared drive, artifact repository (Artifactory, CodeArtifact), or private PyPI server. Teammates install with pip install path/to/etl_tools-1.0.0.whl. For public tools, publish to PyPI with twine upload dist/*. For internal teams, a private PyPI (like AWS CodeArtifact or Azure Artifacts) is the standard approach.

Wrapping Up

Packaging and CLI tools are the final step in professionalizing your Python data engineering work. Start with argparse for quick scripts, graduate to click for team-facing tools, configure everything in pyproject.toml, and distribute with wheels. The difference between “clone my repo and run python3 scripts/pipeline.py with these 7 arguments” and “pip install etl-tools and run etl run –source orders” is the difference between a script and a product.

Related posts:ETL Patterns (extract, transform, load)Testing with pytestModules, Packages & Virtual EnvironmentsLogging (levels, formatters, handlers)



Naveen Vuppula is a Senior Data Engineering Consultant and app developer based in Ontario, Canada. He writes about Python, SQL, AWS, Azure, and everything data engineering at DriveDataScience.com.

Leave a Comment

Your email address will not be published. Required fields are marked *

Scroll to Top
Share via
Copy link