Skip to content

Python

Language Overview

Python is a high-level, interpreted, multi-paradigm programming language known for its readability, simplicity, and extensive ecosystem. Python is widely used in DevOps for automation, infrastructure management, data processing, and web services.

Key Characteristics

  • Paradigm: Multi-paradigm (object-oriented, functional, procedural, imperative)
  • Typing: Dynamically typed with optional static type hints (PEP 484)
  • Runtime: CPython interpreter (default), also PyPy, Jython, IronPython
  • Primary Use Cases:
  • Infrastructure automation and configuration management
  • CI/CD pipeline scripting and orchestration
  • API development (FastAPI, Flask, Django)
  • Data processing and analysis
  • Cloud automation (AWS boto3, Azure SDK, Google Cloud)
  • Testing and validation frameworks

This Style Guide Covers

  • PEP 8 compliance with modern best practices
  • Type hints and static type checking
  • Documentation standards (docstrings, comments)
  • Testing requirements and coverage
  • Security best practices for DevOps
  • Performance optimization patterns
  • Tool configuration (Black, Flake8, mypy, pytest)

Supported Versions

Version Support Status EOL Date Recommended
3.13.x Active 2029-10 ✅ Yes
3.12.x Active 2028-10 ✅ Yes
3.11.x Active 2027-10 ✅ Yes
3.10.x Security 2026-10 ⚠️ Maintenance
3.9.x EOL 2025-10 ❌ No
3.8.x EOL 2024-10 ❌ No

Recommendation: Use Python 3.11+ for new projects. Python 3.10+ is supported but consider upgrading to get the latest performance improvements and features.

EOL Policy: We recommend upgrading before the EOL date to continue receiving security updates. Once a version reaches EOL, it will no longer receive security patches from the Python core team.

Version Features:

  • Python 3.13: Performance improvements, better error messages, experimental JIT compiler
  • Python 3.12: Improved error messages, performance enhancements, typing improvements
  • Python 3.11: 10-25% faster than 3.10, better error locations, exception groups
  • Python 3.10: Pattern matching (match/case), better error messages, union types with |

Quick Reference

Category Convention Example Notes
Naming
Variables snake_case user_count, api_response Descriptive, lowercase with underscores
Constants UPPER_SNAKE_CASE MAX_RETRIES, API_URL Module-level constants, all uppercase
Functions snake_case get_user(), validate_input() Verbs, descriptive action names
Classes PascalCase UserProfile, DataProcessor Nouns, capitalize each word
Methods snake_case calculate_total(), is_valid() Like functions, instance/class methods
Modules snake_case user_manager.py, api_client.py Short, lowercase, no hyphens
Packages snake_case data_utils/, auth_service/ Short, lowercase, avoid underscores if possible
Private _leading_underscore _internal_method(), _cache Indicates internal use only
Formatting
Line Length 88 characters # Black default Max 88 (Black), 79 (PEP 8) acceptable
Indentation 4 spaces if condition: Never tabs, always 4 spaces
Blank Lines 2 between top-level class Foo:\n\n\nclass Bar: 2 blank lines between classes/functions
String Quotes Double quotes "hello world" Prefer double, single for avoiding escapes
Imports
Order stdlib, 3rd-party, local import os\nimport boto3\nfrom .utils import x Alphabetical within each group
Style Absolute imports from myapp.utils import helper Avoid relative imports except in packages
Documentation
Docstrings """Triple double quotes""" """Returns user by ID.""" All public modules, classes, functions
Type Hints Required for functions def foo(x: int) -> str: All function signatures
Files
Naming snake_case.py user_service.py, __init__.py Lowercase, underscores, .py extension
Encoding UTF-8 # -*- coding: utf-8 -*- Default, explicit if non-ASCII

Naming Conventions

Variables

Convention: snake_case

## Good
user_count = 10
max_retry_attempts = 3
api_response_data = fetch_data()

## Bad
UserCount = 10  # PascalCase for variables
maxRetryAttempts = 3  # camelCase
apiresponsedata = fetch_data()  # No separation

Guidelines:

  • Use descriptive names that indicate purpose
  • Avoid single-letter names except loop counters (i, j, k) and comprehensions
  • Boolean variables should ask a question: is_active, has_permission, should_retry
  • Avoid abbreviations unless universally understood (http, api, url)

Constants

Convention: UPPER_SNAKE_CASE

## Good
MAX_CONNECTION_POOL_SIZE = 100
API_BASE_URL = "https://api.example.com"
DEFAULT_TIMEOUT_SECONDS = 30

## Bad
max_connection_pool_size = 100  # Looks like a variable
MaxConnectionPoolSize = 100  # Not a constant style

Functions and Methods

Convention: snake_case

## Good
def get_user_by_id(user_id: int) -> User:
    """Retrieve user from database by ID."""
    return database.query(User).filter(User.id == user_id).first()

def calculate_monthly_cost(instances: List[Instance]) -> Decimal:
    """Calculate total monthly cost for EC2 instances."""
    return sum(instance.hourly_rate * 730 for instance in instances)

## Bad
def GetUserById(user_id: int):  # PascalCase
    pass

def calcCost(inst):  # camelCase, abbreviations
    pass

Guidelines:

  • Use verb-noun format: get_user(), calculate_total(), validate_input()
  • Keep names concise but descriptive (avoid process(), handle(), do_stuff())
  • Private methods start with single underscore: _internal_helper()
  • Name-mangled methods start with double underscore: __private_method()

Classes

Convention: PascalCase

## Good
class UserRepository:
    """Handles database operations for User entities."""
    pass

class AWSResourceManager:
    """Manages AWS resources lifecycle."""
    pass

class HTTPConnectionPool:
    """Pool of reusable HTTP connections."""
    pass

## Bad
class user_repository:  # snake_case
    pass

class awsResourceManager:  # camelCase
    pass

Guidelines:

  • Use noun phrases: User, PaymentProcessor, ConfigValidator
  • Exception classes end with Error or Exception: ValidationError, ConfigurationException
  • Abstract base classes can prefix with Abstract or Base: AbstractRepository, BaseHandler

Files and Modules

Convention: snake_case.py

## Good
user_repository.py
aws_resource_manager.py
http_client.py

## Bad
UserRepository.py  # PascalCase
awsResourceManager.py  # camelCase
httpClient.py  # camelCase

Guidelines:

  • Match file name to primary class when file contains single class: user.py contains class User
  • Use __init__.py for package initialization
  • Test files: test_<module>.py or <module>_test.py

Code Formatting

Indentation

  • Style: Spaces only (no tabs)
  • Size: 4 spaces per indentation level
## Good
def process_data(items):
    for item in items:
        if item.is_valid:
            result = transform(item)
            save(result)

## Bad - 2 spaces
def process_data(items):
  for item in items:
    if item.is_valid:
      result = transform(item)

Line Length

  • Maximum: 88 characters (Black default) or 100 characters
  • Exception: Long strings, URLs, import statements can exceed
## Good - line broken appropriately
user_data = database.query(User).filter(
    User.is_active == True,
    User.created_at > start_date
).all()

## Good - long URL on its own line
API_ENDPOINT = (
    "https://api.example.com/v2/resources/users/search?filter=active&limit=100"
)

## Bad - line too long
user_data = database.query(User).filter(User.is_active == True, User.created_at > start_date, User.department == "Engineering").all()

Blank Lines

  • Between top-level functions and classes: 2 blank lines
  • Between methods in a class: 1 blank line
  • Within functions: Use sparingly to separate logical blocks
  • File end: Exactly 1 blank line
import os

def function_one():
    """First function."""
    pass

def function_two():
    """Second function."""
    pass

class MyClass:
    """Example class."""

    def method_one(self):
        """First method."""
        pass

    def method_two(self):
        """Second method."""
        pass

Imports

Order: Standard library, third-party, local modules

## Good - organized imports
import os
import sys
from pathlib import Path

import boto3
import requests
from fastapi import FastAPI, HTTPException

from app.models.user import User
from app.services.auth import AuthService
from app.utils.validators import validate_email

## Bad - mixed order, grouped incorrectly
from app.models.user import User
import requests
import os
from fastapi import FastAPI

Guidelines:

  • Use absolute imports for better clarity
  • Group imports with blank lines between groups
  • Use isort to automatically organize imports
  • Avoid wildcard imports: from module import * (except in __init__.py when appropriate)

Documentation Standards

Module-Level Documentation

Required for: All Python files

"""
@module user_authentication
@description Handles user authentication, session management, and JWT token generation
@dependencies fastapi, pyjwt, passlib, python-dotenv
@version 1.2.0
@author Tyler Dukes
@last_updated 2025-10-28
@status stable
@security_classification internal
@python_version >= 3.9
"""

import jwt
from fastapi import APIRouter, HTTPException
from passlib.context import CryptContext

Function/Method Documentation

Required for: All public functions and complex logic

def authenticate_user(username: str, password: str) -> Optional[User]:
    """
    Authenticate user credentials and return user object if valid.

    Args:
        username: User's username or email address
        password: Plain text password to verify

    Returns:
        User object if authentication succeeds, None otherwise

    Raises:
        DatabaseError: If database connection fails
        ValidationError: If username format is invalid

    Example:
        >>> user = authenticate_user("john@example.com", "secret123")
        >>> if user:
        ...     print(f"Welcome {user.name}")
    """
    if not validate_username(username):
        raise ValidationError("Invalid username format")

    user = get_user_by_username(username)
    if user and verify_password(password, user.password_hash):
        return user
    return None

Type Hints

Required for: All function signatures in production code

from typing import List, Dict, Optional, Union, Tuple

## Good - comprehensive type hints
def get_active_users(
    department: str,
    limit: int = 100,
    include_archived: bool = False
) -> List[User]:
    """Get list of active users from specified department."""
    pass

def parse_config(
    config_path: Path
) -> Dict[str, Union[str, int, bool]]:
    """Parse configuration file and return settings dictionary."""
    pass

## Bad - no type hints
def get_active_users(department, limit=100):
    pass

Error Handling

Exception Handling

Strategy: Fail-fast, raise specific exceptions, clean up resources

## Good - specific exceptions and cleanup
def fetch_remote_data(url: str) -> Dict:
    """Fetch data from remote API with retry logic."""
    try:
        response = requests.get(url, timeout=10)
        response.raise_for_status()
        return response.json()
    except requests.Timeout:
        logger.error(f"Timeout fetching data from {url}")
        raise APITimeoutError(f"Request to {url} timed out")
    except requests.HTTPError as e:
        logger.error(f"HTTP error {e.response.status_code}: {url}")
        raise APIError(f"Failed to fetch data: {e}")
    except ValueError:
        logger.error(f"Invalid JSON response from {url}")
        raise DataFormatError("Response is not valid JSON")

## Bad - catching generic Exception
def fetch_remote_data(url):
    try:
        response = requests.get(url)
        return response.json()
    except Exception:  # Too broad
        pass  # Silent failure

Custom Exceptions

class APIError(Exception):
    """Base exception for API-related errors."""
    pass

class APITimeoutError(APIError):
    """Raised when API request times out."""
    pass

class DataFormatError(APIError):
    """Raised when API response has invalid format."""
    pass

## Usage
try:
    data = fetch_remote_data("https://api.example.com/users")
except APITimeoutError:
    # Handle timeout specifically
    use_cached_data()
except APIError as e:
    # Handle other API errors
    logger.error(f"API error: {e}")

Context Managers

Use for: Resource cleanup (files, connections, locks)

## Good - guaranteed cleanup
from contextlib import contextmanager

@contextmanager
def database_session():
    """Context manager for database sessions."""
    session = Session()
    try:
        yield session
        session.commit()
    except Exception:
        session.rollback()
        raise
    finally:
        session.close()

## Usage
with database_session() as session:
    user = session.query(User).first()
    user.last_login = datetime.now()

Testing Requirements

Coverage Requirements

  • Unit Tests: 80%+ coverage for business logic
  • Integration Tests: All API endpoints and external integrations
  • Test Files: Located in tests/ directory, mirror source structure

Test Naming

Convention: test_should_<behavior>_when_<condition>

def test_should_return_user_when_valid_id_provided():
    """Test get_user_by_id returns user for valid ID."""
    user_id = 123
    user = get_user_by_id(user_id)
    assert user.id == user_id
    assert user is not None

def test_should_raise_error_when_user_not_found():
    """Test get_user_by_id raises NotFoundError for invalid ID."""
    with pytest.raises(NotFoundError):
        get_user_by_id(999999)

Test Structure

Pattern: Arrange-Act-Assert

import pytest
from app.services.user_service import UserService

@pytest.fixture
def user_service():
    """Fixture providing UserService instance."""
    return UserService(database_url="sqlite:///:memory:")

def test_should_create_user_with_valid_data(user_service):
    # Arrange
    user_data = {
        "username": "john_doe",
        "email": "john@example.com",
        "password": "secure_password123"
    }

    # Act
    user = user_service.create_user(**user_data)

    # Assert
    assert user.username == "john_doe"
    assert user.email == "john@example.com"
    assert user.password != "secure_password123"  # Should be hashed

Mocking

from unittest.mock import Mock, patch, MagicMock

## Good - mock external dependencies
@patch('app.services.email.send_email')
def test_should_send_welcome_email_after_signup(mock_send_email):
    """Test welcome email is sent after user signup."""
    user_service = UserService()
    user = user_service.signup("john@example.com", "password123")

    mock_send_email.assert_called_once_with(
        to=user.email,
        subject="Welcome!",
        template="welcome"
    )

Security Best Practices

Input Validation

from pydantic import BaseModel, EmailStr, validator

class UserCreate(BaseModel):
    """Validated user creation request."""
    username: str
    email: EmailStr
    password: str

    @validator('username')
    def username_alphanumeric(cls, v):
        """Ensure username is alphanumeric."""
        if not v.isalnum():
            raise ValueError('Username must be alphanumeric')
        return v

    @validator('password')
    def password_strength(cls, v):
        """Ensure password meets minimum requirements."""
        if len(v) < 8:
            raise ValueError('Password must be at least 8 characters')
        return v

SQL Injection Prevention

## Good - parameterized queries
from sqlalchemy import text

def get_user_by_email(email: str) -> Optional[User]:
    """Get user by email using parameterized query."""
    query = text("SELECT * FROM users WHERE email = :email")
    result = db.execute(query, {"email": email})
    return result.first()

## Bad - string concatenation (NEVER DO THIS)
def get_user_by_email(email: str):
    query = f"SELECT * FROM users WHERE email = '{email}'"  # Vulnerable!
    return db.execute(query)

Secret Management

import os
from functools import lru_cache

## Good - environment variables
@lru_cache()
def get_settings():
    """Get application settings from environment."""
    return {
        "database_url": os.getenv("DATABASE_URL"),
        "api_key": os.getenv("API_KEY"),
        "secret_key": os.getenv("SECRET_KEY")
    }

## Bad - hardcoded secrets (NEVER DO THIS)
DATABASE_URL = "postgresql://user:password@localhost/db"  # Exposed!
API_KEY = "sk_live_abc123xyz..."  # Committed to git!

Formatters

  • Black: Opinionated code formatter
  • Installation: pip install black
  • Configuration: pyproject.toml

  • Run: black .

  • isort: Import statement organizer

  • Installation: pip install isort
  • Run: isort .

Linters

  • Flake8: Style guide enforcement
  • Installation: pip install flake8
  • Configuration: .flake8 or setup.cfg
  • Run: flake8 .
# .flake8
[flake8]
max-line-length = 100
extend-ignore = E203, W503
exclude = .git, __pycache__, .venv, dist, build
per-file-ignores =
    tests/*: S101
  • Pylint: Comprehensive code analysis
  • Installation: pip install pylint
  • Run: pylint src/
# .pylintrc (generate full file: pylint --generate-rcfile > .pylintrc)
[MESSAGES CONTROL]
disable = C0114, C0115, C0116

[FORMAT]
max-line-length = 100

Type Checkers

  • mypy: Static type checker
  • Installation: pip install mypy
  • Configuration: mypy.ini
  • Run: mypy src/
# mypy.ini
[mypy]
python_version = 3.11
warn_return_any = True
warn_unused_configs = True
disallow_untyped_defs = True
ignore_missing_imports = True

Testing

  • pytest: Testing framework
  • Installation: pip install pytest pytest-cov
  • Run: pytest --cov=src tests/

Pre-commit Configuration

## .pre-commit-config.yaml
repos:
  - repo: https://github.com/psf/black
    rev: 24.10.0
    hooks:
      - id: black
        language_version: python3.11

  - repo: https://github.com/pycqa/isort
    rev: 5.13.2
    hooks:
      - id: isort
        args: ["--profile", "black"]

  - repo: https://github.com/pycqa/flake8
    rev: 7.1.1
    hooks:
      - id: flake8
        args: ["--max-line-length=88", "--extend-ignore=E203"]

  - repo: https://github.com/pre-commit/mirrors-mypy
    rev: v1.11.2
    hooks:
      - id: mypy
        additional_dependencies: [types-requests]

Complete Example

FastAPI Application with Best Practices

"""
@module user_api
@description RESTful API for user management with authentication
@dependencies fastapi, pydantic, sqlalchemy, python-jose
@version 1.0.0
@author Tyler Dukes
@last_updated 2025-10-28
@status stable
@api_endpoints POST /users, GET /users/{id}, PUT /users/{id}, DELETE /users/{id}
"""

from typing import List, Optional
from datetime import datetime

from fastapi import FastAPI, HTTPException, Depends, status
from pydantic import BaseModel, EmailStr
from sqlalchemy.orm import Session

from app.database import get_db
from app.models.user import User
from app.services.auth import get_current_user

app = FastAPI(title="User Management API")

class UserCreate(BaseModel):
    """Schema for user creation."""
    username: str
    email: EmailStr
    password: str

class UserResponse(BaseModel):
    """Schema for user response."""
    id: int
    username: str
    email: EmailStr
    created_at: datetime

    class Config:
        """Pydantic configuration."""
        from_attributes = True

@app.post("/users", response_model=UserResponse, status_code=status.HTTP_201_CREATED)
def create_user(
    user_data: UserCreate,
    db: Session = Depends(get_db)
) -> UserResponse:
    """
    Create new user account.

    Args:
        user_data: User creation data
        db: Database session

    Returns:
        Created user information

    Raises:
        HTTPException: If username or email already exists
    """
    # Check for existing user
    existing_user = db.query(User).filter(
        (User.username == user_data.username) | (User.email == user_data.email)
    ).first()

    if existing_user:
        raise HTTPException(
            status_code=status.HTTP_409_CONFLICT,
            detail="Username or email already exists"
        )

    # Create new user
    user = User(
        username=user_data.username,
        email=user_data.email,
        password_hash=hash_password(user_data.password),
        created_at=datetime.utcnow()
    )

    db.add(user)
    db.commit()
    db.refresh(user)

    return user

@app.get("/users/{user_id}", response_model=UserResponse)
def get_user(
    user_id: int,
    db: Session = Depends(get_db),
    current_user: User = Depends(get_current_user)
) -> UserResponse:
    """
    Get user by ID (requires authentication).

    Args:
        user_id: User ID to retrieve
        db: Database session
        current_user: Authenticated user

    Returns:
        User information

    Raises:
        HTTPException: If user not found
    """
    user = db.query(User).filter(User.id == user_id).first()

    if not user:
        raise HTTPException(
            status_code=status.HTTP_404_NOT_FOUND,
            detail=f"User {user_id} not found"
        )

    return user

Common Pitfalls

Late Binding in Closures

Issue: Lambda functions and closures bind variables by reference, not value, causing unexpected behavior in loops.

Example:

## Bad - All functions reference same 'i'
functions = []
for i in range(5):
    functions.append(lambda: i)  # Binds to 'i' reference

for f in functions:
    print(f())  # Prints: 4 4 4 4 4 (not 0 1 2 3 4)

Solution: Use default arguments to capture values at definition time.

## Good - Capture value with default argument
functions = []
for i in range(5):
    functions.append(lambda x=i: x)  # Captures current value of i

for f in functions:
    print(f())  # Prints: 0 1 2 3 4

## Good - Use list comprehension
functions = [lambda x=i: x for i in range(5)]

Key Points:

  • Closures bind variables by reference, not value
  • Use default arguments lambda x=i: x to capture values
  • List comprehensions create new scope per iteration
  • Affects lambdas, nested functions, and class definitions

Import Circular Dependencies

Issue: Two modules importing each other causes ImportError or incomplete module initialization.

Example:

## Bad - module_a.py
from module_b import function_b  # Imports module_b

def function_a():
    return function_b()

## Bad - module_b.py
from module_a import function_a  # Imports module_a (circular!)

def function_b():
    return function_a()  # ImportError or AttributeError

Solution: Restructure code, use local imports, or import modules rather than functions.

## Good - Import module, not function
## module_a.py
import module_b  # Import module, not specific function

def function_a():
    return module_b.function_b()  # Access via module

## Good - Local import (deferred)
## module_b.py
def function_b():
    from module_a import function_a  # Import only when needed
    return function_a()

## Better - Refactor to remove circular dependency
## shared.py (new file)
def shared_function():
    pass

## module_a.py
from shared import shared_function

## module_b.py
from shared import shared_function

Key Points:

  • Circular imports fail or create incomplete modules
  • Import modules, not functions: import module vs from module import func
  • Use local imports as temporary workaround
  • Best solution: refactor to remove circular dependency
  • Extract shared code to third module

Dictionary Iteration During Modification

Issue: Modifying a dictionary while iterating causes RuntimeError in Python 3.

Example:

## Bad - RuntimeError: dictionary changed size during iteration
user_data = {"alice": 25, "bob": 30, "charlie": 35}
for key in user_data:
    if user_data[key] > 28:
        del user_data[key]  # RuntimeError!

Solution: Iterate over a copy or build a new dictionary.

## Good - Iterate over copy of keys
user_data = {"alice": 25, "bob": 30, "charlie": 35}
for key in list(user_data.keys()):  # list() creates copy
    if user_data[key] > 28:
        del user_data[key]

## Good - Dictionary comprehension (most Pythonic)
user_data = {"alice": 25, "bob": 30, "charlie": 35}
user_data = {k: v for k, v in user_data.items() if v <= 28}

## Good - Filter and rebuild
user_data = dict(filter(lambda item: item[1] <= 28, user_data.items()))

Key Points:

  • Cannot modify dictionary during iteration
  • Use list(dict.keys()) to iterate over copy
  • Dictionary comprehension is cleaner and faster
  • Applies to sets as well (not lists, which support modification)

Asyncio Event Loop Blocking

Issue: Calling blocking I/O or CPU-intensive code in async functions blocks the entire event loop.

Example:

## Bad - Blocks event loop for 5 seconds
import asyncio
import time

async def fetch_data():
    time.sleep(5)  # Blocks entire event loop!
    return "data"

async def main():
    # All tasks blocked while one sleeps
    await asyncio.gather(
        fetch_data(),
        fetch_data(),
        fetch_data()
    )  # Takes 15 seconds sequentially, not 5 seconds concurrently

Solution: Use async-compatible libraries or run blocking code in executor.

## Good - Use asyncio.sleep for async operations
import asyncio

async def fetch_data():
    await asyncio.sleep(5)  # Non-blocking sleep
    return "data"

async def main():
    # Runs concurrently
    await asyncio.gather(
        fetch_data(),
        fetch_data(),
        fetch_data()
    )  # Takes 5 seconds total

## Good - Run blocking code in executor
import asyncio
from concurrent.futures import ThreadPoolExecutor

def blocking_operation():
    time.sleep(5)
    return "data"

async def fetch_data():
    loop = asyncio.get_event_loop()
    result = await loop.run_in_executor(None, blocking_operation)
    return result

Key Points:

  • time.sleep() blocks event loop; use await asyncio.sleep()
  • Blocking I/O prevents other tasks from running
  • Use aiohttp, aiomysql, etc. for async I/O
  • Run blocking code in executor: loop.run_in_executor()
  • CPU-intensive work should use ProcessPoolExecutor

List Comprehension Memory Issues

Issue: List comprehensions load entire result into memory, causing MemoryError with large datasets.

Example:

## Bad - Loads 100 million integers into memory
result = [i * 2 for i in range(100_000_000)]  # MemoryError!
total = sum(result)

## Bad - Reads entire large file into memory
lines = [line.strip() for line in open("huge_log.txt")]  # MemoryError!

Solution: Use generator expressions for lazy evaluation.

## Good - Generator expression (lazy evaluation)
result = (i * 2 for i in range(100_000_000))  # Parentheses, not brackets
total = sum(result)  # Processes one item at a time

## Good - Generator for file processing
with open("huge_log.txt") as f:
    lines = (line.strip() for line in f)  # Lazy evaluation
    for line in lines:
        process(line)  # One line at a time

## Good - When you need a list, consider chunking
def process_in_chunks(iterable, chunk_size=10000):
    chunk = []
    for item in iterable:
        chunk.append(item)
        if len(chunk) >= chunk_size:
            yield chunk
            chunk = []
    if chunk:
        yield chunk

Key Points:

  • List comprehensions [...] load everything into memory
  • Generator expressions (...) evaluate lazily
  • Use generators for large datasets
  • Generator expressions can only be iterated once
  • Convert to list only when necessary: list(generator)

Exception Handling Anti-Pattern

Issue: Catching broad exceptions hides bugs and makes debugging impossible.

Example:

## Bad - Catches KeyboardInterrupt, SystemExit, etc.
try:
    result = risky_operation()
    process(result)
except:  # Catches EVERYTHING, including Ctrl+C!
    pass  # Silent failure

## Bad - Too broad exception handling
try:
    data = json.loads(response.text)
    user_id = data["user"]["id"]
except Exception as e:  # Catches ALL exceptions
    return None  # Hides TypeError, KeyError, JSONDecodeError

Solution: Catch specific exceptions and handle appropriately.

## Good - Catch specific exceptions
import json
from requests.exceptions import RequestException

try:
    response = requests.get(url)
    response.raise_for_status()
    data = response.json()
    user_id = data["user"]["id"]
except RequestException as e:
    logger.error(f"Network error: {e}")
    raise
except json.JSONDecodeError as e:
    logger.error(f"Invalid JSON: {e}")
    raise
except KeyError as e:
    logger.error(f"Missing key in response: {e}")
    raise

## Good - Allow critical exceptions to propagate
try:
    result = operation()
except (ValueError, TypeError) as e:  # Only catch expected errors
    logger.warning(f"Operation failed: {e}")
    result = fallback_value

Key Points:

  • Never use bare except: (catches KeyboardInterrupt, SystemExit)
  • Avoid catching Exception unless truly necessary
  • Catch specific exceptions: ValueError, KeyError, etc.
  • Log exceptions before swallowing them
  • Use raise to re-raise after logging

Anti-Patterns

Mutable Default Arguments

Problem: Using mutable objects (lists, dicts) as default arguments causes unexpected behavior.

Bad:

def add_item(item, items=[]):  # ❌ Mutable default
    items.append(item)
    return items

## Unexpected behavior
list1 = add_item("a")  # ["a"]
list2 = add_item("b")  # ["a", "b"] - unexpected!

Good:

def add_item(item, items=None):  # ✅ Use None as default
    if items is None:
        items = []
    items.append(item)
    return items

## Expected behavior
list1 = add_item("a")  # ["a"]
list2 = add_item("b")  # ["b"] - correct!

Bare Except Clauses

Problem: Catching all exceptions hides bugs and makes debugging impossible.

Bad:

def process_data(data):
    try:
        result = complex_operation(data)
        return result
    except:  # ❌ Catches everything, including KeyboardInterrupt!
        return None

Good:

def process_data(data):
    try:
        result = complex_operation(data)
        return result
    except (ValueError, TypeError) as e:  # ✅ Catch specific exceptions
        logger.error(f"Failed to process data: {e}")
        return None

String Formatting with % or format()

Problem: Old-style string formatting is less readable and more error-prone.

Bad:

## Old % formatting
message = "User %s has %d points" % (username, points)  # ❌ Hard to read

## Old .format()
message = "User {} has {} points".format(username, points)  # ❌ Positional

Good:

## f-strings (Python 3.6+)
message = f"User {username} has {points} points"  # ✅ Clear and concise

## With expressions
message = f"User {username} has {points * 2} bonus points"  # ✅ Powerful

Missing Type Hints

Problem: Without type hints, IDEs can't help with autocomplete and type checking.

Bad:

def calculate_total(items):  # ❌ No type information
    return sum(item['price'] for item in items)

Good:

from typing import List, Dict, Any

def calculate_total(items: List[Dict[str, Any]]) -> float:  # ✅ Clear types
    """Calculate total price from list of items."""
    return sum(float(item['price']) for item in items)

Using Global Variables

Problem: Global variables make code hard to test and reason about.

Bad:

## Module level
user_cache = {}  # ❌ Global mutable state

def get_user(user_id):
    if user_id in user_cache:
        return user_cache[user_id]
    user = fetch_user(user_id)
    user_cache[user_id] = user
    return user

Good:

class UserCache:  # ✅ Encapsulated state
    def __init__(self):
        self._cache: Dict[int, User] = {}

    def get_user(self, user_id: int) -> User:
        if user_id in self._cache:
            return self._cache[user_id]
        user = self._fetch_user(user_id)
        self._cache[user_id] = user
        return user

    def _fetch_user(self, user_id: int) -> User:
        # Implementation
        pass

Not Using Context Managers

Problem: Manual resource management leads to resource leaks.

Bad:

def read_config():
    file = open("config.json")  # ❌ No guarantee file will be closed
    data = json.load(file)
    file.close()  # May not execute if exception occurs
    return data

Good:

def read_config():
    with open("config.json") as file:  # ✅ Automatically closed
        return json.load(file)

## Or for multiple resources
def process_files(input_file, output_file):
    with open(input_file) as infile, open(output_file, 'w') as outfile:
        for line in infile:
            outfile.write(line.upper())

Checking for Empty Containers with len()

Problem: Using len() to check if a container is empty is unnecessarily verbose.

Bad:

if len(items) == 0:  # ❌ Verbose
    print("No items")

if len(users) > 0:  # ❌ Unnecessary
    process_users(users)

Good:

if not items:  # ✅ Pythonic and clear
    print("No items")

if users:  # ✅ Direct boolean context
    process_users(users)

Ignoring List Comprehensions

Problem: Using loops for simple transformations is less readable and slower.

Bad:

## Creating a new list
squares = []  # ❌ Verbose
for x in range(10):
    squares.append(x**2)

## Filtering
evens = []  # ❌ Multiple lines
for x in range(10):
    if x % 2 == 0:
        evens.append(x)

Good:

## Creating a new list
squares = [x**2 for x in range(10)]  # ✅ Concise

## Filtering
evens = [x for x in range(10) if x % 2 == 0]  # ✅ Clear intent

## With transformation and filtering
upper_names = [name.upper() for name in names if len(name) > 3]  # ✅ Powerful

Not Using enumerate()

Problem: Manual index tracking is error-prone and not Pythonic.

Bad:

items = ["apple", "banana", "cherry"]
index = 0  # ❌ Manual index management
for item in items:
    print(f"{index}: {item}")
    index += 1

Good:

items = ["apple", "banana", "cherry"]
for index, item in enumerate(items):  # ✅ Built-in enumeration
    print(f"{index}: {item}")

## With custom start index
for index, item in enumerate(items, start=1):  # ✅ Start from 1
    print(f"{index}: {item}")

String Concatenation in Loops

Problem: Concatenating strings in loops creates many intermediate objects.

Bad:

result = ""  # ❌ Inefficient
for word in words:
    result += word + " "

Good:

## For simple joining
result = " ".join(words)  # ✅ Efficient and clear

## For complex building
parts = []  # ✅ Build list first
for word in words:
    parts.append(f"<item>{word}</item>")
result = "".join(parts)

Best Practices

Virtual Environments

Always use virtual environments to isolate project dependencies:

# Create virtual environment
python3 -m venv venv

# Activate (Linux/macOS)
source venv/bin/activate

# Activate (Windows)
venv\Scripts\activate

# Install dependencies
pip install -r requirements.txt

Dependency Management

Pin exact versions for reproducibility:

# requirements.txt
requests==2.31.0
pytest==7.4.3
black==23.12.1

Use requirements-dev.txt for development dependencies:

# requirements-dev.txt
-r requirements.txt
pytest-cov==4.1.0
mypy==1.7.1
flake8==6.1.0

Implement Robust Error Handling

Good - Specific exceptions:

try:
    user = get_user_by_id(user_id)
except UserNotFoundError as e:
    logger.error(f"User {user_id} not found: {e}")
    raise
except DatabaseConnectionError as e:
    logger.critical(f"Database connection failed: {e}")
    return None

Bad - Bare except:

try:
    user = get_user_by_id(user_id)
except:  # ❌ Catches everything including KeyboardInterrupt
    return None

Code Organization

Organize imports using isort standards:

# Standard library
import os
import sys
from pathlib import Path

# Third-party
import requests
from fastapi import FastAPI, HTTPException

# Local imports
from .models import User
from .utils import validate_email

Testing Best Practices

Use pytest fixtures for setup/teardown:

import pytest

@pytest.fixture
def database_connection():
    """Provide a database connection for tests."""
    conn = create_connection()
    yield conn
    conn.close()

def test_user_creation(database_connection):
    user = create_user(database_connection, "test@example.com")
    assert user.email == "test@example.com"

Type Hints Best Practices

Use type hints consistently:

from typing import List, Optional, Dict, Any

def process_users(
    users: List[Dict[str, Any]],
    filter_active: bool = True
) -> List[str]:
    """Extract names from user dicts.

    Args:
        users: List of user dictionaries
        filter_active: Whether to filter for active users only

    Returns:
        List of user names
    """
    return [
        user["name"]
        for user in users
        if not filter_active or user.get("active", False)
    ]

Always Use Context Managers

Always use context managers for resource management:

# Good - Automatic cleanup
with open("file.txt") as f:
    data = f.read()

# Good - Multiple resources
with open("input.txt") as infile, open("output.txt", "w") as outfile:
    outfile.write(infile.read())

# Custom context manager
from contextlib import contextmanager

@contextmanager
def database_transaction(conn):
    try:
        yield conn
        conn.commit()
    except Exception:
        conn.rollback()
        raise

Logging Best Practices

Use structured logging:

import logging

logger = logging.getLogger(__name__)

def process_order(order_id: str) -> bool:
    logger.info(
        "Processing order",
        extra={
            "order_id": order_id,
            "user_id": current_user.id
        }
    )
    try:
        result = process(order_id)
        logger.info("Order processed successfully", extra={"order_id": order_id})
        return result
    except ProcessingError as e:
        logger.error(
            "Order processing failed",
            extra={"order_id": order_id, "error": str(e)},
            exc_info=True
        )
        raise

Performance Optimization

Use generators for large datasets:

# Good - Memory efficient
def read_large_file(filename: str):
    with open(filename) as f:
        for line in f:
            yield process_line(line)

# Use list comprehensions for small, simple operations
squares = [x**2 for x in range(100)]

# Use generator expressions for large or complex operations
sum_of_squares = sum(x**2 for x in range(1000000))

Follow Security Best Practices

Never hardcode secrets:

# Bad - Hardcoded secrets
API_KEY = "sk_live_abc123"  # ❌

# Good - Environment variables
import os
API_KEY = os.getenv("API_KEY")
if not API_KEY:
    raise ValueError("API_KEY environment variable not set")

# Good - Use python-dotenv for local development
from dotenv import load_dotenv
load_dotenv()
API_KEY = os.getenv("API_KEY")

Sanitize user input:

from html import escape

def display_user_input(user_text: str) -> str:
    """Safely display user-provided text."""
    return escape(user_text)  # Prevents XSS

# Use parameterized queries for databases
def get_user(conn, email: str):
    # Good - Parameterized query
    cursor = conn.execute(
        "SELECT * FROM users WHERE email = ?",
        (email,)
    )

    # Bad - String interpolation (SQL injection risk)
    # cursor = conn.execute(f"SELECT * FROM users WHERE email = '{email}'")

Documentation Best Practices

Write comprehensive docstrings:

def calculate_discount(
    price: float,
    discount_percent: float,
    member_tier: str = "standard"
) -> float:
    """Calculate final price after discount.

    Applies percentage discount and additional member tier benefits.

    Args:
        price: Original price in dollars
        discount_percent: Discount percentage (0-100)
        member_tier: Membership tier ('standard', 'premium', 'vip')

    Returns:
        Final price after all discounts applied

    Raises:
        ValueError: If discount_percent is not between 0 and 100
        ValueError: If member_tier is invalid

    Examples:
        >>> calculate_discount(100.0, 10.0, "standard")
        90.0
        >>> calculate_discount(100.0, 20.0, "premium")
        75.0
    """
    if not 0 <= discount_percent <= 100:
        raise ValueError("Discount must be between 0 and 100")

    valid_tiers = ["standard", "premium", "vip"]
    if member_tier not in valid_tiers:
        raise ValueError(f"Invalid tier. Must be one of: {valid_tiers}")

    # Apply percentage discount
    discounted = price * (1 - discount_percent / 100)

    # Apply tier benefits
    tier_discount = {"standard": 0, "premium": 0.05, "vip": 0.10}
    return discounted * (1 - tier_discount[member_tier])

Code Quality Tools

Configure tools in pyproject.toml:

[tool.black]
line-length = 100
target-version = ['py311']

[tool.isort]
profile = "black"
line_length = 100

[tool.mypy]
python_version = "3.11"
warn_return_any = true
warn_unused_configs = true
disallow_untyped_defs = true

[tool.pytest.ini_options]
minversion = "7.0"
testpaths = ["tests"]
python_files = ["test_*.py"]
python_classes = ["Test*"]
python_functions = ["test_*"]

References

Official Documentation

Community Style Guides

Tools Documentation

See Also

Development Tools & Practices

Testing & Quality

CI/CD Integration

Templates & Examples

Core Documentation