pipenv/docs/docker.md

Using Pipenv with Docker

This guide provides comprehensive instructions for integrating Pipenv with Docker, including best practices, optimization techniques, and example configurations for different scenarios.

Docker and Pipenv: Core Concepts

Docker containers provide isolated, reproducible environments for applications, while Pipenv manages Python dependencies. When used together, they create a powerful workflow for Python application deployment.

Why Use Pipenv with Docker?

• Dependency consistency: Ensure the same packages are installed in development and production
• Reproducible builds: Lock files guarantee identical environments across deployments
• Security: Hash verification prevents supply chain attacks
• Simplified workflow: Manage dependencies with a familiar tool inside containers

Basic Docker Integration

Simple Dockerfile Example

Here's a basic Dockerfile that uses Pipenv:

FROM python:3.10-slim

# Install pipenv
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy Pipfile and Pipfile.lock
COPY Pipfile Pipfile.lock ./

# Install dependencies
RUN pipenv install --deploy --system

# Copy application code
COPY . .

# Run the application
CMD ["python", "app.py"]

This approach:

1. Starts with a Python base image
2. Installs Pipenv
3. Copies dependency files
4. Installs dependencies system-wide
5. Copies application code
6. Runs the application

Key Flags for Docker Environments

• --system: Installs packages to the system Python instead of creating a virtual environment
• --deploy: Ensures the Pipfile.lock is up-to-date and fails if it isn't
• --ignore-pipfile: Uses only the lock file for installation, ignoring the Pipfile

System Installation vs. Virtual Environments in Docker

Docker containers provide isolated environments, which raises the question: should you use --system to install packages globally, or create a virtual environment inside the container?

Using --system (Recommended for most Docker use cases)

• Simpler setup with no nested isolation
• Smaller image size (no virtualenv overhead)
• Packages are directly accessible without activation
• Works well with pipenv run for executing scripts defined in Pipfile
• Best for single-application containers following the "one process per container" principle

Using a Virtual Environment

• Provides an extra layer of isolation from system Python packages
• Useful if your container runs multiple Python applications or system tools that depend on Python
• Preferred when extending base images that have Python-based system utilities (e.g., add-apt-repository in Ubuntu)
• Consider this approach if you use apt-get install python3-* packages alongside your application

For most production containers that run a single Python application, --system is the simpler and more efficient choice. The concerns about breaking system Python tools (raised in older guidance) are less relevant when:

• Your container runs only your application
• You're using slim/minimal base images
• You're not installing Python-based system packages via apt/yum

Optimized Docker Builds

Multi-Stage Builds

Multi-stage builds create smaller, more secure images by separating the build environment from the runtime environment:

FROM python:3.10-slim AS builder

# Install pipenv
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY Pipfile Pipfile.lock ./

# Install dependencies
RUN pipenv install --deploy --system

FROM python:3.10-slim

# Set working directory
WORKDIR /app

# Copy installed packages from builder stage
COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin

# Copy application code
COPY . .

# Run as non-root user
RUN useradd -m appuser
USER appuser

# Run the application
CMD ["python", "app.py"]

This approach:

1. Uses a builder stage to install dependencies
2. Copies only the necessary files to the final image
3. Runs the application as a non-root user
4. Results in a smaller, more secure image

Layer Caching Optimization

To take advantage of Docker's layer caching and speed up builds:

FROM python:3.10-slim

# Install pipenv
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files only
COPY Pipfile Pipfile.lock ./

# Install dependencies
RUN pipenv install --deploy --system

# Copy application code (changes more frequently)
COPY . .

# Run the application
CMD ["python", "app.py"]

This separates dependency installation from code copying, so dependencies are only reinstalled when Pipfile or Pipfile.lock change.

Development vs. Production Configurations

Development Dockerfile

For development environments, you might want to include development dependencies:

FROM python:3.10-slim

# Install pipenv
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY Pipfile Pipfile.lock ./

# Install dependencies including development packages
RUN pipenv install --dev --system

# Copy application code
COPY . .

# Run the development server
CMD ["python", "manage.py", "runserver", "0.0.0.0:8000"]

Production Dockerfile

For production, focus on security and minimal image size:

FROM python:3.10-slim AS builder

# Install pipenv and dependencies
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY Pipfile Pipfile.lock ./

# Install only production dependencies
RUN pipenv install --deploy --system

FROM python:3.10-slim

# Set working directory
WORKDIR /app

# Copy installed packages from builder stage
COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin

# Copy application code
COPY . .

# Create and use non-root user
RUN useradd -m appuser
USER appuser

# Set production environment variables
ENV PYTHONUNBUFFERED=1 \
    PYTHONDONTWRITEBYTECODE=1 \
    PYTHONPATH=/app

# Run the application with gunicorn (for web applications)
CMD ["gunicorn", "app:app", "--bind", "0.0.0.0:8000", "--workers", "4"]

Docker Compose Integration

Basic Docker Compose Example

version: '3.8'

services:
  web:
    build: .
    ports:
      - "8000:8000"
    volumes:
      - .:/app
    environment:
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/app
    depends_on:
      - db

  db:
    image: postgres:14
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_DB=app

volumes:
  postgres_data:

Development Environment with Hot Reload

version: '3.8'

services:
  web:
    build:
      context: .
      dockerfile: Dockerfile.dev
    ports:
      - "8000:8000"
    volumes:
      - .:/app
    environment:
      - DEBUG=True
      - DATABASE_URL=postgresql://postgres:postgres@db:5432/app
    depends_on:
      - db
    command: python manage.py runserver 0.0.0.0:8000

  db:
    image: postgres:14
    volumes:
      - postgres_data:/var/lib/postgresql/data
    environment:
      - POSTGRES_PASSWORD=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_DB=app

volumes:
  postgres_data:

Advanced Docker Techniques

Using Project-Local Virtual Environments

For some workflows, you might want to use a project-local virtual environment:

FROM python:3.10-slim

# Install pipenv
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY Pipfile Pipfile.lock ./

# Create a project-local virtual environment
ENV PIPENV_VENV_IN_PROJECT=1
RUN pipenv install --deploy

# Copy application code
COPY . .

# Run the application using the virtual environment
CMD ["./.venv/bin/python", "app.py"]

Handling Different Python Versions

If your application requires a specific Python version:

FROM python:3.10-slim

# Install pipenv
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY Pipfile Pipfile.lock ./

# Install dependencies
RUN pipenv install --deploy --system --python 3.10

# Copy application code
COPY . .

# Run the application
CMD ["python", "app.py"]

Using Custom Package Indexes

For private packages or custom indexes:

FROM python:3.10-slim

# Install pipenv
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY Pipfile Pipfile.lock ./

# Set environment variables for private repository authentication
ARG PRIVATE_REPO_USERNAME
ARG PRIVATE_REPO_PASSWORD
ENV PIP_EXTRA_INDEX_URL=https://${PRIVATE_REPO_USERNAME}:${PRIVATE_REPO_PASSWORD}@private-repo.example.com/simple

# Install dependencies
RUN pipenv install --deploy --system

# Copy application code
COPY . .

# Run the application
CMD ["python", "app.py"]

Security Best Practices

Running as Non-Root User

Always run your application as a non-root user:

FROM python:3.10-slim

# Install pipenv and dependencies
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY Pipfile Pipfile.lock ./

# Install dependencies
RUN pipenv install --deploy --system

# Copy application code
COPY . .

# Create and use non-root user
RUN useradd -m appuser && \
    chown -R appuser:appuser /app
USER appuser

# Run the application
CMD ["python", "app.py"]

Handling Secrets Securely

Use build arguments and environment variables for secrets:

FROM python:3.10-slim

# Install pipenv
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY Pipfile Pipfile.lock ./

# Use build arguments for secrets (only available during build)
ARG API_KEY
ENV API_KEY=${API_KEY}

# Install dependencies
RUN pipenv install --deploy --system

# Copy application code
COPY . .

# Run the application
CMD ["python", "app.py"]

Then build with:

$ docker build --build-arg API_KEY=your-secret-key -t your-image .

For runtime secrets, use environment variables or Docker secrets.

Scanning for Vulnerabilities

Integrate security scanning into your Docker workflow:

FROM python:3.10-slim AS builder

# Install pipenv
RUN pip install pipenv

# Set working directory
WORKDIR /app

# Copy dependency files
COPY Pipfile Pipfile.lock ./

# Install dependencies
RUN pipenv install --deploy --system

# Scan for vulnerabilities
RUN pipenv scan

FROM python:3.10-slim

# Set working directory
WORKDIR /app

# Copy installed packages from builder stage
COPY --from=builder /usr/local/lib/python3.10/site-packages /usr/local/lib/python3.10/site-packages
COPY --from=builder /usr/local/bin /usr/local/bin

# Copy application code
COPY . .

# Run the application
CMD ["python", "app.py"]

CI/CD Integration

GitHub Actions Example

name: Build and Push Docker Image

on:
  push:
    branches: [ main ]

jobs:
  build:
    runs-on: ubuntu-latest

    steps:
    - uses: actions/checkout@v3

    - name: Set up Docker Buildx
      uses: docker/setup-buildx-action@v2

    - name: Login to DockerHub
      uses: docker/login-action@v2
      with:
        username: ${{ secrets.DOCKERHUB_USERNAME }}
        password: ${{ secrets.DOCKERHUB_TOKEN }}

    - name: Build and push
      uses: docker/build-push-action@v4
      with:
        context: .
        push: true
        tags: yourusername/yourapp:latest

GitLab CI Example

stages:
  - build
  - test
  - deploy

build:
  stage: build
  image: docker:20.10.16
  services:
    - docker:20.10.16-dind
  script:
    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA .
    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA

test:
  stage: test
  image: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
  script:
    - python -m pytest

deploy:
  stage: deploy
  image: docker:20.10.16
  services:
    - docker:20.10.16-dind
  script:
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:latest
    - docker push $CI_REGISTRY_IMAGE:latest

Troubleshooting

Common Issues and Solutions

Permission Denied Errors

If you encounter permission issues:

# Add this to your Dockerfile
RUN pip install --user pipenv
ENV PATH="/root/.local/bin:${PATH}"

Package Installation Failures

For packages with system dependencies:

FROM python:3.10-slim

# Install system dependencies
RUN apt-get update && \
    apt-get install -y --no-install-recommends \
    build-essential \
    libpq-dev \
    && rm -rf /var/lib/apt/lists/*

# Install pipenv
RUN pip install pipenv

# Continue with your Dockerfile...

Slow Builds

To speed up builds:

# Use a specific pip version
RUN pip install --upgrade pip==22.2.2 pipenv==2022.8.5

# Use pip cache
ENV PIP_CACHE_DIR=/var/cache/pip

# Install with multiple workers
RUN pipenv install --deploy --system --extra-pip-args="--use-feature=fast-deps"

Best Practices

1. Use multi-stage builds to create smaller, more secure images

2. Separate dependency installation from code copying to leverage Docker's layer caching

3. Run applications as non-root users to improve security

4. Use --deploy flag to ensure Pipfile.lock is up-to-date

5. Use --system for single-application containers to install dependencies system-wide, or use a virtual environment if your container runs multiple Python applications

6. Include only necessary files in your Docker image

7. Set appropriate environment variables like PYTHONUNBUFFERED=1 for better logging

8. Scan for vulnerabilities as part of your build process

9. Use build arguments for build-time configuration

10. Use environment variables or Docker secrets for runtime configuration

Conclusion

Integrating Pipenv with Docker creates a powerful workflow for Python application deployment. By following the best practices and examples in this guide, you can create efficient, secure, and reproducible Docker images for your Python applications.

Remember that Docker and Pipenv are both tools that help with reproducibility and dependency management. When used together correctly, they complement each other and provide a robust solution for Python application deployment.