pipenv/docs/locking.md

Locking Dependencies

Dependency locking is a critical feature of Pipenv that ensures consistent, reproducible environments across development, testing, and production systems. This guide explains how Pipenv's locking mechanism works and how to use it effectively.

Understanding Dependency Locking

What is a Lock File?

The Pipfile.lock is a JSON file that contains:

1. Exact versions of all direct and transitive dependencies
2. Cryptographic hashes for each package to verify integrity
3. Metadata about the environment and sources
4. Dependency markers for platform-specific packages

This ensures that everyone using your project gets exactly the same dependencies, preventing "works on my machine" problems.

Why Use Lock Files?

• Deterministic builds: Ensures the same packages are installed every time
• Security: Verifies package integrity through hash checking
• Dependency resolution: Captures the complete dependency graph, including sub-dependencies
• Reproducibility: Makes it easy to recreate the exact environment on any system

Creating and Updating Lock Files

Generating a Lock File

Pipenv automatically creates a Pipfile.lock file when you install packages:

$ pipenv install requests

To explicitly generate or update the entire lock file based on your current Pipfile:

$ pipenv lock

This command:

1. Resolves all dependencies specified in your Pipfile
2. Determines compatible versions for all packages
3. Calculates hashes for each package
4. Writes the complete dependency graph to Pipfile.lock

Updating Specific Dependencies

To update a specific package in your lock file:

$ pipenv upgrade requests

This updates only the specified package and its dependencies in the lock file without installing them.

To update and install the package:

$ pipenv update requests

Viewing the Current Lock Status

To see the currently locked dependencies:

$ pipenv graph

Example output:

requests==2.28.1
  - certifi [required: >=2017.4.17, installed: 2022.6.15]
  - charset-normalizer [required: >=2.0.0,<2.1.0, installed: 2.0.12]
  - idna [required: >=2.5,<4, installed: 3.3]
  - urllib3 [required: >=1.21.1,<1.27, installed: 1.26.10]

Installing from Lock Files

Basic Installation

To install all dependencies exactly as specified in the lock file:

$ pipenv sync

This command installs the exact versions from Pipfile.lock without updating the lock file.

Development vs. Production

For development environments, including development dependencies:

$ pipenv sync --dev

For production environments, using only production dependencies:

$ pipenv sync

Deployment Scenarios

In deployment or CI/CD pipelines, use the --deploy flag to ensure the lock file is up-to-date:

$ pipenv install --deploy

This will fail if the Pipfile.lock is out of date or doesn't exist, preventing accidental use of incorrect dependencies.

Lock File Verification

Verifying Lock File Integrity

To verify that your lock file is up-to-date with your Pipfile:

$ pipenv verify

This is useful in CI/CD pipelines to ensure the lock file has been properly updated after changes to the Pipfile.

Handling Hash Verification Failures

If you encounter hash verification failures:

$ pipenv install --ignore-pipfile

This forces Pipenv to use the lock file and ignore the Pipfile, which can help diagnose whether the issue is with the lock file or the Pipfile.

Advanced Locking Features

Locking with Pre-release Versions

To include pre-release versions in your lock file:

$ pipenv lock --pre

Locking for Specific Python Versions

To generate a lock file for a specific Python version:

$ pipenv lock --python 3.9

Locking Specific Package Categories

To lock only specific package categories:

$ pipenv lock --categories="docs,tests"

Clearing the Cache

If you encounter issues with dependency resolution:

$ pipenv lock --clear

This clears Pipenv's cache and forces a fresh resolution of all dependencies.

Lock File Structure

The Pipfile.lock is a JSON file with the following structure:

{
    "_meta": {
        "hash": {
            "sha256": "<hash-of-pipfile-contents>"
        },
        "pipfile-spec": 6,
        "requires": {
            "python_version": "3.9"
        },
        "sources": [
            {
                "name": "pypi",
                "url": "https://pypi.org/simple",
                "verify_ssl": true
            }
        ]
    },
    "default": {
        "requests": {
            "hashes": [
                "sha256:6a1b267aa90cac58ac3a765d067950e7dbbf75b1da07e895d1f594193a40a38b",
                "sha256:9c443e7324ba5b85070c4a818ade28bfabedf16ea10206da1132edaa6dda237e"
            ],
            "index": "pypi",
            "version": "==2.28.1"
        },
        "urllib3": {
            "hashes": [
                "sha256:8298d6d56d39be0e3bc13c1c97d133f9b45d797169a0e7e64a9e85268445e93b",
                "sha256:879ba4d1e89654d9769ce13121e0f94310ea32e8d2f8cf587b77c08bbcdb30d6"
            ],
            "markers": "python_version >= '2.7' and python_version not in '3.0, 3.1, 3.2, 3.3, 3.4, 3.5'",
            "version": "==1.26.10"
        }
    },
    "develop": {
        "pytest": {
            "hashes": [
                "sha256:13d0e3ccfc2b6e26be000cb6568c832ba67ba32e719443bfe725814d3c42433c",
                "sha256:a06a0425453864a270bc45e71f783330a7428defb4230fb5e6a731fde06ecd45"
            ],
            "index": "pypi",
            "version": "==7.1.2"
        }
    }
}

Key sections:

• _meta: Contains metadata about the lock file
• default: Production dependencies
• develop: Development dependencies
• Custom categories: Any additional package categories you've defined

Multi-Platform Considerations

Platform-Specific Lock Files

Pipfile.lock is generated based on the current platform where pipenv lock is run. This means:

• Package hashes included in the lock file are only those for wheels/sdists compatible with the platform and Python version used during locking.
• If your team develops on macOS but deploys to Linux, the lock file generated on macOS may include different hashes than one generated on Linux (especially for packages with platform-specific binary wheels).

This is expected behavior, but it is important to be aware of when working across multiple platforms or operating systems.

Working Across Multiple Platforms

If your project runs on multiple platforms (e.g., developers on Windows/macOS and production on Linux), consider these strategies:

Option 1: Lock on the target platform

Generate the Pipfile.lock on the same platform (or a Docker container matching it) as your production environment:

# Lock inside a Linux container
$ docker run --rm -v $(pwd):/app -w /app python:3.11-slim \
    sh -c "pip install pipenv && pipenv lock"

Option 2: Use PIPENV_VENV_IN_PROJECT with CI/CD

Generate the lock file in CI on your target platform and commit it to version control, ensuring all developers install with pipenv sync instead of re-locking locally.

Option 3: Use --extra-pip-args for cross-platform wheels

For packages that need specific platform wheels, you can use pip's --platform flag during installation (but not during locking):

$ pipenv install --extra-pip-args="--platform=manylinux2014_x86_64 --only-binary=:all:"

Native cross-platform lock file generation (a single `Pipfile.lock` with hashes for
all target platforms) is not currently supported by Pipenv. This is a known limitation.
If this is a hard requirement, consider generating separate lock files per platform in
CI and selecting the appropriate one at deploy time.

Best Practices

Version Control

Always commit both Pipfile and Pipfile.lock to version control. The lock file ensures that everyone on your team and your deployment pipeline uses the exact same dependencies.

Regular Updates

Regularly update your dependencies to get security fixes and improvements:

# Check for outdated packages
$ pipenv update --outdated

# Update all packages
$ pipenv update

Lock File Review

When updating dependencies, review the changes in the lock file to understand the impact:

# After updating, check what changed
$ git diff Pipfile.lock

CI/CD Integration

In your CI/CD pipeline:

# Verify the lock file is up-to-date
$ pipenv verify

# Install dependencies from the lock file
$ pipenv sync

Handling Conflicts

If you encounter dependency conflicts:

1. Use pipenv graph to visualize dependencies and identify conflicts
2. Try relaxing version constraints in your Pipfile
3. Use pipenv lock --clear to clear the cache and try again
4. Consider using custom package categories to manage conflicting dependencies

Troubleshooting

Common Issues

Lock File Out of Date

If you see "Pipfile.lock out of date":

$ pipenv lock

Dependency Resolution Failures

If Pipenv can't resolve dependencies:

# Try with verbose output
$ pipenv lock --verbose

# Clear the cache and try again
$ pipenv lock --clear

Hash Mismatch Errors

If you encounter hash verification failures:

# Regenerate the lock file
$ pipenv lock

# Or force installation from the lock file
$ pipenv install --ignore-pipfile

Helpful Commands

Passing Additional Arguments to pip

You can supply additional arguments to pip during locking:

$ pipenv lock --extra-pip-args="--use-feature=truststore --proxy=127.0.0.1"

Debugging Lock Issues

For detailed debugging information:

$ PIPENV_VERBOSE=1 pipenv lock

Conclusion

Locking dependencies with Pipenv is a powerful way to ensure consistent, reproducible environments. By understanding and using Pipenv's locking mechanism effectively, you can avoid many common dependency management issues and ensure your projects run reliably across different environments.