The following user-facing changes are included here:
- `aws-lc` is used instead of `ring` for a cryptography backend
- Expands our certificate signature algorithm support to include
ECDSA_P256_SHA512, ECDSA_P384_SHA512, ECDSA_P521_SHA256,
ECDSA_P521_SHA384, and ECDSA_P521_SHA512
- `--native-tls` is deprecated in favor of a new `--system-certs` flag,
avoiding confusion with the TLS implementation used (we use `rustls` not
`native-tls`, see prior confusion at
https://github.com/astral-sh/uv/issues/11595)
- NASM is a new build requirement on Windows, it is required by `aws-lc`
on x86-64 and i386
- `rustls-platform-verifier` is used instead of `rustls-native-certs`
for system certificate verification
- On macOS, certificate validation is now delegated to
`Security.framework` (`SecTrust`). Performance when using
`--system-certs` is improved by avoiding exporting and parsing all the
certificates from the keychain at startup.
- On Windows, certificate validation is now delegated to
`CertGetCertificateChain` and `CertVerifyCertificateChainPolicy`
- On Linux, certificate validation should be approximately unchanged
- Some previously failing chains may succeed, and some previously
accepted chains may fail; generally, this should result in behavior
closer matching browsers and other native applications
- macOS and Windows may now perform live OCSP fetches for early
revocation, which could add latency to some requests
- Empty `SSL_CERT_FILE` values are ignored (for consistency with
`SSL_CERT_DIR`)
The following internal changes are included here:
- Certificate loading has been refactored to use a newtype with helper
methods
- The certificate tests have been rewritten
- We use `webpki-root-certs` instead of `webpki-roots`, see
https://github.com/astral-sh/uv/pull/17543#discussion_r2820187691
- We request `identity` encoding for range requests, see
https://github.com/astral-sh/async_http_range_reader/pull/3#discussion_r2700194798
- Various dependencies (including forks) updates to versions which use
reqwest 0.13+
This is a replacement of #17543 with an updated description. See that
pull request for prior discussion. I've made the following changes from
the initial approach there:
- Previously, the `native-tls` TLS implementation was added which
included an OpenSSL build. We don't currently use the `native-tls`
implementation, but the `--native-tls` flag there was erroneously
updated to enable it.
- Previously, there was a `--tls-backend` flag to toggle between
`native-tls` and `rustls`. Since we currently always use `rustls`, this
is deferred to future work (if we need it at all).
- Previously, there were unintentional breaking changes to
`SSL_CERT_FILE` and `SSL_CERT_DIR` handling, including merging with the
base certificates instead of replacing them, dropping support for
OpenSSL hash-named certificate files, skipping deduplication of
certificates. Here, we retain use of `rustls-native-certs` for loading
certificates from the system as it handles these edge cases.
Closes https://github.com/astral-sh/uv/issues/17427
---------
Co-authored-by: salmonsd <22984014+salmonsd@users.noreply.github.com>
Persists snapshots as artifacts on test failure in CI and adds a script
to apply them locally
```
❯ ./scripts/apply-ci-snapshots.sh
Found pull request #18424 for branch 'zb/ci-snapshots'...
Found latest CI run 23022983761
Downloading pending snapshot artifacts...
Downloaded 3 artifacts
Applying 2 snapshot changes...
accepted:
crates/uv/tests/it/pip_install.rs:13679 (transitive_dependency_config_settings_invalidation-2)
crates/uv/tests/it/python_install.rs:1694 (python_install_default-5)
```
We infer the target run via the `gh` CLI. You may also provide the run
ID directly.
We'll merge snapshot artifacts from multiple platforms, so if there are
platform-specific failures on both Linux and Windows we'll apply both.
This policy is derived from
https://github.com/ghostty-org/ghostty/blob/main/AI_POLICY.md: We have
the problem that there are autonomous contributions, or users trying to
get the LLM to solve the hard problems, where it usually fails.
I've kept the section intentionally concise without laying out all the
branches of valid and invalid LLM usage. For example, it's totally
possible to first have the LLM write the change, review that, then
develop a design from that and iterate on it and in the end have a fully
human-understood PR. What I want to discourage is that contributors
think they can outsource e.g. the hard algorithmic parts to the LLM,
without understanding the existing structure, where the LLM currently
inevitably fails.
---------
Co-authored-by: Tomasz Kramkowski <tom@astral.sh>
Had another PR this morning where I was just talking to an LLM. This is
the minimum to LLM usage, not yet addressing yet how to use LLMs for
coding responsibly.
* ✅ **Impact**: Edited the contributing guide
* ✅ **Actionable**: The change addressed an evolving situation
* ✅ **Correctness**: Performed a grammar pass – this is now ready for
review
Uses a nextest setup hook to sign the uv and test binaries before
running the tests. This allows you to grant permission to the test suite
_once_ when running native authentication tests on macOS. Otherwise, you
get prompted on every access on every binary change.
This also removes the file-specific targets from prettier execution
which means we're including `.json`, `.css`, and `.html` files, which
seems like an improvement.
Co-authored-by: Claude <noreply@anthropic.com>
## Summary
Follow up to https://github.com/astral-sh/uv/pull/15563
Closes https://github.com/astral-sh/uv/issues/13485
This is a first-pass at adding support for conditional support for Git
LFS between git sources, initial feedback welcome.
e.g.
```
[tool.uv.sources]
test-lfs-repo = { git = "https://github.com/zanieb/test-lfs-repo.git", lfs = true }
```
For context previously a user had to set `UV_GIT_LFS` to have uv fetch
lfs objects on git sources. This env var was all or nothing, meaning you
must always have it set to get consistent behavior and it applied to all
git sources. If you fetched lfs objects at a revision and then turned
off lfs (or vice versa), the git db, corresponding checkout lfs
artifacts would not be updated properly. Similarly, when git source
distributions were built, there would be no distinction between sources
with lfs and without lfs. Hence, it could corrupt the git, sdist, and
archive caches.
In order to support some sources being LFS enabled and other not, this
PR adds a stateful layer roughly similar to how `subdirectory` works but
for `lfs` since the git database, the checkouts and the corresponding
caching layers needed to be LFS aware (requested vs installed). The
caches also had to isolated and treated entirely separate when handling
LFS sources.
Summary
* Adds `lfs = true` or `lfs = false` to git sources in pyproject.toml
* Added `lfs=true` query param / fragments to most relevant url structs
(not parsed as user input)
* In the case of uv add / uv tool, `--lfs` is supported instead
* `UV_GIT_LFS` environment variable support is still functional for
non-project entrypoints (e.g. uv pip)
* `direct-url.json` now has an custom `git_lfs` entry under VcsInfo
(note, this is not in the spec currently -- see caveats).
* git database and checkouts have an different cache key as the sources
should be treated effectively different for the same rev.
* sdists cache also differ in the cache key of a built distribution if
it was built using LFS enabled revisions to distinguish between non-LFS
same revisions. This ensures the strong assumption for archive-v0 that
an unpacked revision "doesn't change sources" stays valid.
Caveats
* `pylock.toml` import support has not been added via git_lfs=true,
going through the spec it wasn't clear to me it's something we'd support
outside of the env var (for now).
* direct-url struct was modified by adding a non-standard `git_lfs`
field under VcsInfo which may be undersirable although the PEP 610 does
say `Additional fields that would be necessary to support such VCS
SHOULD be prefixed with the VCS command name` which could be interpret
this change as ok.
* There will be a slight lockfile and cache churn for users that use
`UV_GIT_LFS` as all git lockfile entries will get a `lfs=true` fragment.
The cache version does not need an update, but LFS sources will get
their own namespace under git-v0 and sdist-v9/git hence a cache-miss
will occur once but this can be sufficient to label this as breaking for
workflows always setting `UV_GIT_LFS`.
## Test Plan
Some initial tests were added. More tests likely to follow as we reach
consensus on a final approach.
For IT test, we may want to move to use a repo under astral namespace in
order to test lfs functionality.
Manual testing was done for common pathological cases like killing LFS
fetch mid-way, uninstalling LFS after installing an sdist with it and
reinstalling, fetching LFS artifacts in different commits, etc.
PSA: Please ignore the docker build failures as its related to depot
OIDC issues.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
Co-authored-by: konstin <konstin@mailbox.org>
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
This PR adds instructions to install a C compiler on Fedora-based Linux
distributions.
## Test Plan
```
# Start Fedora container interactively (can probably be done on Docker as well)
podman run -it registry.fedoraproject.org/fedora
# From now on, run all commands inside the container.
# Install Rustup
curl --proto '=https' --tlsv1.2 -sSf https://sh.rustup.rs | sh
# Add cargo bin folder to PATH
export PATH="${HOME}/.cargo/bin:${PATH}"
# Install git, clone uv project and get into its folder
dnf install git
git clone https://github.com/astral-sh/uv.git
cd uv
# Try to compile uv and fail (error: linker `cc` not found)
cargo build
# Install C compiler
dnf install gcc
# Try to compile uv again. This time, successfully.
cargo build
```
Signed-off-by: Mateus Devino <mdevino@ibm.com>
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
Sets the correct path to the builder dockerfile in the contributing
docs.
---------
Co-authored-by: konsti <konstin@mailbox.org>
## Summary
I believe `zlib-rs` is now a better choice on ARM and x86, so I'm just
going to assume it's a better choice everywhere. It's much easier to
build (removes our CMake dependency), and in my benchmarking, it's
substantially faster on ARM and faster or ~exactly even on my x86
Windows machine.
We migrated to `zlib-rs` once before (#9184); however, I later reverted
it as I learned that they were only doing compile-time feature
detection, and so `zlib-rs` was meaningfully slower on x86. They now
perform runtime feature detection:
https://trifectatech.org/blog/zlib-rs-is-faster-than-c/.
To benchmark, I wrote a script to create a local Simple API-compliant
registry (see the commit history) for a single package. Then I ran the
`install-cold` benchmark against that registry to install NumPy.
On ARM:
```
❯ uv run resolver --uv-pip-path ../../zlib-ng --uv-pip-path ../../zlib-rs \
--benchmark install-cold \
req.txt --warmup 10 --min-runs 30
Benchmark 1: ../../zlib-ng (install-cold)
Time (mean ± σ): 165.7 ms ± 34.7 ms [User: 64.4 ms, System: 93.2 ms]
Range (min … max): 141.8 ms … 293.2 ms 30 runs
Benchmark 2: ../../zlib-rs (install-cold)
Time (mean ± σ): 150.9 ms ± 16.2 ms [User: 57.4 ms, System: 86.4 ms]
Range (min … max): 135.3 ms … 202.4 ms 30 runs
Summary
../../zlib-rs (install-cold) ran
1.10 ± 0.26 times faster than ../../zlib-ng (install-cold)
```
I benchmarked this about 100 times on my Windows machine and found it
difficult to conclude anything beyond "They're nearly the same". Here's
an example:
```
PS C:\Users\crmar\workspace\puffin> hyperfine --prepare "uv venv" "zlib-rs.exe pip sync ./scripts/benchmark/req.txt" "zlib-ng.exe pip sync ./scripts/benchmark/req.txt" "zlib-rs.exe pip sync ./scripts/benchmark/req.txt" "zlib-ng.exe pip sync ./scripts/benchmark/req.txt" --runs 10 --warmup 5
Benchmark 1: zlib-rs.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 240.6 ms ± 10.8 ms [User: 6.1 ms, System: 92.2 ms]
Range (min … max): 229.4 ms … 267.9 ms 10 runs
Benchmark 2: zlib-ng.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 241.3 ms ± 6.2 ms [User: 7.7 ms, System: 90.6 ms]
Range (min … max): 233.9 ms … 252.1 ms 10 runs
Benchmark 3: zlib-rs.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 242.8 ms ± 7.7 ms [User: 6.2 ms, System: 23.4 ms]
Range (min … max): 236.1 ms … 262.8 ms 10 runs
Benchmark 4: zlib-ng.exe pip sync ./scripts/benchmark/req.txt
Time (mean ± σ): 245.9 ms ± 5.7 ms [User: 1.5 ms, System: 59.4 ms]
Range (min … max): 240.9 ms … 257.3 ms 10 runs
Summary
zlib-rs.exe pip sync ./scripts/benchmark/req.txt ran
1.00 ± 0.05 times faster than zlib-ng.exe pip sync ./scripts/benchmark/req.txt
1.01 ± 0.06 times faster than zlib-rs.exe pip sync ./scripts/benchmark/req.txt
1.02 ± 0.05 times faster than zlib-ng.exe pip sync ./scripts/benchmark/req.txt
```
Closes#11885.
## Summary
Testing with `UV_PYTHON_INSTALL_DIR` environment variable has some
problems. This PR fix them.
- `UV_PYTHON_INSTALL_DIR` must be an absolute path.
- Cargo tries to find Python executables from each crates in test. If it
is relative path, cargo searches in different directories for each
tests.
- Skip the test asserting help messages.
- Clap shows the current value of the environment variables. If
`UV_PYTHON_INSTALL_DIR` is set, the test fails.
## Test Plan
<!-- How was it tested? -->
All tests pass with
`UV_PYTHON_INSTALL_DIR=/path/to/my/home/uv/target/testpython`.
<!--
Thank you for contributing to uv! To help us out with reviewing, please
consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title?
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
## Test Plan
<!-- How was it tested? -->
To enforce the 100 character line limit in markdown files introduced in
https://github.com/astral-sh/uv/pull/5635, and to automate the
formatting of markdown files, i've added prettier and formatted our
markdown files with it.
I've excluded the changelog and the generated references documentation
from this for having too many changes, but we can also include them.
I'm not particular on which style we use. My main motivations are
(major) not having to reflow markdown files myself anymore and (minor)
consistence between all markdown files. I've chosen prettier for similar
reason as we chose black, it's a single good style that's automated and
shared in the community. I do prefer prettier's style of not breaking
inside of a link name though.
This PR is in two parts, the first adds prettier to CI and documents
using it, while the second actually formats the docs. When merge
conflicts arise, we can drop the last commit and regenerate it with `npx
prettier --prose-wrap always --write BENCHMARKS.md CONTRIBUTING.md
README.md STYLE.md docs/*.md docs/concepts/**/*.md docs/guides/**/*.md
docs/pip/**/*.md`.
---------
Co-authored-by: Zanie Blue <contact@zanie.dev>
## Summary
Removes the legacy `benchmark` directory (we'll always have it in Git)
and renames `bench` to `benchmark` for clarity. Fixes a variety of
commands and references.
## Summary
Spotted some issues in the settings documentation, and room for small
improvements by linking to PEPs/RFCs.
Also updating contribution documentation to mention that it's necessary
to enable the virtual environment before running `mkdocs serve`.
## Test Plan
Running documentation locally.
## Summary
I noticed the command to install multiple Python versions was wrong as
it was failing cause `toolchain` is not a known command.
I looked in the `ci.yml` workflow to see which command is used there and
updated the instructions accordingly.
## Test Plan
I just ran the command locally. :)
Whew this is a lot.
The user-facing changes are:
- `uv toolchain` to `uv python` e.g. `uv python find`, `uv python
install`, ...
- `UV_TOOLCHAIN_DIR` to` UV_PYTHON_INSTALL_DIR`
- `<UV_STATE_DIR>/toolchains` to `<UV_STATE_DIR>/python` (with
[automatic
migration](https://github.com/astral-sh/uv/pull/4735/files#r1663029330))
- User-facing messages no longer refer to toolchains, instead using
"Python", "Python versions" or "Python installations"
The internal changes are:
- `uv-toolchain` crate to `uv-python`
- `Toolchain` no longer referenced in type names
- Dropped unused `SystemPython` type (previously replaced)
- Clarified the type names for "managed Python installations"
- (more little things)
## Summary
We currently rely on libgit2 for most git-related functionality.
However, libgit2 has long-standing performance issues, as well as lags
significantly behind git in terms of new features. For these reasons we
now use the git CLI by default for fetching repositories
(https://github.com/astral-sh/uv/pull/1781). This PR completely drops
libgit2 in favor of the git CLI for all git-related functionality, which
should allow us to use features such as partial clones and sparse
checkouts in the future for performance.
There is also a lot of technical debt in the current git code as it's
mostly taken from Cargo. Switching to the git CLI *vastly* simplifies
the `uv-git` codebase.
Eventually we might want to look into switching to
[`gitoxide`](https://github.com/Byron/gitoxide), but it's currently too
immature for our use case.
Extends #3726
Moves toolchain storage out of `UV_BOOTSTRAP_DIR` (`./bin`) into the
proper user data directory as defined by #3726.
Replaces `UV_BOOTSTRAP_DIR` with `UV_TOOLCHAIN_DIR` for customization.
Installed toolchains will be discovered without opt-in, but the idea is
still that these are not yet user-facing.
See https://github.com/astral-sh/uv/issues/2617
Note this also includes:
- #2918
- #2931 (pending)
A first step towards Python toolchain management in Rust.
First, we add a new crate to manage Python download metadata:
- Adds a new `uv-toolchain` crate
- Adds Rust structs for Python version download metadata
- Duplicates the script which downloads Python version metadata
- Adds a script to generate Rust code from the JSON metadata
- Adds a utility to download and extract the Python version
I explored some alternatives like a build script using things like
`serde` and `uneval` to automatically construct the code from our
structs but deemed it to heavy. Unlike Rye, I don't generate the Rust
directly from the web requests and have an intermediate JSON layer to
speed up iteration on the Rust types.
Next, we add add a `uv-dev` command `fetch-python` to download Python
versions per the bootstrapping script.
- Downloads a requested version or reads from `.python-versions`
- Extracts to `UV_BOOTSTRAP_DIR`
- Links executables for path extension
This command is not really intended to be user facing, but it's a good
PoC for the `uv-toolchain` API. Hash checking (via the sha256) isn't
implemented yet, we can do that in a follow-up.
Finally, we remove the `scripts/bootstrap` directory, update CI to use
the new command, and update the CONTRIBUTING docs.
<img width="1023" alt="Screenshot 2024-04-08 at 17 12 15"
src="https://github.com/astral-sh/uv/assets/2586601/57bd3cf1-7477-4bb8-a8e9-802a00d772cb">
* Document good first issues
* Document `scripts` directory, as far as useful for contributors
* Remove compare with pip script, we don't need it anymore
I think this closes#817.
---------
Co-authored-by: Jo <10510431+j178@users.noreply.github.com>
Zanie Blue