## Summary
Per the spec:
> A named tuple class can be subclassed, but any fields added by the
subclass are not considered part of the named tuple type. Type checkers
should enforce that these newly-added fields do not conflict with the
named tuple fields in the base class.
The spec then provides this example:
```python
class Point(NamedTuple):
x: int
y: int
units: str = "meters"
class PointWithName(Point):
name: str # OK
x: int # Type error (invalid override of named tuple field)
```
We take a slightly more expansive view here, rejecting any attributes
(with or without an annotation, with or without a default) and
properties in a `NamedTuple` subclass. This seems to match Pyright and
Pyrefly, though Mypy doesn't flag these.
Shadowing an attribute in this way leads to odd behavior that is
probably never what you want, e.g.:
```python
from typing import NamedTuple
class A(NamedTuple):
x: int
class B(A):
x: int = 42
b = B(1)
print(b.x) # 42
print(b[0]) # 1
print(repr(b)) # B(x=1)
b.x = 99
print(b.x) # 99
print(b[0]) # 1
print(repr(b)) # B(x=1)
```
## Summary
We now use the same error code for all of these, with the same range,
and it's a warning by default:
```python
N = NamedTuple("O", [])
TD = TypedDict("TE", {})
T = TypeVar("U")
P = ParamSpec("Q")
NT = NewType("N", int)
```
It also implements more graceful recovery for `TypeVar`,
`TypeAliasType`, `ParamSpec`, and `NewType`.
Closes https://github.com/astral-sh/ty/issues/3255.
## Summary
There are three special projects in mypy_primer's list: they are all
part of the cpython repository, but in different subfolders. Previously,
`good.txt` only included the pattern "cpython" to include all of them
for mypy_primer runs. However, ecosystem-analyzer expects an exact
match. I don't really want to change that behavior in
ecosystem-analyzer, so instead, we now include the full project names,
and map those back to "cpython" for mypy_primer (which matches the regex
against the projects URL, not the name).
## Test Plan
- Made sure that mypy_primer still runs as expected, with the new
selector (regex-escaping leads to a slightly different project selector)
- Made sure that CPython projects are now included in ecosystem-analyzer
runs
## Summary
Enable Ruff's own `S602` rule (`subprocess-popen-with-shell-equals-true`
from flake8-bandit) in `pyproject.toml` to enforce no-shell subprocess
calls going forward, and fix the two existing violations it catches.
**`pyproject.toml`** - adds `S602` to the lint `select` list so any
future `shell=True` usage in the scripts is caught automatically by the
linter.
**`scripts/setup_primer_project.py`** - `project.install_cmd` and
`project.deps` come from the mypy-primer project registry (an
externally-fetched third-party config). Both were joined into shell
strings and executed with `shell=True`, making them a supply-chain
injection vector. Fixed by tokenising with `shlex.split()` and dropping
`shell=True`.
**`scripts/ty_benchmark/src/benchmark/snapshot.py`** - `command.prepare`
was passed to `subprocess.run(..., shell=True)`. While no current caller
sets this field, it is a latent injection point. Fixed by tokenising
with `shlex.split()` and dropping `shell=True`; adds the missing `import
shlex`.
## Test Plan
Both scripts are developer-only utilities. The changes are semantically
equivalent for well-formed inputs - `shlex.split()` produces the same
argument list the shell would have constructed, while refusing to pass
metacharacters through to a shell process.
Verified no other `shell=True` uses remain in `scripts/` or `python/`
that would be newly flagged by S602.
## Summary
This PR renames TID254 to `lazy-import-mismatch` and expands it into a
single rule with two complementary settings: `require-lazy` and
`ban-lazy`. Both settings accept `"all"`, `list[str]`, or `{ include =
..., exclude = [...] }`, and we validate that the two selectors don’t
overlap.
This covers the two main use-cases users asked for:
- Require lazy imports by default, except for known side-effectful
modules. Example:
```toml
require-lazy = { include = "all", exclude = ["sitecustomize"] }
```
This enforces lazy imports everywhere Ruff can rewrite them, while
leaving sitecustomize eager.
- Forbid lazy imports for modules that must stay eager, or even forbid
lazy imports by default with a small allowlist.
```toml
# Require `sitecustomize` to stay eager.
ban-lazy = ["sitecustomize"]
# Ban lazy imports everywhere except `typing`.
ban-lazy = { include = "all", exclude = ["typing"] }
```
## Summary
`TID254` enforces the use of `lazy` imports. You can specify a set of
modules, similar to `banned-module-level-imports`, or `"all"`:
```toml
# Require every module-level import to be lazy.
banned-eager-imports = "all"
# Require lazy imports for specific modules.
banned-eager-imports = [
"boto3",
"botocore",
]
```
## Summary
Part of https://github.com/astral-sh/ty/issues/1240
Stop unioning `Unknown` into the types of un-annotated container
literals.
We discussed perhaps continuing to union `Unknown` if the inferred type
is a singleton type like `None`. I'd like to explore this as a separate
change so we can see the ecosystem impact more clearly.
## Test Plan
Adjusted many mdtest expectations.
There's one test case that regresses with this change, because we don't
fully support union type contexts (it can require a lot of repeat
inference in pathological cases). So `x10: list[int | str] | list[int |
None] = [1, 2, 3]` previously passed only because we inferred the RHS as
`list[Unknown | int]` -- now we infer it as `list[int]` and the
assignment fails due to invariance. I've kept this test as a TODO since
it's not trivial to fix. Mypy errors in the same way we now do,
suggesting it's not necessarily a huge priority either.
## Ecosystem
This change is expected to cause new diagnostics and some false
positives, since we are replacing very-forgiving gradual types with
non-gradual inference heuristics.
Many of these issues could be solved or significantly mitigated by
https://github.com/astral-sh/ty/issues/1473, depending how far we are
able to go with that, and particularly whether we can afford to apply it
also to container literals which are not empty at construction. The
downside of broad application of this approach is that in some cases it
could cause us to widen container types when the user actually just made
a mistake and added the wrong thing to a container, and would prefer an
error at that location.
Some categories of new error that show up in the ecosystem report:
### Implicit TypedDicts
These are cases where the dictionary is heterogeneous and would ideally
be typed as a `TypedDict` but isn't, for example:
```py
def make_person(photo: bytes | None):
person = {"name": "Pat", age: 29}
if photo is not None:
person["photo"] = photo
```
We (and pyrefly, and pyright in strict mode) error on the last line here
because we already inferred `dict[str, str | int]`, so we can't add a
`bytes` value.
Mypy prefers common-base joins over union joins, so it infers `dict[str,
object]`, which avoids the error adding a `bytes` value. This means the
value type is less precise, which theoretically means potentially more
errors using values from the dict later. But in practice with this
heterogeneous pattern, either `object` or the union will cause similar
problems when using values from the dict -- in either case you'd
probably have to cast or narrow.
Pyright (in non-strict mode) has a special case where it falls back to
`Unknown` when it sees heterogenous value types, so it infers this as
`dict[str, Unknown]`.
I think we could consider either the mypy or pyright approaches here,
but we don't need to do it in this PR; we can file an issue and consider
it as a follow-up.
Another symptom of this same root cause is repetitive diagnostics
arising from a large union inferred as value type; the same fixes would
address this.
### Negative intersections, particularly with e.g. `~AlwaysFalsy` or
`~None`.
Example:
```py
class A: ...
def _(a: A | None) -> dict[str, A]:
if a:
d = {"a": a}
return d
return {}
```
We error on `return d` because "expected `dict[str, A]`, found
`dict[str, A & ~AlwaysFalsy]`". This is an issue specific to
intersection types, so no other type checker has this problem.
I think when we "promote literals" (we may need to give this operation a
broader name -- it's really "type promotion to give a better inferred
type when invariance means too-precise is bad") we should also eliminate
all negative types from intersections. I would prefer to do this as a
separate PR for easier review and better visibility of ecosystem impact,
but I think it's high priority to land soon after this PR (ideally
before a release).
### Overly-precise inference for singleton `None`
This did show up, to the tune of ~100 new diagnostics
([example](https://github.com/pytorch/ignite/blob/b73a4c20e991b3e14949f2a69651ed2a7219f2fd/tests/ignite/engine/test_engine.py#L158)),
so I think it is worth addressing as a follow-up.
## Summary
This PR fixes the last remaining conformance failure on
`enums_members.py` in the conformance suite.
Implements a new lint rule `invalid-enum-member-annotation` that detects
type annotations on enum members. According to the typing spec, enum
members should not have explicit type annotations, as the actual runtime
type is the enum class itself, not the annotated type.
The rule:
- Flags annotated enum members (e.g., `DOG: int = 2` in an `Enum` class)
- Allows bare `Final` annotations (which don't specify a type)
- Excludes dunder names, private names, and special sunder names like
`_value_` and `_ignore_`
- Excludes pure declarations without values (non-members)
## Test Plan
mdtests
Just to be sure, I ran the example from #23587 again on this branch:
```console
~/astral/ruff on brent/0.15.4 [$] is 📦 v0.15.4 via 🐍 v3.14.2 via 🦀 v1.93.0
❯ echo 'x = id' | uvx ruff@latest --isolated check - --preview --select ANN003,PLR1712
error: Ruff crashed. If you could open an issue at:
https://github.com/astral-sh/ruff/issues/new?title=%5BPanic%5D
...quoting the executed command, along with the relevant file contents and `pyproject.toml` settings, we'd be very appreciative!
thread 'main' (1681253) panicked at crates/ruff_python_semantic/src/definition.rs:285:26:
index out of bounds: the len is 0 but the index is 0
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
~/astral/ruff on brent/0.15.4 [$] is 📦 v0.15.4 via 🐍 v3.14.2 via 🦀 v1.93.0
❯ echo 'x = id' | just run --isolated check - --preview --select ANN003,PLR1712
cargo run -p ruff -- --isolated check - --preview --select ANN003,PLR1712
Compiling ruff_python_semantic v0.0.0 (/home/brent/astral/ruff/crates/ruff_python_semantic)
Compiling ruff v0.15.4 (/home/brent/astral/ruff/crates/ruff)
Compiling ruff_linter v0.15.4 (/home/brent/astral/ruff/crates/ruff_linter)
Compiling ruff_graph v0.1.0 (/home/brent/astral/ruff/crates/ruff_graph)
Compiling ruff_workspace v0.0.0 (/home/brent/astral/ruff/crates/ruff_workspace)
Compiling ruff_markdown v0.0.0 (/home/brent/astral/ruff/crates/ruff_markdown)
Compiling ruff_server v0.2.2 (/home/brent/astral/ruff/crates/ruff_server)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 23.14s
Running `target/debug/ruff --isolated check - --preview --select ANN003,PLR1712`
warning: Detected debug build without --no-cache.
All checks passed!
```
## Summary
This PR implements the following paragraph in the typing spec:
> Type checkers should infer a final attribute that is initialized in a
class body as being a class variable, except in the case of
[Dataclasses](https://typing.python.org/en/latest/spec/dataclasses.html),
where `x: Final[int] = 3` creates a dataclass field and instance-level
final attribute `x` with default value `3`; `x: ClassVar[Final[int]] =
3` is necessary to create a final class variable with value `3`. In
non-dataclasses, combining `ClassVar` and `Final` is redundant, and type
checkers may choose to warn or error on the redundancy.
>
>
https://typing.python.org/en/latest/spec/qualifiers.html#semantics-and-examples
## Test Plan
New Markdown tests
## Summary
This PR adds the new default rule set in preview. This ended up being
pretty non-invasive because the `DEFAULT_SELECTORS` are only used in one
place where `preview` isn't set to the default value of `false`.
I've currently listed each rule with a separate `RuleSelector`, which I
generated with the script below. I thought about trying to be more
clever by finding the smallest set of prefix selectors that yield the
same rule set, but I figured this would be the easiest way to add and
remove rules in the future anyway.
<details><summary>Script</summary>
<p>
```py
import json
import subprocess
import tomllib
from pathlib import Path
from string import ascii_uppercase
RULES = {
rule["code"]: rule["linter"]
for rule in json.loads(
subprocess.run(
["ruff", "rule", "--all", "--output-format=json"],
check=True,
text=True,
capture_output=True,
).stdout
)
}
for code, linter in RULES.items():
if linter == "Ruff-specific rules":
RULES[code] = "Ruff"
def kebab_to_pascal(s: str) -> str:
return "".join(part.title() for part in s.split("-"))
rules = tomllib.loads(Path("proposal.toml").read_text())["lint"]["select"]
for rule in rules:
linter = kebab_to_pascal(RULES[rule])
suffix = rule.lstrip(ascii_uppercase)
prefix = "_"
match linter:
case "Flake8Comprehensions":
suffix = suffix.removeprefix("4")
case "Pycodestyle":
prefix = ""
suffix = rule
case "Flake8Gettext":
linter = "Flake8GetText"
case "Pep8Naming":
linter = "PEP8Naming"
case "Pylint":
prefix = ""
suffix = rule.removeprefix("PL")
case "Flake8Debugger":
suffix = suffix.removeprefix("10")
print(
" " * 4,
f"RuleSelector::rule(RuleCodePrefix::{linter}("
f"codes::{linter}::{prefix}{suffix})),"
f" // {rule}",
sep="",
)
```
</p>
</details>
## Test Plan
A new CLI test showing the preview default rules. I filtered down the
snapshot to just the `linter.rules.enabled` section, which isn't
strictly necessary but was a lot shorter. I also tested manually in VS
Code to make sure I didn't miss wiring up the preview defaults in the
server:
<img width="681" height="340" alt="image"
src="https://github.com/user-attachments/assets/9f7a0cd4-968e-40d6-abf0-dfbfa496531d"
/>
I also tested in the playground.
We now infer specializations that involve generic protocols. This
includes recursing into the methods of the protocol, matching up the
signatures of the class's methods with the signatures of the protocol,
and adding bindings for any typevars that appear in the protocol
signatures.
This required several performance updates, which were pulled out and
merged as separate PRs. There is still a performance and memory hit, but
I think a reasonable one, given the new functionality that this opens
up.
There are still some ecosystem false positives that relate to how this
interacts with overload resolution, but I plan to tackle that in
follow-on PRs.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
A couple small improvements to the memory report:
- Consistent sorting between the summary and expanded view.
- Human readable memory size.
- Strip the inner whitespace from type names.
## Summary
`uv run scripts/setup_primer_project.py <projname> <directory>` will
clone the mypy-primer project `<projname>` to `<directory>` and then
install it and its dependencies (in the way defined by mypy-primer) into
`.venv` in that directory, setting it up for easy reproduction of the
primer results.
Also added instructions to `CLAUDE.md` so Claude will use this script
when reproducing ecosystem results. This can short-circuit a lot of "it
doesn't reproduce! oh I didn't install the dependencies" churn.
## Test Plan
`uv run scripts/setup_primer_project.py artigraph /tmp/artigraph` set
things up such that I was able to just run two versions of ty in
`/tmp/artigraph` and quickly repro an ecosystem diff from CI.
Fixes https://github.com/astral-sh/ty/issues/1762. If a type assertion
fails, but the actual type is an unspellable subtype of the asserted
type, we now emit `assert-type-unspellable-subtype` rather than
`type-assertion-failure`. We also ignore this error in
`scripts/conformance.py`. This results in 17 fewer false positives on
the conformance suite, and it's also something that our users have asked
for.
Charlie Marsh