Serhiy Storchaka
561d1b95b9
[3.13] gh-138907: Support RFC 9309 in robotparser (GH-138908) (GH-149376)
...
* empty lines are always ignored instead of separating groups
* the "user-agent" line after a rule starts a new group
* groups matching the same user agent are now merged
* the rule with the longest match wins instead of the first matching rule
* in case of equal matches, the “Allow” rule wins over “Disallow”
* special characters “$” and “*” are now supported in rules
* prefer full match for user agent
(cherry picked from commit bc285e5832 )
2026-05-04 18:43:03 +00:00
..
2025-08-02 16:19:20 +02:00
2024-07-04 23:05:02 +00:00
2024-10-11 08:20:46 +00:00
2025-02-26 11:02:53 +01:00
2024-11-17 22:20:32 +02:00
2024-12-23 04:40:59 +00:00
2026-04-14 17:53:11 +00:00
2026-03-26 19:52:07 +00:00
2026-05-02 20:05:24 +03:00
2024-11-17 22:20:32 +02:00
2026-03-21 12:06:05 +00:00
2026-04-09 14:07:54 +00:00
2023-07-05 08:07:02 -07:00
2023-07-29 08:48:10 +03:00
2026-04-09 20:22:59 +05:30
2024-09-02 16:00:04 +02:00
2023-07-21 12:40:37 +03:00
2024-01-10 15:01:18 +02:00
2026-03-21 18:41:50 +00:00
2025-12-17 16:53:14 +00:00
2024-08-30 20:14:48 +03:00
2025-10-04 22:12:03 +05:30
2026-03-21 18:41:50 +00:00
2025-08-16 16:29:26 +05:30
2026-03-21 18:41:50 +00:00
2025-11-06 05:45:18 +00:00
2024-11-17 22:20:32 +02:00
2024-05-08 19:52:39 +00:00
2024-11-17 22:20:32 +02:00
2024-03-05 17:41:53 +02:00
2026-01-23 19:19:22 +00:00
2025-12-10 22:46:50 +00:00
2024-05-08 19:52:39 +00:00
2026-03-01 00:20:20 +01:00
2024-10-21 00:42:06 +00:00
2024-05-08 19:52:39 +00:00
2026-05-03 17:42:39 +01:00
2024-11-17 22:20:32 +02:00
2024-11-17 22:20:32 +02:00
2024-11-17 22:20:32 +02:00
2025-10-14 08:32:41 +00:00
2024-05-08 19:52:39 +00:00
2025-11-17 19:55:54 +02:00
2025-08-15 17:50:47 +00:00
2025-05-20 12:17:10 +00:00
2025-10-18 19:03:38 +00:00
2024-05-08 19:52:39 +00:00
2025-10-05 20:28:30 +00:00
2026-03-26 19:52:07 +00:00
2024-07-19 09:13:08 +00:00
2025-06-01 08:27:26 +00:00
2026-03-09 16:43:01 +00:00
2023-07-29 08:48:10 +03:00
2024-11-05 03:04:32 +00:00
2025-09-05 13:47:19 -07:00
2026-03-02 21:19:05 +00:00
2026-01-27 11:39:20 +01:00
2025-05-26 10:29:53 +00:00
2024-05-08 19:52:39 +00:00
2025-10-09 17:22:01 +00:00
2023-05-25 15:45:46 +02:00
2025-11-11 16:13:19 -08:00
2025-12-16 10:56:32 +01:00
2024-05-08 19:52:39 +00:00
2024-05-08 19:52:39 +00:00
2026-02-19 12:13:53 +00:00
2025-10-05 20:28:30 +00:00
2026-03-10 13:38:34 +02:00
2026-02-19 12:13:53 +00:00
2025-12-17 21:42:35 +05:30
2023-05-23 19:40:02 +02:00
2023-09-27 16:07:28 +03:00
2026-02-19 12:13:53 +00:00
2026-03-21 18:41:50 +00:00
2026-01-07 22:42:48 +00:00
2024-11-17 22:20:32 +02:00
2025-08-18 15:05:12 +00:00
2024-05-08 19:52:39 +00:00
2025-07-07 19:51:04 +03:00
2025-01-18 18:40:30 +00:00
2024-05-08 19:52:39 +00:00
2025-03-03 12:04:51 +02:00
2023-07-16 19:14:08 +02:00
2024-05-08 19:52:39 +00:00
2025-06-26 20:13:46 -04:00
2025-11-04 19:35:14 +00:00
2024-05-08 19:52:39 +00:00
2026-01-06 15:32:56 -05:00
2024-05-08 19:52:39 +00:00
2026-03-11 08:08:51 +00:00
2026-01-06 15:32:56 -05:00
2024-05-08 19:52:39 +00:00
2024-07-19 09:13:08 +00:00
2025-11-17 19:55:54 +02:00
2026-04-14 17:06:49 -07:00
2025-01-28 00:19:06 +00:00
2026-03-11 08:08:51 +00:00
2024-05-08 19:52:39 +00:00
2026-02-19 12:13:53 +00:00
2025-02-04 14:25:18 +00:00
2024-06-12 14:51:35 +00:00
2024-12-23 04:40:59 +00:00
2025-01-13 10:35:26 +00:00
2025-08-05 14:54:31 +01:00
2025-08-15 17:50:47 +00:00
2024-07-13 14:45:32 +00:00
2026-04-15 17:15:31 +01:00
2026-03-07 17:15:16 +00:00
2025-09-07 10:53:56 +03:00
2024-12-23 04:40:59 +00:00
2024-08-30 20:14:48 +03:00
2024-05-08 19:52:39 +00:00
2026-04-09 10:51:12 +00:00
2024-05-08 19:52:39 +00:00
2026-02-19 12:13:53 +00:00
2025-11-17 19:55:54 +02:00
2025-10-02 17:17:55 -07:00
2025-11-06 17:37:31 +00:00
2025-09-03 06:27:17 +00:00
2024-05-08 19:52:39 +00:00
2026-04-27 22:31:35 +08:00
2025-09-17 21:54:16 +08:00
2025-12-14 10:18:08 +00:00
2025-11-04 14:11:15 +01:00
2026-01-25 17:09:22 +00:00
2025-10-14 08:24:31 +00:00
2026-04-15 19:01:04 +03:00
2026-01-20 09:32:55 +00:00
2025-10-07 14:01:51 +02:00
2024-11-17 22:20:32 +02:00
2024-11-17 22:20:32 +02:00
2025-03-27 15:42:47 +02:00
2026-01-08 13:46:33 +00:00
2026-04-11 17:10:53 -07:00
2025-10-16 20:10:31 +02:00
2025-08-15 17:50:47 +00:00
2026-01-23 11:35:27 +01:00
2023-05-04 10:17:12 +02:00
2024-09-24 00:13:45 +00:00
2025-05-21 17:07:03 +00:00
2026-05-04 13:10:17 +00:00
2026-04-08 17:56:10 +00:00
2025-11-08 10:30:15 +00:00
2024-01-11 18:25:18 -06:00
2024-05-08 19:52:39 +00:00
2024-05-08 19:52:39 +00:00
2025-12-16 11:32:19 +00:00
2025-06-18 17:44:43 +00:00
2025-11-04 18:57:32 +00:00
2025-12-25 12:41:24 +00:00
2024-05-08 19:52:39 +00:00
2026-02-17 12:56:29 +00:00
2024-11-17 22:20:32 +02:00
2024-07-19 09:13:08 +00:00
2025-06-30 14:49:35 +00:00
2024-07-26 22:20:01 +00:00
2025-09-21 11:05:10 +00:00
2024-05-08 19:52:39 +00:00
2024-11-17 22:20:32 +02:00
2026-02-19 12:13:53 +00:00
2026-04-27 01:48:39 +00:00
2024-05-08 19:52:39 +00:00
2025-06-22 20:12:20 +00:00
2024-11-17 22:20:32 +02:00
2024-11-17 22:20:32 +02:00
2025-08-05 14:54:31 +01:00
2024-05-08 19:52:39 +00:00
2024-12-23 04:40:59 +00:00
2026-02-03 14:16:28 +00:00
2026-03-11 08:08:51 +00:00
2024-11-17 22:20:32 +02:00
2026-04-09 10:51:12 +00:00
2025-11-17 19:55:54 +02:00
2026-02-17 12:56:29 +00:00
2024-05-08 19:52:39 +00:00
2024-11-17 22:20:32 +02:00
2026-04-07 12:26:51 +02:00
2025-09-17 15:05:56 +00:00
2026-03-21 18:41:50 +00:00
2024-05-08 19:52:39 +00:00
2026-02-19 12:13:53 +00:00
2025-10-12 16:56:03 +00:00
2025-08-16 10:56:21 -04:00
2026-02-19 12:13:53 +00:00
2026-02-19 12:13:53 +00:00
2024-05-08 19:52:39 +00:00
2024-05-08 19:55:12 +00:00
2024-10-29 23:23:18 +00:00
2026-02-17 12:56:29 +00:00
2023-09-05 12:35:52 +01:00
2025-07-25 08:48:28 -06:00
2024-05-08 19:52:39 +00:00
2026-01-13 00:25:55 +00:00
2026-04-23 02:46:31 +00:00
2026-02-19 12:13:53 +00:00
2024-11-17 22:20:32 +02:00
2024-05-08 19:52:39 +00:00
2026-02-19 12:13:53 +00:00
2024-05-08 19:52:39 +00:00
2024-05-08 19:52:39 +00:00
2024-05-08 19:52:39 +00:00
2026-02-17 12:56:29 +00:00
2025-07-07 09:10:20 +00:00
2026-02-17 12:56:29 +00:00
2026-02-17 12:56:29 +00:00
2026-02-17 12:56:29 +00:00
2026-02-17 12:56:29 +00:00
2026-02-17 12:56:29 +00:00
2026-03-26 16:49:46 +01:00
2026-02-17 12:56:29 +00:00
2024-11-17 22:20:32 +02:00
2025-12-16 11:27:42 +00:00
2024-11-17 22:20:32 +02:00
2026-03-28 05:53:21 -04:00
2025-11-20 16:35:54 +00:00
2024-11-17 22:20:32 +02:00
2026-04-14 17:06:49 -07:00
2026-03-21 18:41:50 +00:00
2024-05-08 19:52:39 +00:00
2024-07-21 06:10:06 +00:00
2026-04-09 13:40:28 +00:00
2026-03-21 18:41:50 +00:00
2024-05-08 19:52:39 +00:00
2026-04-27 20:22:47 +00:00
2026-04-05 18:14:03 +00:00
2024-11-17 22:20:32 +02:00
2024-12-23 04:40:59 +00:00
2024-10-08 17:58:47 +10:00
2023-06-01 09:14:02 +02:00
2025-11-04 18:36:40 +00:00
2026-02-17 12:56:29 +00:00
2025-09-17 13:22:36 +00:00
2026-02-19 12:13:53 +00:00
2024-05-08 19:52:39 +00:00
2025-07-20 14:16:26 +01:00
2024-11-17 22:20:32 +02:00
2026-01-06 17:14:42 +00:00
2026-02-19 12:13:53 +00:00
2026-01-08 18:12:51 +02:00
2024-05-26 08:20:37 +00:00
2025-05-16 12:25:22 +00:00
2025-11-08 15:03:07 +00:00
2024-05-08 19:52:39 +00:00
2025-08-15 17:50:47 +00:00
2026-02-19 12:13:53 +00:00
2026-02-19 12:13:53 +00:00
2026-02-19 12:13:53 +00:00
2026-02-19 12:13:53 +00:00
2026-01-19 12:26:26 +00:00
2026-02-19 12:13:53 +00:00
2024-05-08 19:52:39 +00:00
2025-03-24 12:34:23 +00:00
2025-06-08 17:55:12 -04:00
2025-01-20 00:11:27 +00:00
2026-05-02 12:13:57 +01:00
2026-02-17 12:56:29 +00:00
2025-03-02 17:22:48 +00:00
2026-02-17 12:56:29 +00:00
2026-02-19 12:13:53 +00:00
2025-11-11 19:42:01 +00:00
2024-10-09 20:03:30 +01:00
2026-04-22 19:57:40 -07:00
2025-09-24 10:36:37 +02:00
2026-03-21 18:41:50 +00:00
2025-08-08 14:08:01 +00:00
2026-04-14 23:45:32 +00:00
2025-08-15 17:50:47 +00:00
2024-05-08 19:52:39 +00:00
2026-02-05 15:47:52 +00:00
2025-08-03 17:24:26 +00:00
2026-05-04 18:43:03 +00:00
2024-05-08 19:52:39 +00:00
2024-11-17 22:20:32 +02:00
2025-02-25 15:24:46 +02:00
2026-02-05 15:47:52 +00:00
2025-10-18 11:03:03 +00:00
2026-03-03 12:51:20 +00:00
2024-08-11 20:57:19 +00:00
2025-11-17 19:55:54 +02:00
2026-02-19 12:13:53 +00:00
2026-02-19 12:13:53 +00:00
2026-02-05 18:22:07 +02:00
2024-11-17 22:20:32 +02:00
2026-02-17 12:56:29 +00:00
2025-11-19 13:07:07 +02:00
2024-05-08 19:52:39 +00:00
2025-07-07 09:10:20 +00:00
2025-11-05 19:06:37 +00:00
2025-11-29 07:14:23 +00:00
2024-05-08 19:52:39 +00:00
2025-07-07 09:10:20 +00:00
2025-11-17 13:02:01 +00:00
2026-03-21 18:41:50 +00:00
2024-12-10 10:45:30 +00:00
2026-03-21 18:41:50 +00:00
2024-08-30 20:14:48 +03:00
2025-11-29 07:18:06 +00:00
2025-08-31 00:59:47 +00:00
2025-10-21 14:45:02 -07:00
2026-03-21 18:41:50 +00:00