Miss Islington (bot)
3b0a3c4738
[3.14] gh-138907: Support RFC 9309 in robotparser (GH-138908) (GH-149374)
...
* empty lines are always ignored instead of separating groups
* the "user-agent" line after a rule starts a new group
* groups matching the same user agent are now merged
* the rule with the longest match wins instead of the first matching rule
* in case of equal matches, the “Allow” rule wins over “Disallow”
* special characters “$” and “*” are now supported in rules
* prefer full match for user agent
(cherry picked from commit bc285e5832 )
Co-authored-by: Serhiy Storchaka <storchaka@gmail.com >
2026-05-04 18:28:54 +00:00
..
2026-02-06 14:06:50 -05:00
2025-01-21 19:52:21 +00:00
2024-10-17 20:10:55 +02:00
2025-02-25 13:38:47 +01:00
2024-11-11 17:49:48 -05:00
2024-12-23 14:17:19 +10:00
2025-10-21 16:22:29 +00:00
2025-05-21 14:24:23 +00:00
2026-04-14 10:42:10 -07:00
2026-03-26 19:55:25 +00:00
2026-05-03 16:04:14 +00:00
2024-11-11 17:49:48 -05:00
2026-03-21 12:05:06 +00:00
2026-04-09 14:07:55 +00:00
2026-04-09 20:51:16 +05:30
2025-02-11 19:16:29 +02:00
2024-10-16 16:50:46 -06:00
2025-01-14 13:52:58 +02:00
2026-03-21 20:19:07 +02:00
2025-12-17 22:23:27 +05:30
2024-12-27 15:13:41 +00:00
2025-10-04 16:40:21 +00:00
2026-03-21 20:19:07 +02:00
2025-08-29 07:37:39 +03:00
2026-03-21 20:19:07 +02:00
2025-11-06 05:46:15 +00:00
2024-11-11 17:49:48 -05:00
2026-02-06 14:06:50 -05:00
2024-11-11 17:49:48 -05:00
2025-05-06 18:33:52 +03:00
2026-01-23 19:24:06 +00:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-28 19:28:14 +00:00
2024-10-20 17:36:27 -07:00
2026-02-06 14:06:50 -05:00
2026-04-27 21:00:46 +01:00
2024-11-11 17:49:48 -05:00
2024-11-11 17:49:48 -05:00
2024-11-11 17:49:48 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2025-11-17 14:57:26 +00:00
2025-08-16 11:04:59 +03:00
2025-05-20 12:15:34 +00:00
2026-02-11 08:09:56 -05:00
2026-02-06 14:06:50 -05:00
2025-10-07 19:13:35 +01:00
2026-03-26 19:55:25 +00:00
2026-02-06 14:06:50 -05:00
2025-06-01 08:24:41 +00:00
2025-10-04 16:45:35 +03:00
2026-01-05 20:01:08 +00:00
2025-06-12 08:19:26 -06:00
2026-03-09 16:36:00 +00:00
2025-11-06 17:19:30 +00:00
2025-06-12 08:19:26 -06:00
2026-02-06 14:06:50 -05:00
2025-09-11 12:22:37 +03:00
2026-03-02 21:18:12 +00:00
2026-01-27 11:39:13 +01:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2025-10-09 17:19:35 +00:00
2026-02-06 14:06:50 -05:00
2026-03-23 14:07:24 +01:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-19 11:51:18 +00:00
2025-10-07 19:13:35 +01:00
2026-02-18 15:14:44 +01:00
2026-02-18 15:14:44 +01:00
2026-03-10 13:38:20 +02:00
2026-02-19 11:51:18 +00:00
2026-02-06 14:06:50 -05:00
2026-02-19 11:51:18 +00:00
2026-03-21 20:19:07 +02:00
2026-02-06 14:06:50 -05:00
2024-11-11 17:49:48 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2025-07-07 19:50:56 +03:00
2025-01-18 20:34:40 +02:00
2026-02-11 08:09:56 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2025-11-04 19:35:20 +00:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-03-11 08:08:09 +00:00
2026-02-11 08:09:56 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-04-14 17:06:41 -07:00
2025-01-28 01:12:45 +01:00
2026-03-23 23:31:32 +01:00
2025-06-29 15:05:40 +00:00
2026-02-19 11:51:18 +00:00
2026-02-06 14:06:50 -05:00
2024-12-23 14:17:19 +10:00
2025-07-19 09:56:55 +00:00
2026-02-06 14:06:50 -05:00
2025-08-16 11:04:59 +03:00
2026-02-06 14:06:50 -05:00
2026-04-15 17:15:42 +01:00
2026-03-07 17:14:45 +00:00
2026-04-26 21:12:52 +03:00
2024-12-23 14:17:19 +10:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-04-09 11:00:23 +00:00
2026-02-06 14:06:50 -05:00
2026-02-19 11:51:18 +00:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-01-13 09:36:59 +00:00
2025-09-03 06:26:29 +00:00
2026-04-27 22:30:00 +08:00
2025-09-17 21:58:17 +08:00
2025-12-14 10:17:54 +00:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-04-15 18:27:06 +03:00
2026-01-20 09:34:29 +00:00
2026-02-06 14:06:50 -05:00
2024-11-11 17:49:48 -05:00
2024-11-11 17:49:48 -05:00
2026-04-30 20:42:35 +01:00
2026-01-08 13:45:57 +00:00
2026-04-30 20:42:35 +01:00
2026-02-11 08:09:56 -05:00
2026-02-25 16:57:32 +01:00
2026-02-06 14:06:50 -05:00
2024-09-23 23:09:53 +00:00
2026-02-06 14:06:50 -05:00
2026-05-04 13:10:51 +00:00
2026-04-24 05:52:35 +00:00
2026-02-11 08:09:56 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2025-11-10 15:20:00 +01:00
2026-02-17 12:53:42 +00:00
2024-11-11 17:49:48 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-04-23 00:41:40 +03:00
2025-09-21 21:24:10 +01:00
2024-11-11 17:49:48 -05:00
2026-02-19 11:51:18 +00:00
2026-04-27 01:49:27 +00:00
2025-06-22 20:12:57 +00:00
2024-11-11 17:49:48 -05:00
2024-11-11 17:49:48 -05:00
2025-10-07 19:15:22 +01:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-04-23 00:41:40 +03:00
2024-11-11 17:49:48 -05:00
2026-04-09 11:00:23 +00:00
2026-02-11 08:09:56 -05:00
2026-02-17 12:53:42 +00:00
2026-02-06 14:06:50 -05:00
2024-11-11 17:49:48 -05:00
2026-04-07 12:48:29 +02:00
2026-02-06 14:06:50 -05:00
2026-03-21 20:19:07 +02:00
2026-02-06 14:06:50 -05:00
2026-02-19 11:51:18 +00:00
2026-02-06 14:06:50 -05:00
2026-02-11 08:09:56 -05:00
2026-02-19 11:51:18 +00:00
2026-02-19 11:51:18 +00:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2024-10-29 23:12:02 +00:00
2026-02-17 12:53:42 +00:00
2025-06-12 08:19:26 -06:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-04-23 02:45:46 +00:00
2026-02-19 11:51:18 +00:00
2024-11-11 17:49:48 -05:00
2026-02-19 11:51:18 +00:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-17 12:53:42 +00:00
2025-07-07 09:08:41 +00:00
2026-02-17 12:53:42 +00:00
2026-02-17 12:53:42 +00:00
2026-02-17 12:53:42 +00:00
2026-02-17 12:53:42 +00:00
2026-02-17 12:53:42 +00:00
2026-03-26 16:49:43 +01:00
2026-02-17 12:53:42 +00:00
2024-11-11 17:49:48 -05:00
2026-02-06 14:06:50 -05:00
2024-11-11 17:49:48 -05:00
2026-03-28 05:53:14 -04:00
2026-02-06 14:06:50 -05:00
2024-11-11 17:49:48 -05:00
2026-04-14 17:06:41 -07:00
2026-03-21 20:19:07 +02:00
2026-02-06 14:06:50 -05:00
2026-04-09 13:39:39 +00:00
2026-03-21 20:19:07 +02:00
2025-10-05 07:41:43 +03:00
2026-02-06 14:06:50 -05:00
2026-04-27 23:07:00 +03:00
2026-04-05 18:13:29 +00:00
2024-11-11 17:49:48 -05:00
2024-12-23 14:17:19 +10:00
2026-02-06 14:06:50 -05:00
2025-04-14 13:24:46 +00:00
2026-04-11 15:30:24 +00:00
2026-02-17 12:53:42 +00:00
2026-02-06 14:06:50 -05:00
2026-02-19 11:51:18 +00:00
2026-02-06 14:06:50 -05:00
2024-11-11 17:49:48 -05:00
2026-02-06 14:06:50 -05:00
2026-02-19 11:51:18 +00:00
2026-02-11 08:09:56 -05:00
2025-07-22 13:32:00 +03:00
2026-02-06 14:06:50 -05:00
2025-06-27 00:05:53 +00:00
2026-03-18 14:28:52 +01:00
2025-11-08 15:02:35 +00:00
2026-02-06 14:06:50 -05:00
2025-08-16 11:04:59 +03:00
2026-02-19 11:51:18 +00:00
2026-02-19 11:51:18 +00:00
2026-02-19 11:51:18 +00:00
2026-02-19 11:51:18 +00:00
2026-02-06 14:06:50 -05:00
2026-02-19 11:51:18 +00:00
2026-02-06 14:06:50 -05:00
2025-03-19 16:42:11 +01:00
2026-02-18 17:59:17 +01:00
2026-02-06 14:06:50 -05:00
2026-05-01 14:42:33 +02:00
2026-02-17 12:53:42 +00:00
2025-03-02 18:16:51 +01:00
2026-02-17 12:53:42 +00:00
2026-02-19 11:51:18 +00:00
2026-02-11 08:09:56 -05:00
2025-03-04 11:44:19 -08:00
2026-04-22 19:59:18 -07:00
2025-09-24 18:11:20 +01:00
2026-03-21 20:19:07 +02:00
2026-02-06 14:06:50 -05:00
2026-04-14 23:45:18 +00:00
2025-08-16 11:04:59 +03:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-11 08:09:56 -05:00
2026-05-04 18:28:54 +00:00
2024-11-11 17:49:48 -05:00
2026-02-06 14:06:50 -05:00
2026-02-05 17:24:24 +02:00
2026-02-06 14:06:50 -05:00
2026-03-03 12:50:45 +00:00
2026-02-11 08:09:56 -05:00
2026-02-06 14:06:50 -05:00
2026-02-19 11:51:18 +00:00
2026-02-19 11:51:18 +00:00
2026-02-11 08:09:56 -05:00
2024-11-11 17:49:48 -05:00
2026-02-17 12:53:42 +00:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-11 08:09:56 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-02-06 14:06:50 -05:00
2026-03-21 20:19:07 +02:00
2024-12-10 12:22:37 +02:00
2026-03-21 20:19:07 +02:00
2026-02-06 14:06:50 -05:00
2026-03-20 16:20:39 +00:00
2026-02-06 14:06:50 -05:00
2025-11-10 15:20:00 +01:00
2026-03-21 20:19:07 +02:00