Ruben Vorderman
eae7dad402
gh-95534: Improve gzip reading speed by 10% ( #97664 )
...
Change summary:
+ There is now a `gzip.READ_BUFFER_SIZE` constant that is 128KB. Other programs that read in 128KB chunks: pigz and cat. So this seems best practice among good programs. Also it is faster than 8 kb chunks.
+ a zlib._ZlibDecompressor was added. This is the _bz2.BZ2Decompressor ported to zlib. Since the zlib.Decompress object is better for in-memory decompression, the _ZlibDecompressor is hidden. It only makes sense in file decompression, and that is already implemented now in the gzip library. No need to bother the users with this.
+ The ZlibDecompressor uses the older Cpython arrange_output_buffer functions, as those are faster and more appropriate for the use case.
+ GzipFile.read has been optimized. There is no longer a `unconsumed_tail` member to write back to padded file. This is instead handled by the ZlibDecompressor itself, which has an internal buffer. `_add_read_data` has been inlined, as it was just two calls.
EDIT: While I am adding improvements anyway, I figured I could add another one-liner optimization now to the python -m gzip application. That read chunks in io.DEFAULT_BUFFER_SIZE previously, but has been updated now to use READ_BUFFER_SIZE chunks.
2022-10-16 19:10:58 -07:00
..
2022-09-14 14:34:40 +01:00
2022-08-04 18:12:35 +02:00
2022-06-21 10:24:33 +02:00
2022-09-08 11:52:07 -07:00
2022-10-12 17:26:21 +02:00
2022-10-15 16:09:30 -07:00
2022-09-26 17:27:44 +01:00
2022-07-25 09:17:25 +03:00
2022-06-27 20:27:19 +02:00
2022-10-16 15:00:39 -04:00
2022-09-24 14:38:53 +03:00
2022-08-04 18:12:35 +02:00
2022-05-25 08:34:14 -07:00
2022-08-01 12:25:16 +02:00
2022-09-27 14:05:05 +03:00
2022-06-27 16:19:47 +02:00
2022-10-15 07:57:53 -07:00
2022-09-27 14:05:05 +03:00
2022-10-04 17:29:18 -07:00
2022-08-13 21:56:08 +02:00
2022-10-07 11:06:23 -07:00
2022-10-07 11:23:06 -07:00
2022-06-15 18:49:14 +02:00
2022-05-19 20:23:53 +01:00
2022-10-02 17:41:01 -07:00
2022-07-14 11:57:18 +02:00
2022-05-03 07:21:42 -06:00
2022-10-07 20:17:08 +03:00
2022-10-03 10:42:54 +03:00
2022-08-28 14:27:42 -07:00
2022-05-19 17:46:15 +02:00
2022-06-16 13:44:58 +02:00
2022-10-04 15:31:16 -07:00
2022-06-13 16:56:03 +02:00
2022-06-13 16:56:03 +02:00
2022-08-05 17:41:29 -07:00
2022-10-05 14:21:16 +03:00
2022-07-27 20:28:06 +02:00
2022-07-14 11:57:18 +02:00
2022-06-15 14:09:56 +02:00
2022-08-05 17:41:29 -07:00
2022-10-15 11:40:22 -07:00
2022-05-16 16:02:37 +02:00
2022-07-01 15:33:35 +01:00
2022-05-08 17:49:09 +03:00
2022-06-15 11:42:10 +02:00
2022-05-12 11:26:11 -04:00
2022-08-25 06:18:38 -05:00
2022-10-05 01:34:03 +01:00
2022-09-30 09:43:02 +01:00
2022-05-03 10:17:54 +03:00
2022-06-16 13:44:58 +02:00
2022-06-13 16:56:03 +02:00
2022-10-07 20:17:08 +03:00
2022-07-02 15:53:43 +09:00
2022-10-01 20:57:17 -07:00
2022-06-22 14:05:45 +02:00
2022-06-13 16:56:03 +02:00
2022-09-06 19:37:18 -04:00
2022-07-14 16:53:06 -07:00
2022-09-10 22:44:10 +02:00
2022-06-13 16:56:03 +02:00
2022-09-27 23:23:42 +01:00
2022-10-10 08:28:41 -07:00
2022-06-07 10:44:29 +02:00
2022-10-05 01:34:03 +01:00
2022-10-15 21:18:59 -07:00
2022-08-09 16:04:05 -07:00
2022-10-03 13:37:15 -07:00
2022-10-07 14:41:35 -07:00
2022-09-27 23:23:42 +01:00
2022-10-03 13:55:45 -07:00
2022-05-03 21:36:52 -06:00
2022-10-04 17:30:03 -07:00
2022-05-02 12:37:48 +03:00
2022-06-13 16:56:03 +02:00
2022-09-16 17:37:30 +03:00
2022-10-05 01:34:03 +01:00
2022-10-05 08:52:35 +01:00
2022-06-03 00:44:48 +02:00
2022-07-28 11:20:10 +02:00
2022-08-04 18:12:35 +02:00
2022-05-23 21:09:59 +02:00
2022-05-24 15:41:32 +02:00
2022-05-05 17:39:33 +01:00
2022-06-09 10:12:43 +03:00
2022-05-08 17:10:11 +03:00
2022-10-07 16:12:28 -07:00
2022-07-14 11:57:18 +02:00
2022-10-07 11:15:34 -07:00
2022-04-26 07:59:56 +03:00
2022-10-06 17:11:59 -07:00
2022-06-06 19:24:11 +02:00
2022-09-08 19:32:40 -07:00
2022-09-08 19:32:40 -07:00
2022-09-02 09:35:08 -07:00
2022-06-13 16:56:03 +02:00
2022-06-13 16:56:03 +02:00
2022-08-04 18:12:35 +02:00
2022-08-08 14:00:17 +03:00
2022-08-30 17:43:23 -05:00
2022-10-07 16:30:23 -07:00
2022-05-05 16:22:32 -07:00
2022-07-11 07:40:53 +01:00
2022-06-03 22:14:31 +02:00
2022-10-05 01:34:03 +01:00
2022-10-03 13:55:45 -07:00
2022-10-05 15:25:55 -07:00
2022-06-26 10:34:06 +02:00
2022-07-08 19:01:47 +01:00
2022-08-17 07:24:53 +02:00
2022-10-05 01:34:03 +01:00
2022-09-15 10:33:13 +01:00
2022-08-04 18:12:35 +02:00
2022-07-24 11:42:11 +09:00
2022-08-04 18:12:35 +02:00
2022-08-10 19:25:39 +01:00
2022-06-13 16:56:03 +02:00
2022-06-05 11:46:29 +03:00
2022-09-02 11:10:58 -05:00
2022-10-08 07:57:09 -07:00
2022-07-16 19:51:53 +01:00
2022-05-03 10:17:54 +03:00
2022-10-07 10:36:40 -07:00
2022-06-13 16:56:03 +02:00
2022-06-14 01:09:23 +02:00
2022-10-04 00:36:52 +01:00
2022-05-08 07:24:54 -07:00
2022-05-02 08:29:49 +03:00
2022-05-16 16:02:37 +02:00
2022-07-28 07:40:36 +03:00
2022-08-04 18:12:35 +02:00
2022-06-13 16:56:03 +02:00
2022-05-08 17:49:09 +03:00
2022-05-08 17:49:09 +03:00
2022-08-04 18:12:35 +02:00
2022-08-06 18:52:53 +02:00
2022-05-02 23:11:06 -06:00
2022-06-28 11:51:13 +02:00
2022-06-13 16:56:03 +02:00
2022-08-18 13:16:33 +02:00
2022-06-13 16:56:03 +02:00
2022-08-30 11:11:44 -07:00
2022-06-21 13:16:57 -07:00
2022-06-13 16:56:03 +02:00
2022-04-13 10:47:41 -07:00
2022-08-25 14:19:16 -05:00
2022-10-07 15:24:17 -07:00
2022-10-03 13:55:45 -07:00
2022-06-24 08:46:53 +02:00
2022-09-24 14:38:53 +03:00
2022-10-05 01:34:03 +01:00
2022-10-03 13:50:30 -07:00
2022-06-14 13:45:50 -07:00
2022-05-19 12:43:16 +02:00
2022-09-05 20:06:30 +01:00
2022-06-30 10:18:18 +01:00
2022-09-29 00:07:07 +02:00
2022-06-13 16:56:03 +02:00
2022-06-26 12:41:19 +02:00
2022-09-03 13:38:38 +01:00
2022-10-06 05:16:16 -07:00
2022-05-23 10:39:57 +02:00
2022-06-13 16:56:03 +02:00
2022-10-05 01:34:03 +01:00
2022-09-10 22:44:10 +02:00
2022-05-03 15:17:57 -06:00
2022-08-23 09:16:02 -07:00
2022-10-05 11:42:26 -07:00
2022-06-09 08:32:35 +02:00
2022-04-14 12:50:11 -07:00
2022-06-15 18:49:14 +02:00
2022-06-06 19:24:11 +02:00
2022-06-13 16:56:03 +02:00
2022-05-08 17:49:09 +03:00
2022-06-10 16:59:55 +01:00
2022-09-04 13:15:59 +01:00
2022-07-18 10:10:22 -07:00
2022-07-28 23:48:33 +01:00
2022-04-25 16:26:43 -07:00
2022-08-12 14:23:41 -07:00
2022-07-13 09:13:10 -07:00
2022-07-08 14:40:35 +02:00
2022-09-13 13:03:41 +01:00
2022-09-01 02:47:40 -07:00
2022-06-28 10:01:43 +03:00
2022-04-21 19:28:34 -07:00
2022-09-07 21:09:20 +01:00
2022-05-08 17:49:09 +03:00
2022-05-03 10:17:54 +03:00
2022-09-17 10:09:28 -07:00
2022-10-06 09:11:47 -07:00
2022-06-06 19:24:11 +02:00
2022-05-08 17:10:11 +03:00
2022-05-28 20:25:51 -07:00
2022-08-06 18:53:19 +02:00
2022-06-27 20:27:19 +02:00
2022-08-18 11:16:07 +01:00
2022-10-06 15:40:22 -07:00
2022-08-04 18:12:35 +02:00
2022-06-11 11:55:11 +02:00
2022-07-13 09:46:04 -05:00
2022-08-03 16:19:36 -07:00
2022-06-16 21:48:26 +02:00
2022-09-08 13:51:44 -05:00
2022-05-02 16:36:29 -06:00
2022-05-16 16:02:37 +02:00
2022-05-06 03:41:24 +02:00
2022-09-20 18:34:13 -07:00
2022-06-13 16:56:03 +02:00
2022-05-16 16:02:37 +02:00
2022-08-04 18:12:35 +02:00
2022-06-13 16:56:03 +02:00
2022-07-04 15:29:19 +02:00
2022-10-05 12:48:59 +03:00
2022-06-16 13:44:58 +02:00
2022-06-24 11:48:53 +01:00
2022-08-05 17:41:29 -07:00
2022-04-22 15:48:03 -07:00
2022-08-24 18:37:18 -07:00
2022-06-17 13:16:51 +02:00
2022-08-01 18:02:09 +02:00
2022-09-07 14:23:54 -07:00
2022-04-23 14:48:17 -07:00
2022-07-27 09:19:49 +02:00
2022-08-08 14:12:05 +02:00
2022-06-06 19:24:11 +02:00
2022-08-18 13:48:27 -05:00
2022-09-24 13:43:16 +03:00
2022-09-25 14:32:48 +01:00
2022-10-04 17:47:49 -07:00
2022-04-25 16:26:43 -07:00
2022-06-15 11:42:10 +02:00
2022-09-17 10:09:28 -07:00
2022-07-05 20:18:47 +02:00
2022-09-19 14:02:24 -07:00
2022-09-02 09:35:08 -07:00
2022-09-11 09:51:23 +02:00
2022-10-07 20:17:08 +03:00
2022-06-25 12:53:37 +02:00
2022-06-13 16:56:03 +02:00
2022-04-26 10:45:08 -07:00
2022-10-04 23:37:33 +01:00
2022-06-15 13:52:13 +02:00
2022-06-13 16:56:03 +02:00
2022-08-24 23:21:39 +01:00
2022-07-27 20:28:06 +02:00
2022-10-03 17:12:36 -07:00
2022-05-08 17:10:11 +03:00
2022-10-06 10:39:17 -07:00
2022-10-15 23:13:33 +01:00
2022-09-06 17:37:47 +01:00
2022-08-12 19:27:50 +02:00
2022-09-30 09:43:02 +01:00
2022-09-13 19:35:52 -07:00
2022-08-13 21:56:08 +02:00
2022-05-25 15:57:26 +02:00
2022-10-15 11:40:22 -07:00
2022-09-21 14:52:40 +02:00
2022-06-13 16:56:03 +02:00
2022-04-17 14:04:29 -04:00
2022-09-05 20:14:50 +03:00
2022-04-13 19:19:16 -07:00
2022-08-04 18:12:35 +02:00
2022-05-24 14:09:52 +02:00
2022-05-23 10:39:57 +02:00
2022-05-23 10:39:57 +02:00
2022-09-19 16:06:25 -07:00
2022-06-06 19:24:11 +02:00
2022-10-14 16:58:54 +01:00
2022-06-16 13:44:58 +02:00
2022-06-16 13:44:58 +02:00
2022-09-23 16:08:21 +01:00
2022-06-14 07:18:03 +03:00
2022-05-22 18:54:24 -07:00
2022-09-08 22:02:04 +01:00
2022-04-29 18:22:10 -07:00
2022-08-01 09:52:39 -07:00
2022-09-02 09:35:08 -07:00
2022-06-23 16:48:28 -07:00
2022-06-06 19:24:11 +02:00
2022-09-03 08:58:25 +03:00
2022-09-03 08:58:25 +03:00
2022-07-16 18:07:53 -07:00
2022-10-16 19:10:58 -07:00