The default memory allocation with pipes was too passive/static,
resulting in not allocating enough memory to enable threading.
By dynamically reallocating the buffer when reading from
unknown sized inputs we better use available memory and threads.
$ time seq 10000000 -1 0 | sort-old >/dev/null
real 0m16.523s
user 0m16.900s
sys 0m0.167s
$ time seq 10000000 -1 0 | sort-old -S1G >/dev/null
real 0m12.263s
user 0m29.646s
sys 0m0.527s
$ time seq 10000000 -1 0 | sort-new >/dev/null
real 0m12.994s
user 0m31.266s
sys 0m0.716s
It also avoids the overhead of writing to temp files
for modestly sized inputs. For example the following
input would induce interaction with temp storage:
$ seq 125000 | wc -c
763895
* src/sort.c (sort_buffer_size): Rename to ...
(sort_buffer_policy): ... here, and adjust to set
an initial size and limit, rather than just a size.
(fillbuf): Add a POLICY parameter, and use that
to call maybe_growbuf() as needed.
(maybe_growbuf): Return true if POLICY dictates we
should grow the buffer, and try_growbuf() was
able to reallocate the larger buffer.
* tests/sort/sort-buffer-size.sh: Add a new test.
* tests/local.mk: Reference new test.
* NEWS: Mention the improvement.
Related to https://bugs.gnu.org/10877
This fixes test failures seen on Haiku.
* tests/ls/recursive.sh: Run 'ls' even if ulimit fails.
* tests/split/r-chunk.sh: Run 'split' even if ulimit fails.
* tests/sort/sort-merge-fdlimit.sh: Skip test if 'ulimit -n' cannot set
file descriptor limits.
Reported by Bruno Haible.
Recently we've not built our kill command by default,
so reduce test dependence on that.
* tests/sort/sort-compress-proc.sh: `kill -l [exit_status]` is well
supported, and is a POSIX requirement.
* tests/timeout/timeout.sh: There were actually no kill invocations
in this test at all.
* src/sort.c (main): Ignore SIGPIPE so we've more control over
how we handle for stdout and compression programs.
(sort_die): Handle EPIPE from stdout and mimic a standard SIGPIPE,
otherwise reverting to a standard exit(SORT_FAILURE);
* tests/sort/sort-compress-proc.sh: Add a test case.
* NEWS: Mention the bug fix.
* NEWS: Mention the improvement. Mention that 'sort' will continue
without compressing temporary files if the program specified by
--compress-program cannot be executed.
* doc/coreutils.texi (sort invocation): Document the behavior when the
program specified by --compress-program cannot be executed.
* src/sort.c: Include spawn.h.
(MAX_FORK_TRIES_COMPRESS, MAX_FORK_TRIES_DECOMPRESS): Remove definition.
(MAX_TRIES_COMPRESS, MAX_TRIES_DECOMPRESS): New definitions based on
MAX_FORK_TRIES_COMPRESS and MAX_FORK_TRIES_DECOMPRESS.
(async_safe_die): Remove function.
(posix_spawn_file_actions_move_fd): New function.
(pipe_fork): Remove function.
(pipe_child): New function based on pipe_fork. Return an error number
instead of a pid. Use posix_spawnp instead of calling fork and expecting
the caller to exec.
(maybe_create_temp): Call pipe_child instead of pipe_fork. Print a
warning to standard error if --compress-program cannot be executed and
the error is different than the previous call. Remove code for the child
process.
(open_temp): Remove code for the child process. Improve error message.
* tests/sort/sort-compress.sh: Add a test case for when the program
specified by --compress-program does not exist.
* cfg.mk (sc_standard_outputs): Add a grep command for source files.
* src/du.c (main): Use standard input instead of stdin, standard output
instead of stdout, and standard error instead of stderr in messages.
* src/nohup.c (main): Likewise.
* src/sort.c (main): Likewise.
* src/split.c (main): Likewise.
* src/stdbuf.c (main): Likewise.
* src/wc.c (main): Likewise.
* tests/du/files0-from.pl (@Tests): Adjust test case to new messages.
* tests/sort/sort-files0-from.pl: Likewise.
* tests/wc/wc-files0-from.pl: Likewise.
Problem reported by Cosima Neidahl <https://bugs.gnu.org/78985#13>.
* tests/sort/sort-float.sh: At top level, use C locale at first.
(dbl_minima_order): Assume C locale.
Use string comparison for the fractional parts.
2025-07-10 Paul Eggert <eggert@cs.ucla.edu>
tests: fix integer overflow in sort-float
Problem reported by Cosima Neidahl <https://bugs.gnu.org/78985>.
* tests/sort/sort-float.sh (dbl_minima_order):
Use expr instead of test, to avoid problems with integers
too large for the shell.
Problem reported by Cosima Neidahl <https://bugs.gnu.org/78985>.
* tests/sort/sort-float.sh (dbl_minima_order):
Use expr instead of test to compare fractions,
to avoid problems with integers too large for the shell.
* src/sort.c (begfield): Check pointer adjustment
to avoid Out-of-range pointer offset (CWE-823).
(limfield): Likewise.
* tests/sort/sort-field-limit.sh: Add a new test,
which triggers with ASAN or Valgrind.
* tests/local.mk: Reference the new test.
* NEWS: Mention bug fix introduced in v7.2 (2009).
Fixes https://bugs.gnu.org/78507
* tests/sort/sort-h-thousands-sep.sh: sv_SE defaults to UTF-8
on macOS 18, so avoid the test for multi-byte separators.
Fixes https://bugs.gnu.org/77509
* src/sort.c (key_warnings): Remove "note " from the start
of a usually informational message, as this simplifies translation.
* tests/sort/sort-debug-warn.sh": Adjust accordingly.
Fixes https://bugs.gnu.org/75763
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...
* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Likewise.
* tests/sample-test: Adjust to use the single most recent year.
* tests/df/problematic-chars.sh: Rely on gnulib setting
this to "none" where not usable.
* tests/misc/sleep.sh: Likewise.
* tests/printf/printf-mb.sh: Likewise.
* tests/printf/printf-quote.sh: Likewise.
* tests/sort/sort-debug-keys.sh: Likewise.
Update to latest gnulib with new copyright year.
Run "make update-copyright" and then...
* gnulib: Update included in this commit as copyright years
are the only change from the previous gnulib commit.
* tests/init.sh: Sync with gnulib to pick up copyright year.
* bootstrap: Manually update copyright year,
until we fully sync with gnulib at a later stage.
* tests/sample-test: Adjust to use the single most recent year.
* tests/misc/numfmt.pl: Determine the thousands grouping character
in use, rather than skipping locale tests when it's not a space.
For example fr_FR.UTF-8 uses "NARROW NO-BREAK SPACE" as the grouping
char on modern glibc systems at least.
* tests/sort/sort-h-thousands-sep.sh: Likewise.
Most of this just affects commentary and documentations. The only
significant behavior change is translating author names via
proper_name_lite rather than proper_name_utf8, or not translating
them at all. proper_name_lite is good enough for coreutils and
avoids the bloat that had coreutils not using Gnulib proper_name.
* bootstrap.conf (gnulib_modules): Use propername-lite instead
of propername.
(XGETTEXT_OPTIONS): Look for proper_name_lite instead of for
proper_name_utf8.
* cfg.mk (local-checks-to-skip): Remove
sc_proper_name_utf8_requires_ICONV, since we no longer use
proper_name_utf8.
(old_NEWS_hash): Update.
(sc_check-I18N-AUTHORS): Remove; no longer needed.
* tests/sort/sort-continue.sh: Use ulimit -n 7 not -n 6. On
Solaris 10 'sort' uses Gnulib mkostemp, which calls Gnulib
getrandom, which opens /dev/urandom to calculate the temp file's
name, which means 'sort' needs one more file descriptor to work.
* tests/sort/sort-debug-keys.sh: Decimal point was seen to be '.'
on fr_FR.UTF-8 on Alpine Linux 3.18, so add an extra guard
to ensure we've a ',' as the decimal point on this locale.
Fixes https://bugs.gnu.org/65310
* tests/check.mk (vc_exe_in_TESTS): Adjust regexp not to match the
"TESTS = $x-tests" line in each Test.pm-derived Makefile.am file.
* tests/sort/main: Remove this unused file.
* Transform all Makefile.am files so that when running "make check",
CU_TEST_NAME is set to the name of the test. This is so that when I
run valgrind-enabled (--log-file-qualifier=CU_TEST_NAME) "make check"
on the entire package it is more convenient to map a leak or error
found in a valgrind log file back to the offending test.
Use this command:
(echo tests/Makefile.am.in; find tests -name Makefile.am) \
|xargs perl -pi -e '/^(\s*)PATH=...VG_PATH_PREFIX/ and ' \
-e 'print $1,q|CU_TEST_NAME=`basename $(abs_srcdir)`,$$tst |,"\\\n"'
so that commands like "sort -k 18446744073709551616" no longer fail merely
because 18446744073709551616 doesn't fit in uintmax_t. The trick is that
these fields can all be treated as effectively infinity; their exact
values don't matter, since no internal buffer can be that long.
* src/join.c (string_to_join_field): Verify that SIZE_MAX <= ULONG_MAX
if the code assumes this. Silently truncate too-large values to SIZE_MAX,
as the remaining code will do the right thing in this case.
* src/sort.c (parse_field_count): Likewise.
* src/uniq.c (size_opt, main): Likewise.
* tests/join/Test.pm (bigfield): New test.
* tests/sort/Test.pm (bigfield): New test.
* tests/uniq/Test.pm (121): New test.
Signed-off-by: Jim Meyering <jim@meyering.net>
Remove from CVS, since ./bootstrap generates them automatically.
* .cvsignore: Add INSTALL, Makefile.in, *.cache, *.lineno, *.log.
Remove more-specific entries. This catches files like configure.lineno.
* man/.cvsignore: Add Makefile.in.
* src/.cvsignore: Add Makefile.in.
Remove .version, dir.c, install, mvdir, stamp-v, vdir.c, version.c.
For .cvsignore file under the tests directory:
Add Makefile.in. Sort entries if necessary. Remove *.I, *.E,
*.X, *.O, *-tests, build-script, mk-script if they're never
created in this directory.
* build-aux/.cvsignore: New file.
* doc/.cvsignore: Add Makefile.in, coreutils.html, coreutils.pdf,
coreutils.ps, coreutils.tps. Remove coreutils.cm (dunno what it
is, but the makefile doesn't mention it). Remove coreutils.info
as it is subsumed by coreutils.info*.
* lib/.cvsignore: Add Makefile.in, getdate.tab.h.
Remove stat.c, sysexit.h.
* m4/.cvsignore: Remove Makefile, dev-ino.m4, glibc2.m4, intdiv0.m4,
inttypes-h.m4, inttypes-pri.m4, jm-glibc-io.m4, lcmessage.m4,
lock.m4, printf-posix.m4, same-inode.m4, size_max.m4,
uintmax_t.m4, ulonglong.m4, visibility.m4, xsize.m4. Many of
these are put in automatically by ../bootstrap, but that's a
bootstrap bug that I plan to fix shortly.
* po/.cvsignore: More ../bootstrap-related fixes, plus remove old
cruft. Add *.po, LINGUAS, Makevars. Remove *.cat, *.msg,
cat-id-dbl.c, messages.mo, stamp-cat-id.
backed out due to updates provoked by the copyright changes.
* tests/Makefile.am.in (PATH): Prepend $(VG_PATH_PREFIX), so that
it propagates to the derived Makefile.am files.
($(srcdir)/Makefile.am): Mark generated .am files as read-only,
so we don't mistakenly edit them again.
* tests/cut/Makefile.am: Regenerate.
* tests/head/Makefile.am: Likewise.
* tests/join/Makefile.am: Likewise.
* tests/pr/Makefile.am: Likewise.
* tests/sort/Makefile.am: Likewise.
* tests/tac/Makefile.am: Likewise.
* tests/tail/Makefile.am: Likewise.
* tests/test/Makefile.am: Likewise.
* tests/tr/Makefile.am: Likewise.
* tests/uniq/Makefile.am: Likewise.
* tests/wc/Makefile.am: Likewise.
Pádraig Brady