make -j6 minitest failing

[hvds]

This failed three times in a row on blead@8fb00d81c6. The specific directory was different each time, but the general pattern was the same - shortly before the first "Permission denied" error it tries to make miniperl a second time for no obvious reason.

This was my default debug build: ./Configure -des -Dcc=gcc -Dprefix=/opt/scratch -Doptimize='-g -O6' -DDEBUGGING -Dusedevel -Uversiononly. make -j6 minitest_prep did not fail, which is all the wierder (maybe I just got lucky). make minitest also did not fail, which is less surprising.

Any clues how to debug this?

make output (click to expand)

gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings pad.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings regcomp.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings util.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings dump.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings mg.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings reentr.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings mro_core.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings keywords.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings builtin.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings hv.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings av.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings run.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings pp_hot.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings sv.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings pp.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings scope.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings pp_ctl.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings pp_sys.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings doop.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings doio.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings regexec.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings utf8.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings taint.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings deb.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings globals.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings perlio.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings numeric.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings mathoms.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings locale.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings pp_pack.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings pp_sort.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings caretx.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings dquote.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings time64.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings miniperlmain.c
echo @`sh  cflags "optimize='-g -O6'" opmini.o`  -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB opmini.c
@gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB opmini.c
make[1]: Entering directory '/src/perl/git'
echo @`sh  cflags "optimize='-g -O6'" perlmini.o`  -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
@gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
echo @`sh  cflags "optimize='-g -O6'" universalmini.o`  -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB universalmini.c
@gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB universalmini.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings gv.c
echo @`sh  cflags "optimize='-g -O6'" perlmini.o`  -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
@gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings toke.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings perly.c
echo @`sh  cflags "optimize='-g -O6'" universalmini.o`  -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB universalmini.c
@gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB universalmini.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings gv.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings toke.c
gcc -c -DPERL_CORE -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c99 -g -O6 -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings perly.c
gcc -fstack-protector-strong -L/usr/local/lib -o miniperl \
    opmini.o perlmini.o universalmini.o  gv.o toke.o perly.o pad.o regcomp.o dump.o util.o mg.o reentr.o mro_core.o keywords.o builtin.o hv.o av.o run.o pp_hot.o sv.o pp.o scope.o pp_ctl.o pp_sys.o doop.o doio.o regexec.o utf8.o taint.o deb.o globals.o perlio.o numeric.o mathoms.o locale.o pp_pack.o pp_sort.o caretx.o dquote.o time64.o  miniperlmain.o  -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
./miniperl -w -Ilib -Idist/Exporter/lib -MExporter -e '<?>' || sh -c 'echo >&2 Failed to build miniperl.  Please run make minitest; exit 1'
./miniperl -Ilib -f write_buildcustomize.pl
./miniperl -Ilib make_patchnum.pl
Updating 'git_version.h' and 'lib/Config_git.pl'
./miniperl -Ilib configpm
written lib/Config.pod
updated lib/Config.pm
updated lib/Config_heavy.pl
./miniperl -Ilib make_ext.pl cpan/Archive-Tar/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl dist/Attribute-Handlers/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/AutoLoader/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/CPAN/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl cpan/CPAN-Meta/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Attribute-Handlers directly
./miniperl -Ilib make_ext.pl cpan/CPAN-Meta-Requirements/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/AutoLoader directly
./miniperl -Ilib make_ext.pl cpan/CPAN-Meta-YAML/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/CPAN-Meta-YAML directly
./miniperl -Ilib make_ext.pl cpan/Config-Perl-V/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/Config-Perl-V directly
./miniperl -Ilib make_ext.pl dist/Devel-SelfStubber/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Carp directly
./miniperl -Ilib make_ext.pl cpan/Digest/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Generating a Unix-style Makefile
Writing Makefile for CPAN
Generating a Unix-style Makefile
Writing Makefile for Archive::Tar
make[2]: Entering directory '/src/perl/git/cpan/CPAN'
Generating a Unix-style Makefile
Writing Makefile for CPAN::Meta
make[2]: Entering directory '/src/perl/git/cpan/Archive-Tar'
make[2]: Entering directory '/src/perl/git/cpan/CPAN-Meta'
Running pm_to_blib for dist/Devel-SelfStubber directly
Running pm_to_blib for cpan/Digest directly
./miniperl -Ilib make_ext.pl dist/Dumpvalue/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl dist/Env/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
make[2]: Leaving directory '/src/perl/git/cpan/CPAN-Meta'
make[2]: Leaving directory '/src/perl/git/cpan/Archive-Tar'
./miniperl -Ilib make_ext.pl ext/Errno/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
./miniperl -Ilib make_ext.pl dist/Exporter/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Dumpvalue directly
./miniperl -Ilib make_ext.pl dist/ExtUtils-CBuilder/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Env directly
./miniperl -Ilib make_ext.pl cpan/ExtUtils-Constant/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
make[2]: Leaving directory '/src/perl/git/cpan/CPAN'
./miniperl -Ilib make_ext.pl cpan/ExtUtils-Install/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/Exporter directly
./miniperl -Ilib make_ext.pl cpan/ExtUtils-MakeMaker/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for dist/ExtUtils-CBuilder directly
./miniperl -Ilib make_ext.pl cpan/ExtUtils-Manifest/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/ExtUtils-Constant directly
./miniperl -Ilib make_ext.pl ext/ExtUtils-Miniperl/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/ExtUtils-Install directly
./miniperl -Ilib make_ext.pl cpan/ExtUtils-PL2Bat/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/ExtUtils-Manifest directly
Running pm_to_blib for ext/ExtUtils-Miniperl directly
./miniperl -Ilib make_ext.pl dist/ExtUtils-ParseXS/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Running pm_to_blib for cpan/ExtUtils-PL2Bat directly
./miniperl -Ilib make_ext.pl cpan/File-Fetch/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Generating a Unix-style Makefile
Writing Makefile for Errno
make[2]: Entering directory '/src/perl/git/ext/Errno'
Running pm_to_blib for dist/ExtUtils-ParseXS directly
Running pm_to_blib for cpan/File-Fetch directly
./miniperl -Ilib make_ext.pl ext/File-Find/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Generating a Unix-style Makefile
Writing Makefile for ExtUtils::MakeMaker
make[2]: Entering directory '/src/perl/git/cpan/ExtUtils-MakeMaker'
Generating a Unix-style Makefile
Writing Makefile for ExtUtils::Manifest
make[2]: Entering directory '/src/perl/git/cpan/ExtUtils-Manifest'
"../../miniperl" "-I../../lib" Errno_pm.PL Errno.pm
Running pm_to_blib for ext/File-Find directly
./miniperl -Ilib make_ext.pl cpan/File-Path/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
Generating a Unix-style Makefile
Writing Makefile for ExtUtils::ParseXS
make[2]: Entering directory '/src/perl/git/dist/ExtUtils-ParseXS'
gcc -fstack-protector-strong -L/usr/local/lib -o miniperl \
    opmini.o perlmini.o universalmini.o  gv.o toke.o perly.o pad.o regcomp.o dump.o util.o mg.o reentr.o mro_core.o keywords.o builtin.o hv.o av.o run.o pp_hot.o sv.o pp.o scope.o pp_ctl.o pp_sys.o doop.o doio.o regexec.o utf8.o taint.o deb.o globals.o perlio.o numeric.o mathoms.o locale.o pp_pack.o pp_sort.o caretx.o dquote.o time64.o  miniperlmain.o  -lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
Running pm_to_blib for cpan/File-Path directly
/bin/sh: 1: /src/perl/git/cpan/ExtUtils-MakeMaker/../../miniperl: Permission denied
Makefile:865: recipe for target 'pm_to_blib' failed
make[2]: *** [pm_to_blib] Error 126
make[2]: Leaving directory '/src/perl/git/cpan/ExtUtils-MakeMaker'
/bin/sh: 1: /src/perl/git/dist/ExtUtils-ParseXS/../../miniperl: Permission denied
Makefile:374: recipe for target 'blib/script/.exists' failed
make[2]: *** [blib/script/.exists] Error 126
make[2]: *** Waiting for unfinished jobs....
make[2]: Entering directory '/src/perl/git/cpan/ExtUtils-MakeMaker'
/bin/sh: 1: /src/perl/git/cpan/ExtUtils-Manifest/../../miniperl: Permission denied
Makefile:818: recipe for target 'pm_to_blib' failed
make[2]: *** [pm_to_blib] Error 126
make[2]: *** Waiting for unfinished jobs....
./miniperl -Ilib make_ext.pl cpan/File-Temp/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
make[2]: Leaving directory '/src/perl/git/dist/ExtUtils-ParseXS'
/bin/sh: 1: ./miniperl: Permission denied
makefile:594: recipe for target 'cpan/File-Temp/pm_to_blib' failed
make[1]: *** [cpan/File-Temp/pm_to_blib] Error 126
make[1]: *** Waiting for unfinished jobs....
make[2]: Entering directory '/src/perl/git/dist/ExtUtils-ParseXS'
make[2]: Leaving directory '/src/perl/git/cpan/ExtUtils-Manifest'
/bin/sh: 1: /src/perl/git/dist/ExtUtils-ParseXS/../../miniperl: Permission denied
Makefile:374: recipe for target 'blib/script/.exists' failed
make[2]: *** [blib/script/.exists] Error 126
make[2]: Leaving directory '/src/perl/git/dist/ExtUtils-ParseXS'
Unsuccessful make(dist/ExtUtils-ParseXS): code=512 at make_ext.pl line 584.
/bin/sh: 1: /src/perl/git/cpan/ExtUtils-MakeMaker/../../miniperl: Permission denied
Makefile:865: recipe for target 'pm_to_blib' failed
make[2]: *** [pm_to_blib] Error 126
make[2]: Leaving directory '/src/perl/git/cpan/ExtUtils-MakeMaker'
Unsuccessful make(cpan/ExtUtils-MakeMaker): code=512 at make_ext.pl line 584.
makefile:594: recipe for target 'dist/ExtUtils-ParseXS/pm_to_blib' failed
make[1]: *** [dist/ExtUtils-ParseXS/pm_to_blib] Error 25
makefile:594: recipe for target 'cpan/ExtUtils-MakeMaker/pm_to_blib' failed
make[1]: *** [cpan/ExtUtils-MakeMaker/pm_to_blib] Error 25
make[2]: Entering directory '/src/perl/git/cpan/ExtUtils-Manifest'
/bin/sh: 1: /src/perl/git/cpan/ExtUtils-Manifest/../../miniperl: Permission denied
Makefile:818: recipe for target 'pm_to_blib' failed
make[2]: *** [pm_to_blib] Error 126
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/src/perl/git/cpan/ExtUtils-Manifest'
Unsuccessful make(cpan/ExtUtils-Manifest): code=512 at make_ext.pl line 584.
makefile:594: recipe for target 'cpan/ExtUtils-Manifest/pm_to_blib' failed
make[1]: *** [cpan/ExtUtils-Manifest/pm_to_blib] Error 25
/bin/sh: 1: /src/perl/git/ext/Errno/../../miniperl: Permission denied
Makefile:826: recipe for target 'pm_to_blib' failed
make[2]: *** [pm_to_blib] Error 126
make[2]: Leaving directory '/src/perl/git/ext/Errno'
make[2]: Entering directory '/src/perl/git/ext/Errno'
/bin/sh: 1: /src/perl/git/ext/Errno/../../miniperl: Permission denied
Makefile:826: recipe for target 'pm_to_blib' failed
make[2]: *** [pm_to_blib] Error 126
make[2]: *** Waiting for unfinished jobs....
make[2]: Leaving directory '/src/perl/git/ext/Errno'
Unsuccessful make(ext/Errno): code=512 at make_ext.pl line 584.
makefile:594: recipe for target 'ext/Errno/pm_to_blib' failed
make[1]: *** [ext/Errno/pm_to_blib] Error 25
make[1]: Leaving directory '/src/perl/git'
makefile:838: recipe for target 'minitest_prep' failed
make: [minitest_prep] Error 2 (ignored)
 
You may see some irrelevant test failures if you have been unable
to build lib/Config.pm, or the Unicode data files.
 
cd t && (rm -f perl; /bin/ln -s ../miniperl perl)
./miniperl -w -Ilib -Idist/Exporter/lib -MExporter -e '<?>' || sh -c 'echo >&2 Failed to build miniperl.  Please run make minitest; exit 1'
./miniperl -Ilib -f write_buildcustomize.pl
cd t &&   ./perl -Ilib -I. TEST base/*.t comp/*.t cmd/*.t run/*.t io/*.t re/*.t opbasic/*.t op/*.t uni/*.t perf/*.t </dev/tty
t/base/cond .................... ok
...

[jkeenan]

This failed three times in a row on blead@8fb00d81c6. The specific directory was different each time, but the general pattern was the same - shortly before the first "Permission denied" error it tries to make miniperl a second time for no obvious reason.

This was my default debug build: ./Configure -des -Dcc=gcc -Dprefix=/opt/scratch -Doptimize='-g -O6' -DDEBUGGING -Dusedevel -Uversiononly. make -j6 minitest_prep did not fail, which is all the wierder (maybe I just got lucky). make minitest also did not fail, which is less surprising.

If you use either a different version of gcc or a different C compiler (g++, clang), can you reproduce the failures?

[jkeenan]

There does indeed appear to be a compiler-specific aspect to this problem.

On FreeBSD-12, I configured with a somewhat stripped-down version of your configuration invocation, using first gcc, then clang (version 10) of each. While I got through make minitest_prep with gcc, I had failures with make minitest before getting to the actual tests.

$ ./Configure -des -Dcc=gcc -Dusedevel -Doptimize='-g -O6' -DDEBUGGING make -j6 minitest
...

First, some build-time warnings:

148 --- minitest_prep ---
149 op.c:1895:9: warning: 'keysv' may be used uninitialized in this function [-Wmaybe-uninitialized]
150  1895 |         Perl_warner(aTHX_ packWARN(WARN_SYNTAX), msg,
151       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
152  1896 |                 SVfARG(name), lbrack, SVfARG(keysv), rbrack,
153       |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
154  1897 |                 SVfARG(name), lbrack, SVfARG(keysv), rbrack);
155       |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
156 --- opmini.o ---
157 op.c:1895:9: warning: 'keysv' may be used uninitialized in this function [-Wmaybe-uninitialized]
158  1895 |         Perl_warner(aTHX_ packWARN(WARN_SYNTAX), msg,
159       |         ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
160  1896 |                 SVfARG(name), lbrack, SVfARG(keysv), rbrack,
161       |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
162  1897 |                 SVfARG(name), lbrack, SVfARG(keysv), rbrack);
163       |                 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
...

Then:

174 --- minitest_prep ---
175 ./miniperl -w -Ilib -Idist/Exporter/lib -MExporter -e '<?>' || sh -c 'echo >&2 Failed to build miniperl.  Please run make minitest; exit 1'
176 sh: ./miniperl: Text file busy
177 Failed to build miniperl. Please run make minitest
178 *** [lib/buildcustomize.pl] Error code 1
179 
180 make[1]: stopped in /usr/home/jkeenan/gitwork/perl
181 1 error
182 
183 make[1]: stopped in /usr/home/jkeenan/gitwork/perl
184 
185 You may see some irrelevant test failures if you have been unable
186 to build lib/Config.pm, or the Unicode data files.
187 
188 cd t && (rm -f perl; /bin/ln -s ../miniperl perl)
189 --- lib/buildcustomize.pl ---
190 ./miniperl -w -Ilib -Idist/Exporter/lib -MExporter -e '<?>' || sh -c 'echo >&2 Failed to build miniperl.  Please run make minitest; exit 1'
191 ./miniperl -Ilib -f write_buildcustomize.pl

With clang10, I get many instances of this build-time warning:

$ ./Configure -des -Dcc=clang -Dusedevel -Doptimize='-g -O6' -DDEBUGGING make -j6 minitest
...
$ make -j6 minitest
...
warning: optimization level '-O6' is not supported; using '-O3' instead

However, I get through to the start of make minitest, which then PASSes.

I don't often run make minitest, and I rarely run with -Doptimize='-g -O6' or with -DDEBUGGING, so I don't know what to expect "normally."

[jkeenan]

With gcc, with -DDEBUGGING but without optimization:

$ ./Configure -des -Dcc=gcc -Dusedevel -DDEBUGGING
...
$ make -j6 minitest
...
gcc -c -DPERL_CORE -DHAS_FPSETMASK -DHAS_FLOATINGPOINT_H -fwrapv -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_FORTIFY_SOURCE=2 -std=c99 -O2 -g -Wall -Werror=pointer-arith -Werror=vla -Wextra -Wno-long-long -Wno-declaration-after-statement -Wc++-compat -Wwrite-strings dump.c
--- minitest_prep ---
"dd834db38c.freebsd.gcc.debugging.minitest.output.txt.gz" [noeol] 27L, 8201B            1,1           Top
/usr/local/bin/ld: globals.o:(.data+0x7f8): undefined reference to `Perl_ck_fun'
/usr/local/bin/ld: globals.o:(.data+0x800): undefined reference to `Perl_ck_fun'
/usr/local/bin/ld: globals.o:(.data+0x808): more undefined references to `Perl_ck_fun' follow
/usr/local/bin/ld: globals.o:(.data+0x830): undefined reference to `Perl_ck_select'
...
/usr/local/bin/ld: globals.o:(.data+0xd90): more undefined references to `Perl_ck_null' follow
/usr/local/bin/ld: perlio.o: in function `PerlIO_find_layer':
/usr/home/jkeenan/gitwork/perl/perlio.c:742: undefined reference to `Perl_load_module'
/usr/local/bin/ld: perlio.o: in function `Perl_boot_core_PerlIO':
/usr/home/jkeenan/gitwork/perl/perlio.c:1088: undefined reference to `Perl_newXS'
/usr/local/bin/ld: /usr/home/jkeenan/gitwork/perl/perlio.c:1089: undefined reference to `Perl_newXS'
/usr/local/bin/ld: mathoms.o: in function `Perl_ref':
/usr/home/jkeenan/gitwork/perl/mathoms.c:87: undefined reference to `Perl_doref'
/usr/local/bin/ld: mathoms.o: in function `Perl_newSUB':
/usr/home/jkeenan/gitwork/perl/mathoms.c:784: undefined reference to `Perl_newATTRSUB_x'
collect2: error: ld returned 1 exit status
*** [lib/buildcustomize.pl] Error code 1

make[1]: stopped in /usr/home/jkeenan/gitwork/perl
1 error

make[1]: stopped in /usr/home/jkeenan/gitwork/perl

$ zgrep -c 'undefined reference' dd834db38c.freebsd.gcc.debugging.minitest.output.txt.gz
875

[jkeenan]

With gcc but without debugging or optimization, the output is similar to with debugging but without optimization:

$ ./Configure -des -Dcc=gcc -Dusedevel
...
$ make -j6 minitest

[jkeenan]

Compiling with g++ version 10 instead of gcc: make minitest ran smoothly and PASSed.

[jkeenan]

gcc12 failed similarly to gcc10 but with 908 undefined reference warnings.

$ sh ./Configure -des -Dusedevel -Dcc=gcc12

$ make -j6 minitest
...
/usr/local/bin/ld: mathoms.o: in function `Perl_ref':
mathoms.c:(.text+0x6): undefined reference to `Perl_doref'
/usr/local/bin/ld: mathoms.o: in function `Perl_newSUB':
mathoms.c:(.text+0xa89): undefined reference to `Perl_newATTRSUB_x'
collect2: error: ld returned 1 exit status
*** [lib/buildcustomize.pl] Error code 1

make[1]: stopped in /usr/home/jkeenan/gitwork/perl
1 error

[jkeenan]

gcc12 failed similarly to gcc10 but with 908 undefined reference warnings.

$ sh ./Configure -des -Dusedevel -Dcc=gcc12

$ make -j6 minitest
...
/usr/local/bin/ld: mathoms.o: in function `Perl_ref':
mathoms.c:(.text+0x6): undefined reference to `Perl_doref'
/usr/local/bin/ld: mathoms.o: in function `Perl_newSUB':
mathoms.c:(.text+0xa89): undefined reference to `Perl_newATTRSUB_x'
collect2: error: ld returned 1 exit status
*** [lib/buildcustomize.pl] Error code 1

make[1]: stopped in /usr/home/jkeenan/gitwork/perl
1 error

However, notwithstanding the failure in make minitest above, when I then said make -j6 test_prep, that target completed without incident and make -j6 test_harness PASSed.

[jkeenan]

There appears to be a "test jobs" aspect to this problem in addition to the C-compiler problem.

With the following configuration on the same (FreeBSD-12) machine,

sh ./Configure -des -Dusedevel -Dcc=gcc12 && make minitest

... if I then run make minitest without any -j6 setting, the program reaches the test suite, which then PASSes.

...
Updating 'mktables.lst'
`lib/unicore/TestProp.pl' is up to date.
`lib/unicore/CombiningClass.pl' is up to date.
`lib/unicore/Name.pl' is up to date.
`lib/unicore/UCD.pl' is up to date.
`lib/unicore/Name.pm' is up to date.
`lib/unicore/mktables.lst' is up to date.

You may see some irrelevant test failures if you have been unable
to build lib/Config.pm, or the Unicode data files.

cd t && (rm -f perl; /bin/ln -s ../miniperl perl)
cd t &&   ./perl -Ilib -I. TEST base/*.t comp/*.t cmd/*.t run/*.t io/*.t re/*.t opbasic/*.t op/*.t uni/*.t perf/*.t </dev/tty
t/base/cond .................... ok
t/base/if ...................... ok
...
t/perf/taint ................... skipped
All tests successful.
Elapsed: 141 sec
u=1.78  s=0.38  cu=57.05  cs=8.71  scripts=361  tests=462237

[jkeenan]

This problem emerged somewhere in the 5.35 development cycle. I compiled at the v5.36.0 tag with gcc12 (no debugging or optimization) and then called make -j6 minitest and got failures similar to that already reported.

However, I'm currently compiling and building at the v5.34.1 tag and make -j6 minitest has completed successfully.

[jkeenan]

This problem emerged somewhere in the 5.35 development cycle. I compiled at the v5.36.0 tag with gcc12 (no debugging or optimization) and then called make -j6 minitest and got failures similar to that already reported.

However, I'm currently compiling and building at the v5.34.1 tag and make -j6 minitest has completed successfully.

@hvds, in principle this should be bisectable, but I haven't been able to figure out the syntax for an invocation of Porting/bisect.pl. Any ideas?

[hvds]

@hvds, in principle this should be bisectable, but I haven't been able to figure out the syntax for an invocation of Porting/bisect.pl. Any ideas?

I'll be busy with family stuff for a day or two, but I'll try a manual bisect when I have time - I don't have a clear failure mode for bisect.pl to detect.

I do recall that there were recent changes to the build process that I thought risked impacting parallel builds and appeared to gave me occasional failures on make miniperl or make minitest_prep, but I didn't report them since I couldn't demonstrate a reproducible failure. I have a nasty feeling that after bisect I'll be thinking "oh that one, yeah, that was a dodgy one, I should have said something".

[hvds]

On FreeBSD-12, I configured with a somewhat stripped-down version of your configuration invocation, using first gcc, then clang (version 10) of each. While I got through make minitest_prep with gcc, I had failures with make minitest before getting to the actual tests.

$ ./Configure -des -Dcc=gcc -Dusedevel -Doptimize='-g -O6' -DDEBUGGING make -j6 minitest
...

First, some build-time warnings:

148 --- minitest_prep ---
149 op.c:1895:9: warning: 'keysv' may be used uninitialized in this function [-Wmaybe-uninitialized]

This is unrelated, it deserves a separate issue.

[hvds]

There does indeed appear to be a compiler-specific aspect to this problem.

On FreeBSD-12, I configured with a somewhat stripped-down version of your configuration invocation, using first gcc, then clang (version 10) of each. [...]
Then:

174 --- minitest_prep ---
175 ./miniperl -w -Ilib -Idist/Exporter/lib -MExporter -e '<?>' || sh -c 'echo >&2 Failed to build miniperl.  Please run make minitest; exit 1'
176 sh: ./miniperl: Text file busy

This is potentially a symptom of the same underlying problem.

With clang10, I get many instances of this build-time warning:

$ ./Configure -des -Dcc=clang -Dusedevel -Doptimize='-g -O6' -DDEBUGGING make -j6 minitest
...
$ make -j6 minitest
...
warning: optimization level '-O6' is not supported; using '-O3' instead

This is irrelevant, feel free to invoke with -O3 instead.