make_ext.pl direct pm_to_blib handling causes unnecessary rebuilds

[p5pRT]

Migrated from rt.perl.org#126710 (status was 'resolved')

Searchable as RT126710$

[p5pRT]

From @tonycoz

For pure-perl modules with no Makefile.PL, make_ext.pl avoids the work in creating
standard Makefile.PL and processing it and instead calls ExtUtils​::Install​::pm_to_blib() directly.

Unfortunately it always "touches" the module's pm_to_blib, even if no files were copied.

Since perlmain.c depends on ext/ExtUtils-Miniperl/pm_to_blib that file is always rebuilt leading to other files being rebuilt.

Under normal usage, when ExtUtils​::Install​::pm_to_blib() is called from a Makefile, the target depends on the .pm files involved, so the pm_to_blib file is only touched when those the .pm files are updated, so it's only a problem with make_ext.pl.

make_ext.pl should check the .pm files are newer than pm_to_blib before touching pm_to_blib.

Tony

---

Summary of my perl5 (revision 5 version 23 subversion 6) configuration​:
  Derived from​: 96b6870
  Platform​:
  osname=linux, osvers=3.16.0-4-amd64, archname=x86_64-linux
  uname='linux mars 3.16.0-4-amd64 #1 smp debian 3.16.7-ckt11-1+deb8u5 (2015-10-09) x86_64 gnulinux '
  config_args='-des -Dusedevel'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -D_FORTIFY_SOURCE=2',
  optimize='-O2',
  cppflags='-fwrapv -fno-strict-aliasing -pipe -fstack-protector-strong -I/usr/local/include'
  ccversion='', gccversion='4.9.2', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -fstack-protector-strong -L/usr/local/lib'
  libpth=/usr/local/lib /usr/lib/gcc/x86_64-linux-gnu/4.9/include-fixed /usr/include/x86_64-linux-gnu /usr/lib /lib/x86_64-linux-gnu /lib/../lib /usr/lib/x86_64-linux-gnu /usr/lib/../lib /lib
  libs=-lpthread -lnsl -lgdbm -ldb -ldl -lm -lcrypt -lutil -lc -lgdbm_compat
  perllibs=-lpthread -lnsl -ldl -lm -lcrypt -lutil -lc
  libc=libc-2.19.so, so=so, useshrplib=false, libperl=libperl.a
  gnulibc_version='2.19'
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=so, d_dlsymun=undef, ccdlflags='-Wl,-E'
  cccdlflags='-fPIC', lddlflags='-shared -O2 -L/usr/local/lib -fstack-protector-strong'

Characteristics of this binary (from libperl)​:
  Compile-time options​: HAS_TIMES PERLIO_LAYERS PERL_COPY_ON_WRITE
  PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_PRESERVE_IVUV PERL_USE_DEVEL USE_64_BIT_ALL
  USE_64_BIT_INT USE_LARGE_FILES USE_LOCALE
  USE_LOCALE_COLLATE USE_LOCALE_CTYPE
  USE_LOCALE_NUMERIC USE_LOCALE_TIME USE_PERLIO
  USE_PERL_ATOF
  Locally applied patches​:
  uncommitted-changes
  Built under linux
  Compiled at Nov 23 2015 09​:50​:05
  %ENV​:
  PERLBREW_BASHRC_VERSION="0.43"
  PERLBREW_HOME="/home/tony/.perlbrew"
  PERLBREW_PATH="/home/tony/perl5/perlbrew/bin"
  PERLBREW_ROOT="/home/tony/perl5/perlbrew"
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.23.6/x86_64-linux
  /usr/local/lib/perl5/site_perl/5.23.6
  /usr/local/lib/perl5/5.23.6/x86_64-linux
  /usr/local/lib/perl5/5.23.6
  .

[p5pRT]

From @jkeenan

On Sun Nov 22 15​:35​:15 2015, tonyc wrote​:

For pure-perl modules with no Makefile.PL, make_ext.pl avoids the work
in creating
standard Makefile.PL and processing it and instead calls
ExtUtils​::Install​::pm_to_blib() directly.

Can you provide a list of those pure-perl modules (or a formula to derive such a list)?

Thank you very much.
Jim Keenan

Unfortunately it always "touches" the module's pm_to_blib, even if no
files were copied.

Since perlmain.c depends on ext/ExtUtils-Miniperl/pm_to_blib that file
is always rebuilt leading to other files being rebuilt.

Under normal usage, when ExtUtils​::Install​::pm_to_blib() is called
from a Makefile, the target depends on the .pm files involved, so the
pm_to_blib file is only touched when those the .pm files are updated,
so it's only a problem with make_ext.pl.

make_ext.pl should check the .pm files are newer than pm_to_blib
before touching pm_to_blib.

Tony

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @tonycoz

On Sun, Nov 22, 2015 at 03​:53​:19PM -0800, James E Keenan via RT wrote​:

On Sun Nov 22 15​:35​:15 2015, tonyc wrote​:

For pure-perl modules with no Makefile.PL, make_ext.pl avoids the work
in creating
standard Makefile.PL and processing it and instead calls
ExtUtils​::Install​::pm_to_blib() directly.

Can you provide a list of those pure-perl modules (or a formula to derive such a list)?

In a built perl tree run​:

  grep make_ext `find . -name pm_to_blib`

$ grep make_ext `find . -name pm_to_blib` | wc -l
75

So lots, but I'm not sure than any of the others cause rebuilds.

Tony

[p5pRT]

From @tonycoz

On Sun Nov 22 15​:35​:15 2015, tonyc wrote​:

Unfortunately it always "touches" the module's pm_to_blib, even if no
files were copied.

Since perlmain.c depends on ext/ExtUtils-Miniperl/pm_to_blib that file
is always rebuilt leading to other files being rebuilt.

Under normal usage, when ExtUtils​::Install​::pm_to_blib() is called
from a Makefile, the target depends on the .pm files involved, so the
pm_to_blib file is only touched when those the .pm files are updated,
so it's only a problem with make_ext.pl.

make_ext.pl should check the .pm files are newer than pm_to_blib
before touching pm_to_blib.

Something like the attached.

Tony

[p5pRT]

From @tonycoz

0001-perl-126710-only-touch-pm_to_blib-if-files-are-copie.patch

From 942c9d10af7307004b6be235a7d5967c1d2978c7 Mon Sep 17 00:00:00 2001
From: Tony Cook <tony@develop-help.com>
Date: Tue, 1 Dec 2015 16:10:55 +1100
Subject: [perl #126710] only touch pm_to_blib if files are copied

Add checks similar to what the Makefile would do: only copy the
files if the source file is newer than pm_to_blib
---
 make_ext.pl | 45 +++++++++++++++++++++++++++++++--------------
 1 file changed, 31 insertions(+), 14 deletions(-)

diff --git a/make_ext.pl b/make_ext.pl
index 0745049..787faba 100644
--- a/make_ext.pl
+++ b/make_ext.pl
@@ -708,20 +708,37 @@ sub just_pm_to_blib {
     }
     # This is running under miniperl, so no autodie
     if ($target eq 'all') {
-        local $ENV{PERL_INSTALL_QUIET} = 1;
-        require ExtUtils::Install;
-        ExtUtils::Install::pm_to_blib(\%pm, '../../lib/auto');
-        open my $fh, '>', $pm_to_blib
-            or die "Can't open '$pm_to_blib': $!";
-        print $fh "$0 has handled pm_to_blib directly\n";
-        close $fh
-            or die "Can't close '$pm_to_blib': $!";
-	if (IS_UNIX) {
-            # Fake the fallback cleanup
-            my $fallback
-                = join '', map {s!^\.\./\.\./!!; "rm -f $_\n"} sort values %pm;
-            foreach my $clean_target ('realclean', 'veryclean') {
-                fallback_cleanup($return_dir, $clean_target, $fallback);
+        my $need_update = 1;
+        if (-f $pm_to_blib) {
+            # avoid touching pm_to_blib unless there's something that
+            # needs updating, see #126710
+            $need_update = 0;
+            my $test_at = -M _;
+            while (my $from = each(%pm)) {
+                if (-M $from < $test_at) {
+                    ++$need_update;
+                    last;
+                }
+            }
+            keys %pm; # reset iterator
+        }
+
+        if ($need_update) {
+            local $ENV{PERL_INSTALL_QUIET} = 1;
+            require ExtUtils::Install;
+            ExtUtils::Install::pm_to_blib(\%pm, '../../lib/auto');
+            open my $fh, '>', $pm_to_blib
+                or die "Can't open '$pm_to_blib': $!";
+            print $fh "$0 has handled pm_to_blib directly\n";
+            close $fh
+                or die "Can't close '$pm_to_blib': $!";
+            if (IS_UNIX) {
+                # Fake the fallback cleanup
+                my $fallback
+                    = join '', map {s!^\.\./\.\./!!; "rm -f $_\n"} sort values %pm;
+                foreach my $clean_target ('realclean', 'veryclean') {
+                    fallback_cleanup($return_dir, $clean_target, $fallback);
+                }
             }
         }
     } else {
-- 
2.1.4

[p5pRT]

From @jkeenan

On Mon Nov 30 21​:11​:45 2015, tonyc wrote​:

On Sun Nov 22 15​:35​:15 2015, tonyc wrote​:

Unfortunately it always "touches" the module's pm_to_blib, even if no
files were copied.

Since perlmain.c depends on ext/ExtUtils-Miniperl/pm_to_blib that file
is always rebuilt leading to other files being rebuilt.

Under normal usage, when ExtUtils​::Install​::pm_to_blib() is called
from a Makefile, the target depends on the .pm files involved, so the
pm_to_blib file is only touched when those the .pm files are updated,
so it's only a problem with make_ext.pl.

make_ext.pl should check the .pm files are newer than pm_to_blib
before touching pm_to_blib.

Something like the attached.

Tony

I ran 'make' at commit 406d554 and transcribed the output.

I then created a branch, reconfigured, applied your patch, ran 'make' and again transcribed the output.

The differences were very small; see attachment. Should I have expected to see more differences?

Thank you very much.

--
James E Keenan (jkeenan@​cpan.org)

[p5pRT]

From @jkeenan

blead.126710.make.diff

--- blead.make.typescript	2015-12-05 09:13:12.864841463 -0500
+++ 126710.make.typescript	2015-12-05 09:13:20.052841254 -0500
@@ -1,4 +1,4 @@
-Script started on Sat 05 Dec 2015 09:03:24 AM EST
+Script started on Sat 05 Dec 2015 09:10:00 AM EST
 [�[01;31mperl�[00m] �[01;33m1 �[00m$ make
 echo @`sh  cflags "optimize='-O2'" perlmini.o`  -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
 @cc -c -DPERL_CORE -fwrapv -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -D_LARGEFILE_SOURCE -D_FILE_OFFSET_BITS=64 -std=c89 -O2 -Wall -Werror=declaration-after-statement -Wextra -Wc++-compat -Wwrite-strings -DPERL_IS_MINIPERL -DPERL_EXTERNAL_GLOB perlmini.c
@@ -335,14 +335,14 @@
 Generating a Unix-style Makefile
 Writing Makefile for Pod
 make[1]: Entering directory `/home/jkeenan/gitwork/perl/cpan/podlators'
-"../../miniperl" "-I../../lib" "-I../../lib" "-I../../lib" "-I../../lib" pod2text.PL pod2text
-Extracting pod2text (with variable substitutions)
-cp pod2text blib/script/pod2text
-/home/jkeenan/gitwork/perl/cpan/podlators/../../miniperl "-I../../lib" "-I../../lib" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/pod2text
 "../../miniperl" "-I../../lib" "-I../../lib" "-I../../lib" "-I../../lib" pod2man.PL pod2man
 Extracting pod2man (with variable substitutions)
 cp pod2man blib/script/pod2man
 /home/jkeenan/gitwork/perl/cpan/podlators/../../miniperl "-I../../lib" "-I../../lib" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/pod2man
+"../../miniperl" "-I../../lib" "-I../../lib" "-I../../lib" "-I../../lib" pod2text.PL pod2text
+Extracting pod2text (with variable substitutions)
+cp pod2text blib/script/pod2text
+/home/jkeenan/gitwork/perl/cpan/podlators/../../miniperl "-I../../lib" "-I../../lib" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/pod2text
 make[1]: Leaving directory `/home/jkeenan/gitwork/perl/cpan/podlators'
 ./miniperl -Ilib make_ext.pl cpan/Pod-Parser/pm_to_blib  MAKE="make" LIBPERL_A=libperl.a
 Generating a Unix-style Makefile
@@ -1147,10 +1147,10 @@
 	  
 chmod 755 ../../lib/auto/Encode/Encode.so
 /home/jkeenan/gitwork/perl/cpan/Encode/../../miniperl "-I../../lib" "-I../../lib" -MExtUtils::Command::MM -e 'cp_nonempty' -- Encode.bs ../../lib/auto/Encode/Encode.bs 644
-cp bin/enc2xs blib/script/enc2xs
-/home/jkeenan/gitwork/perl/cpan/Encode/../../miniperl "-I../../lib" "-I../../lib" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/enc2xs
 cp bin/encguess blib/script/encguess
 /home/jkeenan/gitwork/perl/cpan/Encode/../../miniperl "-I../../lib" "-I../../lib" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/encguess
+cp bin/enc2xs blib/script/enc2xs
+/home/jkeenan/gitwork/perl/cpan/Encode/../../miniperl "-I../../lib" "-I../../lib" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/enc2xs
 cp bin/piconv blib/script/piconv
 /home/jkeenan/gitwork/perl/cpan/Encode/../../miniperl "-I../../lib" "-I../../lib" -MExtUtils::MY -e 'MY->fixin(shift)' -- blib/script/piconv
 make[1]: Leaving directory `/home/jkeenan/gitwork/perl/cpan/Encode'
@@ -1870,5 +1870,5 @@
 	Everything is up to date. Type 'make test' to run test suite.
 [�[01;31mperl�[00m] �[01;33m2 �[00m$ exit
 
-Script done on Sat 05 Dec 2015 09:06:41 AM EST
+Script done on Sat 05 Dec 2015 09:12:44 AM EST
 

[p5pRT]

From @tonycoz

On Sat Dec 05 06​:24​:27 2015, jkeenan wrote​:

On Mon Nov 30 21​:11​:45 2015, tonyc wrote​:

Something like the attached.

I ran 'make' at commit 406d554 and
transcribed the output.

I then created a branch, reconfigured, applied your patch, ran 'make'
and again transcribed the output.

The differences were very small; see attachment. Should I have
expected to see more differences?

The differences would be in a second build of the same tree.

Without the change perlmain.c and its dependents are rebuilt each time.

Tony

[p5pRT]

From @cpansprout

If I run a simple ‘make’ in the perl build directory multiple times, it recompiles perlmain.c every time.

(This causes problems for me, since I often do ‘make && sudo make install’ *followed* by ‘make test’, which then fails since perlmain.o is owned by root and cannot be overwritten. I do things in this order so as to test CPAN modules against blead as soon as possible, while in the mean time running perl’s tests to make sure they pass in the configuration I am using.)

./perl -Ilib -V output​:

Summary of my perl5 (revision 5 version 25 subversion 2) configuration​:
  Snapshot of​: cbef69c
  Platform​:
  osname=darwin, osvers=12.5.0, archname=darwin-2level
  uname='darwin pint.local 12.5.0 darwin kernel version 12.5.0​: sun sep 29 13​:33​:47 pdt 2013; root​:xnu-2050.48.12~1release_x86_64 x86_64 '
  config_args='-DDEBUGGING -de -Dusedevel -Accflags=-DPERL_BOOL_AS_CHAR'
  hint=recommended, useposix=true, d_sigaction=define
  useithreads=undef, usemultiplicity=undef
  use64bitint=define, use64bitall=define, uselongdouble=undef
  usemymalloc=n, bincompat5005=undef
  Compiler​:
  cc='cc', ccflags ='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.8 -DPERL_BOOL_AS_CHAR -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include -DPERL_USE_SAFE_PUTENV',
  optimize='-O3 -g',
  cppflags='-fno-common -DPERL_DARWIN -mmacosx-version-min=10.8 -DPERL_BOOL_AS_CHAR -DDEBUGGING -fno-strict-aliasing -pipe -fstack-protector -I/usr/local/include'
  ccversion='', gccversion='4.2.1 Compatible Apple LLVM 4.2 (clang-425.0.27)', gccosandvers=''
  intsize=4, longsize=8, ptrsize=8, doublesize=8, byteorder=12345678, doublekind=3
  d_longlong=define, longlongsize=8, d_longdbl=define, longdblsize=16, longdblkind=3
  ivtype='long', ivsize=8, nvtype='double', nvsize=8, Off_t='off_t', lseeksize=8
  alignbytes=8, prototype=define
  Linker and Libraries​:
  ld='cc', ldflags =' -mmacosx-version-min=10.8 -fstack-protector -L/usr/local/lib'
  libpth=/usr/local/lib /usr/bin/../lib/clang/4.2/lib /usr/lib
  libs=-lpthread -ldbm -ldl -lm -lutil -lc
  perllibs=-lpthread -ldl -lm -lutil -lc
  libc=, so=dylib, useshrplib=false, libperl=libperl.a
  gnulibc_version=''
  Dynamic Linking​:
  dlsrc=dl_dlopen.xs, dlext=bundle, d_dlsymun=undef, ccdlflags=' '
  cccdlflags=' ', lddlflags=' -mmacosx-version-min=10.8 -bundle -undefined dynamic_lookup -L/usr/local/lib -fstack-protector'

Characteristics of this binary (from libperl)​:
  Compile-time options​: DEBUGGING HAS_TIMES PERLIO_LAYERS PERL_BOOL_AS_CHAR
  PERL_COPY_ON_WRITE PERL_DONT_CREATE_GVSV
  PERL_HASH_FUNC_ONE_AT_A_TIME_HARD PERL_MALLOC_WRAP
  PERL_OP_PARENT PERL_PRESERVE_IVUV PERL_USE_DEVEL
  PERL_USE_SAFE_PUTENV USE_64_BIT_ALL USE_64_BIT_INT
  USE_LARGE_FILES USE_LOCALE USE_LOCALE_COLLATE
  USE_LOCALE_CTYPE USE_LOCALE_NUMERIC USE_LOCALE_TIME
  USE_PERLIO USE_PERL_ATOF
  Built under darwin
  Compiled at May 26 2016 19​:45​:13
  @​INC​:
  lib
  /usr/local/lib/perl5/site_perl/5.25.2/darwin-2level
  /usr/local/lib/perl5/site_perl/5.25.2
  /usr/local/lib/perl5/5.25.2/darwin-2level
  /usr/local/lib/perl5/5.25.2
  /usr/local/lib/perl5/site_perl
  .

--

Father Chrysostomos

[p5pRT]

From @cpansprout

On Thu May 26 22​:11​:33 2016, sprout wrote​:

If I run a simple ‘make’ in the perl build directory multiple times,
it recompiles perlmain.c every time.

Bisect​:

$ ../perl.git/Porting/bisect.pl -- perl -e '$age = -M "perlmain.o"; system "make -j25"; $age == -M "perlmain.o" or die'
...
f6dd0e4 is the first bad commit
commit f6dd0e4
Author​: Nicholas Clark <nick@​ccl4.org>
Date​: Thu Feb 27 11​:04​:19 2014 +0100

  For simple extensions make_ext.pl can emulate the entire MakeMaker/make dance.

--

Father Chrysostomos

[p5pRT]

The RT System itself - Status changed from 'new' to 'open'

[p5pRT]

From @cpansprout

On Thu Jun 09 22​:45​:58 2016, sprout wrote​:

On Thu May 26 22​:11​:33 2016, sprout wrote​:

If I run a simple ‘make’ in the perl build directory multiple times,
it recompiles perlmain.c every time.

Bisect​:

$ ../perl.git/Porting/bisect.pl -- perl -e '$age = -M "perlmain.o";
system "make -j25"; $age == -M "perlmain.o" or die'
...
f6dd0e4 is the first bad commit
commit f6dd0e4
Author​: Nicholas Clark <nick@​ccl4.org>
Date​: Thu Feb 27 11​:04​:19 2014 +0100

For simple extensions make_ext.pl can emulate the entire
MakeMaker/make dance.

The problem is that make_ext.pl​:just_pm_to_blib touches pm_to_blib unconditionally even if the files have not changed.

perlmain.c depends on ext/ExtUtils-Miniperl/pm_to_blib.

--

Father Chrysostomos

[p5pRT]

From @tonycoz

On Sat, Jun 11, 2016 at 07​:32​:59PM -0700, Father Chrysostomos via RT wrote​:

On Thu Jun 09 22​:45​:58 2016, sprout wrote​:

On Thu May 26 22​:11​:33 2016, sprout wrote​:

If I run a simple ‘make’ in the perl build directory multiple times,
it recompiles perlmain.c every time.

Bisect​:

$ ../perl.git/Porting/bisect.pl -- perl -e '$age = -M "perlmain.o";
system "make -j25"; $age == -M "perlmain.o" or die'
...
f6dd0e4 is the first bad commit
commit f6dd0e4
Author​: Nicholas Clark <nick@​ccl4.org>
Date​: Thu Feb 27 11​:04​:19 2014 +0100

For simple extensions make_ext.pl can emulate the entire
MakeMaker/make dance.

The problem is that make_ext.pl​:just_pm_to_blib touches pm_to_blib unconditionally even if the files have not changed.

perlmain.c depends on ext/ExtUtils-Miniperl/pm_to_blib.

Does the patch in #126710 fix this for you?

Tony

[p5pRT]

From @cpansprout

On Sat Jun 11 20​:46​:17 2016, tonyc wrote​:

Does the patch in #126710 fix this for you?

Yes, it does. (I see this ticket is a duplicate. I’ll merge them.)

Another approach would be to allow EUMM to handle ExtUtils​::Miniperl. I can’t say which is better. What is our goal here? To keep make_ext.pl as simple as possible, or to make things more ‘correct’ from the point of view of proper dependency management.

Nevertheless, I would like to see this fixed in blead soon, because I keep having to chown after sudo make install.

--

Father Chrysostomos