Opened 4 years ago
Last modified 3 years ago
#62621 new defect
mpbb takes too long to install dependencies
Reported by: | ryandesign (Ryan Carsten Schmidt) | Owned by: | admin@… |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | buildbot/mpbb | Version: | |
Keywords: | Cc: | mascguy (Christopher Nielsen) | |
Port: |
Description
It's dumb that mpbb takes 80 minutes to install the dependencies of a port that takes only a minute to build.
https://build.macports.org/builders/ports-11_x86_64-builder/builds/25468
Yes, there are 280 dependencies. That's still 17 seconds per dependency, which is too much.
It's also dumb that the port has that many dependencies.
Change History (16)
comment:1 Changed 4 years ago by jmroot (Joshua Root)
comment:2 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)
In c8db5a3f36610a13f3c99915e32a6fa853982be9/mpbb (master):
comment:3 Changed 4 years ago by mascguy (Christopher Nielsen)
Cc: | mascguy added |
---|
comment:4 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)
Cc: | mascguy removed |
---|
Here we have a pull request updating auto-multiple-choice-devel, a port with 554 dependencies. It has taken 4 hours so far to attempt to install all dependencies. The Azure Pipelines build terminated after ~3.5 hours but its install-dependencies.log is already 1GB in size and contains 9669 occurrences of "activate phase started at" which is wildly excessive. There should be 554 occurrences, one for each port, but for each of the 554 we appear to be activating and deactivating that port and its dependencies, so many ports keep getting activated and deactivated over and over. Why? Can we stop doing that? This wears out build machine SSDs (all of the buildbot SSDs bought in 2016 died in 4-5 years) and wastes time.
$ egrep '^(----> Installing dependency|---> Activating|---> Deactivating)' install-dependencies.log | head -n 50 ----> Installing dependency (1 of 554) 'gperf' with variants '' (requesting '') ---> Activating gperf @3.1_0 ---> Deactivating gperf @3.1_0 ----> Installing dependency (2 of 554) 'libiconv' with variants '' (requesting '') ---> Activating libiconv @1.16_1 ---> Deactivating libiconv @1.16_1 ----> Installing dependency (3 of 554) 'ncurses' with variants '' (requesting '') ---> Activating ncurses @6.2_1 ---> Deactivating ncurses @6.2_1 ----> Installing dependency (4 of 554) 'gettext' with variants '' (requesting '') ---> Activating libiconv @1.16_1 ---> Activating ncurses @6.2_1 ---> Activating gettext @0.19.8.1_2 ---> Deactivating gettext @0.19.8.1_2 ---> Deactivating libiconv @1.16_1 ---> Deactivating ncurses @6.2_1 ----> Installing dependency (5 of 554) 'm4' with variants '' (requesting '') ---> Activating m4 @1.4.18_2 ---> Deactivating m4 @1.4.18_2 ----> Installing dependency (6 of 554) 'xz' with variants '' (requesting '') ---> Activating libiconv @1.16_1 ---> Activating ncurses @6.2_1 ---> Activating gettext @0.19.8.1_2 ---> Activating xz @5.2.5_0 ---> Deactivating xz @5.2.5_0 ---> Deactivating gettext @0.19.8.1_2 ---> Deactivating libiconv @1.16_1 ---> Deactivating ncurses @6.2_1 ----> Installing dependency (7 of 554) 'autoconf' with variants '' (requesting '') ---> Activating libiconv @1.16_1 ---> Activating ncurses @6.2_1 ---> Activating gettext @0.19.8.1_2 ---> Activating m4 @1.4.18_2 ---> Activating autoconf @2.71_1 ---> Deactivating autoconf @2.71_1 ---> Deactivating gettext @0.19.8.1_2 ---> Deactivating libiconv @1.16_1 ---> Deactivating m4 @1.4.18_2 ---> Deactivating ncurses @6.2_1 ----> Installing dependency (8 of 554) 'automake' with variants '' (requesting '') ---> Activating automake @1.16.3_0 ---> Deactivating automake @1.16.3_0 ----> Installing dependency (9 of 554) 'libffi' with variants '' (requesting '') ---> Activating libffi @3.3_1 ---> Deactivating libffi @3.3_1 ----> Installing dependency (10 of 554) 'libtool' with variants '' (requesting '') ---> Activating libtool @2.4.6_11 ---> Deactivating libtool @2.4.6_11 ----> Installing dependency (11 of 554) 'bzip2' with variants '' (requesting '') ---> Activating bzip2 @1.0.8_0
comment:5 follow-ups: 6 8 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)
These are two somewhat different cases. On the buildbot we keep all ports installed, and mpbb-install-dependencies prints "Already installed, nothing to do" and does nothing more for each direct dependency that's already installed. But on the CI build machines no ports are installed to begin with so it goes through the whole process of fetching the archive, installing and activating it (and its dependencies), then deactivating everything, for each direct dependency.
I expect the reason why it deactivates everything between dependencies is so that if any dependency was not already built, then that dependency will build from source with a clean slate, without any other random ports activated that might affect its build. However since #60935 was implemented, that should never be the case anymore. Packages for all dependencies should already exist by the time any build is attempted. So maybe we can streamline mpbb-install-dependencies so that if archive fetch fails, the entire build fails.
That might be too simple though. CI build machines sometimes lose access to the private packages server; in that case, it would be better to take extra time to build those nondistributable dependencies again in the PR rather than fail the PR. We could move "${option_prefix}/bin/port" -fp deactivate active
so that it is only run right before we install a dependency from source, rather than right after installing any dependency. We should be able to determine whether fetching the archive succeeded to decide whether or not to then deactivate active. And if any dependency was built from source, we might have to deactivate again at the end of installing all dependencies to ensure none of the dependencies' build dependencies are present for the main port build.
It would be even better if we could deactivate just the ports that should not be active for the next build, rather than deactivate everything. For example, looking at the dependency list from auto-multiple-choice-devel above, suppose we were able to install and activate gperf, libiconv, and ncurses from binaries, and then the gettext package couldn't be fetched and needed to be built from source. Rather than deactivating gperf, libiconv, and ncurses, and then building gettext from source, which would reactivate libiconv and ncurses again, it would be better if we could just deactivate gperf. But I don't think we have a way to do that yet.
comment:6 follow-up: 7 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)
Replying to ryandesign:
CI build machines sometimes lose access to the private packages server
There might also be a chance that one of the port's distributable dependencies has been updated so recently that its package has not yet made it to the public packages server.
It would be even better if we could deactivate just the ports that should not be active for the next build [...] But I don't think we have a way to do that yet.
or would that be as simple as "${option_prefix}/bin/port" -fp deactivate active and not depof:${depname}
?
comment:7 Changed 4 years ago by ryandesign (Ryan Carsten Schmidt)
Replying to ryandesign:
Replying to ryandesign:
It would be even better if we could deactivate just the ports that should not be active for the next build [...] But I don't think we have a way to do that yet.
or would that be as simple as
"${option_prefix}/bin/port" -fp deactivate active and not depof:${depname}
?
Hmm, no because it needs to be recursive. And it's not "${option_prefix}/bin/port" -fp deactivate active and not rdepof:${depname}
either because it would leave all ports' build dependencies active.
comment:8 follow-up: 15 Changed 4 years ago by jmroot (Joshua Root)
Replying to ryandesign:
Packages for all dependencies should already exist by the time any build is attempted. So maybe we can streamline mpbb-install-dependencies so that if archive fetch fails, the entire build fails.
That might be too simple though.
There are other possibilities than the ones you already mentioned that make this unworkable even on the buildbot—the bugs that make builds sometimes run out of order, and noarch ports that are skipped because they will be built by another builder, but then are needed as dependencies for something else, to name two.
So a check for whether an archive is available for each dependency is definitely needed before skipping the deactivation before installation.
Something like the opposite of tools/dependencies.tcl
would probably be needed to deactivate only the unneeded ports.
comment:9 Changed 4 years ago by mascguy (Christopher Nielsen)
Cc: | mascguy added |
---|
comment:10 Changed 4 years ago by mascguy (Christopher Nielsen)
Large dependencies - specifically, ports consisting of many thousands of files, like the various texlive
-related components - would also benefit from speedup of the file registration process (issue:56793).
comment:11 Changed 3 years ago by mascguy (Christopher Nielsen)
Update: Josh's fix for issue:56793 will improve dependency installation times in some situations. And for the most extreme cases - ports that depend on huge components like texlive-fonts-extra
- that fix is a game-changer! (Both for buildbot/CI jobs, as well as the user experience in-general.)
Net-Net: For ports like auto-multiple-choice
, which fit into the latter category, we'll see overall dependency installation time reduced dramatically. Great stuff!
comment:12 Changed 3 years ago by mascguy (Christopher Nielsen)
Cc: | mascguy removed |
---|
comment:13 Changed 3 years ago by jmroot (Joshua Root)
In f83c5266f670e1334e6a3eb1591194c6c0efe593/mpbb (master):
comment:14 Changed 3 years ago by jmroot (Joshua Root)
The timing breakdown for the latest auto-multiple-choice build is now:
init took 1 seconds deactivating unneeded ports took 61 seconds calculating deps took 8 seconds checking failcache took 16 seconds installing deps took 2 seconds activating deps took 2375 seconds
386 dependencies needed to be activated, of which 5 needed to be installed (from binary archives) first. https://build.macports.org/builders/ports-11_x86_64-builder/builds/69530
The build of auto-multiple-choice-devel right after that really sees the benefits, of course:
init took 2 seconds deactivating unneeded ports took 89 seconds calculating deps took 0 seconds checking failcache took 0 seconds installing deps took 0 seconds activating deps took 0 seconds
https://build.macports.org/builders/ports-11_x86_64-builder/builds/69531
comment:15 Changed 3 years ago by jmroot (Joshua Root)
Replying to jmroot:
the bugs that make builds sometimes run out of order
A textbook example of this, BTW: https://build.macports.org/builders/ports-12_x86_64-builder/builds/27325
The build of gcc-devel with priority 7001.2 ran before the build of libgcc-devel with priority 7001.1. The good news at least is that the mpbb code seems to have handled this perfectly.
comment:16 Changed 3 years ago by mascguy (Christopher Nielsen)
Cc: | mascguy added |
---|
Part of it is probably how many of the dependencies are being uninstalled between builds.