base does not handle dependencies correctly from tests variant

[barracuda156]

For example, I add a tests variant for R-bamlss:

variant tests description "Enable testing" {
   # Some optional deps are omitted.
    depends_test-append \
                    port:R-BayesX \
                    port:R-bit \
                    port:R-ff \
                    port:R-fields \
                    port:R-gamlss \
                    port:R-gamlss.dist \
                    port:R-glmnet \
                    port:R-glogis \
                    port:R-interp \
                    port:R-knitr \
                    port:R-mapdata \
                    port:R-maps \
                    port:R-rjags \
                    port:R-rmarkdown \
                    port:R-scoringRules \
                    port:R-sdPrior \
                    port:R-sf \
                    port:R-spdep \
                    port:R-splines2 \
                    port:R-statmod \
                    port:R-zoo

    test.run        yes
}

And then this does not work:

36-142% sudo port -v test R-bamlss +tests
Warning: configured user/group macports does not exist, will build as root
--->  Computing dependencies for R-bamlss..
The following dependencies will be installed:  R-sdPrior
Continue? [Y/n]: y
Warning: configured user/group macports does not exist, will build as root
--->  Fetching distfiles for R-sdPrior
--->  sdPrior_1.0-0.tar.gz does not exist in /opt/local/var/macports/distfiles/R-sdPrior
--->  Attempting to fetch sdPrior_1.0-0.tar.gz from https://cran.r-project.org/src/contrib
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 71871  100 71871    0     0  38905      0  0:00:01  0:00:01 --:--:-- 38912
--->  Verifying checksums for R-sdPrior
--->  Checksumming sdPrior_1.0-0.tar.gz
--->  Extracting R-sdPrior
--->  Extracting sdPrior_1.0-0.tar.gz
Executing:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_R_R-sdPrior/R-sdPrior/work" && /usr/bin/gzip -dc '/opt/local/var/macports/distfiles/R-sdPrior/sdPrior_1.0-0.tar.gz' | /usr/bin/gnutar --no-same-owner -xf - 
--->  Configuring R-sdPrior
Executing:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_R_R-sdPrior/R-sdPrior/work/sdPrior" && /opt/local/bin/R CMD build . --no-manual --no-build-vignettes --keep-empty-dirs 
* checking for file ‘./DESCRIPTION’ ... OK
* preparing ‘sdPrior’:
* checking DESCRIPTION meta-information ... OK
* checking for LF line-endings in source and make files and shell scripts
* checking for empty or unneeded directories
Omitted ‘LazyData’ from DESCRIPTION
* building ‘sdPrior_1.0-0.tar.gz’

--->  Building R-sdPrior
xinstall: mkdir /opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_R_R-sdPrior/R-sdPrior/work/build
Executing:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_R_R-sdPrior/R-sdPrior/work/sdPrior" && /opt/local/bin/R CMD INSTALL . --library=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_R_R-sdPrior/R-sdPrior/work/build --install-tests 
ERROR: dependency ‘pscl’ is not available for package ‘sdPrior’
* removing ‘/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_R_R-sdPrior/R-sdPrior/work/build/sdPrior’
Command failed:  cd "/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_R_R-sdPrior/R-sdPrior/work/sdPrior" && /opt/local/bin/R CMD INSTALL . --library=/opt/local/var/macports/build/_opt_local_var_macports_sources_rsync.macports.org_macports_release_tarballs_ports_R_R-sdPrior/R-sdPrior/work/build --install-tests 
Exit code: 1
Error: Failed to build R-sdPrior: command execution failed

However, there is no problem with R-sdPrior as such, it has needed dependencies specified:

depends_lib-append  port:R-doParallel \
                    port:R-GB2 \
                    port:R-mvtnorm \
                    port:R-pscl

If I install it directly, it works though:

36-142% sudo port -v install R-sdPrior
Warning: configured user/group macports does not exist, will build as root
Warning: configured user/group macports does not exist, will build as root
--->  Computing dependencies for R-pscl.
--->  Fetching distfiles for R-pscl
--->  pscl_1.5.9.tar.gz does not exist in /opt/local/var/macports/distfiles/R-pscl
--->  Attempting to fetch pscl_1.5.9.tar.gz from https://cran.r-project.org/src/contrib
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 2948k  100 2948k    0     0   658k      0  0:00:04  0:00:04 --:--:--  678k
--->  Verifying checksums for R-pscl

The same issue with R-GB2, which began to install without updating dependencies, but did work correctly when invoked on its own.

[jmroot (Joshua Root)]

Can't reproduce with a set of simple test ports.

% port deps testport1
testport1 @1.0_0 has no dependencies.
% port deps testport1 +tests
Full Name: testport1 @1.0_0+tests
Test Dependencies:    testport2
% port deps testport2       
Full Name: testport2 @1.1_0
Library Dependencies: testport3
% sudo port test testport1 +tests
--->  Computing dependencies for testport1
--->  Dependencies to be installed: testport2 testport3
--->  Fetching archive for testport3
--->  Attempting to fetch testport3-1.0_0.darwin_23.x86_64.tbz2 from file:///opt/local/var/macports/packages/testport3
--->  Fetching distfiles for testport3
--->  Verifying checksums for testport3
--->  Extracting testport3
--->  Configuring testport3
--->  Building testport3
--->  Staging testport3 into destroot
--->  Installing testport3 @1.0_0
--->  Activating testport3 @1.0_0
--->  Cleaning testport3
--->  Fetching archive for testport2
--->  Attempting to fetch testport2-1.1_0.darwin_23.x86_64.tbz2 from file:///opt/local/var/macports/packages/testport2
--->  Fetching distfiles for testport2
--->  Verifying checksums for testport2
--->  Extracting testport2
--->  Configuring testport2
--->  Building testport2
--->  Staging testport2 into destroot
--->  Installing testport2 @1.1_0
--->  Activating testport2 @1.1_0
--->  Cleaning testport2
--->  Fetching distfiles for testport1
--->  Verifying checksums for testport1
--->  Extracting testport1
--->  Configuring testport1
--->  Building testport1
--->  Staging testport1 into destroot
--->  Testing testport1
Warning: testport1 is configured to build for the architecture(s) 'x86_64', but did not install any Mach-O files.

Can you give a reproduction recipe that only uses versions of ports that exist in the current macports-ports master branch? Preferably ones that don't have large dependency lists?

[ryandesign (Ryan Carsten Schmidt)]

Didn't we already decide that a dependency cycle exists regarding R ports used for testing? Behavior is undefined when a dependency cycle exists.

[ryandesign (Ryan Carsten Schmidt)]

I'm referring to #67521 here, although I haven't identified a specific dependency cycle here yet.

[barracuda156]

Replying to ryandesign:

Didn't we already decide that a dependency cycle exists regarding R ports used for testing? Behavior is undefined when a dependency cycle exists.

I still do not understand why the base considers that a dependency cycle, since it is not (a real dependency cycle makes installation impossible, but it is not the case with test dependencies), and it was working at some point correctly, but this issue appears different (has to be checked in fact, could be the same also).

[barracuda156]

Replying to jmroot:

Can you give a reproduction recipe that only uses versions of ports that exist in the current macports-ports master branch? Preferably ones that don't have large dependency lists?

I will try to reconstruct a case. I think what may be happenning is that the base for some reason fails to trigger updating dependencies in this scenario. They are not found because they were installed for R 4.3, and R 4.4 cannot see them.

[jmroot (Joshua Root)]

Replying to barracuda156:

I think what may be happenning is that the base for some reason fails to trigger updating dependencies in this scenario.

OK, that makes more sense but is very different to what you seemed to be claiming originally.

[jmroot (Joshua Root)]

I still can't reproduce that in the simplified case though:

 port installed testport2
The following ports are currently installed:
  testport2 @1.0_0 (active)
% port info --version testport2
version: 1.1
% sudo port test testport1 +tests
--->  Fetching archive for testport3
--->  Attempting to fetch testport3-1.0_0.darwin_23.x86_64.tbz2 from file:///opt/local/var/macports/packages/testport3
--->  Fetching distfiles for testport3
--->  Verifying checksums for testport3
--->  Extracting testport3
--->  Configuring testport3
--->  Building testport3
--->  Staging testport3 into destroot
--->  Installing testport3 @1.0_0
--->  Activating testport3 @1.0_0
--->  Cleaning testport3
--->  Computing dependencies for testport2
--->  Fetching archive for testport2
--->  Attempting to fetch testport2-1.1_0.darwin_23.x86_64.tbz2 from file:///opt/local/var/macports/packages/testport2
--->  Fetching distfiles for testport2
--->  Verifying checksums for testport2
--->  Extracting testport2
--->  Configuring testport2
--->  Building testport2
--->  Staging testport2 into destroot
--->  Installing testport2 @1.1_0
--->  Cleaning testport2
--->  Computing dependencies for testport2
--->  Deactivating testport2 @1.0_0
--->  Cleaning testport2
--->  Activating testport2 @1.1_0
--->  Cleaning testport2
--->  Computing dependencies for testport1
--->  Fetching distfiles for testport1
--->  Verifying checksums for testport1
--->  Extracting testport1
--->  Configuring testport1
--->  Building testport1
--->  Staging testport1 into destroot
--->  Testing testport1
Warning: testport1 is configured to build for the architecture(s) 'x86_64', but did not install any Mach-O files.

[ryandesign (Ryan Carsten Schmidt)]

Replying to barracuda156:

I still do not understand why the base considers that a dependency cycle,

MacPorts base doesn't contain any code that "considers" something to be a dependency cycle. Rather, MacPorts base was designed under the assumption that dependency cycles do not exist. If that assumption is violated, behavior is undefined. Anything could happen. MacPorts base might do what you want or it might not.

Imagine port R-foo depends on port R-bar and R-bar depends on R-foo. (I'm referring to dependencies that affect installation, like depends_build or depends_lib, not depends_test.) That's a dependency cycle. It cannot be satisfied. Imagine MacPorts base trying naively to install R-foo. It finds that R-foo depends on R-bar so it tries to install R-bar first. It then looks at R-bar, finds that it depends on R-foo, so it tries to install R-foo first. And so on forever. Infinite loop.

I've never observed MacPorts get into an infinite loop so I assume it must contain some safeguard against that, probably for performance reasons. Imagine that both R-foo and R-bar also depend on R-CRAN-recommended. It would be a waste of time for MacPorts to check twice whether it still needed to install R-CRAN-recommended so it probably keeps track of ports it has already considered. So now, when asked to install R-foo, it checks its dependencies. Let's say R-CRAN-recommended is already installed so then it looks at the other dependency, R-bar, which isn't installed yet, so it checks its dependencies. It finds that R-bar depends on R-CRAN-recommended, which is already installed, and R-foo, which isn't installed yet but it has already checked R-foo's dependencies, therefore it must be time to install R-foo, so it does, even though its dependency R-bar has not been installed yet. It would have been if the dependency cycle had not existed. I have not read the MacPorts base code so I do not know if this accurately describes what it does but it is the behavior you observed and the explanation seems logical to me.

since it is not (a real dependency cycle makes installation impossible, but it is not the case with test dependencies),

I'm imagining that one of the test dependencies eventually depends on the port being tested, hence a cycle. I cannot see such a cycle with the ports on my system but perhaps you have other local modifications that introduce such a cycle.

and it was working at some point correctly,

That's the tricky thing with dependency cycles. If you already have one of the ports in the cycle installed, you won't notice the problem. That's how cycles get inadvertently introduced into MacPorts. "It was working", and then someone adds a dependency to one port, not realizing that the dependency they just added eventually depends on the port they added the dependency to.