BUG: 'port -f -R upgrade' rebuilds too much

[yaseppochi (Stephen J. Turnbull)]

Here's a log (partially completed, and of course I have no idea how much longer it's going to run) of port -f -R upgrade openssl. Both -f and -R are necessary AFAICT because port doesn't have versioned dependencies for at least some packages (eg gettext and libiconv) so upgrading those packages silently breaks others, even with -R, because dyld *does* track library versions, and the upgraded library won't link. This wouldn't be a problem---I'd just have an unupgraded package whose old version probably still works---except that port often has deactivated some library in the process of upgrading and doesn't reactivate it on abort due to build breakage. Thus -f is needed.

Net effect---I don't dare upgrade anything unless I can afford literally days of downtime.

On second thought, it looks like posting the log as text is too much work; I'll cross my fingers and hope there's an opportunity to attach it later.

[yaseppochi (Stephen J. Turnbull)]

Log of port rebuilding the same package many times in a row

[yaseppochi (Stephen J. Turnbull)]

On my system, `port dependents gconf | uniq -c' returns

95 gnome-vfs depends on gconf

It seems to be adding another redundancy to the count every time gnome-vfs gets rebuilt (the command logged in dpsux.txt is still running...).

Port 1.400, updated from svn about a week ago.

[yaseppochi (Stephen J. Turnbull)]

port 1.500

I gather that nobody cares, but FWIW, I just did

port -f -k upgrade git-core +doc

and among other things it went and started to install XFree86 (even though I have a perfectly usable self-built X.org installation from ancient days).

All I wanted to do was to rebuild git-core with the doc, leaving source in place so I can see why most of the docs don't seem to get installed....

[markd@…]

We care, but upgrade is an area that needs a lot of work and it just hasn't been done yet. I believe that -n will not follow dependencies and improves the situation a bit.

[markd@…]

Changing summary just a bit.

[markd@…]

For more examples of upgrade problems, see duplicate #10813.

[boeyms@…]

Cc'ing to reporter and coders with greater MacPorts-base-fu than I; I hope nobody minds.

In theory, there is a simple solution to this problem: for each of the ports to be upgraded, traverse its dependency graph depth-first, appending a port in the tree to a list once all of its dependencies have been visited or skipping the port if it has already been visited, and then upgrading the ports in order of appearance in the list. This would also enable fetching a port and its dependencies all at once, as well as enabling detection of circular dependencies (the search requires us to keep track of the dependency trail from the root of the graph to the current dependency, so if the end of trail appears in the body too, we have a loop).

My problem is that I can't figure out where in the code this ought to go. If nobody understands (or has the time) to write this from scratch, I'll try to write some Tcl-ish pseudocode that can hopefully be added with only minor modifications.

[frstan@…]

it also downloads items multiple times due to the creation of multiple subdirs once each repition at least in some cases: upgrading libpng with -R produced: -> Fetching gimp2 DEBUG: Executing org.macports.fetch (gimp2) ---> gimp-2.4.1.tar.bz2 doesn't seem to exist in /opt/local/var/macports/distfiles/gimp2/gimp2/gimp2 I have seen as many as a dozen levels in other cases.

The repition seems to arise from the dependency register getting a new entry with each new install of an item while the preceeding uninstall (during a port -R upgrade foo) does not cause the existing entry to be deleted. Therefore that dependent will get rebuilt multiple times, and next time even more times.......

we badly need a utility to purge the dependentcy register of non-unique entries. port clean --reg_duplicates --all .

[jmroot (Joshua Root)]

For reference, the issue with duplicate dep map entries noted in comment:1 is #8763. The subdirectory creation noted in comment:12 is #11971.

The method proposed in comment:11 is probably the Right Thing, but I think that after the resolution of the other two tickets, this would be mostly solved by making -R imply -n. Upgrade normally applies recursively to dependencies, but if you're recursing in the other direction, you usually don't want to go back the other way as well. Would we then need an option to complement -n in case you really do want the both-ways recursion?

[jmroot (Joshua Root)]

make -R imply -n

[jmroot (Joshua Root)]

Attached a one-liner to make -R imply -n.

[raimue (Rainer Müller)]

I am against committing intermediate solutions without at least adding a comment there why -R should imply -n. This will just become the default and noone will ever remember why -R implies -n, even once the other bugs are fixed.

[jmroot (Joshua Root)]

I think that #7361 being fixed should also help with the initial problem described in the summary.

[jmroot (Joshua Root)]

OK, the proper fix isn't to make -R imply -n. It is to turn off -R for the dependencies of the requested port. We want to upgrade its dependents, not its dependencies' dependents. The attached patch does this, and can make a huge difference to the amount of upgrade checks (which are all actual rebuilds when using -f).

The other thing to do to eliminate the last of the unnecessary upgrading would be to use the depscache for dependents as well as dependencies.

[jmroot (Joshua Root)]

Updated the patch to use the depscache for dependents. Now no port should be touched more than once.

[jmroot (Joshua Root)]

turn off -R while upgrading dependencies, and use the dep cache for dependents too

[jmroot (Joshua Root)]

Committed the fix to trunk in r37046. Thanks to Rainer for reviewing.

I think this should go into 1.6.1, so I'm leaving this ticket open until the fix is merged into the branch, or retargeted at a later release.

[raimue (Rainer Müller)]

Marking fixed, targeted for 1.7.0.