Use primary port category for fetched distfile layout

[wsiegrist@…]

The distfile directory ($prefix/var/macports/distfiles/) currently uses 1 level of directory based on port name. This means mirroring also uses the single directory level.

I propose to use the primary category to layout mirrors and distfiles with an additional directory. The main reason for this is the layout on distfiles.macports.org. We need the 2 layers of directories to make browsing more managable.

The change will be mostly invisible to users as far as their local installations are concerned. This will make distfiles match the layout of Portfiles in the svn and rsync repositories.

The simplest fix, and the one I provide a patch for here, is to read the categories value and set distpath accordingly during fetch_init. This affects all fetching and mirroring. The only impact I see to end users is if they pre-fetch something, upgrade to the patched code, then try to install. The distfile would be in the wrong place and re-fetched. This case seems rare and a minor inconvenience at that.

[wsiegrist@…]

add primary category to distpath for better distfile/mirror layout

[jmroot (Joshua Root)]

Distfiles do tend to stick around in my experience, so I think selfupdate will need to rearrange the distfiles directory. Otherwise users would end up with distfiles in the wrong place that are not removed by clean --all.

Also, this has the side effect of changing the behaviour of dist_subdir. Setting it will now only work as expected if all the ports sharing a dist_subdir also have the same primary category. Not sure how much of an issue this is in practice.

[raimue (Rainer Müller)]

First of all, there are other places where this variable is used in portclean.tcl which would also need the new location for proper cleaning.

We could change the default of dist_subdir to fix it in all locations without much rewriting. I didn't test it, but I mean something like this:

default dist_subdir {[lindex categories 0]/${portname}}

This would still allow to override the default dist_subdir with the fetch directory of another port by using dist_subdir category/portname. The downside is that we would need to change it in all existing Portfiles using dist_subdir (I counted 190 ports).

[afb@…]

Sigh, I wish I had added that port dist addition to survive the transitions...

(it would output the distfiles directory for a port, much like port work now)

[wsiegrist@…]

Good point on port clean. I originally was trying to minimize the impact and only change the mirror procedure, but then moved it to fetch. If we did make this change, I agree with changing the default line instead so all targets get the change.

A script to move existing distfiles seems simple, but I didnt want this to be such a monumental transition. So I'm tempted to just drop this and move the distfiles after each port mirror on the server. I was just trying to leverage/extend the existing code.

What about an option in macports.conf that defaults to the current behavior? And provide a simple script to rearrange existing files? That way people have to opt in but can otherwise ignore this. But of course, I dont want to spend the effort on a feature no one will use, when I can just make the layout work on the server and leave base alone.

[ryandesign (Ryan Carsten Schmidt)]

@afb: I was looking for something like "port dist" recently. Didn't know it had been considered before. Would love to see it implemented!

@wsiegrist: I would tend to vote against the change proposed in this ticket, because I don't see a benefit, and I do see many opportunities for problems as a result, as already discussed in this ticket and by email. (Orphaned distfiles or needing a migration script, dist_subdir being manipulated by many ports already which would break.) Also, I "cd" to my various ports' distfiles directories fairly often, and it's already quite enough typing to get there without having the additional category directory name, not to mention that I don't always remember my ports' categories. For the portfiles, I'm a big fan of "port cddir foo" (my local alias for "cd `port dir foo`") and why I'd love a "port dist foo" (for which I could write an alias "port cddist foo" to do "cd `port dist foo`").

[wsiegrist@…]

Ok, there's too much risk and no benefits to the change, so I'm closing the ticket. Thanks.