github portgroup: ability to download files from more than one github project

[ryandesign (Ryan Carsten Schmidt)]

Some ports need to download distfiles from more than one GitHub project.

I could see a good implementation of this being complicated and requiring a rewrite of much of the portgroup's existing functionality.

[mojca (Mojca Miklavec)]

See also #50708.

[ryandesign (Ryan Carsten Schmidt)]

It would also be nice to be able to include the github portgroup and use it to add a github distfile to a port that doesn't use github for its primary distfile, e.g. opencv.

[mojca (Mojca Miklavec)]

I was looking for other tickets (I'm sure there are some other relevant ones, I just couldn't find them right now).

As you already noted, apart from the ability to fetch from multiple GitHub projects, MacPorts suffers an equally big problems that one cannot:

• download and extract files of different type (.tar.gz and .zip for example)
• fetch from one git and one subversion repository
• fetch some parts from subversion and some parts from tarball ... or basically any other combination of those

It would be nice to find a way to combine an arbitrary number of fetching types.

[kencu (Ken)]

Without getting too fancy about it, homebrew has a concept of adding a resource that is specified simply by a complete URL. It is downloaded without further fanfare, and then can be used as desired.

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

I was looking for other tickets (I'm sure there are some other relevant ones, I just couldn't find them right now).

As you already noted, apart from the ability to fetch from multiple GitHub projects, MacPorts suffers an equally big problems that one cannot:

• download and extract files of different type (.tar.gz and .zip for example)
• fetch from one git and one subversion repository
• fetch some parts from subversion and some parts from tarball ... or basically any other combination of those

It would be nice to find a way to combine an arbitrary number of fetching types.

True, but I don't consider that in-scope for this ticket.

[ryandesign (Ryan Carsten Schmidt)]

Replying to kencu:

Without getting too fancy about it, homebrew has a concept of adding a resource that is specified simply by a complete URL. It is downloaded without further fanfare, and then can be used as desired.

And I consider that typical lack of fanfare to be one of homebrew's weaknesses, and what we do to be one of our strengths.

What happens if, for example, GitHub changes their URLs? They've done it once before, and it was nice that we could make the needed changes centrally in the portgroup instead of having to modify hundreds of Portfiles.

Further, in the case of GitHub, the portgroup does more than just download a file. It sets proper livecheck and homepage defaults too. And in the case of automatically-generated tarballs, it renames the extracted directories to a predictable name. You don't want to have to duplicate all that code to each portfile either.

And very simply, MacPorts has the idea that a single distfile might be distributed by a network of mirror servers. MacPorts lets a port specify the distfile (basename and extension) once, then define the list of servers and server directories where that file might be found. We even have central lists of known mirror server groups, so that all ports that download from, say, GNU, can use it, and if a new GNU mirror appears, or if one disappears, it can be updated centrally and all ports using that group can benefit.

All of this is not possible if one simply, and without fanfare, specifies complete URLs of files to download. Which is why MacPorts doesn't do that.

[ryandesign (Ryan Carsten Schmidt)]

I'm trying to decide what interface should be exposed to ports for this functionality. One option is to simply have a proc that takes all the parameters needed to specify a GitHub distfile, and then it adds the needed master_sites and distfiles and post-extract blocks to handle it. For example, for the libpsl port:

# hypothetical code
github.add_distfile publicsuffix list 85fa8fbdf73a0f2fcf5f4790c204394557dfbaf3 {} archive

But this becomes increasingly messy as more parameters are needed. It can be hard to remember which parameter is which, and skipping parameters requires entering empty strings. Would it make more sense to try to use named parameters? The checksums line comes to mind as an example of an existing option that uses key-value pairs. So it could instead be:

# hypothetical code
github.add_distfile author  publicsuffix \
                    project list \
                    tag     85fa8fbdf73a0f2fcf5f4790c204394557dfbaf3 \
                    type    archive

It's clear what each parameter is for and it's easy to omit unneeded parameters. But it's different from how github.setup is invoked.

This could be an opportunity to have a whole new github-2.0 portgroup with a different github.setup signature. Or, since we don't really like .setup procs and have tried to get away from them in other portgroup, maybe we can find an interface that doesn't require one here either. For example, instead of having github.setup set name and version, have the portfile author set name and version and let the portgroup take its defaults from that. Only github.author need be specified separately. But then we have the tag prefix which often needs to be specified too.

A single proc to add a distfile might not be sufficient. What if we want to add a distfile, but not extract it? We don't have its filename easily available.

[mascguy (Christopher Nielsen)]

Add keyword portgroup, to pg-related tickets