Base should support fetching git submodules (git submodule update)

[mojca (Mojca Miklavec)]

If ports fetched via git (or via the github portgroup) have submodules, those submodules are left out (and should usually be fetched via git submodule update).

It would be great if the base would provide a straightforward way to also fetch the submodules automatically.

Examples:

multimarkdown and git-flow ports use

# https://github.com/fletcher/MultiMarkdown-4/issues/7
fetch.type          git
post-fetch {
    system -W ${worksrcpath} "git submodule update --init"
}

and

post-fetch {
    # Does the github portgroup support 'clone --recursive'?
    # This will be removed once the python re-write is complete.
    system -W ${worksrcpath} "git submodule update --init"
}

​René used an even more complex approach.

See also #16373.

[RJVB (René Bertin)]

I cannot remember exactly why I ended up with the complex approach, and whether it had to do with peculiarities of the remote git repo. I've already encountered several occasions where the latest submodule commit wasn't compatible with the latest Qt-Creator commit and I had to use an earlier commit. I do remember though that I couldn't determine with certainty what revision you get when doing a submodule update --init. The whole approach does appear to be geared towards getting the latest commit. Which makes sense, but not so much so for MacPorts.

If we use git to fetch sources for a port, the assumption is that we grab a known version ("commit") which we have tested sufficiently to be reasonably sure it work at least as good as the latest official release. That should also apply to submodule checkouts, which is why in the end I went with an approach which more or less treats the submodule as an independent repo that just happens to be checkout out under another working copy.

[ci42]

Cc Me!

[mojca (Mojca Miklavec)]

Indeed. Submodules should be specified together with the shasum.

[ryandesign (Ryan Carsten Schmidt)]

Sounds like the current instructions for how to use submodules in the github portgroup are insufficient, because they don't mention using a shasum for submodule updates. If so, please update the instructions. Many ports follow those instructions and will need updates too.

[mojca (Mojca Miklavec)]

Do you man the following free text in github-1.0.tcl?

# Some projects' tag- or commit-based distfiles will not contain all the
# necessary files, if the project uses git submodules. If available, use a
# distfile from "releases" or "downloads" instead, as described above. If the
# project does not provide those, encourage the project's developers to provide
# releases. Until they do, fetch from git instead of from a distfile, and add a
# post-fetch block to fetch the submodules:
#
#   fetch.type          git
#
#   post-fetch {
#       system -W ${worksrcpath} "git submodule update --init"
#   }

I would need to study this a bit more. After reading the first answer from ​http://stackoverflow.com/questions/1777854/git-submodules-specify-a-branch-tag I'm even more confused about whether or not the commit is fixed or not and whether that depends on the version of git being used by developers.

(And perhaps also double-check the situation in SVN.)

[RJVB (René Bertin)]

Understand now why I use the approach I use?

Also, if you have to do part of the fetching explicitly anyway, using a git(hub) PortGroup uses part of its interest (but leaves the additional parsing overhead).

[mojca (Mojca Miklavec)]

I keep reading the git documentation and it seems to me that submodules are well-defined rather than arbitrary commits from the included third-party repository. (The question is: what if some port maintainers urgently need a different commit than what upstream envisioned?)

Now it seems that once a user has a sufficiently recent git available (which is already the case for MacPorts), it could be enough to use

git clone --recursive ${git.url}

or rather: I don't currently see a problem with "git submodule update --init". The question is then: are the any reasons why these steps shouldn't be executed by default? Or perhaps with something like

git.submodules yes

could trigger fetching the submodules.

I tried to read your code, René, but I don't yet completely understand what you tried to do / which problems you were facing.

[neverpanic (Clemens Lang)]

Submodules are always referenced using a well-defined commit. Checking out a different commit in a submodule isn't supported.

For Github, we usually prefer HTTP downloads instead of fetching with git. It would be nice if that also worked for repositories with submodules.

See Homebrew's approach of resources and "install resource" calls for the target directories (this example is with git trees, but this also works with HTTP tarballs): ​https://github.com/Homebrew/homebrew/blob/master/Library/Formula/v8.rb#L86-L92

[RJVB (René Bertin)]

The problem I've faced at several occasions is that a simple checkout of a known commit followed by the standard submodule fetch left me with a mismatch between the main and the sub modules. So what I do now is a git clone --recursive followed by a git checkout -q ${git.branch} in the working copy. That takes care of the main source tree. In order to get the known & tested submodule version I need to do some extra work in the post-fetch. If we ignore the reinplace in the .git/config file which is there for other reasons, I do git submodule update --init ; git submodule update in the toplevel working copy followed by a series of git commands in the submodule working copy that leave me with the known commit. It might be possible to do that in less than the 3 commands I'm using, but if so I haven't yet found out how.

Part of this complexity may stem from the fact that I've been trying to support the use of a local working copy as the git, something I clearly dropped when it became impossible. I probably ought to revisit the submodule post-fetch code to see if it cannot be simplified, but that's got a bit too much of a fixing something that ain't broke thing (and the repo is big so takes a bit too much time to check out repeatedly).

[neverpanic (Clemens Lang)]

Whenever you change the version of the base repository, you need to run git submodule update, preferably with the --recursive and --init flags, just in case there are nested submodules, or submodules have been added/removed. So yes, it is expected behavior that you get a mismatch if you do

git checkout $branch
git submodule fetch

because you never did a submodule update. Fetch operations never change the state of a working copy in git.

So for a proper implementation in MacPorts, we need:

1. git clone
2. git checkout $yourbranch_hash_or_tag
3. git submodule update --init --recursive

git clone --recursive does not buy as anything, because the state of the submodules at our target commit $yourbranch_hash_or_tag can be different from origin/master, which is what the recursive clone would give us. We cannot use git clone -b $branch_or_tag --recursive either, because that doesn't work for commit hashes.

[raimue (Rainer Müller)]

Fetching of git submodules is implemented on the ​vcs-fetch branch.

[mojca (Mojca Miklavec)]

See also #55455.

[neverpanic (Clemens Lang)]

The ​vcs-fetch branch already supports that and we're planning that for 2.5, so let's also plan this for 2.5.

[jmroot (Joshua Root)]

Ticket retargeted after milestone closed

[jmroot (Joshua Root)]

Ticket retargeted after milestone closed