Lots of golang ports are downloading dependencies at build time

[amake (Aaron Madlon-Kay)]

With the introduction of the Go modules system, it's very easy to accidentally make a port that downloads dependencies at build time.

I want to add GOPROXY=off GO111MODULE=off to build.env to prevent this, but there are many ports that currently fail in my testing.

Discover ports using the golang-1.0 portgroup (currently 85):

find . -name Portfile | xargs grep -l -E 'PortGroup +golang' | xargs -n 1 dirname | xargs -n 1 basename

Of those, ports that failed to build with GOPROXY=off GO111MODULE=off appended to build.env:

port	maintainer	status
annie	@l2dy,openmaintainer	​fixed
aws-vault	@herbygillot,openmaintainer	​fixed
certigo	@herbygillot,openmaintainer	​fixed
chezmoi	@herbygillot,openmaintainer	​fixed
cloudmonkey	@herbygillot,openmaintainer	​fixed
copilot	@herbygillot,openmaintainer	Non-trivial: has a complicated asset generating step that relies on multiple golang tools that should be ports of their own: mockgen, packr2
croc	@herbygillot,openmaintainer	​fixed
elvish	@herbygillot,openmaintainer	​fixed
evans	@herbygillot,openmaintainer	​fixed
fzf	@cardi,openmaintainer	​fixed
gitqlite	@herbygillot,openmaintainer	​fixed
glow	@herbygillot,openmaintainer	​fixed
go-migrate	@herbygillot,openmaintainer	Non-trivial: depends on multiple modules housed in same repo but with different versions
golangci-lint	@herbygillot,openmaintainer	Non-trivial: import path error
gore	@herbygillot,openmaintainer	​fixed
gotop	@i0ntempest,openmaintainer	​fixed
grpcurl	@herbygillot,openmaintainer	Non-trivial: depends on multiple modules housed in same repo but with different versions
hugo	@cardi,openmaintainer	Non-trivial: depends on multiple modules housed in same repo but with different versions
ipfs	@sirn,openmaintainer	Non-trivial: extracted source file name collisions
istioctl	@nickolaev,openmaintainer	due to #61184
jenkins-cli	@harens,openmaintainer	​fixed
k9s	@breun,openmaintainer	Non-trivial: import path error
krew	@herbygillot,openmaintainer	​fixed
kubergrunt	@herbygillot,openmaintainer	#61185; Non-trivial: import path error
kustomize	@breun,openmaintainer	Non-trivial: depends on multiple modules housed in same repo but with different versions
micro	@herbygillot,openmaintainer	​fixed
mole	@herbygillot,openmaintainer	​fixed
newreleases	@herbygillot,openmaintainer	​fixed
pulumi	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
rclone	@eborisch,openmaintainer	Non-trivial: depends on multiple modules housed in same repo but with different versions
scw	@dgsb,openmaintainer	​pending
staticcheck	@herbygillot,openmaintainer	​fixed
syncthing	@lbschenkel,openmaintainer	​pending
tektoncd-cli	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
terragrunt	@mjrc,openmaintainer	Non-trivial: depends on multiple modules housed in same repo but with different versions
trivy	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
uni	@herbygillot,openmaintainer	​fixed
up	@herbygillot,openmaintainer	​fixed
webify	@harens,openmaintainer	​fixed
wtfutil	@herbygillot,openmaintainer	Non-trivial: contains numerous dependencies from unsupported domains
yq	@herbygillot,openmaintainer	​fixed
nebula	@herbygillot,openmaintainer	​fixed
qri	@herbygillot,openmaintainer	Non-trivial: depends on multiple modules housed in same repo but with different versions

I have not yet checked that each of these failures is from being unable to download dependencies, as opposed to e.g. something more esoteric about GO111MODULE.

[ryandesign (Ryan Carsten Schmidt)]

Herby, you may wish to know about this since you've been doing a lot of go-based ports.

[herbygillot (Herby Gillot)]

Thank you @ryandesign, @amake

I will look to transition more of the Go ports over time to using go.vendor. It will be cumbersome, but will give it a go.

[herbygillot (Herby Gillot)]

go2port goes not seem to work very well... the checksums that it generates for go.vendor don't always match what MacPorts sees.... so when attempting to build a Portfile derived from go2port, the build process immediately fails on checksum mismatches.

[amake (Aaron Madlon-Kay)]

go2port is very much a "best effort" tool :(

I just had a go at fixing the glow port ​here and noted the following tricky parts:

1. go2port 20200217 will duplicate some dependencies. I just fixed this and pushed an update to the port, so you should see it soon.
2. There were some checksum mismatches as you mentioned; surprisingly enough it seemed like GitHub sometimes served truncated tarballs, because deleting and re-downloading fixed it. This is quite troubling, but I'm not sure what I can do about it.
3. Simply pasting the generated go.vendors block is not enough; you also have to give the main distfile a filename. Until you do so, the checksum phase will fail in a way that makes it look like issue (2) above.

One issue you might run into is that custom domains are generally not supported by either go2port or the golang portgroup*. So far all the packages with custom domains I've seen have been mere redirects to a supported domain like github.com. If you encounter something else then it might be quite difficult to handle.

*Someone has ​offered a patch to support arbitrary domains in go2port but since the golang portgroup would also need a lot of work I haven't yet properly reviewed it.

[Aaron Madlon-Kay <amake@…>]

In 9876801804d06272278ebfdbb8512532440a0f83/macports-ports (master):

glow: don't fetch dependencies in build phase

See #61192

[herbygillot (Herby Gillot)]

I feel like instead of having go2port reinvent the wheel, we should download deps _exactly_ as Go would... and so perhaps the go mod download command could be of some use here. The only downside is that Go seems to prefer using .zip files, which won't match MacPort's preferred .tar.gz suffix used in most places.

Within a Go project's source tree, go mod download -json will print the manifest that Go would use to download the project's dependencies if that project is using Go modules. This can lead to a much more robust way for MacPorts to vendor the Go deps.

Sample output:

{
        "Path": "cloud.google.com/go",
        "Version": "v0.26.0",
        "Info": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.info",
        "GoMod": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.mod",
        "Zip": "/Users/herby/go/pkg/mod/cache/download/cloud.google.com/go/@v/v0.26.0.zip",
        "Dir": "/Users/herby/go/pkg/mod/cloud.google.com/go@v0.26.0",
        "Sum": "h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ=",
        "GoModSum": "h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw="
}
{
        "Path": "github.com/BurntSushi/toml",
        "Version": "v0.3.1",
        "Info": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.info",
        "GoMod": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.mod",
        "Zip": "/Users/herby/go/pkg/mod/cache/download/github.com/!burnt!sushi/toml/@v/v0.3.1.zip",
        "Dir": "/Users/herby/go/pkg/mod/github.com/!burnt!sushi/toml@v0.3.1",
        "Sum": "h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=",
        "GoModSum": "h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU="
}
...

[amake (Aaron Madlon-Kay)]

Some background:

I made go2port before the modern go module system existed, when there were several competing dependency managers (glide, gopkg, glock); it's modeled after other portfile generators like pypi2port and cpan2port. I made the golang-1.0 portgroup largely based on other portgroup idioms and the peco portfile.

I am not a golang guy. I just wanted to make and maintain ports for tools that happened to be in golang. go2port was my first (and last, so far) golang project. So it sounds like you have a much better understanding of the ecosystem than I do.

There is no mandate to use either go2port or the golang-1.0 portgroup. If you have better ideas that better meet MacPorts requirements then by all means we should move on.

The go mod download -json output sounds very useful, but I'm not seeing a way to use that *within* a portfile that lets MacPorts manage the fetching (which is what we're really concerned with here). The best I can imagine is using it to generate the distfiles and checksums sections, but that's basically what go2port already does.

[amake (Aaron Madlon-Kay)]

I should add: In the pre-go-module-system world, there was no danger of dependencies being automatically downloaded, because all projects (as far as I know) either committed their dependencies or used a third-party tool like glide, et al. It's only recently that this has become a problem.

[i0ntempest]

What's the best way of getting the names/urls of all the dependencies used by Go? Once I know that I'll try fixing gotop.

[amake (Aaron Madlon-Kay)]

Replying to i0ntempest:

What's the best way of getting the names/urls of all the dependencies used by Go? Once I know that I'll try fixing gotop.

Currently it's to use ​go2port. When it works well it's good, but there are often little stumbling blocks. See the "fixed" links in the table for details.

[i0ntempest]

So gotop needs to fetch stuff from howett.net/plist which go.vendors doesn't support. Alternative is ​https://github.com/DHowett/go-plist, that fetches it to ${gopath}/src/github.com/DHowett/go-plist, then I'll need to link it to ${gopath}/src/howett.net/plist, only then it will build. Don't know if there's a better way but it works. PR: ​https://github.com/macports/macports-ports/pull/8486

[breun (Nils Breunese)]

I created and maintain the k9s and kustomize ports, because I needed those tools, but I am not very familiar with Go and I’m not really sure what I need to do. If anyone could outline the process that would’ve helpful. Or maybe someone can even create pull requests for these ports?

[Aaron Madlon-Kay <amake@…>]

In 6640210512cc56fe6ce46af3a0a4710d9d54edba/macports-ports (master):

yq: don't fetch dependencies in build phase

See #61192

[i0ntempest]

Run go2port on your software repo and it will output a portfile. You take the checksum block and the go.vendors block and put them into your portfile (replace the old checksum block), add GOPROXY=off and GO111MODULE=off to build.env, and see if it works. You'll most likely need to do a bit of additional fixing and tweaking.

[Aaron Madlon-Kay <amake@…>]

In 00bbcb613bfaf1aa0d87b2ac85000e441f520197/macports-ports (master):

up: don't fetch dependencies in build phase

See #61192

[i0ntempest]

In cc761aa313fab209e261105a7332d22fc80b91e1/macports-ports (master):

gotop: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 439ecf855855cedac020127bb66c3f4ceb8d1c8a/macports-ports (master):

uni: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In a153862bd9542074156089436102d37273ba8501/macports-ports (master):

staticcheck: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 397d9d31d286b03813c6be3f4b30ad9450f457b2/macports-ports (master):

mole: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In ba97efd7ca98ad4d750042744abcddaa796ee644/macports-ports (master):

newreleases: don't fetch dependencies in build phase

See #61192

[herbygillot (Herby Gillot)]

In 64cac642068e98d6f2ddcbbad398df2f6e664bc1/macports-ports (master):

yaegi: update to 0.9.1

• ensure no Go dependencies are to be fetched from the Internet during

build time with GOPROXY=off

• See: #61192

[herbygillot (Herby Gillot)]

Seeing another issue where things fail when go.vendors contains:

• two modules of the same name even though they live in different repos
• or two or more versions of the exact same module (main project may require vX, but a dependency uses vY)

[amake (Aaron Madlon-Kay)]

Replying to herbygillot:

Seeing another issue where things fail when go.vendors contains:

• two modules of the same name even though they live in different repos
• or two or more versions of the exact same module (main project may require vX, but a dependency uses vY)

I fixed one limitation relating to this recently in ​https://github.com/macports/macports-ports/commit/3ad6f3ce0a0c011466bbd3e076295879f1148975, but there are still limitations relating to how GitHub names tarballs: tarballs fetched from a tag ref have the tag's hash in the name, so to identify the extracted directory you must either

• Specify the hash in addition to the tag (cumbersome), or
• Only use the hash and never a tag (also cumbersome, harder to understand)

The workaround currently used by both golang-1.0 and github-1.0 portgroups is to use globs (${author}-${project}-*). This works for 99% of cases but obviously fails in the case of multiple tarballs from the same repo.

[breun (Nils Breunese)]

I tried using go2port for the k9s port and I got a Portfile with really long list of go.vendors. I then added the other metadata, like description, license, etc. from the current k9s Portfile.

However, when I run portindex I get this error:

Error: go.vendors can't handle dependencies from k8s.io
Failed to parse file sysutils/k9s/Portfile: can't set "go.vendors": invalid command name "unsupported dependency domain"

All entries with github.com URLs in go.vendors look fine, but the ones for k8s.io, sigs.k8s.io, google.golang.org, go.uber.org, helm.sh, go.opencensus.io, rsc.io, vbom.ml all have rmd160, sha256 and size set to 0. It seems that these domains are not supported by go.vendors. Is there a way around this?

[amake (Aaron Madlon-Kay)]

Replying to breun:

All entries with github.com URLs in go.vendors look fine, but the ones for k8s.io, sigs.k8s.io, google.golang.org, go.uber.org, helm.sh, go.opencensus.io, rsc.io, vbom.ml all have rmd160, sha256 and size set to 0. It seems that these domains are not supported by go.vendors. Is there a way around this?

Those domains are indeed not supported by go.vendors. Golang allows arbitrary domains in package identifiers; often those domains simply redirect to well-known hosts like GitHub, but at the moment we don't have a way to figure that out.

The only way to handle this right now would be

1. Manually resolve those package IDs
2. If the actual host is a known one like GitHub, then replace the package ID with the GitHub, etc., equivalent
3. In the post-extract phase, move the extracted directory to its correct place in the GOPATH according to its original package ID

If the package resolves to a host that is not understood by the golang-1.0 portgroup, then you would have to set up the distfile manually the usual way with distfiles and master_sites.

I have an idea for an improvement. I will ping you with more later.

[amake (Aaron Madlon-Kay)]

I've updated both go2port and the golang-1.0 portgroup to support auto-resolving custom domains (thanks to ​@mohd-akram). With this, a lot of the dependencies for k9s will be handled nicely automatically. However some will not:

• Any deps "deeper" than the top level (like github.com/Azure/go-autorest/autorest/adal) don't work right. The root is also a dependency so you can delete those.
• google.golang.org/protobuf is hosted at go.googlesource.com; tarballs from here ​are not deterministic. You can manually replace it with github.com/protocolbuffers/protobuf-go
• The Microsoft dependencies reference github.com/Microsoft, but this has been renamed to github.com/microsoft. This needs to be manually resolved.

Unfortunately after fixing all of that, I got build errors like:

# github.com/deislabs/oras/pkg/content
../../deislabs/oras/pkg/content/file.go:23:2: cannot use &FileStore literal (type *FileStore) as type ProvideIngester in assignment:
	*FileStore does not implement ProvideIngester (wrong type for ReaderAt method)
		have ReaderAt(context.Context, "github.com/opencontainers/image-spec/specs-go/v1".Descriptor) (content.ReaderAt, error)
		want ReaderAt(context.Context, "github.com/containerd/containerd/vendor/github.com/opencontainers/image-spec/specs-go/v1".Descriptor) (content.ReaderAt, error)

The errors seem to have something to do with incorrectly resolving full type names in the GOPATH. I don't know enough about go to fix it.

I'll attach the complete portfile in case it helps.

[amake (Aaron Madlon-Kay)]

k9s portfile with go.vendors

[mohd-akram (Mohamed Akram)]

I'm also having an issue updating ​caddy because of two similarly named packages (qtls and qtls-go1-15). I think fetching from /archive/ instead of /tarball/ on GitHub can fix this issue since it uses more sane naming. Perhaps there could be an option akin to github.tarball_from to not break existing ports. In the meantime, filtering the globbed list via regex could help too.

[herbygillot (Herby Gillot)]

In d7f7ca6dabc2f935ccc179e60fe5ff5200976ffa/macports-ports (master):

aws-vault: update to 6.2.0

• switch deps to go.vendor, disable downloading dependencies at build time
• See: #61192

[herbygillot (Herby Gillot)]

In 236e0c2f119ed5b5fd1d6f2da5b3ec670d36706b/macports-ports (master):

elvish: vendor go dependencies, don't download them at build time

• See: #61192

[herbygillot (Herby Gillot)]

In 76f74db8796dc5f03c9fcaf6b686dfd9ce1addca/macports-ports (master):

gore: vendor Go dependencies, do not download dependencies at build time

• See: #61192

[harens (Haren S)]

In 1d5abbe395918dc567f95b0e65e31646b513d41b/macports-ports (master):

webify: don't fetch dependencies in build phase

[amake (Aaron Madlon-Kay)]

I tried to go.vendorize the kubergrunt port and ran into basically the same issues described ​above. I'll attach the portfile just in case someone figures out how to fix it.

[amake (Aaron Madlon-Kay)]

kubergrunt portfile with go.vendors

[amake (Aaron Madlon-Kay)]

Similar import issue for golangci-lint as with kubergrunt and k9s.

[amake (Aaron Madlon-Kay)]

golangci-lint portfile with go.vendors

[Aaron Madlon-Kay <amake@…>]

In 22fba12fc00e5732fd60ca024c2e2324bc91670e/macports-ports (master):

krew: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 15d58e010e86a746536a2dd7d1d4e4b553267e76/macports-ports (master):

micro: don't fetch dependencies in build phase

See #61192

[breun (Nils Breunese)]

I should add: In the pre-go-module-system world, there was no danger of dependencies being automatically downloaded, because all projects (as far as I know) either committed their dependencies or used a third-party tool like glide, et al. It's only recently that this has become a problem.

I must admit I do not completely understand what the problem exactly is. I maintain a couple of Go ports, but I'm not very familiar with the Go ecosystem. I am however very familiar with the Java ecosystem and the Maven build tool specifically and I'm used to build tools downloading dependencies at build time (when not previously downloaded and locally cached). Maven uses Maven repositories to download dependencies, which support hashes and signatures, etc. Is the problem that Go builds download dependencies without any checks? If not, what is the problem exactly?

[breun (Nils Breunese)]

k9s portfile with go.vendors

Thanks for creating this. However, when I try to build this locally, it takes quite a long time and then fails to build with some Go build error that I don't understand. I attached k9s-with-go.vendors-failure.txt to this ticket. Anyone know what's wrong here?

[amake (Aaron Madlon-Kay)]

Replying to breun:

I must admit I do not completely understand what the problem exactly is.

There are two problems:

1. Downloading dependencies outside of MacPorts' fetch mechanism prevents MacPorts from mirroring distfiles. That's bad for a number of reasons, such as: upstream could disappear, or we send a lot of traffic their way putting a strain on their infrastructure.
2. Some build-time dependency fetching schemes lack a locking mechanism, so the exact version of the dependency used can differ depending on when exactly the build was performed. This is bad for reproducibility.

I am however very familiar with the Java ecosystem and the Maven build tool specifically and I'm used to build tools downloading dependencies at build time (when not previously downloaded and locally cached).

Sure, that's fine and good, but suboptimal for a MacPorts port.

Maven uses Maven repositories to download dependencies, which support hashes and signatures, etc. Is the problem that Go builds download dependencies without any checks? If not, what is the problem exactly?

No, Maven is also problematic for (1) above. For (2), Maven does allow version ranges but they are not used that often in my experience, so that's good. But in my experience it also doesn't provide a lockfile-like mechanism that would ensure that dependencies can't be silently replaced upstream.

Are there ports building with Maven that are downloading their dependencies at build time? If so, I guess the main reason they have been given a pass in the past is, as I understand it:

1. Until recently there was no Java port, and Java could not be assumed to be available, so we basically didn't build any Java ports from source; they have mostly been grabbing pre-compiled (JAR) distfiles
2. No one was paying attention

[amake (Aaron Madlon-Kay)]

Replying to breun:

k9s portfile with go.vendors

Thanks for creating this. However, when I try to build this locally, it takes quite a long time and then fails to build with some Go build error that I don't understand. I attached k9s-with-go.vendors-failure.txt to this ticket. Anyone know what's wrong here?

Yes, that's what I ​reported earlier.

I think the problem is that some of the dependencies have their own vendored dependencies (in a vendor dir at the dependency's root), and when GO111MODULE=off this is unexpected: instead it's expected that all packages should be flattened into the GOPATH.

I'm not sure what to do about it.

[Aaron Madlon-Kay <amake@…>]

In 3bf12bd32bb5e0c007481c11c9a31dfb687904d5/macports-ports (master):

evans: don't fetch dependencies in build phase

See #61192

[dgsb (David Bariod)]

I was wondering if it is wise to reimplement something that the go compiler already does. I think the go build problem may not be trivial to replicate in macports.

[Aaron Madlon-Kay <amake@…>]

In c74c831164a7309e1b6dd142e9d5799910ffe535/macports-ports (master):

chezmoi: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <amake@…>]

In bc0da04788973104df02dd4267f675015602dcc7/macports-ports (master):

cloudmonkey: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 9c12bfc242d45fbe70c2c538e46018db0e3f9696/macports-ports (master):

certigo: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 2e02c5335d3bd0da951e330bb7450725c83bad62/macports-ports (master):

gitqlite: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In aa1427b26d7ca24dff559d0d51c32bc7fdbb0edf/macports-ports (master):

annie: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <aaron@…>]

In 500aa1d2f53673b781d77ebe95dabcd2faf60803/macports-ports (master):

fzf: don't fetch dependencies in build phase

See #61192

[Aaron Madlon-Kay <amake@…>]

In 25c9ac3c7d7d066eb33dd90c0c26b954e58c1ab2/macports-ports (master):

jenkins-cli: don't fetch dependencies in build phase

See #61192

[breun (Nils Breunese)]

Replying to amake:

I think the problem is that some of the dependencies have their own vendored dependencies (in a vendor dir at the dependency's root), and when GO111MODULE=off this is unexpected: instead it's expected that all packages should be flattened into the GOPATH.

I'm not sure what to do about it.

Neither am I. I maintain these ports because I use these tools myself, but I am not familiar with how Go software gets built. I was already struggling quite a bit to get an initial Portfile for k9s working, and it seems that this go.vendor stuff adds a whole new dimension of complexity. Are there any people familiar enough with Go builds in this thread that could help out?

[Aaron Madlon-Kay <amake@…>]

In b34a811028a6700b822a2c4aeb691f83ce37acd3/macports-ports (master):

syncthing: don't fetch dependencies in build phase

See #61192

[i0ntempest]

I'm submitting cloudflared to macports (​https://github.com/macports/macports-ports/pull/8662), it has two problematic dependencies. One is known to be unsolvable yet (multiple versions of the same dependency, go-oidc v1 and v2), another one seem to be a new problem (giving error: no files matched glob pattern "/opt/local/var/macports/build/_Users_Admin_Ports_net_cloudflared/cloudflared/work/kshvakov-clickhouse-*") However it builds and works anyway without these two dependencies. Please take a look if interested.

[herbygillot (Herby Gillot)]

tektoncd-cli can be marked as fixed since it vendors all its dependencies already. It builds with GOPROXY=off, which has been added as of version 0.13.0:

​https://github.com/macports/macports-ports/commit/cc7c646f38bc9729fbf1763df76abdbdff9d3a03

[herbygillot (Herby Gillot)]

In 93f8502df9e9150e6f055a1a9f6cc06bbb3825c7/macports-ports (master):

ultralist: update to 1.5.1

• vendor Go dependencies, do not download dependencies at build time

See: #61192

[i0ntempest]

Another problematic port: ​https://github.com/macports/macports-ports/pull/8721

[Aaron Madlon-Kay <amake@…>]

In f9eab1b28611e634049daae2d268dc570243ce50/macports-ports (master):

nebula: don't fetch dependencies in build phase (​https://github.com/macports/macports-ports/pull/8810)

See #61192

[Aaron Madlon-Kay <amake@…>]

In ce448c8a13df8398ee867fffc9f2b6ea71a3c6f2/macports-ports (master):

golang-1.0: disable fetching packages during build, test phases

See #61192

[Aaron Madlon-Kay <amake@…>]

In a25e7b41049aeea1f57565369e77a2a8c6fab1ad/macports-ports (master):

Various: temporarily allow fetching deps during build

See #61192

[Aaron Madlon-Kay <amake@…>]

In d79aad1b301412f3ff83438f4e0c23d3c513aeb2/macports-ports (master):

Various: remove superfluous golang envars

These are now set in the golang-1.0 portgroup

See #61192

[herbygillot (Herby Gillot)]

In 0290dd1c8d056701778cb9a48a051eff77eba981/macports-ports (master):

mkcert: update to 1.4.2

• use golang portgroup
• vendor Go dependencies See: #61192

[herbygillot (Herby Gillot)]

In 8ea3b202f9e30e23b0cf4d5f252ca4e579c5daf0/macports-ports (master):

vgrep: update to 2.5.0

• vendor Go dependencies See: #61192

[herbygillot (Herby Gillot)]

In 4270625f2748334ad06ed76f450e48b76b46d1e3/macports-ports (master):

promu: update to 0.7.0

• use the golang portgroup
• vendor Go dependencies to prevent downloading them at build time See: #61192

[herbygillot (Herby Gillot)]

In 62b049b5aa9c1c2e21604fab30e066c8f80643e7/macports-ports (master):

copilot: update to 0.6.0

• add packr and go-mockgen as build dependencies
• vendor Go dependencies, do not download them at build time See: #61192

[emcrisostomo (Enrico Maria Crisostomo)]

I'm trying to update kompose to v. 1.22.0 and I'm hitting this issue: it won't build with GOPROXY=off. Any guidance?

[herbygillot (Herby Gillot)]

In 0f8835630589663ff4b462f987adae644b056333/macports-ports (master):

kubergrunt: update to 0.6.9

• vendor Go dependencies
• Fixes: #61185
• See: #61192

[Aaron Madlon-Kay <amake@…>]

In 85ad5183de0036aae804c59c7d1a71e2f7d00bf5/macports-ports (master):

scw: don't fetch dependencies in build phase

See #61192

[gpanders (Gregory Anders)]

I was looking at upgrading the caddy port, which is now quite old. Unfortunately, it is no longer possible to generate the caddy binary with version information without using go modules.

However, it looks like it is possible to both 1) download module dependencies before the build step (the issue which this ticket addresses) and 2) still use go's new module system. This can be done using the -mod vendor flag to go build.

The only thing this requires is to create a vendor directory that contains all of the vendored dependencies. Basically, instead of moving downloaded dependencies to gopath/src/, simply move them to vendor/. This also requires a file called vendor/modules.txt that contains all of the module dependencies; however, this can be generated once when the port is updated and kept as a patch alongside the Portfile.

[quentinmit (Quentin Smith)]

GO111MODULES=off is deprecated and scheduled to be removed soon: ​https://go.dev/blog/go116-module-changes

Replying to amake:

Replying to breun:

I must admit I do not completely understand what the problem exactly is.

There are two problems:

1. Downloading dependencies outside of MacPorts' fetch mechanism prevents MacPorts from mirroring distfiles. That's bad for a number of reasons, such as: upstream could disappear, or we send a lot of traffic their way putting a strain on their infrastructure.
2. Some build-time dependency fetching schemes lack a locking mechanism, so the exact version of the dependency used can differ depending on when exactly the build was performed. This is bad for reproducibility.

(2) is a core feature of Go's module system; in fact *every* module dependency is pinned to a specific version by the go.mod and go.sum files in the repo. Turning off GO111MODULE actually causes the latest HEAD of each dependency to be used, so setting GO111MODULE=off does the opposite of what you are hoping for.

I don't think there's an easy answer here; go mod download can be used as part of the fetch step to download all the dependencies, but that's still not going to use MacPorts's distfiles mechanism. Honestly I wouldn't worry about the load on the Go module proxies; they're provisioned at least as well as the distfiles network. But if you are worried, probably the best option would be to have the fetch step download module zip files from the module proxy, and prefill the local module cache in the extract step. Then go build will use the module zip files, and the module zip files can be downloaded as distfiles.

You can also use go mod vendor to download all the dependencies as a source tree, but I don't know how to integrate that with distfiles. You would probably need extra magic to generate a tarball of the output of go mod vendor in the distfiles system.

[cooljeanius (Eric Gallager)]

add vfkit to this list, too; see issue #69683