Checksum mismatches for generated tarballs from GitHub

[raimue (Rainer Müller)]

Apparently, GitHub upgraded their software and changed the way git tarballs are generated due to a ​bugfix in upstream git. The produced tarballs will differ from those that have been generated before.

The checksums of the following ports have changed due to this (ran against [ffdb5922289aca17f166d14e3e91ecbe52278c10/macports-ports]):

Port	Maintainer
ALPSCore	galexv@…
apbs	howarth.at.macports@…,openmaintainer
box2d	stromnov@…,openmaintainer
certbot	mps@…,openmaintainer
cgminer
chapel	takeshi@…,openmaintainer
cjdns	l2dy@…,openmaintainer
coffee-script	ryandesign@…,openmaintainer
cot	glenn@…
csvToLedger	ben@…
cusp	sean@…,openmaintainer
DepreciateForLedger	ben@…
docker
docker-machine
elixir	me@… milmazz
expense.txt	ben@…
faust-devel	ryandesign@…,aggraef@…
faust2-devel	ryandesign@…,aggraef@…
fbthrift	sean@…,openmaintainer
fbthrift	sean@…,openmaintainer
git-lfs	raimue@…,openmaintainer
gst-plugins-cairovis	aronnax@… lpsinger
GTDtoCSV	ben@…
Gyoto	thibaut@…,openmaintainer
HermesApp	ryandesign@… openmaintainer
htcondor	aronnax@… lpsinger
hub	lists@…
ifeffit	joe.fowler@…,openmaintainer
Io
iTerm2	emer@…
jake	ryandesign@…,openmaintainer
just	casey@…
keybase	sean@…,openmaintainer
keybinder	ryandesign@…,openmaintainer
keybinder-3.0	ryandesign@…,openmaintainer
kops	pebrc,openmaintainer
LedgerScheduler	ben@…
libcdata
libcdatetime
libcerror
libcfile
libclocale
libcnotify
libcsplit
libcthreads
libewf
libgit2	juanrgar@…,devans@…
libiodbc	snc@…,openmaintainer
libphonenumber-cpp	devans@…,openmaintainer
litecoin	mk@…,openmaintainer
lua-cgilua
lua-rings
lua-wsapi
lua-xavante
LucenePlusPlus
mame	mascguy@…,openmaintainer
meson
mogenerator
mongo-cxx-driver	ryandesign@…,openmaintainer
mongo-tools	ryandesign@…,openmaintainer
MongoHub	ryandesign@…,openmaintainer
mongrel2
mosesdecoder
myrepos	gjasny@…,openmaintainer
mythtv-core.25	ctreleaven@…,openmaintainer
mythtv-core.26	ctreleaven@…,openmaintainer
nu2
ocaml-async-core
OpenCoarrays	fanfarillo.gcc@…
pear-TCPDF
pficommon
phoenix	g5pw@…,openmaintainer
phpsh
povray
QGIS	vince@…
QLStephen
qoauth
racer	sean@…,openmaintainer
rekall
root5	jonesc@…
scm_breeze	g5pw@…,openmaintainer
ShiftIt
subsurface
syncthing-0.14	leonardo@…,openmaintainer
synergy
tDOM	neumann@…
tbb	sean@…
teleport	graemelawes@…,openmaintainer
texworks	mojca@…,openmaintainer
tidyp
time.txt	ben@…
todotxt	snc@…,openmaintainer
tpkg	jheiss@…,openmaintainer
uzbl	ryandesign@…,openmaintainer
webkit-sharp	ryandesign@…,openmaintainer
xctool	sean@…,openmaintainer
xmlroff
zorba

I do not know if GitHub actually invalidated the generated tarballs from their download caches, so the following might not be conclusive and checksums that are deemed correct might be subject to change in the future.

To track the progress, please do not edit above table as-is but remove fixed ports from the port field in this ticket.

See also this similar report for Homebrew: ​https://github.com/Homebrew/homebrew-core/issues/18044

For reference, above list was generated with these commands:

$ grep -l -R '^github\.setup' . | xargs -I{} sh -c 'basename $(dirname {})' | while read port; do sudo port clean --all \; fetch --no-mirrors \ checksum || echo $port > github-failed.txt; done
$ while read port; do port info --line --name --maintainer $port; done < <(sort -f github-failed.txt) | awk -F$'\t' '{printf "|| %-40s || %s ||\n", $2, $1}'

[raimue (Rainer Müller)]

In order to fix these ports, please apply the recipe from wiki:PortfileRecipes#stealth-updates by adding a dist_subdir.

[raimue (Rainer Müller)]

In 73194e3f3ba6e6435f28a4de68060cb7c98827d5/macports-ports:

git-lfs: Fix checksums for stealth distfile update

See: #54839

[nerdling (Jeremy Lavergne)]

I did get a changed distfile for todotxt (07842f8/macports-ports), but libiodbc seems to still be correct.

[raimue (Rainer Müller)]

Replying to nerdling:

but libiodbc seems to still be correct.

You are right, I only checked if the command was successful and the problem with libiodbc was unrelated and temporary (502 Bad Gateway). I noticed now that the list also contains some other errors such as 404 Not Found for some ports. However, they need to be looked into anyway, so I am not re-running the whole thing.

[Schamschula (Marius Schamschula)]

In 8de513662f8647017c9f850c0146606ed9121154/macports-ports:

certbot: revision bump got stealth update 2 GitHub

See: #54839

[lbschenkel (Leonardo Brondani Schenkel)]

There's a new syncthing release scheduled to happen in 3 days; since installing the pre-built binaries still works and this port has no variants that would force building from source, I think that the number of impacted users is on the low side and we can afford to wait so I can kill two birds with one stone. Sounds reasonable?

[raimue (Rainer Müller)]

Users of a non-default prefix will not receive binary archives, although I agree there are probably not many. Fixing the checksums will not affect existing installations in any way. We can assume the content of the distfiles has not changed as they are still based on the same git commit. There is no need to increase the revision when fixing the checksums and it will not cause any rebuilds.

[mojca (Mojca Miklavec)]

I planned to seize the opportunity to update texworks to 0.6.2 to avoid doing workarounds, but I cannot make it compile. I'll see if I can fix the problems quickly. If not, I'll do a workaround.

[l2dy (Zero King)]

In d7d06e889adaad74b8bd92d356d0bcd7f9aea989/macports-ports:

cjdns: Fix checksums for stealth distfile update

See: #54839

[Schamschula (Marius Schamschula)]

In 14721aee908d3a7a98becf1e6119d7c7893aebce/macports-ports:

iTerm2: update to version 3.0.15 (maintainer timeout)

Closes: #54806
also fixes GitHub stealth update:
See: #54839

[lbschenkel (Leonardo Brondani Schenkel)]

syncthing-0.14 has now been upgraded to a new upstream version, that also addresses the checksum problem

[joefowler (Joe Fowler)]

See #54890: I tried to follow the instructions and update port ifeffit. But as @ryandesign has politely pointed out, I did not actually have to change the checksums.

So can you check again that ifeffit belongs on this list? If it does, then I guess I am totally confused about what to do.

[raimue (Rainer Müller)]

I can confirm ifeffit was a false positive, follow-up in #54890.

[Dan R. K. Ports <drkp@…>]

In 7c281354f28c9ddbcdb19dddec135e7df2f030ff/macports-ports:

osxfuse: set dist_subdir to prevent github-caused checksum mismatch

See #54839
Closes: #54892
Closes: #54898

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #54910.

[cjones051073 (Chris Jones)]

tbb fixed (also updated to 2017_U8).

[bokertov]

Any update on when this will get fixed? This is preventing me from install packages such as Julia (see #55117). Thanks!

[raimue (Rainer Müller)]

As soon as you provide updated checksums. It just needs someone to work through these ports and update the checksums.

[lbschenkel (Leonardo Brondani Schenkel)]

Btw: syncthing-0.14 has been updated a while ago, I forgot to notify in this ticket.

[michaellass (Michael Lass)]

At least for libgit2 I created a pull request some time ago which just needs to be merged: ​https://github.com/macports/macports-ports/pull/900

[Michael Lass <michael.lass@…>]

In 1f90ea9f36086e27cbbf90ba36652d1202ea8c7d/macports-ports:

libgit2: set dist_subdir and update checksum

See #54839

[ctreleaven (Craig Treleaven)]

In aabf04dc178a3c445472383a47446ffecb4d7aab/macports-ports:

mythtv-core.28: update to fixes as of 2017-10-10

see #54839

[ctreleaven (Craig Treleaven)]

mythtv-core.28 was not on the list but was affected by this issue. Now fixed.

Will sudo port distcheck blah verify if a port suffers from this issue? Because I would expect mythweb.28 to be affected and it appears not to be.

[neverpanic (Clemens Lang)]

In e6a8d60bb52bcba99ddb01fc6470a7f9a9ce5ac9/macports-ports:

keybase: update to 1.0.33, openmaintainer

Fixes checksum failure due to GitHub regenerating their tarballs.

Closes: #54807
See: #54839

[mojca (Mojca Miklavec)]

In c95b459817df041b5281fc14ae9ce65e4bfd659c/macports-ports:

texworks: update to 0.6.2

See: #54839

[Alexander Gaenko <galexv@…>]

In e2515ef1caa7184e3e0fd46eb8ee30833488d0f0/macports-ports:

ALPSCore: update to 1.0.0

A new release. Also should resolve the tarball checksum mismatch.
See: #54839

[ctreleaven (Craig Treleaven)]

In 0185d8ff5ae99fbc3454163b59b8c65e72e21410/macports-ports:

myth*.27: bitrot fixes

mythtv-core.27, mythtv-plugins.27, mythweb.27 -

Update to most recent commit in Fixes/0.27 branch for each
which fixes mangled checksums problem (54839)
Update to use Python 5.2.6 (55208)
Change MythWeb paths to account for recent changes to Apache2

See #54839
See #55208

[ctreleaven (Craig Treleaven)]

Ports now obsolete and therefore checksums no longer relevant:

• mythtv-core.25: [14829fec8a6e4d888910197a0f47fff10b17032f/macports-ports]
• mythtv-core.26: [056d04d825a60266f409e0c94d89c48e627389c4/macports-ports]
• mythweb.25: [98b164f39ceeb9f7ebd2ff0866defde9fc74be5a/macports-ports]
• mythweb.26: [01c9b70862011ca711cda1bd3a298e29cd23a5d9/macports-ports]

[ryandesign (Ryan Carsten Schmidt)]

Has duplicate #55412 (docker).

[ryandesign (Ryan Carsten Schmidt)]

In 8449249ddc3ab5f608c3e3a5ecdcd19504ff68b2/macports-ports:

docker: Update to 17.05.0-ce

See: #54839

[mojca (Mojca Miklavec)]

root5 looks ok to me.

[lpsinger (Leo Singer)]

I fixed htcondor in [f52105c01a0a2905b62d04d9fb98b3bb4e418553/macports-ports].

[Eric A. Borisch <eborisch@…>]

In acca590955b08c0922fbfb3dc43855f38fceeb50/macports-ports:

rclone: Fix checksum & upgrade to 1.3.8

See also #54839 and #54959

[mf2k (Frank Schima)]

In 12bb6c1b2ff2a548677f23080b115c8e3ab46796/macports-ports:

mame: Update to version 0.193

• Explicitly link with Macports libsdl2
• Add size to checksums

Fixes: #55667
See: #54839

[mojca (Mojca Miklavec)]

Short testings seems to suggest that a number of ports were fixed in the meantime. Likely fixed:

• DepreciateForLedger
• GTDtoCSV
• LedgerScheduler
• elixir
• expense.txt
• for libcomerr
• for faust-devel
• faust2-devel
• gst-plugins-cairovis
• keybinder-3.0
• lua-cgilua
• lua-rings
• lua-wsapi
• mame
• mongo-cxx-driver
• racer
• xctool
• tDOM
• teleport
• tidyp
• tpkg

Still broken:

• Gyoto
• HermesApp
• Io
• LucenePlusPlus
• MongoHub
• OpenCoarrays
• QLStephen
• ShiftIt
• apbs
• box2d
• cgminer
• chapel
• coffee-script
• cot
• cusp
• fbthrift
• hub
• jake
• just
• keybinder
• kops
• libcdata
• libcdatetime
• libcerror
• libcfile
• libclocale
• libcnotify
• libcsplit
• libcthreads
• libewf
• litecoin
• lua-xavante
• mogenerator
• mongrel2
• mosesdecoder
• myrepos
• ocaml-async-core
• pear-TCPDF
• phoenix
• povray
• qoauth
• rekall
• scm_breeze
• subsurface
• uzbl
• webkit-sharp
• xmlroff
• zorba

I would prefer a second pair of eyes before removing the ports from the list though.

[raimue (Rainer Müller)]

coffee-script fixed in [9283464eae694e7567a008210fa5d000616559bf/macports-ports].

[pturley0]

kops still broken.

:notice:checksum --->  Verifying checksums for kops
:debug:checksum Executing org.macports.checksum (kops)
:info:checksum --->  Checksumming kops-1.5.3.tar.gz
:debug:checksum Calculated (rmd160) is ee2308247148138ea30b74225831f2856860aefd
:error:checksum Checksum (rmd160) mismatch for kops-1.5.3.tar.gz
:info:checksum Portfile checksum: kops-1.5.3.tar.gz rmd160 714ad9177149dda1d8e01ffe61d2c398ef1c8ece
:info:checksum Distfile checksum: kops-1.5.3.tar.gz rmd160 ee2308247148138ea30b74225831f2856860aefd
:debug:checksum Calculated (sha256) is 9ca94df33d8a1fbfcbcb00df65c4ca82238ff1f0970b8d4a83e93c748d4160f8
:error:checksum Checksum (sha256) mismatch for kops-1.5.3.tar.gz
:info:checksum Portfile checksum: kops-1.5.3.tar.gz sha256 a49c94d6bc1296f5dc7662547e379c4811c931b7dc4638cafce9ea3778012175
:info:checksum Distfile checksum: kops-1.5.3.tar.gz sha256 9ca94df33d8a1fbfcbcb00df65c4ca82238ff1f0970b8d4a83e93c748d4160f8
:info:checksum The correct checksum line may be:
:info:checksum checksums           rmd160  ee2308247148138ea30b74225831f2856860aefd \
:info:checksum                     sha256  9ca94df33d8a1fbfcbcb00df65c4ca82238ff1f0970b8d4a83e93c748d4160f8 \
:info:checksum                     size    50897087
:error:checksum Failed to checksum kops: Unable to verify file checksums
:debug:checksum Error code: NONE
:debug:checksum Backtrace: Unable to verify file checksums
:debug:checksum     while executing
:debug:checksum "$procedure $targetname"

[mf2k (Frank Schima)]

Near as I can tell, only the following remain:

port	maintainer	status
cgminer		changeset:7333cbd46d5bbb0171c865f92adee3a6848561b0/macports-ports
chapel	@tenomoto,openmaintainer
jake	ryandesign,openmaintainer
kops	pebrc,openmaintainer
uzbl	@ryandesign,openmaintainer

[mf2k (Frank Schima)]

In 7333cbd46d5bbb0171c865f92adee3a6848561b0/macports-ports (master):

cgminer: Fix checksums

• Add size

See: #54839

[mojca (Mojca Miklavec)]

In ad1ab4cea369c49306925663caf5dbbcad15f28b/macports-ports (master):

FScript: switch to GitHub PortGroup

See: #44345
See: #54839

[mf2k (Frank Schima)]

In 91016f1660b1ffe02d30bb36ae839dbc1e79f911/macports-ports (master):

povray: Fix checksums

Fixes: #57696
See: #54839

[myan18]

Unfortunately I still have no clue how to get Povray installed. I read PortfileRecipes#stealth-updates but am not even sure if the instruction is written for developers or person who installs. Where is a Portfile located by the way? Best, Max

[lpsinger (Leo Singer)]

In 1d92100a330248d7e7f352764f9e48eedc22be9d/macports-ports (master):

gst-plugins-cairovis: fix checksum due to GitHub changes

See #54839

[lpsinger (Leo Singer)]

I fixed gst-plugins-cairovis in 1d92100a330248d7e7f352764f9e48eedc22be9d/macports-ports.

[ryandesign (Ryan Carsten Schmidt)]

Replying to lpsinger:

I fixed gst-plugins-cairovis in 1d92100a330248d7e7f352764f9e48eedc22be9d/macports-ports.

gst-plugins-cairovis was not affected by this ticket. This ticket is only about automatically-generated tarballs, some of which changed because GitHub changed the method by which they were generated. gst-plugins-cairovis did not use an automatically-generated tarball; it used a manually-generated download that was provided by its developer (which I see is you), and there's no reason why its checksum should change unless the developer replaced the file with a different one, which has not happened for this port.

[lpsinger (Leo Singer)]

Replying to ryandesign:

Replying to lpsinger:

I fixed gst-plugins-cairovis in 1d92100a330248d7e7f352764f9e48eedc22be9d/macports-ports.

gst-plugins-cairovis was not affected by this ticket. This ticket is only about automatically-generated tarballs, some of which changed because GitHub changed the method by which they were generated. gst-plugins-cairovis did not use an automatically-generated tarball; it used a manually-generated download that was provided by its developer (which I see is you), and there's no reason why its checksum should change unless the developer replaced the file with a different one, which has not happened for this port.

Well, in any case, I switched to an automatically generated tarball, so the stealth update workaround was needed anyway.

[ryandesign (Ryan Carsten Schmidt)]

Replying to myan18:

Unfortunately I still have no clue how to get Povray installed. I read PortfileRecipes#stealth-updates but am not even sure if the instruction is written for developers or person who installs.

PortfileRecipes are suggestions for portfile authors, not users.

Frank has followed those instructions for the povray port, fixing it; all you need to do to receive the fix, as with any other MacPorts fix, is to run sudo port selfupdate and try to install the port again.

Where is a Portfile located by the way? Best, Max

The command port file povray will show you where povray's Portfile is located.

[chrstphrchvz (Christopher Chavez)]

In f4ad27cdaa7d34828b602236cbb86730afd065f1/macports-ports (master):

tpkg: delete port

Upstream project is dormant. A port abandoned ticket was opened;
the author of tpkg, who also used to maintain this port,
responded suggesting that this port be deleted.

Closes: #47985
See: #54839
See: #58399

[pmetzger (Perry E. Metzger)]

In a161124315ae59e99481271f6b087b4e42b3d527/macports-ports (master):

Remove PHP PEAR modules and PortGroup

Per ​https://github.com/macports/macports-ports/pull/4192, which after
nine months received no objections. Manually committed because of
merge conflicts.

Closes: #56424
Closes: #34063
Closes: #35228
Closes: #37119
Closes: #37524
Closes: #37683
Closes: #38910
Closes: #39869
Closes: #46688
Closes: #53510
Closes: #53632
Closes: #57316
Closes: #57318
Closes: #57320
Closes: #57321
See: #54839

[ryandesign (Ryan Carsten Schmidt)]

In 254eb356c904a34e95679eb341530a653ba7ac1b/macports-ports (master):

keybinder: Fix checksum mismatch due to new owner

Switch to from tag to archive tarball to avoid this problem in the
future.

Closes: #60585
See: #54839

[Dave-Allured (Dave Allured)]

Please remove OpenCoarrays from the port field on this ticket. I do not have change privilege for this ticket.

Checksum for this port was fixed here: ​https://github.com/macports/macports-ports/commit/a478ca5577be40732ff328f3856f28f0a7ff4dba

[mascguy (Christopher Nielsen)]

I'm guessing most of these have been fixed already, apart from a few older ports that haven't been updated in years.

In terms of ports that haven't been updated, the libcXXX ports are likely still affected, given that they haven't been touched in years. Though @barracuda156 has pending PRs for those in particular.

[mascguy (Christopher Nielsen)]

libc* ports fixed:

​15518 - update libc ports: libcdata, libcdatetime, libcerror, libcfile, libclocale, libcnotify, libcsplit, libcthreads

[mascguy (Christopher Nielsen)]

Removed deleted ports from list: cot faust2-devel ocaml-async-core pear-TCPDF tpkg webkit-sharp

[Christopher Nielsen <mascguy@…>]

In dc167183037b86a6e7ec50a3c7e3246b6931df85/macports-ports (master):

qoauth: stealth update: fix checksums

• Also eliminate defunct master site, and source from github

See: #54839

[mascguy (Christopher Nielsen)]

Verified checksums for all of these, and only a few stragglers remain:

--->  Verifying checksums for HermesApp
Error: Checksum (rmd160) mismatch for Hermes-1.3.1.tar.gz
--->  Verifying checksums for LucenePlusPlus
Error: Checksum (rmd160) mismatch for LucenePlusPlus-3.0.7.tar.gz
--->  Verifying checksums for MongoHub
Error: Checksum (rmd160) mismatch for MongoHub-Mac-2.7b10.tar.gz
--->  Verifying checksums for apbs
Error: Checksum (rmd160) mismatch for apbs-pdb2pqr-d4e78c62e6a07f92c26924318d83a159fa6af063.tar.gz
--->  Verifying checksums for cusp
Error: Checksum (rmd160) mismatch for cusplibrary-0.5.1.tar.gz
--->  Verifying checksums for lua-xavante
Error: Checksum (md5) mismatch for xavante-2.2.1.tar.gz
--->  Verifying checksums for mosesdecoder
Error: Checksum (rmd160) mismatch for mosesdecoder-f4471692ba51aeec3cf9867fcabe2c4fc7d4cb33.tar.gz
--->  Verifying checksums for scm_breeze
Error: Checksum (rmd160) mismatch for scm_breeze-1.0.0.tar.gz
--->  Verifying checksums for subsurface
Error: Checksum (rmd160) mismatch for subsurface-3.1.1.tar.gz
--->  Verifying checksums for zorba
Error: Checksum (rmd160) mismatch for zorba-3.1.tar.gz

[Christopher Nielsen <mascguy@…>]

In 8ca873b2ada410a884ba643bc8f23317fb77204c/macports-ports (master):

ports: stealth update; fix checksums
Fixes: #54839