MacPorts sync failed

[Behinder (behinder)]

Hello. yesterday everything was OK. Today I wanted to sync my port tree and the result is:

Willkommen auf dem RSYNC-server auf ftp.fau.de.
Nicht all unsere Mirror sind per rsync verfuegbar.

Welcome to the RSYNC daemon on ftp.fau.de.
Not all of our mirrors are available through rsync.


receiving file list ... rsync: link_stat "/tarballs/base.tar" (in release) failed: No such file or directory (2)
done

sent 4 bytes  received 9 bytes  26.00 bytes/sec
total size is 0  speedup is 0.00
rsync error: some files could not be transferred (code 23) at /BuildRoot/Library/Caches/com.apple.xbs/Sources/rsync/rsync-47/rsync/main.c(1400) [receiver=2.6.9]
Command failed: /usr/bin/rsync -rtzv --delete-after rsync://rsync.macports.org/release/tarballs/base.tar /opt/local/var/macports/sources/rsync.macports.org/release/tarballs
Exit code: 23
Error: Error synchronizing MacPorts sources: command execution failed
To report a bug, follow the instructions in the guide:
    http://guide.macports.org/#project.tickets
Error: /opt/local/bin/port: port selfupdate failed: Error synchronizing MacPorts sources: command execution failed

[raimue (Rainer Müller)]

Please remember to use WikiFormatting.

There was a similar report on IRC. The reporter noted it was working again quickly afterwards. I could not see the failure condition myself, as the tarball existed when I checked. Looks like some temporary glitch happened, maybe the mirror was updated at the same time?

$ date
Mon Oct 17 14:48:40 CEST 2016
$ rsync --no-motd 'rsync://rsync.macports.org/macports/release/tarballs/*.tar*'
-rw-r--r--     27,013,120 2016/10/17 14:01:50 base.tar
-rw-r--r--            512 2016/10/17 14:01:52 base.tar.rmd160
-rw-r--r--     64,327,680 2016/10/17 14:01:51 ports.tar
-rw-r--r--            512 2016/10/17 14:01:52 ports.tar.rmd160

[Behinder (behinder)]

Probably. Now it work. You may close ticket.

[ryandesign (Ryan Carsten Schmidt)]

The buildbot saw a similar problem on the rsync server yesterday and missed a lot of builds as a result.

It seems clear that the script that generates these tarballs and signatures, which is supposed to replace the old tarballs and signatures with the new ones atomically, is not doing so.

I already intend to completely rewrite this script to be triggered from the buildbot instead and will resolve this issue then.

The solution until then is to wait an hour and try again.

[ryandesign (Ryan Carsten Schmidt)]

In 74c01e8/macports-infrastructure:

mprsyncup: Improve generation and deployment of tarballs and portindexes

Only generate and sign base.tar if base changed.

Only generate and sign ports.tar and portindexes if ports changed.

Incorporate functionality of porttree_tarball.sh to generate
ports.tar.gz if ports changed.

No longer try to update files in a temporary directory. It wasn’t
atomic and was causing more problems than it solved, presumably
including: #52629

The new goal is to reduce the amount of time that the files on the
rsync server are in an inconsistent state. Use GNU mv to atomically
replace individual files.

Use hardlinks because it's faster than copying, reduces server disk
space, and hopefully reduces rsync transfer time.