make signed sync/selfupdate faster

[sdavids@…]

dragonfly:~ sdavids$ sw_vers
ProductName:	Mac OS X
ProductVersion:	10.7
BuildVersion:	11A511
dragonfly:~ sdavids$ port version
Version: 2.0.1
dragonfly:~ sdavids$ date && sudo port -dv selfupdate && date
Tue Aug  9 13:50:13 CEST 2011
--->  Updating the ports tree
DEBUG: Synchronizing ports tree(s)
Synchronizing local ports tree from rsync://rsync.macports.org/release/tarballs/ports.tar
DEBUG: /usr/bin/rsync -rtzv --delete-after  rsync://rsync.macports.org/release/tarballs/ports.tar /opt/local/var/macports/sources/rsync.macports.org/release/tarballs
receiving file list ... done

sent 36 bytes  received 70 bytes  14.13 bytes/sec
total size is 45267456  speedup is 427051.47
DEBUG: /usr/bin/rsync -rtzv --delete-after  rsync://rsync.macports.org/release/tarballs/ports.tar.rmd160 /opt/local/var/macports/sources/rsync.macports.org/release/tarballs
receiving file list ... done

sent 36 bytes  received 77 bytes  45.20 bytes/sec
total size is 512  speedup is 4.53
DEBUG: successful verification with key /opt/local/share/macports/macports-pubkey.pem
DEBUG: /usr/bin/tar -C /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/tmp -xf /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports.tar
Creating port index in /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports

Total number of ports parsed:	0 
Ports successfully parsed:	0 
Ports failed:			0 
Up-to-date ports skipped:	8301

DEBUG: MacPorts sources location: /opt/local/var/macports/sources/rsync.macports.org/release/tarballs
--->  Updating MacPorts base sources using rsync
receiving file list ... done

sent 36 bytes  received 69 bytes  30.00 bytes/sec
total size is 3219456  speedup is 30661.49
receiving file list ... done

sent 36 bytes  received 76 bytes  32.00 bytes/sec
total size is 512  speedup is 4.57
DEBUG: successful verification with key /opt/local/share/macports/macports-pubkey.pem
DEBUG: /usr/bin/tar -C /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/tmp -xf /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/base.tar
MacPorts base version 2.0.1 installed,
DEBUG: Rebuilding and reinstalling MacPorts if needed
MacPorts base version 2.0.1 downloaded.
--->  MacPorts base is already the latest version
DEBUG: Setting MacPorts sources ownership to root

The ports tree has been updated. To upgrade your installed ports, you should run
  port upgrade outdated
Tue Aug  9 13:56:18 CEST 2011
dragonfly:~ sdavids$ echo $PATH
/opt/local/bin:/usr/bin:/bin:/opt/local/sbin:/usr/sbin:/sbin:/usr/local/bin:/usr/X11/bin
dragonfly:~ sdavids$ tar --version
bsdtar 2.8.3 - libarchive 2.8.3
dragonfly:~ sdavids$ ls -la /usr/bin/tar
lrwxr-xr-x  1 root  wheel  6 Jul 21 23:45 /usr/bin/tar -> bsdtar
dragonfly:~ sdavids$ otool -hLv /usr/bin/bsdtar
/usr/bin/bsdtar:
Mach header
      magic cputype cpusubtype  caps    filetype ncmds sizeofcmds      flags
MH_MAGIC_64  X86_64        ALL LIB64     EXECUTE    15       2040   NOUNDEFS DYLDLINK TWOLEVEL PIE
	/usr/lib/libarchive.2.dylib (compatibility version 9.0.0, current version 9.2.0)
	time stamp 2 Thu Jan  1 01:00:02 1970
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 159.0.0)
	time stamp 2 Thu Jan  1 01:00:02 1970
dragonfly:~ sdavids$ ls -la /opt/local/var/macports/sources/rsync.macports.org/release/tarballs
total 103816
drwxr-xr-x  10 root  admin       340 Aug  9 13:56 .
drwxr-xr-x   3 root  admin       102 Jul 25 15:54 ..
-rw-r--r--   1 root  admin   4649199 Aug  9 09:00 PortIndex
-rw-r--r--   1 root  admin       512 Aug  9 13:32 PortIndex.rmd160
drwxr-xr-x  24 root  wheel       816 Aug  2 07:30 base
-rw-r--r--   1 root  admin   3219456 Aug  9 13:32 base.tar
-rw-r--r--   1 root  admin       512 Aug  9 13:32 base.tar.rmd160
drwxr-xr-x  51 root  wheel      1734 Aug  9 13:56 ports
-rw-r--r--   1 root  admin  45267456 Aug  9 13:32 ports.tar
-rw-r--r--   1 root  admin       512 Aug  9 13:32 ports.tar.rmd160

The bulk of the time (~ 5 minutes) is between these two lines:

DEBUG: /usr/bin/tar -C /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/tmp -xf /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports.tar
Creating port index in /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports

It appears as if not only the delta is extracted but always the entire posts tar.

[jmroot (Joshua Root)]

How exactly would you go about making it so that "only the delta is extracted"?

[afb@…]

Wonder why extracting ports.tar takes 5 minutes ? Should be more like 5 seconds, for about 43M.

But you could always go back to the previous sources.conf line, without the trailing ports.tar ?

[jmroot (Joshua Root)]

Indeed. Selfupdate takes barely a minute on my old PPC machine FTR. Patches welcome.

[mojca (Mojca Miklavec)]

I don't know how exactly the system works, but why not using something like

rsync -av --delete rsync://rsync.macports.org/release/ports/ /opt/local/var/macports/sources/rsync.macports.org/release/tarballs/ports/

as opposed to fetching the whole tar file? On my computer it takes 17s. The selfupdate take 44s, but it probably also does a bit more. (OK, there could be a problem in case that the second folder is not set up properly since the complete content might be deleted. I'm not sure how macports currently deals with deleting files if user does something stupid.)

[neverpanic (Clemens Lang)]

The reason the tarball is synced rather than the files themselves is that the tarball is cryptographically signed with a detached signature.

Since the tarball does not use any compression, rsync should be able to copy only the deltas, i.e. syncing via tarball shouldn't be that much slower than syncing the files directly, especially since you should have less file operations when syncing the tarball (one file vs. a couple thousand).

Switching to the non-tarball sync method will automatically disable signature verification, i.e. you can no longer be sure the portfiles haven't been tampered with along the way, so I'd suggest not going this way.

So the correct thing to do here is speed up the extraction of the tarball, if possible.

[neverpanic (Clemens Lang)]

Cannot reproduce and I don't see a problem with the way we currently do things. We could certainly speed things up, but probably by using something like git to sync instead.