Opened 11 years ago
Closed 11 years ago
#39652 closed defect (fixed)
Base/ports tarballs and PortIndexes are not being updated
Reported by: | mndavidoff (Monte Davidoff) | Owned by: | jmroot (Joshua Root) |
---|---|---|---|
Priority: | High | Milestone: | |
Component: | server/hosting | Version: | 2.1.3 |
Keywords: | Cc: | wsiegrist@…, neverpanic (Clemens Lang), su-v, b.forbort@…, MarcusCalhoun-Lopez (Marcus Calhoun-Lopez), larryv (Lawrence Velázquez), jdgleeson, drkp (Dan Ports), mail@…, nonstop.server@…, skymoo (Adam Mercer), jwhowse4, stromnov (Andrey Stromnov), mojca (Mojca Miklavec), cjones051073 (Chris Jones) | |
Port: |
Description
It appears port sync
is not downloading an up-to-date ports.tar. It seems to be stuck downloading a ports.tar that's currently about 20 hours old.
$ date Sat Jul 6 10:45:29 PDT 2013 $ sudo port sync ---> Updating the ports tree $ ls -l /opt/local/var/macports/sources/rsync.macports.org/release/tarballs total 130176 -rw-r--r-- 1 root admin 9759688 Jul 5 14:31 PortIndex -rw-r--r-- 1 root admin 512 Jun 16 10:31 PortIndex.rmd160 drwxr-xr-x 24 root wheel 816 Jan 31 09:30 base -rw-r--r-- 1 root admin 3594240 Jul 5 15:03 base.tar -rw-r--r-- 1 root admin 512 Jul 5 15:03 base.tar.rmd160 drwxr-xr-x 51 root wheel 1734 Jul 6 10:46 ports -rw-r--r-- 1 root admin 53278720 Jul 5 15:03 ports.tar -rw-r--r-- 1 root admin 512 Jul 5 15:03 ports.tar.rmd160 $ date Sat Jul 6 10:46:25 PDT 2013
Change History (36)
comment:1 Changed 11 years ago by cooljeanius (Eric Gallager)
comment:2 Changed 11 years ago by neverpanic (Clemens Lang)
Cc: | cal@… added |
---|---|
Component: | ports → server/hosting |
Owner: | changed from macports-tickets@… to wsiegrist@… |
Priority: | Normal → High |
I was actually just writing a mail for an issue originally reported in IRC. I think this is best assigned to our admins.
comment:4 Changed 11 years ago by larryv (Lawrence Velázquez)
Summary: | port sync downloads out-of-date ports.tar → rsync tarballs and PortIndexes are not being updated |
---|
It’s not just ports.tar
.
% TZ=UTC rsync rsync://rsync.macports.org/release/tarballs/ drwxr-xr-x 4096 2012/07/20 20:37:09 . -rw-r--r-- 3594240 2013/07/05 22:03:03 base.tar -rw-r--r-- 512 2013/07/05 22:03:05 base.tar.rmd160 -rw-r--r-- 53278720 2013/07/05 22:03:04 ports.tar -rw-r--r-- 512 2013/07/05 22:03:05 ports.tar.rmd160 drwxr-xr-x 4096 2013/07/05 22:02:16 PortIndex_darwin_10_i386 drwxr-xr-x 4096 2013/07/05 22:02:19 PortIndex_darwin_11_i386 drwxr-xr-x 4096 2013/07/05 22:02:21 PortIndex_darwin_12_i386 drwxr-xr-x 4096 2013/07/05 22:02:09 PortIndex_darwin_8_i386 drwxr-xr-x 4096 2013/07/05 22:02:06 PortIndex_darwin_8_powerpc drwxr-xr-x 4096 2013/07/05 22:02:14 PortIndex_darwin_9_i386 drwxr-xr-x 4096 2013/07/05 22:02:11 PortIndex_darwin_9_powerpc %
comment:6 Changed 11 years ago by MarcusCalhoun-Lopez (Marcus Calhoun-Lopez)
Cc: | mcalhoun@… added |
---|
Cc Me!
comment:8 follow-up: 10 Changed 11 years ago by larryv (Lawrence Velázquez)
Summary: | rsync tarballs and PortIndexes are not being updated → Base/ports tarballs and PortIndexes are not being updated |
---|
As John just noted, the daily tarball is out of date, too. Which makes sense, since it’s generated by the rsync server.
% curl -s https://distfiles.macports.org/ports.tar.gz | TZ=UTC pax -vzT 201307050000 | cut -c 43- | sort -r | head -n 20 Jul 5 22:02 ports/PortIndex_darwin_9_powerpc/PortIndex.quick Jul 5 22:02 ports/PortIndex_darwin_9_powerpc Jul 5 22:02 ports/PortIndex_darwin_9_i386/PortIndex.quick Jul 5 22:02 ports/PortIndex_darwin_9_i386 Jul 5 22:02 ports/PortIndex_darwin_8_powerpc/PortIndex.quick Jul 5 22:02 ports/PortIndex_darwin_8_powerpc Jul 5 22:02 ports/PortIndex_darwin_8_i386/PortIndex.quick Jul 5 22:02 ports/PortIndex_darwin_8_i386 Jul 5 22:02 ports/PortIndex_darwin_12_i386/PortIndex.quick Jul 5 22:02 ports/PortIndex_darwin_12_i386 Jul 5 22:02 ports/PortIndex_darwin_11_i386/PortIndex.quick Jul 5 22:02 ports/PortIndex_darwin_11_i386 Jul 5 22:02 ports/PortIndex_darwin_10_i386/PortIndex.quick Jul 5 22:02 ports/PortIndex_darwin_10_i386 Jul 5 22:02 ports/PortIndex.quick Jul 5 22:02 ports/PortIndex Jul 5 21:31 ports/devel/qt4-creator-mac/Portfile Jul 5 21:31 ports/devel/qt4-creator-mac Jul 5 21:31 ports/PortIndex_darwin_9_powerpc/PortIndex Jul 5 21:31 ports/PortIndex_darwin_9_i386/PortIndex %
Does anyone have any idea what’s going on here? It’s been the better part of a week now.
comment:10 Changed 11 years ago by jmroot (Joshua Root)
Cc: | wsiegrist@… added |
---|---|
Owner: | changed from wsiegrist@… to admin@… |
Replying to larryv@…:
Does anyone have any idea what’s going on here? It’s been the better part of a week now.
Presumably the job is not being run or is stuck for some reason. I don't know why Bill hasn't been able to respond.
comment:11 Changed 11 years ago by drkp (Dan Ports)
Cc: | dports@… added |
---|
The distfiles mirror has also not been updating.
comment:14 Changed 11 years ago by wsiegrist@…
I'm investigating. Sorry for the delays, it is a combination of some vacation and hectic work loads.
comment:15 Changed 11 years ago by wsiegrist@…
Owner: | changed from admin@… to cal@… |
---|
The sync script builds MacPorts first, and [107864] broke the build on Linux.
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic -DHAVE_CONFIG_H -I.. -I. -I"/usr/include" -fPIC rmd160cmd.c -o rmd160cmd.o gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic -DHAVE_CONFIG_H -I.. -I. -I"/usr/include" -fPIC sha256cmd.c -o sha256cmd.o gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic -DHAVE_CONFIG_H -I.. -I. -I"/usr/include" -fPIC readline.c -o readline.o gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic -DHAVE_CONFIG_H -I.. -I. -I"/usr/include" -fPIC uid.c -o uid.o gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic -DHAVE_CONFIG_H -I.. -I. -I"/usr/include" -fPIC tracelib.c -o tracelib.o tracelib.c:49:23: error: sys/event.h: No such file or directory tracelib.c: In function ‘send_file_map’: tracelib.c:340: warning: implicit declaration of function ‘strlcat’ tracelib.c: In function ‘TracelibOpenSocketCmd’: tracelib.c:517: error: ‘struct sockaddr_un’ has no member named ‘sun_len’ tracelib.c:519: error: ‘struct sockaddr_un’ has no member named ‘sun_len’ tracelib.c: At top level: tracelib.c:541: error: array type has incomplete element type tracelib.c: In function ‘TracelibRunCmd’: tracelib.c:543: error: storage size of ‘kev’ isn’t known tracelib.c:548: warning: implicit declaration of function ‘kqueue’ tracelib.c:572: warning: implicit declaration of function ‘EV_SET’ tracelib.c:572: error: ‘EVFILT_READ’ undeclared (first use in this function) tracelib.c:572: error: (Each undeclared identifier is reported only once tracelib.c:572: error: for each function it appears in.) tracelib.c:572: error: ‘EV_ADD’ undeclared (first use in this function) tracelib.c:572: error: ‘EV_RECEIPT’ undeclared (first use in this function) tracelib.c:573: warning: implicit declaration of function ‘kevent’ tracelib.c:583: error: ‘EV_ERROR’ undeclared (first use in this function) tracelib.c:692: error: ‘EV_EOF’ undeclared (first use in this function) tracelib.c:543: warning: unused variable ‘kev’ make[2]: *** [tracelib.o] Error 1 make[2]: Leaving directory `/var/macports/trunk/base/src/pextlib1.0' make[1]: *** [all] Error 1 make[1]: Leaving directory `/var/macports/trunk/base/src' make: *** [all] Error 1
comment:16 Changed 11 years ago by wsiegrist@…
Actually, the time of the change and this bug don't support my theory. So maybe its not that change that started the problem, but that is where the sync script ends currently.
comment:17 Changed 11 years ago by jmroot (Joshua Root)
How is it after the latest changes to trunk? And can we add a Linux buildslave for base with the same environment as the server?
comment:20 follow-up: 21 Changed 11 years ago by wsiegrist@…
Yes, we can get a linux slave going, I assume that would be base only?
Here is the latest error:
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic -DHAVE_CONFIG_H -I.. -I. -I"/usr/include" -fPIC strlcat.c -o strlcat.o strlcat.c:23: error: expected ‘=’, ‘,’, ‘;’, ‘asm’ or ‘__attribute__’ before ‘strlcat’ make[2]: *** [strlcat.o] Error 1 make[2]: Leaving directory `/var/macports/trunk/base/src/pextlib1.0' make[1]: *** [all] Error 1 make[1]: Leaving directory `/var/macports/trunk/base/src' make: *** [all] Error 1
comment:21 follow-up: 23 Changed 11 years ago by larryv (Lawrence Velázquez)
Replying to wsiegrist@…:
Yes, we can get a linux slave going, I assume that would be base only?
It wouldn’t be a bad idea to have a ports buildslave also, for debugging issues with Portfile parsing and so forth.
comment:23 follow-up: 24 Changed 11 years ago by jmroot (Joshua Root)
Replying to larryv@…:
It wouldn’t be a bad idea to have a ports buildslave also, for debugging issues with Portfile parsing and so forth.
We don't want a ports buildslave like the others running on Linux, since a ton of stuff probably doesn't build. A new builder type that generates a PortIndex would be OK. Let's just go with base for now since that's really easy to set up.
comment:24 Changed 11 years ago by cooljeanius (Eric Gallager)
Replying to jmr@…:
Replying to larryv@…:
It wouldn’t be a bad idea to have a ports buildslave also, for debugging issues with Portfile parsing and so forth.
We don't want a ports buildslave like the others running on Linux, since a ton of stuff probably doesn't build.
How about one that just tries to build all the ports that include linux
in their platform
field in their portfiles?
Edit: That happens to be the following ports:
Local-Admins-MacBook-Pro:RubyMotion ericgallager$ port echo platforms:linux bodr btrfs-progs bzip2 cbmbasic chemical-mime-data fizmo gettext gnupg12 go gtk-doc hexedit libiconv libmcs libpng logtalk logtalk-devel makepak monit mutt netpbm openbox perl5 perl5.8 perl5.10 perl5.12 perl5.14 perl5.16 poppler processing py-pylibacl py-xattr py24-xattr py25-pylibacl py25-xattr py26-pylibacl py26-xattr py27-pylibacl py27-xattr python24-doc rox-all rox-filer rpm rpm2cpio rpm45 rpm50 rpm51 rpm52 rpm53 rpm54 tf tom xar-devel xdg-utils zeroinstall-injector
That's just 54 ports so far, so that shouldn't be too many.
comment:26 Changed 11 years ago by jmroot (Joshua Root)
Bill, could you please incorporate r108145, r108146, and r108147 into the copy of mprsyncup on the server? That should fix the immediate problem. Please also delete the base checkouts for both trunk and release (should be /var/tmp/macports/trunk/base
and /var/tmp/macports/release/base
) and let it check out fresh copies, since there are some dylibs in there that we don't want on the rsync server.
comment:29 Changed 11 years ago by wsiegrist@…
Next error:
gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic -DHAVE_CONFIG_H -I.. -I. -I"/usr/include" -fPIC mktemp.c -o mktemp.o gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic -DHAVE_CONFIG_H -I.. -I. -I"/usr/include" -fPIC realpath.c -o realpath.o gcc -c -DUSE_TCL_STUBS -DTCL_NO_DEPRECATED -g -O2 -Wextra -Wall -pedantic -DHAVE_CONFIG_H -I.. -I. -I"/usr/include" -fPIC strlcat.c -o strlcat.o gcc -shared -g -O2 -Wextra -Wall -pedantic -fPIC -Wl,-soname,Pextlib.so Pextlib.o strsed.o fgetln.o md5cmd.o setmode.o xinstall.o fs-traverse.o strcasecmp.o vercomp.o filemap.o base32cmd.o sha1cmd.o curl.o rmd160cmd.o sha256cmd.o readline.o uid.o tracelib.o tty.o readdir.o pipe.o flock.o system.o mktemp.o realpath.o strlcat.o -o Pextlib.so -L/usr/lib64 -ltclstub8.5 -lcurl -lcrypto ../registry2.0/registry.so warning: error while loading Pextlib.so: couldn't load file "./Pextlib.so": registry.so: cannot open shared object file: No such file or directory make[2]: *** [pkgIndex.tcl] Error 1 make[2]: Leaving directory `/var/macports/trunk/base/src/pextlib1.0' make[1]: *** [all] Error 1 make[1]: Leaving directory `/var/macports/trunk/base/src' make: *** [all] Error 1
comment:30 Changed 11 years ago by jmroot (Joshua Root)
We can fix that later; for now please just resolve as per comment:26.
comment:31 Changed 11 years ago by cooljeanius (Eric Gallager)
Someone in IRC said it was working again...
comment:32 Changed 11 years ago by wsiegrist@…
Owner: | changed from cal@… to jmr@… |
---|
I applied the patches, looks like its working for me too. Josh, can you confirm?
comment:33 Changed 11 years ago by jmroot (Joshua Root)
Resolution: | → fixed |
---|---|
Status: | new → closed |
Yes, rsync and distfiles are updating again. Thanks.
comment:34 Changed 11 years ago by mndavidoff (Monte Davidoff)
Resolution: | fixed |
---|---|
Status: | closed → reopened |
It looks like it may have stopped working again. There have not been any updates for 16 hours.
$ TZ=UTC date Wed Jul 17 07:01:35 UTC 2013 $ TZ=UTC rsync rsync://rsync.macports.org/release/tarballs/ drwxr-xr-x 4096 2012/07/20 20:37:09 . -rw-r--r-- 4915200 2013/07/16 14:57:37 base.tar -rw-r--r-- 512 2013/07/16 14:57:39 base.tar.rmd160 -rw-r--r-- 53258240 2013/07/16 14:57:38 ports.tar -rw-r--r-- 512 2013/07/16 14:57:39 ports.tar.rmd160 drwxr-xr-x 4096 2013/07/16 14:57:20 PortIndex_darwin_10_i386 drwxr-xr-x 4096 2013/07/16 14:57:22 PortIndex_darwin_11_i386 drwxr-xr-x 4096 2013/07/16 14:57:25 PortIndex_darwin_12_i386 drwxr-xr-x 4096 2013/07/16 14:57:11 PortIndex_darwin_8_i386 drwxr-xr-x 4096 2013/07/16 14:57:08 PortIndex_darwin_8_powerpc drwxr-xr-x 4096 2013/07/16 14:57:17 PortIndex_darwin_9_i386 drwxr-xr-x 4096 2013/07/16 14:57:14 PortIndex_darwin_9_powerpc
comment:35 Changed 11 years ago by wsiegrist@…
My fault, I left the mprsyncup job disabled after troubleshooting it manually. It is running now and should be done in about 30m.
comment:36 Changed 11 years ago by wsiegrist@…
Resolution: | → fixed |
---|---|
Status: | reopened → closed |
We're caught up again and the sync job is enabled.
cal might want to be cc-ed on this, as he brought up the same issue on macports-dev: https://lists.macosforge.org/pipermail/macports-dev/2013-July/023223.html