Opened 9 years ago
Closed 5 years ago
#50875 closed defect (wontfix)
mirror-macports.sh runs daily but takes > 24 hours to complete
Reported by: | ryandesign (Ryan Carsten Schmidt) | Owned by: | admin@… |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | server/hosting | Version: | |
Keywords: | Cc: | jmroot (Joshua Root), neverpanic (Clemens Lang) | |
Port: |
Description
The browser:trunk/base/portmgr/jobs/mirror_macports.sh script runs every day at 2:15 am on rsync-origin.macports.org, but because of the number of ports we have and because it tries to mirror each port for 8 different OS X versions (darwin 8 through darwin 15), it now takes more than 24 hours to complete.
When a file is not found or the checksum doesn't match, the fetch is attempted 8 times, one for each OS X version, wasting time and bandwidth.
It mirrors for each version of OS X separately because a small subset of ports declare different distfiles for different OS X versions. However the vast majority of ports don't do that.
I think it's also supposed to be mirroring for each variant, since some variants might add distfiles.
We should find a way to rewrite the script so that it gets a list of URLs to mirror for all permutations of the port, but only attempts to fetch from each URL once.
The script on the server has been updated quite a bit from the version currently in the repository, and there are changes in the repository that have not been reflected on the server yet.
Change History (5)
comment:1 Changed 9 years ago by ryandesign (Ryan Carsten Schmidt)
comment:2 Changed 9 years ago by ryandesign (Ryan Carsten Schmidt)
Really, we shouldn't need to mirror all ports' files very often. That's just a safeguard against what should be the primary mirroring method, which is mirroring a port's files right before the buildbot is going to build it.
comment:3 Changed 9 years ago by ryandesign (Ryan Carsten Schmidt)
I have written a new mirroring script but have not deployed it yet because I haven't done a full test run of it yet.
comment:4 Changed 8 years ago by ryandesign (Ryan Carsten Schmidt)
Owner: | changed from admin@… to admin@… |
---|---|
Status: | new → assigned |
comment:5 Changed 5 years ago by ryandesign (Ryan Carsten Schmidt)
Resolution: | → wontfix |
---|---|
Status: | assigned → closed |
We no longer use mirror-macports.sh. We now have a mirror-distfiles command in mpbb which we use in our buildbot setup.
Replying to ryandesign@…:
I cut it back to running only twice a week, on Wednesday and Saturday mornings.
I'm working on a new script.