Opened 6 years ago

Closed 3 years ago

#57343 closed defect (fixed)

mpbb mirror-distfiles repeatedly downloads distfiles with checksum mismatches

Reported by: ryandesign (Ryan Carsten Schmidt) Owned by: admin@…
Priority: Normal Milestone:
Component: buildbot/mpbb Version:
Keywords: Cc:
Port:

Description

Consider this run of mpbb mirror-distfiles which ran after the qt58 port was modified to fix a typo. Upstream may have done a stealth update, because 31 of the subports' distfiles had checksum mismatches. But mpbb downloaded each files 12 times, because mpbb tries mirroring separately for each of our platform versions and archs. The 31 files totaled 339MB, downloaded 12 times is 4GB, which took 53 minutes, during which time no other builds could be started. And because we still haven't optimized our buildbot setup to trigger the mirror task only once per batch of commits, and instead we trigger a mirror task from each of the 12 portwatchers for each batch of commits, this 4GB 53-minute download was repeated 12 times.

The reason why we impersonate each platform while mirroring is so that we mirror files that a port only uses on a particular platform. But if a file fails to download or checksum properly for one platform, we should probably remember that and not try downloading that same file again for another platform during this mirroring run.

Change History (1)

comment:1 Changed 3 years ago by jmroot (Joshua Root)

Resolution: fixed
Status: newclosed

In bdea90b82f3fa0c91da3dc9522140daab275d9ca/mpbb (master):

mirror-multi.tcl: don't repeatedly mirror distfiles

Skip mirroring if any of the port's distfiles already had a checksum
mismatch this run. Also skip if all of the port's distfiles were
already successfully mirrored, since that information becomes readily
available in solving the aforementioned problem.

Fixes: #57343

Note: See TracTickets for help on using tickets.