Opened 6 years ago
Last modified 2 years ago
#57464 new enhancement
Builtbot: Don't keep huge ports installed — at Initial Version
Reported by: | ryandesign (Ryan Carsten Schmidt) | Owned by: | admin@… |
---|---|---|---|
Priority: | Normal | Milestone: | |
Component: | buildbot/mpbb | Version: | |
Keywords: | Cc: | ||
Port: |
Description
Our buildbot setup currently keeps all built ports installed on each worker. This is supposed to speed up builds so that we don't waste time constantly downloading and installing ports that are dependencies of a lot of other ports.
But there are also some inordinately large ports, such as some games, which aren't commonly used as dependencies. Here are some of the larger ports installed on the highsierra buildbot worker:
$ cd /opt/local/var/macports/software $ du -sch alienarena-data/* geant*-data/* knp/* supertuxkart/* texlive-latex-extra/* typesafe-activator/* wesnoth/* 551M alienarena-data/alienarena-data-7.66-20130827_0.darwin_16.noarch.tbz2 427M geant4.10.0-data/geant4.10.0-data-4.10.0_0.darwin_16.noarch.tbz2 428M geant4.10.1-data/geant4.10.1-data-4.10.1_0.darwin_16.noarch.tbz2 432M geant4.10.2-data/geant4.10.2-data-4.10.2_1.darwin_16.noarch.tbz2 1005M geant4.10.3-data/geant4.10.3-data-4.10.3_0.darwin_16.noarch.tbz2 1.1G geant4.10.4-data/geant4.10.4-data-4.10.4_0.darwin_16.noarch.tbz2 266M geant4.9.6-data/geant4.9.6-data-4.9.6_0.darwin_16.noarch.tbz2 1.0G knp/knp-4.14_0.darwin_16.x86_64.tbz2 512M supertuxkart/supertuxkart-0.9.3_0.darwin_16.x86_64.tbz2 453M texlive-latex-extra/texlive-latex-extra-47410_0+doc.darwin_16.noarch.tbz2 663M typesafe-activator/typesafe-activator-1.3.12_0.darwin_16.noarch.tbz2 365M wesnoth/wesnoth-1.12.6_2.darwin_16.x86_64.tbz2 7.1G total
It's getting to the point where builds are failing because there's not enough free disk space, and I am manually uninstalling some of these ports on the buildbot workers to free up space. If we could have mpbb automatically uninstall large ports, it would free up a good deal of disk space.
I'm not sure what do about dependencies. For example, geant4.10.4 is not large, but it depends on geant4.10.4-data which is.
Do we force the uninstallation of the large ports, leaving the ports depending on them broken, on the assumption that when the buildbot asks to install them again it will reinstall the dependencies and fix them?
Or do we have the buildbot uninstall not only the large ports but also all ports depending on them? That would be good for geant, but bad for texlive-latex-extra, on which lots of things depend. We could base the decision of whether to uninstall on how many ports would be uninstalled. (It it's, say, 5 or fewer ports, uninstall them, otherwise leave them installed.)
Or do we instead make a general rule (not based on size) that we don't keep a port installed if nothing depends on it?
Some of these ports might not be distributable. Once we have nondistributable archives available on a private server that won't be a big problem anymore. We can wait until we resolve that ticket before deploying an implementation for this one.