mpbb-list-subports is slow

[ryandesign (Ryan Carsten Schmidt)]

Would be great if mpbb-list-subports could be made faster. Listing all subports of the php port takes 2 minutes 30 seconds on my 2016 MacBook Pro. Listing all of the py-* ports' subports ​took our buildbot 2 hours, listing the p5-* subports ​on the ppc buildbot took 26 hours. If we ever want to try to build all ports, at this rate, just listing the subports would probably take a week.

[jmroot (Joshua Root)]

This is a direct result of all the exclusion checks. Among other things, it queries the packages server with curl and runs port_binary_distributable.tcl for each subport.

[ryandesign (Ryan Carsten Schmidt)]

The curl check is probably already as fast as it can get, since it's just checking the response code from an http request to a server on the same gigabit network.

What about the rest? Each tcl script we run, and we run several for each subport, has to start tclsh and initialize the ports system. Is there a way we can initialize the ports system just once per subport and get all the information we need?

We have $(sysctl -n hw.activecpu) CPU cores. Is there a way we can check that many ports at the same time?

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

Is there a way we can check that many ports in parallel?

e.g. with ​GNU parallel?

[mojca (Mojca Miklavec)]

Curl might be as fast as is gets, but it would be even faster if we had a local database with all available files.

I don't know enough about Tcl to be able to tell how to avoid re-initializing everything every single time.

[ryandesign (Ryan Carsten Schmidt)]

Replying to mojca:

Curl might be as fast as is gets, but it would be even faster if we had a local database with all available files.

Maybe. Seems like something that could easily get out of sync. And how much time would this save? How quickly can we do a (SQLite?) database query? The curl check takes about 0.05 seconds, and it looks like we could cut that to 0.02 by using http instead of https.

I don't know enough about Tcl to be able to tell how to avoid re-initializing everything every single time.

I meant: instead of running archive-path.tcl and port_binary_distributable.tcl and portgroups.tcl and supported-archs.tcl for each subport, could we combine those into a single script? Move some of the logic from bash into Tcl? Even though we don't currently run all four scripts for all ports (some are run conditionally), I feel like probably most of the time is spent in initialization, such that combining them into a single script that does all four tasks would still be faster.

[jmroot (Joshua Root)]

This is about as good as it's going to get now. If you want to check if each archive has been uploaded, there's no getting around the fact that getting the archive names requires executing each portfile.