Import new builds into ports web site

[ryandesign (Ryan Carsten Schmidt)]

The ports web site hasn't imported builds after Feb 20:

​https://ports.macports.org/ports/all_builds/

That was the day the buildmaster's SSD died and it took me a week to find time to figure out how and where to bring the buildmaster back. But it's back now and has done many builds since then.

I wasn't able to recover data from the SSD so I had to go to backups which meant that information about some builds that happened (and that were already imported into the ports web site) was lost. I didn't know exactly how many builds were lost in each queue so I skipped ahead approximately 100 build numbers in each queue. That way anyone who clicks on a link to one of the lost builds will get a 404 error rather than information about an unrelated build (mostly: I let a couple builds go through before I skipped ahead).

If the web app keeps track of the last imported build number in each queue, maybe you could reset it to the last build number that we were able to restore:

Port	Builder	Build Number	Start Time	Elapsed Time	Watcher	Build Status
dyncall	10.6	48687	2021-02-20 14:12:11	0:05:57	15083	build successful
py39-u-msgpack-python	10.6_i386	26668	2021-02-20 13:08:50	0:05:59	15124	build successful
py38-mypy	10.7	44788	2021-02-20 13:07:41	0:01:57	15121	build successful
py38-mypy	10.8	38711	2021-02-20 14:11:21	0:02:00	12979	build successful
gcc-devel	10.9	138837	2021-02-20 13:49:32	1:47:07	42385	build successful
go	10.10	135541	2021-02-20 11:17:51	0:01:54	42712	failed install-port
postgresql95-server	10.11	134031	2021-02-20 11:48:14	0:01:43	42651	build successful
fluent-bit	10.12	139112	2021-02-20 13:02:58	0:04:50	41647	build successful
go	10.13	103763	2021-02-20 12:59:20	0:10:26	33613	build successful
go	10.14	79558	2021-02-20 13:12:31	0:10:40	24308	build successful
squirrel-ime	11_arm64	12483	2021-02-20 14:55:18	0:01:59	3270	failed install-port
octave-nan	11_x86_64	22001	2021-02-20 15:50:11	0:16:04	3097	build successful

And then import starting after there. If the current import process checks successive build numbers and stops when it gets a 404 error, please do a one-time change that continues on through the 404 errors for up to 100 or so further build numbers in each queue to find where we started again after the skip, and then import normally from there.

10.15 is not in the table because 10.15 hasn't done any builds since then and I didn't skip its build numbers ahead. The 10.15 builder is offline for unrelated reasons and will come back eventually.

The buildmaster is temporarily running on a very slow disk so if your import process normally performs multiple requests to the server simultaneously, please limit it to just one request at a time for now, and it would also help to delay 10 seconds between each request to give the disk room to serve other requests that might be coming in from GitHub to schedule builds.

[neverpanic (Clemens Lang)]

I believe this is happening because the webapp has a bug when a port only changes in case, not in name, which causes an SQL exception in the transaction and stops importing new ports.

I've been getting emails about this for a while now, and I have fixed a similar case manually before, but it was such a long time ago that I don't remember how I did it and will have to re-read to code to find out.

Unfortunately, I haven't gotten around to doing that.

[ryandesign (Ryan Carsten Schmidt)]

Ah ok. Yes some case-only port renames have been happening lately and while I'm in favor of the web app handling port names case-insensitively it does need to be able to handle case-only renames.

[arjunsalyan (Arjun Salyan)]

The duplicate ports can be identified by looking for the following line(s) in the logs of the update cronjob:

Could not append {1} as a dependency to {2}, because the depspec does not uniquely identify a port.

And then we need to delete the duplicate entries from the ports table.

The presence of the new version (which fixes all of these issues) prevents me from making changes to the existing version. But this one keeps repeating and might better be fixed.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

The buildmaster is temporarily running on a very slow disk so if your import process normally performs multiple requests to the server simultaneously, please limit it to just one request at a time for now, and it would also help to delay 10 seconds between each request to give the disk room to serve other requests that might be coming in from GitHub to schedule builds.

Buildmaster is back on a fast SSD now so feel free to access it as much and as frequently as needed.

[neverpanic (Clemens Lang)]

See ​https://github.com/macports/macports-webapp/issues/263 and ​https://github.com/macports/macports-webapp/issues/265.

[neverpanic (Clemens Lang)]

In addition to the two known cases, there are also a few other case-only renames that now cause the same problem again:

webapp=> select count(*), lower(name) from ports_port group by lower(name) having count(*) > 1;
 count |      lower
-------+-----------------
     2 | transformerkit
     2 | hnhui
     2 | py35-nuitka
     2 | py34-whoosh
     2 | py36-nuitka
     2 | keepasskit
     2 | kissxml
     2 | py34-sqlalchemy
     2 | py27-nuitka
(9 rows)