"base" seemingly locking up for sometimes very long

[RJVB (René Bertin)]

I'm not the only one who has noticed or thought at times that the port command "hung" because apparently nothing was happening e.g. at the end of a command. I've also already had a registry corruption at least once in the past because I had interrupted (^C) a command that to the best of my recollection wasn't even one that should have touched the registry.

I know I'm also not the only one who has observed the "write-ahead log" registry.db-wal file getting larger than the db itself, and sometimes just persisting until for some unclear reason it disappears (which usually means one of the aforementioned "freezes").

I have a custom script written in Tcl that uses code from "base" to check a newly generated destroot tree for conflicts, new files, updated vs. touched files or files that will go missing compared to the currently installed version/variant of a port. This command routinely takes a comparatively long time to start up the 1st time after a port destroot command (curiously more noticeably on my Mac than on my significantly less powerful Linux set-up).

That all comes across as a bit sub-optimal:

• long operations should be announced and if possible (optionally) provide a progress indicator
• leaving db operations "in limbo" in the -wal file seems like a recipe for disaster (esp. if they're not just backups)

All in all it sounds like port is missing the equivalent of the fsck and/or vacuum maintenance commands provided by KDE PIM's akonadictl driver (KDE Mail stores emails, filters etc. in -by default an sqlite3- database).

Given how large the registry can become and how well it compresses (better than 2x with just LZ4 online compression on ZFS) it seems like an optimisation command would be welcome too.

I've already tried to use the maintenance/optimisation commands in sqlitebrowser but that utility warns about custom tables that made be abort the attempt.

On a related note: it wouldn't hurt to have an automatic backup mechanism that is removed after succesful and total completion of an operation except in case of a DB version update. I'm also not the only person who found out the hard way that you can't (always?) roll back from a MacPorts "base" update, and that DB updates can happen unexpectedly because of operator error e.g. after creating a secondary MacPorts install.

[RJVB (René Bertin)]

(My DB knowledge is just good enough to know that this is not an area where I should go around hacking in the code to come up with a proposition...!)

[barracuda156]

Not too long ago with some earlier MacPorts version I had frequent “freeze-like” conditions when port command apparently did nothing and could not be quit normally: typically that happened when I ran sudo port upgrade for a port which version was not higher than installed one (i.e. -force was needed, which I forgot to add), but also after normal successful updates of some ports. Then the issue resolved, and I probably never had it, until 2.10.0. Today I had a similar freeze, and had to restart the machine.

While admittedly I have many ports installed on the Quad, so indexing is expected not to be instantaneous, it feels like some change after 2.9.3 made it worse. (This is a one-day experience, so take it FWIW. No strict measurements and repeated tests we done.)

[RJVB (René Bertin)]

I can only compare to v2.7.1 (.55, i.e. master branch) I've been running until recently, and overall 2.10.0 is about 2x slower. Or, probably more exactly, *startup* is about 2x slower (as in simple commands like port work, port dir, port info etc.), to the point that I had to edit a number of my personal helper scripts that evoke multiple such commands and became just too slow.

That however has nothing to do with the original issue.

[RJVB (René Bertin)]

I think this is about the only possible DB maintenance we can trigger with only routines from "base": a forced VACUUM.

It does seem to clean up lingering -wal files.

port-vacuum.tcl

#!/usr/bin/env port-tclsh

set t0 [clock clicks -millisec]
package require macports 1.0
package require registry 1.0
puts "Startup took [expr {([clock clicks -millisec]-$t0)/1000.}] sec"

array set ui_options        {}
array set global_options    {}
array set global_variations {}

set ui_options(ports_debug) yes
set ui_options(ports_verbose) yes

### init
puts -nonewline "Initialising MacPorts ..."
flush stdout
set t0 [clock clicks -millisec]
mportinit ui_options global_options global_variations
puts " done ([expr {([clock clicks -millisec]-$t0)}] ms)"

ui_msg "MacPorts version [macports::version], registry version [registry::metadata get version]"

set maybe ""
if {[catch {registry::set_needs_vacuum}]} {
     ui_warn "Your MacPorts version doesn't allow to program a registry VACUUM operation"
     set maybe "maybe "
} else {
     ui_msg "Called registry::set_needs_vacuum"
}

puts -nonewline "Closing registry and ${maybe}performing vacuum ..."
flush stdout
set t0 [clock clicks -millisec]
mportshutdown
puts " done ([expr {([clock clicks -millisec]-$t0)}] ms)"

(edited for the timing freaks like me)

[ryandesign (Ryan Carsten Schmidt)]

As of MacPorts 2.0.0, it does vacuum the registry; see [7d950c17e4be1c12ea13d487997806e5885e8f20/macports-base].

It used to do it after every operation but this was slow every time. As of MacPorts 2.9.0, it only vacuums if it's worth it, so now it's only slow at those times; see [c82a2699117181b1fedd490c62a3efffca7c8947/macports-base].