sqlite error: library routine called out of sequence (21)

[ryandesign (Ryan Carsten Schmidt)]

​Odd activation failure on the 10.12 buildbot worker:

--->  Activating expat @2.2.6_1
DEBUG: Using /usr/bin/tar
DEBUG: Using /usr/bin/bzip2
DEBUG: system: /usr/bin/bzip2 -d -c /opt/local/var/macports/software/expat/expat-2.2.6_1.darwin_16.x86_64.tbz2 | ( /usr/bin/tar -xvpf - )
x ./
x ./+COMMENT
x ./+CONTENTS
x ./+DESC
x ./+PORTFILE
x ./+STATE
x ./opt/
x ./opt/local/
x ./opt/local/bin/
x ./opt/local/include/
x ./opt/local/lib/
x ./opt/local/share/
x ./opt/local/share/doc/
x ./opt/local/share/examples/
x ./opt/local/share/man/
x ./opt/local/share/man/man1/
x ./opt/local/share/man/man1/xmlwf.1.gz
x ./opt/local/share/examples/expat/
x ./opt/local/share/examples/expat/elements.c
x ./opt/local/share/examples/expat/outline.c
x ./opt/local/share/doc/expat/
x ./opt/local/share/doc/expat/AUTHORS
x ./opt/local/share/doc/expat/COPYING
x ./opt/local/share/doc/expat/Changes
x ./opt/local/share/doc/expat/README.md
x ./opt/local/share/doc/expat/changelog
x ./opt/local/share/doc/expat/html/
x ./opt/local/share/doc/expat/html/expat.png
x ./opt/local/share/doc/expat/html/reference.html
x ./opt/local/share/doc/expat/html/style.css
x ./opt/local/share/doc/expat/html/valid-xhtml10.png
x ./opt/local/lib/libexpat.1.6.8.dylib
x ./opt/local/lib/libexpat.1.dylib
x ./opt/local/lib/libexpat.a
x ./opt/local/lib/libexpat.dylib
x ./opt/local/lib/pkgconfig/
x ./opt/local/lib/pkgconfig/expat.pc
x ./opt/local/include/expat.h
x ./opt/local/include/expat_config.h
x ./opt/local/include/expat_external.h
x ./opt/local/bin/xmlwf
DEBUG: Activation failed, rolling back.
Error: Failed to activate expat: sqlite error: library routine called out of sequence (21)
DEBUG: Error code: registry::sqlite-error
DEBUG: Backtrace: sqlite error: library routine called out of sequence (21)
    while executing
"$port activate $imagefiles"
    invoked from within
"try {
                $port activate $imagefiles
                foreach file $files {
                    if {[_activate_file "${extracted_dir}${file..."
    invoked from within
"registry::write {
            # Activate it, and catch errors so we can roll-back
            try {
                $port activate $imagefiles
       ..."
    invoked from within
"try {
        registry::write {
            foreach file $imagefiles {
                set srcfile "${extracted_dir}${file}"

                # To be ..."
    (procedure "_activate_contents" line 20)
    invoked from within
"_activate_contents $requested $rename_list"
    (procedure "portimage::activate" line 61)
    invoked from within
"registry_activate $subport $version $revision $portvariants $optionlist"
    (procedure "portactivate::activate_main" line 15)
    invoked from within
"$procedure $targetname"
Error: See /opt/local/var/macports/logs/_opt_bblocal_var_buildworker_ports_build_ports_textproc_expat/expat/main.log for details.
Error: Follow https://guide.macports.org/#project.tickets to report a bug.
Error: Processing of port expat failed

I had just forced many python builds on all the workers, and this error is happening on this worker anytime expat needs to be activated, but is not happening on the other workers, so I'm taking this worker offline for now.

I also see an error from /opt/local/bin/port installed requested:

Error: /opt/local/bin/port: port installed failed: an invalid entry was passed

I was manually uninstalling some large ports on this worker to free up disk space, and while the new registry was being written to disk, the VM froze and had to be rebooted from within VMware. SQLite3 is supposed to be robust against crashes, but this looks corrupt databasey to me.

An SQLite3 ​integrity check on the registry is not happy:

$ sqlite3 /opt/local/var/macports/registry/registry.db
SQLite version 3.25.2 2018-09-25 19:08:10
Enter ".help" for usage hints.
sqlite> .load /tmp/macports.sqlext
sqlite> pragma integrity_check;
*** in database main ***
On tree page 76852 cell 303: Rowid 18741471 out of order
On tree page 76852 cell 301: Rowid 18741430 out of order
On tree page 76852 cell 299: Rowid 18741387 out of order
On tree page 76852 cell 296: Rowid 18741324 out of order
On tree page 76852 cell 294: Rowid 18741280 out of order
On tree page 76852 cell 293: Rowid 18741258 out of order
On tree page 76852 cell 288: Rowid 18741147 out of order
On tree page 76852 cell 274: Rowid 18740770 out of order
On tree page 76852 cell 270: Rowid 18740688 out of order
On tree page 76852 cell 268: Rowid 18740623 out of order
On tree page 76852 cell 264: Rowid 18740484 out of order
On tree page 76852 cell 257: Rowid 18740236 out of order
On tree page 76852 cell 256: Rowid 18740203 out of order
On tree page 76852 cell 255: Rowid 18740166 out of order
On tree page 76852 cell 254: Rowid 18740133 out of order
On tree page 76852 cell 253: Rowid 18740112 out of order
On tree page 76852 cell 252: Rowid 18740091 out of order
On tree page 76852 cell 251: Rowid 18740070 out of order
On tree page 76852 cell 250: Rowid 18740047 out of order
On tree page 76852 cell 249: Rowid 18740026 out of order
On tree page 76852 cell 248: Rowid 18740007 out of order
On tree page 76852 cell 247: Rowid 18739986 out of order
On tree page 76852 cell 246: Rowid 18739967 out of order
On tree page 76852 cell 245: Rowid 18739948 out of order
On tree page 76852 cell 244: Rowid 18739928 out of order
On tree page 76852 cell 243: Rowid 18739906 out of order
On tree page 76852 cell 242: Rowid 18739875 out of order
On tree page 76852 cell 241: Rowid 18739844 out of order
On tree page 76852 cell 240: Rowid 18739810 out of order
On tree page 76852 cell 239: Rowid 18739768 out of order
On tree page 76852 cell 238: Rowid 18739733 out of order
On tree page 76852 cell 237: Rowid 18739698 out of order
On tree page 76852 cell 236: Rowid 18739662 out of order
On tree page 76852 cell 235: Rowid 18739625 out of order
On tree page 76852 cell 234: Rowid 18739591 out of order
On tree page 76852 cell 233: Rowid 18739555 out of order
On tree page 76852 cell 232: Rowid 18739531 out of order
On tree page 76852 cell 231: Rowid 18739510 out of order
On tree page 76852 cell 230: Rowid 18739491 out of order
On tree page 76852 cell 229: Rowid 18739469 out of order
On tree page 76852 cell 228: Rowid 18739448 out of order
On tree page 76852 cell 227: Rowid 18739427 out of order
On tree page 76852 cell 226: Rowid 18739405 out of order
On tree page 76852 cell 225: Rowid 18739384 out of order
On tree page 76852 cell 224: Rowid 18739363 out of order
On tree page 76852 cell 223: Rowid 18739342 out of order
On tree page 76852 cell 222: Rowid 18739321 out of order
On tree page 76852 cell 221: Rowid 18739299 out of order
On tree page 76852 cell 220: Rowid 18739278 out of order
On tree page 76852 cell 219: Rowid 18739258 out of order
On tree page 76852 cell 218: Rowid 18739237 out of order
On tree page 76852 cell 217: Rowid 18739215 out of order
On tree page 76852 cell 216: Rowid 18739194 out of order
On tree page 76852 cell 215: Rowid 18739175 out of order
On tree page 76852 cell 214: Rowid 18739154 out of order
On tree page 76852 cell 213: Rowid 18739133 out of order
On tree page 76852 cell 212: Rowid 18739111 out of order
On tree page 76852 cell 211: Rowid 18739088 out of order
On tree page 76852 cell 209: Rowid 18739044 out of order
On tree page 76852 cell 208: Rowid 18739022 out of order
On tree page 76852 cell 207: Rowid 18739001 out of order
On tree page 76852 cell 206: Rowid 18738978 out of order
On tree page 76852 cell 205: Rowid 18738951 out of order
On tree page 76852 cell 204: Rowid 18738915 out of order
On tree page 76852 cell 203: Rowid 18738879 out of order
On tree page 76852 cell 202: Rowid 18738844 out of order
On tree page 76852 cell 201: Rowid 18738813 out of order
On tree page 76852 cell 200: Rowid 18738779 out of order
On tree page 76852 cell 199: Rowid 18738743 out of order
On tree page 76852 cell 198: Rowid 18738721 out of order
On tree page 76852 cell 197: Rowid 18738699 out of order
On tree page 76852 cell 196: Rowid 18738679 out of order
On tree page 76852 cell 195: Rowid 18738656 out of order
On tree page 76852 cell 194: Rowid 18738635 out of order
On tree page 76852 cell 193: Rowid 18738616 out of order
On tree page 76852 cell 192: Rowid 18738596 out of order
On tree page 76852 cell 191: Rowid 18738578 out of order
On tree page 76852 cell 190: Rowid 18738558 out of order
On tree page 76852 cell 189: Rowid 18738538 out of order
On tree page 76852 cell 188: Rowid 18738518 out of order
On tree page 76852 cell 187: Rowid 18738493 out of order
On tree page 76852 cell 186: Rowid 18738461 out of order
On tree page 76852 cell 185: Rowid 18738429 out of order
On tree page 76852 cell 184: Rowid 18738393 out of order
On tree page 76852 cell 183: Rowid 18738355 out of order
On tree page 76852 cell 182: Rowid 18738320 out of order
On tree page 76852 cell 181: Rowid 18738286 out of order
On tree page 76852 cell 180: Rowid 18738251 out of order
On tree page 76852 cell 179: Rowid 18738210 out of order
On tree page 76852 cell 178: Rowid 18738176 out of order
On tree page 76852 cell 176: Rowid 18738107 out of order
On tree page 76852 cell 175: Rowid 18738087 out of order
On tree page 76852 cell 174: Rowid 18738067 out of order
On tree page 76852 cell 173: Rowid 18738046 out of order
On tree page 76852 cell 171: Rowid 18738004 out of order
On tree page 76852 cell 170: Rowid 18737983 out of order
On tree page 76852 cell 169: Rowid 18737962 out of order
On tree page 76852 cell 166: Rowid 18737899 out of order
On tree page 76852 cell 165: Rowid 18737878 out of order
On tree page 76852 cell 164: Rowid 18737857 out of order
sqlite>

Any suggestions?

[ryandesign (Ryan Carsten Schmidt)]

See also #57709 which is basically a duplicate.

I ​asked the sqlite mailing list for help. They suggested trying to .dump the registry database and .read it into a new one. This worked, but revealed that the database contained some duplicate entries which MacPorts didn't like. Since I don't know in what other ways the data in this registry might be compromised, uninstalling MacPorts and starting fresh seems like the most reliable option. So that that process goes a bit faster, I first want to configure buildbot and mpbb to download non-distributable archives from a private server.