Source upgrade from 2.0.4 to 2.1.0 fails with sqlite error on OS X 10.4.11

[ned-deily (Ned Deily)]

Attempting to upgrade from 2.0.4 to 2.1.0 on a PPC 10.4.11 machine, the upgrade fails with an sqlite error.

 port selfupdate; port self outdated
--->  Updating MacPorts base sources using rsync
MacPorts base version 2.0.4 installed,
MacPorts base version 2.1.0 downloaded.
--->  Updating the ports tree
--->  MacPorts base is outdated, installing new version 2.1.0
Installing new MacPorts release in /macports as root:admin; permissions 0755; Tcl-Package in /Library/Tcl

Error: /macports/bin/port: port selfupdate failed: Error installing new MacPorts base: shell command failed (see log for details)
sqlite error: near "ADD": syntax error (1) while executing query: ALTER TABLE registry.files ADD COLUMN binary BOOL
    while executing
"registry::open $db_path"
    (procedure "mportinit" line 633)
    invoked from within
"mportinit ui_options global_options global_variations"
Error: /macports/bin/port: Failed to initialize MacPorts, sqlite error: near "ADD": syntax error (1) while executing query: ALTER TABLE registry.files ADD COLUMN binary BOOL

2.0.4 -> 2.1.0 source upgrades on similar configured 10.5 PPC, 10.6, and 10.7 systems worked fine.

[neverpanic (Clemens Lang)]

Which version of sqlite does Apple ship with 10.4?

[ned-deily (Ned Deily)]

$ /usr/bin/sqlite3 
SQLite version 3.1.3

[neverpanic (Clemens Lang)]

​http://www.sqlite.org/changes.html reveals "ALTER TABLE ADD COLUMN" wasn't added until 3.2.0. I guess the only way around this would be creating a new table with the correct layout, move the data over, delete the old table and rename the temporary table to the correct name.

[ultrajoe@…]

Cc Me!

[ultrajoe@…]

To make things worse, the update totally disables the port command. Nothing will run after that without producing the same error, not even "port echo foo"! Instructions on how to fix this without using the port command are needed ASAP!

[ultrajoe@…]

A thought: those users who have the sqlite3 port installed may be able to use that to recover manually. (Since I cannot find where the command is being generated, I'm at a loss.)

[neverpanic (Clemens Lang)]

No, the sqlite port doesn't help in this case, because you'd need MacPorts built against the library installed by itself, which is practically asking for trouble when upgrading this port.

Quickfix back to a working port command is to install 2.0.4 right over 2.1.0. The only data change from 2.0.4 to 2.1.0 is the one that just failed for you (and it was wrapped in a transaction, so should have been automatically reverted or at least revertible), so you should be safe doing that. Keeping a backup of your registry database ($prefix/var/macports/registry/registry.db) is a good idea nonetheless (also when testing any patches to solve this issue).

[phriedrich]

Cc Me!

[ryandesign (Ryan Carsten Schmidt)]

Oops, I filed a duplicate, #34478. To follow cal's suggestion of downgrading back to 2.0.4, you'd have to ​build 2.0.4 from source. Installing from the 2.0.4 disk image won't work, because the disk image installers conclude by running selfupdate.

[ultrajoe@…]

Another problem with dropping back to 2.0.4 is that there is no 2.0.4 dmg for Tiger.

Regarding my sqlite3 idea, I was only thinking in terms of correcting the problem, i.e. getting the port command adequately functional to apply an update.

[ultrajoe@…]

OK, I'm back up, at least. If somebody could tell me how to package up the base package for Tiger distribution, I'd be glad to attach it here.

Otherwise, here are some simpler instructions for those less inclined to figure out what's what.

1. Download either ​the bz2 source package or ​the gzipped one for MacPorts 2.0.4. (The .bz2 one is slightly smaller, but both download quickly on anything above dial-up.)
2. Open up a Terminal window, and cd to the directory where you downloaded one of the above files.
3. Type (or copy & paste) this into the Terminal:

   cd MacPorts-2.0.4
   ./configure && make && sudo make install
   

   (You will be asked for your admin password near the end.)
4. Type (or copy & paste) this into the Terminal to clean up:

   cd ../
   rm -rf MacPorts-2.0.4
   

5. Finally, type sudo port sync, NOT sudo port selfupdate!!! In fact, do not selfupdate until this problem is fixed!

You may note that the rm command above doesn't wipe out the original download from step 1. That's intentional, just in case you accidentally type port selfupdate before the fix. You can go back to step 2 and try again.

[ultrajoe@…]

Oh, crap! I forgot one very important part: unpacking the download!

1. Download either ​the bz2 source package or ​the gzipped one for MacPorts 2.0.4. (The .bz2 one is slightly smaller, but both download quickly on anything above dial-up.)
2. Open up a Terminal window, and cd to the directory where you downloaded one of the above files.
3. Type (or copy & paste) one of the following into the Terminal:
   1. For the .bz2 download, type tar xjvf MacPorts-2.0.4.tar.bz2
   2. For the .gz download, type tar xzvf MacPorts-2.0.4.tar.gz
4. Type (or copy & paste) this into the Terminal:

   cd MacPorts-2.0.4
   ./configure && make && sudo make install
   

   (You will be asked for your admin password near the end.)
5. Type (or copy & paste) this into the Terminal to clean up:

   cd ../
   rm -rf MacPorts-2.0.4
   

6. Finally, type sudo port sync, NOT sudo port selfupdate!!! In fact, do not selfupdate until this problem is fixed!

You may note that the rm command above doesn't wipe out the original download from step 1. That's intentional, just in case you accidentally type port selfupdate before the fix. You can go back to step 2 and try again.

(I hereby give an admin permission to replace my earlier post with this one, if desired.)

[neverpanic (Clemens Lang)]

One could add the column port(1) tries to add in the update using a newer version of SQLite, but that would make the database unreadable with the older version, so there's no point in doing so.

[p3k (Tobi Schäfer)]

Cc Me!

[neverpanic (Clemens Lang)]

Please test the attached patch (but make sure you have a backup of your registry.db).

[ned-deily (Ned Deily)]

The patch does not seem to work at least as I'm trying to do the upgrade. I tried it both on my existing source installation which has a non-default prefix, then with a vanilla default 2.0.4 built from source. Basic steps: download 2.0.4 tarball, ./configure, make, sudo make install, sudo port sync, sudo port install bzip2 (just to test) -> OK. Then download 2.1.0 tarball, download and apply patch to src/cregistry/sql.c. Run make which compiles sql.c with warnings:

gcc -c -DUSE_TCL_STUBS -g -O2 -Wextra -Wall -pedantic  -DHAVE_CONFIG_H -I.. -I.  -I"/usr/include" -fno-common sql.c -o sql.o
sql.c:269:17: warning: C++ style comments are not allowed in ISO C90
sql.c:269:17: warning: (this will be reported only once per input file)
ar cr cregistry.a registry.o entry.o sql.o vercomp.o util.o file.o

Then sudo make install which eventually errors with:

===> making install in tests
make[1]: Nothing to be done for `install'.
[ ! -f /macports/etc/macports/mp_version ] || rm -vf /macports/etc/macports/mp_version
/usr/bin/install -c -o root -g admin -m 444 setupenv.bash  /macports/share/macports/
/usr/bin/install -c -o root -g admin -m 444 macports-pubkey.pem  /macports/share/macports/
/usr/local/bin/tclsh src/upgrade_sources_conf_default.tcl "/macports"
/usr/local/bin/tclsh src/images_to_archives.tcl "/macports/share/macports/Tcl"
sqlite error: near "ADD": syntax error (1) while executing query: ALTER TABLE registry.files ADD COLUMN binary BOOL
    while executing
"registry::open $db_path"
    (procedure "mportinit" line 633)
    invoked from within
"mportinit ui_options"
    (file "src/images_to_archives.tcl" line 18)
make: *** [install] Error 1

[ned-deily (Ned Deily)]

Sorry, I just noticed I pasted the output from the initial attempt using my existing non-default configuration which has a /prefix=/macports. The output from the vanilla configuration (i.e. defaulting to /opt/local) is similar:

===> making install in tests
make[1]: Nothing to be done for `install'.
[ ! -f /opt/local/etc/macports/mp_version ] || rm -vf /opt/local/etc/macports/mp_version
/usr/bin/install -c -o root -g admin -m 444 setupenv.bash  /opt/local/share/macports/
/usr/bin/install -c -o root -g admin -m 444 macports-pubkey.pem  /opt/local/share/macports/
/usr/bin/tclsh src/upgrade_sources_conf_default.tcl "/opt/local"
/usr/bin/tclsh src/images_to_archives.tcl "/opt/local/share/macports/Tcl"
sqlite error: near "ADD": syntax error (1) while executing query: ALTER TABLE registry.files ADD COLUMN binary BOOL
    while executing
"registry::open $db_path"
    (procedure "mportinit" line 633)
    invoked from within
"mportinit ui_options"
    (file "src/images_to_archives.tcl" line 18)
make: *** [install] Error 1

[ryandesign (Ryan Carsten Schmidt)]

Cal, you wrote "#ifdef SQLITE_VERSION_NUMBER >= 3002000". Is that accurate or should that be "#if SQLITE_VERSION_NUMBER >= 3002000"?

[neverpanic (Clemens Lang)]

Ryan, thank you, that might well have been causing the behavior I didn't expect. Please try again with mp-base-src-cregistry-sql.c-support-dbupgrade-on-sqlite-before-3.2.0.patch (please ignore all other patches, as they have been attached in error; Ryan, can you delete those?)

I also fixed the compiler warnings you saw in this new patch.

[ned-deily (Ned Deily)]

Sorry, I should have noticed the #ifdef! The revised patch produces a new sqlite syntax error:

===> making install in tests
make[1]: Nothing to be done for `install'.
[ ! -f /opt/local/etc/macports/mp_version ] || rm -vf /opt/local/etc/macports/mp_version
/usr/bin/install -c -o root -g admin -m 444 setupenv.bash  /opt/local/share/macports/
/usr/bin/install -c -o root -g admin -m 444 macports-pubkey.pem  /opt/local/share/macports/
/usr/bin/tclsh src/upgrade_sources_conf_default.tcl "/opt/local"
/usr/bin/tclsh src/images_to_archives.tcl "/opt/local/share/macports/Tcl"
sqlite error: near " ": syntax error (1) while executing query: CREATE TEMPORARY TABLE mp_files_backup (id INTEGER, path TEXT, actual_path TEXT, active INT, mtime DATETIME, md5sum TEXT, editable INT, 
    while executing
"registry::open $db_path"
    (procedure "mportinit" line 633)
    invoked from within
"mportinit ui_options"
    (file "src/images_to_archives.tcl" line 18)
make: *** [install] Error 1

[neverpanic (Clemens Lang)]

Patch against base/src/cregistry/sql.c to use a different update mechanism on systems with SQLite < 3.2.0

[ultrajoe@…]

Results of my attempt to apply patch & rebuild. Note the detection of sqlite3.h (I have sqlite3 port installed), which may be pertinent.

[neverpanic (Clemens Lang)]

The line had a comma where it should not have been. Fixed in the new version of the patch, attached as mp-base-src-cregistry-sql.c-support-dbupgrade-on-sqlite-before-3.2.0.patch. Please try again.

[ned-deily (Ned Deily)]

The most recent patch looks good. With it, the default instance upgraded to 2.1.0. Also, my main, non-default instance also upgraded and I've done a "port sync" and "port upgrade" of one port all the way through the final steps:

--->  Updating database of binaries: 100.0%
--->  Scanning binaries for linking errors: 100.0%
--->  No broken files found.

Thanks!

[ultrajoe@…]

Current patch works for me as well.

[jmroot (Joshua Root)]

ultrajoe: Just note that if you have fink installed (which is unsupported), you should at least remove /sw/bin from your PATH before you configure.

[neverpanic (Clemens Lang)]

Commited in r93256, will cherry-pick into release 2.1 as soon as Ryan confirms this patch, too.

[ryandesign (Ryan Carsten Schmidt)]

Confirmed, this works for me too.

[neverpanic (Clemens Lang)]

Cherry-picked into release_2_1 in r93266.

[ccarey@…]

Following up on ultrajoe’s comment:6 — for those people with a broken 2.0.4 → 2.1.0 upgrade on Tiger who prefer to not take the manual-patch-application approach, will there be a 2.1.0 .dmg installer made for Tiger that includes the cherry-picked r93266? Or would such people need to wait for a 2.1.1 .dmg installer for Tiger to be available?

[ryandesign (Ryan Carsten Schmidt)]

I assume the release of 2.1.1 is imminent. I'll make a Tiger installer dmg at that time.

[sebastic@…]

Cc Me!

[200309@…]

Replying to ryandesign@…:

I assume the release of 2.1.1 is imminent. I'll make a Tiger installer dmg at that time.

Have you made the 2.1.1 dmg for Tiger? If yes, where is the download?

[ryandesign (Ryan Carsten Schmidt)]

I have not.

I believe you can just download and install the 2.0.3 disk image, which should upgrade you to 2.1.1 at the end.