couchdb 1.6.1: couchdb no longer starts

[akimd (Akim Demaille)]

Hi!

I've been using couchdb (well, not me directly, but it's used by some project I use) for a long time, until recently, and unfortunately I have no idea what might have changed. I did a fresh reinstall, to no avail.

When run as myself, I get this:

$ couchdb
Apache CouchDB 1.6.1 (LogLevel=info) is starting.
[error] [<0.153.0>] {error_report,<0.58.0>,
                     {<0.153.0>,crash_report,
                      [[{initial_call,{couch_file,init,['Argument__1']}},
                        {pid,<0.153.0>},
                        {registered_name,[]},
                        {error_info,
                         {error,
                          {badmatch,{error,eacces}},
                          [{couch_file,init,1,
                            [{file,"couch_file.erl"},{line,314}]},
                           {gen_server,init_it,2,
                            [{file,"gen_server.erl"},{line,365}]},
                           {gen_server,init_it,6,
                            [{file,"gen_server.erl"},{line,333}]},
                           {proc_lib,init_p_do_apply,3,
                            [{file,"proc_lib.erl"},{line,247}]}]}},
                        {ancestors,[<0.152.0>]},
                        {message_queue_len,0},
                        {messages,[]},
                        {links,[#Port<0.2098>,<0.152.0>]},
                        {dictionary,[]},
                        {trap_exit,true},
                        {status,running},
                        {heap_size,376},
                        {stack_size,27},
                        {reductions,826}],
                       [{neighbour,
                         [{pid,<0.152.0>},
                          {registered_name,[]},
                          {initial_call,{erlang,apply,2}},
                          {current_function,{proc_lib,sync_wait,2}},
                          {ancestors,[]},
                          {message_queue_len,1},
                          {links,[<0.116.0>,<0.153.0>]},
                          {trap_exit,false},
                          {status,runnable},
                          {heap_size,233},
                          {stack_size,20},
                          {reductions,36},
                          {current_stacktrace,
                           [{proc_lib,sync_wait,2,
                             [{file,"proc_lib.erl"},{line,348}]},
                            {couch_file,open,2,
                             [{file,"couch_file.erl"},{line,49}]},
                            {couch_db,open_db_file,2,
                             [{file,"couch_db.erl"},{line,48}]},
                            {couch_db,start_link,3,
                             [{file,"couch_db.erl"},{line,38}]},
                            {couch_server,'-open_async/5-fun-0-',4,
                             [{file,"couch_server.erl"},{line,289}]}]}]}]]}}
{"init terminating in do_boot",{{badmatch,{error,{bad_return,{{couch_app,start,[normal,["/opt/local/etc/couchdb/default.ini","/opt/local/etc/couchdb/local.ini"]]},{'EXIT',{{badmatch,{error,{shutdown,{failed_to_start_child,couch_secondary_services,{shutdown,{failed_to_start_child,replicator_manager,{{{case_clause,{{badmatch,{error,eacces}},[{couch_file,init,1,[{file,"couch_file.erl"},{line,314}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}},[{couch_server,handle_info,2,[{file,"couch_server.erl"},{line,442}]},{gen_server,try_dispatch,4,[{file,"gen_server.erl"},{line,616}]},{gen_server,handle_msg,6,[{file,"gen_server.erl"},{line,686}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]},{gen_server,call,[couch_server,{open,<<"_replicator">>,[{before_doc_update,#Fun<couch_replicator_manager.before_doc_update.2>},{after_doc_read,#Fun<couch_replicator_manager.after_doc_read.2>},sys_db,sys_db,{user_ctx,{user_ctx,null,[<<"_admin">>,<<"_replicator">>],undefined}},nologifmissing]},infinity]}}}}}}}},[{couch_server_sup,start_server,1,[{file,"couch_server_sup.erl"},{line,98}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}}}}},[{couch,start,0,[{file,"couch.erl"},{line,18}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({{badmatch,{error,{bad_return,{_}}}},[{couch,start,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: erl_crash.dump...done

When run via sudo, I get something different:

$ sudo couchdb
Password:
Apache CouchDB 1.6.1 (LogLevel=info) is starting.
[error] [<0.173.0>] {error_report,<0.58.0>,
                     {<0.173.0>,crash_report,
                      [[{initial_call,{couch_uuids,init,['Argument__1']}},
                        {pid,<0.173.0>},
                        {registered_name,[]},
                        {error_info,
                         {error,undef,
                          [{crypto,rand_bytes,"\r",[]},
                           {couch_uuids,new_prefix,0,
                            [{file,"couch_uuids.erl"},{line,84}]},
                           {couch_uuids,state,0,
                            [{file,"couch_uuids.erl"},{line,100}]},
                           {couch_uuids,init,1,
                            [{file,"couch_uuids.erl"},{line,50}]},
                           {gen_server,init_it,2,
                            [{file,"gen_server.erl"},{line,365}]},
                           {gen_server,init_it,6,
                            [{file,"gen_server.erl"},{line,333}]},
                           {proc_lib,init_p_do_apply,3,
                            [{file,"proc_lib.erl"},{line,247}]}]}},
                        {ancestors,
                         [couch_secondary_services,couch_server_sup,<0.59.0>]},
                        {message_queue_len,0},
                        {messages,[]},
                        {links,[<0.122.0>]},
                        {dictionary,[]},
                        {trap_exit,false},
                        {status,running},
                        {heap_size,610},
                        {stack_size,27},
                        {reductions,234}],
                       []]}}
{"init terminating in do_boot",{{badmatch,{error,{bad_return,{{couch_app,start,[normal,["/opt/local/etc/couchdb/default.ini","/opt/local/etc/couchdb/local.ini"]]},{'EXIT',{{badmatch,{error,{shutdown,{failed_to_start_child,couch_secondary_s",[]},{couch_uuids,new_prefix,0,[{file,"couch_uuids.erl"},{line,84}]},{couch_uuids,state,0,[{file,"couch_uuids.erl"},{line,100}]},{couch_uuids,init,1,[{file,"couch_uuids.erl"},{line,50}]},{gen_server,init_it,2,[{file,"gen_server.erl"},{line,365}]},{gen_server,init_it,6,[{file,"gen_server.erl"},{line,333}]},{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,247}]}]}}}}}}},[{couch_server_sup,start_server,1,[{file,"couch_server_sup.erl"},{line,98}]},{application_master,start_it_old,4,[{file,"application_master.erl"},{line,273}]}]}}}}}},[{couch,start,0,[{file,"couch.erl"},{line,18}]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({{badmatch,{error,{bad_return,{_}}}},[{couch,start,0,[{_},{_}]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: erl_crash.dump...done

The directories have proper permissions I believe:

$ ls -la /opt/local/*/couchdb
-rwxr-xr-x  1 root  admin  11626  9 avr 15:36 /opt/local/bin/couchdb

/opt/local/etc/couchdb:
total 48
drwxrwxr-x   7 couchdb  couchdb    238  4 jul 17:23 .
drwxrwxr-x  33 root     wheel     1122  4 jul 16:40 ..
drwxr-xr-x   2 couchdb  couchdb     68  4 jul 17:23 default.d
-rw-rw-rw-   1 couchdb  couchdb  15550  9 avr 15:36 default.ini
drwxr-xr-x   2 couchdb  couchdb     68  4 jul 17:22 local.d
-rw-rw-rw-   1 couchdb  couchdb   3767  9 mar 11:53 local.ini
-rw-rw-rw-   1 couchdb  couchdb   3727  9 avr 15:36 local.ini.sample

/opt/local/lib/couchdb:
total 0
drwxr-xr-x     4 root  admin    136  4 jul 16:40 .
drwxrwxr-x  1349 root  wheel  45866  4 jul 16:40 ..
drwxr-xr-x     3 root  admin    102  4 jul 16:40 bin
drwxr-xr-x     3 root  admin    102  4 jul 16:40 erlang

/opt/local/share/couchdb:
total 0
drwxr-xr-x    4 root  admin   136  4 jul 16:40 .
drwxrwxr-x  125 root  wheel  4250  4 jul 16:40 ..
drwxr-xr-x    4 root  admin   136  4 jul 16:40 server
drwxr-xr-x   22 root  admin   748  4 jul 16:40 www

and I have the right rights.

$ id
uid=505(akim) gid=20(staff) groups=20(staff),505(couchdb),12(everyone),61(localaccounts),79(_appserverusr),80(admin),81(_appserveradm),98(_lpadmin),399(com.apple.access_ssh),701(com.apple.sharepoint.group.1),33(_appstore),100(_lpoperator),204(_developer),395(com.apple.access_ftp),398(com.apple.access_screensharing)

I have tried to follow the instructions from ​http://docs.couchdb.org/en/2.0.0/install/troubleshooting.html, but I can't get this to work:

%% test SSL support. If this fails, ensure you have the OTP erlang-crypto library installed
crypto:md5_init().

I guess that's because I don't set properly {/path/to/couchdb/lib} in

erl -env ERL_LIBS $ERL_LIBS:/path/to/couchdb/lib -couch_ini -s crypto

Reading the files installed by the port, I failed to guess what {/path/to/couchdb/lib} should be.

[mf2k (Frank Schima)]

Per the port notes, did you run the following?

sudo port load couchdb

[akimd (Akim Demaille)]

Yep.

[ajdalshov (Anders Johan Dalshov)]

I had the exact same problem after I upgraded all ports recently, the problem seems to be the combination:

couchdb@1.6.1 and erlang@20.0_0+hipe+ssl

After going back to erlang@19.3_0+hipe+ssl CouchDB starts up normally again.

[akimd (Akim Demaille)]

@ajdalshov Thanks a lot! You saved my life...

But it is still a PITA to specify an older version to install with MacPorts. For those interested, this is what I used:

sudo port uninstall -f erlang
cd /tmp
git clone --single-branch https://github.com/macports/macports-ports.git
cd macports-ports
git checkout e80897a5cc8f3583eac1bff12a62db6dc8ce4f99
cd lang/erlang
sudo port install

[kencu (Ken)]

couchdb build deps are here. <​https://github.com/apache/couchdb/blob/master/INSTALL.Unix.md>. There are mentions of patches on google for erlang 20, but not officially supported yet. So you're kinda stuck until they update it, if they do.

[akimd (Akim Demaille)]

It would be good for the package to require the appropriate version of erlang. Or for erlang 20 to conflict with couchdb, I don't know Portfiles well enough to know how the diagnostic can be given to the user, rather than breaking couchdb silently.

[ajdalshov (Anders Johan Dalshov)]

One feature I've been missing with ports is the ability to lock a port to a specific version, so that when doing port selfupdate and port upgrade outdated new versions of the ports marked as locked would get installed but not activated. A notification could then be shown for the ports that have gotten a new version installed but it has to be activated manually if wanting it.

Eg. doing:

port lock erlang@19.3_0+hipe+ssl

Listing installed ports would then look something like:

erlang @19.3_0+hipe+ssl (active) (locked)

erlang @20.0_0+hipe+ssl

Activating the 20.0 version would then require: port unlock erlang@19.3_0+hipe+ssl and port activate erlang@20.0_0+hipe+ssl

...just a thought.

[akimd (Akim Demaille)]

I agree it would be nice. In the meanwhile sudo port upgrade outdated and not erlang is ok.

[kencu (Ken)]

You can 'lock' or 'peg' a port at certain version fairly easily -- if you already have the version you want to keep, and the new version builds. Just sudo port selfupdate and sudo port upgrade outdated, and if that works, you can sudo port activate erlang and select the older version of erlang (or any other port). You're good to go then -- for all intents and purposes, you're locked. IIRC, it won't ask again to upgrade that port, even when new versions come along.

The real problem is when the new version won't build -- then you have a situation. MacPorts will ask over and over for you to upgrade it, and you can't. It often won't build anything else in that case. Similarly, you have a problem if you can't build (or don't want) the current version of a port, but the version three updates back will build on your system or is the version you need.

For THAT issue, you need to make a private local repository and shadow the ports you want to peg at a certain version. It's easy enough to do, but it does have a learning curve.

Someday MacPorts will probably be able to do this automatically, I would venture. Maybe not.

[chrstphrchvz (Christopher Chavez)]

See ticket:53948#comment:7 : the most recent versions of couchdb claim to be compatible with erlang 21.x, so updating it should avoid the need to manually activate an old version of erlang, or keep an old version of erlang in the macports-ports tree.