"---> Attempting to fetch" doesn't start in a new line

[mojca (Mojca Miklavec)]

There seem to be some issues with newlines when fetching with curl:

--->  Attempting to fetch glib2-2.48.0_0.darwin_11.x86_64.tbz2 from http://fco.it.packages.macports.org/mirrors/macports-packages/glib2/glib2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0--->  Attempting to fetch glib2-2.48.0_0.darwin_11.x86_64.tbz2 from http://nue.de.packages.macports.org/glib2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0--->  Attempting to fetch glib2-2.48.0_0.darwin_11.x86_64.tbz2 from http://lil.fr.packages.macports.org/glib2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0
  0     0    0     0    0     0      0      0 --:--:--  0:00:01 --:--:--     0--->  Fetching distfiles for glib2
--->  glib-2.48.0.tar.xz does not exist in /opt/local/var/macports/distfiles/glib2
--->  Attempting to fetch glib-2.48.0.tar.xz from http://fco.it.distfiles.macports.org/mirrors/macports-distfiles/glib2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 11 7312k   11  815k    0     0  3298k      0  0:00:02 --:--:--  0:00:02 3289k
100 7312k  100 7312k    0     0  6957k      0  0:00:01  0:00:01 --:--:-- 6964k--->  Verifying checksums for glib2
--->  Checksumming glib-2.48.0.tar.xz

From src/port1.0/portfetch.tcl​:

foreach site $urlmap($url_var) {
    ui_notice "$UI_PREFIX [format [msgcat::mc "Attempting to fetch %s from %s"] $distfile $site]"
    set file_url [portfetch::assemble_url $site $distfile]
    try -pass_signal {
        curl fetch {*}$fetch_options $file_url "${distpath}/${distfile}.TMP"
        file rename -force "${distpath}/${distfile}.TMP" "${distpath}/${distfile}"
        set fetched 1
        break
    } catch {{*} eCode eMessage} {
        ui_debug [msgcat::mc "Fetching distfile failed: %s" $eMessage]
        set lastError $eMessage
    } finally {
        file delete -force "${distpath}/${distfile}.TMP"
    }
}

Similar code is in src/package1.0/portarchivefetch.tcl​.

Nothing seems unusual, but maybe curl fetch doesn't add a newline or perhaps character deletion (which is supposed to delete a line before printing a new one to implement a "progress bar") doesn't work as expected? These are blind guesses though.

(I believe that this has only been a problem in trunk and that it works fine with the released version. But it's not entirely clear to me where the difference could come from. Maybe I need some search via bisection :)

[raimue (Rainer Müller)]

I cannot reproduce this with current master. Was this fixed already?

[mojca (Mojca Miklavec)]

I just updated to the latest version and I still experience the problem. OK, it's a slightly older OS, but I tested the released version on 10.6 and 10.12 and I don't see the problem there. I'm using bash from MacPorts as my shell, but switching to /bin/bash didn't help.

I can try to investigate a bit further (for example by installing master to 10.6, I would be grateful for suggestions about what I could test locally) to figure out what exactly could cause the problem or why the problem wasn't fixed for me (in case it was fixed for others).

[raimue (Rainer Müller)]

Did you experience this originally with an older version of macOS? Perhaps this only occured due to a bug in the progress output of libcurl which was missing a final newline?

To clarify my findings a bit more, I tested on macOS 10.12 Sierra. I increasing the version of a port and then run sudo port install <portname>. I did not see the problem with either 2.3.5 or current master.

[ryandesign (Ryan Carsten Schmidt)]

We can use the buildbot logs to see what happens on various OS versions.

Picking a build at random (libgcrypt):

​We don't see the problem on Snow Leopard (curl 7.19.7):

--->  Attempting to fetch libgcrypt-1.7.5_0.darwin_10.i386.tbz2 from https://packages.macports.org/libgcrypt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
DEBUG: Fetching archive failed:: The requested URL returned error: 404

​We don't see the problem on Lion (curl 7.21.4):

--->  Attempting to fetch libgcrypt-1.7.5_0.darwin_11.x86_64.tbz2 from https://packages.macports.org/libgcrypt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
DEBUG: Fetching archive failed:: The requested URL returned error: 404

​We don't see the problem on Mountain Lion (curl 7.24.0):

--->  Attempting to fetch libgcrypt-1.7.5_0.darwin_12.x86_64.tbz2 from https://packages.macports.org/libgcrypt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
DEBUG: Fetching archive failed:: The requested URL returned error: 404

​We don't see the problem on Mavericks (curl 7.30.0):

--->  Attempting to fetch libgcrypt-1.7.5_0.darwin_13.x86_64.tbz2 from https://packages.macports.org/libgcrypt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
DEBUG: Fetching archive failed:: The requested URL returned error: 404 Not Found

​We see the problem on Yosemite (curl 7.43.0):

--->  Attempting to fetch libgcrypt-1.7.5_0.darwin_14.x86_64.tbz2 from https://packages.macports.org/libgcrypt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0DEBUG: Fetching archive failed:: The requested URL returned error: 404 Not Found

​We see the problem on El Capitan (curl 7.43.0):

--->  Attempting to fetch libgcrypt-1.7.5_0.darwin_15.x86_64.tbz2 from https://packages.macports.org/libgcrypt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0DEBUG: Fetching archive failed:: The requested URL returned error: 404 Not Found

​We don't see the problem on Sierra (curl 7.49.1):

--->  Attempting to fetch libgcrypt-1.7.5_0.darwin_16.x86_64.tbz2 from https://packages.macports.org/libgcrypt
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed

  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
DEBUG: Fetching archive failed:: The requested URL returned error: 404 Not Found

This suggests it is a curl bug that was introduced after curl 7.30.0 and fixed before curl 7.49.1. But it's also possible Apple made custom modifications to the copies of curl included with macOS that could account for this.

These types of bugs in the system curl are a reason why I've suggested that we should bundle a newer copy of curl so that we get consistent behavior across OS versions.

[mojca (Mojca Miklavec)]

For me the problem is present on Lion as well, even though I don't remember seeing it in the stable version.