py-nbconvert @7.0.0: install fails, due to build-time artifact fetch

[agl2015]

Build of py39-nbconvert on a 2014 27” iMac running the latest version of Catalina (10.15.7) fails when not able to retrieve css from:

​https://unpkg.com/@jupyterlab/nbconvert-css@3.1.11/style/index.css.

I am attaching the main.log file for more info.

[agl2015]

main.log

[mascguy (Christopher Nielsen)]

This port was broken with update to 7.0.0, via this PR:

​https://github.com/macports/macports-ports/pull/15943

Perhaps this can be fixed by explicitly fetching artifacts by the port itself, rather that letting it occur during the Python build...?

[mascguy (Christopher Nielsen)]

Renee/All, is there any Python magic/helpers, to assist with projects that try to fetch via their build scripts? Or will we need to go through the usual port motions: declaring additional aliased master sites, distfiles, etc...?

[mascguy (Christopher Nielsen)]

The new approach in upstream version 7.x isn't ideal, as the CSS artifacts are now fetched at build time.

[mascguy (Christopher Nielsen)]

Reviewing what's done in other ports, I suppose we can handle this via direct use of wget/curl.

The following is semi-complete, apart from copying the artifacts to the appropriate location(s) in the work area. (It's slightly more involved than would otherwise be required, due to multiple artifacts named variables.css.)

    depends_fetch-append \
                    path:bin/wget:wget

    post-fetch {
        # Explicitly fetch CSS artifacts; all of this must remain in-sync with project build script
        set notebook_css_ver            5.4.0
        set jupyterlab_css_ver          3.1.11
        set jupyterlab_theme_light_ver  3.1.11
        set jupyterlab_theme_dark_ver   3.1.11

        set notebook_css_url \
            https://cdn.jupyter.org/notebook/${notebook_css_ver}/style/style.min.css
        set jupyterlab_css_url \
            https://unpkg.com/@jupyterlab/nbconvert-css@${jupyterlab_css_ver}/style/index.css
        set jupyterlab_theme_light_url \
            https://unpkg.com/@jupyterlab/theme-light-extension@${jupyterlab_theme_light_ver}/style/variables.css
        set jupyterlab_theme_dark_url \
            https://unpkg.com/@jupyterlab/theme-dark-extension@${jupyterlab_theme_dark_ver}/style/variables.css

        set css_fetch_dir \
            ${distpath}/css
        set wget_fetch_cmd \
            "${prefix}/bin/wget --no-verbose --no-check-certificate --timestamping"

        xinstall -d \
            ${css_fetch_dir}/notebook_css \
            ${css_fetch_dir}/jupyterlab_css \
            ${css_fetch_dir}/jupyterlab_theme_light \
            ${css_fetch_dir}/jupyterlab_theme_dark

        system -W ${css_fetch_dir}/notebook_css \
            "${wget_fetch_cmd} ${notebook_css_url}"
        system -W ${css_fetch_dir}/jupyterlab_css \
            "${wget_fetch_cmd} ${jupyterlab_css_url}"
        system -W ${css_fetch_dir}/jupyterlab_theme_light \
            "${wget_fetch_cmd} ${jupyterlab_theme_light_url}"
        system -W ${css_fetch_dir}/jupyterlab_theme_dark \
            "${wget_fetch_cmd} ${jupyterlab_theme_dark_url}"
    }

I'll leave this out here for a day or two, in case folks have any suggestions to improve/streamline the approach.

And ultimately we should reach out to upstream, to see if they can eliminate the need for all of this long-term.

[mascguy (Christopher Nielsen)]

Adding @cjones for awareness, since this is blocking builds of root6 and other dependent ports.

[cjones051073 (Chris Jones)]

I personally really do not like the above...

Before restorting to anything like that, I think first we need to understand what is it about the buildbots causing problems, as I can build py310-nbconvert just fine locally

> sudo port -v build py310-nbconvert
<snip>
--->  Fetching distfiles for py310-nbconvert
--->  Verifying checksums for py310-nbconvert
--->  Checksumming nbconvert-7.0.0.tar.gz
--->  Extracting py310-nbconvert
--->  Extracting nbconvert-7.0.0.tar.gz
Executing:  cd "/opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work" && /usr/bin/gzip -dc '/opt/local/var/macports/distfiles/py-nbconvert/nbconvert-7.0.0.tar.gz' | /usr/bin/tar -xf - 
--->  Configuring py310-nbconvert
--->  Building py310-nbconvert
Executing:  cd "/opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0" && /opt/local/Library/Frameworks/Python.framework/Versions/3.10/bin/python3.10 -m build --wheel --no-isolation --outdir /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work 
Downloading CSS: https://unpkg.com/@jupyterlab/nbconvert-css@3.1.11/style/index.css
Downloaded Notebook CSS to /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0/share/templates/lab/static/index.css
Downloading CSS: https://unpkg.com/@jupyterlab/theme-light-extension@3.1.11/style/variables.css
Downloaded Notebook CSS to /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0/share/templates/lab/static/theme-light.css
Downloading CSS: https://unpkg.com/@jupyterlab/theme-dark-extension@3.1.11/style/variables.css
Downloaded Notebook CSS to /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0/share/templates/lab/static/theme-dark.css
Downloading CSS: https://cdn.jupyter.org/notebook/5.4.0/style/style.min.css
Downloaded Notebook CSS to /opt/local/var/macports/build/_Users_chris_Projects_MacPorts_ports_python_py-nbconvert/py310-nbconvert/work/nbconvert-7.0.0/share/templates/classic/static/style.css
* Getting dependencies for wheel...
* Building wheel...
Successfully built nbconvert-7.0.0-py3-none-any.whl

so what is it about the buildbots causing the problem ?

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

Before resorting to anything like that, I think first we need to understand what is it about the buildbots causing problems

While that would be nice, the reality is that arbitrary artifacts shouldn't be fetched by the build script. Particularly not if we're interested in #ReproducibleBuilds.

Regardless of the SSL issue, we need to explicitly fetch and mirror these anyway. And I'm currently engaging with upstream, to see if this process can be simplified.

[mascguy (Christopher Nielsen)]

Replying to mascguy:

While that would be nice, the reality is that arbitrary artifacts shouldn't be fetched by the build script. Particularly not if we're interested in #ReproducibleBuilds.

In case you or anyone else isn't aware of it, there have also been recent reports of bad actors introducing spyware and viruses in some Python components. And this is yet another reason to avoid arbitrary, uncontrolled downloads via build scripts.

[cjones051073 (Chris Jones)]

I don't disagree, builds should not fetch during the build.

But still, why does this, as undesirable as it is, work outside the buildbots but not on them ?

[mascguy (Christopher Nielsen)]

Replying to cjones051073:

I don't disagree, builds should not fetch during the build.

But still, why does this, as undesirable as it is, work outside the buildbots but not on them ?

I'll let @ryandesign speak to that.

[Christopher Nielsen <mascguy@…>]

In afe944bfb3b417fa12c6700814cffd6affb38189/macports-ports (master):

py-nbconvert: explicitly fetch css artifacts
Fixes: #65809

[ryandesign (Ryan Carsten Schmidt)]

In 01f698f3b3b8e3ceeb4813b773af08518cf2246e/macports-ports (master):

py-nbconvert: Remove custom fetch code

Closes: #66011
See: #65809