py-ansible-base 2.10.11: Path replacement breaks custom facts

[F30 (Felix Dreissig)]

Problem:

​Local custom facts do not get picked up from "/etc/ansible/facts.d" if Ansible was installed via MacPorts.

To reproduce, place a JSON or INI file into "/etc/ansible/facts.d" on the remote host, e.g. "/etc/ansible/facts.d/demo.fact":

{
  "foo": "bar"
}

Then, try to read these facts by running something like ansible -m ansible.builtin.setup -a 'filter=ansible_local'. The output will not contain the custom fact.

Alternatively, try accessing ansible_local['demo']['foo'] from a playbook. That will cause Ansible to crash:

fatal: [<snip>]: FAILED! => 
  msg: |-
    The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'demo'

Cause:

The py-ansible-base Portfile does a global replacement of all "/etc/ansible" paths to "/opt/local/etc/ansible". This is valid for local paths of the machine running Ansible. However, it does not take into account that the Ansible code also contains paths relevant for remote hosts, which may have nothing to do with MacPorts.

The Ansible docs clearly state a default fact path of "/etc/ansible/facts.d" and (at least for me) it is very unexpected that MacPorts changes that.

This is a fundamental problem and might affect other places besides facts. The following files from the Ansible repository are currently edited by the path replacement:

examples/hosts.yaml
examples/scripts/uptime.py
lib/ansible/config/base.yml
lib/ansible/config/manager.py
lib/ansible/inventory/manager.py
lib/ansible/module_utils/urls.py
lib/ansible/modules/setup.py
lib/ansible/plugins/loader.py
test/integration/targets/ansible-galaxy/cleanup.yml
test/integration/targets/gathering/implicit.yml
test/integration/targets/gathering/smart.yml
test/integration/targets/gathering_facts/test_gathering_facts.yml
test/integration/targets/gathering_facts/test_run_once.yml
test/units/cli/galaxy/test_execute_list_collection.py
test/units/config/manager/test_find_ini_config_file.py
test/units/executor/test_play_iterator.py
test/units/playbook/role/test_include_role.py
test/units/playbook/role/test_role.py
test/units/playbook/test_included_file.py
test/units/playbook/test_play.py
test/units/vars/test_variable_manager.py

I suppose we can ignore everything under "examples" and "test". This leaves 30 changed lines across 6 files under "lib". Most of these affect the default paths of plugins, collections, roles and similar. These really are relevant for the local machine only.

In fact, the default fact_path from "lib/ansible/modules/setup.py" is the only remote path I could cursorily identify. Of course, it does not need to stay like that in future versions.

Workaround:

The default fact path may get adjusted by setting the ANSIBLE_FACT_PATH environment variable or adding [defaults]/fact_path to an Ansible config file.

However, as stated ​by the Ansible docs, this only works for implicit fact gathering (i.e. the playbook example from above). Explicit calls to ansible.builtin.setup (such as the non-playbook example above) will always use the built-in default or the path directly specified from the call.

[ryandesign (Ryan Carsten Schmidt)]

Replying to F30:

The Ansible docs clearly state a default fact path of "/etc/ansible/facts.d" and (at least for me) it is very unexpected that MacPorts changes that.

It is customary for ports in MacPorts to be programmed to look for configuration files and so forth within the MacPorts prefix (e.g. in /opt/local/etc rather than /etc). I don't know anything about ansible so if that is not appropriate for ansible it can certainly be changed. I defer to the maintainers.

[adfernandes (Andrew Fernandes)]

I am not in favor of using the system /etc directory for anything.

macOS is not Linux, not even POSIX, and as far as I know the similarities, use, and even existence of /etc is a bit of a coincidental anachronism.

How many other Macports packages use the system /etc directory for ... anything?

(I just checked my large collection of installed ports, and the answer is, I think, "none of them".)

Anyone else have thoughts about this?

[F30 (Felix Dreissig)]

Replying to ryandesign:

It is customary for ports in MacPorts to be programmed to look for configuration files and so forth within the MacPorts prefix (e.g. in /opt/local/etc rather than /etc). I don't know anything about ansible so if that is not appropriate for ansible it can certainly be changed. I defer to the maintainers.

Adjusting the paths is certainly appropriate for local configuration on the machine where Ansible and MacPorts are installed. However, I'll argue that Ansible is special because it is used to execute code on remote hosts. Ansible facts, which we're talking about here, do get collected on the remote.

Replying to adfernandes:

I am not in favor of using the system /etc directory for anything.

macOS is not Linux, not even POSIX, and as far as I know the similarities, use, and even existence of /etc is a bit of a coincidental anachronism.

Well, the remote host might very well be Linux. In fact, it often is in practice. But it really could be anything – a default of "/opt/local/etc/ansible" on the remote will only work if it happens to be a Mac with MacPorts.

How many other Macports packages use the system /etc directory for ... anything?

(I just checked my large collection of installed ports, and the answer is, I think, "none of them".)

See above, this is not about local configuration on hosts with MacPorts installed.

To me, saying the remote configuration path should be set by MacPorts feels a bit like saying ssh installed via MacPorts should add "/opt/local/bin" to $PATH on remote hosts.

[mpallante]

Replying to adfernandes:

Anyone else have thoughts about this?

This changes in Ansible configuration is simply wrong because it breaks the usual Ansible behaviour.

As you can see from its documentation (​https://docs.ansible.com/ansible/latest/collections/ansible/builtin/setup_module.html), the setup module expects default value of fact_path to be /etc/ansible/facts.d and must be a directory on the remote host in which Ansible go and search files and scripts that enrich the variables with information that need to be collected on the remote host. So, everything is referred to the remote host.

I just wasted a full day trying to solve an issue with a playbook that configures a backup server, which at the end I tracked down to be due to this misconfigured path.

I also couldn't use the workarounds cited by F30 because our playbook uses directly the setup module or indirectly by importing 3rd parts roles which in turn use the setup module.

I can also confirm py-ansible-core 2.11.6 has the same wrong behaviour.