Consider building Python with +ucs4 by default

[Tatsh (Andrew Udvare)]

From what I can tell, most distros of Linux come with a wide version of Python by default, however OS X does not. This may seem a reason to avoid building a wide version of Python as it does not match the OS built-in but I think it is better that a replacement be better than the built-in system version.

Many packages would benefit from this change. Code that uses UTF-8 characters beyond 0xffff will not throw errors that are unexpected. Packages like ​Unidecode will benefit from this change.

The scope of this change would be that dependent packages will need +ucs4 and if anyone uses -ucs4 they are going to have problems. Basically it would be the reverse of the current situation, where -python27_ucs4 would be required for packages like gobject-introspection. This would apply to all versions of Python in the tree.

In terms of bugs, I really would like to know what kind of issues could be run into if +ucs4 is enabled as I have not had a problem with a wide-version Python on any Linux distro or dependencies.

[ryandesign (Ryan Carsten Schmidt)]

I know that we have had other ports that fail to build if a python is compiled with ucs4. I don't remember which ones or why.

[dbevans (David B. Evans)]

I don't remember either but I suspect the problem was with older packages that didn't properly support UTF-8. It's been a while since this issue was visited and so it probably would make sense to determine where the problems were in the past and see if things have changed in the interim. At the very least, it would be worth while to document any problems that remain. At best, perhaps we could get rid of this python variant and just build with this support enabled. Probably would mean, though, that all ports that link with libpython would need to be rebuilt.

[ned-deily (Ned Deily)]

I would strongly discourage changing the default to ucs4. On OS X, the convention for Python 2 has always been to use ucs2; I'm not aware of any other Python 2 distribution on OS X that uses ucs4. There are many implications: such a change could impact use of precompiled wheels; for some programs it could have a big impact on memory usage, and it can have an impact for any C extension modules that pass Unicode arguments. And, in the end, ucs4 is kind of a bandage anyway. If use of the full Unicode code set is important to you, you should certainly be using the latest versions of Python 3, whose major feature is native support of Unicode, not Python 2.

[ryandesign (Ryan Carsten Schmidt)]

Replying to ryandesign:

I know that we have had other ports that fail to build if a python is compiled with ucs4. I don't remember which ones or why.

For example #36107 and #55586.

Can we remove the ucs4 variants from the python ports? It would prevent users from encountering this problem and prevent us from having to tell people about it over and over again.

[jmroot (Joshua Root)]

Closing as per comment:3. This would also break pyobjc IIRC.