xdvipdfmx not able to use /System/Library/*.dfont fonts, freetype --with-old-mac-fonts involved

[pguyot (Paul Guyot)]

xdvipdfmx (port texlive-bin) is tricked into believing it is not configured on a Mac. The reason is that it needs FreeType function FT_GetFilePath_From_Mac_ATS_Name, which is only provided with FreeType 2.4.6 if configured with --with-old-mac-fonts.

As a result, xdvipdfmx cannot work with MacOS-specific font files, including both old (resource-based) fonts and new (data-fork based) fonts, such as /System/Library/*.dfont (e.g. Helvetica and Helvetica Neue in 10.6.8).

FreeType is no longer compiled with --with-old-mac-fonts since r42513 because of bug #15909.

If the conflict with gd2 is still present, we could make this a variant of freetype. If it is not, I believe we should provide freetype with-old-mac-fonts by default as we used to.

Attached are conservative diffs with a variant for freetype and texlive-bin.

[pguyot (Paul Guyot)]

Patch for freetype

[pguyot (Paul Guyot)]

Patch for texlive-bin

[ryandesign (Ryan Carsten Schmidt)]

"Old Mac fonts" are those having a resource fork. As far as I remember, Tiger was the last time Apple shipped any of those; since Leopard, it has shipped fonts using the data fork only. If tdvipdfmx checks for freetype's ability to load resource-fork-based fonts as some kind of indication of whether it can work at all on a Mac, then that sounds like a bug in tdvipdfmx that needs to be fixed.

[drkp (Dan Ports)]

I don't think that's quite correct -- freetype's --without-old-mac-fonts appears to disable all of the Mac-specific API, including FT_GetFilePath_From_Mac_ATS_Name, which doesn't seem resource-fork-specific. That's what xdvipdfmx is looking for.

[drkp (Dan Ports)]

The functions it needs are simple enough that we could probably build them into xdvipdfmx without too much trouble, though...

[drkp (Dan Ports)]

Does this patch solve your problem? (patch is against port dir texlive-bin; make sure you're using the regular freetype without --with-old-mac-fonts, and build texlive-bin +atsui)

[ryandesign (Ryan Carsten Schmidt)]

Replying to dports@…:

make sure you're using the regular freetype without --with-old-mac-fonts, and build texlive-bin +atsui

​ATSUI itself is of course deprecated in Leopard and up as well.

[pguyot (Paul Guyot)]

"Old Mac fonts" are those having a resource fork. As far as I remember, Tiger was the last time Apple shipped any of those; since Leopard, it has shipped fonts using the data fork only. If tdvipdfmx checks for freetype's ability to load resource-fork-based fonts as some kind of indication of whether it can work at all on a Mac, then that sounds like a bug in tdvipdfmx that needs to be fixed.

I don't think that's quite correct -- freetype's --without-old-mac-fonts appears to disable all of the Mac-specific API, including FT_GetFilePath_From_Mac_ATS_Name, which doesn't seem resource-fork-specific. That's what xdvipdfmx is looking for.

Not passing --with-old-mac-fonts to freetype does more than just dropping support for resource-based fonts. It entirely disables dependency on OS X frameworks, including the FT_GetFilePath_From_Mac_ATS_Name function which is not resource-fork specific. In fact, it removes all Mac-specific code. The real issue with gd2 was not the use of these functions, it was that freetype is linked with a framework whenever --with-old-mac-fonts is passed. This is different but not totally unrelated to ​radr://problem/7209349 : strange things happen when a framework is first loaded within a non-main thread. The ​radr://problem/7209349 bug is not fixed in 10.6.8 (did not test it in 10.7 yet), but the gd2 bug may have been fixed in 10.6. Hence my question about the current state of gd2 and bug #15909 because passing --with-old-mac-fonts on OS X seems a good idea (using Mac-specific code provides better handling of Mac fonts) and it is probably what other distributions of texlive do.

The functions it needs are simple enough that we could probably build them into xdvipdfmx without too much trouble, though... Does this patch solve your problem? (patch is against port dir texlive-bin; make sure you're using the regular freetype without --with-old-mac-fonts, and build texlive-bin +atsui)

I thought about providing just the functions indeed, yet tests I have just conducted with your patch show it is not sufficient.

For the record, I do not think it is related to the ATSUI problem of texlive. ATSUI is deprecated and most functions are just no longer available on 10.7 and on 10.5+ 64 bits. However, the ATS functions xdvipdfmx and Freetype need/use have no equivalent in CoreText and therefore are still available, although not recommended, even in 10.7 according to ​http://developer.apple.com/legacy/mac/library/documentation/Carbon/reference/ATS/ATS.pdf.

Your patch compiles fine (generates a 64bits binary) without +atsui, even if the line you added within the "if {![variant_isset atsui]}" block is removed (i.e. xdvipdfmx is linked against ApplicationServices).

However, passing --with-old-mac-fonts to freetype does more than just providing the couple of functions that was missing to xdvipdfmx. It provides an alternate version of FT_New_Face which, I believe, is able to look into dfont files beyond the first sfnt with the help of parameter face_index.

I did several tests with a simple tex file that uses *.dfont fonts available in 10.6.8 in /System/Library/. You'll find attached the (newer) diff to texlive-bin that I used for these tests (allowing a non-atsui 64bits xdvipdfmx build that links against OS X frameworks) as well as several outputs. You'll notice that Courier does not work, but other fonts only work in all their styles if Freetype is compiled with --with-old-mac-fonts only. The PDFs were generated with xelatex test.tex which invokes xdvipdfmx by default.

After this test, I strongly suggest rehabilitating --with-old-mac-fonts to Freetype.

I wanted to test with +atsui, but texlive-bin currently does not compile with +atsui (this should be in another ticket...)

[pguyot (Paul Guyot)]

Updated texlive-bin patch: xdvipdfmx compiles in 64 bits even with Freetype functions

[pguyot (Paul Guyot)]

Simple output with trunk freetype and texlive-bin with the current patch

[pguyot (Paul Guyot)]

Freetype and texlive-bin with with_old_mac_fonts patches

[pguyot (Paul Guyot)]

Simple output with freetype --with-old-mac-fonts and texlive-bin that includes Freetype functions (which are therefore duplicated)

[drkp (Dan Ports)]

If freetype is unable to handle dfont files containing multiple fonts, then that sounds like something that's worth fixing in freetype. Does that really not work without --with-old-mac-fonts? I'd gotten the impression that was supported even on non-Mac builds of freetype.

[pguyot (Paul Guyot)]

Freetype should be able to handle dfont files on non-Mac builds. Yet, without --with-old-mac-fonts, freetype does not return the proper font to xdvipdfmx as shown by the tests I ran.

If texlive-bin is compiled as on non-Mac (i.e. as in trunk), xdvipdfmx fails in cidtype2.c, line 969 :

#ifdef XETEX_MAC /* disable the index check here because of how .dfonts are handled */
    offset = 0;
#else
    if (opt->index > 0) {
      ERROR("Invalid TTC index (not TTC font): %s", name);
    } else {
      offset = 0;
    }
#endif

This is because opt->index is greater than 0 which is the index of the style in the given face (and might refer to the index of the sfnt resource ?).

[drkp (Dan Ports)]

Replying to pguyot@…:

Freetype should be able to handle dfont files on non-Mac builds. Yet, without --with-old-mac-fonts, freetype does not return the proper font to xdvipdfmx as shown by the tests I ran.

Yes, I see that now (I happened to try it with a font that did work correctly.) I'm still surprised that it doesn't work...

If texlive-bin is compiled as on non-Mac (i.e. as in trunk), xdvipdfmx fails in cidtype2.c, line 969 :

Right, that definitely should be fixed. The patch I provided will take care of that (or will be unnecessary), if we can get the freetype problem resolved one way or the other.

[pguyot (Paul Guyot)]

freetype is able to parse dfont files that contains several sfnt resources that contain a single font each. It cannot handle dfont files that contains one or more sfnt resources that contain several fonts. But this is not the case here. See the 2006 introduction to the code handling this here : ​http://lists.gnu.org/archive/html/freetype-devel/2006-10/msg00025.html

However, freetype computes face_index differently when --with-old-mac-fonts is enabled. When this option is not enabled, it uses a POSIX-compliant resource fork parsing code that probably index sfnt resources in the stream order. When this option is enabled, it uses Get1IndResource which gets resources in ascending ResID order if I remember correctly.

On the other hand, xdvipdfmx (normally) on non-Mac platforms uses fontconfig and the index it is passed, but will just fail to parse truetype fonts with an index greater than 0. On Mac, xdvipdfmx gets the face index from FT_GetFilePath_From_Mac_ATS_Name, which happens to be compatible with the index computed by Get1IndResource.

The proposed patch (texlive-bin.patch and the updated version texlive-bin.2.patch) gets xdvipdfmx to use FT_GetFilePath_From_Mac_ATS_Name without freetype --with-old-mac-fonts. Therefore, xdvipdfmx will invoke freetype with a face_index that will not match the proper style, as freetype expects an index based on the stream and not based on the ResID. As a result, styles will get mixed as shown in the PDFs that were generated and attached to this ticket. However, this does not explain the output of Helvetica Neue Medium and Helvetica Neue Condensed Bold...

Besides configuring freetype with --with-old-mac-fonts, two other fixes are possible :

• get xdvipdfmx to handle TrueType fonts with an index greater than 0. Just replacing #ifdef XETEX_MAC with #ifdef XETEX in the snippet above does not work, though.
• get freetype to index fonts alike with or without --with-old-mac-fonts (except for the FOND/sfnt difference ?) and compile xdvipdfmx according to the proposed patch (i.e. to use FT_GetFilePath_From_Mac_ATS_Name).

[pguyot (Paul Guyot)]

Attached is a patch that gets xdvipdfmx to handle dfonts.

[pguyot (Paul Guyot)]

Patch to texlive-bin to fix xdvipdfmx and get it to work with freetype in trunk (without --with-old-mac-fonts)

[drkp (Dan Ports)]

So does everything work OK with that patch? If so, that seems like the best approach. I'd be in favor of using --with-old-mac-fonts if it would work, but it sounds like the problems with that (#15909 etc) were pretty bad and I don't see any indication they've been fixed.

[pguyot (Paul Guyot)]

Replying to dports@…:

So does everything work OK with that patch? If so, that seems like the best approach. I'd be in favor of using --with-old-mac-fonts if it would work, but it sounds like the problems with that (#15909 etc) were pretty bad and I don't see any indication they've been fixed.

Indeed, this latest patch fixes the problem. For the record, issues with Courrier (the weird font in the PDFs) are caused by the X11 font files (in /usr/X11R6/lib/X11/fonts/Type1/).

[drkp (Dan Ports)]

Yes, we'll go with that, then. I have a couple other updates pending to texlive-bin, so I'll put this in with them.

[drkp (Dan Ports)]

r82979.

[drkp (Dan Ports)]

For the record, this patch has been incorporated into upstream xdvipdfmx, so it won't be necessary starting with texlive 2013.