gettext @0.19.8.1: Running msgfmt to build Java2 resources crashes with specific string lengths of msgctxt and msgid

[p3k (Tobi Schäfer)]

This is kind of a weird bug which so far I only could observe using the msgfmt binary (version 0.19.8.1) on macos.

The following de.po file causes msgfmt to crash with ”Abort trap: 6“ when running msgfmt --java2 --source -d/tmp --resource=foo.bar.Messages --locale=de de.po

# SOME DESCRIPTIVE TITLE.
# Copyright (C) YEAR THE PACKAGE'S COPYRIGHT HOLDER
# This file is distributed under the same license as the PACKAGE package.
# FIRST AUTHOR <EMAIL@ADDRESS>, YEAR.
#
msgid ""
msgstr ""
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Transfer-Encoding: 8bit\n"
"Plural-Forms: nplurals=2; plural=(n != 1);\n"

msgctxt "123456789"
msgid "abcdef"
msgstr "whatever"

The expected output should be this Java file (and in fact, this is what I get with a Linux binary of msgfmt, same version):

/* Automatically generated by GNU msgfmt.  Do not modify!  */
package foo.bar;
public class Messages_de extends java.util.ResourceBundle {
  private static final java.lang.String[] table;
  static {
    java.lang.String[] t = new java.lang.String[4];
    t[0] = "";
    t[1] = "MIME-Version: 1.0\nContent-Type: text/plain; charset=UTF-8\nContent-Transfer-Encoding: 8bit\nPlural-Forms: nplurals=2; plural=(n != 1);\n";
    t[2] = "1a\u000423456789012345";
    t[3] = "whatever";
    table = t;
  }
  public java.lang.Object handleGetObject (java.lang.String msgid) throws java.util.MissingResourceException {
    int hash_val = msgid.hashCode() & 0x7fffffff;
    int idx = (hash_val % 2) << 1;
    java.lang.Object found = table[idx];
    if (found != null && msgid.equals(found))
      return table[idx + 1];
    return null;
  }
  public java.util.Enumeration getKeys () {
    return
      new java.util.Enumeration() {
        private int idx = 0;
        { while (idx < 4 && table[idx] == null) idx += 2; }
        public boolean hasMoreElements () {
          return (idx < 4);
        }
        public java.lang.Object nextElement () {
          java.lang.Object key = table[idx];
          do idx += 2; while (idx < 4 && table[idx] == null);
          return key;
        }
      };
  }
  public java.util.ResourceBundle getParent () {
    return parent;
  }
}

The problem seems to be related to the combined string lenghts of the msgctxt and msgid values.

For example, the following values crash msgfmt, too (msgstr can be arbitrary):

msgctxt "four"
msgid "sixplusfive"

msgctxt "context"
msgid "can hurt"

It looks like if the sum of both values’ lengths equals 15, msgfmt is going to crash – slightly modified values prevent the crash and create the Java message file accordingly:

msgctxt "123456789x"
msgid "abcdef"

msgctxt "123456789"
msgid "abcdef_"

msgctxt "four"
msgid "six-plus-five"

msgctxt "context"
msgid "is hard"

What could cause this behaviour? And how could it sneak into the macos binary only – not in the Linux one (assuming the same source code is used for both platforms)?

---

The issue could be verified on two different macos installations, one of them with the following specs:

# sw_vers
ProductName:    Mac OS X
ProductVersion: 10.14.3
BuildVersion:   18D109

# msgfmt -V
msgfmt (GNU gettext-tools) 0.19.8.1
Copyright (C) 1995-1998, 2000-2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.
Written by Ulrich Drepper.

# port version
Version: 2.5.4

Please let me know if additional information is required.

[p3k (Tobi Schäfer)]

Also see issue for Brew binary: ​https://github.com/Homebrew/homebrew-core/issues/38808

[jmroot (Joshua Root)]

Could you attach the crash log? It should be in either /Library/Logs/DiagnosticReports or ~/Library/Logs/DiagnosticReports.

[p3k (Tobi Schäfer)]

Replying to jmroot:

Could you attach the crash log? It should be in either /Library/Logs/DiagnosticReports or ~/Library/Logs/DiagnosticReports.

sure, did so. thanks for the quick reply.

[jmroot (Joshua Root)]

There's an off-by-one error in msgid_hashcode which results in a string not being null-terminated correctly, which leads to a stack overflow. I guess malloc is allocating in 16-byte chunks and the memory is usually starting zeroed, so unless your string ends exactly on the boundary you don't see the issue.

[jmroot (Joshua Root)]

OK to commit this, Ryan?

[p3k (Tobi Schäfer)]

Thanks for investigating and finding the culprit so fast.

Do you have any idea, though, why the error does not occur on Linux?

[jmroot (Joshua Root)]

Could be many things; malloc allocating in different sized chunks, something else being in the adjacent memory, etc. The problem is masked even by building with -O0 -g instead of -Os.

[ryandesign (Ryan Carsten Schmidt)]

Replying to jmroot:

OK to commit this, Ryan?

Sure. I filed a bug report with the developers: ​https://savannah.gnu.org/bugs/?56112

[jmroot (Joshua Root)]

In a4c79a0b578a7c230518cb8408e5653b5bef430d/macports-ports (master):

gettext: fix crash due to buffer overflow

Fixes: #58323

[jmroot (Joshua Root)]

Looks like the same mistake is made in write_java_msgid as well, though that one happens not to cause a crash.

[ryandesign (Ryan Carsten Schmidt)]

Upstream has committed your fixes, plus another similar fix in write-csharp.c. Looks like they're gearing up for a 0.20 release.