feat: sometimes interpret char[] mutations as single bytes

[florianGla]

When mutating char[] randomly interpret the bytes from libFuzzer as individual (single byte) chars. This helps to make use of libFuzzers table of recent compare entries (encoded as CESU8) if the char[] is used as a String inside the fuzz test.

[simonresch]

LGTM. Thanks!

[oetr]

This doesn't seem to respect tha length provided by @WithLength annotation.

Oh, it does, never mind!

[Marcono1234]

Out of curiosity: Doesn't the current code here (specifically convertWithLength) assume that 2 byte = 1 char?
Wouldn't this assumption be violated here because:

• 1 CESU-8 byte can be converted to 1 char (for ASCII chars) → array becomes too large
• 3 CESU-8 byte can be converted to 1 char (for chars >= U+0800) → array becomes too short

Maybe I am overlooking something though; sorry for the trouble in that case.

(Also side note: Would it make sense to store the Charset.forName("CESU-8") in a static final field?)

[oetr]

I also couldn't see how the length gets respected here, and did some fuzzing with various @WithLength(min=..., max=...) values, and couldn't get the array length out of bounds. So it seems to be respected, for some reason!

Would it make sense to store the Charset.forName("CESU-8") in a static final field?

It would make sense, I think.

[kyakdan]

@oetr The length annotation is enforced by the innerMutator, which is used by toPrimitive and toPrimitiveAfterMutate: see https://github.com/CodeIntelligenceTesting/jazzer/blob/CIF-1863-string-compares-on-char-arrays/src/main/java/com/code_intelligence/jazzer/mutation/mutator/lang/PrimitiveArrayMutatorFactory.java#L95

[kyakdan]

@Marcono1234 You are right regarding the convertWithLength assuming two bytes per char, which could lead to enforcing incorrect length constraints when interepreting the byte array as "CESU-8"-encoded. I adjusted the convertWithLength method to assume a maximum of 6 bytes for a char (the maximum number of bytes used with CESU-8). Then, when we do the actual conversion to a char array, we ensure to enforce the length constraints. @oetr Could you have a look at the last commit?

[oetr]

Nice catch @Marcono1234 , thanks!

[Marcono1234]

Isn't this * 6 for a complete Unicode code point, specifically a code point >= 0x10000?
However a Java char is UTF-16, so those code points >= 0x10000 would actually require 2 char.

So wouldn't it suffice to do * 3 here?

Or at worst * 4 in case a malformed 4-byte CESU-8 encoding can lead to a single replacement ? char.

[oetr]

Looks good, just one issue from my side!

[oetr]

can be removed

[oetr]

Here, the newly padded chars might not be in range.

Maybe we can do a single-pass copy and forceInRange here? Or is Java smart enough to optimize it at runtime?
WDYT about the following:

       int targetLength = Math.min(Math.max(chars.length, minLength), maxLength);

        if (chars.length == targetLength) {
          for (int i = 0; i < chars.length; i++) {
            chars[i] = (char) forceInRange(chars[i], minRange, maxRange);
          }
          return chars;
        } else {
          char[] result = new char[targetLength];
          for (int i = 0; i < targetLength; i++) {
            result[i] = i < chars.length
                  ? (char) forceInRange(chars[i], minRange, maxRange)
                  : (char) forceInRange(0, minRange, maxRange);
          }
          return result;
        }