Thanks!

Cherry-picked to 4.5

After updating from 4.5 to 4.5.1 I started seeing a bunch of errors like this:

  ERROR: Unicode parsing error, some characters were replaced with � (U+FFFD): Unexpected NUL character
  ERROR: Unicode parsing error, some characters were replaced with � (U+FFFD): Unexpected NUL character
  ERROR: Unicode parsing error, some characters were replaced with � (U+FFFD): Unexpected NUL character
  ERROR: Unicode parsing error, some characters were replaced with � (U+FFFD): Unexpected NUL character
  ERROR: Unicode parsing error, some characters were replaced with � (U+FFFD): Unexpected NUL character

Is this change related? I ask because I can't see anything else in the release notes that might be relevant.

Yes, that's this check triggering. Can you pinpoint when the error messages are printed? What's causing it?

As far as I can tell, there's no error callstack associated with the print, so I cannot easily tell where it is coming from. However, after a bit more searching I discovered this: #111792

And in our game we do have quite extensive key handling and those error prints begin after another print related to our custom key rebinding system doing its thing. So it may be triggered by trying to convert keycodes and apply input settings. Though we don't have an absolute ton of startup logging, so it could also be triggered later.

Edit: I should have looked a tiny bit further, as going to our keybinding menu sure enough I see where the character replacements happened:

Edit 2: for reference here is how it looks with Godot 4.5:

(those ö, and å letters do show up in 4.5.1 but I cropped the screenshot differently as I was doing it quickly)

As far as I can tell, there's no error callstack associated with the print, so I cannot easily tell where it is coming from. However, after a bit more searching I discovered this: #111792

And in our game we do have quite extensive key handling and those error prints begin after another print related to our custom key rebinding system doing its thing. So it may be triggered by trying to convert keycodes and apply input settings. Though we don't have an absolute ton of startup logging, so it could also be triggered later.

Edit: I should have looked a tiny bit further, as going to our keybinding menu sure enough I see where the character replacements happened:

Edit 2: for reference here is how it looks with Godot 4.5: (those ö, and å letters do show up in 4.5.1 but I cropped the screenshot differently as I was doing it quickly)

Yea, that would be the kind of bug introduced by these changes I made to string parsing. I'm very sorry about that!

In your custom key handling code, do you happen to call String::chr with keycodes? Keycodes being converted through String::chr has been the primary cause of this error message being printed in Godot's own codebase (hopefully, all fixed in 4.5.1?).
String::chr used to (in 4.4) silently return an empty string. In 4.5, it returned a string with the \0 character in it, which was unintended, because it can cause all kinds of string handling bugs. In 4.5.1, it errors and returns a string with the replacement character instead.

I did a bit of debugging and I do not think our code calls keycode to character conversion (like I initially thought). However, I found out from where the incorrect text was coming from.

It looks like DisplayServer.KeyboardGetLabelFromPhysical does not work correctly in 4.5.1

In 4.5 if I run the following code:

DisplayServer.KeyboardGetLabelFromPhysical(Key.Left)

The result is Key.Left as expected.

But in 4.5.1 I get the result from the same line of code that it is: 65533 which is not a valid value in the Key enum. So somehow that call to the engine is scrambling up the key codes.

I put this code in my game's startup and it makes it quit in 4.5.1 whereas 4.5 works normally:

        if (DisplayServer.KeyboardGetLabelFromPhysical(Key.Left) != Key.Left)
        {
            GD.PrintErr("Key code changed!");
            GetTree().Quit();
            return;
        }

That code in 4.5.1 also triggers the print I saw:

Unicode parsing error, some characters were replaced with � (U+FFFD): Unexpected NUL character

So I think that is the thing our game does that triggers the error print from the engine as well.

Should I open a new issue for this?

Edit: I checked the documentation just in case but it doesn't seem like the return value of the function should have changed: https://docs.godotengine.org/en/stable/classes/class_displayserver.html#class-displayserver-method-keyboard-get-label-from-physical

Thank you for digging.
I checked the code of the keyboard_get_label_from_physical function implementations.

Looks like I was mistaken, the x11 fix for the keyboard issue hasn't been released yet in 4.5.1, it will only be part of 4.5.2 (if there will be a 4.5.2 release), or 4.6: #111795

Looking at macOS and Windows, I don't think they have the same issue.

Thanks. I'll keep an eye out for the next Godot release then and not try further with 4.5.1.