X Tutup
Skip to content

Handle NaN and Infinity in JSON stringify function#111498

Merged
Repiteo merged 1 commit intogodotengine:masterfrom
aaronfranke:json-handle-nan-inf
Oct 30, 2025
Merged

Handle NaN and Infinity in JSON stringify function#111498
Repiteo merged 1 commit intogodotengine:masterfrom
aaronfranke:json-handle-nan-inf

Conversation

@aaronfranke
Copy link
Member

@aaronfranke aaronfranke commented Oct 11, 2025

Fixes #89777, restores PR #108837, reverts the revert PR #111496. Depends on #111522 which should be merged first.

Regardless of what the user desires, we should pick something sane as the default behavior to ensure that the JSON is valid. The old behavior is bad.

The revert PR #111496 argues:

create just as many problems since we are working outside the boundaries of the JSON spec

No, 1e99999 is a valid numeric literal according to the JSON spec.

and there is no consensus on how NaN and INF should be treated by other programs.

This is a common approach used by other programs.

Even if there is no consensus, that just means any behavior is acceptable, except the behavior in master which produces invalid JSON, which is the only behavior that is unacceptable.

Any option is liable to break more JSON parsers than it appeases.

Very very very false, the old behavior produces invalid JSON, so it breaks 100% of JSON parsers. It is literally impossible to break more than 100% of JSON parsers, this is a nonsense argument.


Imagine if you edit an image in an image editor and you have transparency in the image.

Then you try to save the image to JPEG, a format which doesn't support transparency.

It would be completely unacceptable to write an invalid JPEG file that can't be opened by anything!


The JSON specification does not support NaN or Infinity values. However, it does support scientific notation. https://www.json.org/json-en.html

Without this PR, trying to export JSON containing NaN or Infinity will result in invalid JSON, which cannot be parsed back by Godot, nor can it be parsed by web browsers.

With this PR, trying to export JSON containing NaN and Infinity will result in valid JSON, with Infinity being replaced with extremely large 1e99999 numbers, and NaN being replaced with null. These large 1e99999 values are read in as Infinity both by Godot and by web browsers, and this seems to be a common trick in the JSON ecosystem. See graphite-project/graphite-web#813

The unit tests added in this PR include examples of round-trip parsing, and the same tests fail without the other changes.

Why 1e99999: This is greater than the maximum value of an IEEE-standard 256-bit octuple-precision float, which is a bit over 1e78913, pretty much guaranteeing an overflow to infinity since it will happen for all IEEE-standard floating point sizes. Also, just as a general note, this number is big enough that human readers should be able to tell that this is not intended to be a finite float value.

@wareya
Copy link
Contributor

wareya commented Oct 11, 2025

I support this PR.

Replacing NaNs with null is what the Javascript JSON.dumps implementations does in at least Firefox and Chrome (likely mandated by web standards), so that aspect of this PR matches standard Javascript usage of the Javascript Object Notation format.

Replacing infinities with 1e99999 causes any real-world decoder to produce the same number as was used to encode the JSON. It is incredibly rare to parse JSON with bignums, and if anyone is doing so, they're not matching Javascript behavior (e.g. they're going to get incorrectly-rounded decimal values out of it). Javascript JSON.dumps produces null instead, but there's absolutely no reason to do that; 1e99999 is an entirely 100% unobjectionable perfect fix.

This PR causes Godot to fall in line with the rest of the JSON ecosystem and fixes compatibility problems. The only Godot users it breaks are people who are incorrectly using JSON as a generic stringification method AND ALSO depending on the textual values of broken JSON strings. But in principle, this is no more of a breakage to such misuse of the JSON API than changing the number of digits that normal, finite floating point values are printed with, which nobody would object to if it was a fix for a hypothetical bug where Godot wasn't producing floating-point text with enough precision to reproduce them accurately during parsing.

@dalexeev
Copy link
Member

What do you think about not converting these values ​​at all? Or converting them as you did but generating an error/warning? I don't see any error/warning prints or documentation changes in your PR.

Since the JSON specification doesn't support NaN and infinities, strictly speaking, they are unrepresentable (there is no bijective mapping). While you can still rely on scientific notation and overflow for infinities (which may still be implicit and not guaranteed to work for 100% of parsers), you can't represent NaN at all and you are forced to replace it with null or "NaN".

@wareya
Copy link
Contributor

wareya commented Oct 11, 2025

Yeah, the NaN situation is very unfortunate. But I think that the browser Javascript APIs converting it to null is in strong favor of going with that; it's consistent with the overwhelmingly most-used JSON encoding API. I poked around and it's rare for the built-in JSON APIs of dynamically typed languages to throw errors when encoding strange values. gdscript's JSONizer doesn't throw errors when you try to convert Vector3s to json, for example, it just stringifies them.

@aaronfranke
Copy link
Member Author

@dalexeev What does "not converting" actually mean? Returning null? Returning an empty string? This function has to return some value, even if it prints an error or warning.

For NaN, my PR originally had null, but there was feedback that it should be "NaN", so I changed it. I would be happy to change it back to null if that is the consensus. I don't mind either way, but I am opinionated that Infinity is best serialized in JSON as 1e99999.

@wareya
Copy link
Contributor

wareya commented Oct 11, 2025

Oh, I didn't realize you changed it from null to "NaN"! I'm okay with either, but I'd prefer null for consistency with browsers.

@dsnopek
Copy link
Contributor

dsnopek commented Oct 13, 2025

My memory of the core team meeting was that we were fine with correcting this behavior, but that we wanted to match the behavior of another big project, so that we weren't putting another unique solution to this problem out into the world.

Which other project's behavior does the current version of this PR match?

@dalexeev
Copy link
Member

What does "not converting" actually mean? Returning null? Returning an empty string? This function has to return some value, even if it prints an error or warning.

If we supported exceptions, the stringify() method would probably throw an exception in this case. If we supported nullable, union, or generic types, the return type would be String?, Optional[String], String|Error, Result[String], or something similar.

With the current approach, we could use an empty string as an error indicator and document it, but that's probably not a good idea, given how many users forget to check the result before using it. So we should probably try to fallback to the most reasonable value on a best-effort basis.

In my opinion, we should use your approach or something similar to get a valid JSON, but additionally generate an error/warning to let the user know that NaN and infinities ​​are not supported in JSON in the first place. But I'm not insisting that this should be done in this PR, since the stringify() method is currently generally quite tolerant of invalid values ​​and silent about possible user errors.

@aaronfranke
Copy link
Member Author

aaronfranke commented Oct 13, 2025

@dsnopek As written in the description of the PR, graphite-project/graphite-web#813 is what I found in a quick search (IIRC, it was linked to in several places), and as such when I originally opened PR #108837 it had the same behavior as that (big number literals for infinity, and null for NaN). Then there was feedback in the comments of PR #108837 that strings are desired, since that is opt-in behavior in C#, with the default being to throw System.ArgumentException, but we don't have exceptions in Godot's C++ code, so I then changed the PR to use "NaN" for NaN. Similarly to C#, Python throws ValueError by default, but unlike C#, its opt-in option is to emit invalid JSON, which is bad.

As for infinity, big number literals are still the unambiguous best option for encoding Infinity, it can be decoded correctly everywhere as far as I'm aware. The tricky one is NaN. Since both JavaScript and Graphite output null by default, it may be best to change this PR to null, as I originally opened the first iteration of PR #108837.

I would be happy to take any other language or project's behavior into consideration... just name it.

@aaronfranke
Copy link
Member Author

I updated this PR with these changes:

Copy link
Contributor

@Repiteo Repiteo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Feel free to list me as a co-author if you wanna lift changes wholesale from #111608; I have no strong preferences towards which PR ultimately fixes the issue

[b]Note:[/b] The JSON specification does not define integer or float types, but only a [i]number[/i] type. Therefore, converting a Variant to JSON text will convert all numerical values to [float] types.
[b]Note:[/b] If [param full_precision] is [code]true[/code], when stringifying floats, the unreliable digits are stringified in addition to the reliable digits to guarantee exact decoding.
The [param indent] parameter controls if and how something is indented; its contents will be used where there should be an indent in the output. Even spaces like [code]" "[/code] will work. [code]\t[/code] and [code]\n[/code] can also be used for a tab indent, or to make a newline for each indent respectively.
[b]Warning:[/b] The JSON specification only permits numbers written as numeric literals. Any instances of [constant @GDScript.INF] will be replaced with [code]1e99999[/code], -[constant @GDScript.INF] will be replaced with [code]-1e99999[/code], and [constant @GDScript.NAN] will be replaced with [code]null[/code]. If you rely on [constant @GDScript.NAN] or prefer an explicit way to encode [constant @GDScript.INF], consider passing your data through [method from_native] first.
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm in favor of a warning like this, though the OP should be updated to note it's dependant on #111522

ERR_PRINT_OFF
CHECK(JSON::stringify(non_finite_array) == "[1e99999,-1e99999,null]");

Array non_finite_round_trip = JSON::parse_string(JSON::stringify(non_finite_array));
Copy link
Contributor

@Repiteo Repiteo Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should avoid implying it's a round-trip conversion. best_effort_round_trip or partial_round_trip might be better monikers.

Copy link
Member Author

@aaronfranke aaronfranke Oct 13, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I would still call it a round-trip, just a lossy one. The data is converted back into the original format, just with some of the data lost.

Copy link
Contributor

@dsnopek dsnopek left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I updated this PR with these changes:

Ok, so with these changes, I think this is now satisfying the requirements (as I remember them) that were set out at the core team meeting:

  1. We match the behavior of another big project. I don't know how "big" graphite-web is, but serializing NaN as null also matches the behavior of JSON.stringify() in web browsers (if I open Chrome's devtools and type JSON.stringify({"a": NaN"}) into the console it outputs '{"a":null}') which is certainly a big enough target :-)
  2. We print a warning, so developers have a clue to follow if they get unexpected results from a web service or whatever application they are giving the JSON to

So, I approve! However, since this was reverted, it should probably get a few more approvals before going forward

Copy link
Contributor

@Repiteo Repiteo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I stand by my initial approval; even moreso with the added documentation and using null instead of "NaN". It does still need core consensus before going anywhere, so we'll be sure to discuss the refactored implementation at tomorrow's core meeting

Copy link
Member

@Ivorforce Ivorforce left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Discussed in the Godot core meeting.
This implementation matches the criteria we set at a previous Godot meeting:

  • It does not produce invalid JSON
  • It matches an existing implementation

This problem cannot be perfectly solved, but after our meeting we're convinced that this implementation will create the least problems.

Co-authored-by: Thaddeus Crews <repiteo@outlook.com>
Co-authored-by: Lukas Tenbrink <lukas.tenbrink@gmail.com>
@Repiteo Repiteo modified the milestones: 4.x, 4.6 Oct 30, 2025
@Repiteo Repiteo merged commit a176e4f into godotengine:master Oct 30, 2025
20 checks passed
@Repiteo
Copy link
Contributor

Repiteo commented Oct 30, 2025

Thanks (again)!

@aaronfranke aaronfranke deleted the json-handle-nan-inf branch October 30, 2025 22:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

JSON.stringify (and JSON.print in 3.x) produces invalid JSON for INF/NaN floats

8 participants

X Tutup