Handle NaN and Infinity in JSON stringify function#111498
Handle NaN and Infinity in JSON stringify function#111498Repiteo merged 1 commit intogodotengine:masterfrom
Conversation
|
I support this PR. Replacing NaNs with Replacing infinities with This PR causes Godot to fall in line with the rest of the JSON ecosystem and fixes compatibility problems. The only Godot users it breaks are people who are incorrectly using JSON as a generic stringification method AND ALSO depending on the textual values of broken JSON strings. But in principle, this is no more of a breakage to such misuse of the JSON API than changing the number of digits that normal, finite floating point values are printed with, which nobody would object to if it was a fix for a hypothetical bug where Godot wasn't producing floating-point text with enough precision to reproduce them accurately during parsing. |
|
What do you think about not converting these values at all? Or converting them as you did but generating an error/warning? I don't see any error/warning prints or documentation changes in your PR. Since the JSON specification doesn't support |
|
Yeah, the NaN situation is very unfortunate. But I think that the browser Javascript APIs converting it to |
|
@dalexeev What does "not converting" actually mean? Returning null? Returning an empty string? This function has to return some value, even if it prints an error or warning. For NaN, my PR originally had |
|
Oh, I didn't realize you changed it from |
|
My memory of the core team meeting was that we were fine with correcting this behavior, but that we wanted to match the behavior of another big project, so that we weren't putting another unique solution to this problem out into the world. Which other project's behavior does the current version of this PR match? |
If we supported exceptions, the With the current approach, we could use an empty string as an error indicator and document it, but that's probably not a good idea, given how many users forget to check the result before using it. So we should probably try to fallback to the most reasonable value on a best-effort basis. In my opinion, we should use your approach or something similar to get a valid JSON, but additionally generate an error/warning to let the user know that |
|
@dsnopek As written in the description of the PR, graphite-project/graphite-web#813 is what I found in a quick search (IIRC, it was linked to in several places), and as such when I originally opened PR #108837 it had the same behavior as that (big number literals for infinity, and null for NaN). Then there was feedback in the comments of PR #108837 that strings are desired, since that is opt-in behavior in C#, with the default being to throw As for infinity, big number literals are still the unambiguous best option for encoding Infinity, it can be decoded correctly everywhere as far as I'm aware. The tricky one is NaN. Since both JavaScript and Graphite output null by default, it may be best to change this PR to I would be happy to take any other language or project's behavior into consideration... just name it. |
b88537f to
5f2a61b
Compare
|
I updated this PR with these changes:
|
doc/classes/JSON.xml
Outdated
| [b]Note:[/b] The JSON specification does not define integer or float types, but only a [i]number[/i] type. Therefore, converting a Variant to JSON text will convert all numerical values to [float] types. | ||
| [b]Note:[/b] If [param full_precision] is [code]true[/code], when stringifying floats, the unreliable digits are stringified in addition to the reliable digits to guarantee exact decoding. | ||
| The [param indent] parameter controls if and how something is indented; its contents will be used where there should be an indent in the output. Even spaces like [code]" "[/code] will work. [code]\t[/code] and [code]\n[/code] can also be used for a tab indent, or to make a newline for each indent respectively. | ||
| [b]Warning:[/b] The JSON specification only permits numbers written as numeric literals. Any instances of [constant @GDScript.INF] will be replaced with [code]1e99999[/code], -[constant @GDScript.INF] will be replaced with [code]-1e99999[/code], and [constant @GDScript.NAN] will be replaced with [code]null[/code]. If you rely on [constant @GDScript.NAN] or prefer an explicit way to encode [constant @GDScript.INF], consider passing your data through [method from_native] first. |
There was a problem hiding this comment.
I'm in favor of a warning like this, though the OP should be updated to note it's dependant on #111522
| ERR_PRINT_OFF | ||
| CHECK(JSON::stringify(non_finite_array) == "[1e99999,-1e99999,null]"); | ||
|
|
||
| Array non_finite_round_trip = JSON::parse_string(JSON::stringify(non_finite_array)); |
There was a problem hiding this comment.
We should avoid implying it's a round-trip conversion. best_effort_round_trip or partial_round_trip might be better monikers.
There was a problem hiding this comment.
I would still call it a round-trip, just a lossy one. The data is converted back into the original format, just with some of the data lost.
5f2a61b to
effcdd2
Compare
92163a1 to
2382e23
Compare
There was a problem hiding this comment.
I updated this PR with these changes:
- Serialize NaN as
nullinstead of"NaN".- Print a warning when encoding NaN similar to PR Core: Enforce strict JSON for non-finite numbers, document native workaround #111608 (not the same message, and not for Infinity)
- Add a warning to the documentation similar to PR Core: Enforce strict JSON for non-finite numbers, document native workaround #111608 (not the same docs, but similar, thanks @Repiteo).
Ok, so with these changes, I think this is now satisfying the requirements (as I remember them) that were set out at the core team meeting:
- We match the behavior of another big project. I don't know how "big" graphite-web is, but serializing
NaNasnullalso matches the behavior ofJSON.stringify()in web browsers (if I open Chrome's devtools and typeJSON.stringify({"a": NaN"})into the console it outputs'{"a":null}') which is certainly a big enough target :-) - We print a warning, so developers have a clue to follow if they get unexpected results from a web service or whatever application they are giving the JSON to
So, I approve! However, since this was reverted, it should probably get a few more approvals before going forward
Repiteo
left a comment
There was a problem hiding this comment.
I stand by my initial approval; even moreso with the added documentation and using null instead of "NaN". It does still need core consensus before going anywhere, so we'll be sure to discuss the refactored implementation at tomorrow's core meeting
Ivorforce
left a comment
There was a problem hiding this comment.
Discussed in the Godot core meeting.
This implementation matches the criteria we set at a previous Godot meeting:
- It does not produce invalid JSON
- It matches an existing implementation
This problem cannot be perfectly solved, but after our meeting we're convinced that this implementation will create the least problems.
Co-authored-by: Thaddeus Crews <repiteo@outlook.com> Co-authored-by: Lukas Tenbrink <lukas.tenbrink@gmail.com>
0fc6914 to
cc13a37
Compare
|
Thanks (again)! |
Fixes #89777, restores PR #108837, reverts the revert PR #111496. Depends on #111522 which should be merged first.
Regardless of what the user desires, we should pick something sane as the default behavior to ensure that the JSON is valid. The old behavior is bad.
The revert PR #111496 argues:
No,
1e99999is a valid numeric literal according to the JSON spec.This is a common approach used by other programs.
Even if there is no consensus, that just means any behavior is acceptable, except the behavior in master which produces invalid JSON, which is the only behavior that is unacceptable.
Very very very false, the old behavior produces invalid JSON, so it breaks 100% of JSON parsers. It is literally impossible to break more than 100% of JSON parsers, this is a nonsense argument.
Imagine if you edit an image in an image editor and you have transparency in the image.
Then you try to save the image to JPEG, a format which doesn't support transparency.
It would be completely unacceptable to write an invalid JPEG file that can't be opened by anything!
The JSON specification does not support NaN or Infinity values. However, it does support scientific notation. https://www.json.org/json-en.html
Without this PR, trying to export JSON containing NaN or Infinity will result in invalid JSON, which cannot be parsed back by Godot, nor can it be parsed by web browsers.
With this PR, trying to export JSON containing NaN and Infinity will result in valid JSON, with Infinity being replaced with extremely large
1e99999numbers, and NaN being replaced withnull. These large1e99999values are read in as Infinity both by Godot and by web browsers, and this seems to be a common trick in the JSON ecosystem. See graphite-project/graphite-web#813The unit tests added in this PR include examples of round-trip parsing, and the same tests fail without the other changes.
Why
1e99999: This is greater than the maximum value of an IEEE-standard 256-bit octuple-precision float, which is a bit over1e78913, pretty much guaranteeing an overflow to infinity since it will happen for all IEEE-standard floating point sizes. Also, just as a general note, this number is big enough that human readers should be able to tell that this is not intended to be a finite float value.