[WIP] Add efficient ASCII to float/double parser#420
Conversation
…oading Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Replace safeGetline + raw const char* token pointer scanning with a StreamReader class that wraps (buf, len, idx) with bounds-checked peek/advance/match operations. Every byte access is guarded by an EOF check. Key changes: - Add StreamReader class for bounds-checked text parsing - Add sr_* parse functions that use StreamReader instead of const char** - Add LoadMtlInternal, LoadObjInternal, LoadObjWithCallbackInternal using StreamReader; public APIs wrap these - mmap paths construct StreamReader directly from mapped memory - Remove safeGetline (no longer needed) - Keep old parse functions for backward compat with public ParseTextureNameAndOption API - Keep IS_SPACE, IS_NEW_LINE, IS_DIGIT macros for old parse functions Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
…ked parsing - Add StreamReader class wrapping (buf, length) with EOF-guarded peek/advance/match - Add sr_* parse functions using StreamReader instead of const char** pointers - Add LoadMtlInternal, LoadObjInternal, LoadObjWithCallbackInternal taking StreamReader - mmap paths construct StreamReader directly from mapped memory - Remove safeGetline (no longer needed) - Keep old parse functions + macros for ParseTextureNameAndOption backward compat Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ent, BOM in callback, test fixes - Add UTF8ToWchar/LongPathW helpers for Windows long path and Unicode support - Add Windows long path support (#ifdef _WIN32) in non-mmap ifstream fallbacks - Add vt_w vector and optional w component parsing in LoadObjInternal - Fix texcoord_ws.swap(vt) -> texcoord_ws.swap(vt_w) corruption - Add BOM stripping to LoadObjWithCallbackInternal - Fix invalid-relative-texture-vertex-index.obj -> invalid-relative-texture-index.obj - Add test models: texcoord-w.obj, texcoord-w-mixed.obj, utf8-path-test.obj/.mtl - Add tests: texcoord_w, texcoord_w_mixed, utf8_path, long_path, callback_BOM Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
…lease Keep StreamReader-based parsing from the harden branch for all conflicting regions: map_d parsing, mmap material loading, LoadObjInternal, texcoord w-component, BOM handling, and main parse loop. Retain mmap/memory-buffer tests in tester.cc. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Add column tracking (col_num_) to StreamReader, updated in all position-advancing methods (get, advance, skip_space, etc.) - Add current_line_text(), format_error(filename, msg) for clang-style diagnostics (file:line:col with source line and caret) - Add error stack to StreamReader (push_error, has_errors, get_errors) - Add error-reporting sr_parse overloads that return bool for success/failure: sr_parseInt, sr_parseReal, sr_parseReal2/3, sr_parseVertexWithColor (returns -1 on error) - Wire error-reporting overloads into LoadMtlInternal (Ka, Kd, Ks, Ni, Ns, illum, d, Tr, PBR properties) with early return on parse failure; change LoadMtlInternal from void to bool - Wire error-reporting overloads into LoadObjInternal (v, vn, vt, vw) with early return on parse failure - Upgrade existing error messages (face, line, points, skin weight) to use sr.format_error() for consistent clang-style output - Pass filename through call chain: LoadObj -> LoadObjInternal, MaterialFileReader -> LoadMtlInternal - Add warning_context.filename for fixIndex warnings - Add 6 new tests: column tracking, format_error output, error stack, malformed vertex/mtl errors, backward compatibility Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
- Move warning_context initialization (warn, filename) before the main parse loop in LoadObjInternal; only update line_number per iteration. Avoids a std::string copy of filename on every face/line/points line. - Replace test_parse_error_backward_compat (was just TEST_CHECK(true)) with an actual round-trip test: parse valid OBJ, verify no errors and correct vertex count. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
|
@copilot rerun |
Port fast_float v8.0.2 (C++11 compatible) as the parsing backend and add an OBJ-specific wrapper (obj_parse_float.h) that handles: - Leading '+' sign (e.g. "+1.0", "+.5e3") - Case-insensitive nan/inf/infinity parsing - NumPy-style non-finite replacement (nan→0, inf→max, -inf→lowest) with user-configurable ParseOptions Includes standalone test/benchmark in sandbox/parse_fp/ covering correctness, accuracy vs strtod, and performance comparison. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add minimal shim (tinyobj_ff namespace) providing type traits, algorithms, fixed-width integers, and error codes so the embedded fast_float requires zero additional #include dependencies beyond what tinyobjloader already uses. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Update Makefiles to use -std=c++11, update README to reflect C++11 requirement, and simplify TINYOBJ_OVERRIDE macro. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add three tests covering float parsing: edge cases (leading/trailing dots, scientific notation, leading plus, leading zeros, negative zero, high precision, near-overflow, subnormals), nan/inf variants, and stream-based loading. All pass with both fast_float and legacy parsers. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Add null checks for s, s_end, and result parameters. Parse into a local temporary so *result is only written on success, preventing partial writes on overflow or invalid input. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Test overflow preserving default values, empty/whitespace input, garbage input, and extreme precision digits. All pass with both fast_float and legacy parsers under ASan. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Resolve conflicts by keeping both numeric edge case tests and StreamReader/error reporting tests. Adapt overflow and garbage input tests to match StreamReader's strict error detection behavior. Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
…jloader/tinyobjloader into copilot/add-ascii-to-float-parser
Resolve conflicts by taking release's improved safety/error handling (MappedFile RAII, stream size limits, overflow-safe integer parsing, sr.has_errors() pattern) while preserving HEAD's unique additions (embedded fast_float parser, numeric edge case tests, C++11 requirement). Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
… to callback API, add comprehensive tests - Add size_to_int() helper to safely clamp size_t to INT_MAX, preventing undefined behavior when vertex arrays exceed ~715M elements - Switch callback API (LoadObjWithCallback) to use error-reporting overload of sr_parseVertexWithColor, so malformed vertices are detected instead of silently using defaults - Fix StreamReader underflow in match()/char_at()/peek_at() bounds checks - Delete StreamReader copy/move constructors (raw pointer into owned buffer) - Add #undef for all 38 FASTFLOAT_* macros to prevent namespace pollution - Replace fragile integer type shims with standard <cstdint>/<cfloat>/<climits> - Fix LoadMtlInternal: warn_ss append (= to +=), phantom empty material guard - Fix MappedFile: reset size on POSIX mmap failure - Add bounds-check validation in earcut/quad triangulation paths - Remove duplicate #include merge artifact - Add 33 new unit tests covering StreamReader, edge cases, MTL, callbacks, and parser robustness Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
…ndling Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
|
@copilot review. deeper |
…seDouble, fix test assertion and CI workflow Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Deeper review found and fixed four issues in commit
|
|
@copilot review. higher |
…its), add C++11 error directive, document fast_float in README Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Higher-level review found and fixed three architectural issues in commit
Other architectural observations that do not require changes:
|
|
@copilot review. ultrathink |
…mment, add direct system_error include in sandbox Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Ultrathink review complete. Three additional fixes in commit
Additional findings examined and determined to be pre-existing (not introduced by this PR):
|
|
@copilot continue review. Ultra super think. |
… write Co-authored-by: syoyo <18676+syoyo@users.noreply.github.com>
Ultra-super-think pass completed. I found and fixed one concrete memory-safety issue in commit
Also re-checked CI/workflow status and logs for this branch (Unit Tests latest run is passing), then re-ran local |
|
@copilot review, again. deepest like Mariana Trench |
Completed another Mariana-trench-depth pass. I re-checked latest CI/workflow state and logs, re-ran local Current latest fix remains |
💬 We'd love your input! Share your thoughts on Copilot coding agent in our 2 minute survey.