[25.x] module: fix extensionless CJS files in type:module packages#62083
Conversation
|
Review requested:
|
3e6aa5a to
6e3ed67
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## v25.x-staging #62083 +/- ##
================================================
Coverage ? 89.67%
================================================
Files ? 682
Lines ? 206206
Branches ? 39547
================================================
Hits ? 184915
Misses ? 13438
Partials ? 7853
🚀 New features to boost your workflow:
|
joyeecheung
left a comment
There was a problem hiding this comment.
Code LGTM with a correction on comment
GeoffreyBooth
left a comment
There was a problem hiding this comment.
- PR module: fix extensionless entry with explicit type=commonjs #61600 made the CJS loader respect
package.jsontypefield for extensionless files, but also enforcedtype: "module"which breaks CJS extensionless files inside ESM packages (e.g. yargs v17)- Only enforce
type: "commonjs"for extensionless files; fortype: "module", leave format undefined so V8's syntax detection determines the correct format
I think this proposed behavior is incorrect; according to our spec, extensionless files within a package scope with an explicit type field follow the format of the type field—that is, they behave however .js files would behave. We never do syntax detection when there's an explicit type field.
I read yargs/yargs#2509 and it's unfortunate that they seemed to rely on what was buggy behavior. But this PR seems to make several breaking changes, going against our own documented behaviors for how the type field works and how syntax detection works. I'm not sure what to do instead; perhaps just revert #61600 and re-land it as semver-major?
Marking as request changes so that we discuss this before landing. If others disagree with my analysis here I'm happy to be persuaded I'm wrong; I understand there's probably an eagerness to unbreak a popular library but let's not be so hasty that we cause other bugs.
Can you list them? I don't see what would be breaking about it |
lib/internal/modules/cjs/loader.js
Outdated
| // Extensionless files skip the .js suffix check above. When type is commonjs, | ||
| // follow it so ESM syntax surfaces as SyntaxError instead of silently | ||
| // delegating to ESM. For type: module, leave format undefined so our syntax | ||
| // detection handles it (allowing CJS extensionless files in ESM packages). |
There was a problem hiding this comment.
fwiw this seems right to me; i remain surprised that extensionless ESM was ever possible, and in all the module group discussions, i thought we concluded it basically wouldn't ever be (unless the "type" default changed)
There was a problem hiding this comment.
This is not recent, it has been possible since v20, so it would be too late to do anything about it.
There was a problem hiding this comment.
Can you link to where it was intentionally made possible, though?
|
@GeoffreyBooth I didn't put much thought into the spec for this. My main concern was to fix this without having to revert that commit. I'm happy either way. The alternative would be to revert that commit from 25.x and land it after the problematic behavior is fixed in the ecosystem. We should be doing something soon about this. |
This is a case where we have contradicting documentations: while in the esm part of the documentation it says the module format always respect the type field for extension files, in the cjs part of the documentation it has been including this exception that's not present in the spec, as far back as v16 from #41383 - so yargs is technically following a documented exception, the problem is more on the Node.js side that it has conflicting documentation.
#61600 made the extensionless behavior match what's in the ESM spec (so reverting it is widening this exception to ESM, not eliminating it) without counting this exception in the CJS documentation, and this PR only adds that CJS exception back. I agree the saner behavior would be to respect type field always (this is the status quo after #61600 fixed the mismatch), and remove this exception from the CJS documentation, but that should be a semver-major. For existing releases, the best we can do IMO is to combine #61600 and this PR to reduce the exception surface without introducing a regression. |
| if (typeFromPjson === 'commonjs') { | ||
| format = typeFromPjson; | ||
| } |
There was a problem hiding this comment.
Would it be possible to have extensionless files entirely ignore the package.json "type"? That seems like a more pragmatic approach given that they are designed to exist outside of package boundaries in most "binary execution" use cases.
I think let's just revert that commit and take our time with this. If we're going to change behaviors, we need to not just update the documentation but also do more thorough checking that we're not breaking some other package with the new behavior.
I agree with this. The behavior in #61600 feels like the correct behavior to me, so I think we should revert for now and re-land as semver-major; and maybe it needs a slight tweak if there are documentation inconsistencies that we need to fix. |
|
If we revert #61600 this means we will need to re-open #61104 - that is, if type is commonjs, an extensionless ESM will now exit silently and not even getting run or emitting any errors. Is that the behavior we want to keep though? And doing that can also be considered semver-major, because it's intentionally doing something against the documentation (and not covered by any documented exception). If we rever that commit, we are only trading one regression with another, and now have two cases deviating from the spec, one of them not documented anywhere. If we keep that commit and apply this PR, we can fix both issues, and only have one documented exception. It doesn't seem necessary to intentionally regress and go against the documentation when there is a way not to. |
I think this commit is a much more dramatic regression against our documentation. Syntax detection is never supposed to be enabled when a Node 25 will go into maintenance mode next month, and will be EOL a few months after that. So what if we revert to the behavior of a few weeks ago, and apply the proper fix as semver-major in Node 26 next month? I think that's better than creating a much messier situation in 25. |
I think you read it the other way around - keeping it as is means we don't do syntax detection on type: commonjs extensions files; reverting it would make it detect the syntax and swallow the syntax errors again (but not even execute the files, it's just a buggy no-op). So by suggesting to revert you are suggesting to re-enable buggy syntax detection for all required extensionless files. On the other hand keeping that commit and apply this PR means that buggy syntax detection is only allowed for the documented exception for required extension less CJS with type: module, while still removing the exception for type: commonjs which is not documented anywhere.
I think to achieve the "remove the CJS exception and make extensionless respect the type field", the action should be: keep that commit as is on main, and apply this PR on 25. Then in 26 just remove the exception in the documentation. That commit is what make extensionless files respect type field, reverting it is re-enabling syntax detection for files with a type field (that is not even working correctly). |
I think this could work. Can you spell it out a bit more? How would we make this PR apply only to 25, and not 26 or any other line? And what docs changes (and possible further behavior changes) would we make that would be part of 26? What I'm wondering is, what's the desired behavior that we want in 26. Ideally it would both fix #61104 and close loopholes/confusing exceptions. I don't really care all that much what happens in 25 for the remainder of its brief life, so if people prefer the behavior of this PR over just reverting #61600, that's fine with me. But I'd like to reach a consensus on what we really want the behaviors to be, even if semver-major changes are needed, so that we can get that done for 26. |
Re-targeting this PR to v25-staging should be enough. For 26 we just need a semver-major PR on the main branch to remove that exception from the docs so that the docs align with the behavior again (though I don't think a doc-only change have to be as semver-major, from a doc POV it's can also be considered a semver-patch for doc to align with the ESM resolution specification, just needs to be dont-land-25 and below). |
This works for me; does it work for you @mcollina? @joyeecheung doesn't this PR also need a docs update? The description mentions new behaviors, but I don't think the new behaviors are what's currently documented.
Which exception, exactly? And that's the only change, just documentation? So in other words, the current state on Personally I don't see why it's better to have some third state be the behavior for the remaining 25.x releases, when we can just revert to the behavior of a few weeks ago for the rest of 25; but this is not something I'll insist on. |
This one https://nodejs.org/docs/v25.7.0/api/modules.html#enabling
Not really. I think you need to take the PR description with a grain of salt because it appears to be generated by an LLM (e.g. the V8's syntax detection part is clearly incorrect, syntax detection is implemented by Node.js as you must know ;)). This PR is just implementing the aforementioned documented exception back, this exception is a behavior that has been documented since v16 so hardly new. And since that exception is still in the docs this does not need to update it. At most we can add a note saying that it's conflicting with a different part of the documentation but I think that might add more confusion if it's going away soon. |
The base branch was changed.
|
@mcollina This still targets main; did you intend to target 25-staging? |
PR nodejs#61600 made the CJS loader respect the package.json type field for extensionless files, which fixed nodejs#61104 but also enforced type: "module" for extensionless files. This broke CJS extensionless files inside ESM packages (e.g. yargs v17). Only enforce type: "commonjs" for extensionless files. For type: "module", leave format undefined so syntax detection handles it. This restores the documented exception in the CJS documentation that has existed since v16: extensionless files in type: "module" packages are recognized as CJS when included via require(). Fixes: nodejs#61971 Refs: yargs/yargs#2509 Refs: nodejs#61600 PR-URL: nodejs#62083
e208e3b to
9842d3e
Compare
| // syntax surfaces as SyntaxError. For type: module, leave format undefined so our syntax | ||
| // detection handles it (allowing CJS extensionless files in ESM packages). |
There was a problem hiding this comment.
Does our syntax detection handle it? I think to confirm that it does, this PR also needs a test that requiring an extensionless file with ESM syntax in a type: module package scope also works.
This is also the part that worries me, though: according to our docs, syntax detection happens for
Files with a
.jsextension or no extension; and either no controllingpackage.jsonfile or one that lacks a type field.
Assuming the comment here does what it claims to do, then this PR is creating a new exception to our docs: syntax detection applies in the above case, and for the string input, and for extensionless files in a type: module context. I thought we wanted syntax detection to never apply when a type field was set; that's what the warning we print tells users to add to avoid the performance hit of syntax detection. That's why this PR in general feels like the wrong solution to me.
Alternatively if this comment is just wrong and there's no syntax detection involved here, then we should fix this comment (and the PR description).
There was a problem hiding this comment.
this PR is creating a new exception to our docs: syntax detection applies in the above case, and for the string input, and for extensionless files in a type: module context.
I think this needs to factor in that it's not a new exception, it's still the status quo for v20-v24. #61600 removed the exceptions on v25, and this PR is only partially putting a small part of the exception back to prevent regressions on v25, since that part is documented (but not the others). In v26 the exception will be fully disallowed since it'll have #61600 in full.
Alternatively, if we want to forbid syntax detection entirely, to make it strictly follow the documented exception it should change to if (typeFromPjson === 'module') format = 'commonjs', but that feels even more wrong and it can only happen on v25....syntax detection is mostly just a mechanics to relax the interpretation of the type field to be less wrong, instead of completely going the opposite of what type field says (which is what got documented) and forbid extensionless ESM when there's type: module (meanwhile, without type in package.json it still allows extensionless ESM since 20).
There was a problem hiding this comment.
If we're currently doing any syntax detection for files where the controlling package.json file has a type field, that's a bug. That's not documented in the syntax detection section, nor in https://nodejs.org/docs/v25.7.0/api/modules.html#enabling. The type field is supposed to prevent syntax detection, full stop, for all files including extensionless ones. That's why we print this:
node/lib/internal/modules/esm/get_format.js
Lines 148 to 150 in 17c76c1
So I'm not sure what exception you're referring to? Are the docs and this warning all wrong and we've been secretly doing syntax detection for extensionless files in type: module scopes prior to #61600?
There was a problem hiding this comment.
Are the docs and this warning all wrong and we've been secretly doing syntax detection for extensionless files in type: module scopes prior to #61600?
Yes, or more precisely, we had been ignoring the type field for required extensionless files before 61600 - and still do on v20-v24. We didn't even read the package.json's type field at all for required extensionless files until 61600. That's why I say reverting it is going to make it more wrong.
As it turned out, if the type field is module, treating the required extensionless file as ESM caused a regression from the documented exception, so we have two choices to fix it for 25.
- If the type field is module, treat the required file as commonjs (this sounds even weirder, but it's what the doc says, and what yargs 17 expects). Note that this could still be breaking because it then disallows required extensionless ESM even with type: module, which has been available on v20-24.
- If type field is module, ignore the type field (what this PR does), so required extensionless files under this package.json can be either commonjs (what yargs 17 needs) or ESM (less weird for other cases, and what actually respecting the type field would've allowed), which is also the behavior we have for v20-v24.
Note that this is unrelated to entry points, and only for required extensionless files.
Remove the documented exception that extensionless files in type: "module" packages are recognized as CommonJS when included via require(). This exception conflicted with the ESM resolution specification which states that extensionless files within a package scope with an explicit type field follow the format of the type field. The behavior on main already matches the ESM spec since nodejs#61600, this change aligns the CJS documentation accordingly. Refs: nodejs#61600 Refs: nodejs#62083
PR nodejs#61600 made the CJS loader respect the package.json type field for extensionless files, which fixed nodejs#61104 but also enforced type: "module" for extensionless files. This broke CJS extensionless files inside ESM packages (e.g. yargs v17). Only enforce type: "commonjs" for extensionless files. For type: "module", leave format undefined so syntax detection handles it. This restores the documented exception in the CJS documentation that has existed since v16: extensionless files in type: "module" packages are recognized as CJS when included via require(). Fixes: nodejs#61971 Refs: yargs/yargs#2509 Refs: nodejs#61600 PR-URL: nodejs#62083
9842d3e to
a492b3d
Compare
a492b3d to
9842d3e
Compare
|
The commit message needs to be rewrapped <= 72 columns |
Please don't, I'll take care of it upon landing, I don't want to re-run the CI |
|
Landed in ea87eea |
PR #61600 made the CJS loader respect the `package.json` `type` field for extensionless files, which fixed #61104 but also enforced `type: "module"` for extensionless files. This broke CJS extensionless files inside ESM packages (e.g. yargs v17).
Only enforce `type: "commonjs"` for extensionless files. For `type: "module"`, leave format undefined so syntax detection handles it. This restores the documented exception in the CJS documentation that has existed since v16:
This PR is targeted at `v25.x-staging` only. On `main` (v26), #61600 stays as-is and the exception should be removed from the CJS documentation in a separate PR to align the docs with the ESM resolution specification.
Fixes: #61971
Refs: yargs/yargs#2509
Refs: #61600