Add AsyncPCK support (+ Web implementation)#114690
Add AsyncPCK support (+ Web implementation)#114690adamscott wants to merge 5 commits intogodotengine:masterfrom
Conversation
a9d5e51 to
6024634
Compare
Also: - Add a new node: `AsyncPCKInstaller`. - Add Web platform support to export a project using AsyncPCK. - Add Web platform backend code to support AsyncPCK `OS` calls.
6024634 to
678cfcb
Compare
First off, I really appreciate that others are seeing my Resource Remaps implementation to be helpful and necessary! Secondly, I'm curious if it might be possible to promote my plugin to be an official plugin and/or link to it directly from official docs, etc.? The benefit to this would be to allow the plugin to get some real-world use before being implemented "permanently" into core. The obvious downsides are needing to keep the plugin in your project source files and needing to install it to all projects. That said, I would be very happy to help test a native port of this plugin! I wrote the scripts to be easy to do a 1:1 port into C++ code of the editor. I'm indifferent about whether the plugin should be linked to, promoted to official, or integrated into core -- I just figured it would be good to mention some options that exist. We can maybe open a different proposal to discuss the options... |
| // TODO: Keep track of deps. | ||
| return List<String>(); | ||
| const HashSet<String> &get_dependencies() const { | ||
| return dependencies; |
There was a problem hiding this comment.
Sorry I haven't checked the non GDScript parts of this PR, but unless you added cyclic dependency support to the resource system this seems pretty dangerous.
There was a problem hiding this comment.
For all the tests I did, I never encountered issues even once. But I may just be lucky.
Also, the feature kinda relies on this. Because it falls apart without it (if we can't detect which files are needed and such, we cannot know if a scene needs a file and makes the process really flimsy.)
This comment was marked as resolved.
This comment was marked as resolved.
59c4f53 to
33141b3
Compare
33141b3 to
3d766b5
Compare
|
I know it's bad form to use this as a place for compliments, but... This is FANTASTIC. I've fought with web load times for more hours than I care to admit, and I'm thrilled to see the problem being wrestled with from within Godot now! Thanks all! |
|
Async background loading is a super-desirable feature for web exports. In combination with an out-of-the-box brotli compression system, this could be a major step to get more Godot games shipped on the web. |
This could be referred to as "registering". I think "installing" implies a filesystem operation that causes I/O to happen on disk (like the UNIX |
|
Or maybe "acquire", "obtain", or "attain". Acquire may be my preferred "load"/"download" alternative. Edit: I guess I think I like |
| if (p_path.ends_with(".gd")) { | ||
| String source = file->get_as_utf8_string(); | ||
| if (source.is_empty()) { | ||
| return; | ||
| } | ||
| err = parser.parse(source, p_path, false); | ||
| } else { | ||
| // Path ends with ".gdc". | ||
| PackedByteArray source; | ||
| uint64_t source_size = FileAccess::get_size(p_path); | ||
| source.resize(source_size); | ||
| uint64_t actual_size = file->get_buffer(source.ptrw(), FileAccess::get_size(p_path)); | ||
| if (source_size != actual_size) { | ||
| source.resize(actual_size); | ||
| } | ||
|
|
||
| err = parser.parse_binary(source, p_path); | ||
| } |
There was a problem hiding this comment.
This does not account for builtin scripts.
There was a problem hiding this comment.
You're right. I'm gonna check this out. Thanks!
|
We just discussed this PR (or rather its ideas and API design) in the core meeting. Here are the meeting notes (by request of adam) for quick online reference. If I may attempt a summary: The feature seems unquestionably useful, including the provided API. However, attendants raised that most users will likely not use this API, and therefore it would be nice to also integrate async loading into existing APIs made for async loading (as well as possible), and provide users with warnings about improving load times further. |
|
I agree that loading a resource should install it automatically if not installed already (can print a warning or something, to inform the user). Also not sure why async loading needs a node. Like, you need 2 methods to handle installing (downloading) assets: You didn't mention this, but in the meeting there was also an idea to rename As for debugging tools, the editor is able to start a server to host web build locally, no? Maybe it could be extended to allow simulating slow download times? |
|
Quick amendment to my comment from the meeting: not only do slow downloads need to be testable by the game developer, but also unstable connections or connection drops. For example, if you enter a tunnel on a train and loose your internet connection entirely for a period of time. Some games may want to handle this gracefully or automatically via this feature's APIs. |
|
Commenting here, because I can't get the other PR to build on my end, but probably the same issue applies: I tested whether this PR reintroduces #91726 and indeed when initially importing the linked project I get error spam.
|
Note: This PR has been superseded by #116673.
Table of Contents
AsyncPCKInstallernode to the rescue!Introducing AsyncPCKs.
What are AsyncPCKs?
This PR adds AsyncPCK as a new resource pack "format" for Godot.
AsyncPCK are essentially an "unpacked" version of standard
.pckfile in a.asyncpckdirectory instead.Why AsyncPCKs are needed?
Currently, the Web platform export games essentially the same way desktop games do. On export, it creates these files:
.wasmfile containing the engine executable.my_project.exewhen exporting to Windows..pckfile.my_project.pckwhen exporting on desktop platforms.On Desktop and Mobile platforms, disk accesses are essentially taken as granted. So, when asking for
res://my_image.png, the Godot engine opens the.pckfile and finds the image file at a given offset on the disk.On the Web platform, resources are remote. This complicates things, as the engine (and most game engines) expect resources to be available near instantly.
To fix this issue, the Godot JavaScript files make sure to download and install (in the virtual filesystem for the WASM file) the project
.pckfile before launching the game. So everything's fine, right?The problem for the Web platform lies with the size of that
.pckfile. It can get big quite quickly (by Web standards).By default, the exported
.pckcontains every resource of the project. This means that the game will only start the game once every asset is loaded, including the final boss' ones. This is bad. This is not ideal because players are usually impatient, especially on specialized websites like Poki or CrazyGames. On these websites, players land on your game to play something, not necessarily to play your game in particular (vs opening a gamejam entry on itch.io).How does it work?
As AsyncPCK are essentially just "unpacked" PCKs, it works 98% the same, but the files are exposed in a directory on export. The export process must add metadata for each exported asset. These
<asset name>.deps.jsonfiles contain the list of every dependencies (recursively). The dependencies are checked only after export, as it's not possible to do so before on the export dialog (limitation of Godot due to the possibility of export plugins to modify or remap resources).menu.gd.deps.jsongenerated from Catburglar{ "dependencies": { "res://scripts/menu.gd": [ "res://scripts/settings.gd", "res://sprites/ui/menu/cursor.png" ], "res://scripts/settings.gd": [], "res://sprites/ui/menu/cursor.png": [] }, "resources": { "res://scripts/menu.gd": { "files": { "res://scripts/menu.gd.remap": { "size": 39 }, "res://scripts/menu.gdc": { "size": 2613 } }, "totalSize": 2652 }, "res://scripts/settings.gd": { "files": { "res://scripts/settings.gd.remap": { "size": 43 }, "res://scripts/settings.gdc": { "size": 5634 } }, "totalSize": 5677 }, "res://sprites/ui/menu/cursor.png": { "files": { "res://.godot/imported/cursor.png-9b2f19c8ebb7ebafbffb55bc49257263.ctex": { "size": 150 }, "res://sprites/ui/menu/cursor.png.import": { "size": 195 } }, "totalSize": 345 } } }This file is quite handy on the Web for the Godot JavaScript middlelayer. This makes it possible for it to download and install in the virutal filesystem every dependency listed.
Here's a diagram of the
AsyncPCKinstall process as implemented for the Web platform.--- title: "[Web] AsyncPCK install and loading process" --- sequenceDiagram participant Game participant OS as OS<br>(singleton) participant Browser as Browser<br>(Godot middlelayer) participant Server Game ->>+ OS: `OS.async_pck_is_supported()` note over Game, OS: [Web] Returns `true`.<br>[Other] Returns `false`. OS ->>- Game: `true` Game ->>+ OS: `OS.async_pck_is_file_installable("res://level1.tscn")` note over Game, OS: [AsyncPCK export] Returns `true`.<br>[Standard PCK export] Returns `false`. OS ->>- Game: `true` %% [Install file] Game ->>+ OS: `OS.async_pck_install_file("res://level1.tscn")` OS ->> OS: Get the related PCK of "res://level1.tscn". OS ->>+ Browser: Install "res://level1.tscn" from "index.asyncpck" Browser ->>+ Server: Fetch "index.asyncpck/assets/level1.tscn.deps.json" Server ->>- Browser: ["level1.tscn.deps.json" contents] note over Browser, Server: Lists "level1.tscn" (50KiB) and "level1.webp" (500KiB). Browser ->>+ Server: Fetch files listed in "level1.tscn.deps.json" Browser ->>- OS: `Error.OK` OS ->>- Game: `Error.OK` %% [Install file] END %% [Fetch progress] loop Fetch progress Server ->> Browser: Update fetch progress of files listed in "level1.tscn.deps.json". end %% [Fetch progress] END %% [Get status file] loop Update install status Game ->>+ OS: `OS.async_pck_install_file_get_status("res://level1.tscn")` OS ->> OS: Get the related PCK of "res://level1.tscn". OS ->>+ Browser: Get install status of "res://level1.tscn" from "index.asyncpck" Browser ->>- OS: Install status of "res://level1.tscn" OS ->>- Game: Install status of "res://level1.tscn" end note right of Game: The game can display<br>a progress bar. %% [Get status file] END Server ->>- Browser: Fetch done. Browser ->> Browser: Install files in WASM virtual file system. note over Game, Browser: The game will realize that files are installed<br> in the [Update install status] loop. Game ->> Game: `load("res://level1.tscn")`We are installing files, now?
"Install" is the most appropriate word I could find to describe this process, as "load" is already a term users actually use in their projects to load assets in their game.
In the context of AsyncPCKs, "installing" means to make the file available to "load". For the Web platform, it implies downloading the file first.
AsyncPCKInstallernode to the rescue!In order to simplify this heavy install process for users, this PR also introduces the new
AsyncPCKInstallernode.It's a very simple node that does the heavy polling and checks in the background and triggers signals for users instead.
It's made to be compatible with non-AsyncPCK exports too! Ideally, you can create your game all around supporting AsyncPCKs even if they are not needed for specific exports. The node will just emit the signals signalling that the files are ready pretty much instantly.
See it for yourself (demo)
Note
If you're interested in porting an existing game to the Web using this PR, please read the following about the "Resource Remaps" addon and about my plea to merge the feature into core.
"Resource Remaps", a must-have addon for porting to the Web
I'm talking about the Resource Remaps addon by @allenwp.
One of the first things I realized when I started using my PR to port Catburglar (itch.io link) was the need to be able to remap assets to one and another for the Web. In Catburglar, it uses a required intro video that is at least 1.5MiB in size. It's quite a chunky download that adds quite a lot to the initial download size.
I created a compressed version (it shows a little bit), but I didn't necessarily want to replace the original video. Following the "Godot way" of doing things, only one game project should be needed for every exports.
So I had two choices. The first was to get deep into the code in order to add dynamic checks to check the current platform and replace each place that requires the video (animation players, scenes, scripts and more) depending on the result. This is huge and really not efficient.
The other is just to remap the video to another one on export. This is exactly what the Resource Remaps addon does.
Honestly, as the Web platform is so different of others in terms of soft requirements, I really think that one cannot port an existing game to the Web without interchanging assets. Or even whole scenes.
That's why I hereby plead to merge this addon in the Engine natively. (i.e. to reopen godotengine/godot-proposals#10051)
Conclusion
This PR will revolutionize IMHO Godot's pertinence as a tool to create games to the Web platform. By making the initial download as small as possible, users will be able to use their favorite game engine to create legitimate and competitive games on the platform.
PR metadata
- Fixes godotengine/godot-proposals#13625