X Tutup
Skip to content

Add AsyncPCK support (+ Web implementation)#114690

Open
adamscott wants to merge 5 commits intogodotengine:masterfrom
adamscott:async-resource-loader-mk6
Open

Add AsyncPCK support (+ Web implementation)#114690
adamscott wants to merge 5 commits intogodotengine:masterfrom
adamscott:async-resource-loader-mk6

Conversation

@adamscott
Copy link
Member

@adamscott adamscott commented Jan 7, 2026

Note: This PR has been superseded by #116673.


Table of Contents

Introducing AsyncPCKs.

What are AsyncPCKs?

This PR adds AsyncPCK as a new resource pack "format" for Godot.

AsyncPCK are essentially an "unpacked" version of standard .pck file in a .asyncpck directory instead.

Why AsyncPCKs are needed?

Currently, the Web platform export games essentially the same way desktop games do. On export, it creates these files:

  • the required Web files for the browser (HTML and style files).
  • Godot JavaScript files to interface between the engine and the browser.
  • a WebAssembly .wasm file containing the engine executable.
    • Equivalent to my_project.exe when exporting to Windows.
  • the assets of the project in a .pck file.
    • Equivalent to my_project.pck when exporting on desktop platforms.

On Desktop and Mobile platforms, disk accesses are essentially taken as granted. So, when asking for res://my_image.png, the Godot engine opens the .pck file and finds the image file at a given offset on the disk.

On the Web platform, resources are remote. This complicates things, as the engine (and most game engines) expect resources to be available near instantly.

To fix this issue, the Godot JavaScript files make sure to download and install (in the virtual filesystem for the WASM file) the project .pck file before launching the game. So everything's fine, right?

The problem for the Web platform lies with the size of that .pck file. It can get big quite quickly (by Web standards).

By default, the exported .pck contains every resource of the project. This means that the game will only start the game once every asset is loaded, including the final boss' ones. This is bad. This is not ideal because players are usually impatient, especially on specialized websites like Poki or CrazyGames. On these websites, players land on your game to play something, not necessarily to play your game in particular (vs opening a gamejam entry on itch.io).

How does it work?

As AsyncPCK are essentially just "unpacked" PCKs, it works 98% the same, but the files are exposed in a directory on export. The export process must add metadata for each exported asset. These <asset name>.deps.json files contain the list of every dependencies (recursively). The dependencies are checked only after export, as it's not possible to do so before on the export dialog (limitation of Godot due to the possibility of export plugins to modify or remap resources).

menu.gd.deps.json generated from Catburglar
{
  "dependencies": {
    "res://scripts/menu.gd": [
      "res://scripts/settings.gd",
      "res://sprites/ui/menu/cursor.png"
    ],
    "res://scripts/settings.gd": [],
    "res://sprites/ui/menu/cursor.png": []
  },
  "resources": {
    "res://scripts/menu.gd": {
      "files": {
        "res://scripts/menu.gd.remap": {
          "size": 39
        },
        "res://scripts/menu.gdc": {
          "size": 2613
        }
      },
      "totalSize": 2652
    },
    "res://scripts/settings.gd": {
      "files": {
        "res://scripts/settings.gd.remap": {
          "size": 43
        },
        "res://scripts/settings.gdc": {
          "size": 5634
        }
      },
      "totalSize": 5677
    },
    "res://sprites/ui/menu/cursor.png": {
      "files": {
        "res://.godot/imported/cursor.png-9b2f19c8ebb7ebafbffb55bc49257263.ctex": {
          "size": 150
        },
        "res://sprites/ui/menu/cursor.png.import": {
          "size": 195
        }
      },
      "totalSize": 345
    }
  }
}

This file is quite handy on the Web for the Godot JavaScript middlelayer. This makes it possible for it to download and install in the virutal filesystem every dependency listed.

Here's a diagram of the AsyncPCK install process as implemented for the Web platform.

---
title: "[Web] AsyncPCK install and loading process"
---
sequenceDiagram
    participant Game
    participant OS as OS<br>(singleton)
    participant Browser as Browser<br>(Godot middlelayer)
    participant Server

    Game ->>+ OS: `OS.async_pck_is_supported()`
    note over Game, OS: [Web] Returns `true`.<br>[Other] Returns `false`.
    OS ->>- Game: `true`

    Game ->>+ OS: `OS.async_pck_is_file_installable("res://level1.tscn")`
    note over Game, OS: [AsyncPCK export] Returns `true`.<br>[Standard PCK export] Returns `false`.
    OS ->>- Game: `true`

    %% [Install file]
    Game ->>+ OS: `OS.async_pck_install_file("res://level1.tscn")`

    OS ->> OS: Get the related PCK of "res://level1.tscn". 
    OS ->>+ Browser: Install "res://level1.tscn" from "index.asyncpck"

    Browser ->>+ Server: Fetch "index.asyncpck/assets/level1.tscn.deps.json"
    Server ->>- Browser: ["level1.tscn.deps.json" contents]
    note over Browser, Server: Lists "level1.tscn" (50KiB) and "level1.webp" (500KiB).

    Browser ->>+ Server: Fetch files listed in "level1.tscn.deps.json"

    Browser ->>- OS: `Error.OK`

    OS ->>- Game: `Error.OK`
    %% [Install file] END

    %% [Fetch progress]

    loop Fetch progress
        Server ->> Browser: Update fetch progress of files listed in "level1.tscn.deps.json".
    end

    %% [Fetch progress] END

    %% [Get status file]

    loop Update install status
        Game ->>+ OS: `OS.async_pck_install_file_get_status("res://level1.tscn")`
        OS ->> OS: Get the related PCK of "res://level1.tscn". 
        OS ->>+ Browser: Get install status of "res://level1.tscn" from "index.asyncpck"
        Browser ->>- OS: Install status of "res://level1.tscn"
        OS ->>- Game: Install status of "res://level1.tscn"
    end

    note right of Game: The game can display<br>a progress bar.

    %% [Get status file] END

    Server ->>- Browser: Fetch done.
    Browser ->> Browser: Install files in WASM virtual file system.

    note over Game, Browser: The game will realize that files are installed<br> in the [Update install status] loop.

    Game ->> Game: `load("res://level1.tscn")`

Loading

We are installing files, now?

"Install" is the most appropriate word I could find to describe this process, as "load" is already a term users actually use in their projects to load assets in their game.

In the context of AsyncPCKs, "installing" means to make the file available to "load". For the Web platform, it implies downloading the file first.

AsyncPCKInstaller node to the rescue!

In order to simplify this heavy install process for users, this PR also introduces the new AsyncPCKInstaller node.

It's a very simple node that does the heavy polling and checks in the background and triggers signals for users instead.

Inspector view Signals view
asyncpckinstaller_inspector asyncpckinstaller_signals

It's made to be compatible with non-AsyncPCK exports too! Ideally, you can create your game all around supporting AsyncPCKs even if they are not needed for specific exports. The node will just emit the signals signalling that the files are ready pretty much instantly.

See it for yourself (demo)

Link Initial load size
Sync Catburglar Link ~32MiB
Async Catburglar Link ~10MiB

Note

If you're interested in porting an existing game to the Web using this PR, please read the following about the "Resource Remaps" addon and about my plea to merge the feature into core.

"Resource Remaps", a must-have addon for porting to the Web

I'm talking about the Resource Remaps addon by @allenwp.

One of the first things I realized when I started using my PR to port Catburglar (itch.io link) was the need to be able to remap assets to one and another for the Web. In Catburglar, it uses a required intro video that is at least 1.5MiB in size. It's quite a chunky download that adds quite a lot to the initial download size.

I created a compressed version (it shows a little bit), but I didn't necessarily want to replace the original video. Following the "Godot way" of doing things, only one game project should be needed for every exports.

So I had two choices. The first was to get deep into the code in order to add dynamic checks to check the current platform and replace each place that requires the video (animation players, scenes, scripts and more) depending on the result. This is huge and really not efficient.

The other is just to remap the video to another one on export. This is exactly what the Resource Remaps addon does.

Honestly, as the Web platform is so different of others in terms of soft requirements, I really think that one cannot port an existing game to the Web without interchanging assets. Or even whole scenes.

That's why I hereby plead to merge this addon in the Engine natively. (i.e. to reopen godotengine/godot-proposals#10051)


Conclusion

This PR will revolutionize IMHO Godot's pertinence as a tool to create games to the Web platform. By making the initial download as small as possible, users will be able to use their favorite game engine to create legitimate and competitive games on the platform.


PR metadata
- Fixes godotengine/godot-proposals#13625

Also:
- Add a new node: `AsyncPCKInstaller`.
- Add Web platform support to export a project using AsyncPCK.
- Add Web platform backend code to support AsyncPCK `OS` calls.
@adamscott adamscott force-pushed the async-resource-loader-mk6 branch from 6024634 to 678cfcb Compare January 7, 2026 17:39
@allenwp
Copy link
Contributor

allenwp commented Jan 7, 2026

That's why I hereby plead to merge this addon in the Engine natively. (i.e. to reopen godotengine/godot-proposals#10051)

First off, I really appreciate that others are seeing my Resource Remaps implementation to be helpful and necessary!

Secondly, I'm curious if it might be possible to promote my plugin to be an official plugin and/or link to it directly from official docs, etc.? The benefit to this would be to allow the plugin to get some real-world use before being implemented "permanently" into core. The obvious downsides are needing to keep the plugin in your project source files and needing to install it to all projects.

That said, I would be very happy to help test a native port of this plugin! I wrote the scripts to be easy to do a 1:1 port into C++ code of the editor.

I'm indifferent about whether the plugin should be linked to, promoted to official, or integrated into core -- I just figured it would be good to mention some options that exist. We can maybe open a different proposal to discuss the options...

// TODO: Keep track of deps.
return List<String>();
const HashSet<String> &get_dependencies() const {
return dependencies;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sorry I haven't checked the non GDScript parts of this PR, but unless you added cyclic dependency support to the resource system this seems pretty dangerous.

See #111422 (comment)

Copy link
Member Author

@adamscott adamscott Jan 8, 2026

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For all the tests I did, I never encountered issues even once. But I may just be lucky.

Also, the feature kinda relies on this. Because it falls apart without it (if we can't detect which files are needed and such, we cannot know if a scene needs a file and makes the process really flimsy.)

cc. @vnen @dalexeev @mihe

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't say much about the ResourceLoader limitations, but there is some more history here. See #90643 and the corresponding PR (#90860) that ended up being reverted.

@AThousandShips

This comment was marked as resolved.

@adamscott adamscott force-pushed the async-resource-loader-mk6 branch from 59c4f53 to 33141b3 Compare January 8, 2026 14:55
@adamscott adamscott force-pushed the async-resource-loader-mk6 branch from 33141b3 to 3d766b5 Compare January 8, 2026 14:56
@michaelharmonart
Copy link

I know it's bad form to use this as a place for compliments, but...

This is FANTASTIC. I've fought with web load times for more hours than I care to admit, and I'm thrilled to see the problem being wrestled with from within Godot now!

Thanks all!

@BlooRabbit
Copy link

Async background loading is a super-desirable feature for web exports. In combination with an out-of-the-box brotli compression system, this could be a major step to get more Godot games shipped on the web.

@Calinou
Copy link
Member

Calinou commented Jan 9, 2026

We are installing files, now?

"Install" is the most appropriate word I could find to describe this process, as "load" is already a term users actually use in their projects to load assets in their game.

In the context of AsyncPCKs, "installing" means to make the file available to "load". For the Web platform, it implies downloading the file first.

This could be referred to as "registering". I think "installing" implies a filesystem operation that causes I/O to happen on disk (like the UNIX install command), while "registering" doesn't.

@allenwp
Copy link
Contributor

allenwp commented Jan 13, 2026

Or maybe "acquire", "obtain", or "attain". Acquire may be my preferred "load"/"download" alternative.

Edit: I guess AsyncPCKAquirer is super awkward, though. So maybe AsyncPCKObtainer, AsyncPCKAttainer, or AsyncPCKFetcher

I think I like AsyncPCKFetcher the most...

Comment on lines +3107 to +3124
if (p_path.ends_with(".gd")) {
String source = file->get_as_utf8_string();
if (source.is_empty()) {
return;
}
err = parser.parse(source, p_path, false);
} else {
// Path ends with ".gdc".
PackedByteArray source;
uint64_t source_size = FileAccess::get_size(p_path);
source.resize(source_size);
uint64_t actual_size = file->get_buffer(source.ptrw(), FileAccess::get_size(p_path));
if (source_size != actual_size) {
source.resize(actual_size);
}

err = parser.parse_binary(source, p_path);
}
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This does not account for builtin scripts.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right. I'm gonna check this out. Thanks!

@Ivorforce
Copy link
Member

Ivorforce commented Jan 21, 2026

We just discussed this PR (or rather its ideas and API design) in the core meeting.

Here are the meeting notes (by request of adam) for quick online reference.

If I may attempt a summary: The feature seems unquestionably useful, including the provided API. However, attendants raised that most users will likely not use this API, and therefore it would be nice to also integrate async loading into existing APIs made for async loading (as well as possible), and provide users with warnings about improving load times further.
In addition, there was some disagreement about whether misusing the async API on platforms without async pck implementations should perform the requests as expected, or error out (for consistency to web).

Allen: Which tools will we provide to work with / understand / debug async loading?

- Allen: For example, a tool that blocks loading as if you had a very slow internet connection

Holon: Suggested falling back to installing a file automatically if there is a load request, to make it easier to adopt this / not need to develop specifically for it

- e.g. ResourceLoader::load_threaded_request, ResourceLoader::load_threaded_get_status , ResourceLoader::load_threaded_get
  - Adam: This API is not available in single-threaded web games (which are most web games)

Holon: Voiced concerns about existing async resource loading APIs

- Adam: Most godot web games are single threaded, PR defers downloading to browser to make use of this 'multithreading'
- Holon: Freezing as a fallback would be fine worst case (games adopting this explicitly would of course deal with it better)
- Clay: The API should integrate with existing methods and synergize with it
  - For example, if somebody is already async loading, we can async load the pck (even in single-threaded web), and then sync-load into RAM
- Holon: The current async pck node is a power user implementation, normal users won't make much use of it
  - Integrate async loading with existing methods (e.g. threaded load functions, which in single threaded mode won't actually use threads but async functions)
    - ... But also offer the 'power user' option for those that want to squeeze out even better load times
- Adam: Integrating into existing APIs is interesting, but have concerns about mixing unrelated techniques
  - Would prefer to keep the API isolated for users that need it, but not integrate it

Allenwp: How to handle loading screens?

- Ivorius: Would be nice if it was really easy to do this, as a 'low effort' integration option

Clay: A silo'd off solution would not be adopted, async pcks should integrate strongly with Godot

- Adam: Disagree that it's silo'd off, users would just need to commit to the 'async workflow'
- Clay: Suggest the engine kicking off download automatically, and only start the actual load when it's done downloading

Clay: Pointed out similarity to downloading from GPU, where calling some APIs can stall the game

- These APIs return the result as expected, but print a warning to point the user to better APIs
  - Important point is, it works, just not perfectly. 
  - Adam: Would prefer to error out and not return the expected result
    - Clay: In this case, we would add a footgun where you make a game that works in the editor, but fails in web
    - Clay: Re-discuss some other time

Clay: When trying to load an async pck resource that is not 'installed' / downloaded yet, we should at minimum show an error like "the resource is part of an unloaded async pck"

- This should be in all platforms, not just web
- Otherwise, users will complain about 'bugs' in the engine / not understand why web isn't working as expected
- Adam: There are no async pcks for desktop
  - Clay: It should be the same anyway, just reject the load on desktop if not loaded explicitly
  - Adam: People should just use AsyncPck Node, then it 'just works'
- cerberus: People will spend most time testing locally on their platform, and only export to web at the end
  - Worried that games will break last minute when testing in web

@KoBeWi
Copy link
Member

KoBeWi commented Jan 21, 2026

I agree that loading a resource should install it automatically if not installed already (can print a warning or something, to inform the user). Also not sure why async loading needs a node. Like, you need 2 methods to handle installing (downloading) assets: is_installed() and install() (and maybe get_install_progress() to get progress). They can be in ResourceLoader and is_installed() would always return true in non-web platforms.

You didn't mention this, but in the meeting there was also an idea to rename load_threaded_request() to load_async_request(), to better convey that it's a generic async loading method, if we change it to automatically handle installing. If an asset is not yet installed, the "threaded" loading will simply take longer and require no changes to the code. The new install-related methods would optionally allow to better strategize when to install something.

As for debugging tools, the editor is able to start a server to host web build locally, no? Maybe it could be extended to allow simulating slow download times?

@allenwp
Copy link
Contributor

allenwp commented Jan 28, 2026

Quick amendment to my comment from the meeting: not only do slow downloads need to be testable by the game developer, but also unstable connections or connection drops. For example, if you enter a tunnel on a train and loose your internet connection entirely for a period of time. Some games may want to handle this gracefully or automatically via this feature's APIs.

@HolonProduction
Copy link
Member

Commenting here, because I can't get the other PR to build on my end, but probably the same issue applies:

I tested whether this PR reintroduces #91726 and indeed when initially importing the linked project I get error spam.

  • the errors are different than in the old issue, so maybe it's not GDScript related. Hard to say with such a big PR
  • the error spam does not occur after the initial import
  • there is no error spam in the project on master and 4.6

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Async PCKs support
X Tutup