X Tutup
Skip to content
This repository was archived by the owner on Mar 23, 2026. It is now read-only.

CFn: configure wait time for polling resources#13455

Merged
simonrw merged 3 commits intomainfrom
cfn/configurable-sleep-time
Dec 4, 2025
Merged

CFn: configure wait time for polling resources#13455
simonrw merged 3 commits intomainfrom
cfn/configurable-sleep-time

Conversation

@simonrw
Copy link
Contributor

@simonrw simonrw commented Dec 2, 2025

Motivation

While deploying our re:Invent demo stack I found it takes ~85 seconds for a warm deploy on my system. This time is dominated by deploying lambda functions which each take 5 seconds.

When we deploy resources, we create a loop where

  • the first iteration deploys the resource
  • the remaining iterations poll the resource for completion

AWS CloudFormation waits between polling loops (to reduce burden on their networking and services). On LocalStack we don't need to do this - a lot of our resources are in-memory so are instantly available. Therefore we set the sleep time for the first iteration of this loop only to 0 seconds, and wait 5 seconds for every other iteration.

Since the lambda functions take a minimum of 5 seconds each, I suspect the deployment loop is going round more than twice.

Increasing the number of loop iterations that we don't sleep for would reduce this deployment time greatly.

Changes

  • Add a configurable CFN_NO_WAIT_ITERATIONS to control the number of loop iterations we don't wait 5 seconds between, defaulting to 5 iterations
  • This change was added to the legacy provider since the resource provider executor is shared between the two
  • Decrease the default sleep time from 5 to 1 second, which should speed everything up (thanks @dfangl for the suggestion)
  • Now that we are iterating quicker, I have increased the minimum number of iterations to 10 (if for some reason the user specifies a CFN_PER_RESOURCE_TIMEOUT of e.g. 0!)

After these changes, our re:Invent demo stack takes ~30 seconds, leading to a 2.8x total speedup 🎉

Tests

  • Test deploying a large stack with lambda functions with this setting at its previous default of 1, and with the new default of 5 and observe the total time to deploy has decreased

Related

Resolves UNC-122

@simonrw simonrw added semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes notes: needed Pull request should be mentioned in the release notes labels Dec 2, 2025
CFN_IGNORE_UNSUPPORTED_RESOURCE_TYPES = is_env_not_false("CFN_IGNORE_UNSUPPORTED_RESOURCE_TYPES")

# Decrease the waiting time for resource deployment
CFN_NUM_NO_WAIT: str | int | None = os.environ.get("CFN_NUM_NO_WAIT")
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am not happy with this name. Any better ideas?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CFN_NO_WAIT_ITERATIONS

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah that's better, thanks!

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

S3 Image Test Results (AMD64 / ARM64)

    2 files    2 suites   8m 13s ⏱️
  544 tests 492 ✅  52 💤 0 ❌
1 088 runs  984 ✅ 104 💤 0 ❌

Results for commit ecee01c.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

Test Results - Preflight, Unit

22 889 tests  ±0   21 075 ✅ ±0   6m 19s ⏱️ -3s
     1 suites ±0    1 814 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit ecee01c. ± Comparison against base commit 9bf66b4.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 3s ⏱️ -24s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit ecee01c. ± Comparison against base commit 9bf66b4.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

Test Results - Alternative Providers

587 tests   331 ✅  18m 11s ⏱️
  1 suites  256 💤
  1 files      0 ❌

Results for commit ecee01c.

♻️ This comment has been updated with latest results.

@simonrw simonrw added docs: needed Pull request requires documentation updates and removed docs: skip Pull request does not require documentation changes labels Dec 2, 2025
@github-actions
Copy link

github-actions bot commented Dec 2, 2025

Test Results (amd64) - Integration, Bootstrap

    5 files      5 suites   2h 33m 43s ⏱️
5 478 tests 4 926 ✅ 552 💤 0 ❌
5 484 runs  4 926 ✅ 558 💤 0 ❌

Results for commit ecee01c.

♻️ This comment has been updated with latest results.

@github-actions
Copy link

github-actions bot commented Dec 2, 2025

LocalStack Community integration with Pro

    2 files  ±0      2 suites  ±0   1h 56m 14s ⏱️ - 7m 46s
5 104 tests ±0  4 712 ✅ ±0  392 💤 ±0  0 ❌ ±0 
5 106 runs  ±0  4 712 ✅ ±0  394 💤 ±0  0 ❌ ±0 

Results for commit ecee01c. ± Comparison against base commit 9bf66b4.

♻️ This comment has been updated with latest results.

@simonrw simonrw marked this pull request as ready for review December 2, 2025 14:38
@simonrw simonrw force-pushed the cfn/configurable-sleep-time branch 2 times, most recently from 263dad2 to c299bf7 Compare December 3, 2025 13:05
Copy link
Member

@pinzon pinzon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice improvements 👍

CFN_IGNORE_UNSUPPORTED_RESOURCE_TYPES = is_env_not_false("CFN_IGNORE_UNSUPPORTED_RESOURCE_TYPES")

# Decrease the waiting time for resource deployment
CFN_NUM_NO_WAIT: str | int | None = os.environ.get("CFN_NUM_NO_WAIT")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

CFN_NO_WAIT_ITERATIONS

if current_iteration == 0:
time.sleep(0)
# This should have been set up in the provider
assert isinstance(config.CFN_NUM_NO_WAIT, int)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Question: is it necessary to assert this in every iteration?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

That's a fair point, thanks!

@simonrw simonrw force-pushed the cfn/configurable-sleep-time branch from c299bf7 to 35bd89f Compare December 3, 2025 20:39
Since we perform 5 updates in quick succession, we probably want to use
a higher number of minimum iterations
@simonrw simonrw force-pushed the cfn/configurable-sleep-time branch from 35bd89f to ecee01c Compare December 3, 2025 20:42
@simonrw simonrw merged commit fe00d2b into main Dec 4, 2025
72 of 73 checks passed
@simonrw simonrw deleted the cfn/configurable-sleep-time branch December 4, 2025 00:59
@simonrw simonrw added this to the 4.12 milestone Dec 4, 2025
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

docs: needed Pull request requires documentation updates notes: needed Pull request should be mentioned in the release notes semver: patch Non-breaking changes which can be included in patch releases

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants

X Tutup