X Tutup
Skip to content
This repository was archived by the owner on Mar 23, 2026. It is now read-only.

Logs: harden dict view iterations#13272

Merged
nik-localstack merged 1 commit intomainfrom
logs/improve-dictionary-handling
Oct 15, 2025
Merged

Logs: harden dict view iterations#13272
nik-localstack merged 1 commit intomainfrom
logs/improve-dictionary-handling

Conversation

@nik-localstack
Copy link
Contributor

@nik-localstack nik-localstack commented Oct 15, 2025 •
edited
Loading

Motivation

The following issue came up when looking in telemetry for issues with the new Batch service. Batch provider calls the DescribeLogGroups API when creating a new task.

exception while calling logs.DescribeLogGroups: dictionary changed size during iterationTraceback (most recent call last):
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/rolo/gateway/chain.py", line 166, in handle
    handler(self, self.context, response)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/handlers/service.py", line 116, in __call__
    handler(chain, context, response)
    ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/handlers/service.py", line 86, in __call__
    skeleton_response = self.skeleton.invoke(context)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/skeleton.py", line 155, in invoke
    return self.dispatch_request(serializer, context, instance)
           ~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/skeleton.py", line 169, in dispatch_request
    result = handler(context, instance) or {}
             ~~~~~~~^^^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/forwarder.py", line 137, in _call
    return handler(context, req)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/skeleton.py", line 117, in __call__
    return self.fn(*args, **kwargs)
           ~~~~~~~^^^^^^^^^^^^^^^^^
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/aws/api/core.py", line 181, in operation_marker
    return fn(*args, **kwargs)
  File "/opt/code/localstack/.venv/lib/python3.13/site-packages/localstack/services/logs/provider.py", line 108, in describe_log_groups
    copy_groups = copy.deepcopy(region_backend.groups)
  File "/usr/local/lib/python3.13/copy.py", line 137, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.13/copy.py", line 222, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/copy.py", line 163, in deepcopy
    y = _reconstruct(x, memo, *rv)
  File "/usr/local/lib/python3.13/copy.py", line 260, in _reconstruct
    state = deepcopy(state, memo)
  File "/usr/local/lib/python3.13/copy.py", line 137, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.13/copy.py", line 222, in _deepcopy_dict
    y[deepcopy(key, memo)] = deepcopy(value, memo)
                             ~~~~~~~~^^^^^^^^^^^^^
  File "/usr/local/lib/python3.13/copy.py", line 137, in deepcopy
    y = copier(x, memo)
  File "/usr/local/lib/python3.13/copy.py", line 221, in _deepcopy_dict
    for key, value in x.items():
                      ~~~~~~~^^
localstack.aws.api.core.CommonServiceException: exception while calling logs.DescribeLogGroups: dictionary changed size during iteration

The error occurs because the dictionary region_backend.groups is being modified by another thread while copy.deepcopy() is trying to iterate over it. This is a classic race condition where concurrent modifications to the dictionary cause the "dictionary changed size during iteration" error.

Changes

  • Fixes the race condition, by creating a shallow copy first to avoid the race condition, then do the deep copy:

@nik-localstack nik-localstack self-assigned this Oct 15, 2025
@nik-localstack nik-localstack added semver: patch Non-breaking changes which can be included in patch releases docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes labels Oct 15, 2025
@github-actions
Copy link

github-actions bot commented Oct 15, 2025

Test Results - Preflight, Unit

22 337 tests  ±0   20 587 ✅ ±0   15m 20s ⏱️ -31s
     1 suites ±0    1 750 💤 ±0 
     1 files   ±0        0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

@github-actions
Copy link

github-actions bot commented Oct 15, 2025

Test Results (amd64) - Acceptance

7 tests  ±0   5 ✅ ±0   3m 20s ⏱️ -4s
1 suites ±0   2 💤 ±0 
1 files   ±0   0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

@github-actions
Copy link

github-actions bot commented Oct 15, 2025

LocalStack Community integration with Pro

    2 files  ±    0      2 suites  ±0   53m 43s ⏱️ - 1h 7m 10s
2 016 tests  - 2 801  1 923 ✅  - 2 557  93 💤  - 244  0 ❌ ±0 
2 018 runs   - 2 801  1 923 ✅  - 2 557  95 💤  - 244  0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

This pull request removes 2801 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

@github-actions
Copy link

github-actions bot commented Oct 15, 2025

Test Results (amd64) - Integration, Bootstrap

    5 files  ±    0      5 suites  ±0   1h 17m 4s ⏱️ - 1h 24m 15s
2 040 tests  - 3 151  1 949 ✅  - 2 745  91 💤  - 406  0 ❌ ±0 
2 046 runs   - 3 151  1 949 ✅  - 2 745  97 💤  - 406  0 ❌ ±0 

Results for commit f49043e. ± Comparison against base commit 395f363.

This pull request removes 3151 tests. 
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_lambda_dynamodb
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_opensearch_crud
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_search_books
tests.aws.scenario.bookstore.test_bookstore.TestBookstoreApplication ‑ test_setup
tests.aws.scenario.kinesis_firehose.test_kinesis_firehose.TestKinesisFirehoseScenario ‑ test_kinesis_firehose_s3
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_destination_sns
tests.aws.scenario.lambda_destination.test_lambda_destination_scenario.TestLambdaDestinationScenario ‑ test_infra
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_prefill_dynamodb_table
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input0-SUCCEEDED]
tests.aws.scenario.loan_broker.test_loan_broker.TestLoanBrokerScenario ‑ test_stepfunctions_input_recipient_list[step_function_input1-SUCCEEDED]
…

@nik-localstack nik-localstack marked this pull request as ready for review October 15, 2025 11:14
pinzon
pinzon approved these changes Oct 15, 2025
View reviewed changes
Copy link
Member

@pinzon pinzon left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@nik-localstack nik-localstack merged commit 814a597 into main Oct 15, 2025
56 checks passed
@nik-localstack nik-localstack deleted the logs/improve-dictionary-handling branch October 15, 2025 13:50
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Reviewers

@steffyP steffyP Awaiting requested review from steffyP

1 more reviewer

@pinzon pinzon pinzon approved these changes

Reviewers whose approvals may not affect merge requirements

Assignees

@nik-localstack nik-localstack

Labels

docs: skip Pull request does not require documentation changes notes: skip Pull request does not have to be mentioned in the release notes semver: patch Non-breaking changes which can be included in patch releases

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@nik-localstack @pinzon
X Tutup