X Tutup
Skip to content

OCPBUGS-77949: OCPBUGS-77948: TNF node replacement test updates#30846

Open
jaypoulz wants to merge 4 commits intoopenshift:mainfrom
jaypoulz:tnf-node-replacement-fixes
Open

OCPBUGS-77949: OCPBUGS-77948: TNF node replacement test updates#30846
jaypoulz wants to merge 4 commits intoopenshift:mainfrom
jaypoulz:tnf-node-replacement-fixes

Conversation

@jaypoulz
Copy link
Contributor

@jaypoulz jaypoulz commented Mar 6, 2026

  • Tightens up timeouts in core test loop
  • Fixes podman-etcd logging to feature human-readable output
  • Fixes a bug with IPv6 IP address formatting in URL

Summary by CodeRabbit

Release Notes

  • Tests
    • Enhanced node replacement test reliability with improved concurrency and timeout handling for job waits
    • Strengthened resource cleanup logic with finalizer-based deletion support
    • Expanded debugging capabilities with verbose pacemaker status logging and extended error reporting
    • Improved SSH command output logging with automatic truncation to prevent excessive log sizes
    • Added automatic pod log capture for job completion monitoring

jaypoulz added 4 commits March 6, 2026 15:11
…logs

- Use PcsStatusFull / PcsStatusFullViaDebug for human-readable pacemaker
  status instead of XML
- Truncate long SSH stdout/stderr in logs (e.g. avoid pcs status xml dumps)
- On WaitForNodesOnline timeout, log full pacemaker status for debugging

Made-with: Cursor
…nHostPort for IPv6

- Use net.JoinHostPort(RedfishIP, port) so IPv6 addresses are bracketed (RFC 3986)
- BMH template placeholder {REDFISH_HOST_PORT} replaces {REDFISH_IP}; port 8000 in code

Made-with: Cursor
…e replacement

- Capture job pod logs and gate on survivor update-setup run
- Force-delete stuck BMH/Machine after 1m; shorten recovery and BMH timeouts;
  rename timeouts to match duration (3m/7m/10m)
- Wait for survivor and target update-setup jobs in parallel

Made-with: Cursor
CSR approval was never observed in testing; remove the wait to simplify recovery.

Made-with: Cursor
@openshift-ci-robot
Copy link

Pipeline controller notification
This repo is configured to use the pipeline controller. Second-stage tests will be triggered either automatically or after lgtm label is added, depending on the repository configuration. The pipeline controller will automatically detect which contexts are required and will utilize /test Prow commands to trigger the second stage.

For optional jobs, comment /test ? to see a list of all defined jobs. To trigger manually all jobs from second stage use /pipeline required command.

This repository is configured in: automatic mode

@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 6, 2026
@openshift-ci-robot
Copy link

@jaypoulz: This pull request references Jira Issue OCPBUGS-77949, which is invalid:

  • expected the bug to target the "4.22.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

Details

In response to this:

  • Tightens up timeouts in core test loop
  • Fixes podman-etcd logging to feature human-readable output
  • Fixes a bug with IPv6 IP address formatting in URL

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@coderabbitai
Copy link

coderabbitai bot commented Mar 6, 2026

Walkthrough

Enhanced the two-node test suite with improved concurrency handling, extended timeout management, and better diagnostic logging. Updated BMC address construction, added new job and resource lifecycle helpers, and strengthened error reporting and cleanup mechanisms across test utilities.

Changes

Cohort / File(s) Summary
Test Configuration
test/extended/testdata/two_node/baremetalhost-template.yaml
Updated BMC address field to use parameterized host:port format instead of hardcoded IP and port.
Test Implementation
test/extended/two_node/tnf_node_replacement.go
Introduced parallelized resource waits, extended timeouts, new OpenShift resource lifecycle helpers (with finalizer removal), expanded BMC preparation logic using net.JoinHostPort, and improved Pacemaker cluster restoration flow.
Logging & SSH Utilities
test/extended/two_node/utils/core/ssh.go
Added output truncation for SSH command logs with configurable byte limit and truncation indicators.
Common Utilities
test/extended/two_node/utils/common.go
Updated Pacemaker status retrieval to use verbose debug output method.
Job & Resource Management
test/extended/two_node/utils/services/etcd.go
Added DumpJobPodLogs function, refactored WaitForJobCompletion with log dumping, and introduced WaitForSurvivorUpdateSetupJobCompletion with enhanced completion handling and pod creation time gating.
Pacemaker Status Utilities
test/extended/two_node/utils/services/pacemaker.go
Changed PcsStatusXMLResponse.Online field from bool to string, added PcsStatusFullViaDebug helper for debug-container-based status retrieval, and enhanced WaitForNodesOnline with post-timeout debugging logs.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~20 minutes

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly references both OCPBUGS-77949 and OCPBUGS-77948, and describes the change as 'TNF node replacement test updates', which aligns with the substantive changes across multiple test files and utilities that improve timeouts, logging, and IPv6 address handling in the node replacement test flow.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.
Stable And Deterministic Test Names ✅ Passed PR refactors test infrastructure without modifying Ginkgo test names to include dynamic information like pod names, timestamps, IPs, or random identifiers.
Test Structure And Quality ✅ Passed The node replacement test is structured as a single cohesive Ginkgo It block that exercises one end-to-end scenario, uses BeforeEach/AfterEach for setup and cleanup, passes explicit timeout parameters to all wait helpers, and all Expect calls include informative failure messages.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Post copyable unit tests in a comment

Warning

There were issues while running some tools. Please review the errors and either fix the tool's configuration or disable the tool if it's a critical failure.

🔧 golangci-lint (2.5.0)

Error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions
The command is terminated due to an error: can't load config: unsupported version of the configuration: "" See https://golangci-lint.run/docs/product/migration-guide for migration instructions


Comment @coderabbitai help to get the list of available commands and usage tips.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 6, 2026

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: jaypoulz
Once this PR has been reviewed and has the lgtm label, please assign xueqzhan for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@jaypoulz
Copy link
Contributor Author

jaypoulz commented Mar 6, 2026

/jira refresh

@openshift-ci-robot openshift-ci-robot added jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. and removed jira/invalid-bug Indicates that a referenced Jira bug is invalid for the branch this PR is targeting. labels Mar 6, 2026
@openshift-ci-robot
Copy link

@jaypoulz: This pull request references Jira Issue OCPBUGS-77949, which is valid. The bug has been moved to the POST state.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state ASSIGNED, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (dhensel@redhat.com), skipping review request.

Details

In response to this:

/jira refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci-robot
Copy link

@jaypoulz: This pull request references Jira Issue OCPBUGS-77949, which is valid.

3 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.22.0) matches configured target version for branch (4.22.0)
  • bug is in the state POST, which is one of the valid states (NEW, ASSIGNED, POST)

No GitHub users were found matching the public email listed for the QA contact in Jira (dhensel@redhat.com), skipping review request.

Details

In response to this:

  • Tightens up timeouts in core test loop
  • Fixes podman-etcd logging to feature human-readable output
  • Fixes a bug with IPv6 IP address formatting in URL

Summary by CodeRabbit

Release Notes

  • Tests
  • Enhanced node replacement test reliability with improved concurrency and timeout handling for job waits
  • Strengthened resource cleanup logic with finalizer-based deletion support
  • Expanded debugging capabilities with verbose pacemaker status logging and extended error reporting
  • Improved SSH command output logging with automatic truncation to prevent excessive log sizes
  • Added automatic pod log capture for job completion monitoring

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

Copy link

@coderabbitai coderabbitai bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@test/extended/two_node/utils/core/ssh.go`:
- Around line 136-141: The current log truncation slices the UTF-8 string bytes
with out[:maxLogOutputBytes], which can cut a multi-byte rune and produce
invalid UTF-8 in logs; update the truncation logic around stdout.String()/out
and the e2e.Logf call to perform rune-safe truncation (for example, convert to
runes or iterate runes until adding the next rune would exceed
maxLogOutputBytes) and then log the safely truncated string along with the total
byte length using the existing maxLogOutputBytes and e2e.Logf call sites.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Repository: openshift/coderabbit/.coderabbit.yaml

Review profile: CHILL

Plan: Pro

Run ID: 3ecbedc9-3c0f-4263-b6e6-a6a05314b2c2

📥 Commits

Reviewing files that changed from the base of the PR and between 35bab74 and 547c9eb.

📒 Files selected for processing (6)
  • test/extended/testdata/two_node/baremetalhost-template.yaml
  • test/extended/two_node/tnf_node_replacement.go
  • test/extended/two_node/utils/common.go
  • test/extended/two_node/utils/core/ssh.go
  • test/extended/two_node/utils/services/etcd.go
  • test/extended/two_node/utils/services/pacemaker.go

Comment on lines +136 to +141
out := stdout.String()
if len(out) > maxLogOutputBytes {
e2e.Logf("SSH stdout: %q ... (truncated, total %d bytes)", out[:maxLogOutputBytes], len(out))
} else {
e2e.Logf("SSH stdout: %q", out)
}
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

⚠️ Potential issue | 🟡 Minor

Minor: Truncation may split multi-byte UTF-8 characters.

Slicing out[:maxLogOutputBytes] by byte count could split a multi-byte UTF-8 character (e.g., if present in file paths or error messages), producing malformed output in logs. Since this is debug logging and the full output is returned to the caller unchanged, the impact is cosmetic.

💡 Optional fix using rune-safe truncation
-		if len(out) > maxLogOutputBytes {
-			e2e.Logf("SSH stdout: %q ... (truncated, total %d bytes)", out[:maxLogOutputBytes], len(out))
+		if len(out) > maxLogOutputBytes {
+			truncated := out[:maxLogOutputBytes]
+			// Ensure we don't split a UTF-8 multi-byte character
+			for len(truncated) > 0 && truncated[len(truncated)-1]&0xC0 == 0x80 {
+				truncated = truncated[:len(truncated)-1]
+			}
+			e2e.Logf("SSH stdout: %q ... (truncated, total %d bytes)", truncated, len(out))
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
out := stdout.String()
if len(out) > maxLogOutputBytes {
e2e.Logf("SSH stdout: %q ... (truncated, total %d bytes)", out[:maxLogOutputBytes], len(out))
} else {
e2e.Logf("SSH stdout: %q", out)
}
out := stdout.String()
if len(out) > maxLogOutputBytes {
truncated := out[:maxLogOutputBytes]
// Ensure we don't split a UTF-8 multi-byte character
for len(truncated) > 0 && truncated[len(truncated)-1]&0xC0 == 0x80 {
truncated = truncated[:len(truncated)-1]
}
e2e.Logf("SSH stdout: %q ... (truncated, total %d bytes)", truncated, len(out))
} else {
e2e.Logf("SSH stdout: %q", out)
}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@test/extended/two_node/utils/core/ssh.go` around lines 136 - 141, The current
log truncation slices the UTF-8 string bytes with out[:maxLogOutputBytes], which
can cut a multi-byte rune and produce invalid UTF-8 in logs; update the
truncation logic around stdout.String()/out and the e2e.Logf call to perform
rune-safe truncation (for example, convert to runes or iterate runes until
adding the next rune would exceed maxLogOutputBytes) and then log the safely
truncated string along with the total byte length using the existing
maxLogOutputBytes and e2e.Logf call sites.

@openshift-ci
Copy link
Contributor

openshift-ci bot commented Mar 6, 2026

@jaypoulz: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/verify 547c9eb link true /test verify

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

Copy link
Contributor

@eggfoobar eggfoobar left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great, just had some small suggestions.

automatedCleaningMode: metadata
bmc:
address: redfish+https://{REDFISH_IP}:8000/redfish/v1/Systems/{UUID}
address: redfish+https://{REDFISH_HOST_PORT}/redfish/v1/Systems/{UUID}
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe, we should just rename this to REDFISH_HOST, the port can cause confusion

// to empty so the test can proceed without blocking on controller cleanup.
func deleteOcResourceWithRetry(oc *exutil.CLI, resourceType, resourceName, namespace string) error {
return core.RetryWithOptions(func() error {
done := make(chan error, 1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of doing this timer, can we just use the old core.RetryWithOptions and if that fails we fall to force delete?

"{CREDENTIALS_NAME}": testConfig.TargetNode.BMCSecretName,
"{BOOT_MAC_ADDRESS}": newMACAddress,
"{BMH_NAME}": testConfig.TargetNode.BMHName,
"{REDFISH_HOST_PORT}": redfishHostPort,
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same here on updating to REDFISH_HOST

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants

X Tutup