Enhanced worker crash handling with integrated crash telemetry

[raujaiswal]

Context

#AB2333253

• Fixed worker crash reporting logic.
• Added Kusto telemetry to capture and report worker‑crash events.
• Added try/catch handling in a crash‑prone scenario to prevent unhandled worker failures.
• changes are behind Feature flag

Description

Problem (in Brief):

There were instances where the worker crashed, yet the pipeline continued running until it eventually hit the job‑level timeout. While the worker ideally should never crash, any unhandled exception causing a crash results in unnecessary resource consumption and prolonged pipeline execution—raising concerns about product reliability and efficiency.

How this problem gets noticed?

We identified this issue through an IcM where a regression was detected. Sharing the retrospective below for reference:
https://portal.microsofticm.com/imp/v5/retrospectives/Internal/1250792

Root Cause (Worker Crash – Brief Summary)

In the current Plan v8+ flow, the worker is responsible for reporting job status (Succeeded/Failed/Completed) back to the server. However, due to an unhandled exception, the worker process was crashing before sending the completion event. As a result, the server never received a signal to stop the job, and the pipeline continued running until the job-level timeout, leading to unnecessary resource consumption.

To address this, we implemented improvements focusing on two areas:

1. Preventing Worker Crashes
   The worker ideally should never crash. Wherever crashes were caused by unhandled exceptions, we identified those scenarios and added the required try/catch handling to ensure the worker exits gracefully.

2. Fallback Handling via Listener
   In cases where the worker still crashes unexpectedly, the listener now takes responsibility for notifying the server to stop the job. We also added telemetry signals to track and diagnose worker crashes more effectively.

Repro Steps:

• Added exception handling inside the CancelAndKillProcessTree function, which is invoked during step‑level timeouts.

• Pipeline Script

trigger: none
pool:
  name: Default
  demands:
    - Agent.Name -equals selfhosted2

stages:
- stage: ReleaseStage
  displayName: ModelD Service Deployment in Public Cloud
  jobs:
  - job: LockboxApprovalRequest
    displayName: Lockbox Approval Request
    dependsOn: []
    timeoutInMinutes: 7
    steps:
    - task: CmdLine@2
      timeoutInMinutes: 1
      continueOnError: true
      inputs:
        script: |
          echo Write your commands here
 
          echo Hello world
    - task: PowerShell@2
      displayName: 'Loop with Wait Script'
      timeoutInMinutes: 3
      inputs:
        targetType: 'inline'
        script: |
          Write-Host "Starting loop script..."
          for ($i = 1; $i -le 5000; $i++) {
              Write-Host "Iteration $i of 10"
              Start-Sleep -Seconds 10
              Write-Host "Completed iteration $i"
          }
          Write-Host "Loop script completed."
    - task: CmdLine@2
      inputs:
        script: |-
          echo Write your commands here
 
          echo Hello world
 
 

• Job keep on running even worker get crashed: Pipeline

• worker crashed logs in terminal:

Risk Assessment (Low / Medium / High)

Assess the risk level and justify your assessment. For example: code path sensitivity, usage scope, or backward compatibility concerns.

---

Unit Tests Added or Updated (Yes / No)

Indicate whether unit tests were added or modified to reflect the changes.

---

Additional Testing Performed

List manual or automated tests performed beyond unit tests (e.g., integration, scenario, regression).

• Verified worker‑crash reporting via Listener (FF enabled locally): Pipeline

• Verified crash‑prevention logic after adding try/catch in the identified scenario: Pipeline

• Also tested on-prem 2022 ADO version

Change Behind Feature Flag (Yes / No)

Can this change be behine feature flag, if not why?

---

Tech Design / Approach

• Design has been written and reviewed.
• Any architectural decisions, trade-offs, and alternatives are captured.

Design Doc (deep dive in Plan v7, plan v8+): https://microsoftapc.sharepoint.com/:w:/r/teams/ADOTasksandAgents/_layouts/15/Doc.aspx?sourcedoc=%7BDFED2317-D226-4EC9-A6AB-1E3E53D91934%7D&file=worker%20crash%20handling.docx&action=default&mobileredirect=true

Documentation Changes Required (Yes/No)

Indicate whether related documentation needs to be updated.

• User guides, API specs, system diagrams, or runbooks are updated.

---

Logging Added/Updated (Yes/No)

• Appropriate log statements are added with meaningful messages.
• Logging does not expose sensitive data.
• Log levels are used correctly (e.g., info, warn, error).

---

Telemetry Added/Updated (Yes/No)

• Custom telemetry (e.g., counters, timers, error tracking) is added as needed.
• Events are tagged with proper metadata for filtering and analysis.
• Telemetry is validated in staging or test environments.

Kusto Query:

CustomerIntelligence
| where Area == "AzurePipelinesAgent" 
| where Feature == "WorkerCrash"
| extend data = parse_json(Properties)
| extend tracepoint = tostring(data.properties.TracePoint)
| extend exitCode= tostring(data.properties.ExitCode)
| extend HostId  = tostring(data.hostId)
| extend JobId = tostring(data.properties.JobId)
| project TIMESTAMP, tracepoint, exitCode, HostId, JobId, data

Rollback Scenario and Process (Yes/No)

• Rollback plan is documented.

---

Dependency Impact Assessed and Regression Tested (Yes/No)

• All impacted internal modules, APIs, services, and third-party libraries are analyzed.
• Results are reviewed and confirmed to not break existing functionality.

[raujaiswal]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[raujaiswal]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[sanjuyadav24]

should we rename this to worker failed to send status to server?
as in that case this can be extended to any events in future

[sanjuyadav24]

this logic is similar to the logic in method LogWorkerProcessUnhandledException, could we please check if we can refactor this

[raujaiswal]

Thank you for suggestion, refactored the part.

[sanjuyadav24]

please convert this to runtime knob

[sanjuyadav24]

shouldn't we have sendEvent to server method here as well, as here only we are logging if worker is terminated due to unhandled exception

[Copilot]

@raujaiswal I've opened a new pull request, #5423, to work on those changes. Once the pull request is ready, I'll request review from you.

[raujaiswal]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[raujaiswal]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).

[sanjuyadav24]

should we add a retry here? I do see we have a retry in Jobrunner for RaisePlanEventAsync

[raujaiswal]

updated

[sanjuyadav24]

Please add design doc, other documentation link in work item with anything related to this issue

Also please check, I think we should also add a catch condition the actual block of worker which threw unhandled exception and process crashed

[raujaiswal]

/azp run

[azure-pipelines]

Azure Pipelines successfully started running 1 pipeline(s).