Investigate Visual Studio integration test failures from Azure DevOps builds. Use when investigating integration test timeouts, crashes, or failures in the roslyn-integration-CI pipeline. Also use when asked "why are integration tests failing", "integration test timeout", "VS integration tests", or given AzDO build URLs from the roslyn-integration-CI pipeline.
Investigate Visual Studio integration test failures in the roslyn-integration-CI Azure DevOps pipeline. Downloads and analyzes published test artifacts including exception logs, screenshots, MEF errors, and VSIX installer logs.
Workflow: Run the script with a BuildId → it fetches the build timeline, identifies failed/canceled integration test jobs, downloads the published test artifacts, and parses exception XMLs and error logs → read the structured output and [INTEGRATION_TEST_SUMMARY] JSON → synthesize root cause analysis. The script collects data; you generate the diagnosis.
Use this skill when:
roslyn-integration-CI pipelinedev.azure.com/dnceng-public with integration test jobs (VS_Integration_*)⚠️ This skill is specific to Roslyn VS integration tests. For general CI analysis (compiler tests, Helix failures, build errors), use the
ci-analysisskill instead.
# Ensure powershell tool is restored
dotnet tool restore
# Analyze integration test failures by build ID
dotnet pwsh -File .github/skills/integration-test-analysis/scripts/Get-IntegrationTestStatus.ps1 -BuildId 1354185
# Analyze with verbose artifact download (downloads and parses exception XMLs)
dotnet pwsh -File .github/skills/integration-test-analysis/scripts/Get-IntegrationTestStatus.ps1 -BuildId 1354185 -DownloadArtifacts
# Limit artifact download size (default 200MB)
dotnet pwsh -File .github/skills/integration-test-analysis/scripts/Get-IntegrationTestStatus.ps1 -BuildId 1354185 -DownloadArtifacts -MaxArtifactSizeMB 500
| Parameter | Required | Default | Description |
|---|---|---|---|
-BuildId | Yes | — | Azure DevOps build ID from the roslyn-integration-CI pipeline |
-DownloadArtifacts | No | $false | Download and analyze published test artifacts (exception XMLs, screenshots, MEF errors) |
-MaxArtifactSizeMB | No | 200 | Maximum artifact size in MB to download. Artifacts larger than this are skipped with a warning |
-Organization | No | dnceng-public | Azure DevOps organization |
-Project | No | public | Azure DevOps project |
VS_Integration_*) and their status (succeeded/failed/canceled)For each failed or canceled integration test job:
-DownloadArtifacts)| File Pattern | What It Contains |
|---|---|
*.Activity.xml | Exception logs per test — contains full stack traces, VS activity log entries |
*.png / *.jpg (in Screenshots/) | Screenshots taken during test execution — shows VS state at failure |
*.png / *.jpg (at root level) | Screenshots from timeouts — shows VS state when the job was killed |
MEFErrors*.txt | MEF composition errors — missing exports, failed service activation |
VSIXInstaller*.log | VSIX installation log — shows if Roslyn VSIX sideloading succeeded |
ServiceHub*.log | ServiceHub process logs — shows OOP service activation failures |
StartingBuild.png | Screenshot taken when the test harness first starts VS |
For each *.Activity.xml file found:
VisualStudioErrorReportingService, GlobalBrokeredServiceContainer)When integration test jobs are canceled after ~150 minutes:
When exception XMLs show ServiceActivationFailedException:
System.Runtime version mismatch)When MEFErrors*.txt files are present:
When VSIXInstaller*.log shows errors:
FileNotFoundException during install — VS instance state is corrupted or lockedThe script outputs human-readable console text followed by a structured JSON block:
[INTEGRATION_TEST_SUMMARY]
{
"buildId": 1354185,
"buildUrl": "https://dev.azure.com/...",
"buildResult": "failed",
"pipelineTimeout": 150,
"jobs": [
{
"name": "VS_Integration_Debug_64",
"result": "canceled",
"durationMinutes": 153.7,
"timedOut": true,
"configuration": "Debug",
"testRunnerStatus": "1 running, 3 queued, 0 completed",
"vsixInstallSuccess": true,
"lastTestActivity": "01:11:59"
}
],
"artifacts": {
"downloaded": true,
"exceptionFiles": ["01.15.40-CSharpTyping.TypingInPartialType-AggregateException.Activity.xml"],
"screenshotFiles": ["StartingBuild.png", "01.15.40-Timeout.png"],
"mefErrorFiles": [],
"vsixLogFiles": ["VSIXInstaller-xxx.log"],
"parsedExceptions": [
{
"fileName": "01.15.40-CSharpTyping.TypingInPartialType-AggregateException.Activity.xml",
"testName": "CSharpTyping.TypingInPartialType",
"exceptionType": "AggregateException",
"rootCausePattern": "assembly-load-failure",
"primaryError": "Could not load file or assembly 'System.Runtime, Version=10.0.0.0'",
"totalErrors": 147,
"affectedFeatures": ["Solution Events", "Asset synchronization", "..."],
"errorSources": { "GlobalBrokeredServiceContainer": 80, "VisualStudioErrorReportingService": 67 },
"vsVersion": "17.14.0"
}
],
"parsedMefErrors": [],
"parsedVsixLogs": [
{ "fileName": "VSIXInstaller-xxx.log", "success": true, "errors": [] }
]
}
}
[/INTEGRATION_TEST_SUMMARY]
Before running the script:
roslyn-integration-CI onlypwsh -File .github/skills/integration-test-analysis/scripts/Get-IntegrationTestStatus.ps1 -BuildId <ID> -DownloadArtifacts
[INTEGRATION_TEST_SUMMARY] JSON for structured dataartifacts.parsedExceptions, artifacts.parsedMefErrors, and artifacts.parsedVsixLogs to understand what went wrongtimedOut, testRunnerStatus, vsixInstallSuccess) with the parsed artifact data to synthesize a root cause