Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
19 changes: 19 additions & 0 deletions .github/workflows/workflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -69,6 +69,25 @@ jobs:
if errorlevel 1 exit /b %errorlevel%
dir

- name: dotnet test
shell: powershell
run: |
$testProject = "$env:project_root\CollectSFData.Tests\CollectSFData.Tests.csproj"
$runSettings = "$env:project_root\CollectSFData.Tests\CollectSFData.Tests.runsettings"
foreach ($framework in @("net8.0", "net48")) {
write-host "dotnet test $testProject -c Release -f $framework --settings $runSettings"
dotnet test $testProject -c Release -f $framework --settings $runSettings --logger "console;verbosity=normal"
if ($LASTEXITCODE -ne 0) { exit $LASTEXITCODE }
}

- name: upload test results
if: always()
uses: actions/upload-artifact@ea165f8d65b6e75b540449e92b4886f43607fa02 # v4.6.2
with:
name: TestResults
path: ${{ env.project_root }}/CollectSFData.Tests/TestResults
if-no-files-found: warn

- name: build output
shell: cmd
run: |
Expand Down
12 changes: 12 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -329,3 +329,15 @@ ASALocalRun/
# MFractors (Xamarin productivity tool) working folder
.mfractor/
**/tst/*
# Test audit & coverage artifacts (added by automated test auditing fixture)
test-audits/
latest.json
coverage.opencover.xml
coverage.cobertura.xml
coverage.info
*.trx
# Potential crash / blame dumps
*.dmp

# IDE language-service caches
*.lscache
119 changes: 116 additions & 3 deletions docs/contributing.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,10 @@ To report issue with collectsfdata, create a new issue in github repo [new issue
[CI CollectSFData](../.github/workflows/workflow.yml) builds pull requests targeting
`main` or `master` on opening, reopening, and subsequent updates with a read-only
contents token and no persisted checkout credentials. The existing documentation,
configuration, and Kusto path exclusions still apply.
configuration, and Kusto path exclusions still apply. After the build, CI runs the
`CollectSFData.Tests` project on net8.0 and net48 in the Release configuration;
a test failure or zero discovered tests fails the run. TRX and coverage results
are uploaded as the `TestResults` artifact even when the test step fails.

Pushes to `main` or `master` build the exact pushed commit independently of PR
artifacts. A release is published only after that build succeeds and the GitHub
Expand Down Expand Up @@ -192,6 +195,7 @@ collectSfDataDllTestProperties.json diff:
```

using output from .\scripts\azure-az-create-aad-application-spn.ps1, enter:

- AzureClientId = 'application id'
- AzureClientCertificate = 'clientsecret'
- AzureTenantId = 'tenant id'
Expand Down Expand Up @@ -265,7 +269,116 @@ data ingestion uri: https://ingest-collectsfdataunittest.eastus.kusto.windows.ne

## Testing

todo
### Test Project Targets

The test project targets `net48` and `net8.0`, one full-framework and one modern runtime, with C# latest, nullable and implicit usings enabled for both.

### Running Tests (Quick Start)

```powershell
# Fast run for latest framework only
dotnet test src/CollectSFData.Tests/CollectSFData.Tests.csproj -f net8.0

# Run all frameworks (slower)
dotnet test src/CollectSFData.Tests/CollectSFData.Tests.csproj

# Use custom runsettings (stabilizes host & disables parallelization)
dotnet test src/CollectSFData.Tests/CollectSFData.Tests.csproj -f net8.0 --settings src/CollectSFData.Tests/CollectSFData.Tests.runsettings

# Filter to a subset (example new process tests)
dotnet test src/CollectSFData.Tests/CollectSFData.Tests.csproj -f net8.0 --settings src/CollectSFData.Tests/CollectSFData.Tests.runsettings --filter "FullyQualifiedName~FileManagerProcessTests"
```

### Runsettings Rationale

`CollectSFData.Tests.runsettings` was added to mitigate intermittent file locking and PowerShell runspace errors by:

- Disabling AppDomain reuse (`DisableAppDomain=true`) so each run uses a clean test host.
- Forcing single-threaded execution (`MaxCpuCount=1`, xUnit parallelization off) to avoid simultaneous copies of large dependent assemblies.
- Increasing session timeout to allow slower PowerShell-invoking tests to complete without abrupt host termination.
- Auto-detecting the target framework instead of forcing net8.0 for the net48 run.
- Failing the run if no tests are discovered (`TreatNoTestsAsError=true`).

The settings enable the `XPlat code coverage` collector and TRX logger. Results
are written below `src/CollectSFData.Tests/TestResults` unless overridden with
`--results-directory`. The optional crash-dump collector requires the signed
Microsoft Sysinternals `procdump64.exe` on `PATH` or in `PROCDUMP_PATH` for net48.

### Test Audit Output (opt-in)

Set `COLLECTSFDATA_TEST_AUDIT=1` to have the test run write per-run metadata
(run id, OS, framework, git commit, coverage and TRX summaries) under the
gitignored `test-audits/` folder. Without the variable nothing is written.

### File Lock (MSB3021 / MSB3027) Mitigation Guide

Symptoms:

- Build warnings `MSB3026` escalating to errors `MSB3027` / `MSB3021` mentioning `CollectSFDataDll.dll` locked by `testhost`.
- Subsequent builds for the same TFM fail until the locking process exits.

Root Causes:

- Parallel test executions across target frameworks holding open file handles.
- Post-build PowerShell scripts executing while the prior testhost still loads the DLL.

Mitigations (apply in order):

1. Use single target with `--framework net8.0` while iterating.
2. Add `--no-build` when re-running quickly after a successful build.
3. Use the provided runsettings to disable parallelization & AppDomain reuse.
4. If locks persist, close IDE test sessions or manually terminate `testhost.exe` processes (outside constrained environments).
5. For CI: run target frameworks sequentially (`dotnet test -f net48` then `-f net8.0`) instead of simultaneous matrix on a single workspace if locks reappear.

### PowerShell Runspace Exception

Error: `System.Management.Automation.PSInvalidOperationException: There is no Runspace available…`

Causes & Remedies:

- Host teardown while a timer in a PowerShell-invoking component still references a disposed runspace.
- Long-running or asynchronous PowerShell script triggered during test cleanup.

Workarounds implemented:

- Longer `TestSessionTimeout` (10 minutes) to reduce abrupt cancellation.
- Serial execution eliminating overlapping teardown phases.

### Adding New Tests – Guidelines

Focus areas still needing coverage:

- Serialization chunking thresholds (CSV & JSON) for large collections.
- Compression branches (when configuration enables compression).
- Observer/ETL timeout & error pathways.
- Ingestion mode branching (Kusto vs Log Analytics vs none) with mocked dependencies.
- Concurrency safety of collection modifications under simulated parallel enumeration.

Use patterns already present (FluentAssertions + xUnit). Prefer small, fast unit tests; only integrate PowerShell/IO when asserting critical behaviors.
Exercise the production parser and timestamp setters in time-validation tests.
Exact-string mocks of normalized local timestamps are time-zone dependent and
can pass on a developer machine but fail on a UTC CI runner. Use explicit input
offsets and assert the resulting UTC values, including invalid and reversed ranges.

### Traits / Filtering (Optional Enhancement)

Introduce `[Trait("Category","Fast")]` vs `"Integration"` on future tests to enable:

```powershell
dotnet test --filter "Category=Fast"
```

### Coverage

Run with the checked-in settings to collect coverage:

```powershell
dotnet test src/CollectSFData.Tests/CollectSFData.Tests.csproj -c Release -f net8.0 --settings src/CollectSFData.Tests/CollectSFData.Tests.runsettings
```

The collector emits Cobertura, LCOV, and OpenCover reports. CI also runs net48
sequentially and retains both runtimes' results in the `TestResults` artifact.
Coverage thresholds and report badges remain future roadmap work.

## Creating pull requests

Expand Down Expand Up @@ -341,4 +454,4 @@ The executable is strong-name signed, which is not an Authenticode signature.
generate a new version; inspect existing assets before retrying publication.

Live publication must be verified on GitHub after deployment. Local linting and
release-policy tests do not prove a release has been published.
release-policy tests do not prove a release has been published.
92 changes: 92 additions & 0 deletions docs/testing-roadmap.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
# Testing & Quality Improvement Roadmap

Status: Active (Phase 2 in progress)

Owner: jagilber

Last Updated: 2026-11-18

## Goals

Establish sustainable, high-confidence test & quality framework for CollectSFData enabling safe refactors, feature velocity, and reliability.

## Success Metrics

- Progressive code coverage thresholds (core logic): 50% → 70% → 85%+
- <5% flaky test rate over rolling 30 days
- Zero secret tokens/keys in logs or saved configs (automated grep gate)
- Mutation score (core) ≥65% by Phase 5
- Time to add new ingestion type reduced (qualitative dev feedback)

## Phase Plan

| Phase | Focus | Deliverables | Target Coverage | Status |
| ------- | ------- | ------------- | ----------------- | -------- |
| 0 | Baseline setup | Test infra, initial unit tests (time, file type, table naming), coverage reporting | ~10-15% | ✅ Complete |
| 1 | Testability baseline | Test-only time and filesystem adapters; production decomposition deferred | 25-35% | Baseline complete |
| 2 | Core Logic | Validation tests, merge precedence, golden-file parser fixtures | 40-50% | 🔄 In Progress |
| 3 | Integration Harness | Azurite setup, Kusto stub, E2E CLI happy/empty/error | 55-65% | Pending |
| 4 | Concurrency & Reliability | Replace CustomTaskManager, cancellation & timeout tests | 65-75% | Pending |
| 5 | Performance & Mutation | Benchmarks, Stryker, hotspot optimizations | 70-80% | Pending |
| 6 | Security & Hardening | Secret redaction, threat model checklist | 75-85% | Pending |
| 7 | Legacy Rationalization | TFM review (drop net462?), simplify conditionals | 80-85% | Pending |
| 8 | Governance & Polish | Analyzer enforcement, docs, contributor test guidelines | 85%+ | Pending |

## Baseline Checklist (Phase 0)

- [x] Add test project `CollectSFData.Tests` targeting net48 and net8.0
- [x] Cover time parsing, table-name normalization, and file type parsing
- [x] Configure Coverlet collection and TRX results through shared runsettings
- [x] Run both test targets sequentially in Release in GitHub CI
- [x] Retain test results and coverage artifacts on successful and failed CI runs
- [ ] Add generated coverage reports, measured baselines, and thresholds
- [ ] Add automated secret-redaction regression coverage

## Risk Register

| Risk | Impact | Mitigation |
| ------ | -------- | ------------ |
| Static singletons limit test isolation | Slows test adoption | Introduce seam interfaces early (Phase 1) |
| Custom scheduler complexity | Flaky concurrency tests | Replace with channels workers (Phase 4) |
| Large refactor churn | Merge conflicts | Short-lived incremental PRs per phase |
| Legacy TFMs blocking APIs | Limits modernization | Evaluate usage metrics before removal |

## Test Strategy Summary

See prior analysis for detailed rationale. Key layers: unit (pure logic), integration (local Azurite + stubs), E2E CLI, performance, mutation, security redaction.

## Progress Log

| Date | Phase | Change | Author |
| ------ | ------- | -------- | -------- |
| 2025-09-11 | 0 | Roadmap file created, test project scaffolded | (fill) |
| 2025-09-11 | 0 | Multi-target framework support (net462;net48;net6.0;net8.0) | (fill) |
| 2025-09-11 | 0 | Initial ConfigurationOptions unit tests (time parsing, file type, table naming) | (fill) |
| 2025-09-11 | 0→1 | Tests passing on modern frameworks (net6.0, net8.0); legacy signing issue identified | (fill) |
| 2025-09-11 | 1 | Implemented dependency injection pattern with ITimeProvider and IFileSystemAbstraction | (fill) |
| 2025-09-11 | 1 | Created TestableConfigurationOptions wrapper for dependency injection testing | (fill) |
| 2025-09-11 | 1 | Added Moq-based testing with mocked dependencies, 12 tests passing on modern frameworks | (fill) |
| 2025-09-11 | 1 | Phase 1 complete: Added test project to solution, ready for Phase 2 core logic expansion | (fill) |

## Current Status (2026-11-18)

- Branch `feature/test-project` and upstream PR #185 contain the test baseline.
- Local Release validation passes 104 tests on net48 and 104 on net8.0, with none skipped.
- The shipped library still targets net462, net48, net6.0, and net8.0; tests cover one runtime from each family.
- Time and filesystem adapters are test-only helpers, not production dependency-injection abstractions.
- Time parser and range checks exercise the production implementation with explicit UTC offsets, invalid inputs, and equal/reversed ranges.
- The UTC CI failure was caused by exact-string mocks of local timestamp normalization, not production timestamp parsing.
- Shared runsettings auto-detect the runtime, disable parallelism, collect coverage, and reject runs that discover no tests.
- GitHub CI validation and upstream merge status must be checked on PR #185; local results alone do not prove delivery.
- Coverage percentages have not been established by the phase table; those values are targets, not measured results.

## Next Steps

1. Require green hosted CI and maintainer review for the baseline PR.
2. Measure coverage and prioritize untested production branches rather than test-only adapters.
3. Add local integration and CLI happy/empty/error scenarios in Phase 3.
4. Add secret-redaction regression tests before claiming security coverage.

---

This document is the authoritative plan; update table & log on each material change.
48 changes: 48 additions & 0 deletions src/CollectSFData.Tests/AuditFixtureTests.cs
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
using System;
using System.IO;
using System.Linq;
using Xunit;

namespace CollectSFData.Tests
{
[Collection("TestAuditCollection")] // ensure fixture has run
public class AuditFixtureTests
{
[Fact]
public void AuditLatestJsonExists()
{
if (!TestAuditFixture.Enabled)
{
return; // audit output is opt-in, nothing to verify
}

// Discover repo root similarly to fixture logic (climb for .git or solution)
var dir = new DirectoryInfo(AppContext.BaseDirectory);
DirectoryInfo? solutionRoot = null;
DirectoryInfo? repoRoot = null;
for (int i = 0; i < 12 && dir != null; i++)
{
if (solutionRoot == null && dir.GetFiles("*.sln").Any())
{
solutionRoot = dir;
}
if (dir.GetDirectories(".git").Any())
{
repoRoot = dir;
break;
}
dir = dir.Parent;
}
repoRoot ??= solutionRoot ?? new DirectoryInfo(AppContext.BaseDirectory);
var auditsDir = Path.Combine(repoRoot.FullName, "test-audits");

// The fixture writes latest.json during construction (phase = start), so it should exist now.
var latestPath = Path.Combine(auditsDir, "latest.json");
Assert.True(File.Exists(latestPath), $"Expected audit latest.json at {latestPath} but it was not found.");

// Basic sanity: file should have non-trivial content
var length = new FileInfo(latestPath).Length;
Assert.True(length > 50, $"latest.json seems unexpectedly small ({length} bytes).");
}
}
}
39 changes: 39 additions & 0 deletions src/CollectSFData.Tests/CollectSFData.Tests.csproj
Original file line number Diff line number Diff line change
@@ -0,0 +1,39 @@
<Project Sdk="Microsoft.NET.Sdk">

<PropertyGroup>
<!-- One full-framework and one modern target cover both runtime families of CollectSFDataDll. -->
<TargetFrameworks>net48;net8.0</TargetFrameworks>
<LangVersion>latest</LangVersion>
<ImplicitUsings>enable</ImplicitUsings>
<Nullable>enable</Nullable>
<SignAssembly>false</SignAssembly>

<IsPackable>false</IsPackable>
<IsTestProject>true</IsTestProject>
</PropertyGroup>

<ItemGroup>
<PackageReference Include="coverlet.collector" Version="6.0.0">
<IncludeAssets>runtime; build; native; contentfiles; analyzers; buildtransitive</IncludeAssets>
<PrivateAssets>all</PrivateAssets>
</PackageReference>
<PackageReference Include="FluentAssertions" Version="6.12.0" />
<PackageReference Include="Microsoft.NET.Test.Sdk" Version="17.8.0" />
<PackageReference Include="Moq" Version="4.20.70" />
<PackageReference Include="xunit" Version="2.5.3" />
<PackageReference Include="xunit.runner.visualstudio" Version="2.5.3" />
</ItemGroup>

<ItemGroup Condition="'$(TargetFramework)' == 'net48'">
<PackageReference Include="System.Text.Json" Version="8.0.5" />
</ItemGroup>

<ItemGroup>
<Using Include="Xunit" />
</ItemGroup>

<ItemGroup>
<ProjectReference Include="..\CollectSFDataDll\CollectSFDataDll.csproj" />
</ItemGroup>

</Project>
Loading
Loading