# CI Resolve: enterprise engineering automation research

**Research date:** 4 November 2026  
**Decision:** Evolve CI Resolve into an enterprise platform for CI investigation, isolated reproduction, repair, and verification.  
**Scope:** Product strategy, competitor research, target architecture, implementation sequence, evaluation, and investment assumptions.  
**Status:** Proposed enterprise design. The existing application remains version 0.1.0, a local workbench; the systems described here are not implemented unless explicitly identified as existing.

> **Implementation update:** Version 0.2 now implements a bounded unattended GitHub → reproduction → repair → protected verification → PR → CI/merge reconciliation path. See [current implementation and limitations](../README.md) and [validation evidence](VALIDATION.md). The enterprise architecture below remains a target; organization identity, PostgreSQL, microVM fleets and other later systems are not implemented. The original 0.1 gap assessment is retained as historical research.

## 1. The product we should build

**An engineering reliability platform that automatically turns failed CI jobs into reproducible investigations and independently verified repair proposals.**

The main workflow should be:

**Observe failure → reconstruct execution context → reproduce → test explanations → generate a bounded repair → independently verify → open a reviewable pull request → observe required CI and merge results → detect recurrence.**

This is substantially more demanding than importing a log, matching error patterns, and recording a command exit code. Its core engineering problems are distributed systems, reproducible execution, inference under uncertainty, untrusted-code isolation, verification, and integration with an organization's existing delivery controls.

There is no useful upper bound on “most complex.” A credible enterprise product should earn complexity through outcomes: less engineering investigation time, fewer incorrect repairs, and reliable resolution across real repositories. The goal is a substantial engineering business with measurable technical depth.

**Proposed initial market:** Engineering organizations with roughly 100–2,000 developers, many repositories or a substantial monorepo, frequent CI failures, and a platform/developer-productivity team. These size boundaries are customer-selection hypotheses, not evidence of demand.

**First supported environment:** GitHub Actions, Linux, Node.js/TypeScript, reproducible test/build/type-check failures, and disposable PostgreSQL/Redis service dependencies. Expand languages, providers, operating systems, and failure types after demonstrating repeatability.

**Long-term scope:** Multiple CI providers, languages, private runners, dependency and infrastructure regressions, cross-repository compatibility, build performance, and integrations with production incidents. Production remediation is a separate later capability with a separate authorization model.

## 2. What the current application actually contains

The local source was inspected during this research. The application is a useful workflow prototype, but it does not yet implement the enterprise product.

| Area | Current implementation | Enterprise requirement |
| --- | --- | --- |
| Ingestion | Manual text/JSON imports with user-entered metadata | Authenticated provider collection, webhooks, reconciliation and source verification |
| Persistence | Local SQLite with cases, runs, events and verifications | Tenant-scoped relational state, object storage, retention, migrations and recovery |
| Diagnosis | Ordered diagnostic regex rules and a signature | Multiple failure instances, historical evidence, environment differences and experiments |
| Grouping | Repository/workflow/job plus one diagnostic signature | Versioned clusters with explainable similarity and split/merge operations |
| Reproduction | User explicitly invokes a local command | Captured job recipe, isolated disposable environment and comparable baseline |
| Fix generation | Absent | Constrained candidate generation, diff review and candidate budgets |
| Verification | Exit code plus supported checkout fingerprint before/after | Independent oracle, pre-patch failure, candidate success and required original CI checks |
| Resolution | Latest passing local check for the latest imported occurrence | PR/check/merge identity, evidence freshness, policy decision and recurrence monitoring |
| Organization model | Single local user | Organizations, installations, membership, RBAC/SSO, audit and tenant isolation |
| Operations | Local process | Durable workflows, scheduling, fleet health, quotas, incident response and billing |

### Specific limitations that shape the redesign

- A user can select an irrelevant command that exits zero. Stable checkout identity does not establish that the original failure was exercised.
- Imported repository/run/commit information is user-provided, rather than fetched and authenticated against the provider.
- One selected diagnostic category per job cannot represent several independent failing tests or a shared upstream service failure.
- The checkout fingerprint excludes environment state, installed dependency contents, service data and ignored files; these can determine whether CI passes.
- Local verification history is not a remotely verifiable attestation. A user with database access can alter it.
- New occurrences reopen a case, which is useful, but the lifecycle has no PR head/base revision, merge-group commit, runner recipe or patch identity.

These are observed scope limits, not a claim that the local implementation is broken. The enterprise design changes the meaning of “verified.”

## 3. Research approach and claim boundaries

Research used official product documentation, vendor engineering articles, open-source project documentation, standards, and original research papers. No customer interviews, vendor trials, enterprise contracts, paid pricing quotes, or private market databases were available.

- Competitor descriptions state documented or advertised capability; they are not independent quality assessments.
- An omitted capability is **not verified in this research**, rather than proof that a vendor lacks it.
- Architecture choices, differentiators, staffing, timelines, prices and target metrics are proposals.
- A benchmark result is not a forecast of success on a customer's CI failures.
- No claim of an empty market, global uniqueness, market share, or market size is made.

The source register gives direct links, with citations placed beside externally supported claims. Product names and capabilities were checked on the research date; provider documentation can change.

## 4. Competitive landscape

| Product or platform | Publicly documented capability relevant to us | Implication |
| --- | --- | --- |
| Harness | Broad SDLC platform connecting delivery, testing, security, governance and software-delivery context | A broad “autonomous SDLC” positioning already has strong competition |
| Trunk | Flaky-test detection, clustering, quarantine and ticket workflows | Test analytics and automatic quarantine are established features |
| Buildkite Test Engine | Test analytics and test execution/orchestration capabilities | Test optimization and historical execution evidence are occupied territory |
| CircleCI Smarter Testing | Test selection/splitting and failed-test rerun workflows | Retrying only failures is not a defensible new product |
| Devin | Repository-connected engineering agent, environments, automations and CI-fix positioning | An agent that opens a patch from a failure is already a direct competitive concept |
| Cursor cloud/background agents | Remote engineering work with repository/environment controls | Hosted coding execution and environment setup are not sufficient differentiation |
| Sentry Seer | Debugging workflow using application context, with automated fix/PR workflows | Moving from errors to proposed repairs is an established adjacent category |
| Datadog Bits AI agents | Incident investigation and remediation-related capabilities across observability context | Broad AI SRE expansion has substantial incumbent competition |
| Greptile | Repository-aware automated pull-request review | AI review of a generated patch can be bought or integrated; review alone is not proof |
| Aviator | Review/merge workflow and merge queue automation | Respect and integrate existing merge systems rather than recreate them by default |

Official references: [Harness](https://www.harness.io/products/platform), [Trunk documentation](https://docs.trunk.io/flaky-tests/overview), [Buildkite](https://buildkite.com/platform/test-engine/), [CircleCI](https://circleci.com/docs/guides/test/smarter-testing-overview/), [Devin documentation](https://docs.devin.ai/get-started/devin-intro) and [CI-fix positioning](https://app.devin.ai/), [Cursor](https://prod.cursor.com/help/ai-features/background-agents), [Sentry engineering workflow](https://blog.sentry.io/automated-debugging-workflow-sentry/), [Datadog](https://www.datadoghq.com/product/ai/bits-ai-agents/), [Greptile](https://www.greptile.com/docs/introduction), [Aviator](https://www.aviator.co/).

### The differentiation to validate

**CI-provider-independent, privately executable reproduction and repair evidence, including controlled diagnosis experiments and validation against the exact current change.**

This is a hypothesis. “Provider independent,” “private runners,” “evidence,” and “agents” individually are not demonstrated moats. A buyer should compare the full workflow against their incumbent platform and an existing coding agent with repository-specific scripts.

The defensible asset could become a customer's execution recipes, trusted integrations, labeled failure history, successful repairs, and accurate decision policy. That asset requires consent and actual usage. Source code volume, a vector database, or the number of agents does not establish defensibility.

## 5. Alternative complex products considered

| Direction | Technical depth | Daily engineering relevance | Entry difficulty | Decision |
| --- | --- | --- | --- | --- |
| CI investigation, reproduction and repair platform | Very high: untrusted execution, experiments, distributed workflows and validation | High where CI failures are frequent | High, with direct competition | **Recommended continuation** |
| General internal developer platform | High: catalogs, environments, identity and self-service | High for platform organizations | Crowded; substantial enterprise configuration | Integrate with existing portals |
| Autonomous production SRE platform | Very high: live topology, observability and operational remediation | High for operations teams | Very high trust and integration requirements | Later adjacency, not the first scope |
| Cross-repository modernization factory | Very high: program analysis, transformations and coordinated rollout | More campaign-based than daily | High; language/framework-specific | Later repair and migration campaigns |
| New CI execution/build platform | Very high: scheduling, caches, workers and toolchains | High | Expensive replacement sale against established CI | Reuse existing CI and execution infrastructure |

The selected direction has a complete end-to-end engineering loop while preserving the user's choice of CI Failure Resolution.

## 6. Target user workflows

### A. A developer's PR fails

The system detects the exact failed job and attempt, extracts individual failures, identifies an existing related investigation, and attaches a concise check result. If a trusted execution recipe exists, it reproduces the failure in a disposable environment. It presents the failed test, reproduction evidence, proposed patch and independent verification results in the PR.

The developer reviews the proposed repair. A new push invalidates evidence for the old head. Required original CI still runs. Resolution is recorded after the configured PR/merge outcome, rather than immediately after a local pass.

### B. A platform engineer handles a failure storm

Several repositories fail after a runner-image or package-registry change. The system relates affected execution contexts, tests a previous environment against the same source, and identifies the shared suspected dependency. It proposes a controlled recipe/image rollback or infrastructure ticket instead of opening dozens of unrelated code patches.

### C. A test owner investigates nondeterminism

The system observes contradictory results for a comparable test identity, records seeds/order/worker concurrency where available, and runs a budgeted experiment. It can recommend temporary quarantine with an owner, expiry and monitoring. It does not label a test safe merely because several retries passed.

### D. A dependency update breaks consumers

The system compares lockfiles and API/type changes, tests current and previous dependency sets, proposes a migration or constrained version adjustment, and verifies relevant consumers. Native contract-test records and full CI results remain the acceptance evidence.

[Pact's compatibility matrix](https://docs.pact.io/pact_broker/can_i_deploy) demonstrates an existing approach to recorded consumer/provider compatibility. Integrate it where installed; do not claim to infer every consumer from source search.

## 7. Product modules and the real engineering work

| Module | Responsibilities | Why it is difficult | Deliverable |
| --- | --- | --- | --- |
| Provider ingestion | GitHub/GitLab/Buildkite/CircleCI events, polling, logs/artifacts | Redelivery, ordering, missing artifacts, permissions, rate limits | Complete source-linked execution records |
| Failure extraction | TAP/JUnit/framework JSON, steps, stack traces | Formats disagree; test identities and parameterization change | Individual normalized failure instances |
| Investigation engine | Grouping, owner mapping, historical comparisons | Similar text can hide distinct causes; one cause can have many symptoms | Explainable candidate relationships |
| Environment reconstruction | Images, toolchains, lockfiles, service fixtures and execution recipe | External state and hosted-runner drift are incompletely observable | Reproduction manifest with explicit gaps |
| Private execution fleet | Provisioning, queues, quotas, cancellation and cleanup | Untrusted code, network isolation, crashes and customer networking | Bounded disposable runs |
| Diagnostic experiments | Compare source/environment/dependency variants | Confounding variables, noisy tests, compute cost | Supported or rejected hypotheses |
| Repair engine | Bounded edits, candidate variants, command planning | Agents can optimize away the test instead of fixing behavior | Reviewable patch proposals |
| Independent verifier | Protected tests, regression suite, artifacts and CI reconciliation | Oracle quality and stale evidence | Structured verification decision |
| Change integration | Checks, PRs, reviews, merge queues and recurrence | Remote side-effect ambiguity and changing commits | Traceable repair lifecycle |
| Enterprise control | Identity, policies, retention, audit, data residency | Tenant boundaries and customer-specific requirements | Operable enterprise service |
| Evaluation and economics | Benchmarks, cost ledger and quality gates | Selection bias, leakage, expensive unsuccessful attempts | Measured release decisions |

These are logical boundaries. They do not require eleven independent microservices on day one.

## 8. Target architecture

```mermaid
flowchart TB
    SCM[GitHub and later CI providers] --> IN[Authenticated ingestion and reconciliation]
    IN --> DB[(PostgreSQL metadata and outbox)]
    IN --> OBJ[(Tenant scoped artifact storage)]
    DB --> WF[Durable investigation workflows]
    WF --> CTX[Failure and execution context]
    CTX --> EXP[Experiment planner]
    EXP --> FLEET[Customer or managed execution fleet]
    FLEET --> BASE[Baseline and reproduction runs]
    BASE --> REPAIR[Bounded patch generation]
    REPAIR --> VERIFY[Independent verification workers]
    VERIFY --> EVID[Evidence and policy decision]
    EVID --> SCM
    EVID --> UI[Investigation and PR review interface]
    POLICY[Identity policy credentials and quotas] --> WF
    POLICY --> FLEET
    WF --> COST[Cost ledger and observability]
    FLEET --> OBJ
    VERIFY --> OBJ
```

### Deployment boundaries

**Control plane:** Organization identity, provider installations, policies, investigation state, scheduling, billing, public API and UI. It should not execute repository code.

**Execution plane:** Customer-managed or managed disposable workers. Repository code, dependency installation and tests execute here. Credentials and network access are scoped to each run.

**Evidence plane:** Source metadata, manifests, artifacts, observations and signed execution records. Stored content has tenant scope, retention and provenance. Signing is an integrity/identity mechanism; it does not prove a repair is correct.

**Inference plane:** Versioned diagnostic and patch-generation services. They receive only allowed evidence and are charged against per-investigation budgets. Model output cannot directly approve its own tool permissions or verification result.

## 9. Ingestion and provider correctness

Start with a GitHub App installed on selected repositories. Separate read-only ingestion permissions from branch/PR writing capabilities. Installation tokens expire after one hour according to [GitHub's authentication documentation](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/authenticating-as-a-github-app-installation); generate them when needed and retain installation identity separately.

### Required behavior

1. Validate the raw webhook body using the provider's documented signature method before processing. GitHub recommends the HMAC-SHA256 signature header. [Webhook validation](https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries).
2. Resolve installation and repository membership server-side. An organization name in a payload is not an authorization decision.
3. Persist delivery receipt, payload digest and durable processing state before acknowledging accepted work.
4. Deduplicate transport deliveries and business events separately. An attacker changing an unsigned delivery-ID header must not create a new authorized action from a replayed signed payload.
5. Fetch authoritative run/job/attempt/check details. Preserve workflow definition identity, PR head, base, and any tested synthetic merge SHA.
6. Paginate jobs, attempts and artifacts; retain acquisition failures as incomplete evidence.
7. Reconcile by polling after outages or lost events. Webhooks alone are not a completeness guarantee.
8. Bound download size, decompression, file count and path traversal. Treat report files and archives as untrusted inputs.
9. Follow only provider-approved artifact redirects, with credentials scoped to the correct destination. Signed URLs must not enter public logs.
10. Handle uninstallation, permission changes and repository transfer by revoking or suspending work.

GitHub explicitly identifies risks when privileged workflows process untrusted PR code or artifacts. The ingest/execute split must account for that behavior. [GitHub secure-use reference](https://docs.github.com/en/actions/reference/security/secure-use).

## 10. Reproduction: the central technical product

A commit SHA identifies source, not the complete computation. Toolchain versions, container image contents, environment configuration, services, fixtures, test order, architecture, timezone and external dependencies can change the result. Bazel's [hermeticity documentation](https://bazel.build/basics/hermeticity) explains the value of isolating builds from host/tool changes.

### Reproduction manifest

Record immutable or explicitly versioned references for:

- Source tree, submodules, LFS content and generated-code inputs where relevant.
- Workflow file, reusable workflows, action revisions and resolved command sequence.
- Operating system, architecture, image digest, runtime, compiler and package-manager versions.
- Lockfiles, package integrity references, registry/mirror configuration and cache trust class.
- Service images, startup/readiness checks, schema, fixture version and snapshot lineage.
- Nonsecret environment configuration, locale, timezone, seed, clock strategy and concurrency.
- Secret **references and versions**, never secret values in the manifest.
- Network policy, permitted external dependencies and evidence of unavailable state.
- Test selector set, expected test count, timeout and verification policy version.

Use opaque secret versions rather than publishing hashes of low-entropy secret values. A hash can still disclose a guessable secret.

### Reproduction fidelity levels

| Level | Meaning | Permitted claim |
| --- | --- | --- |
| R0 | Log/context only | Diagnostic suggestion |
| R1 | Same source, partial local recipe | Local observation under recorded differences |
| R2 | Pinned supported recipe and services | Failure reproduced in the supported recipe |
| R3 | Comparable baseline and controlled experiments | Evidence supports a specific explanation under stated assumptions |
| R4 | Candidate passes independent validation and required original CI | Verified for the recorded candidate and policy scope |

These are proposed internal levels, not an industry standard or proof of perfect equivalence. Some jobs will remain unreproducible because secrets, production fixtures, hardware, network state or original images cannot be reconstructed. That must be a first-class result.

## 11. Diagnostic experiments instead of confident summaries

Maintain hypotheses with evidence for and against them. Examples: source regression, dependency change, runner-image drift, service readiness, resource exhaustion, nondeterminism, and unavailable external service.

### Example: source or environment?

| Experiment | Source revision | Environment recipe | What it helps distinguish |
| --- | --- | --- | --- |
| A | Candidate | Current | Confirm the observed symptom |
| B | Baseline | Current | Whether the source change is necessary under this environment |
| C | Candidate | Previous | Whether the environment change is necessary under this source |
| D | Baseline | Previous | Control for the earlier known state |

If A fails and the others pass, an interaction is plausible. It is not automatically proven: fixture differences, timing and unavailable external state can confound the result. Select experiments for expected diagnostic value and compute budget; do not automatically run every combination.

### Failure relationship model

Keep separate:

- **Observed facts:** test exit, exception, service response, image digest, duration.
- **Associations:** failures correlated with a runner image or dependency.
- **Hypotheses:** proposed causal explanation.
- **Interventions:** a controlled modification and its observed result.
- **Resolution evidence:** a repair accepted under a stated verification policy.

Confidence must be calibrated on held-out cases. A self-reported model probability is not a validated confidence score.

### Flaky tests

Contradictory outcomes establish observed nondeterminism under the recorded conditions, not the underlying cause. Investigate seeds, ordering, parallelism, time and external dependencies. A burst of retry passes is weak evidence for a rare failure.

For illustration, with 30 independent runs and zero failures, the one-sided 95% upper bound on the failure probability is approximately `1 - 0.05^(1/30) = 9.5%`. Real retries may not be independent, making that simplified calculation optimistic. Use this to avoid overstating evidence, not as a universal quarantine threshold.

## 12. Isolated execution and private runner fleet

Repository code, dependencies, tests, logs and agent-suggested commands are untrusted execution inputs. A subprocess on the control server is not an acceptable managed tenant-isolation boundary.

### Execution choices

| Option | Appropriate use | Constraint |
| --- | --- | --- |
| Hardened containers on dedicated customer workers | Early supported execution inside a customer-controlled trust domain | Shared-kernel risks and host access require explicit review |
| Managed microVM workers | Stronger per-investigation isolation for Linux workloads | Host/KVM provisioning, image maintenance and fleet engineering |
| Managed sandbox service | Faster implementation of disposable execution | Verify networking, residency, identity, lifecycle and pricing contract |
| Full virtual machines | Windows, specialized toolchains and broader OS support | Slower/heavier provisioning and image lifecycle |

[Firecracker](https://firecracker-microvm.github.io/) is a relevant component, but it requires Linux/KVM. It is not a native drop-in runner on this Mac workspace. Use a Linux execution host or a suitable managed service; local development can use a separate supported backend.

### Required fleet capabilities

Enrollment with short-lived credentials; capability/architecture labels; lease and fencing epoch; per-tenant concurrency quotas; CPU/memory/disk/network budgets; cancellation and forced termination; artifact upload independent of worker success; cleanup confirmation; orphan sweeper; image patching and revocation; tenant/trust-scoped cache policy; health telemetry.

Prefer an outbound connection from private runners. A customer should not expose a general remote shell to the control plane.

A crashed worker may have completed a side effect before losing its response. The coordinator must reconcile the side effect rather than blindly retrying an external action.

## 13. Repair engine and independent verification

### Repair planning

Classify permitted repair families: source-code change, generated-code regeneration, lockfile reconciliation, test determinism, service readiness, or execution-recipe change. Each family has different acceptance rules.

A patch proposal includes:

- Hypothesis and source evidence.
- Exact base revision and reproduction manifest.
- Allowed file paths and maximum edit budget.
- New/modified tests with purpose.
- Verification plan and non-goals.
- Model/tool/prompt versions and resource budget.
- Remaining uncertainty and any human decision required.

The repair engine may propose a few competing candidates. Bound attempts by elapsed time, compute and token budget, and stop when the expected benefit falls below the remaining cost.

### Verification ladder

1. **Precondition:** Original symptom is reproduced or explicitly marked unproven.
2. **Diff inspection:** No unapproved test deletion, assertion weakening, skipped checks, swallowed exceptions, reduced coverage requirements or permission changes.
3. **Targeted test:** Original failing behavior passes after the change and fails before it under comparable conditions.
4. **Independent oracle:** Existing protected tests, separately authored regression tests, property checks or differential checks evaluate the intended behavior.
5. **Broader validation:** Build, type checks, lint, relevant integration suites and customer-required security checks.
6. **Original CI:** Provider checks run on the exact candidate/merge-group commit required by repository policy.
7. **Review/merge:** Required human reviews and native branch rules remain effective.
8. **Recurrence:** Observe the configured follow-up window and record a new occurrence or escaped regression.

The verifier should be unable to overwrite the protected oracle or silently reduce its scope. A second model reading the same flawed assumptions is not independent verification.

### Merge freshness

Bind approval to repository, head/base or merge-group SHA, diff digest, policy version, required test set and evidence bundle. A force push, rebase, relevant base change or recipe change invalidates applicable evidence.

GitHub's merge queue requires checks on merge-group commits and documents the `merge_group` trigger. Integrate that identity rather than treating a successful PR-head run as equivalent. [Merge queue documentation](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue).

## 14. Workflow durability and state model

Use durable orchestration for long-running work, timers, retries, cancellation and approvals. Temporal is a candidate. Its [retry documentation](https://docs.temporal.io/encyclopedia/retry-policies) distinguishes retry behavior and emphasizes idempotent activities. It does not make arbitrary GitHub or infrastructure side effects exactly once.

### Investigation lifecycle

`received → source_verified → evidence_ready → recipe_ready → reproducing → reproduced / unreproduced → experimenting → candidate_ready → verifying → verification_failed / policy_review → pr_open → required_checks_passed → merged → observing → resolved`

Additional explicit states: `blocked_permissions`, `blocked_environment`, `budget_exhausted`, `superseded`, `cancelled`, `expired`, and `needs_human`.

Do not encode the full lifecycle as one editable `status` string. Store typed records and transition events with optimistic concurrency and constraints.

### Side-effect correctness

- Persist an outbox event with the domain transaction.
- Give each logical external action a stable operation ID.
- Use provider idempotency mechanisms where available.
- For APIs without them, persist intent, correlate external results and reconcile before retry.
- Use leases/fencing to reject late worker results from cancelled or superseded attempts.
- Separate retryable transport errors from deterministic invalid inputs.
- Compensate by closing an obsolete draft PR or revoking an unused resource where appropriate; do not assume every action has a reversible inverse.

An ambiguous branch push or PR creation remains an unknown state until provider reconciliation determines the outcome.

## 15. Data model and evidence contract

### Core entities

| Domain | Principal records |
| --- | --- |
| Organization | Organization, Membership, Role, ProviderInstallation, RepositoryBinding |
| Execution source | WorkflowDefinition, WorkflowRun, RunAttempt, Job, Step, CheckRun, Artifact |
| Investigation | FailureInstance, FailureClusterVersion, Investigation, Hypothesis, Observation, Experiment |
| Reproduction | RecipeVersion, EnvironmentManifest, ServiceFixture, CacheNamespace, Runner, Lease, ExecutionAttempt |
| Repair | RepairPlan, CandidatePatch, VerificationPlan, VerificationResult, ReviewDecision |
| Change outcome | ExternalOperation, PullRequestLink, HeadRevision, MergeGroup, MergeOutcome, Recurrence |
| Governance | PolicyBundle, CredentialReference, EvidenceBundle, AuditEvent, RetentionRule, UsageLedger |

Use PostgreSQL for authoritative workflow metadata and tenant-scoped relationships; object storage for large logs, reports, snapshots and evidence bundles. Keep a graph of relationships as relational tables initially. Introduce a separate graph/search/analytics engine only when query workload and latency justify it.

### Evidence bundle

Every verification decision should reference an immutable bundle containing source identities, tested candidate digest, environment and recipe versions, selected and executed tests, test counts, baseline/candidate results, logs/artifacts, policy inputs/decision, executor identity, timestamps, and stated coverage gaps.

SLSA provenance provides a relevant model for recording build inputs and execution context, not a substitute for functional testing. [SLSA provenance](https://slsa.dev/spec/v1.2/provenance).

Signing can establish the issuer and integrity of a bundle under a trust policy. It cannot establish that a test was relevant, a worker was uncompromised, or a repair was semantically correct. Private evidence should not be accidentally disclosed through public transparency infrastructure.

### Isolation

Include organization scope in every reference and object key. Enforce authorization both at the API and database/object-access boundary. Test cross-tenant reads, writes, joins, exports, search indexes, caches and signed URLs. Administrative support access requires explicit controls and a customer-visible audit trail.

## 16. Technology and build-versus-buy decisions

| Concern | Proposed choice | Reason and decision trigger |
| --- | --- | --- |
| Control API and UI | TypeScript API with a structured web application | Fits current engineering context; organization model and contracts matter more than framework choice |
| Durable metadata | PostgreSQL | Transactions, constraints, scoped access and mature backup tooling |
| Orchestration | Temporal or comparable durable workflow engine | Avoid rebuilding long-running retry/approval/recovery semantics |
| Runners | Go or another suitable systems language, plus execution backends | Process/fleet control and small distribution footprint; validate team expertise |
| Large evidence | S3-compatible object storage with explicit tenant scope | Avoid putting full artifacts in hot relational rows |
| Telemetry | OpenTelemetry plus chosen metrics/log backend | Observe workflows, attempts and cost; version adopted conventions |
| Policy | Versioned typed policy; evaluate OPA if customer rules justify it | Policies must be replayable and testable; avoid bespoke authorization sprawl |
| LLM access | Provider-neutral inference interface with usage accounting | Route by capability/privacy/budget; test actual models on private failure cases |
| Search/relationships | PostgreSQL indexes and explicit edges initially | Add vector/graph infrastructure only for measured retrieval needs |
| CI execution | Existing CI and supported private/sandbox execution | Preserve customer workflows instead of requiring CI replacement |

[OpenTelemetry CI/CD conventions](https://opentelemetry.io/docs/specs/semconv/cicd/) offer shared terminology; verify maturity/version before locking schemas. [OPA](https://www.openpolicyagent.org/docs) provides an existing policy-evaluation model. [Bazel remote execution](https://docs.bazel.build/versions/main/remote-execution.html) is an integration direction for suitable customers, not a requirement for every repository.

The exact software stack remains an architecture decision after connector and execution spikes. Do not introduce Kafka, Kubernetes, a graph database, multiple caches and many services merely to appear complex.

## 17. Enterprise controls that affect the product

Customer requirements include organization identity, scoped roles, SSO where required, repository-specific policy, private execution, controlled outbound network, model data-handling choices, retention/deletion, export, data residency, audit and emergency disable.

The planner must treat logs, repository instructions, issue descriptions and tool output as untrusted content. They cannot expand permissions, choose secret access, alter policy or authorize an external action. The tool gateway evaluates a structured action against policy independently of model prose.

Separate read-only observation, sandbox execution, draft-PR creation, branch push, merge and production action capabilities. Organizations should be able to enable the useful lower-trust capabilities without granting everything.

The product must handle customer deletion across raw objects, metadata, search indexes, caches, model context stores and backups according to the documented retention policy. Do not promise immediate removal from immutable backups without an actual retention and key-management design.

## 18. User experience for a substantial platform

### Main surfaces

1. **Organization command center:** Failure volume, affected repositories, active investigations, queue health, cost and accepted repairs.
2. **Investigation workspace:** Source timeline, failure instances, hypotheses, environment differences and evidence.
3. **Experiment view:** Baseline/candidate comparison, controlled variables, results and remaining confounders.
4. **Repair review:** Diff, intent, protected-oracle results, required checks and policy explanation.
5. **Execution fleet:** Runner capabilities, leases, image versions, capacity, health and active resource consumption.
6. **Policy and access:** Repository permissions, allowed actions, budgets, required evidence and approval rules.
7. **Evaluation laboratory:** Replay cohorts, baseline competitors, quality regressions and model/version comparison.

Complexity should be behind the workflow. A developer opening a failed PR should see the original symptom, what was reproduced, the proposed repair, its verification limits, and the next required decision.

A graph visualization is useful when tracing an actual shared failure or change dependency. It should not be the mandatory home screen for every user.

## 19. Scale and operating assumptions

The following workload is an illustrative engineering scenario, not measured demand or a forecast:

- 20 customer organizations.
- 100,000 CI jobs/day across the service.
- 5% failed job attempts: 5,000/day.
- Deduplication and related-failure aggregation yield 2,000 investigation opportunities/day.
- Budgets and eligibility admit 400 investigations/day to active experiments/repair.
- Four sandbox attempts per admitted investigation, ten minutes each, 4 vCPU and 8 GiB memory.

This implies **1,600 sandbox attempts/day**, **64,000 vCPU-minutes/day**, **128,000 GiB-minutes/day**, and approximately **11.1 average concurrent sandboxes** if uniformly distributed over 24 hours. A tenfold arrival burst suggests roughly 111 active slots before adding headroom or considering queueing and provisioning delays. These are arithmetic capacities, not performance guarantees.

At 200 KiB of retained raw evidence per CI job, intake is approximately 19.1 GiB/day or 572 GiB over 30 days before indexes, replicas and reproduction artifacts. Real logs and browser traces can be much larger.

### Proposed service objectives

- Acknowledge durably accepted webhooks within a one-second p95 target under the agreed load.
- Produce an initial evidence-based summary within two minutes for supported available artifacts.
- Reproduce and propose a repair within 15–30 minutes for eligible bounded jobs, excluding customer approval and external queues.
- Make cancellation prevent new work promptly and terminate leased execution within a defined backend-specific bound.
- Begin with a 99.9% control-plane availability target after production hardening; define maintenance, degradation and measurement explicitly.

All are proposed acceptance targets requiring load tests and operational evidence.

## 20. Economics and commercial model

### Unit cost model

`Investigation cost = environment startup + CPU/memory execution + inference + artifact storage/transfer + control-plane allocation + support/review + unsuccessful attempts.`

Charge for useful completed workflow capacity, with a base organization fee, included investigation budget, transparent execution/inference usage and private-deployment support. Avoid unlimited autonomous attempts and a fee structure that rewards repeated ineffective fixes.

### Illustrative monthly scenario

These are hypothetical input rates, **not cloud/provider price quotes**:

| Cost component | Assumption | Monthly arithmetic |
| --- | --- | ---: |
| vCPU | 64,000 vCPU-min/day × $0.04/vCPU-hour × 30 days | $1,280 |
| Memory | 128,000 GiB-min/day × $0.005/GiB-hour × 30 days | $320 |
| Inference | 400 investigations/day × $0.40 × 30 days | $4,800 |
| Storage/network/control allocation | Placeholder | $1,000 |
| Customer delivery/support allocation | Placeholder | $2,500 |
| **Total example delivery cost** | Excludes product development and sales | **$9,900/month** |

At 20 customers paying an assumed $2,000/month, revenue would be $40,000/month and this simplified delivery contribution would be $30,100, or 75.25%. It is not an audited gross-margin estimate. Idle capacity, reserved hosts, richer inference, full-suite reruns, third-party minimums and enterprise support could materially raise costs.

If inference cost per admitted investigation rises from $0.40 to $2, the example adds $19,200/month in inference expense alone. Cost-aware scheduling and a demonstrated repair success rate are central product requirements.

### Commercial experiments

- Paid failure-triage/reproduction pilot: $10,000–$30,000 for a bounded period and agreed workload.
- Initial team/organization subscription: $2,000–$8,000/month plus explicit usage limits.
- Enterprise/private deployment: quoted after security, runner, support and residency requirements are known.

These are pricing hypotheses. Customer interviews and paid pilots must validate both willingness to pay and procurement feasibility.

## 21. Evaluation: how to know it actually works

[SWE-bench](https://www.swebench.com/SWE-bench/faq/) provides software-repair benchmark tasks, including a 500-instance Verified set. It is useful for an initial agent harness check, but our CI incidents include environment drift, services, dependencies, tests and provider behavior beyond that benchmark's scope.

[SWE-agent research](https://arxiv.org/abs/2405.15793) supports the importance of the agent/computer interaction design. It does not prove our proposed product's success. [SWE-rebench research](https://arxiv.org/abs/2505.20411) discusses fresh task collection and contamination concerns, reinforcing the need for customer-specific held-out evaluation.

### Build a private evaluation corpus

Start with at least 200 labeled real failure instances from consenting design partners, then expand toward 1,000+ across repositories and time periods. These are proposed sample targets, not existing data.

Stratify by failure category, language, repository, environment complexity and severity. Include unsupported and unreproducible cases. Separate training/tuning from held-out repositories and later time periods; do not allow future fixes to leak into retrieval for earlier incidents.

### Measure the whole funnel

| Metric | Definition |
| --- | --- |
| Ingestion completeness | Provider-visible eligible jobs with complete normalized records / all eligible jobs |
| Failure extraction quality | Correct test/step failure instances against human labels |
| Grouping quality | Pairwise precision/recall and human-reviewed incorrect cluster merges |
| Reproduction rate | Cases reproducing the original symptom under a documented fidelity level / eligible cases |
| End-to-end resolution rate | Accepted, verified repairs / all eligible assigned failures, including abstentions and failures |
| Patch acceptance | Human-accepted candidates / proposed candidates; report edits separately |
| Unsafe proposal rate | Proposals that hide tests, change permissions, introduce defects or exceed repair policy |
| Verification false acceptance | Incorrect candidates accepted by the verifier / candidates evaluated with a valid oracle |
| Time saved | Measured hands-on investigation effort against a matched or randomized baseline |
| Cost per useful resolution | All attempted execution/inference/support cost / accepted verified repairs |
| Recurrence and escape | Related recurrence or introduced regression within an explicitly stated window |

Compare against the customer's existing process, native CI diagnostics, the chosen incumbent tool, and a general coding agent with a reasonable repository-specific setup. Report denominators and unsupported cases rather than only successful demos.

### Proposed pilot release gates

- At least three paying design partners with actual failure access.
- At least 60% reproduction on a predefined supported deterministic cohort, with a confidence interval and explicit exclusions.
- At least 30% median hands-on triage reduction in a valid within-team comparison.
- No hidden test suppression or privilege expansion in accepted pilot patches; investigate every observed violation.
- A measured cost ceiling and successful cleanup/cancellation tests.
- No claim of a low rare-error rate from a small sample. For example, zero harmful accepted patches in 1,500 independent cases gives an approximate 95% upper bound of 0.2% under the rule-of-three model, not proof of zero risk. Real cases may be correlated.

Datadog's [evaluation engineering article](https://www.datadoghq.com/blog/engineering/bits-ai-eval-platform/) is relevant evidence that serious incident agents need dedicated evaluation infrastructure; it is not a validation of our thresholds.

## 22. Implementation roadmap and acceptance gates

| Stage | Indicative period | Engineering outcome | Acceptance gate |
| --- | --- | --- | --- |
| 0. Discovery and technical spikes | Weeks 1–4 | Real failure corpus, buyer workflow, provider permissions, private runner/environment spike | Confirm repeated pain and viable data/execution access |
| 1. Enterprise evidence foundation | Months 2–3 | Organization model, GitHub App collector, artifact pipeline, typed failures, Postgres state and durable orchestration | Redelivery/order/reconciliation and tenant-isolation tests on real runs |
| 2. Reproduction platform | Months 3–6 | Versioned recipes, disposable services, private runners, manifest comparison and replay UI | Reproduce supported partner failures and demonstrate cancellation/cleanup |
| 3. Repair and independent verifier | Months 5–9 | Candidate engine, protected oracle, diff constraints, PR/check integration and cost budgets | Held-out evaluation and paid pilot gates |
| 4. Enterprise production hardening | Months 8–12 | SSO, operational recovery, audit/export/retention, signed evidence and production support | Customer acceptance and operational readiness review |
| 5. Provider/language expansion | Months 12–18 | Second provider, more languages, shared-environment diagnosis and contract integrations | Repeatable deployment economics and quality maintained by cohort |
| 6. Broader engineering reliability | Months 18–30 | Modernization campaigns, incident-to-regression loops and deeper cross-repository repair | Separate validated customer demand and permissions for each adjacency |

Timelines overlap because specialist work can proceed concurrently within an appropriately staffed organization. They are planning assumptions, not commitments to implement enterprise scope in one coding session.

### First substantial engineering milestone

Deliver one complete production-shaped loop:

**GitHub App event → authoritative failed-job collection → isolated recipe-based reproduction → bounded candidate patch → protected regression verification → draft PR with evidence → original CI reconciliation.**

This milestone is the product's technical core. Additional dashboard tabs do not substitute for it.

## 23. Staffing and investment planning

A production enterprise platform needs expertise in CI/build systems, distributed orchestration, untrusted execution, program repair, evaluation, enterprise identity, integrations, operations and customer deployment.

**Proposed initial group, 8–12 people:** Engineering/product lead, 2–3 backend/distributed-systems engineers, 2 execution/build engineers, 1–2 applied AI/evaluation engineers, frontend/product engineer, and customer/deployment engineer. Security and domain specialists may be shared initially.

**Production expansion, 25–40 people:** Dedicated provider adapters, runtime images, runner fleet, verification/evaluation, enterprise platform, reliability, developer experience and customer success.

**Broader platform, 50–90 people:** Multiple providers/languages/OS families, international enterprise delivery, 24-hour support coverage, partner integrations and differentiated product modules.

A 24-month funding envelope of **$8 million–$20 million** is a planning scenario for substantial mixed-location engineering, infrastructure and go-to-market. It is not a compensation survey or funding requirement established by market evidence. Geography, seniority, managed-service choices, sales motion and security obligations can move this materially.

Release capital against reproduction, repair-quality, paid-adoption and delivery-margin gates. Large ambition does not require funding every later module before proving the central loop.

## 24. Major failure modes and design responses

| Failure | Required response |
| --- | --- |
| Agent makes tests green by weakening behavior | Protected oracles, diff constraints and independent review |
| Candidate passes only because the environment changed | Comparable baseline and explicit environment evidence |
| Correlated failures incorrectly merge | Versioned clusters, evidence, human split/merge and reversible classification |
| A repair is verified against an obsolete PR head | Invalidate evidence on identity changes; reverify required commits |
| Worker finishes after cancellation | Fenced leases and rejection of obsolete result/action submissions |
| PR creation succeeds but response is lost | Reconcile a persisted external-operation identity before retry |
| An untrusted log instructs the agent to reveal secrets | Data/instruction separation, independent authorization and constrained secret access |
| Cache moves untrusted output into trusted execution | Tenant/trust-scoped caches and explicit promotion rules |
| Artifact parsing exhausts memory or escapes paths | Bounded parsing/decompression, sandboxed parsers and archive path validation |
| Large jobs make every investigation uneconomic | Admission control, partial experiments, cost caps and abstention |
| Customer's service state cannot be reproduced | Report blocked/unreproduced state and request the missing execution contract |
| Product improves summaries but not engineer effort | Measured workflow comparison and willingness to stop the thesis |

## 25. What to preserve, replace, and prove next

**Preserve:** The investigation workflow concept, explicit distinction between suggestions and evidence, deduplication principles, occurrence history, owner/notes UI patterns, and local-first ability to inspect data.

**Redesign:** The authoritative schema, organization/installation identity, execution contract, verification semantics, artifact storage and lifecycle state machine. Migrate local records as legacy user-imported evidence; do not silently upgrade them to provider-authenticated or independently attested records.

**Prove first:** Access to real failure data, a reproducible supported cohort, one isolated execution backend, and a verification oracle that cannot be gamed by trivial “green” changes.

**Investment recommendation:** Pursue the enterprise CI investigation/reproduction/repair platform if paying design partners confirm the need and the full workflow outperforms incumbent CI plus a general coding agent. The opportunity is technically substantial, but the competitive advantage is still unvalidated.

## 26. Source register

Official sources and original research consulted on 4 November 2026. Unquoted architecture and commercial recommendations are our analysis. Vendor claims are not independently audited.

| # | Source | What it supports |
| --- | --- | --- |
| 1 | [Harness platform](https://www.harness.io/products/platform) | Broad autonomous SDLC competition |
| 2 | [Trunk Flaky Tests overview](https://docs.trunk.io/flaky-tests/overview) | Existing flaky-test investigation and quarantine workflows |
| 3 | [Buildkite Test Engine](https://buildkite.com/platform/test-engine/) | Existing test analytics/orchestration platform |
| 4 | [CircleCI Smarter Testing](https://circleci.com/docs/guides/test/smarter-testing-overview/) | Existing selection, splitting and rerun capabilities |
| 5 | [Devin introduction](https://docs.devin.ai/get-started/devin-intro) | Repository/environment/automation agent scope |
| 6 | [Devin application positioning](https://app.devin.ai/) | Explicit CI-fix competitive positioning |
| 7 | [Cursor background agents](https://prod.cursor.com/help/ai-features/background-agents) | Existing remote coding agent/environment controls |
| 8 | [Sentry automated debugging workflow](https://blog.sentry.io/automated-debugging-workflow-sentry/) | Existing error-to-plan/patch/PR workflow |
| 9 | [Datadog Bits AI agents](https://www.datadoghq.com/product/ai/bits-ai-agents/) | Incident investigation/remediation competition |
| 10 | [Greptile introduction](https://www.greptile.com/docs/introduction) | Repository-aware automated code review |
| 11 | [Aviator](https://www.aviator.co/) | Review and merge workflow integration |
| 12 | [GitHub installation authentication](https://docs.github.com/en/apps/creating-github-apps/authenticating-with-a-github-app/authenticating-as-a-github-app-installation) | App credentials and token lifetime |
| 13 | [GitHub webhook validation](https://docs.github.com/en/webhooks/using-webhooks/validating-webhook-deliveries) | Raw-body webhook signature verification |
| 14 | [GitHub Actions secure use](https://docs.github.com/en/actions/reference/security/secure-use) | Untrusted workflow/code/artifact boundary |
| 15 | [GitHub merge queues](https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/configuring-pull-request-merges/managing-a-merge-queue) | Merge-group/check identity |
| 16 | [Bazel hermeticity](https://bazel.build/basics/hermeticity) | Source/tool/environment isolation concepts |
| 17 | [Bazel remote execution](https://docs.bazel.build/versions/main/remote-execution.html) | Distributed build/test execution integration |
| 18 | [Firecracker](https://firecracker-microvm.github.io/) | Linux/KVM microVM execution option |
| 19 | [Temporal retry policies](https://docs.temporal.io/encyclopedia/retry-policies) | Retry and idempotent-activity design |
| 20 | [SLSA provenance](https://slsa.dev/spec/v1.2/provenance) | Build input/execution provenance model |
| 21 | [OpenTelemetry CI/CD semantic conventions](https://opentelemetry.io/docs/specs/semconv/cicd/) | CI/CD telemetry vocabulary |
| 22 | [Open Policy Agent](https://www.openpolicyagent.org/docs) | Existing policy evaluation component |
| 23 | [Pact can-i-deploy](https://docs.pact.io/pact_broker/can_i_deploy) | Recorded consumer/provider compatibility |
| 24 | [SWE-bench FAQ](https://www.swebench.com/SWE-bench/faq/) | Repair benchmark scope and datasets |
| 25 | [SWE-agent paper](https://arxiv.org/abs/2405.15793) | Agent/computer interaction research |
| 26 | [SWE-rebench paper](https://arxiv.org/abs/2505.20411) | Fresh task collection and contamination concerns |
| 27 | [Datadog agent evaluation platform](https://www.datadoghq.com/blog/engineering/bits-ai-eval-platform/) | Evaluation-system engineering example |

Local sources inspected: `src/analyze.js`, `src/store.js`, `src/verify.js`, `src/server.js`, `README.md`, and `docs/ARCHITECTURE.md` in the CI Resolve repository. Local implementation observations are separate from external market evidence.