🛠️ Update various documentation, scripts, and configuration templates to enhance clarity, functionality, and maintainability across the project
This commit is contained in:
127
docs/audit-workflow.md
Normal file
127
docs/audit-workflow.md
Normal file
@@ -0,0 +1,127 @@
|
||||
# Audit Workflow
|
||||
|
||||
## Purpose
|
||||
|
||||
The audit workflow reviews the last 30 days of persisted local Copilot
|
||||
artifacts and produces a shortlist of reusable patterns worth promoting into the
|
||||
shared Copilot resource repository.
|
||||
|
||||
## Current Scope
|
||||
|
||||
- macOS first
|
||||
- read-only audit execution
|
||||
- file-based review interface
|
||||
- local per-machine audit history stored inside the repo checkout but excluded
|
||||
from git
|
||||
|
||||
## Command
|
||||
|
||||
```bash
|
||||
resources/scripts/audit-copilot-usage.sh --days 30
|
||||
```
|
||||
|
||||
The runner excludes the current `copilot-resources` repo root from workspace
|
||||
indexing by default so you do not review patterns that already live in the
|
||||
shared repository.
|
||||
|
||||
Optional filters:
|
||||
|
||||
```bash
|
||||
resources/scripts/audit-copilot-usage.sh --days 14 --workspace copilot-resources
|
||||
```
|
||||
|
||||
```bash
|
||||
resources/scripts/audit-copilot-usage.sh --days 30 --include-sources publish-log,repo-memories
|
||||
```
|
||||
|
||||
```bash
|
||||
resources/scripts/audit-copilot-usage.sh --days 30 --exclude-workspace /path/to/another/repo
|
||||
```
|
||||
|
||||
## Storage Model
|
||||
|
||||
Audit runs are written under:
|
||||
|
||||
```text
|
||||
.local/audits/<machine-id>/<timestamp>/
|
||||
```
|
||||
|
||||
The runner creates the machine directory automatically on first use. The entire
|
||||
`.local/` tree is git-ignored so each machine can keep its own audit history
|
||||
without creating commit noise.
|
||||
|
||||
## Inputs
|
||||
|
||||
The first implementation reads from these local sources when they exist:
|
||||
|
||||
- VS Code workspace transcripts under `~/Library/Application Support/Code/User/workspaceStorage/*/GitHub.copilot-chat/transcripts/`
|
||||
- Repo memories under `.../GitHub.copilot-chat/memory-tool/memories/repo/`
|
||||
- Shared resource publish history under `~/.copilot-resources-state/publish-log.tsv`
|
||||
|
||||
## Outputs
|
||||
|
||||
- `audit-summary.md` — high-level inventory and top candidates
|
||||
- `workspace-index.tsv` — workspaceStorage hash to workspace path mapping for the run
|
||||
- `candidates-report.tsv` — scored candidate list with source references plus transcript prompt-cost proxy columns such as `token_cost_signal`, `avg_prompt_chars`, and `repeated_prompt_chars`
|
||||
- `selection-manifest.tsv` — editable approval surface with `decision`, `review_note`, `detail_file`, and the same transcript prompt-cost proxy columns used during triage
|
||||
- `pattern-details/*.md` — per-candidate evidence bundles used during review, including purpose, expected benefit, audit context, score rationale, and caveats
|
||||
|
||||
Transcript prompt-cost fields are character-count proxies derived from persisted
|
||||
user message text. They help surface repeated long prompts that are likely to be
|
||||
more expensive, but they are not exact billing-token measurements.
|
||||
|
||||
## Selection Interface
|
||||
|
||||
Use `selection-manifest.tsv` as the review surface and `pattern-details/*.md` as
|
||||
the one-by-one evidence bundle for each candidate.
|
||||
|
||||
Inside Copilot, use the shared `review-audit-candidates` prompt to work through
|
||||
pending rows one at a time. That prompt should ask for a decision with buttons,
|
||||
capture optional freeform review notes, explain why the candidate exists and
|
||||
why it scored as it did, and update `selection-manifest.tsv` after each answer.
|
||||
|
||||
Recommended decision states:
|
||||
|
||||
- `promote-skill`
|
||||
- `promote-instruction`
|
||||
- `promote-agent`
|
||||
- `promote-hook`
|
||||
- `promote-script`
|
||||
- `promote-prompt`
|
||||
- `template-only`
|
||||
- `docs-only`
|
||||
- `discard`
|
||||
- `needs-sanitization`
|
||||
|
||||
## Promotion Rule
|
||||
|
||||
If a candidate cannot be explained as a portable skill, instruction, prompt
|
||||
adapter, agent behavior, hook, script, or template, it should stay in audit
|
||||
notes and not be promoted into the shared repo.
|
||||
|
||||
## Follow-on Draft Generation
|
||||
|
||||
After the manifest has approved rows, generate next-step artifacts with:
|
||||
|
||||
```bash
|
||||
resources/scripts/prepare-audit-promotions.sh
|
||||
```
|
||||
|
||||
Or target a specific run:
|
||||
|
||||
```bash
|
||||
resources/scripts/prepare-audit-promotions.sh --audit-dir .local/audits/<machine-id>/<timestamp>
|
||||
```
|
||||
|
||||
The command reads approved rows from `selection-manifest.tsv` and writes:
|
||||
|
||||
- `draft-resources/` for straightforward prompt, instruction, agent, and skill drafts
|
||||
- `staging-notes/` for approved rows that still need manual design work
|
||||
- `promotion-summary.md` to summarize what was generated
|
||||
|
||||
## Limits
|
||||
|
||||
- This audits persisted local artifacts only.
|
||||
- It does not capture every inline completion or transient UI suggestion.
|
||||
- Windows parity is deferred until the macOS workflow and report format are
|
||||
proven.
|
||||
Reference in New Issue
Block a user