Agent-HAnS Annotation Audit

How to read this page

Agent-HAnS uses three kinds of markers, called embedded feature annotations, to record which feature a piece of code belongs to. A folder mapping tags a whole folder. A file mapping tags a whole file, listed in a small companion file named .feature-to-file. A fragment mapping tags a specific block or line of code with a comment such as &begin[Groups] ... &end[Groups], or &line[Groups] for a single line. All of them point at an entry in .feature-model, the tree of every feature in the app.

For every annotation we checked two things. First, does the feature it points at still exist? A merge or a split can remove a feature, leaving old annotations pointing at nothing, we call that stale. Second, assuming the feature does exist, does the annotated code actually implement it? If the code doesn't match, or the annotation covers more than it should, we call that misplaced. Everything else is correct. A begin/end block is also either balanced, meaning its &begin has a matching &end, or not.

Final feature model (.feature-model)

ReferenceManager
    Database
    ApiDocs
    Versioning
    Papers
        CreatePaper
        GetPaper
        UpdatePaper
        DeletePaper
        ImportBibtex
        ExportBibtex
        SearchPapers
    Authors
        CreateAuthor
        GetAuthor
        ListAuthors
        UpdateAuthor
        DeleteAuthor
        MergeAuthors
    Groups
        CreateGroup
        GetGroup
        UpdateGroup
        DeleteGroup
        AddPaperToGroup
        RemovePaperFromGroup

25 non-root features, 6 top-level. Tags and Collections existed earlier and were merged into Groups at Evolution 5; Authors was promoted from Papers.Authors to top-level at Evolution 8 (split). Note that Authors has a dedicated ListAuthors feature, but Papers and Groups do not have a corresponding ListPapers / ListGroups — this asymmetry turns out to matter below.

Summary

Being balanced and pointing at a real feature name is a low bar, every begin/end block clears it, and every referenced feature name exists. But reading the actual code inside each block tells a stricter story: 3 of the 36 blocks (8%) are tagged with a feature their code doesn't actually implement. The two charts below show both counts side by side, the two mapping mechanisms Agent-HAnS uses (whole-file mappings and in-code fragment markers) land at different accuracy.

File-level mappings (.feature-to-file) — 20 total

Do the annotations point at the right code?

Checking that &begin[Groups] names a feature that exists is a name lookup, it does not tell you whether the code between &begin[Groups] and &end[Groups] actually implements Groups. To check that, we read every one of the 36 blocks by hand against the code it wraps. Three resolve to a real feature yet are misplaced, meaning the code inside doesn't match, or over-broad, meaning the block covers more than the one feature it names.

// &begin[Groups]
new Paper
{
    Title = "On Using LLMs to 'Featurize' Software",
    ...
},
new Paper { Title = "An IDE Plugin for Clone Management...", ... },
new Paper { Title = "Visualizing Feature-Oriented Software Evolution", ... },
... (4 more Paper object literals)
// &end[Groups]

// &begin[GetGroup]
app.MapGet("/groups", ...)          // list all groups — named "ListGroups"
    .WithName("ListGroups")...

app.MapGet("/groups/{id:int}", ...) // get one group — named "GetGroup"
    .WithName("GetGroup")...
// &end[GetGroup]

// &begin[GetPaper]
app.MapGet("/papers", ...)          // list all papers — named "ListPapers"
    .WithName("ListPapers")...

app.MapGet("/papers/{id:int}", ...) // get one paper — named "GetPaper"
    .WithName("GetPaper")...
// &end[GetPaper]

Full fragment list — 36 begin/end blocks + 14 line annotations

Feature coverage by fragment annotations

All 25 features in the model are referenced by at least one fragment annotation: Database, ApiDocs, Versioning, Papers, CreatePaper, GetPaper, UpdatePaper, DeletePaper, ImportBibtex, ExportBibtex, SearchPapers, Authors, CreateAuthor, GetAuthor, ListAuthors, UpdateAuthor, DeleteAuthor, MergeAuthors, Groups, CreateGroup, GetGroup, UpdateGroup, DeleteGroup, AddPaperToGroup, RemovePaperFromGroup. 25/25 — full coverage, independent of the placement findings above (coverage counts a feature as "referenced" even where the reference is over-broad, e.g. GetGroup / GetPaper).

Coverage-by-feature is not the same question as coverage-by-file. A feature can be "covered" by one annotated file while a second file that also implements it carries nothing. The section below checks the latter.

Files that should have an annotation but don't

So far, every check has started from an annotation that exists and asked whether it's correct. This section asks the reverse question: are there files that clearly implement a feature, yet carry no annotation at all? Of the 41 .cs files in ReferenceManager, 23 carry at least one annotation and 18 carry none. Most of the 18 are legitimately infrastructural, files a human wouldn't annotate either, such as EF-generated *.Designer.cs siblings of already-tagged migrations, the auto-generated AppDbContextModelSnapshot.cs, the test-harness CustomWebApplicationFactory.cs, the three endpoint test suites (discussed separately below), and the generic cross-feature Responses/PagedResult.cs. Two are not, and those two are real gaps.

Test files are excluded from this check by design. Embedded feature annotations track where a feature is implemented, and tests verify a feature rather than implement it, so an unannotated test suite is not a recall failure. The interesting fact about the test suites turns out to run the other way, see below.

public partial class StandaloneAuthors : Migration
{
    protected override void Up(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.CreateTable(name: "Authors", ...);
        migrationBuilder.CreateTable(name: "PaperAuthor", ...);
        // ... migrates existing JSON-embedded authors into standalone rows ...
        migrationBuilder.DropColumn(name: "Authors", table: "Papers");
    }
}
// no &begin[Authors] anywhere in this file

public partial class AddJournalAndBooktitleToPaper : Migration
{
    protected override void Up(MigrationBuilder migrationBuilder)
    {
        migrationBuilder.AddColumn<string>(name: "Booktitle", table: "Papers", ...);
        migrationBuilder.AddColumn<string>(name: "Journal", table: "Papers", ...);
    }
}
// no &line[ImportBibtex] anywhere in this file