Research Candidate Evaluation V1
Research Candidate Evaluation V1
Objective
Define the first bounded evaluation layer for research candidates.
This layer consumes:
research_problem_packetcandidate_search_tree_packetresearch_candidate_packet[]- optional
evidence_packet
It emits:
research_evaluation_packet[]candidate_evaluation_summary_packet
Why Evaluation Comes Next
The research system now has:
- problem materialization
- hypothesis branching
- candidate tree search
Without evaluation, the system still cannot distinguish:
- plausible candidates
- feasible candidates
- evidence-supported candidates
- candidates ready for recommendation
Candidate evaluation is the gate between search and recommendation.
Research Basis
Canonical bibliography:
docs/current-research-bibliography.mdartifacts/planner/research/bibliography-graph.json
The design is grounded in four research threads.
Self-Refineshows that iterative feedback can improve generated outputs, but the feedback signal must be explicit enough to guide refinement: https://arxiv.org/abs/2303.17651Reflexionshows that feedback and retained context can improve agent performance across decision and coding tasks: https://arxiv.org/abs/2303.11366CRITICshows that tool-interactive critique can improve self-correction compared with unsupported introspection: https://arxiv.org/abs/2305.11738SWE-benchshows why software candidates need task-grounded evaluation against realistic code issues rather than plausibility-only scoring: https://arxiv.org/abs/2310.06770
Design consequence:
- candidate evaluation should combine structural checks, evidence support, feasibility priors, and executable or artifact-level checks where available
- the evaluator should emit machine-readable records before any recommendation is allowed
Hard Boundary
This slice must not become:
- candidate generation
- final recommendation
- planner
- governance
- code-writing runtime
It only evaluates candidate packets and records what is known.
Evaluation Model
V1 uses bounded static and contract-level evaluation.
It does not require full execution of candidate code.
Evaluation dimensions:
contract_completenesstraceabilityevidence_supportfeasibilityartifact_readinessrisk_level
The output should be explicit about whether a score is:
- observed from artifacts
- inferred from packet content
- missing and therefore penalized
Evaluation Gates
A candidate should be blocked from recommendation if:
- it does not trace to a source hypothesis
- it has no candidate family
- it has no approach summary
- it has no proposed changes or artifact target
- it has unresolved high-risk feasibility priors and no evidence support
Blocked candidates still receive evaluation packets.
New Packet
candidate_evaluation_summary_packet
Purpose:
- summarize evaluation results for one candidate search run
Required fields:
- all
packet_basefields problem_idcandidate_search_tree_refevaluation_packet_refspromoted_candidate_idsblocked_candidate_idsevaluation_policy
research_evaluation_packet Tightening
Required fields should include:
- all
packet_basefields evaluation_idproblem_idcandidate_idsource_hypothesis_idevaluation_methodmetricsfailureswarningsartifact_refsevidence_refspromotion_status
Scoring
V1 should compute a simple normalized score:
contract_completeness:0.0to1.0traceability:0.0to1.0evidence_support:0.0to1.0feasibility:0.0to1.0artifact_readiness:0.0to1.0risk_penalty:0.0to1.0
Promotion rule:
- candidate can be promoted if total score is at least
0.65 - candidate must not have blocking failures
V1 promotion does not mean final recommendation.
It only means the candidate is eligible for ranking and recommendation.
Acceptance
V1 is successful when:
- every generated candidate receives an evaluation packet
- evaluation records trace back to hypotheses and evidence
- blocked candidates are explicit
- promoted candidates are explicit
- recommendation is impossible without evaluation records