What is this project?

Small open-source repositories (under 500 GitHub stars) accumulate issue debt fast — duplicates, vague bug reports, stale PRs, and requests the maintainer will never action. Maintainers spend their weekends cleaning this up by hand. This project trains an LLM agent to do it for them.

The agent reads a backlog of GitHub-style issues and must: (1) triage each item as keep, close, need-info, or duplicate, (2) link duplicates to their canonical issue, and (3) assign a release priority. A programmatic grader scores the agent against a hidden ground-truth label set.

Architecture — how the files connect

Why does each file exist?

inference.py — the entry point

This is the only file the hackathon validator cares about. It must live at the project root. It reads three environment variables, connects to the LLM, runs the agent loop over all three tasks, and prints the required [START] / [STEP] / [END] lines to stdout. Every other file exists to support it.

env/environment.py — the RL environment

Implements the OpenEnv interface: reset() loads a task's issues into memory, step(action) applies one agent action and returns (observation, reward, done, info), and grade() calls the task's grader and returns a final score. It also handles loop detection — if the agent keeps repeating the same action, penalty accumulates at −0.05 per repeat.

env/models.py — why Pydantic?

Pydantic is imported because it provides data validation with zero boilerplate. Issue, Action, Observation, and Reward are all BaseModel subclasses. This means:

• Type errors are caught immediately when you construct an object
• .model_dump() converts any model to a plain dict (needed by graders)
• .model_copy(deep=True) deep-copies an issue without manual recursion

env/tasks/task_easy/medium/hard.py

Each file contains two things: a list of Issue objects (the synthetic backlog) and a GROUND_TRUTH dict mapping item IDs to the correct triage decision, priority, and duplicate target. The ground truth is never shown to the agent — it's only used by the grader.

env/graders/grader_*.py

Each grader receives the agent's trajectory and final state, then computes a weighted score. All graders are deterministic — same input always gives same output. The weights are: triage accuracy 55%, priority accuracy 35%, duplicate detection 10%. Hard difficulty adds extra penalties for deprioritizing security issues.

openenv.yaml

Metadata consumed by the openenv validate CLI and the hackathon submission system. Lists task IDs, difficulty levels, action space description, and reward range. Without this file, openenv validate . fails.

Dockerfile

Packages the entire project into a container with Python 3.11 and the two required dependencies (openai, pydantic). The hackathon runs submissions inside Docker with 2 vCPU / 8 GB RAM. The entrypoint is python inference.py.

Why are these libraries imported?

Library	Where used	Why it's needed
openai	inference.py	The hackathon mandates the OpenAI client for all LLM calls. It supports any OpenAI-compatible endpoint via base_url, so the same code works with Groq, HuggingFace, OpenRouter, etc.
pydantic	models.py, all tasks	Strict data validation. Ensures Reward.value is always a float, Action.action_type is always a string, etc. Prevents silent bugs from malformed agent output.
os	inference.py	Reads environment variables (HF_TOKEN, API_BASE_URL, MODEL_NAME) via os.getenv().
json	inference.py	Parses the LLM's JSON response into an Action object. Also handles stripping markdown code fences that some models include.
sys	task_*.py	Used to add the project root to Python's module search path so tasks can import from env.models without needing a package install.
copy	environment.py	Deep-copies issue lists on reset() so each episode starts from a clean state without mutating the original template.

Environment variables

How the agent loop works

The three tasks explained

Task	Items	What the agent faces	Key challenges
easy	10	Clean backlog with clear labels. One obvious duplicate pair (#3 and #7, both report the same KeyError). Two items to close outright (Python 2 request, GUI request).	Learning the basic signal: what's a real bug vs noise
medium	30	Noisy descriptions, borderline cases, multiple duplicate clusters, stale items (365-day-old question). Two open PRs that need to stay in the backlog.	Distinguishing "need-info" from "close", finding the right duplicate_of target
hard	61	Two-release milestone scope (v2.6 + v3.0), security issues that carry extra penalties if deprioritized, 6 cascading duplicate clusters, stale PRs from 2 years ago, conflicting community signals.	Global consistency, security sensitivity, release scoping, cascading duplicates

How reward is calculated

Understanding the output you saw

Your run produced scores of easy: 1.0000, medium: 0.9554, hard: 0.2828. Here's what those numbers mean and why the gap exists:

The required stdout format

The hackathon validator reads stdout line by line and parses these three line types exactly:

[START] task=easy env=issue-grooming-env model=Qwen/Qwen2.5-7B-Instruct
[STEP] step=1 action=triage_item({"item_id":1,"decision":"keep"}) reward=0.10 done=false error=null
[STEP] step=2 action=mark_duplicate({"item_id":3,"duplicate_of":2}) reward=0.12 done=false error=null
[STEP] step=10 action=done({}) reward=-0.20 done=true error=null
[END] success=true steps=10 rewards=0.10,0.12,0.08,...,-0.20

Rules: one [START] per task, one [STEP] per env.step() call, one [END] always emitted even on exception. success=true means the final grade was ≥ 0.5.

Real-world use cases

Setup and running

Prerequisites

• Python 3.10, 3.11, or 3.12
• A free API key from Groq (console.groq.com), Google AI Studio, or OpenRouter
• Docker (only needed for hackathon submission)

Install and run (PowerShell / Windows)

python -m venv .venv
.venv\Scripts\activate
pip install -r requirements.txt

$env:HF_TOKEN     = "gsk_your_groq_key"
$env:API_BASE_URL = "https://api.groq.com/openai/v1"
$env:MODEL_NAME   = "llama-3.1-8b-instant"

python inference.py

Install and run (bash / macOS / Linux)

python -m venv .venv && source .venv/bin/activate
pip install -r requirements.txt

export HF_TOKEN="gsk_your_groq_key"
export API_BASE_URL="https://api.groq.com/openai/v1"
export MODEL_NAME="llama-3.1-8b-instant"

python inference.py

Docker (for hackathon submission)

docker build -t issue-grooming-env .
docker run -e HF_TOKEN=$env:HF_TOKEN `
           -e API_BASE_URL=$env:API_BASE_URL `
           -e MODEL_NAME=$env:MODEL_NAME `
           issue-grooming-env

Use as a Python library

from env import IssueGroomingEnv, Action

env = IssueGroomingEnv(task_id="easy")
obs = env.reset()

action = Action(
    action_type="triage_item",
    payload={"item_id": 2, "decision": "keep"}
)
obs, reward, done, info = env.step(action)
print(reward.message)  # "Triaged #2 as 'keep'. Score: +0.10"

final_score = env.grade()  # float in [0.0, 1.0]

Grading formula

Penalties subtract from the raw score before clamping to [0.0, 1.0]. The hard grader adds an extra penalty if security issues are deprioritized.

What's currently implemented vs what's next

Feature	Status
3 tasks (easy / medium / hard) with synthetic issues	Done
Deterministic graders with partial credit	Done
Loop detection and penalties	Done
Hackathon-compliant stdout format	Done
OpenEnv interface (reset / step / state / grade)	Done
Improve hard task score with larger model	In progress
Real GitHub issue data via API (instead of synthetic)	Planned
Maintainer feedback loop for ground truth correction	Planned
Fine-tuning on successful trajectories	Planned