Rubric changelog

Every report records the rubric version it was graded with. The current rubric is v9.3. Scores graded under different versions may not be directly comparable; owners can request a fresh grade after pushing a new commit. How grading works

v9.3: Archetype profiles, domain exemptions, and hackathon judge evaluation

2026-11-13

• Every project is classified into an archetype profile (Developer Tool / MCP Server, Game / Interactive Simulation, Media / Audio Lounge, Mobile / IoT, Automation Pipeline, Web Productivity, or General Software) with tailored evaluation lenses.
• Domain-specific exemptions are honored: developer tools and MCP servers are not penalized for technical CLI/schema terminology or lack of web screen-reader labels; games are evaluated on arcade game loop engagement and feel rather than business forms.
• The 14B model's persona is tuned to a discerning Hackathon Judge and power-user looking for working solo vibe-coding leverage, real-world utility, and creative superpowers rather than corporate enterprise bureaucracy.
• Semantic README parsing bypasses top-of-page badge walls, donation buttons (like Ko-fi), and hero images to capture the developer's true pitch, while recognizing screenshots and diagrams as positive visual proof of work.
• Projects without clear domain signals or with minimal READMEs gracefully fall back to an 'Unknown / General Software' archetype that evaluates on core functionality without assuming specific frameworks or workflows.

v9.2: Graders first read what the project says it is

2026-11-13

• Before any judging, LetsVibe reads the project's own descriptions: its GitHub description and topics, the README's title, opening and table of contents (never the whole file, so a long README counts no more than a short one), recent release notes and commit messages that explain changes.
• The AI turns these into one plain sentence about what the project is and who it is for. Its quotes are checked against the descriptions, and the kind of project is taken from its files when they make it certain (for example an AI-assistant plugin or a browser extension).
• Every checklist sees this first, and the summary opens with it, so a project is described as what it is rather than as the closest common app type. It is context only: a feature the README promises still has to be shown by the app and its code.
• Projects with no README or description are noted as such, which is a fact about how easy they are to get started with.
• Grading is now repeatable: the AI models use a fixed seed and no randomness when they judge, so grading the same project twice gives the same answers instead of tipping close calls either way.

v9.1: Problems are found as reliably as strengths

2026-11-13

• Every checklist question now has a real 'no' answer. Before, questions such as 'does the app tell people when something goes wrong?' could only be answered 'clearly', 'sometimes' or 'not shown', so the code review could only ever add praise.
• A 'no' is backed by the code that shows the gap, for example an error that is only written to a log. 'Not there at all' counts only when every part of the code that could show it was read and none did; a scan that simply missed something is still never held against a project.
• For things people see (messages, confirmations, loading feedback and screen-reader labels), praise must quote something a person actually sees. Lines written to a log, print or console never count.
• Each answer from the code review is checked once more: the smaller AI model is shown the quoted lines and asked whether they really show the claim. Answers whose quotes are real but beside the point are dropped.
• Questions about the whole app (does it work from start to finish, are the features wired together, is the look consistent, do the features work together) need evidence from at least two files. When parts of the code disagree, the answer is the one halfway between them, so a few polished screens never outvote the rest.
• A code-review answer can no longer replace the dossier review's answer without a dispute check.
• 'Needs work' lists every problem and every answer that falls short (for example 'mostly works' or 'only sometimes confirms an action'), with the file and line where the code shows it. Each factor's description starts with its weakest finding, so the words match the score.

v9: Two independent reviews of every project, checked against each other

2026-11-13

• Every project now gets two independent reviews. The first is the App Dossier from v8: a written account of what people see and do, read by the larger AI model. The second reads the code itself, cut into small sections.
• The code review looks at one thing at a time. A smaller AI model reads each section separately for functionality, UI/UX, the user journey and creativity, and only reads the sections that can tell it something about that topic. Real-world value is still judged from the project as a whole.
• Every answer from the code review must quote real lines of code, and a program checks each quote against the file. Answers whose quotes are not in the code are thrown away.
• The two reviews are compared item by item. When they agree, the answer stands. When only one found evidence, that evidence is used, so things a scan could not see are no longer simply given middle points. When they disagree, the larger model rereads the exact quoted lines from both sides and decides which claim the code supports; if neither holds up, the item counts as not shown. Answers are never averaged.
• Scores still come from fixed points per answer, with the same weights as v8.
• Reports now say where findings were seen (file and line) and include a plain-English description of the code's complexity. Complexity is shown for information only and never changes the score.
• A grade that runs longer than 3 hours stops and publishes nothing.

v8: Graded on what people experience, not on how the code is written

2026-11-12

• Projects are graded on five things, in order of weight: real-world value, creativity, the user journey (from installing it to closing it), functionality and UI/UX. Code health is shared with each project's owner only, as notes for the developer, and never counts toward the score.
• No AI reads the code any more. A program first writes an App Dossier of what people see and do: what the project says it is, its screens, the text on them, its own features, how many devices can use it at once, how much setup it needs before first use, how it launches and saves, and any ads, paywalls or forced sign-in.
• The project is compared with common app types people make from a single prompt (to-do lists, stock viewers, space shooters and so on), and the closest match is given to the reviewer as a fact.
• Problems a program can prove (for example back-end features no screen calls, or a config file that cannot be read) count against functionality, per 1,000 lines of code. Anything a scan could not find is never counted as missing.
• The reviewer answers fixed checklists, one topic at a time, and quotes the dossier for every answer. It never picks a number: each answer is worth fixed points, so every score can be traced to the answers behind it.
• The written review only restates the checked answers, and 'Needs work' lists only real problems (such as a heavy setup, nothing being saved, or a standard copy of a common idea), never invented advice.
• Safety is no longer graded: LetsVibe is not a security tool, and a grade never says software is safe to run.

v7: Graded on a curve, with a leaner review

2026-11-11

• Letter grades now come from a curve (a bell curve). AI reviewers hedge toward the middle, so a raw score says little on its own. Your letter depends on how far your score sits from the average, in standard deviations; two published reference projects set the bell at first, and your category's own scores take over as it grows. Everything behind the curve, every run included, is on the Grading curve page.
• Before review, files are sorted by name and folder (no AI): docs, licenses, build/config files and tests are skipped (they reach the reviewers as measurements), and story, dialogue or level content is sampled. Each chunk lists only its own files, which cut the reading by about a fifth.
• The yes/no checklist went from 33 to 23 questions: duplicates were removed, and code-health questions that measurements already answer (tests, complexity, dependencies) or that AI-written code almost always passes (clear names) are no longer asked.
• Safety is always reviewed for code that makes network calls, runs other programs, uses threads or files, loops forever or silently ignores errors, and setup wizards built inside a page are now recognized.

v6: Vibe-coded software, graded as vibe-coded software

2026-11-11

• Projects are compared with other vibe-coded projects (programs written by AI from someone's prompts), never with software written by professional developers. Score bands were rewritten accordingly: 62 is a typical working vibe-coded project.
• New factor: User Journey, the user's path from install and first-run setup, through finding their way (including how deep settings are buried), to finishing the main task and being told it worked.
• Maintainability and Architecture were merged into Code Health. Engineering factors now carry roughly 10-25% of the weight in most categories and are framed as 'what a pro team would do next'.
• Letter grades reflect standing among similar projects once a category has 30 or more; until then the rubric curve is used (62 = C). Reports show their rank within their category.
• The checklist was rephrased for vibe-coded projects, and complexity now counts mainly when it is widespread rather than for a few large functions.

v5: Owner input before publishing, and corrections

2026-11-11

• When the review needs more context about specific code, the report is not published or scored down. Instead the owner is asked to explain those lines; the report is published after review, and requests left unanswered for 7 days are deleted.
• Owners can report an error in their review. Each report is checked against the code; confirmed errors are corrected and marked as corrected on the report.

v4: Device-aware grading and fairer factor scoring

2026-11-11

• Each project is classified first (category, target platforms and hardware) and graded against the device it is built for, so a microcontroller dashboard is not judged like desktop software.
• Screen size, chip and memory are read from the project's own build files and source code.
• Parts of a codebase that contain no user interface, I/O or user-facing features no longer count for or against those factors. If no part contains them, the whole-repository review decides.
• Factor weights are kept inside the published range for each project category.
• Detected secrets are reported by type only; their location is never published.

v3: Reliability fixes

2026-11-11

• Fixed model prompts that were too long for the local models' context window, which caused some evaluations to return generic placeholder results. Evaluations that fail now produce no ranking at all instead of a misleading one.
• Vendored third-party libraries are excluded so projects are graded on their own code.

v2: Calibration

2026-11-11

• Letter-grade curve aligned with the scoring rubric (a typical working project is a C).
• Yes/no evidence checklist blended with model scores to anchor results in concrete facts.

v1: Beta

2026-11-08

• Initial dual-model (7B + 14B) local grading pipeline.