The DiscoverAINow AI Testing Benchmark.
A transparent way to compare AI models on the work software testers actually do—not on generic chatbot demos.
Status: methodology published, benchmark runs pending.We are intentionally publishing the scoring framework before publishing scores. That prevents us from changing the rules after seeing which model performs best.
Six benchmark tasks
- Requirement interpretation: identify ambiguity, dependencies and missing acceptance criteria.
- Test-case generation: produce positive, negative, boundary and state-transition coverage.
- Edge-case discovery: identify meaningful non-obvious failures rather than producing long generic lists.
- API testing: generate useful test scenarios and validations from an API contract.
- Automation generation: create maintainable Selenium/Playwright-style test code from a defined scenario.
- Defect analysis: reason from symptoms/logs to reproducible hypotheses without inventing evidence.
100-point scoring rubric
| Dimension | Weight | What earns points |
|---|---|---|
| Correctness | 25 | Technically and logically correct output; no invented requirements. |
| Coverage | 20 | Important positive, negative, boundary and dependency cases. |
| Risk prioritization | 15 | Finds high-impact risks instead of treating every case equally. |
| Automation quality | 15 | Readable, maintainable and appropriately synchronized code. |
| Hallucination control | 15 | Flags uncertainty instead of inventing facts or interfaces. |
| Manual cleanup | 10 | How close the output is to something a practitioner can use. |
Representative benchmark requirement
Feature: Card payment limit
A customer can make a card purchase when:
- the card is active;
- the transaction is below the remaining daily limit;
- sufficient available credit exists.
The daily purchase limit resets at midnight in the account timezone.
Declined transactions must not reduce available credit or the daily limit.
The same source requirement is supplied to every model. Scorers then compare whether the output catches boundary behavior, timezone reset, concurrent transactions, exact-limit behavior, decline-state integrity and missing/ambiguous requirements.
Why this is useful
Benchmarking makes comparison pages auditable. Instead of saying one model “feels better,” we can show which cases it found, which it missed and how much human correction was required.
No model vendor paid for this methodology. Future commercial relationships will be disclosed and will not change scoring rules.