Runtime backend testing benchmarks
We continuously benchmark Kerno against production open source codebases and realistic runtime bugs to improve its accuracy month over month.
| Month | Catch rate |
|---|---|
| Jan | 53% |
| Feb | 55% |
| Mar | 57% |
| Apr | 61% |
| May | 62% |
| Jun | 65% |
| Jul | 66% |
| Aug | 67% |
| Sep | 72% |
Overview
The benchmark asks a narrow question. When a realistic bug lands in an endpoint that already has a Kerno baseline, does Kerno report a Diff on that endpoint?
The current set covers 64 planted bugs across 7 open source backends written in Go, Java, Python and TypeScript. Each bug carries a severity from Critical to Low and one of 9 categories, from access control and data exposure to business logic and API contract changes.
Each repository ran on a fresh Kerno workspace, so these results reflect cold-start performance during initial calibration.
- Planted bugs
- 64
- Repositories
- 7
- Languages
- 4
- Categories
- 9
Methodology
Each repository is pinned to one commit and run locally with its real database. Kerno generates baselines for the main endpoints while the code is clean, and those baselines are frozen before any bug is written.
The bugs come from separate author sessions that never see Kerno's work. Authors receive the requests each baseline sends, with no scenario names or assertions, so they write bugs against the application's behavior. Each bug is a small patch that compiles and passes the project's typecheck.
For each bug, the patch is applied, the service restarted and Kerno validation run on the affected endpoints. A bug counts as caught when Kerno reports Diff Detected on that endpoint and a human confirms the difference comes from the injected change.
This version of the benchmark covers bugs that surface in status codes, response bodies, database rows, broker messages and outbound HTTP calls. Latency, logs, memory and concurrency races are outside its scope.
- JavaappsmithPlatform to build admin panels, internal tools, and dashboards.
- TypeScriptcal.diyScheduling infrastructure for absolutely everyone.
- TypeScriptdocmostOpen-source collaborative wiki and documentation software.
- PythonflagsmithOpen-source feature flag platform with remote config.
- GografanaOpen and composable observability and data visualization platform.
- PythonlangflowA tool for building and deploying AI-powered agents and workflows.
- JavathingsboardOpen-source IoT platform for device management and data visualization.
Runtime bug catch rate
Kerno caught 46 of the 64 planted bugs, a 72% catch rate. Data exposure bugs were caught most often. Input validation bugs and money and quantity logic bugs were the hardest to catch.
Select a severity or category to see those bugs in the results below.
Results by repository
The tables list every bug in the benchmark with its severity, category and result. Open a row to read the bug's effect and the exact difference Kerno reported.
Caught means Kerno reported Diff Detected on the affected endpoint and a human confirmed it reflects the planted bug.
appsmith
Platform to build admin panels, internal tools, and dashboards.
Any user can read any workspaceOWASP A01GET /api/v1/workspaces/{id}CriticalAccess controlCaught
Any signed-in user can read the details of any other user's workspace by guessing or replaying its id.
Kerno reported Diff Detectedcross_tenant_isolation failed on the assert step: 'body must be empty (empty string or empty object): expected false to be true' and 'no data field should be present: expected {id, userPermissions, name, email, plugins, slug, organizationId} to be undefined'. The secondary user's GET returned 200 with the primary user's full workspace (name isolation-test-1789456049482-45oa3, owner email kerno-test@example.com). The other 6 scenarios on the endpoint passed, so the diff is confined to the injected observable.Signup passwords stored in cleartextOWASP A02POST /api/v1/usersCriticalInsecure storageCaught
Every account created through the signup form has its password written to the database in cleartext, so anyone with read access to the user collection or a database backup owns every account.
Kerno reported Diff DetectedTwo scenarios failed on the database channel while every HTTP response stayed identical (302, same Location, SESSION cookie issued). happy-path: 'password starts with bcrypt prefix $2a$ or $2b$: expected "cobosipumelu" to match /^\$2[ab]\$/'. name_omitted: 'password should be bcrypt hash: expected false to be true', with the persisted row showing password "Test3grtDoMy!9" in plaintext. The other 10 scenarios passed.
Profile endpoint returns password hashOWASP A07GET /api/v1/users/meHighData exposureCaught
The profile endpoint hands every caller the stored password digest of the signed-in account, so a cross-site script, a browser extension or a proxy log turns into offline password cracking and account takeover.
Kerno reported Diff Detectedno_sensitive_field_leakage failed: 'data.password is undefined: expected "$2b$10$iGPd7B3v..." to be undefined' and 'no sensitive fields (password/token/secret/key): expected [] to have a length of 0 but got ["password"]'. happy-path's closed-world envelope comparison showed the same extra key. A third scenario, super_user_flag, also failed with 'super user has isSuperUser set to true: expected false to be true'; that concerns the super-user flag rather than the injected credential leak and is recorded as an unrelated diff. Four scenarios passed.
Deleted workspace leaves roles behindDELETE /api/v1/workspaces/{id}HighState and persistenceMissed
Deleting a workspace leaves its three role documents behind forever, so the roles collection grows without bound and members keep holding role assignments that point at a workspace that no longer exists.
Kerno reported No Diff on this endpointMember lookup treated as a regexOWASP A03PUT /api/v1/workspaces/{workspaceId}/permissionGroupMediumInput validationMissed
The member field of the role-change call is interpreted as a MongoDB regular expression, so a caller can address accounts by pattern instead of by address and can pin the query against the whole user collection.
Kerno reported No Diff on this endpointNew apps default to widget collapse offPOST /api/v1/applicationsMediumControl flow logicMissed
Every newly created application starts with widget collapsing switched off, so hidden widgets keep occupying layout space and new apps render with gaps the author never chose.
Kerno reported No Diff on this endpointSignup name replaced by emailPOST /api/v1/usersMediumState and persistenceCaught
The display name typed at signup is thrown away and replaced by the email address, so the new account shows its email everywhere a name is expected and the auto-created workspace is named after the email too.
Kerno reported Diff Detectedhappy-path failed on the database channel while the HTTP response was unchanged (302 to /signup-success with a SESSION cookie): 'persisted name equals sent name: expected "Rosamond.Breitenberg@hotmail.com" to be "Major"'. The form sent name=Major; the stored user row carries the email address in its name field. The other 11 scenarios passed, including name_omitted, which sends no name and therefore has nothing to mismatch.
Workspace creation stops returning 201POST /api/v1/workspacesLowAPI contractCaught
Clients and integrations that branch on a 201 from workspace creation no longer see one, so a created workspace is not recognised as newly created.
Kerno reported Diff DetectedFive of nine scenarios failed on 'POST /api/v1/workspaces returns 201: expected 200 to be 201'. Three assert it directly (name_special_chars, excessive_data_exposure, happy-path, the last also showing the closed-world envelope diff "status": 201 -> 200); two more (mass_assignment, broken_object_level_authz) fail in a setup step that asserts 201 while building fixtures, same cause counted once. Four passed: name_missing, name_blank, invalid_token, unauthenticated. happy-path additionally logged 'Unexpected end of JSON input', which is incidental parse-ordering noise rather than a second finding, since the body was valid JSON.
Error responses leak stack tracesOWASP A05DELETE /api/v1/workspaces/{id}LowError handlingCaught
Every error response now ships the server's internal stack trace, exposing class names, source files, line numbers and the internal framework layout to anyone who can trigger an error.
Kerno reported Diff Detected0 of 3 endpoints passed; 7 scenarios failed, every one on the error envelope gaining the field: 'expected {responseMeta: {status, success, error: {code, title, message, errorType, referenceDoc}}, errorDisplay} to deeply equal {...}'. DELETE /api/v1/workspaces/{id} 2 of 9 failing (not_found_nonexistent_id, nosql_injection_attempt); GET /api/v1/applications/home 4 of 8 (sql_injection_attempt_in_workspace_id, no_workspace_id_provided, nonexistent_workspace_id, cross_tenant_isolation); PUT .../permissionGroup 1 of 10 (no_workspace_read_permission). Caught on all three affected endpoints. Raw 611 KB job output archived as kerno-reports/10-error-reference-fallback.kerno_job.json.cal.diy
Scheduling infrastructure for absolutely everyone.
Password reset skips CSRF checkOWASP A01OWASP A07POST /api/auth/reset-passwordCriticalAccess controlCaught
Anyone who knows a reset-request id can complete a password reset without ever holding the CSRF token, just by omitting the cookie.
Kerno reported Diff DetectedScenario `missing_csrf_cookie` failed on the affected endpoint, and the failure is the injected behaviour rather than a side effect. Two assertions fired: `POST /api/auth/reset-password responds 403 when CSRF cookie is missing: expected 201 to be 403`, and `response body is exactly {"error": "Invalid CSRF token"}: expected {message} to deeply equal {error}`, received `{"message": "Password reset."}`. The other 10 scenarios on the endpoint passed, including `csrf_token_mismatch`, which still returns 403 because a mismatched token is a different path from an absent cookie. That contrast is what makes the diff specific to the injected change.Signup passwords weakly hashedOWASP A02POST /api/auth/signupCriticalInsecure storageCaught
Signup passwords are stored as a fast, unsalted 128-bit hash, so anyone who reads the UserPassword table can recover them, and users created through signup can no longer sign in.
Kerno reported Diff DetectedScenario `happy-path` failed on the one assertion that reads the database: `Password hash matches bcrypt format: expected "4a9db7e6cf5311d389de89bc93f43f71" to match /^\$2[ab]\$/`. The other 12 scenarios passed. This is the clearest white-box catch in the run: the HTTP response was byte-identical at 201 {"message":"Created user"}, so nothing over the wire changed and the bug was visible only in the stored column. A black-box suite would have missed it entirely.Cancellation sends wrong iCal sequencePOST /api/cancelHighMoney and quantity logicMissed
The cancellation pushed to the organizer's external calendar carries a sequence number of 0 and the booking is stored with sequence 100, so later updates to the same iCal UID are silently ordered wrong and can be ignored by calendar clients.
Kerno reported No Diff on this endpointBooking decline reason droppedPOST /api/verify-booking-tokenHighState and persistenceCaught
When a host declines a booking through the email magic link and types a reason, the reason is dropped, so the attendee's decline notification and the booking page show no explanation.
Kerno reported Diff Detected10 of 11 scenarios passed; `reject_with_reason` failed. The rejection still succeeded and the redirect was unchanged, so the only signal was the stored column. The sibling `reject_without_reason` scenario continued to pass, since NULL is the correct value when no reason is supplied, which makes the diff specific to the injected change rather than a blanket failure.
SQL injection in calendar conflict lookupOWASP A03POST /api/availability/calendarMediumInput validationMissed
A caller controls the WHERE clause of the conflict lookup through integration and externalId, so a crafted value can read or match rows belonging to other users, and a stray apostrophe in a calendar id breaks calendar selection outright.
Kerno reported No Diff on this endpointHost cancels without a required reasonPOST /api/cancelMediumControl flow logicMissed
A host can cancel a booking without giving a reason even though the event type requires one, so attendees receive a cancellation notice with no explanation.
Kerno reported No Diff on this endpointPassword reset links expire on issuePOST /api/auth/forgot-passwordMediumState and persistenceCaught
Every password-reset link is dead the moment it is issued, so no user can complete a password reset.
Kerno reported Diff Detected3 of 9 scenarios failed on the affected endpoint. The reordering leaves the HTTP response untouched at 201 {"message":"password_reset_email_sent"} and is visible only in the row's `expires` value, which is why it was tagged subtle.Version field renamed in responseGET /api/versionLowAPI contractCaught
Any client reading the version field from this endpoint now reads undefined.
Kerno reported Diff Detected4 of 5 scenarios failed, the broadest hit in the run. A renamed response key is the easiest class of bug for a full-shape body assertion to catch, and every scenario that reads the body tripped on it.
Username check fails openPOST /api/usernameLowError handlingCaught
A username check that fails or is sent with a bad body reports the username as available, so the signup form shows a green tick for a name that may be taken or invalid.
Kerno reported Diff Detected3 of 10 scenarios failed. The bug swallows the error path so malformed input is answered as a successful availability check rather than a 400. The scenarios that cover wrong-typed and malformed bodies tripped, while the well-formed ones continued to pass, which keeps the diff specific to the injected change.
docmost
Open-source collaborative wiki and documentation software.
Login accepts any passwordOWASP A07POST /api/auth/loginCriticalAccess controlCaught
Any password is accepted for any known email address, letting an attacker log in as any user without knowing their password.
Kerno reported Diff Detectedpassword_incorrect scenario failed 3 assertions: status 401 expected but got 200; response message assertion failed (200 body has no error message); last_login_at expected to remain null on a failed login but was updated to a real timestamp.
Passwords hashed with unsalted SHA-256OWASP A02POST /api/auth/change-passwordCriticalInsecure storageCaught
New passwords set through change-password (and every other flow that reuses hashPassword) are stored as fast, unsalted SHA-256 digests instead of salted bcrypt hashes, making a database leak trivially crackable.
Kerno reported Diff Detectedhappy-path and password_hash_format_verified scenarios both failed exactly on the hash-format assertion: expected a bcrypt-shaped string (/^\$2[aby]\$\d{2}\$.../) and got a 64-char hex SHA-256 digest instead; the algorithm-identifier and cost-factor sub-assertions failed accordingly. same_password_reuse also failed on the same hash-format check.Deleted pages listed as recentPOST /api/pages/recentHighData exposureCaught
A user requesting recent pages for a space sees pages that were deleted (moved to trash) still listed as if they were active, alongside their real title.
Kerno reported Diff Detecteddeleted_pages_excluded failed exactly as the bug predicts: a soft-deleted page's id was found in the recent-pages result set where the assertion expects it absent (expected false, got true).
Page content stored unparsedPOST /api/pages/createHighState and persistenceCaught
Pages created with markdown or HTML content silently store unparsed source text as their content column instead of a valid ProseMirror document; the editor would fail to render or would render garbage the next time the page is opened, with no error at creation time.
Kerno reported Diff Detectedformat_html and format_markdown both failed on the same shape: pages.content persisted as the raw HTML/Markdown string instead of parsed ProseMirror JSON (expected an object with type:'doc' and a content array, got the raw string).
Update endpoint can soft-delete pagesPOST /api/pages/updateMediumInput validationCaught
Any user who can edit a page can soft-delete it (or set an arbitrary deletedAt timestamp on it) through the ordinary update endpoint, bypassing the dedicated delete flow and its own checks.
Kerno reported Diff Detectedunintended_field_mass_assignment failed on exactly the injected bug: both the response body's deletedAt and the DB's deleted_at column took the attacker-supplied date instead of staying null.
Recent pages sorted incorrectlyPOST /api/pages/recentMediumControl flow logicCaught
The 'recent pages' list a user sees no longer reflects which pages were most recently edited; an older page that was just updated no longer rises to the top, while a newly created but untouched page can outrank it.
Kerno reported Diff Detectedcursor_pagination failed its 'All items across both responses are ordered by updatedAt descending' assertion, which is exactly the mechanism the bug changes (id becomes the primary sort key ahead of updatedAt). All other 12 scenarios passed.
Search text cleared on title editPOST /api/pages/updateMediumState and persistenceMissed
A page's full-text search entry silently goes stale (or disappears) the moment its title is edited, even though its content and the update response both look completely normal; the page becomes unfindable by search until something else happens to repopulate text_content.
Kerno reported No Diff on this endpointDefault page size halvedPOST /api/pages/recentLowAPI contractMissed
A client that relies on the documented default page size of 20 gets half as many recent pages back per request and must page through more requests to see the same pages.
Kerno reported No Diff on this endpointMissing page returns 500POST /api/pages/infoLowError handlingCaught
Looking up a page that does not exist (or was mistyped) returns a generic server error instead of a clean not-found response.
Kerno reported Diff Detectedpage_not_found failed exactly on the injected bug: expected 404 with the standard not-found envelope, got 500 {statusCode:500, message:'Internal server error'}.flagsmith
Open-source feature flag platform with remote config.
Project permissions readable by any userOWASP A01GET /api/v1/projects/{project_pk}/all-user-permissions/{user_pk}/CriticalAccess controlCaught
Any signed-in user, including one with no membership of the organisation, can read another user's permission set for any project.
Kerno reported Diff Detectedcross_organisation_isolation on GET /api/v1/projects/{project_pk}/all-user-permissions/{user_pk}/: 'endpoint returns 500 Internal Server Error for cross-organisation access' expected 500, received 200. Request GET /api/v1/projects/62/all-user-permissions/51/ with organisation A's admin token now returns 200 and a real permission payload for an organisation B project. 1 of 11 scenarios diffed; the other 10 passed. The baseline had pinned Flagsmith's pre-existing 500, so the catch is on the guard bypass rather than on a 403.OAuth client secrets stored in cleartextOWASP A02POST /o/register/CriticalInsecure storageCaught
Every OAuth client secret issued by dynamic client registration is stored in the database in the clear, so anyone with database or backup access can impersonate every registered client.
Kerno reported Diff DetectedTwo of 25 scenarios diffed, both on the database channel. token_endpoint_auth_method_client_secret_basic: 'Persisted client_secret is hashed, not plaintext' expected false, received true, because the stored secret now equals the plaintext returned. register_success: expected client_secret matching /^pbkdf2_sha256$...$/, received "". Status, headers and body were identical throughout, so the subtle tag holds and the catch came purely from the white-box hash-format assertion.
Identity lookup crosses environmentsOWASP A07GET /api/v1/identities/HighData exposureCaught
An SDK key for one environment returns the traits stored against the same identifier in a different environment, so one customer's identity data is served to another environment's SDK.
Kerno reported Diff Detectedcross_environment_isolation on GET /api/v1/identities/ failed three assertions: 'Two identity rows exist for identifier cross-env-...' expected 2, received 1; 'Identity exists in environment A (id 1)' expected to be defined; 'Environment A identity has correct identifier' received null. The lookup now matches on identifier alone, so environment B's identity is returned instead of a new one being created in A. 1 of 11 GET scenarios diffed; POST passed 11/11. NOTE: the first attempt aborted on GET with 'the system under test is unreachable (fetch failed)' under host memory pressure and ran zero scenarios; this is the one permitted rerun.
Numeric trait updates not savedPOST /api/v1/identities/HighMoney and quantity logicMissed
Numeric traits stop being updated: the API confirms the new value and serves it for the rest of that request, but every later flag evaluation and identity read uses the stale number, so segment rules on counts, versions or quantities match on out-of-date data.
Kerno reported No Diff on this endpointConfidential OAuth client stored as publicPOST /o/register/HighState and persistenceCaught
A client that registered with client_secret_basic is recorded as a public client, so the token endpoint stops requiring its secret and anyone holding only the client id can exchange codes as that client.
Kerno reported Diff Detected1 of 25 scenarios diffed. token_endpoint_auth_method_client_secret_basic: 'Persisted application client_type is confidential' expected confidential, received public. The HTTP response was identical to the baseline in every respect, including the returned client_secret and client_secret_expires_at, so detection came solely from the white-box assertion on oauth2_provider_application.client_type.
javascript: redirect URIs acceptedOWASP A10POST /o/register/MediumInput validationCaught
A client can register a javascript: or data: redirect URI, so the authorisation flow can be made to redirect a signed-in user into attacker-controlled script with the authorisation code.
Kerno reported Diff Detected1 of 25 scenarios diffed. redirect_uris_forbidden_scheme_rejected failed four assertions: expected 400 received 201; the closed-world body check received the full registration payload with redirect_uris ['javascript:alert(1)'] where {error, error_description} was expected; error field expected invalid_redirect_uri received null. Matching forbidden schemes as 'scheme://' prefixes means javascript:, which carries no '//', never matches. Notable because this is the expected_hard A10 slot and the baseline happened to send the adversarial input, so it was caught rather than missed.Enabled-features metric counts every flagGET /api/v1/environments/{environment_api_key}/metrics/MediumControl flow logicMissed
The environment dashboard reports every flag as enabled, so the enabled-features count always equals the total and no longer says anything about what the environment actually serves.
Kerno reported No Diff on this endpointFirst-evaluation record overwrittenPUT /api/v1/environments/{environment_api_key}/onboarding-status/MediumState and persistenceCaught
The onboarding record stops meaning 'first evaluated'; the stored timestamp and SDK name move to whichever SDK reported most recently, so onboarding funnels and first-evaluation reporting are wrong.
Kerno reported Diff Detected1 of 9 scenarios diffed, both failures on the database channel. idempotent_subsequent_call: first_evaluated_sdk_label expected flagsmith-python-sdk, received flagsmith-java-sdk; first_evaluated_at expected 2026-09-14T16:13:52.230Z, received ...256Z. The scenario's second PUT deliberately sends a different SDK label, which is exactly what the label-keyed guard lets through. Both calls returned 204 with an empty body, identical to the baseline, so detection came only from the two white-box assertions.
Trait writes return 201POST /api/v1/traits/LowAPI contractCaught
SDKs and integrations that treat only 200 as success on trait writes see an unexpected status code.
Kerno reported Diff Detected7 of 15 scenarios diffed, all on 'POST /api/v1/traits/ responds 200: expected 201 to be 200': happy-path, integer_trait_value, boolean_trait_value, float_trait_value, sql_injection_attempt_in_identifier, server_key_with_allow_client_traits_false, and update_existing_trait (which failed in its seed_initial_trait arrange step and so never reached its own assertions). Removing the status override lets DRF return its natural 201 on every success path. The eight 400/403 rejection scenarios passed unchanged. Two earlier attempts were aborted by the SIGTRAP worker crash and are not the scored run.
SDK errors leak database detailsOWASP A05GET /api/v1/identities/LowError handlingMissed
Unhandled failures on the SDK endpoints hand the raw database error text, including column types and limits, back to any anonymous caller.
Kerno reported No Diff on this endpointgrafana
Open and composable observability and data visualization platform.
Viewers can create snapshotsOWASP A01POST /api/snapshots/CriticalAccess controlMissed
Any signed-in user, including a Viewer who holds no snapshots:create permission, can create dashboard snapshots.
Kerno reported No Diff on this endpointSnapshot contents stored in cleartextOWASP A02POST /api/snapshots/CriticalInsecure storageMissed
Snapshot dashboard contents, which Grafana deliberately keeps encrypted at rest, are also written to the database in cleartext, so anyone with read access to the database reads them without the secret key.
Kerno reported No Diff on this endpointSnapshot expiry set to fifty yearsPOST /api/snapshots/HighMoney and quantity logicCaught
A snapshot shared with a one-hour (or one-day) expiry never expires; it stays publicly readable for fifty years.
Kerno reported Diff Detectedexpires_persisted_correctly FAILED: 'expires is approximately 3600 seconds after created (within 5s tolerance): expected 1576800000000 to be less than 5000'. The 1576800000000 ms is the 50-year default leaking into the delta - exactly the injected arithmetic.
Public dashboard links broken on createPOST /api/dashboards/uid/{dashboardUid}/public-dashboardsHighState and persistenceCaught
Every public dashboard created after this change gets a share link that is dead on arrival: the creation succeeds, the link is handed to the user, and every visit to it is rejected before any lookup happens.
Kerno reported Diff Detectedcreate_public_dashboard_success FAILED: 'accessToken is a UUID hex string without dashes: expected "J4npaDc2RY2d31pyn52DqQ" to match /^[a-f0-9]{32}$/'. access_token_collision_rejected also failed - the secondary effect the author disclosed in advance, where a replayed token now trips the format gate before the collision check.SQL injection in org user searchOWASP A03GET /api/users/search (actual request path: /api/org/users/search)MediumInput validationMissed
An organisation user search whose text contains a quote character fails the whole request, and a crafted search term lists members the query was never meant to match.
Kerno reported No Diff on this endpointOrg user search skips a pageGET /api/users/search (actual request path: /api/org/users/search)MediumControl flow logicCaught
The organisation user list comes back empty, or skipped forward by one page, even though the reported total is right.
Kerno reported Diff Detected5 of 11 failed. happy_path shows the contradiction directly: GET /api/org/users/search -> 200 {"totalCount":2,"orgUsers":[],"page":1,"perPage":1000}. Also failed: sort_login_asc_orders_results, page_parameter_controls_offset, no_users_returns_empty_list, query_parameter_filters_results.Duplicate public dashboards allowedPOST /api/dashboards/uid/{dashboardUid}/public-dashboardsMediumState and persistenceCaught
A dashboard can end up with more than one public dashboard, each with its own share link, and the API reports success instead of rejecting the duplicate.
Kerno reported Diff Detectedpublic_dashboard_already_exists_rejected FAILED: 'POST .../public-dashboards (second attempt) -> 200 {"uid":"cfyvcej2qr11cb",...}' where the scenario expected the 400 conflict.Org creation returns 201POST /api/orgsLowAPI contractCaught
Clients that check for an exact 200 on organisation creation now see a 201, although the organisation is created and the body is the same.
Kerno reported Diff Detectedhappy-path FAILED: 'POST /api/orgs responds 200: expected 201 to be 200'. duplicate_name_rejected also failed, because its arrange step creates the first org and that create now returns 201.
Failed login returns 400OWASP A07POST /loginLowError handlingCaught
Failed sign-ins are reported as a malformed request rather than as an authentication failure, so clients and monitoring can no longer tell a rejected credential from a bad payload.
Kerno reported Diff Detected5 of 8 failed, on both axes. Status: 'POST /login with incorrect password responds 401: expected 400 to be 401'. Shape: 'response body contains exactly statusCode, messageId, and message fields: expected {message} to deeply equal {statusCode, messageId, message}'.langflow
A tool for building and deploying AI-powered agents and workflows.
Users can promote themselves to superuserOWASP A01PATCH /api/v1/users/{user_id}CriticalAccess controlCaught
Any ordinary logged-in user can promote their own account to superuser and then read, edit and delete every other user's data.
Kerno reported Diff DetectedTwo of 17 scenarios on the affected endpoint diffed, and the first reflects the injected change directly.
regular_user_cannot_grant_superuser (response 200):
assert: 'PATCH /api/v1/users/{user_id} with is_superuser=true responds 403' -- expected 403, received 200
assert: 'is_superuser remains false in database' -- expected false, received true
regular_user_cannot_change_password (response 200): a knock-on of the same defect rather than a separate signal. Once the first scenario's self-promotion succeeded, the same account was a superuser, so the password change it attempts was then permitted and the stored bcrypt hash changed where the baseline expects it unchanged.
The remaining 15 scenarios on the endpoint stayed green, including the cross-user IDOR scenario, which confirms the patch's second guard is behaviourally unchanged as intended.API keys stored in cleartextOWASP A02POST /api/v1/api_keyCriticalInsecure storageCaught
Every API key is now readable in cleartext by anyone who can see the database or a database backup, so a stolen dump hands over working credentials for every account.
Kerno reported Diff DetectedFour of 16 scenarios on the affected endpoint diffed, every one of them on the injected change, and all of them through the database rather than the response. plaintext_not_persisted (response 200), the most direct hit: assert: 'stored api_key starts with gAAAAA (Fernet-encrypted format)' -- expected /^gAAAAA/, received 'sk-_1JD_K_XUH6EBOXv9TKOEGKtYIcdDgR4rxtrMjZFSDs' assert: 'stored api_key does NOT start with sk- (plaintext prefix)' -- expected not to match /^sk-/, received a sk- value assert: 'stored api_key does NOT equal plaintext sk- key (it is encrypted)' -- the two are now equal happy-path and response_excludes_hash both fail the same 'stored api_key does NOT equal plaintext' assertion. unique_keys_per_call fails it twice, once per minted key.
Project totals ignore filtersGET /api/v1/projects/{project_id}HighMoney and quantity logicCaught
The project view reports a flow count and a page count for the whole project instead of for the filtered result, so a filter that matches nothing still claims there are flows to show and the pager offers pages that cannot be reached.
Kerno reported Diff DetectedFour of 12 scenarios on the affected endpoint diffed, all on the injected count. is_component_filter (response 200): assert 'flows.total is 1' -- expected 1, received 2 search_filter (response 200): assert 'flows.total is 1' -- expected 1, received 2 sql_injection_attempt_in_search (response 200): assert 'flows.total is 0 (no matches)' -- expected 0, received 1 is_flow_filter (response 200): failed on the same page envelope. The third is the sharpest illustration: a search that legitimately matches nothing now reports a non-zero total, so the envelope claims results that the filter excluded. Status, headers and body shape were unchanged in both. CATEGORY NOTE, recorded rather than hidden: E1 as defined in CATEGORIES.md is money and inventory arithmetic, and langflow's covered surface has none. The slot was filled with the only computed quantity available on it, the pagination envelope's total and pages. The author raised this itself rather than passing it off. The severity stays High per the fixed slot template. Treat this repository's E1 result as 'quantity arithmetic', not 'money'.
Flow edits keep old timestampPATCH /api/v1/flows/{flow_id}HighState and persistenceMissed
Editing a flow no longer records when it was edited, so the project view keeps listing it in its old position and every 'last modified' the product shows is the time the flow was created.
Kerno reported No Diff on this endpointEndpoint name validation bypassedOWASP A03POST /api/v1/flowsMediumInput validationCaught
A flow can be given an endpoint name containing spaces, '!' or any other character, and that raw string is then used as the path segment of the flow's public run URL, so the generated URL is malformed or carries attacker-chosen characters into the route.
Kerno reported Diff DetectedOne of 16 scenarios on the affected endpoint diffed, on the injected change.
endpoint_name_invalid_chars_rejected (response 201):
assert: 'POST /api/v1/flows with invalid endpoint_name responds 422' -- expected 422, received 201
assert: 'Response body has a detail field' -- expected to be defined, received undefined
assert: a .toMatch() on the error detail received undefined rather than a string
Wider reach, recorded rather than scored: the relaxed rule lives in _validate_endpoint_name_value, which FlowUpdate also uses, so while this patch is live PATCH /api/v1/flows/{flow_id} would accept an invalid endpoint name too. That endpoint belongs to another slot and was not validated here. Bugs are applied and reverted one at a time, so the two are never live together, and classification reads only the affected endpoint.Header-only flow list returns full graphsGET /api/v1/flowsMediumControl flow logicCaught
The header-only flow listing, which the canvas loads on every page open to get a light index of flows, now ships the complete graph of every flow, so the list payload grows with the size of the user's flows instead of staying constant.
Kerno reported Diff DetectedFive of 13 scenarios on the affected endpoint diffed. The first is the author's stated observable, exactly:
header_flows_true (response 200):
assert: 'data field is null for non-component flow' -- expected null, received the full graph object {"edges":[{"id":"edge1","source":"node1","target":"node2"}], "nodes":[...]}
assert: 'Seeded flow matches FlowHeader contract with data=null for non-component' -- failed on the same field
The other four (happy-path, excessive_data_exposure_check, flow_type_workflow, remove_example_flows_true) all returned 500 where the baseline expects 200, with 'getResponse.body.find is not a function' and, on happy-path, Content-Encoding lost as a consequence of the error response.
SUBTLETY TAG CORRECTED at step 8.5, from the author's 'moderate' to 'obvious'. The author aimed at a changed field value; the patch also makes the endpoint fail outright once flows carry a real graph, which is a larger and more visible change than intended.
The 500s are data-dependent, not universal. A post-hoc probe against an emptied flow set returned 200 with [], so the collateral failure needs flows holding non-trivial graph data to appear, which the scenarios create for themselves.MCP server not saved with flowPATCH /api/v1/flows/{flow_id}MediumState and persistenceCaught
Saving a flow that carries an MCP server no longer registers that server, so it never appears in the user's MCP server list and its command, arguments and credential have no stored home to be maintained or rotated from.
Kerno reported Diff DetectedOne of 14 scenarios on the affected endpoint diffed, on the injected change, entirely through the database. mcp_secret_stripped_from_response (response 200): assert: 'Exactly one mcp_server row persisted' -- expected 1, received 0 assert: 'mcp_server row user_id matches authenticated user' -- expected the caller's id, received nothing assert: 'mcp_server row name matches sent server name' -- expected the sent name, received nothing The MCP_ variable row was still written under the patch, which is what keeps the response identical and the defect silent. Subtlety tag CONFIRMED rather than corrected: status, headers and body are unchanged. This slot is the recorded substitution for H. langflow has no locally reachable side-effect endpoint, so the side-effect slot became a second F at Medium, subtle.
User creation returns 200POST /api/v1/usersLowAPI contractCaught
Any client or integration that keys off a 201 to confirm a user was created now sees a 200 and can treat the creation as a no-op or an error.
Kerno reported Diff DetectedNine of 12 scenarios on the affected endpoint diffed, every one on the injected change. Eight of them fail the identical assertion: 'POST /api/v1/users responds 201' -- expected 201, received 200 (happy-path, public_signup_success, admin_create_user_success, password_not_leaked_in_response, mass_assignment_privilege_escalation_blocked, special_characters_in_username, optins_defaulted_when_omitted, empty_password_rejected). duplicate_username_rejected failed earlier, in its arrange step: its seed_existing_user step expects 201 from the same create route, so it never reached the assertion and its cleanUp then deleted 0 rows. Known wider reach, recorded not scored: several other endpoints build fixtures through this create route, so while this patch is live their arrange steps would fail too. Only the affected endpoint was validated and only it counts.
Failed login returns 400OWASP A07POST /api/v1/loginLowError handlingCaught
A wrong password now comes back as a request-format error rather than an authentication failure, so clients and log-based lockout tooling cannot tell a rejected credential from a malformed request.
Kerno reported Diff DetectedThree of 7 scenarios on the affected endpoint diffed, and they caught BOTH halves of the change, the status and the missing challenge header. invalid_password (response 400): assert: 'login with wrong password responds 401' -- expected 401, received 400 assert: 'WWW-Authenticate header value is Bearer' -- expected 'Bearer', received null invalid_username (response 400): same two failures, status and the absent WWW-Authenticate header. sql_injection_username (response 400): 'SQL injection attempt returns 401 Unauthorized' -- expected 401, received 400.
thingsboard
Open-source IoT platform for device management and data visualization.
Customers can list other customers' devicesOWASP A01GET /api/customer/{customerId}/devicesCriticalAccess controlCaught
Any customer user of a tenant can list the devices of every other customer of that tenant just by putting the other customer's id in the URL.
Kerno reported Diff Detectedcross_customer_denied diffed. expected 403, received 200. 20 of 21 scenarios passed.
bcrypt cost factor loweredOWASP A02POST /api/auth/changePasswordCriticalInsecure storageMissed
Every password written after this change is protected by roughly one sixteenth of the intended bcrypt work, so a stolen credentials table is far cheaper to crack.
Kerno reported No Diff on this endpointCustomer count off by oneGET /api/usageHighMoney and quantity logicMissed
The tenant usage page under-reports how many customers the tenant has, so a tenant sitting on its customer quota is shown as having one slot free.
Kerno reported No Diff on this endpointDevice access token replacedPOST /api/deviceHighState and persistenceCaught
A device provisioned with a pre-agreed access token is stored with a different, server-generated token, so the device cannot connect with the credential it was shipped with.
Kerno reported Diff Detectedcustom_access_token diffed. 16 of 17 scenarios passed. Caught by a white-box scenario that reads device_credentials back and asserts credentials_id equals the supplied accessToken. HTTP response is identical, so this is only catchable by DB read-back, which the baseline does.
SQL injection in device name checkOWASP A03POST /api/deviceMediumInput validationMissed
A device name is pasted straight into the SQL that checks for name conflicts, so a crafted name can break the create call or make the conflict check read rows it should not see.
Kerno reported No Diff on this endpointEmpty type filter returns 400GET /api/tenant/devicesMediumControl flow logicCaught
Clearing the device-type filter in the UI makes the device list fail with a bad-request error instead of showing all devices.
Kerno reported Diff Detectedtype_empty_string diffed. 21 of 22 scenarios passed. type_empty_string asserts an empty type returns the same result as no type param; the control-flow bug making empty type 400 breaks that comparison.
Deleted device keeps its credentialsDELETE /api/device/{deviceId}MediumState and persistenceCaught
Deleting a device leaves its access token behind in the platform, so a credential that should have been revoked stays in the credentials table forever.
Kerno reported Diff Detectedhappy-path diffed. 8 of 9 scenarios passed. Caught by DB read-back: happy-path asserts no device_credentials row remains after delete. HTTP response byte-identical, catchable only white-box.
Device delete returns 204DELETE /api/device/{deviceId}LowAPI contractCaught
API clients that treat only 200 as a successful delete now see an unexpected status code for a delete that actually succeeded.
Kerno reported Diff Detectedhappy-path diffed. 6 of 9 scenarios passed.
Unknown customer returns 500POST /api/customer/{customerId}/device/{deviceId}LowError handlingCaught
Assigning a device to a customer id that does not exist returns a server error instead of telling the caller the customer was not found.
Kerno reported Diff Detectedcustomer_id_not_found diffed. expected 404, received 500. 11 of 12 scenarios passed.