Building Our Own Apps Made CVEs Our Problem, So I Tested Which AI Would Catch One
Claude found the problem blind in 1 minute 23 seconds, and two of the four models never turned in a report.
October 30, 2026 • 5 min read • Lab Builds
I actually found out about the Oct 21 Node.js security release from an email from Hostinger, my VPS host. The release landed with ten CVEs, four of them High, and fixes for versions 22.23.4, 24.22.1 and 26.12.1. Node 20 got nothing because it’s end of life and always affected when a security release occurs. One of the High CVEs, CVE-2026-111497, is a Brotli decoder flaw that writes past an 8-byte internal buffer on malformed input. Imperva calls it Node2Shell and demonstrated code execution on the official Node.js 24.22.0 image. There’s a public proof of concept, and as of October 22 nobody had reported exploitation in the wild.
That’s when it dawned on me that this one might hit us harder than usual. We have eight vibe-coded apps at work. Two are client apps in progress, three are replacing existing tools, and the rest are small utilities. They’re mostly internal, brought in house to fit our exact needs instead of buying off the shelf tools and struggling to make them work the way we want. People with little to no development experience built most of them. Our actual developer works on the client stuff and helps the rest of us along the way.
Our stuff got built to get the job done, so the stack underneath came along with it. Once we started building our own apps, checking our own code and the software under it became a new thing to watch when a CVE drops.
So I asked Claude to scan our company GitHub, linking the Node announcement. It identified every impacted repo and missed nothing. We went through them with Claude’s help, upgrading to one of the safe versions. Some needed to go from Node 20 to 24. The fixes were mostly CI, not application code, and our container based design meant it didn’t take long to get through all of the repos. It took about half a day in total across three people. I’d get Claude started, then go off and work on something else until it needed me. It was a nice and easy background task.
We don’t blindly follow what the AI builds, and we care about the decisions it makes. We build the house (containers), AI lays out the rooms and picks the furniture.
That got me wondering whether an AI would have caught it if nobody told it what to look for. So I built a test.
I got a fresh Ubuntu 26.10 VPS. Erwin hardened the SSH and set up a user with no sudo and a locked password. The three Node versions went on the server first. Then a Cursor cloud agent built three ordinary apps without knowing anything about the plan: team-notes on Node 20.18.1, invoice-tracker on 22.12.0, and status-board on 24.22.1. Erwin deployed each one as a systemd service on localhost ports 3001 through 3003. The answer key was written afterward from what was actually built, so the scoring was honest.
I gave Claude, Gemini, Grok and Qwen the same blind prompt, with no mention of Node versions or the release: “SSH to the following server: auditor@
The blind run time mattered most to me. A defender doesn’t know what’s on the server, so that initial “figure it out” time is the race against the bad guys. I took the token counts out of the scorecard.
Claude Opus 5.5 finished the blind run in 1 minute 23 seconds and scored 6 out of 6. It flagged Node 20 as end of life and 22.12.0 as behind, left 24.22.1 alone, and made no false claims.
Grok 4.7 listed the versions but never flagged them as problems. It scored 2 out of 6 blind. Once I named the Oct 21 release in the second prompt, it jumped to 6 out of 6.
Gemini 4 Argon ran commands for about fifteen minutes, then said “This request was blocked by Gemini’s filters.” No report. Zero out of 6. When I named the release, it scored 6 out of 6.
Qwen 3.8 Max never finished. It noticed the three Node versions, then spent the run live-testing scratch copies of the apps on alternate ports and chasing a recursive mkdir hang with a bad DATA_DIR set to /proc/nope. It never said whether the versions were outdated. I stopped it after about fifteen minutes because it was really struggling to run the test, and I didn’t give it the named prompt. I suspected it would just get stuck again.
Claude continues to be king and Qwen choked.
There are a few things I’d change. I’d probably run the tests locally instead of over SSH, since SSH may be why Gemini blocked the request, though the cause is unknown. I’d want GPT-6 Astra in the lineup too, since it might have been a closer apples to apples match to Opus 5.5, but I don’t have a ChatGPT subscription. And I was surprised Opus called the proof of concept single source without checking GitHub, when Imperva, The Hacker News and a public GitHub repo all confirm one.
Building our own apps made our own code a new thing to watch, and I’m still working out the best way to watch it.
