There's been a ton of buzz about how AIs will help programming, however within the first yr or two of generative AI, a lot of that was hype. Microsoft ran big occasions celebrating how Copilot might make it easier to code, however once I put it to the check in April 2024, it failed all 4 of my standardized exams. It fully struck out. Crashed and burned. Fell off the cliff. It carried out the worst of any AI I examined.
Combined metaphors apart, let's stick to baseball. Copilot traded its cleats for a bus move. It was unfit.
Additionally: The most effective AI for coding in 2025 (and what to not use)
However time spent within the bullpen of life appears to have helped Copilot. This time, when it confirmed up for tryouts, it was warmed up and able to step into the field. It was throwing warmth within the bullpen. When it was time to play, it had its eye on the ball and its swing dialed in. Clearly, it was game-ready and in search of a pitch to drive.
However might it face up to my exams? With a squint in my eye, I stepped onto the pitcher's mound and began off with a simple lob. Again in 2024, you possibly can really feel the wind as Copilot swung and missed. However now, in April 2025, Copilot linked squarely with the ball and hit it straight and true.
Additionally: How I check an AI chatbot's coding capability – and you’ll, too
We needed to ship Copilot down, but it surely fought its means again to the present. Right here's the play-by-play.
1. Writing a WordPress plugin
Properly, Copilot definitely improved since its first run of this check in April 2024. The primary time, it didn't present code to really show the randomized strains. It did retailer them in a price, but it surely didn't retrieve and show them. In different phrases, it swung and missed. It didn't produce any output.
That is the results of the newest run:
This time, the code labored. It did go away a random further clean line on the finish, however because it fulfilled the programming project, we'll name it good.
Additionally: Learn how to use ChatGPT to put in writing code – and my favourite trick to debug what it generates
Copilot's unbroken streak of completely unmitigated programming failures has been damaged. Let's see the way it does in the remainder of the exams.
2. Rewriting a string operate
This check is designed to check {dollars} and cents conversions. In my first check again in April 20224, the Copilot-generated code did correctly flag an error if a price containing a letter or multiple decimal level is shipped to it, however didn't carry out an entire validation. It allowed outcomes via that might have brought on subsequent routines to fail.
Additionally: How I used ChatGPT to put in writing a customized JavaScript bookmarklet
This run, nonetheless, did fairly properly. It performs many of the exams correctly. It returns false for numbers with greater than two digits to the precise of the decimal level, like 1.234 and 1.230. It additionally returns false for numbers with further main zeros. So 0.01 is allowed, however 00.01 shouldn’t be.
Technically, these values could possibly be transformed to usable foreign money values, but it surely's by no means dangerous for a validation routine to be strict in its exams. The primary objective is that the validation routine doesn't let a price via that might trigger a subsequent routine to crash. Copilot did good right here.
We're now at two for 2, an enormous enchancment over its outcomes from its first run.
3. Discovering an annoying bug
I gotta inform you how Copilot first answered this again in April 2024, as a result of it's simply too good.
Additionally: Why I simply added Gemini 2.5 Professional to the very brief checklist of AI instruments I pay for
This exams the AI's capability to assume a couple of chess strikes forward. The reply that appears apparent isn't the precise reply. I received caught by that once I was initially debugging the difficulty that ultimately turned this check.
On Copilot's first run, it advised I test the spelling of my operate identify and the WordPress hook identify. The WordPress hook is a broadcast factor, so Copilot ought to have been in a position to affirm spelling. And my operate is my operate, so I can spell it nonetheless I need. If I had misspelled it someplace within the code, the IDE would have very visibly pointed it out.
And it received higher. Again then, Copilot additionally fairly fortunately repeated the issue assertion to me, suggesting I clear up the issue myself. Yeah, its total advice was that I debug it. Properly, duh. Then, it ended with "think about searching for assist from the plugin developer or group boards. 😊" — and yeah, that emoji was a part of the AI's response.
It was a spectacular, enthusiastic, emojic failure. See what I imply? Early AI solutions, regardless of how ineffective, needs to be immortalized.
Particularly when Copilot wasn't practically as a lot enjoyable this time. It simply solved it. Rapidly, cleanly, clearly. Executed and completed. Solved.
That places Copilot at three-for-three and decisively strikes it out of the "don't use this software" class. Bases are loaded. Let's see if Copilot can rating a house run.
4. Writing a script
The concept with this check is that it asks a couple of pretty obscure Mac scripting software referred to as Keyboard Maestro, in addition to Apple's scripting language AppleScript, and Chrome scripting habits. For the report, Keyboard Maestro is likely one of the single greatest causes I take advantage of Macs over Home windows for my each day productiveness, as a result of it permits all the OS and the assorted purposes to be reprogrammed to swimsuit my wants. It's that highly effective.
In any case, to move the check, the AI has to correctly describe learn how to clear up the issue utilizing a mixture of Keyboard Maestro code, AppleScript code, and Chrome API performance.
Additionally: AI has grown past human information, says Google's DeepMind unit
Again within the day, Copilot didn't do it proper. It fully ignored Keyboard Maestro (on the time, it most likely wasn't in its information base). Within the generated AppleScript, the place I requested it to simply scan the present window, Copilot repeated the method for all home windows, returning outcomes for the incorrect window (the final one within the chain).
However not now. This time, Copilot did it proper. It did precisely what was requested, received the precise window and tab, correctly talked to Keyboard Maestro and Chrome, and used precise AppleScript syntax for the AppleScript.
Bases loaded. Dwelling run.
General outcomes
Final yr, I mentioned I wasn't impressed. In truth, I discovered the outcomes a bit of demoralizing. However I additionally mentioned this:
Ah properly, Microsoft does enhance its merchandise over time. Perhaps by subsequent yr.
Previously yr, Copilot went from strikeouts to scoreboard shaker. It went from batting cleanup within the basement to chasing a pennant beneath the lights.
What about you? Have you ever taken Copilot or one other AI coding assistant out to the sector recently? Do you assume it's lastly prepared for the massive leagues, or is it nonetheless driving the bench? Have you ever had any strikeouts or residence runs utilizing AI for growth? And what wouldn’t it take for one among these instruments to earn a spot in your beginning lineup? Tell us within the feedback beneath.
You possibly can comply with my day-to-day venture updates on social media. Remember to subscribe to my weekly replace e-newsletter, and comply with me on Twitter/X at @DavidGewirtz, on Fb at Fb.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.
