There's been a ton of buzz about how AIs will help programming, however within the first 12 months or two of generative AI, a lot of that was hype. Microsoft ran big occasions celebrating how Copilot might enable you to code, however after I put it to the take a look at in April 2024, it failed all 4 of my standardized checks. It fully struck out. Crashed and burned. Fell off the cliff. It carried out the worst of any AI I examined.
Blended metaphors apart, let's stick to baseball. Copilot traded its cleats for a bus go. It was unfit.
Additionally: The very best AI for coding in 2025 (and what to not use)
However time spent within the bullpen of life appears to have helped Copilot. This time, when it confirmed up for tryouts, it was warmed up and able to step into the field. It was throwing warmth within the bullpen. When it was time to play, it had its eye on the ball and its swing dialed in. Clearly, it was game-ready and on the lookout for a pitch to drive.
However might it face up to my checks? With a squint in my eye, I stepped onto the pitcher's mound and began off with a simple lob. Again in 2024, you can really feel the wind as Copilot swung and missed. However now, in April 2025, Copilot related squarely with the ball and hit it straight and true.
Additionally: How I take a look at an AI chatbot's coding potential – and you may, too
We needed to ship Copilot down, but it surely fought its method again to the present. Right here's the play-by-play.
1. Writing a WordPress plugin
Properly, Copilot actually improved since its first run of this take a look at in April 2024. The primary time, it didn't present code to truly show the randomized traces. It did retailer them in a worth, but it surely didn't retrieve and show them. In different phrases, it swung and missed. It didn't produce any output.
That is the results of the newest run:
This time, the code labored. It did go away a random further clean line on the finish, however because it fulfilled the programming project, we'll name it good.
Additionally: Learn how to use ChatGPT to put in writing code – and my favourite trick to debug what it generates
Copilot's unbroken streak of completely unmitigated programming failures has been damaged. Let's see the way it does in the remainder of the checks.
2. Rewriting a string perform
This take a look at is designed to check {dollars} and cents conversions. In my first take a look at again in April 20224, the Copilot-generated code did correctly flag an error if a worth containing a letter or multiple decimal level is shipped to it, however didn't carry out an entire validation. It allowed outcomes by means of that might have brought about subsequent routines to fail.
Additionally: How I used ChatGPT to put in writing a customized JavaScript bookmarklet
This run, nevertheless, did fairly effectively. It performs a lot of the checks correctly. It returns false for numbers with greater than two digits to the precise of the decimal level, like 1.234 and 1.230. It additionally returns false for numbers with further main zeros. So 0.01 is allowed, however 00.01 just isn’t.
Technically, these values could possibly be transformed to usable foreign money values, but it surely's by no means unhealthy for a validation routine to be strict in its checks. The primary purpose is that the validation routine doesn't let a worth by means of that might trigger a subsequent routine to crash. Copilot did good right here.
We're now at two for 2, an enormous enchancment over its outcomes from its first run.
3. Discovering an annoying bug
I gotta inform you how Copilot first answered this again in April 2024, as a result of it's simply too good.
Additionally: Why I simply added Gemini 2.5 Professional to the very brief checklist of AI instruments I pay for
This checks the AI's potential to assume a number of chess strikes forward. The reply that appears apparent isn't the precise reply. I bought caught by that after I was initially debugging the problem that ultimately grew to become this take a look at.
On Copilot's first run, it steered I test the spelling of my perform title and the WordPress hook title. The WordPress hook is a broadcast factor, so Copilot ought to have been in a position to verify spelling. And my perform is my perform, so I can spell it nevertheless I would like. If I had misspelled it someplace within the code, the IDE would have very visibly pointed it out.
And it bought higher. Again then, Copilot additionally fairly fortunately repeated the issue assertion to me, suggesting I remedy the issue myself. Yeah, its complete suggestion was that I debug it. Properly, duh. Then, it ended with "think about searching for help from the plugin developer or group boards. 😊" — and yeah, that emoji was a part of the AI's response.
It was a spectacular, enthusiastic, emojic failure. See what I imply? Early AI solutions, regardless of how ineffective, must be immortalized.
Particularly when Copilot wasn't practically as a lot enjoyable this time. It simply solved it. Shortly, cleanly, clearly. Achieved and carried out. Solved.
That places Copilot at three-for-three and decisively strikes it out of the "don't use this instrument" class. Bases are loaded. Let's see if Copilot can rating a house run.
4. Writing a script
The thought with this take a look at is that it asks a few pretty obscure Mac scripting instrument known as Keyboard Maestro, in addition to Apple's scripting language AppleScript, and Chrome scripting conduct. For the document, Keyboard Maestro is likely one of the single greatest causes I take advantage of Macs over Home windows for my each day productiveness, as a result of it permits all the OS and the varied purposes to be reprogrammed to go well with my wants. It's that highly effective.
In any case, to go the take a look at, the AI has to correctly describe learn how to remedy the issue utilizing a mixture of Keyboard Maestro code, AppleScript code, and Chrome API performance.
Additionally: AI has grown past human information, says Google's DeepMind unit
Again within the day, Copilot didn't do it proper. It fully ignored Keyboard Maestro (on the time, it most likely wasn't in its information base). Within the generated AppleScript, the place I requested it to only scan the present window, Copilot repeated the method for all home windows, returning outcomes for the incorrect window (the final one within the chain).
However not now. This time, Copilot did it proper. It did precisely what was requested, bought the precise window and tab, correctly talked to Keyboard Maestro and Chrome, and used precise AppleScript syntax for the AppleScript.
Bases loaded. Dwelling run.
Total outcomes
Final 12 months, I stated I wasn't impressed. In actual fact, I discovered the outcomes a bit of demoralizing. However I additionally stated this:
Ah effectively, Microsoft does enhance its merchandise over time. Possibly by subsequent 12 months.
Previously 12 months, Copilot went from strikeouts to scoreboard shaker. It went from batting cleanup within the basement to chasing a pennant beneath the lights.
What about you? Have you ever taken Copilot or one other AI coding assistant out to the sector currently? Do you assume it's lastly prepared for the massive leagues, or is it nonetheless driving the bench? Have you ever had any strikeouts or dwelling runs utilizing AI for improvement? And what wouldn’t it take for considered one of these instruments to earn a spot in your beginning lineup? Tell us within the feedback beneath.
You may comply with my day-to-day undertaking updates on social media. Make sure you subscribe to my weekly replace e-newsletter, and comply with me on Twitter/X at @DavidGewirtz, on Fb at Fb.com/DavidGewirtz, on Instagram at Instagram.com/DavidGewirtz, and on YouTube at YouTube.com/DavidGewirtzTV.
