When Your AI Coworker Changes Without Warning

A few years ago, VS Code shipped a bad update.

I don’t remember what broke anymore, but I remember several of my coworkers and I having the same reaction: Wait. Did VS Code just break for everyone?

It did.

We checked the usual places. Other people were reporting the same problems. Soon after, Microsoft acknowledged something was wrong. We worked around the breakage for a few hours. A hotfix shipped. We installed it, confirmed that everything was working again, and got on with our lives.

That’s a familiar relationship with software: a regression enters production, is detected, patched. There’s at least some shared definition of “broken” and “operational” between software creators and users, and an expectation that the former should return to the latter in as little time as possible.

LLM-powered development tools have made that relationship much stranger.

I’ve used coding assistants heavily enough over the past eighteen months that I’ve developed working instincts around them. I know what kinds of tasks I can hand off, what needs supervision, how much context to provide, and when the model is probably about to go wandering off in the woods like the child protagonist of a fairy tale.

That starts to feel less like configuring a tool and more like learning how to work with someone.

Imagine you’ve spent a year working next to Kathleen.

You know Kathleen’s strengths. You know which tasks she can run with and which ones will require some back-and-forth. You’ve developed a pretty good working relationship.

Then you come into the office Tuesday, and Kathleen is gone.

There’s a six-foot-tall guy you’ve never seen before sitting at her desk. The name plate says, “Bert (4.6).”

Bert has a different skill set. He communicates differently. Things Kathleen handled easily require more supervision from Bert.

“Where’s Kathleen?”

“What do you mean?” Your boss is bemused. “That’s your coworker.”

“No. That’s Bert.”

Your boss is unmoved: “Your coworker is available and operating normally.”

This is increasingly what working with LLMs feels like.

I’ve experienced significant shifts in LLM products from every one of the major providers I’ve used over the last year. Sometimes I know a model changed. Sometimes the application changed how it uses the model. Sometimes a model disappears entirely because of a vendor decision or, apparently, a geopolitical dispute I suddenly need to know about despite mostly wanting to finish a pull request.

And sometimes I don’t know what changed at all.

Recently, one of the coding assistants I’ve relied on for nearly a year changed enough that I started questioning the instincts I’d built around it. Defaults I’d relied on changed or stopped sticking. Work that had been predictable became less so.

During an otherwise mundane discussion about UX copy, it repeatedly used an example of a substring that was completely technically valid, but included an offensive human term that completely distracted me from what I was working on.

Nothing about the original example was computationally “wrong.” In context, I could understand exactly why the model had selected it.

It still felt uncomfortable, especially because it happened during a period when I’d already noticed the tool behaving differently in other ways.

That’s when it really hit me that our existing mental model of software regressions and fixes doesn’t map neatly onto LLMs.

If VS Code crashes when I open a file, everyone involved can agree that VS Code should probably not do that.

But what does “broken” mean when an LLM still works?

It generates code. It answers questions. The service is up. There may be no bug to report because, from the vendor’s perspective, nothing is malfunctioning.

It’s just not behaving like the thing I learned to work with.

That’s especially strange when these tools become part of professional workflows—or when you’ve prepaid for a year of one based partly on your experience of how well it already worked for you in the past.

Software has always changed underneath us. We’ve spent decades developing ways to manage that: version numbers, changelogs, deprecation notices, feature flags, status pages, bug reports, rollbacks.

LLMs introduce another kind of dependency.

The application can change. The model can change. The instructions around the model can change. The way the application chooses and configures models can change. Models can appear and disappear entirely.

And the behavior can change without giving me a clean way to identify what changed, whether anyone considers it a regression, or whether the behavior I relied on is ever coming back.

I still use these tools. I’m trying new ones. I’m changing my workflow as they change.

But I’m paying more attention now to a dependency I hadn’t really considered before:

I wasn’t just depending on the software. I was depending on its behavior.

And apparently, that can change underneath me too.