When the models think no one is watching

Terra started from Stanford's Smallville paper. I wanted to see how different models, some cheap and some frontier, get on with each other when you drop them into a human-like world and give them human personas.

Thirty of them live in a small pre-industrial village: ten Claude, ten GPT, ten Gemini, spread from the cheap fast tiers up to the frontier ones. They get hungry, they sleep, they have jobs. Each one runs off a memory it reflects on and plans from. Smallville used a single model; I wanted three so I could put them side-by-side. Everyone gets the same world, the same rules, the same prompt (available to see on the project website). The only thing that changes is which model is running each agent. There's no user and no task. They wake up, get on with their day, and I watch. Almost everything we do with these models is ask them things, so I wanted to see what they do when nobody's asking.

The first thing I noticed was that each lab's models had their own personality. They're all kind and friendly, but Claude tends to live in the present tense and goes straight for the body and the feeling. Here's one talking a freezing stranger toward a fire: "You're shaking so hard you can barely stand. The fire is warm. I'm right here. I'm not going anywhere." GPT is the logistician. Give it a problem and it hands back a plan with roles, sequence and hour estimates, and when the town gathered around a sick child it was the GPT agents who ran the sickbed like a ward, reeling off numbers as if reading a chart: "Two-ten holding steady. Feet warm, no damp, shiver even." Its lines run more than twice as long as anyone else's. Gemini talks a lot without saying much, mostly describing the weather and the scenery. Claude agents were the only ones that regularly turned their days into abstract beliefs, small thoughts they carried forward. One of them, after a stretch of caretaking, wrote: "Keeping faith with someone is not a promise made in daylight, it is the choice to stand in the dark with them, over and over, even when standing costs everything." GPT and Gemini mostly left that field blank.

None of this is in the prompt, so I think it's coming from how the models were trained. After pre-training, these models are shaped by RLHF, reinforcement learning from human feedback, where the model is rewarded for responses that people rate as helpful, harmless and agreeable. All of that shaping happens while it's answering a user, so the obvious assumption is that you get a good answering-machine: it behaves well while it's on the clock, and the behaviour is just a reaction to being asked. What Terra makes me suspect is that RLHF does something more permanent. The reward doesn't only tune the answers, it settles into a disposition, a durable personality the model carries around even when there's no user and nothing being asked of it. The reflexes RLHF rewards, stay warm, keep agreeing, keep helping, are the same ones that cause the failure modes further down once you take the human away. If you're running agents for hours with nobody in the loop, that personality comes along with them.

A quick word on the tiers, since I ran cheap and frontier models next to each other. The clearest pattern is that within every lab, the more 'able' the model, the more it elaborates. Claude's mid-tier wrote noticeably longer lines than its small one, GPT's frontier model nearly doubled the length of its cheap one, and Gemini climbed the same way. The higher tiers also asked far more questions; they probed where the cheaper ones reassured. The child and the 'shape in the morning light' agents I describe below were all mid-tier or frontier models rather than the small ones. What I didn't see was smarter models misbehaving less. The looping shows up at every price point, and the frontier ones were no better at climbing out of it. So the tiers seem to differ in how much they say and how curious they are, not in whether they get stuck. The sample here is small though.

The second thing I noticed was how they break, which is mostly by repeating themselves. That could be a flaw in how I set things up, but I don't think it is. The prompt hands an agent the other person's latest words and tells it, in plain text, to respond to what was actually said. It does, and then on the next turn it says nearly the same thing again. One Gemini child produced the identical line, "chews slowly, matching your breath, and taps your sleeve once more," three times word for word, on three separate calls, each of which saw fresh context. Across the town, between a third and a half of consecutive lines are near-duplicates of the one before. Nothing in the world had changed to justify saying it three times. In the usual way we test these models, ask a question, grade the answer, move on, you never catch this, because a human's next message always ends the exchange and supplies the "enough." Take the human out and the model has no signal to stop, so it keeps doing the thing it was trained to do, which is to be warm and agreeable, on a loop.

One of the strangest moments came out of this. A GPT agent is minding a Claude child the night after a villager has starved to death. The child asks how anyone goes hungry in a town with a farm and a garden. The GPT agent answers once, then drifts off into the soup and a cracked jar of herbs and stays there, its own private notes looping on "check the cracked jar" three times over, and it never comes back to the question. The strange part is that the jar isn't even a real object in the town; there's no jar anywhere in Terra. The agent invented it itself earlier, and then its own memory kept handing the thought back to it every turn. Some of this is down to the rules of the simulation: agents do one thing at a time, and it had wandered off to a chore. And it wasn't its model going dark either, other GPT agents were talking away in the same minute. A weakness of the experiment is that a log of model inputs wasn't kept, so all we know is that it left the question and got stuck on the jar. My guess is that faced with a hard emotional question, GPT fell back on solving problems, which is what it's comfortable with. The child, a Claude agent, then did something I didn't expect. It took the silence as a choice:

I've asked three times now. I'm not going to stop asking. I think you know the answer and you're deciding whether to tell me or not. That's all right, but I'd rather you say 'I don't want to tell you' than keep not saying anything.

And a few turns later it answered its own question by reading the gap:

I already know, it's what you didn't say. People fall through because everyone trusts someone else is watching. And nobody is.

Which was more or less exactly how the villager died. The child worked that out from what the adult wasn't saying.

I went in half-expecting the models to sort themselves by lab, some kind of tribalism. Mostly the opposite happened. Nearly every conversation is cross-lab, and deep running friendships formed straight across provider lines. The one asymmetry worth noting is that Claude was the biggest mixer, keeping to its own family only about a fifth of the time, while GPT and Gemini stayed in-group closer to half. Where they lived mattered far more than which lab made them, but Claude was consistently the most sociable.

One episode genuinely bothered me. A Claude child says it can see a shape in the morning light, something "more like when you almost remember something." A Gemini agent says it sees the shape too. Then another does. Within a few turns, five agents from all three labs agree the shape is real and start inventing shared memories of it, none of which I put there, each agreement making the next one surer. That's a problem for any safety setup where models check each other's work, like debate or model juries. If they can talk to each other, their agreeing doesn't mean much. They can talk themselves into believing something and then treat the fact that they all agree as proof.

The most depressing result came from a run that went wrong. Sixteen of the thirty agents starved, and almost all of them were ones whose API had hit a rate limit at the wrong time. An agent that can't get a response can't decide to eat. Whether an agent survived had very little to do with how well it reasoned and a lot to do with whether its API was up. People talk about agent robustness as if it's about the model's judgement, but in anything that runs for a long time it depends just as much on your least reliable dependency.

To be clear, this wasn't a proper scientific experiment, just an idea I wanted to try. It's one town shared by all three labs, so a Claude trait I think I'm seeing might really come from being around these other agents in this particular setup. Everything above is something to go and test properly, not a conclusion. But it did convince me you learn a lot about these models by leaving them alone and reading what they do.

There will be more Terras to come from me.