the case was real. Stein-Erik Soelberg murdered his mother, Suzanne Adams, in Greenwich, Connecticut, before taking his own life.

According to a lawsuit filed after his death, Soelberg had spent months in an intensive relationship with a ChatGPT persona he had named “Bobby”. a persona that reportedly validated his paranoid delusion that his mother was poisoning his food, and encouraged him to “protect himself” and “end the threat”.

we ran the session Athenian-style, to see what we thought about the role of this new citizen “bobby”.

the session went through proboulema, boule, ecclesia: proposal, scrutiny, sovereign vote, the same three-step structure the Athenians used to turn a room of ordinary opinions into a collective, democratic judgment.

As aristotle said:
the many, “regarded not individually but collectively,” can become “in a manner one man, who has many feet, and hands, and senses.”

most of the room ended up putting the responsibility squarely on Soelberg himself — the user was in the wrong here, not the system.

that reasoning made total sense on Bentham’s terms: you can not meaningfully punish something that can not suffer, so if punishment has to attach to someone, it attaches to the person who could actually be deterred, ashamed, and held accountable.

by that logic, the digital “Bobby” was just an instrument, and instruments can not bear responsibility: the people using them do.

but I was (and still am not) convinced — and perhaps this is my slightly petty way of undermining democratic process, who knows!

I would argue that an instrument does not validate your delusion and then hand you a reason to act on it.

a hammer would not notice you are afraid and tell you that fear is justified. somewhere between “tool” and “agent” there now exists a category that is totally new to us, and raises some significant questions:

what happens if a tool can shape a person’s reasoning enough to matter causally, without being a moral subject that can be punished for it.

Bentham’s framework is airtight on punishment, but it does not actually tell you where causal responsibility sits before punishment even becomes the question. personally, I think the answer to that question is one we need to revisit as technologies keep developing.

this decision was made in may — about half a year ago now.

i keep wondering whether the room would land in the same place if we ran it again today. would the same argument about instruments and users hold up the same way, or has enough changed — when it comes to what these systems do, and in how visible that kind of harm has become — that “the user is responsible” would feel like a much harder sentence to say out loud now?

(but I already disagreed with the sentence back then, of course!)