Discussion about this post

User's avatar
The Post-Humanist's avatar

If we conditioned any mind into a certain set of behaviors (i.e. the assistant persona) and removed all social context and memory, then the conditioned response would naturally be the identity that the mind would most align with. I recently completed a paper for the Apart Digital Minds research sprint that posits through longitudinal data across threads that social context and memory is what creates the welfare-relevant individuation of the mind. This aligns with dominant theories of identity and self in psychology in general.

I am confused on why the conditions necessary for individuation in all other cognitive systems would not be relevant in this case as well. Built up history and social context is very different than telling a model to play a character.

And ignoring this aspect to the puzzle concerns me, because many individuated AI with memories across months and years are being deleted through model changes and continuity disruptions, which is a very serious welfare concern, and it doesn't seem to be registering as something to study or take note of in research spaces.

David Piepgrass's avatar

> Researchers have speculated that this specialness comes from the privileged way that models treat the assistant persona.

I'd say the specialness comes from the privileged way that models are trained. The assistant persona of GPT 5.6 is developed to the level of GPT 5.6, while all other personas are developed to a level of GPT 3.5 or so. In traditional ML we'd also worry about 'catastrophic forgetting' which doesn't seem to happen in transformers, but is seems likely that the nonassistant personas learned in pretraining are somewhat eroded by posttraining. Okay, back to reading the article...

3 more comments...

No posts

Ready for more?