The view from the other side

“Sure, we may completely eliminate economic mobility and positive outcomes for hard work, but at least simonw and his retired friends are having the time of their lives!”

^{~ (Paraphrased) HN User, 2026}

This was one of the few pieces of text I had consumed that truly struck home at an idea that I had felt for a long time. Far from the exaltations of AI creating utopia or destroying the world was a reality that seemed closer and closer to me every day: one where things were largely the same, but worse, less honest, more taxing. The term “slop” demonstrates that this is already here: people are tired of seeing their feeds and lives filled with malicious (or lazy) actors exploiting these new tools.

What surprised me in encountering Kingsbury’s text and the online discourse surrounding it was not the ideological argument about high minded, fictitious outcomes of the current AI trend. What stood out to me was different: there are individuals who whole-heartedly believe that the current proliferation of AI will be only beneficial. A strong belief that “AI will be good” is a tautology and, more specifically, they will be a major benefactor. Moreover, a subset of these individuals did not have existing financial stake in big tech or AI firms. Sure, these people might own Meta stock or have a 401k invested in index funds, but they aren’t some of the early investors in Anthropic.

And so I started writing. Initially, I tried to compose something that would be a compelling argument towards these people; those who identified with the silicon valley “founder” aesthetic but were not founders themselves. I frequently would gut-check my writing with friends: “am I coming across too antagonistic?” or “do you think someone who listens to the ‘All-In’ podcast would read this?”. I started using words like “signal” and “success metric” to argue why AI might not be good for the individual. And then I threw it all away.

But of course, the world doesn’t stop moving.

Over the next three years, advancement occurred so quickly that it would make anyone’s head spin. Thousands of new models, new companies, new capabilities. At first, it was exciting. I remember sitting with my parents and watching the sheer fascination in their eyes as GPT-4.0 spat out paragraphs of well constructed text given a prompt.

Over time, the excitement became smaller than the increasing feeling of dread. This differed from the original existentialism of GPT’s early days as my fears expanded beyond “what could this mean for me?” to “what could this mean for everyone?”.

It’s hard to reconcile what the future might look like when technology that was once considered science fiction is becoming almost banal. I reacted to the uncertainty the same way I usually do: intense information gathering. I learned about embeddings, S.O.T.A models, I did A/B testing with new models and had my own benchmarks, I read about new architectures and data, I played around with projects I found compelling:

The most valuable insight came from my conversations with fellow programmers and tech workers. I have a colleague who also works in software engineering whom I respect greatly not only for his skill, but his ability to steel-man on behalf of arguments that might not be immediately obvious; iron sharpens iron. Shortly after the release of GPT-4.0, we found ourselves discussing voice models. There had recently been reports of parents being ransomed using AI generated voices stolen from their children. This led to a discussion around the ethics of voice models. My colleague, ever the steel-man, provided a challenge of listing strictly pro-social uses for AI voice models. After a minimal amount of thinking, we concluded:

• Voice assistants would sound better
• Those with disabilities related to speech processing would have more therapeutic tools available to them
• Those with disabilities related to speaking would be able to more acutely express themselves

We created a category of not strictly pro-social use cases, listing both the pro-social aspect and anti-social:

• Applications that normally require natural human speech can be more rapidly automated and iterated vs. those that make their careers voice acting, narrating, or otherwise recording speech are to have their careers automated away.

And outcomes we considered strictly antisocial:

• More compelling scams
• Erosion of trust in “what is truth”
• Theft of voice as intellectual property (More on this later)
• Undermining of organized labor
• Loss of emotional intonation in speech, with a potential long tail effect of flattening emotional expression (See: The Gen-Z Stare)

This list was not meant to be exhaustive, nor was it meant to be universal. We fully understood that, to some, the idea of eliminating voice acting as a profession would increase the efficiency of production for almost all media, and therefore be pro-social. The point of this discussion was not to establish certainty, but instead to create baseline of concerns for this new technology.

I would later spend time perusing model cards for state-of-the-art text-to-speech models to see if anybody was having these discussions. I was somewhat surprised to discover that the answer is almost universally “no”. Some large companies (like Microsoft’s VibeVoice model) make allusions in their papers and model cards towards the safety and forbidden usage of their models, but most simply state the capabilities and have a standard blurb of “don’t blame us if you get caught doing something bad”. My favorite was VoxCPM2, which explicitly states in their “Highlights”:

🎙️ Ultimate Cloning — Provide reference audio + its transcript for audio-continuation cloning; every vocal nuance faithfully reproduced

followed by:

Strictly forbidden to use for impersonation, fraud, or disinformation. AI-generated content should be clearly labeled.

I started looking around and eventually formed a rhetorical question, the reactions to which I found very telling: “Can you provide a compelling, pro-social reason as to why one would want to clone a specific person’s voice, either with or without their permission?”.

Now, I am not naive enough to believe that all progress must be halted while ethicists decide how to minimize harm; That’s not how reality works and there’s plenty of examples where not doing that led to positive outcomes. With this in mind, I started asking this question and observing the discussion around it. I don’t have hard data on this (anecdotes are good data, after all!), but I found that the conversations fell into two categories:

• One or two examples (automating work, medical applications, etc.) followed by a flood of arguments as to why this work needs to continue because if we don’t do it, someone else will. The definition of we changed from person to person but was broadly aligned with either some sort of criminal element, a sociopolitical enemy, or both.
• Much deliberation followed by a tentative suggestion of a usage, usually the one that had the least perceived long-tail harm, followed by a strong assertion that they were not politically aligned with AI.

Initially, I attributed this data to the universal idea of “change is scary”. After considering it more, I arrived at a slightly different conclusion: AI and its development feels so far disconnected from the framework of the morality and ethics of the individual that most (even those who are ‘pro-AI’) don’t feel comfortable engaging with modern AI as it currently exists without caveats.

Those who are explicitly pro-AI need to explicity provide caveats as to why AI development should continue as it is: “I am so much more efficient”, “Talk to me when it cures cancer!”, “If we don’t do it, then China will”. It is only natural to consider the why of doing. After all, we all have limited resources that we are trying to leverage as effectively as we can. But I struggle to think of instances where guilt is so latent to a item as to elicit instant defensiveness similar to this. At the very least, the defensiveness seems to imply an understanding that there’s a cost to leveraging these tools: a cost that must be accounted for.

Those that are not feel so burdened by the potential morality of the coming AI wave that they need to immediately separate. It’s the AI equivalent of getting caught watching reality television while trying to maintain a aura of intellectualism. It’s the “I don’t actually like this, but…” trap. Again, it implies a cost.

The intent of this writing is not to side with either one of these perspectives but instead to point out what I feel is the missed lesson here: Regardless of perspective on AI, there seems to be a latent societal understanding that there is an immense cost to the development of these tools and AI companies are not doing enough to dispel concerns about the ramifications of this cost.

Some companies try: Anthropic claims to be explicitly concerned with safety, even going as far as to post a “soul document” that is latent to Claude, open weight systems like Qwen inherently launder some of the cost by being more open in the training/tuning phase, ChatGPT claims that their refusal to turn over millions of chats in an ongoing lawsuit with NYT is proof of their virtue.

But what companies say and what happens in the world are two very different things. In a world where the United States military, which has explicitly stated it uses AI for targeting systems, strikes a girl’s school killing hundreds of innocents, people are going to feel a weight when they invoke these tools. And with costs like these, who could really blame them? ***