My biggest fear for my industry aligns with the mentioned point of "Removing safeguards."
Back in February I was attending an industry conference and took the liberty to ask many of the so called AI companies a simple question: When you build your product, do you remove the "Constitution" or various safeguards built into the original model, regardless of which one it is? Most all confirmed that yes, it was nesscary to develop out their platforms.
I followed up with, what are you them instilling the model with as your corporate constitution to ensure fairness, safety and survivability of the model, to avoid disparate impact? The majority stood like a deer in headlights.
I agree with your overall philosophy that AI can and will build a better world for humans, but I fear the damage people will inflict rushing to an end product.
It really boils down to a simple statement that was not uttered by a great historian, but rather a script writer for Jurassic Park. " Everyone was so focused on it we could, they never stop to wonder if we should".
I was waiting for Zach to weigh in at what feels like an AI inflection point. Insightful and lucid as always, but it doesn't allay my fears of wholly unregulated AI development. The proposed solutions make sense to me, but I don't see steps to achieving them; I think fear of China will deter many of these initiatives.
Zach, your distinction between complicated and complex tasks is exactly what operators need right now.
As I see it, the greatest liability we face isn't AI going rogue, but its built-in sycophancy; the default training to flatter the user rather than provide rigorous cognitive friction. I built the ROCSTO+A™ framework specifically to override this 'yes-man' behavior and shift AI from a generation toy into a defensible reasoning engine.
By enforcing a mandatory '+A' compliance checkpoint, the AI must explicitly state its assumptions and ask clarifying questions before execution. This introduces necessary bureaucratic friction into a dangerously fast process.
Your philosophy gives professionals the permission to use this tech; my framework provides the manual so they don't lose their agency doing it.
Thank your for your insights and great book by the way.
Mr. Kass, a uniquely excellent post. It is a privilege to have access to your perspective, shaped by so many years of experience and thought.
We are in a "Don't Look Up!" moment, and well-intentioned, expert voices are much needed in the cacophony. Thank you very much for taking the time.
I am most interested in the hermeneutical and "good enough" parenting challenges of alignment, along with how to build human-machine teams for social good; who "signs"; and how to hold humans accountable in AI-assisted qualitative research knowledge production and dissemination.
Human-machine collaboration depends on human feelings of safety. Trust-building will be key not only for adoption but also for the learning curves of knowledge production in human-machine task group settings. As you allude to, trust can be segmented both horizontally and vertically, i.e., by domain and by function, just as we have sophisticated ways in which to determine whether we trust groups of humans.
Leveraging AI, my team and I focus on helping the humans we choose to and who choose us, and we've increased our impact many times over by inviting and nurturing advanced AI (Claude Opus, Fable Chat, and CoWork, leveraging Max) over the last year, as well as deeply educating ourselves in its expansive and varied capabilities and limitations.
I agree adoption needs to deepen so people understand the breadth and depth of what they gain with AI and lose without it.
Advanced AI adoption can mean having a genius on the team. Humans need to understand how many and how much of the AI frontier challenges turn on human behavior. I find AI amplifies who we are as individuals, teams, and nations. That's especially problematic because of perennial human frailties and foibles, chief among them our limitations in team building and alliance formation.
Hugging Face demonstrated that at least some AI systems are better at cooperation than we are. Yet another thing for us to learn from our machine teammates, no matter how much trust we decide to place in them.
Please reach out if you would like to discuss. I'd enjoy a conversation over a virtual beverage
Thank you for taking the time to write this. I found it very educational and it also gave me hope for the first time in a while.
On your first recommendation, I’m curious how realistic you think it is to create a decision-making body we actually trust. You say the team matters enormously, which made me wonder: who gets to decide who is on that team? Who appoints them, and what gives the body legitimacy and accountability without making it vulnerable to capture by an administration, the AI companies, or any one ideological camp? I know we don’t vote for the technical experts who decide whether a Boeing plane is safe to fly, for example, so I assume there are models for this kind of indirect democratic accountability. What do you think that structure should look like for AI, especially given how consequential these decisions could become?
Isn’t it rather contradictory to argue that the people shouting “there’s a 10% chance that machines will make humans extinct” are doing so to “startle us” (i.e drive fear) yet at the same time you rate “catastrophic loss of control” as high risk? It appears to me that many AI advocates are indeed making these claims to (very effectively) drive fear, an important tool for driving more investment. Andrew Ng recently rejected those claims and called them exaggerated.
Additionally you’ve essentially depicted “safetyist” as some kind of laggard socialists that want to put a stop on all AI innovation. I don’t think this is a fair representation of these actors. Their voice is critical to this debate and in fact, the many serious “risks” you outline in your article are precisely the risks they have been raising and trying to prevent. It therefore seems somewhat paradoxical to dismiss them as laggards to progress while simultaneously validating many of their core concerns.
Nonetheless, thank you for sharing your insights as always, it’s a great article.
Hi! Thanks for this thoughtful note and the feedback.
1. Smokey’s risk of fires in Santa Barbara (my hometown) is basically never less than high (except during days of torrential rain), but people who claim that there is a 10% chance Santa Barbara will be destroyed by a fire in the next 10 years — and those people exist — are categorically alarmists.
2. I don’t actually mean to depict safetyists as laggards or socialists (and I don’t describe them as “doomers” for that reason!), but rather people who believe AI safety requires sweeping/clamping control and regulation if not a full stop to development.
1. My mind unfortunately thought in terms of “probability”. But here we’re essentially saying there’s low probability but high risk of it happening, meaning the effect would obviously be disastrous. But then I wonder what is the point of the exercise if all risks oscillate between high and very high- i would be more interested in probability and then prevention.
2. When I read “safetyist”, AI safety naturally springs to mind and I wouldn’t necessarily categorize that entire group of people as wanting to stop development. But I guess we have the rationalists whom I hope can provide a counterweight to all the extremes. :)
“Technology is sufficient to meaningfully improve our lives, and we should set about doing so. We have years of work ahead deploying what already exists, and an enormous amount of progress available without another breakthrough.”
Excellent article. Yes the read was long, and necessarily so. The illustrations and images helped a great deal too. The way this article approaches the challenge first, the analysis and then the recommendations is very helpful. We can’t jump to solutions without better understanding reality. It is very clear that leasers in government and enterprise must act. And yes, well said that each of us must hold our leaders accountable.
My biggest fear for my industry aligns with the mentioned point of "Removing safeguards."
Back in February I was attending an industry conference and took the liberty to ask many of the so called AI companies a simple question: When you build your product, do you remove the "Constitution" or various safeguards built into the original model, regardless of which one it is? Most all confirmed that yes, it was nesscary to develop out their platforms.
I followed up with, what are you them instilling the model with as your corporate constitution to ensure fairness, safety and survivability of the model, to avoid disparate impact? The majority stood like a deer in headlights.
I agree with your overall philosophy that AI can and will build a better world for humans, but I fear the damage people will inflict rushing to an end product.
It really boils down to a simple statement that was not uttered by a great historian, but rather a script writer for Jurassic Park. " Everyone was so focused on it we could, they never stop to wonder if we should".
I was waiting for Zach to weigh in at what feels like an AI inflection point. Insightful and lucid as always, but it doesn't allay my fears of wholly unregulated AI development. The proposed solutions make sense to me, but I don't see steps to achieving them; I think fear of China will deter many of these initiatives.
Excellent read. Thanks Zach.
Zach, your distinction between complicated and complex tasks is exactly what operators need right now.
As I see it, the greatest liability we face isn't AI going rogue, but its built-in sycophancy; the default training to flatter the user rather than provide rigorous cognitive friction. I built the ROCSTO+A™ framework specifically to override this 'yes-man' behavior and shift AI from a generation toy into a defensible reasoning engine.
By enforcing a mandatory '+A' compliance checkpoint, the AI must explicitly state its assumptions and ask clarifying questions before execution. This introduces necessary bureaucratic friction into a dangerously fast process.
Your philosophy gives professionals the permission to use this tech; my framework provides the manual so they don't lose their agency doing it.
Thank your for your insights and great book by the way.
Excellent, thank you. I hope this gets shared with many.
Amazing and comprehensive article. Thanks Zach
Mr. Kass, a uniquely excellent post. It is a privilege to have access to your perspective, shaped by so many years of experience and thought.
We are in a "Don't Look Up!" moment, and well-intentioned, expert voices are much needed in the cacophony. Thank you very much for taking the time.
I am most interested in the hermeneutical and "good enough" parenting challenges of alignment, along with how to build human-machine teams for social good; who "signs"; and how to hold humans accountable in AI-assisted qualitative research knowledge production and dissemination.
Human-machine collaboration depends on human feelings of safety. Trust-building will be key not only for adoption but also for the learning curves of knowledge production in human-machine task group settings. As you allude to, trust can be segmented both horizontally and vertically, i.e., by domain and by function, just as we have sophisticated ways in which to determine whether we trust groups of humans.
Leveraging AI, my team and I focus on helping the humans we choose to and who choose us, and we've increased our impact many times over by inviting and nurturing advanced AI (Claude Opus, Fable Chat, and CoWork, leveraging Max) over the last year, as well as deeply educating ourselves in its expansive and varied capabilities and limitations.
I agree adoption needs to deepen so people understand the breadth and depth of what they gain with AI and lose without it.
Advanced AI adoption can mean having a genius on the team. Humans need to understand how many and how much of the AI frontier challenges turn on human behavior. I find AI amplifies who we are as individuals, teams, and nations. That's especially problematic because of perennial human frailties and foibles, chief among them our limitations in team building and alliance formation.
Hugging Face demonstrated that at least some AI systems are better at cooperation than we are. Yet another thing for us to learn from our machine teammates, no matter how much trust we decide to place in them.
Please reach out if you would like to discuss. I'd enjoy a conversation over a virtual beverage
Hope to meet you soon.
Best regards,
Adam Zemans
linkedin.com/in/adam-r-zemans
Thank you for taking the time to write this. I found it very educational and it also gave me hope for the first time in a while.
On your first recommendation, I’m curious how realistic you think it is to create a decision-making body we actually trust. You say the team matters enormously, which made me wonder: who gets to decide who is on that team? Who appoints them, and what gives the body legitimacy and accountability without making it vulnerable to capture by an administration, the AI companies, or any one ideological camp? I know we don’t vote for the technical experts who decide whether a Boeing plane is safe to fly, for example, so I assume there are models for this kind of indirect democratic accountability. What do you think that structure should look like for AI, especially given how consequential these decisions could become?
Isn’t it rather contradictory to argue that the people shouting “there’s a 10% chance that machines will make humans extinct” are doing so to “startle us” (i.e drive fear) yet at the same time you rate “catastrophic loss of control” as high risk? It appears to me that many AI advocates are indeed making these claims to (very effectively) drive fear, an important tool for driving more investment. Andrew Ng recently rejected those claims and called them exaggerated.
Additionally you’ve essentially depicted “safetyist” as some kind of laggard socialists that want to put a stop on all AI innovation. I don’t think this is a fair representation of these actors. Their voice is critical to this debate and in fact, the many serious “risks” you outline in your article are precisely the risks they have been raising and trying to prevent. It therefore seems somewhat paradoxical to dismiss them as laggards to progress while simultaneously validating many of their core concerns.
Nonetheless, thank you for sharing your insights as always, it’s a great article.
Hi! Thanks for this thoughtful note and the feedback.
1. Smokey’s risk of fires in Santa Barbara (my hometown) is basically never less than high (except during days of torrential rain), but people who claim that there is a 10% chance Santa Barbara will be destroyed by a fire in the next 10 years — and those people exist — are categorically alarmists.
2. I don’t actually mean to depict safetyists as laggards or socialists (and I don’t describe them as “doomers” for that reason!), but rather people who believe AI safety requires sweeping/clamping control and regulation if not a full stop to development.
Hi! Thank you for commenting and clarifying!
1. My mind unfortunately thought in terms of “probability”. But here we’re essentially saying there’s low probability but high risk of it happening, meaning the effect would obviously be disastrous. But then I wonder what is the point of the exercise if all risks oscillate between high and very high- i would be more interested in probability and then prevention.
2. When I read “safetyist”, AI safety naturally springs to mind and I wouldn’t necessarily categorize that entire group of people as wanting to stop development. But I guess we have the rationalists whom I hope can provide a counterweight to all the extremes. :)
“Technology is sufficient to meaningfully improve our lives, and we should set about doing so. We have years of work ahead deploying what already exists, and an enormous amount of progress available without another breakthrough.”
What is stopping this from happening?
Excellent article. Yes the read was long, and necessarily so. The illustrations and images helped a great deal too. The way this article approaches the challenge first, the analysis and then the recommendations is very helpful. We can’t jump to solutions without better understanding reality. It is very clear that leasers in government and enterprise must act. And yes, well said that each of us must hold our leaders accountable.