The AI Challenges Businesses Are Actually Focused On Right Now

The AI Daily Brief: Artificial Intelligence News and Analysis

This episode explores the disconnect between high-profile AI safety debates and the real-world challenges enterprises face with AI adoption.

Key takeaways

• Businesses are shifting focus from AI safety hype to practical concerns like agent security, data control, and model ownership.
• Anthropic's proposed three-axis framework (R&D automation, oversight coverage, resource allocation) aims to increase transparency but faces calls for standardization across labs.

Main topics

• Enterprise AI adoption challenges
• AI safety transparency and metrics

Notable quotes

"Don't pace building and owning your own AI models and systems as an enterprise, and there will be no doomsday for you." – Arthur Mensch, Mistral CEO

Conclusion

While AI safety discourse dominates headlines, enterprises are responding by prioritizing

Transcript preview

Speaker 1 (0:00) It has been a heck of a last couple of weeks when it comes to the AI discussion in society. And yet, in all of that, one group that's left trying to figure out if anything has actually changed for them, or if they are just on the same path that they were before, is the businesses and enterprises that have been trying to figure out how to maximize AI for their own value for, at this point, a number of years. Today we're discussing both how enterprises are thinking about, if at all, this new era of AI safety, and also digging a little bit deeper to find out what the real concerns that businesses have and the real AI challenges they're facing right now. As always, we're in a moment where new challenges are creating new opportunities as well. The AI Daily Brief is the daily podcast and video about the most important news and discussions in AI. Speaker 1 (0:50) All right, friends, quick announcements before we dive in. First of all, thank you to today's sponsors, KPMG, Blitzy, Robots and Pencils, and HyperAgent. To get an ad-free version of the show, go to patreon.com slash ai-dailybrief, or you can subscribe on Apple Podcasts. And to learn more about sponsoring the show, send us a note at sponsors at ai-dailybrief.ai. We are now two weeks into the AI safety discourse absolutely dominating the conversation. For those who think that awareness of these issues had been sorely lacking, it has been a very good period. Although now seeing polling numbers that suggest that something like 17 % of Americans are completely convinced that AI is going to end humanity, there is certainly some reasonable concern that we might have over-calibrated. Holding that discussion aside for a moment, One thing that I think almost everyone is looking for is increasing specificity. That is specificity of policy, but also specificity of monitoring. And it's to that that Anthropic speaks with their new proposed set of measurements to help the public understand just how quickly advanced AI is developing. Once you move past the scariest headlines, this month's safety debate has been largely about recursive self-improvement and the idea that AI development is A, moving too fast, but B, really about to move much. more quickly. The threat of societal destruction makes for a good headline, but the AI researchers issuing the warnings have had a difficult time describing exactly what they've seen inside the labs. To better inform the public, Anthropic has proposed a three-axis measurement to understand the current pace of AI development. The first axis is AI's ability to build the next version of itself. The second is Anthropic's ability to oversee and intervene in the actions of agents. And the third is the scale of resources that go into AI model development. Now, Anthropic explicitly notes that these measurements only look at how models are built. In other words, they are measuring only the inputs to model development, but they argue that these inputs are correlated to the growth in model capabilities. On the first measure, the use of AI models in future model training, Anthropic put together a measure they called the R &D Automation Index. It seeks to measure every aspect of model training, rate how automated each task currently is, and aggregate those rankings to express them as a single number. By Anthropic's measure, Claude now quote-unquote leads 26 % of their R &D work and quote-unquote collaborates on more than 90%. Anthropic's definition of AI-led R &D was tasks where a human provides a high-level goal and the AI completes it end-to-end with human oversight. And here you see the challenge even as they are trying to be precise. They are talking about AI's leading R &D, but that definition of leading still includes the human providing a high-level goal rather than the AI determining the goal. which is what I think some might assume when they hear AI-led R &D. For their part, Anthropic was careful to note that, quote, Claude is not operating fully autonomously for any measured subset of AI R &D work. Still, they noted that research automation has massively increased since Mythos finished its training run and has continued to ramp in recent months. In March, prior to Mythos, only 1 % of Anthropic's R &D was led by AI. Mythos boosted this to 12 % in May, and it has doubled since then. Regarding oversight of agents, Anthropic measured coverage, review latency, and escalation rates within their agent oversight systems. They said that around 30,000 agents are currently doing research and engineering work at any particular time. Anthropic claims to have 100 % coverage of agentic actions for this work, instant AI review of any flagged actions, and around a 0.002 % escalation rate for real-time review, meaning around 1 in 47,000 actions are blocked by the monitoring system. Anthropic also maintains an after-the-fact review system with 100 % coverage. This system flags around 100,000 agentic transcripts per week that are then parsed for false positives with around 50 escalated to human review. In other words, around one or two transcripts per thousand cause any material concern. Finally, on resource allocation, Anthropic expressed their measure in terms of a percentage of overall compute dedicated to safety systems. Around 6 % of the compute allocated to AI-assisted R &D went towards safety, and around 12 % of the compute allocated to AI-led R &D went towards safety. Anthropic noted that this measure doesn't include classifiers that constantly monitor agentic activity, and is a quote, imperfect proxy for how much a company focuses on safety. This is because safety research consists of individual researchers designing experiments, which is time-consuming even though running the experiments is not particularly compute-intensive. Overall, Anthropic's goal was not to provide perfect measures, Instead, it was to define and propose a set of metrics that any AI lab could report as part of a regulatory system. They conclude, As the world considers pacing the frontier, we should do everything possible to minimize the gap between what frontier labs know and what the public knows. This means better measuring the development of AI, reporting on it publicly, and giving society an opportunity to decide how to use this information. We hope to model that transparency by releasing these measurements, and will continue to do so. Now, the response to this announcement, I think, shows the good, bad, and difficult of this particular moment. On the one hand, it is the rare move where people on all different sides of the AI safety debate largely think that this existing is better than it not existing. AI safety researcher Jeffrey Ledish writes, Great to see this from Anthropic. A few weeks ago, I wrote that the company could be a lot more transparent, and I appreciate how they've stepped up, both with this and the recent incident investigation report and misuse report. At the same time, there are plenty of people who also wanted to see more. Prime Intellect's Eli Bakoush writes, Interesting that OpenAI gives us the breakdown of usage per R &D task and Anthropic gives us automation levels. Would be great to combine both and get the evolution of automation per R &D task. There are also some who argue that the self-reporting while a fine start isn't enough. Arun Rao writes, We need standard measures across all labs to measure progress to RSI and publicly report on it weekly to monthly. Now, one discussion that we'll see a lot more of, especially as markets digest all of this, Is that one way, perhaps a cynical way, perhaps a realistic way, to look at an increase in safety monitoring is to view it as really expensive overhead for R &D, meaning that these measures of percentage of spend on safety is basically a measurement of that overhead. Will markets reward companies that spend more on safety? Or will they punish them for cutting into their own margins? This shows another complication for having these companies operate in a public market environment. For what it's worth, this week also reminded... That recursive self-improvement is not just the province of the U.S. labs. ZAI released a blog post on Thursday called Towards Recursive Self-Improvement, How GLM Built Its Own Inference Infrastructure. The first sentence reads, As we develop GLM, the model sometimes exhibits capabilities that surprise us and even unsettle us. The most recent moment that shook us, GLM is increasingly helping build AI itself. We watched the model complete an infrastructure task that would have previously taken a team of experienced infrastructure engineers weeks. When we realized that this work would directly change how the next generation of models is trained, we became even more convinced. Our successors are the AI systems we are creating ourselves. It's certainly beyond the scope of