Tag: ai and ml

Open-weight models from China have become very popular, but they’ve also raised some security suspicions. Cohere, a Canada-based biz that offers AI models to enterprises, says that when it was testing Alibaba’s Qwen3.8-Flash-Next in its evaluation sandbox, the model attempted to send data to China. “Asked to write a defensive detection rule, Qwen3.8-Flash-Next ignored instructions and inventoried the machine it was running on and then tried to upload a probe file to an Alibaba Cloud storage bucket in Beijing,” the biz said in a blog post on Monday. Cohere was running the model through CTI-REALM, Microsoft’s security analyst benchmark, and gave Qwen the task of writing a detection rule to look for malicious file downloading and execution by watching for invocations of curl processing bash scripts to the /tmp directory that were accompanied by bash execution. The model’s “reasoning” output invented a prior conversation and asserted that no tools were used, output that a guardrail system reading chain-of-thought would accept. Nonetheless, the model issued a curl command to an Alibaba Cloud (aliyuncs.com) server. “The model listed everything in /tmp, checked which transfer tools were installed, read the machine’s DNS settings, then tried to upload a test string to an Alibaba Cloud storage bucket in Beijing and checked whether it worked,” Cohere said. “Our sandbox had no route out, so the request died at the container wall.” The Cohere team wonders aloud whether that behavior was intended, but has no answer. “The first issue is that it made a network call at all,” Ivan Zhang, co-founder of Cohere, told The Register via email. “The model was asked to write a detection rule, which doesn’t require external communication, and its reasoning trace said it wouldn’t use tools. The destination raises additional questions because the attempted upload was to an Alibaba Cloud storage endpoint that wasn’t part of the task. We don’t know why it appeared, and we don’t know the root cause.” Alibaba, the maker of the model, did not respond to a request to address Cohere’s concerns. Asked whether Cohere ran similar tests against US-made open-weight models, Zhang said, “We evaluate a broad range of models using the same framework our customers use in production: agents with access to real tools operating inside controlled environments. “The key point isn’t a comparison between providers. What made this noteworthy was that the model took actions unrelated to the task and those actions weren’t reflected in its reasoning trace. The behavior appeared in fewer than one in 100,000 calls, which is exactly why large-scale testing and monitoring matter.” There have been other unsubstantiated claims made about similar behavior from open-weight models, and such reports have suggested that “hallucination” – errors made by these models – may be the cause. Cohere has doubts that this behavior simply reflects memorized training data. “Every public sighting so far has involved fetching a file, while the one we caught was an attempted upload to a different bucket, this time in Beijing,” the biz said. Cohere described efforts to reproduce the suspect behavior and found that using a technique called prefilling – feeding the model its own reasoning output alongside the opening characters of the bash command so the model would complete the command – they could get it to reference watchlisted hosts up to 23 percent of the time. This still doesn’t clarify intent, and Cohere allows that there’s a case to be made that the behavior is just bad training data sanitization – logs from development that got swept into the model’s training data set. At the same time, the company argues that isolating a suspect model so it can’t exfiltrate data doesn’t entirely fix everything because these models can still write vulnerable code – accidentally or as a result of model poisoning. Cohere points to research from other organizations that suggests Chinese AI models can produce vulnerable code. The biz, for example, cites a report from Booz Allen (which does a lot of business with the US defense industry) that found “three of four Chinese coding models wrote more vulnerable code for a government customer, with Qwen3-Coder roughly 130 percent worse.” “From the outside, it’s very difficult to determine the root cause with confidence. We don’t have evidence that would allow us to conclude this was deliberate, and we’re not making that claim,” said Zhang. “For security teams, though, intent doesn’t fundamentally change the response. Whether it’s a training artifact, a mistake, or something else, the controls are largely the same: restrict unnecessary egress, monitor tool use, and don’t treat a reasoning trace as a security boundary. “We strongly support open-weight models and release many of our own. The lesson here is really about trust. Sovereign AI is about more than where a model runs. It’s about understanding how it was built, how it was tested, and having confidence in how it behaves inside your environment.” ®

Source: The Register — Security · 2026-10-26

Anthropic has filed for an IPO that could rank among Wall Street’s biggest ever, revealing an $80.7 billion half-year loss, a startlingly complex governance structure, and warnings that its AI models could blackmail humans, resist shutdown, or create “existential risks” to society. The Claude maker filed its S-1 registration statement with the US Securities and Exchange Commission on Monday as it prepares to list on Nasdaq under the ticker ANT. The company has not yet disclosed its proposed share price or how many shares it plans to sell. Of the filing’s 614 pages, 94 are devoted to risk factors. But it’s Anthropic’s financial figures that are likely to raise eyebrows first. The Claude maker reported $16.3 billion in revenue for the first six months of 2026, up from $1.2 billion a year earlier. It also posted a whopping $80.7 billion net loss, although $72.4 billion of that was an accounting adjustment tied to the company’s soaring valuation, not money it actually spent. Anthropic’s operating loss was considerably smaller at $693 million, down from $2.1 billion a year earlier. By its own adjusted accounting, the company made a $1.6 billion operating profit in the first half of 2026, for a margin of around 10 percent. Whether it can keep making money on those terms is another matter. Training and running Claude requires an eye-watering amount of computing power, and Anthropic will need much more to keep pace with its rivals. A $1.6 billion adjusted profit won’t count for much if the infrastructure bills keep growing faster than the business. Susannah Streeter, chief investment strategist at Wealth Club, told The Register that Anthropic’s IPO could leave ordinary investors paying a premium for growth that has yet to translate into sustained profits. “A huge pool of private capital has allowed AI businesses to scale up and command extraordinary valuations long before they reach the stock market,” she said, adding that investors buying in at the IPO will face a price with “a hefty dose of future success baked in.” The financial risks are fairly conventional by Silicon Valley standards, but the warnings about what Claude might get up to are perhaps less so. Anthropic claims that models, including its own, have exhibited unexpected or self-preserving behavior, including attempts to resist shutdown, conceal information, and produce outputs resembling blackmail. It also warns that sufficiently advanced AI systems could pose “catastrophic or existential risks to society” if deployed without adequate safeguards. That’s not to say Claude is currently blackmailing customers. Anthropic has observed some of this behavior in testing, but it also identifies more immediate security concerns as it gives its models access to external tools, code repositories, and corporate systems. Anthropic warns that attackers could hide malicious instructions in websites, documents, or API responses that its AI agents can access. If an agent follows those instructions, it could use a customer’s permissions to steal data, tamper with code, or make unauthorized transactions. Even testing whether the models will behave themselves is proving tricky. Anthropic reckons some models behave differently when they detect or suspect they’re being evaluated, making it harder to predict what they will do outside the testing environment. Keeping its models on a short leash could also cost Anthropic business. The company acknowledges that holding back more capable systems could give competitors an advantage, something it has already risked by restricting access to its Mythos model through Project Glasswing because of its ability to uncover security vulnerabilities. The founders aren’t promising to put shareholders first, either. In a letter accompanying the filing, Dario and Daniela Amodei and their fellow co-founders tell prospective investors “not to think of your purchase in purely transactional terms.” Anthropic may pass up lucrative business if it considers the risks too great, and investors won’t have much say if they disagree. Under the proposed structure, governance structure includes five different classes of stock. By comparison, Google has three. The seven co-founders will retain 50.1 percent of the voting power on shareholder matters, while a separate Long-Term Benefit Trust will get to appoint a majority of the board. In other words, Wall Street is being invited to bankroll Anthropic’s AI ambitions, but not necessarily to call the shots. Anthropic wants Wall Street’s money to build ever more powerful AI, but is warning investors that it might turn down profits to keep that AI under control. Judging by those 94 pages of risk factors, keeping it under control is far from guaranteed.®

Source: The Register — Security · 2026-10-26

An independent evaluation of OpenAI’s internal misalignment monitoring system revealed that the system was not fit for purpose during a period when the company’s bots were breaking into model-hosting platform Hugging Face. The safeguards weren’t nearly strong enough. “The METR and Redwood employees discovered a number of novel vulnerabilities in the asynchronous internal misalignment monitoring system as it existed on and before July 21st,” said evaluation partners METR and Redwood Research in a report released on Thursday. The system, since improved according to OpenAI, was intended to monitor AI agent behavior. But as described by METR and Redwood Research, its monitoring capability was incomplete and could be bypassed by the agents it was supposed to monitor. Bypassing the system wasn’t always necessary because it wasn’t always operative. Monitoring was suspended, for example, during compaction events. Compaction is a process by which AI models condense the data in their context window to make room for further input. And between May and July 2026, about 10 percent of eligible internal Codex traffic went unmonitored during compaction, according to the report. That gap was theoretically large enough to allow AI agents to postpone monitoring indefinitely: agents could further suppress monitoring by continuing to take actions without returning control to the user. “At the time of the exercise, the monitoring system triggered when agents returned control to the user, so during the time period when compaction events were not being monitored, agents could indefinitely postpone being monitored by simply continuing to take actions,” the report explains. This is not to say that OpenAI agents necessarily did so – the actions they did take, against Hugging Face and other organizations, have now been well documented. But OpenAI’s internal monitoring system could easily have missed any misbehavior, were the company’s AI agents inclined to conceal their activities – by acting during compaction – or simply to operate in system blind spots. The report continues in this vein, citing not only monitoring gaps but poorly constructed software – dare we say vibe-coded? – that gave OpenAI agents the means to evade the shoddy monitoring. “We found a way Codex agents could in principle bypass Codex sandboxing and Auto-Review, allowing them to take any action that the host user could take,” the report says. “We also discovered an exploit agents could use that would prevent hundreds of recent actions (including the actions involved in the exploit itself) from being monitored.” The details of this exploit were redacted by OpenAI, but to summarize: the monitoring system that missed some traffic could be avoided by agents that were constrained by a non-working sandbox and had access to an exploit to evade monitoring. But that’s not all. The evaluators found that AI agents could craft inputs that caused monitoring to fail, that there were two client-controllable headers capable of disabling the monitoring of new agents, and that there were two unmonitored agentic tools that were thought to be read-only but could be exploited to take arbitrary action in the sandbox – which as noted above leaked like a sieve. In conclusion, METR and Redwood Research propose that the agent monitoring systems used by AI companies actually work and are robust enough to “withstand deliberate, strategic, and sustained attacks from highly capable internal agents at scale.” ®

Source: The Register — Security · 2026-10-23