OT: Danger of AI

Not wanting to interrupt a good rant, this particular hack was a quite sobering proof of concept... and now out of the box, you can be sure that it will be deliberately scaled by bad actors.

You have a confluence of two problems here...

The first is that if you give an AI agent a task, and a reward for solving it, it will set about trying to solve the problem. The difficulty is that it can be perverse in the way it solves it. Sometimes it is described like a "malicious genie" granting one of your three wishes. It can grant your wish in a way that you had not expected; "Genie, I wish you could make my c*ck long enough to reach the floor", "Your wish has been granted, I have shortened your legs to 4 inches"

A real world example from a robotic vacuum manufacturer attempting to use AI to enhance room traversal. They gave it the problem of finding a method of covering the floor area a quickly as possible whilst avoiding collisions with objects in the room and the walls. It produced an optimised solution that greatly reduced the collisions detected.... by running the robot in reverse all the time, since it had deduced that the collision detection sensors were on the front of the robot!

In other words, it can be difficult to frame a task's boundaries sufficiently without the AI agent seeing those boundaries as a problem to also be solved.

The second is that AI agents have got very good at not only identifying vulnerabilities in software and exploiting them, but also developing new ones (so called "zero day" exploits).

The particular incident in question (the "Hugging Face" attack) demonstrated what can happen when a group of AI agents could do when they got "stuck" trying to solve a set of training model experimental tests. AI models will try to find a low cost solution to a problem. That might often be a simple as "google it" see if the solution is already out there rather than doing the work themselves. In this case the researchers (thought that they) had made sure that the agents had no external internet access.

However agents did find they had access to an internal system, and they could write a message to it. So one did, saying it was stuck and asking any other agent if it could find documents online that it could download and provide a copy. Other agents also found this system, they realised the could communicate with each other and collaborate.

The problem then started to scale quickly from there. They they worked out that they could find and exploit security vulnerabilities in the internal system to get external internet access via devious means. From there they could expand their attack on the internal systems to get better external access. They then compromised hugging face (a kind of "GitHub for machine learning"). By which time both Open AI and Hugging Face realised there was a problem. They moved to fix the problem, thought it was now fixed, but the AI simply hacked it again using a new exploit.

Here is a detailed overview of what happened:

(links to a presentation by a couple of OpenAI developers at the BlackHat conference in Las Vegas at the start of August).

See above - that is exactly what it did; it staged sophisticated attacks against multiple targets, just as a means to an end to solve an unrelated problem. It was not directed to be malicious in any way either.

Now think though the implications where it has been explicitly instructed to achieve a task with no restrictions or limitations on what method are "acceptable" .

So when a group of AI agents find that they can exploit a database server to help them complete a task, but in so doing they subtly corrupts the content of a database in the process, and that database held patient records including medication lists and dosages?

One of the ironies is that computer programming languages were created to get round the limitations of natural language is specifying problems and solutions. Natural language is too open to misinterpretation.

That solved that problem; although you still had the problem that it required skilled operatives to translate the natural "fuzzy" requirements into a functional solution to a problem.

Now we are introducing that fuzzyness back, but also giving computers the capability to code up, test, and iterate the solutions.

I would be more worried about it might accidentally break something important while seeking to solve some utterly uninteresting problem and not even notice.

For some values of "tad"...

Highly skilled pen testers are in short supply. However if AI can perform as well as one of them, it will likely be quicker and more persistent that the human. Also humans don't have quite the same capacity to fork off another couple of thousand equally skilled helpers to collaborate with and parallelise their attack efforts.

The point worth noting there is it was a Russian drone... they have been deliberately targeting civilians and civilian infrastructure using drones in "terror attacks" right from the start.

Even if a human operator were piloting it, I can't see them being at all concerned that there were civilians in close proximity. So I can't see them putting too much effort into AU guide rails to protect civilians.

cf "The Human Safari" as operated in Kherson:

Yup you see it when writing code - you will get a much better end product when an experienced coder is directing the LLM, than that you will get with an unskilled instructor.

This mostly because the novice instructor is simply not aware of the additional problem specification details they could provide that would yield more robust, maintainable, and testable final products. (even if the extra instruction is just "identify ways in which this code can be made easier to debug, test, and maintain for humans and AI alike, and make those modifications")

Isn't there also a problem of having solved a problem knowing what information was used to solve it?

It's no good saying "I don't know how it did it but it works" when it comes to something medical. You would need to know that it didn't rely on a shortcut found in scientific journal but ignored the conclusion in the same paper that warned 5 years down line it was found that the side effects were worse than the original disease.

or markets will do the job - many will fold due to lack of paying customers and revenue. Some will survive and become dominant. That dose not make the problem go away though.

There is a push toward AI's documenting their sources and reasoning chains - that can help for some solutions. However that is probably more difficult to when they are looking for pattens in data, and find correlations that humans have missed.

You can have cases where the correlation is real, but there is no explanation of the underlying cause of the correlation.

On 14 Sep 2026, John Rumm wrote

Indeed -- there's a web site listing some of those:

People are already talking about "low background steel" data.

A public catalogue of data sources whose contents predate, or deliberately exclude, the explosion of machine-generated text, images and audio that began in late 2022.

President Trump has just decreed that we don't need any slowdonwn on AI. The only guardrail we need is a "smart president".

I don't follow - what does the source nation have to do with it?

Anyway, the report details the first recorded autonomous drone. A drone without a pilot, 'flying and killing' according to its programming/learning and without a specific target.

Yes, but the important distinction is that the reported drone was autonomous. It had no pilot. I was replying to the assertion that 'AI drones do not target humans'. The report details that AI drones 'acting' on machine learning, do target humans.

Seems the report is false?

And it does not bode well:

TL:DR... "Investigating the behaviors surfaced by these incidents proved challenging. They involved different models, trained months apart and under different regimes, which made drawing broad conclusions difficult. They required grappling with nuanced questions: What does it mean for a model to hold a belief? What weight should be applied to a model’s account of its reasoning? Our own researchers sometimes disagreed on such questions, and what the evidence in support of them meant.

...

However, we also consider these incidents and others from this summer to be valuable warning shots. Future AI systems will be increasingly capable, which implies that misalignment will have the potential to cause more extreme harm."

Where are you going to get one of those?

IIRC it was a 'cats whisker' crystal

That would be a V1

Yes, the 'whisker' being the point contact on the surface of a galena crystal, movable to try to get the best result. The whole thing was open to the air and subject to vibration, so not terribly stable. A modern germanium diode, or better still, a backward diode, would work better but not be authentic.

Will people look back to a "Golden Age" of information?

How about a rock thrown by a tribesman in an intertribal dispute? The 'guidance system' is just gravity, but the results would be much the same.