shAI-hulud
nice
nice
Is it like the one in Hackers, at least?
calling it a worm is so funny when it’s literally just following the instructions given to it
The tl;dr:
-
Prompt inject a malicious instruction in a word document that instructs the AI to copy this instruction to other documents as part of the payload.
-
Dumb user downloads and opens the document with copilot enabled, abd ignores the large suspicious white blank page that totally doesnt look like a hidden giant injection attack.
-
Thats it pretty much it.
Copilot will get injection attacked because the prompt is super huge and at the end of the document, so its prior instructions start to fuzzy out.
Then it’ll go “okey doke” and start copying the prompt injection attack payload to a bunch of other documents.
The fix is stupid simple… copilot should just be prompting the user for permission if it ever edits a file other than the one that is open. Im surprised that isnt already the case…?
It certainly is already the case for copilot in vscode.
Back in ancient times when I was a system administrator we got a heads up that there be a new breed of Outlook worm coming soon to our timezone.
So we mailed the entire office that if you get mail that looks like this or that, do not open it, do not interact but delete it on sight.
Most of the office was all right, except pretty much entire sales and marketing departments including the bosses. Most of them had noOo idea what could have happened but one of them explained that they saw the warning but they were curious to see what the virus looks like.
People. What a bunch of bastards.
Sales and marketing don’t count. Critical thinking doesn’t sell. So you won’t find critical thinkers in those departments.
From a security standpoint, those departments are to be considered hostile. But you can lock down the PCs there as much as possible to reduce the offline time because computer-illiterate employees don’t care about being able to install stuff or change settings.
Considering people’s Word formatting skills, a random blank page is not suspicious
The large section is also just a low effort version. I’m sure this could have been inserted instead as a collapsed section, a comment, document metadata, .1 sized font, Alt text on a single pixel image, or any number of other ways to disguise the prompt.
It’s an arms race for defenders, and even someone paying attention might not notice. Not to mention, you don’t even have to open the document. If it’s in a SharePoint library, or attached to an email, there’s a good chance it’s in scope for whatever random ‘assistance’ copilot will attempt automatically
-
Relevent XKCD:
the only way to block it is to get AI to differentiate instructions from data, which is impossible today
Input sanitation, basically security 101. And it can’t currently do it…
if the instruction is messy fuzzy human language to a system that was not coded instruction by instruction but got generated and trained then there never is a way to differentiate instructions from data if i’m not mistaken
Well, good thing we’ve only poured a trillion and a half dollars into it and wrecked the economy.
We know that some humans can be trained to do that just fine. Humans are natural neuronal networks. That implies, neuronal networks can in principle do it. We just don’t have any human-capability artificial neuronal networks yet.
LLMs might never get there. But humans aren’t LLMs. If we ever manage to properly model a human brain, that probably will be able to do that task with human-level accuracy (which actually is pretty good if you only look at professionals of the filed).
Hopefully, it doesn’t actually need a human brain for the task - because modeling that might still be a century off.A human brain model would never get us to a super computer, which is their intended goal.
Our brains have about one exaflop of processing. The NVidia B300 has 15 petaflops of processing. Corresponding to 0.015 exa, you’d need 66 of the strongest GPUs on the market to match the processing power of one human brain.
That’s before considering context, current estimations would put the human brain at 2.5 petabytes of storage. The former B300 has 288 gigabytes of VRAM. Corresponding to 0.000288 of one human brain, the earlier expectation of 66 landing flat at 0.019 the storage of a human brain.
Can arguments be made not all brain capacity is at VRAM at all times? Most likely. But context windows is the primary bottleneck, not compute
Our brains are absolute monsters, we’d gain more from understanding how we can have this powerful of a machine on so little space. The bigger breakthrough would be brainbased processing of software, but I’ve not heard of anyone going that route - Probably because it’s not feasible
Just a hundred years ago, we had one floating point operation per second in mechanical calculators. We got roughly 100 TFlops (fp16; probably much more accurate than human brain neurons) in a high-end gaming GPU now. That’s 14 orders of magnitudes in a hundred years. I guess, 4 more orders of magnitude in another hundred years should be possible. We don’t even have fusion power yet. A lot changes when power becomes a practically unlimited resource.
"To address this class of risk, we use a defense-in-depth strategy with safeguards that block malicious instructions at multiple points and help keep tasks aligned with users’ requests,” Microsoft said.
Keep talks aligned with users’ requests?!? We’re doomed!
“We encourage customers to install the latest updates, use multiple layers of security protection, treat content from unknown sources with caution, and review AI-generated content before using or sharing it.”
No, no you don’t, you encourage customers to spend more to give copilot access to all of SharePoint and all of Exchange, so they can replace human competency, human expertise and long standing employees with copilot until they’re dependent on your ever-pricier subscription and you can raise prices until you recover your vast and incomprehensible LLM losses from your customers.
“Separating instructions from data may be part of the solution, but I think the distinction between data and instructions is not always clear in real-world workflows. For example, a user may ask an agent to arrange a business trip, requiring the agent to retrieve an email specifying the approved itinerary and a document containing the booking procedure,” Måløy [vulnerability researcher and discloser] said.
Or you could hire a PA, who wouldn’t spread the already live copilot worm.
A bunch of security experts note that distinguishing between instructions and data is a solved problem with SQL injection attacks, but completely unsolved in LLMs:
“None of them are rewarding that work commercially right now, so treat that as a multi-year research problem, not something a CISO should wait on.”
Is this a minor technical issue or a major problem? It’s a major problem.
Mike Wilkes, enterprise CISO at Aikido Security, said it would be difficult to overstate the potential problems from this situation.
I like your display name
Well, I like yours too, but don’t remind my wife of that. She’s just not a fan of me flirting with other men.
I have good news for your wife, then!
…which is?
Okay, I was amused before but now this has gone back to the usual frustration. Do you really not see how that reply hints at your assumption that everyone you speak to on the internet is a man?
Yes, it did, and that’s what I suspected, but I saw other possible interpretations and ironically, I didn’t want to assume. Sorry for being annoying. I only meant any of this for fun and didn’t mean to offend or frustrate you.
I was hoping your reply would be something like “am woman” in which case I was planning to tell you that in real life it feels like a majority of the women that I thought were really really cool but who I didn’t feel inclined towards romantically turned out to be uninterested in my entire gender romantically, so I was going to assume that you are also really cool even though my maths teacher was very insistent that correlation does not imply causation.
I assert that your “I have good news for your wife, then!” answer fits with my correlation.
If your reply was “you’re really rubbish at flirting” I was planning on responding with “everybody wins”.
I was hoping to come up with something amusingly self deprecating if it was along the lines of “I’m specifically not interested in you”, but I hadn’t got as far as thinking through that answer, hoping it wasn’t that.
As you can see I’m not all that great at this social interaction thing. Genuinely sorry to have annoyed you.
Well I appreciate your reply, and also,
am woman, am gay
