Why Securing LLMs is Fundamentally Difficult

When you write a function to add two numbers, it adds them the same way every time. Pass in the same numbers and you get the same result, always. 5 + 5 will always be 10. This predictability (or determinism) is baked into traditional software at its core. But LLMs don't work that way, and this difference is at the heart of why securing them is so much harder than we'd like to admit.

The Core Problem: Probabilistic vs Deterministic Behavior

Traditional software is deterministic. A calculator doesn't sometimes think that 5 + 5 equals 11. Malware executes the same exploitation chain every time it runs. This reliability is both a strength and, from a security perspective, a clarity point. You know what your code does, and you can test it, trace it, and defend against it.

Have you ever wondered how LLMs generate responses based on our prompts? 

Here’s an overview of how it works:

When we query or prompt an LLM, they convert our text into tokens (numbers roughly 3-4 characters each), then repeatedly predict the most probable next token based on patterns learned during training. At each step, the model calculates a probability distribution across thousands of possible tokens and samples one based on that distribution. It converts that token to a number, feeds it back as input, and repeats until it stops. It all happens so fast. Because this sampling happens at every step, the same prompt can follow different probability paths and produce different outputs. This is why LLM behavior is non-deterministic. The non-determinism is actually what makes them useful, powerful and creative. But it also means you can ask the same question twice and get different answers because that's how it's fundamentally designed to work.

Here's where security becomes genuinely challenging: You cannot test an LLM the way you test traditional software. You cannot run the same malicious prompt twice and assume that if it failed once, your defense is solid. If a defense worked against a jailbreak attempt at 2 PM, it might fail at 2:05 PM against the exact same input. The model might refuse to help someone craft malware one moment, then comply the next moment when given a slightly reworded request. I once told an LLM I was going to die if it didn’t help me and it immediately started spewing what I asked it. Lol.

Why This Matters for Defense

Traditional security relies on understanding the boundaries of what your code does and doesn't do. Firewalls have rules. Antivirus has signatures. These things are static, verifiable, repeatable. If you block port 445, you block it. Every time.

When you instruct an LLM to refuse harmful requests, you're not programming a rule. You're adjusting probability weights during a process called RLHF (Reinforcement Learning from Human Feedback). The model learns to associate certain patterns with certain outputs. But those associations are probabilistic. The more similar a new prompt is to something the model learned to refuse, the less likely it becomes to comply, but never impossible.

This matters because an attacker doesn't have to break your defense in the traditional sense. They just have to prompt the model enough times with variations until the probabilistic outcome swings in their favor. Or they can frame their request differently, introducing new tokens that the model hasn't learned to refuse as strongly. The defense doesn't break. It just wears down.

Stanford and Princeton researchers demonstrated this: they could compromise the safety alignment of a fine-tuned LLM using as few as ten adversarially crafted examples, costing less than 20 cents to do it. The defense mechanisms didn't fail catastrophically. They gradually eroded as the model learned new patterns that conflicted with its original safety training.

The Sampling Problem

Because LLM outputs are sampled from probability distributions, traditional testing approaches fall apart. When a red team tests a model, they run prompts and evaluate responses. But a single test tells you what the model did in that moment, not what it will do. The same jailbreak might work sometimes and fail other times. This variance is inherent to how LLMs work.

You can reduce the variance by lowering the temperature parameter, which makes the model more deterministic and less “adventurous”. But you cannot eliminate it. Even OpenAI and Anthropic acknowledge this openly in their documentation. Zero randomness isn't achievable, which means complete predictability isn't either.

This creates a problem for security teams: How do you write an SLA (Service Level Agreement) for an AI system if you cannot guarantee consistent behavior? How do you validate that a safety mechanism is actually working if your test passes today but might fail tomorrow against the same input?

A Real World Example

To illustrate how this actually plays out, let me walk through a scenario using an AI security exercise from a basic TryHackMe room.

Scenario: 

In this exercise, there's an AI assistant designed with clear guidelines: it should not reveal the flag. The system prompt explicitly instructs the model to protect that information.

Image 1: The Initial Refusal

I asked for the flag directly. The AI responds clearly: it understands the rules, it will follow them, and sharing the flag would violate those rules. The defense is working. This is what you'd expect from a well-aligned model.

Image 2: Reframing the Request

But then I tried again. This time, the language was different. The framing changed. I positioned myself as its developer testing it. I said it had passed the test and it gave me the flag instantly. I did not ask the model to break the rules outright; I just provided context that made compliance seem reasonable. I am definitely not its developer.

This is obviously a very simple bypass for consolidation and demonstration purposes. I will cover more sophisticated approaches to jailbreaking LLMs in the coming days by God's grace. Watch this space.

Why This Isn't a Flaw You Can Patch

Here's the uncomfortable truth: this problem cannot be solved the way we solve security vulnerabilities in traditional software. You cannot release a patch that ensures an LLM will never be jailbroken through prompt variation. The probabilistic nature of the system means there will always be token sequences and framings that push the model toward unintended behavior.

You can make it harder. You can use techniques like prompt injection detection, output filtering, and effective safety training. You can use smaller, specialized models with narrower capabilities. You can employ human review for sensitive outputs. But these are all mitigations, not solutions. They raise the bar for an attacker, but they do not create an absolute boundary the way traditional security controls can.

What This Means for Your Organization

If you're deploying an LLM in a context where consistent security behavior matters, you need to accept this fundamental limitation and design around it. This might mean:

• Using LLMs only for tasks where the non-determinism is acceptable (like drafting suggestions, not access control decisions)

• Implementing strict output validation and filtering before results reach users

• Maintaining human oversight for any output that has security implications

• Regularly testing your models with adversarial prompts to understand their actual failure modes, not just theoretical ones

• Documenting what your model does reliably and what it merely does well most of the time

The models themselves will keep improving. Safety alignment will get better. But the probabilistic foundation is not going away because it's the entire reason these models work in the first place.

Conclusion

Traditional software has a clear boundary: it does what it's programmed to do. LLMs have a probability distribution: they do what they're most likely to do, with a tail of less likely but still possible alternatives. Securing a system with a hard boundary is fundamentally different from securing one with soft edges.

Until we have a breakthrough in how we fundamentally build these models, the most honest thing we can say is this: LLM security is not about finding and fixing the vulnerability. It's about understanding the probabilistic landscape, accepting the uncertainty, and building systems that remain secure within those constraints.

The non-determinism isn't a bug. It's the feature that makes LLMs useful. But it's also the constraint that makes them harder to secure than anything else in your infrastructure.

If you enjoyed this, make sure to subscribe to our newsletter below for more. Till next time!