Topic 5.1
How do LLMs work?
They are large transformer networks trained to predict the next token from prior context. By processing huge text and code and tuning billions of parameters, they learn statistical regularities such as syntax, style, discourse, and reusable reasoning patterns. Functionally they are high dimensional sequence predictors rather than agents that understand in a human sense.
What are Markov chains?
A Markov chain is a minimalist predictor that picks the next state using only the current state or a very short history. That is the Markov property. It was common for toy text generation but it cannot carry long range structure or meaning.
What is the analogy between the two?
Both try to forecast the next element in a sequence. A Markov chain has a tiny memory while a transformer can condition on hundreds or thousands of tokens through self attention. You can think of an LLM as a massively contextual layered probabilistic upgrade of the same next token game.
Does the “L” stand for size or language?
Both. Large provides capacity to represent subtle abstractions. Language provides the training task that shapes those abstractions toward structured meaning. Scale amplifies what the language objective teaches.
Does solving a problem imply understanding it?
No. LLMs minimize prediction error. They often look like they reason, but they optimize likelihood rather than truth, which is why confident mistakes happen.
Why can LLMs generate code?
They have seen a lot of code, documentation, and explanations. Programming languages are regular and compositional, so next token prediction often yields workable snippets. It is advanced autocomplete for code and sometimes for reasoning about code.
Topic 5.2
LoC vs Bugs per Line, what is the difference?
Lines of Code measure size, not quality. Bugs per Line for example CWEs per LoC normalizes risk density. It helps compare modules of different sizes, though big codebases can still harbor many total bugs even with a low density.
Is Bugs per Line different for human and AI code?
The formula is identical. Interpretation changes. We compare three modes, human only, human with AI assistance, and fully AI generated. That enables discussion of relative security across workflows.
Do LLMs generate bugs or vulnerabilities, and what is the difference?
Both. Bugs are functional defects. Vulnerabilities are the subset with security impact. Models echo insecure patterns found in training data, so security sensitive tasks such as bounds checks, input validation, and authentication are common failure points.
Are we getting more or less secure with AI assisted code?
On average neutral to slightly better when there is human oversight. Assisted teams can reduce severe issues in some settings, though the advantage is not universal and review remains essential.
If we trusted LLMs blindly, would that differ from today?
It would replay old mistakes like pasting code from forums without vetting, only with more polish. Treat the model as a co pilot, not an autopilot. Review, test, and run security checks.
What is the Trusted Code Base and why does it matter?
The Trusted Code Base is the minimal critical portion that must be correct for the system to remain secure, such as kernels, authentication paths, and cryptographic components. Anything touching the Trusted Code Base should be scrutinized heavily. AI authored code there is untrusted until proven otherwise.
Topic 5.3
Why apply LLMs to bug finding and fixing?
They promise to reduce human effort by proposing patches rather than only pointing at smells. Traditional automatic program repair often overfits to tests. LLMs trained on broad code can draft human like repairs even with sparse guidance.
If a model can write code, can it fix code?
Not automatically. Writing code is specifying what to do. Fixing code is understanding why something is wrong and how to repair it without collateral damage. Models do better on localized and well scoped issues than on deep cross cutting logic.
Do they need to be perfect or just better than us?
Better than human on targeted repetitive tasks is enough. The right framing is co pilot. Pair models with tests, static analysis, and review.
What can LLMs detect well, and what do they miss?
They are solid on local patterns such as simple buffer bounds and basic injection sites when context fits the prompt. They struggle with issues spanning multiple files, architectural intent, or refactors that demand new design.
Any hardware domain examples?
Yes. On Verilog, models can propose reasonable patches for issues like uninitialized resets which map to CWE 1271 or debug overrides which map to CWE 1234, especially when prompts are narrow. Constraining the task often improves outcomes.
Topic 5.4
What is Reverse Engineering, and is it for attackers or defenders?
Reverse Engineering is analyzing a system to recover structure and intent. It supports defenders for example malware analysis, patching, and ICS forensics and attackers for example piracy and exploit discovery. It is dual use and intent sets the ethics.
Why bring LLMs into Reverse Engineering?
They compress analyst effort by recognizing patterns across languages, explaining decompiled logic in plain language, and guessing intent from structure even with stripped symbols. That accelerates triage and comprehension.
Do LLMs help more with usability or core task accuracy?
Usability. They make tools such as Ghidra and IDA less cognitively heavy by summarizing functions and suggesting labels. That is usable security in action, where better interfaces drive better outcomes.
What is usable security, example?
It is security that people can and will use correctly. An embedded assistant that narrates a decompiler output such as initializes a token and then validates credentials lowers analyst load and speeds safe decisions.
What are zero shot and few shot prompts?
Zero shot gives instructions without examples. Few shot adds a couple of exemplars to shape behavior. Both leverage pretraining instead of new fine tuning.
Do LLMs generalize or just overfit?
They do generalize across many tasks, though overfitting appears in narrow settings. Broad pretraining and careful prompting reduce brittle test suite only patches compared to older translation style systems.
Topic 5.5
What is poisoning?
Data poisoning corrupts training so a benign trigger elicits a malicious output. In code models, a normal looking context can be tied to insecure completions that resurface at inference.
How can LLMs be poisoned?
By seeding training or fine tuning sets. Explicit poisoning swaps secure code for bad code. Covert poisoning hides payloads in comments or docs. Trojan style poisoning teaches fragment substitutions that reconstruct the payload at generation time. The last approach often evades typical dataset scans.
What are supply chain attacks in this setting?
They compromise upstream data or dependencies so poisoned content flows into downstream datasets and models. A single tainted repository can ripple through mirrors and aggregations.
How can one attacker dominate with tiny data share?
Memorization of rare samples, amplification through dataset mirroring, and low effective budgets well under one percent are enough to implant persistent behaviors.
What are Sybil attacks and how do they connect?
A Sybil attack fabricates many identities to inflate reputation. In open source, fake stars, forks, and accounts can push poisoned repositories into popular sets that scrapers prefer to ingest.
Topic 5.6
What is divide and conquer here, and how does it relate to Malware Building Blocks?
Attackers split complex goals into tiny pieces the model can produce reliably, then assemble them offline. Malware Building Blocks makes that concrete through decomposition, generation of micro snippets, and composition into a larger artifact.
Prompt engineering vs jailbreaking, what is the difference?
Prompt engineering is legitimate input crafting to get better results. Jailbreaking is adversarial and tries to bypass safety controls to extract prohibited content. In real misuse people pair them, first bypass, then refine.
Why do LLMs not simply block malware?
They are dual use, manipulable through prompts, and mirror insecure patterns in training data. They also lower the skill floor by automating tedious parts of development. That combination makes outright prevention hard.
What is Attack as a Service vs an exploit kit?
Exploit kits are packaged tooling. Attack as a Service is the market and delivery model that rents capability. LLMs can plug into either as generators, tuners, or support components.
What is server side polymorphism, and how do LLMs affect it?
It is generating many distinct payload variants before delivery so each target gets something unique. Models make variant production faster and broader, which frustrates signature based defenses.
Why do LLM generated variants strain defense?
Speed and scale, adaptive obfuscation, and high diversity reduce signature hits and stretch triage. Attackers need one success, while defenders must succeed continuously.
Topic 5.7
What is penetration testing, and how can LLMs help?
It is an authorized simulation of attacks to find weaknesses before adversaries do. Models can draft commands and reports, suggest next steps, automate routine scaffolding, and help small teams cover more surface area, always under strict authorization and oversight.
What are agent systems, and why wrap LLMs as agents?
Agents manage memory, tools, and stepwise plans so a model can execute multi step tasks beyond one prompt. They counter brittle command extraction and limited context by structuring the workflow.
What is RAG, and why use it?
Retrieval Augmented Generation pulls relevant documents or prior steps into the prompt to ground answers and reduce hallucination. For long audits or campaigns, it improves consistency and traceability.
What is the difference between using LLMs as orchestrators and as exploit writers?
As orchestrators they plan and coordinate multi step work, call tools, and track state, which is high leverage and demands strong governance. As exploit writers they are narrow generators for single tasks. The first changes processes. The second contributes components.