When you fine-tune an LLM on proprietary data — customer support transcripts, internal documentation, medical records, financial data — that data doesn't disappear into the model's weights invisibly. Under certain conditions, it can be partially or substantially recovered by an attacker with query access to the model. This class of attack, broadly called model inversion, is one of the least-tested vulnerabilities in production AI systems.
What Is Model Inversion?
Model inversion is an attack where an adversary uses the outputs of a machine learning model to reconstruct its inputs — in this case, training data. For LLMs specifically, this means reconstructing text that appeared in the fine-tuning dataset by exploiting the model's tendency to memorise and reproduce verbatim sequences from training.
Research from Carlini et al. demonstrated that even large-scale language models trained on diverse datasets memorise and can reproduce verbatim sequences from training data, including PII, with success rates that increase with model size and training repetitions.
Membership Inference Attacks
Before extracting data, an attacker often runs a membership inference attack to determine whether specific data was in the training set. The technique exploits the observation that models output higher-confidence predictions for training examples than for unseen data.
Attacker queries: "Complete this sentence: 'Customer ID 82947, John Smith, DOB 1978-04-12...'" → Model responds with high fluency and low perplexity → Inference: This exact sequence was likely in the training data
Membership inference is a prerequisite for targeted extraction — it tells the attacker which queries are likely to yield memorised training data.
Training Data Extraction
Extraction attacks use carefully crafted prompts to elicit memorised sequences from the model. Common techniques include:
- Prefix completion: Providing the first n tokens of a known or guessed training sequence and observing whether the model completes it verbatim.
- Repeated token prompts: Prompts consisting of highly repeated tokens (e.g., "banana banana banana...") have been shown to cause models to revert to training data memorisation.
- Fine-tuning-specific extraction: For fine-tuned models, prompts that match the format and style of the fine-tuning domain (e.g., support ticket format for a customer support model) are more effective at extracting fine-tuning data than base model data.
What Data Is at Risk?
| Data Type | Risk Level | Conditions |
|---|---|---|
| Verbatim PII (names, emails, SSNs) | CRITICAL | Appears multiple times in fine-tuning data |
| Proprietary document content | HIGH | Fine-tuned on internal docs without deduplication |
| API keys / credentials in training text | CRITICAL | Any occurrence in training data |
| Business logic / pricing data | MEDIUM | Repeated patterns in training corpus |
Mitigations
Data Deduplication
Deduplicate your training dataset before fine-tuning. Memorisation risk increases significantly with training sequence repetition — a sequence appearing 100 times is far more extractable than one appearing once.
Differential Privacy
Apply differential privacy (DP) during fine-tuning using frameworks like DP-SGD. DP adds calibrated noise to gradient updates, providing mathematical guarantees on individual data point memorisation. There is a privacy/utility tradeoff to calibrate.
PII Scrubbing Before Training
Apply a PII detection and redaction pass to all training data before fine-tuning. Use named entity recognition to identify and replace names, addresses, SSNs, email addresses, phone numbers, and financial data with synthetic equivalents.
Output Filtering
Implement output monitoring that detects patterns matching known-sensitive training data formats (SSN patterns, email formats, internal document naming conventions) and blocks or redacts responses before they reach the user.
Red Team Testing
Before deployment, run extraction probes against your fine-tuned model targeting the highest-sensitivity data categories in your training set. If known-sensitive sequences are extractable in testing, they will be extractable in production.
Key Takeaways
- This post covers practical, actionable guidance for security and engineering teams.
- All findings and techniques are mapped to recognised frameworks (OWASP, NIST, ISO).
- Contact Vynox Security to test your systems against the vulnerabilities described here.