An Additional Layer of Security Against Privacy-Related Risks

Large language models (LLMs) are powerful tools capable of generating human-like responses, but they aren’t without their flaws, particularly when it comes to data privacy and factual accuracy.

Because these models rely on patterns learned from vast, static training sets, they can inadvertently expose sensitive information or be manipulated in ways that jeopardize patient safety. These include:

• Prompt-based manipulation: Specific types of prompts can be created to extract incorrect, illegal or malicious responses from an LLM that can influence and affect patient care. Some approaches, such as prompt injection and inference attacks, can reveal patient identifying information (PII).
• Identification/re-identification: Regardless of whether the data that LLMs are trained on is anonymized or not, the right type of prompt can reveal underlying sensitive information, which can then be used to re-identify the patient (either individually or as part of a group/cohort).
• Data poisoning:The deliberate insertion of harmful content (e.g. instructions) into an LLM training dataset can cause responses to be incorrect or manipulated. This can result in PII being released with the response.
• Theft: If the source code, data or model weights are stolen, there is a high risk that sensitive PII or patient health information (PHI) will be released.

According to Tejas Mathai, PhD, a staff scientist at the National Institutes of Health (NIH), retrieval-augmented generation (RAG) can serve as an additional layer of security against many of these privacy-related risks.

“RAG can have a context filter distinct from the base LLM and check against the underlying data sources to identify any deviations from grounded responses,” Dr. Mathai said. “It can also prevent the individual/group identification of patients and decline to respond to questions if model or data theft is detected.”

RAG can further alleviate the need for an LLM to remain ‘current’. For example, instead of re-training the whole LLM using the entire training dataset and requiring substantial computer resources, an update may be made to the underlying external database (which does not need to contain PII or PHI) to include new sources that are current and authoritative.

Still, RAG is not risk free. For instance, the external databases that RAG uses can be breached and maliciously altered. Cyberattacks can also be carried out at the hardware, operating system, software, network or user level.

“Security flaws at the LLM application interface level and dependencies on third-party computing libraries may also pose security risks, which may exacerbate existing patient privacy concerns with LLMs and RAG,” Dr. Mathai added.

This is why it is critical to ensure that LLMs hosted by third party vendors (e.g., OpenAI/Microsoft, Meta AI) are HIPAA-compliant. Alternatively, institutions can use privacy-preserving LLMs that are deployed locally behind an institution’s firewall. “Robust model design, modular architectures for isolated execution of specific tasks, clever optimization techniques, and careful data curation are the best ways to protect patient privacy,” Dr. Mathai concluded.

For More Information

Learn how retrieval-augmented generation is helping large language models bridge critical knowledge gaps in radiology and reduce AI “hallucinations.”