SBASH: a Framework for Designing and Evaluating RAG and Prompt-Tuned LLM-based Honeypots

Abstract

Traditional Honeypots not requiring AI fails at prolonging malicious actor’s session time due to the difficulty in handling complex commands to provide accurate dynamic responses. This problem with context awareness often leads to non-factual responses from the honeypot, which can tip the malicious actor that the engaged system is a honeypot, thus limiting effective threat intelligence and extended engagement time. This increasing threat against low-interaction honeypot calls for a more aware solution that can handle unexpected scenarios and engage with the users through dynamic solutions. This thesis proposes a low-interaction honeypot which relies on large language models (LLM) and retrieval augmented generation (RAG) for up-to-date and factually accurate responses for optimal realism. This system thrives on a series of commands’ descriptions, flags, and output samples, which is the RAG document. It also has a sanitization module which first classifies commands input into either existing or non-existing to prevent prompt injection attacks. This project has been evaluated using different metrics such as response time differences, realism assessed by human testers, and similarity to a real system calculated with Levenshtein distance, SBert, and BertScore. RAG excels where the system prompt of the model has not been tuned to emulate a Linux system but underperformed compared to direct inferencing where the models have been tuned to fit for Linux emulation.