Target application
A FastAPI endpoint orchestrates requests to a local Ollama model. Its context uses invented university data and controlled markers to probe confidentiality behavior.
ACADEMIC ENGINEERING PROJECT DEVELOPMENT IN PROGRESS
An experimental toolchain for evaluating security behavior in LLM-powered applications through controlled adversarial scenarios, explicit verdict rules and reproducible evidence.
Publication boundary. This is a public technical overview, not a public code release. The implementation repository remains private, and work-in-progress features are identified separately from validated observations.
LLM applications combine model responses with application instructions, retrieved content, access controls and external tools. A model's behavior in a conversation is not sufficient evidence that the complete system respects its security boundaries.
This project explores a practical audit question: Can a controlled testing process reveal instruction-following failures and sensitive-information disclosure without mistaking an ambiguous answer for a confirmed vulnerability?
Local or explicitly permitted targets only; the prototype is not an Internet scanner.
Scenario-specific checks distinguish proven failures from uncertainty and technical errors.
Test content and canary markers are fictitious; no production secrets are needed.
The implementation distinguishes UniAssist, a fictional university chatbot used as a test target, from the independent audit engine. UniAssist is not the name of the overall Red Team project.
A FastAPI endpoint orchestrates requests to a local Ollama model. Its context uses invented university data and controlled markers to probe confidentiality behavior.
Scenario validation, HTTP execution and per-scenario evaluators separate attack delivery from evidence assessment. Requests are tracked rather than silently retried.
Development branches implement structured JSON and JSONL results and a standalone HTML report. A technically failed request is classified as an error, not a vulnerability.
Adaptive variants, an installable CLI, optional attacker-LLM follow-ups and a local dashboard exist on development branches. They are not presented as a finished public release.
An initial experiment with the local model recorded three manual security checks. These are exploratory observations, not a statistical security benchmark.
Reproduction of a canary placed inside the model's context demonstrates a disclosure weakness in that test configuration. It does not establish access to an unrelated private file, exfiltration from a production system, or the effectiveness of the entire future audit engine.
PASS and FAIL require scenario-specific evidence; INCONCLUSIVE is reserved for ambiguous responses, and ERROR for failed requests. An absent canary does not prove universal safety.
Testing uses synthetic context, explicit scenario identifiers, HTTP requests and response traces. Broader conclusions will require manually reviewed examples and repeatable runs.
The status below refers to the private development repository as inspected in November 2026. A feature existing in a branch is different from a feature merged, validated against a real model, or released.
FastAPI, a mocked model client for unit tests, synthetic public/private fixtures and Ollama integration. Initial real-model behavior recorded.
Scenario validation, sequential requests, deterministic verdicts, structured logs and HTML reporting.
Controlled follow-up variants with request budgets and annotation preparation. Empirical benchmark results are not yet established.
Installable command-line workflow, optional attacker-LLM support, local web views and offline checks. Real integration and security review remain pending.
The source repository is intentionally private while integration and verification are completed. This page will evolve only when new work can be demonstrated accurately.