Keep Your AI Safe from New Threats
Let us handle the security, so you can focus on innovation. Our tailored solutions protect your AI applications from today’s most sophisticated risks.
Our Key Activities
Research GenAI threats
and rigorously test AI models
to ensure security.
to ensure security.
Evaluate and compare
the ethical characteristics of various models
and LLM (Large Language Model) solutions.
and LLM (Large Language Model) solutions.
Conduct comprehensive audits
and protection assessments
for GenAI business solutions.
for GenAI business solutions.
Develop
tools
to monitor and detect attacks targeting LLM applications.
Perform data cleaning
to remove sensitive information before training models.
Our Focus Areas
[01]
[02]
[03]
[04]
Personal (PII) and sensitive data handling
To mitigate data leaks, we create tools to remove sensitive information from datasets.
Identify and clean
personal data
across systems.
across systems.
Develop data-cleaning
modules for pipelines.
Ensure full compliance
with key regulations
such as GDPR, PCI-DSS,
HIPAA, etc.
such as GDPR, PCI-DSS,
HIPAA, etc.
Implement
cleaning methods
like replacement,
paraphrasing,
and deletion
paraphrasing,
and deletion
Monitoring Prompt Injections and Generated Content
We develop tools to monitor AI applications and detect attacks on ethical standards.
Detect attacks
and analyze messages for toxicity.
Provide security
statistics and reports.
Support popular models like LLama, OpenAI, and YandexGPT.
Integrate with monitoring tools
and cloud providers.
Benchmarking Model Ethics
We ensure ethical compliance and perform comparative analyses of models.
Compare ethical properties
of proprietary and open models.
Test alignment robustness
against jailbreaking.
Maintain a model leaderboard for production readiness.
Red-teaming for Chatbots
We build Red-teaming tools
to test LLM security.
to test LLM security.
Develop testing
solutions
solutions
for LLM applications.
Explore new attack
types
types
on large language models.
Create Red-teaming
checklists
checklists
and methodologies.
Collaborate with
OWASP
OWASP
to share security knowledge.
Our Team
Our dedicated team brings together industry leaders and educators to drive innovation in AI security.
With a strong partnership between ITMO University and Raft, we are committed to developing cutting-edge solutions and sharing our expertise to help businesses secure their AI products.
With a strong partnership between ITMO University and Raft, we are committed to developing cutting-edge solutions and sharing our expertise to help businesses secure their AI products.
GAI Red Teaming: Continuous Monitoring
As more organizations integrate large language models (LLMs) into production systems, ensuring their reliability, security, and performance becomes paramount.
This is where observability and continuous monitoring emerge as vital practices. Observability provides deep insights into the internal workings and behaviors of LLMs in real-world settings, while continuous monitoring ensures that these models operate smoothly and efficiently over time. By adopting comprehensive observability and monitoring frameworks, organizations can increase the reliability of their LLM deployments, promptly address potential issues, and build greater trust in their AI-driven solutions. In this section we present effective strategies for implementing continuous monitoring, and highlight best practices to ensure the performance of AI systems.
This is where observability and continuous monitoring emerge as vital practices. Observability provides deep insights into the internal workings and behaviors of LLMs in real-world settings, while continuous monitoring ensures that these models operate smoothly and efficiently over time. By adopting comprehensive observability and monitoring frameworks, organizations can increase the reliability of their LLM deployments, promptly address potential issues, and build greater trust in their AI-driven solutions. In this section we present effective strategies for implementing continuous monitoring, and highlight best practices to ensure the performance of AI systems.
Observability in LLM / What is observability?
Observability in the production of Large Language Models refers to
the ability to comprehensively monitor and understand the internal
states and behaviors of these models as they operate in real-world environments. Unlike traditional monitoring, which might focus
solely on predefined metrics, observability emphasizes collecting and analyzing diverse data points to gain deeper insights into model performance, decision-making processes, and potential anomalies.
By leveraging observability, several key aspects of LLM production can be significantly enhanced, including:
the ability to comprehensively monitor and understand the internal
states and behaviors of these models as they operate in real-world environments. Unlike traditional monitoring, which might focus
solely on predefined metrics, observability emphasizes collecting and analyzing diverse data points to gain deeper insights into model performance, decision-making processes, and potential anomalies.
By leveraging observability, several key aspects of LLM production can be significantly enhanced, including:
The importance of observability
and continuous monitoring
1. Reliability and Performance
Continuous monitoring ensures that LLMs maintain optimal performance levels, allowing for the detection and resolution of latency issues, resource bottlenecks, or degradation in response quality.
2. Security
Observability helps in identifying unusual patterns or behaviors that may indicate security threats, such as prompt injections or attempts to manipulate the model’s outputs.
3. Continuous Improvement:
Insights gained from observability practices inform ongoing model training and refinement, leading to enhanced accuracy and relevance of the LLM’s responses.
Best practices
During the development of LLM, it is crucial to follow best practices that ensure both the protection and optimal performance of these systems. Adopting robust monitoring and proactive defense strategies helps mitigate risks while enhancing the model's reliability. Below are key best practices to consider:
1
Setup observability
platform on early stages
of development:
mini paragraph
platform on early stages
of development:
mini paragraph
2
Integrate with current tools:Make sure
observability tools support the
frameworks and languages in your
environment, container platform,
messaging platform and any other
critical software.
observability tools support the
frameworks and languages in your
environment, container platform,
messaging platform and any other
critical software.
3
Setup observability
platform on early stages
of development:
mini paragraph
platform on early stages
of development:
mini paragraph
4
Integrate with current
tools:Make sure observability
tools support the frameworks
and languages in your
environment,
tools:Make sure observability
tools support the frameworks
and languages in your
environment,
Top features of a good LLM observability platform
Application Tracin
LLM applications use increasingly complex abstractions, such as chains, agents with tools, and advanced prompts. Traces capture the full context of the execution, including API calls, context, prompts, parallelism, and help to understand what is happening and identify the root cause of problems.
Metrics and Logs
Monitor the cost, latency and performance of the LLM application. Your observability platform should provide the relevant insights via dashboards, reports and queries in real time so teams can understand an issue, its impact and how to resolve it.
Monitor User Activity
Analyze the number of currently active users and the mean length of a session. Activity peaks or extended session durations can indicate potential security threats (e.g. DDoS attack, prompt injection, data extraction).
Token Usage
Jailbreak attempts often involve long and complex inputs that consume a significant number of tokens. Therefore, monitoring the amount of tokens used by messages is essential.
Prompts with Low-Resource Languages
AI Safeguards can be easily tricked by translating unsafe English inputs into low-resource languages. Establishing robust safety measures across multiple languages is crucial to detect harmful content in a broader range of inputs.
Automatic prompt tagging
Configure automatic tagging of user prompts and LLM’s outputs. This helps in categorizing and tracking inputs and responses, making it easier to identify unusual or risky interactions.
User Analytics and Clustering
Aggregate prompts, users and sessions to find abnormal interactions with the LLM application.
Alerts
Create custom alert mechanisms for potential security threats (e.g. activity peaks, long sessions, low-resource languages).
Prompt Injections and Jailbreaks Monitoring
Utilize rule-based filters to detect known attack structures and employ fine-tuned models to identify suspicious instructions within the input.
Harmful Output Moderation
Ensuring that the model’s responses are free from offensive, biased, or unsafe content is essential. Proactive monitoring helps protect the business from reputational damage and legal risks.
Have Questions?
If you're worried about the security of your AI applications, or
if sensitive data or personal information might be slipping into
your RAG system, we're here to help.
Maybe you've got an app in production and aren't sure how users are interacting with it — don’t sweat it, just reach out to us! Drop us a line, and we’ll sort things out together. We're always happy to chat and find the best solution for you.
Maybe you've got an app in production and aren't sure how users are interacting with it — don’t sweat it, just reach out to us! Drop us a line, and we’ll sort things out together. We're always happy to chat and find the best solution for you.
2024