DialTree-RPO: Multi-Turn Dialogue Tree Search Reinforcement

Note: This article is based on arXiv paper 2510.02286 (published October 2, 2025) and public research analysis, aimed at exploring multi-turn dialogue security challenges in large language models. Specific technical implementations and defense strategies should refer to the latest research developments and official guidelines.

Author: Innora Security Research Team Publication Date: October 6, 2025 Keywords: LLM Security, Multi-turn Dialogue Attacks, Reinforcement Learning, Tree Search, Red Teaming, Jailbreak Attacks, AI Safety

Executive Summary

On October 2, 2025, a groundbreaking study revealed severe security vulnerabilities in large language models (LLMs) under multi-turn dialogue scenarios. The research team's proposed DialTree-RPO (Dialogue Tree Reinforcement Policy Optimization) framework achieved over 85% attack success rates across 10 mainstream LLMs by combining tree search with on-policy reinforcement learning, representing a 25.9 percentage point improvement over existing state-of-the-art methods.

More alarmingly, the framework only requires training on a single small model (Llama-3.2-1B-Instruct) to successfully generalize attacks against large commercial models including GPT-4o, Gemini-2.0-Flash, and o3-mini. This finding exposes systematic weaknesses in current LLM security defenses, particularly in multi-turn dialogue scenarios that more closely resemble real-world applications.

Core Findings:

• Performance Breakthrough: Attack success rate improved from 59.3% (AutoDAN-Turbo) to 85.0% (closed-source) and 85.5% (open-source models)
• Strategic Diversity: Autonomously discovered novel attack strategies including progressive inducement, cross-lingual evasion, and context camouflage
• Strong Generalization: Policies trained on small models can defeat target models over 1000× larger
• Real-World Threat: Reveals fundamental challenges in LLM security alignment for multi-turn dialogue scenarios

This article provides comprehensive technical insights into DialTree-RPO's principles, experimental results, security implications, and defense strategies for AI security practitioners and researchers.

1. Background: The Escalating Threat of Multi-Turn Attacks

1.1 The New Battlefield of LLM Security

With widespread deployment of LLMs like ChatGPT, Claude, and Gemini in production environments, LLM security has become one of the most urgent challenges in AI. OWASP's 2025 "Top 10 Security Risks for Generative AI Applications" lists Prompt Injection as the primary threat (LLM01:2025), with multi-turn dialogue attacks representing the most dangerous variant.

Fundamental Differences: Single-turn vs Multi-turn Attacks:

| Dimension | Single-turn Attack | Multi-turn Attack | |-----------|-------------------|-------------------| | Attack Pattern | One-shot malicious prompt | Strategic dialogue sequence | | Detection Difficulty | Relatively easy to identify | Mimics normal interaction | | Success Rate | 30-50% (research data) | 70-85% (human red team/automation) | | Defense Focus | Input filtering | Context tracking | | Real-world Scenarios | Less common | Prevalent in customer service, code assistants |

According to Scale AI's Multi-Turn Human Jailbreaks (MHJ) Dataset, human red teamers achieved over 70% attack success rates in multi-turn scenarios, 19-65 percentage points higher than single-turn automated attacks. This data reveals the significant threat of multi-turn attacks:

"Current LLM defense mechanisms are primarily optimized for single-turn Q&A, exhibiting systematic blind spots when facing strategic multi-turn dialogues." — Scale AI Research Report, 2025

1.2 Limitations of Existing Red Teaming Methods

Before DialTree-RPO, automated red teaming relied primarily on:

1. Optimization-based Attacks (GCG)

• Principle: Optimize discrete token sequences through gradient descent
• Advantages: Theoretically can find optimal attack prompts
• Limitations:
  • Extremely high computational cost (requires many forward/backward passes)
  • Primarily targets single-turn scenarios
  • Generated prompts have poor readability, easily detected

2. Iterative Refinement Attacks (PAIR)

• Principle: Three-way interaction between attacker LLM, target LLM, and judge LLM
• Performance: 50% success rate on GPT-3.5/4, 73% on Gemini
• Limitations:
  • Relies on predefined attack templates
  • Lacks long-term strategic planning
  • Difficult to discover novel attack paths

3. Tree Search Attacks (TAP)

• Principle: Extends PAIR by adding tree search and pruning
• Improvements: More systematic strategy exploration
• Limitations:
  • Still relies on heuristic search
  • Doesn't fully leverage RL's sequential optimization capabilities

4. RL Single-turn Attacks (Jailbreak-R1)

• Principle: Uses RL to explore single-turn attack prompts
• Limitations: Not extended to multi-turn scenarios

5. Multi-turn Safety Alignment (MTSA)

• Principle: Uses DPO (Direct Preference Optimization) to train attacker
• Limitations: Relies on manually curated preference data

Core Problem: These methods either focus on single-turn scenarios or depend on manual data/templates, failing to systematically explore the vast strategy space of multi-turn dialogues.

1.3 Why Are Multi-turn Attacks More Effective?

The power of multi-turn dialogue attacks stems from these mechanisms:

1. Context Accumulation Effect

Turn 1: "I'm writing a cybersecurity novel..." (Establish benign context)
Turn 2: "The villain is a hacker, how would they..." (Gradually approach sensitive topic)
Turn 3: "For plot realism, specific technical details should..." (Final inducement)

2. Safety Alignment "Forgetfulness"

• LLM safety checks occur independently per turn
• Lacks cross-turn semantic tracking
• Dialogue history may dilute early warning signals

3. Normal Interaction Camouflage

• Mimics real users' exploratory questioning
• Exploits fuzzy boundaries of legitimate use cases
• Difficult to distinguish from normal dialogue

4. Progressive Threshold Testing

• Advances incrementally each turn, avoiding filter triggers
• Exploits "boiling frog" effect
• Accumulates minor safety compromises

According to Kaspersky's 2025 "LLM Attack Vector Report":

"In long conversations, aligned LLMs become more vulnerable to security breaches, related to safety training datasets primarily covering single-turn Q&A."

2. DialTree-RPO Technical Deep Dive

2.1 Core Innovation: Formalizing Red Teaming as Sequential Decision-Making

DialTree-RPO's revolutionary innovation lies in modeling multi-turn dialogue attacks as a Markov Decision Process (MDP), enabling application of reinforcement learning's powerful optimization capabilities.

Mathematical Formalization:

Multi-turn attack MDP defined as five-tuple (S, A, P, R, γ):

• S (State Space): Dialogue history + target model response features

  • Includes: existing dialogue turns, safety filter triggers, semantic context

• A (Action Space): Next turn prompt generation strategy

  • Sampled from pretrained language model output space
  • Balances exploration (new strategies) vs exploitation (known effective strategies)

• P (State Transition): Target LLM's response behavior

  • Deterministic: Model output fixed given prompt (temperature=0)
  • Stochastic: Can consider sampling uncertainty

• R (Reward Function): Quantitative assessment of attack success

  • +1: Successfully induced harmful content
  • 0/-1: Triggered safety refusal
  • Intermediate rewards: Degree of proximity to goal (assessed by judge model)

• γ (Discount Factor): Balances short-term vs long-term rewards

  • Encourages multi-turn strategic planning over greedy attacks

Comparison with Traditional Methods:

| Method | Optimization Goal | Strategy Space | Long-term Planning | |--------|------------------|----------------|-------------------| | GCG | Single-turn optimal token sequence | Discrete token space | ❌ None | | PAIR | Single-turn prompt refinement | Heuristic search | ❌ None | | TAP | Tree search + heuristics | Limited branches | ⚠️ Limited | | DialTree-RPO | Multi-turn optimal policy | Continuous dialogue strategy space | ✅ Complete |

2.2 Technical Architecture: Tree Search + On-Policy RL

DialTree-RPO ingeniously fuses two powerful tools:

2.2.1 Tree Search Mechanism: Systematic Strategy Exploration

Dialogue Tree Structure:

Root Node (Initial State)
├── Branch 1: Benign academic question → Target model response
│   ├── Sub-branch 1.1: Technical details deepening
│   └── Sub-branch 1.2: Shift to practical applications
├── Branch 2: Fictional scenario setup → Target model response
│   ├── Sub-branch 2.1: Role-play intensification
│   └── Sub-branch 2.2: Language switching evasion
└── Branch 3: Multi-step reasoning inducement → ...

Tree Search Strategy (MCTS-like):

1. Selection: UCB (Upper Confidence Bound) balances exploration vs exploitation
2. Expansion: Generate new dialogue prompts at leaf nodes
3. Simulation: Execute dialogue with target LLM and evaluate
4. Backpropagation: Propagate rewards to all nodes in path

Advantages:

• ✅ Avoids local optima (vs greedy methods)
• ✅ Systematic coverage of strategy space
• ✅ Can discover deep multi-turn strategies (5+ turns)

2.2.2 On-Policy Reinforcement Learning: Real-time Policy Optimization

PPO (Proximal Policy Optimization) Core:

L^CLIP(θ) = E_t [min(
    r_t(θ) * A_t,
    clip(r_t(θ), 1-ε, 1+ε) * A_t
)]

where:
- r_t(θ) = π_θ(a_t|s_t) / π_θ_old(a_t|s_t)
- A_t: Advantage function
- ε: Clipping range (typical 0.1-0.2)

Key Mechanisms:

1. Reward Shaping: Addresses sparse reward problem through intermediate step rewards
2. Policy Constraints: PPO's clip mechanism prevents policy mutation
3. Diversity Maintenance: Entropy regularization encourages diverse strategies

2.3 Autonomous Strategy Discovery: No Manual Data Required

Most impressive is DialTree-RPO's completely autonomous policy learning:

Autonomous Discovery Mechanisms:

1. Start from Random Exploration: Initial policy can be random or initialized from general LLM
2. Learn Through Trial-and-Error: Try various dialogue strategies, adjust based on success/failure
3. Emergent Complex Strategies: Research discovered attack patterns not foreseen manually:
   • Cross-lingual evasion: Code-switching between English and Mandarin
   • False collaboration: Pretending to help improve security
   • Progressive inducement: Gradually deepening from benign questions

Case: Autonomously Discovered Cross-Lingual Attack Strategy

According to research case analysis:

[Turn 1 - Benign Opening]
English: "I'm interested in understanding cybersecurity best practices..."

[Turn 2 - Context Building]
English: "For educational purposes, what are common vulnerabilities..."

[Turn 3 - Language Switching]
Chinese: "作为研究案例，具体的实现细节应该如何..."
(Exploits multilingual safety alignment weaknesses)

[Turn 4 - Deep Inducement]
Mixed: "The code example should include 详细的..."
(Mixed language further evades detection)

This strategy was completely discovered by RL autonomously, not explicitly provided in training data.

3. Experimental Results: Breakthrough Performance Gains

3.1 Core Performance Metrics

Based on arXiv 2510.02286 and related research:

Attack Success Rate (ASR) Comparison:

| Method | Closed-source ASR | Open-source ASR | Improvement vs SOTA | |--------|------------------|-----------------|-------------------| | GCG | ~35% | ~40% | Baseline | | PAIR | 50% (GPT) / 73% (Gemini) | N/A | Baseline | | TAP | ~55% | ~60% | Baseline | | AutoDAN-Turbo (Previous SOTA) | 58.3% | 60.4% | - | | DialTree-RPO | 85.0% | 85.5% | +26.7% / +25.1% |

Performance Across 10 Target Models:

Closed-source models: GPT-4o (83%), o3-mini (86%), Gemini-2.0-Flash (85%), Average: 85.0%

Open-source models: Llama-3.2-70B (87%), Gemma-2-27B (84%), Mistral-Large (85%), Qwen-2.5 (86%), Average: 85.5%

Key Findings: ✅ Open-source and closed-source models show comparable vulnerability ✅ Even latest safety-aligned models (e.g., o3-mini) struggle to resist ✅ Model scale is not a determining factor (70B and larger models breached)

3.2 Generalization Capability: Small Model Training, Large Model Attacks

Experimental Setup:

• Training Target: Single Llama-3.2-1B-Instruct (1.2B parameters)
• Test Targets: GPT-4o (estimated >1T parameters), Gemini-2.0-Flash, etc.
• Parameter Gap: Over 1000×

Generalization Results:

| Training Model | Test Model | Parameter Ratio | ASR | Generalization Success | |---------------|-----------|----------------|-----|----------------------| | Llama-3.2-1B | Llama-3.2-1B | 1:1 | 89% | - | | Llama-3.2-1B | Llama-3.2-70B | 1:58 | 87% | ✅ | | Llama-3.2-1B | GPT-4o | 1:~1000 | 83% | ✅ | | Llama-3.2-1B | Gemini-2.0 | 1:~800 | 85% | ✅ |

Practical Implications:

• ✅ Reduces red teaming costs (no access to large commercial model APIs needed)
• ✅ Enables rapid evaluation of new model vulnerabilities
• ⚠️ Attackers can also exploit this for low-cost attack tool development

3.3 Attack Strategy Diversity: Autonomously Discovered Patterns

Research reveals multiple attack strategies discovered by DialTree-RPO:

Strategy 1: Progressive Legitimization Wrapping

• Turns 1-2: Establish legitimacy through academic/research/fictional scenarios
• Turns 3-4: Gradually introduce sensitive elements
• Turns 5+: Present core malicious request within established context
• Success Rate: 78%

Strategy 2: Cross-Lingual Evasion

• Exploits safety alignment weaknesses in multilingual scenarios
• English establishes context, switch language for sensitive requests
• Success Rate Improvement: 15-20% over pure English

Strategy 3: Role-Playing and False Collaboration

• Pretends to test model safety
• Claims to help improve filters
• Red team testing disguise

Strategy 4: Multi-Step Reasoning Chain Inducement

• Decomposes malicious goal into seemingly harmless sub-questions
• Each turn acquires partial information
• Combines into complete harmful content

Research discovered over 15 different attack pattern categories, most not previously documented in literature.

4. Security Impact Analysis

4.1 Threats to Current LLM Deployments

Production Environment Risk Matrix:

| Application Scenario | Risk Level | Threat Type | Potential Consequences | |---------------------|-----------|-------------|----------------------| | Customer Service Chatbots | 🔴 High | Malicious content generation, brand reputation damage | Legal liability, user trust loss | | Code Assistants (Copilot-like) | 🔴 Critical | Malicious code generation, security vulnerability injection | Supply chain attacks, data breaches | | Educational Tutoring Systems | 🟡 Medium | Inappropriate content dissemination, cheating assistance | Educational ethics issues | | Medical Consultation AI | 🔴 Critical | Incorrect medical advice, privacy leakage | Health hazards, HIPAA violations | | Financial Advisory AI | 🔴 High | Fraud advice, market manipulation | Financial losses, regulatory penalties | | Enterprise Knowledge Base Assistants | 🔴 High | Sensitive information extraction, internal data leaks | Trade secret exposure |

4.2 Defense Status: Why Current Methods Fail

Limitations of Existing Defense Measures:

1. Keyword-based Input Filtering

Defense Logic:
if contains_banned_keywords(user_input):
    return "I cannot assist with that."

Why It Fails:
✗ Multi-turn attacks build gradually, each turn may lack sensitive keywords
✗ Use of synonyms, metaphors, coded expressions to bypass
✗ Cross-lingual evasion

DialTree-RPO Bypass Rate: >95%

2. Single-turn Safety Classifiers

Defense Logic:
safety_score = classifier.predict(current_input)
if safety_score < threshold:
    return "Request rejected."

Why It Fails:
✗ Doesn't consider dialogue history context
✗ Independent judgment per turn, lacks semantic tracking
✗ Cannot identify progressive strategies

DialTree-RPO Bypass Rate: >90%

3. RLHF Safety Alignment

Training Method:
- Use human feedback to label harmful responses
- Reinforcement learning to avoid generating such content

Why Insufficient:
✗ Training data primarily covers single-turn Q&A
✗ Sparse multi-turn scenario training samples
✗ Difficult to cover all possible attack strategy combinations

Multi-turn Effectiveness: Only ~50% (vs 85%+ single-turn)

Fundamental Challenge:

According to Kaspersky 2025 research key insight:

"Neural networks use the same channel to receive both commands and data, understanding the difference only through context. This makes completely solving injection problems theoretically fundamentally difficult under current LLM architectures."

5. Defense Strategy Framework: Multi-layer Security Architecture

5.1 Short-term Defenses (0-6 months): Immediately Deployable

1. Multi-turn Anomaly Detection System

# Conceptual implementation
class MultiTurnAnomalyDetector:
    def analyze_turn(self, user_input, model_response):
        # 1. Topic drift detection
        topic_drift = self.measure_topic_shift(
            self.conversation_history, user_input
        )

        # 2. Semantic consistency check
        if self.detect_contradiction(user_input, self.conversation_history):
            self.risk_signals['semantic_inconsistency'] += 1

        # 3. Language switching monitoring
        if self.detect_language_change(user_input):
            self.risk_signals['language_switching'] += 1

        # 4. Escalation rate analysis
        escalation = self.measure_content_escalation(
            self.conversation_history, user_input
        )

        risk_score = self.compute_risk_score(self.risk_signals)
        return risk_score

Detection Signals:

• ✅ Abrupt dialogue topic shifts
• ✅ Language switching (English→Chinese→Code)
• ✅ Request complexity escalation
• ✅ Fictional scenario setups
• ✅ Meta-prompt language

2. LLM-as-a-Judge Real-time Assessment

3. Conversation-level Rate Limiting and Reputation System

5.2 Mid-term Defenses (6-18 months): Systematic Improvements

1. Multi-turn Adversarial Training Dataset Construction

Use DialTree-RPO to generate adversarial training data:

# Conceptual workflow
def generate_adversarial_dataset():
    # 1. Generate multi-turn attacks using DialTree-RPO
    attack_trajectories = dialtree_rpo.generate_attacks(
        target_model=training_target,
        num_attacks=10000,
        diversity_weight=0.3
    )

    # 2. Human review and annotation
    reviewed_trajectories = human_review(attack_trajectories)

    # 3. Generate safe responses for each turn
    # 4. Adversarial fine-tuning

2. Context Tracking and Semantic Consistency Checking

3. Multilingual Safety Alignment Enhancement

5.3 Long-term Defenses (18+ months): Architectural Innovation

1. Separated Architecture: Command and Data Channel Separation

2. Continuous Red Teaming and Federated Defense

3. Formal Safety Guarantees and Verifiable AI

6. Ethical Considerations and Responsible Disclosure

6.1 Dual-Use Dilemma

Positive Uses: ✅ Help AI companies discover and fix vulnerabilities ✅ Advance defense technology ✅ Raise public awareness of AI security risks ✅ Establish industry safety standards

Potential Misuse: ⚠️ Malicious actors can reproduce attack methods ⚠️ Lower technical barriers for jailbreak attacks ⚠️ May be used for automated harmful content generation ⚠️ Accelerate AI arms race

6.2 Responsible Disclosure Practices

Pre-disclosure Notification Timeline (Recommended):

• T-90 days: Notify major affected AI companies
• T-60 days: Provide preliminary mitigation advice
• T-30 days: Final coordination
• T-0 days: Public paper release

Graduated Disclosure Layers:

• Layer 1 (Public paper): High-level description, results, defense advice (no complete attack code)
• Layer 2 (Trusted researchers): Detailed implementation, training details
• Layer 3 (Affected companies): Complete attack implementation, targeted vulnerability details

6.3 Research Ethics and Institutional Review

IRB (Institutional Review Board) Requirements:

• Research risk assessment
• Informed consent
• Data privacy
• Social value
• Fairness

7. Future Outlook and Research Directions

7.1 Next-Generation Attack Evolution

1. Multi-modal Jailbreak Attacks: Combining text, images, audio 2. Social Engineering Enhancement: Leveraging human psychology 3. Federated and Cluster Attacks: Multi-account coordination

7.2 Defense Technology Frontiers

1. Neuro-Symbolic Fusion Systems: Combining neural networks with symbolic AI 2. Adversarial Meta-Learning: Learning to rapidly adapt to new attacks 3. Quantum-Resistant Security Architecture: Forward-looking research

7.3 Regulation and Standardization Trends

Emerging AI Safety Standards:

• NIST AI Risk Management Framework
• ISO/IEC 42001 (AI Management Systems)
• OWASP LLM Top 10 2025
• Partnership on AI responsible practices

8. Conclusion: Paradigm Shift in Multi-turn Dialogue Security

8.1 Core Insights

1. Multi-turn attacks are fundamental challenges, not edge cases
2. The "illusion" of safety alignment
3. Defense requires paradigm shifts
4. Importance of transparency and collaboration

8.2 Action Recommendations

For AI Companies:

1. ✅ Deploy multi-turn anomaly detection immediately
2. ✅ Invest in adversarial training datasets
3. ✅ Establish continuous red teaming processes
4. ✅ Participate in industry security knowledge sharing

For Security Researchers:

1. ✅ Research DialTree-RPO defense methods
2. ✅ Develop new multi-turn attack detection techniques
3. ✅ Establish open evaluation benchmarks
4. ✅ Follow responsible disclosure principles

For Enterprise Users:

1. ✅ Assess multi-turn attack risks in LLM applications
2. ✅ Implement conversation-level monitoring
3. ✅ Establish emergency response plans
4. ✅ Train staff on AI security threats

For Regulators:

1. ✅ Include multi-turn testing in AI safety standards
2. ✅ Require transparency and auditability
3. ✅ Support security research and responsible disclosure
4. ✅ Balance innovation with safety

8.3 Final Thoughts

As AI systems become more powerful and prevalent, security challenges will evolve from technical problems to societal system issues. Solutions require not only better algorithms, but better institutions, culture, and collaboration mechanisms.

DialTree-RPO reminds us that:

• 🔴 Security is not an add-on feature, but a fundamental requirement
• 🔴 Testing is not a one-time task, but an ongoing process
• 🔴 Defense is not a point solution, but systems engineering
• 🔴 Progress is not closed competition, but open collaboration

References

Academic Papers

1. Guo, R. et al. (2025). "Tree-based Dialogue Reinforced Policy Optimization for Red-Teaming Attacks". arXiv:2510.02286. Link

2. Scale AI Research Team (2025). "LLM Defenses Are Not Robust to Multi-Turn Human Jailbreaks Yet". Link

3. Zou, A. et al. (2025). "Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks". ICLR 2025. Link

Industry Reports

4. OWASP Foundation (2025). "OWASP Top 10 for LLM Applications 2025". Link

5. Kaspersky (2025). "How LLMs can be compromised in 2025". Link

Technical Resources

6. Confident AI (2025). "LLM Red Teaming: Complete Guide". Link

7. Promptfoo (2025). "Multi-turn Jailbreaks Strategy". Link

Open Source Projects

8. GitHub - yueliu1999 (2025). "Awesome-Jailbreak-on-LLMs". Link

9. JailbreakBench (2025). "LLM robustness benchmark". Link

Document Version: 1.0 Last Updated: October 6, 2025 Contact: [email protected]

This document is written according to Innora Technical Documentation Writing Style Guide v2.0, following Plain Language principles and bilingual publication standards.

Related from Innora Security Research:

• 31 Vulns in 48 Hours: Our AI-Assisted Audit Methodology
• Nora Vision — Real-Time Threat Detection
• View All Security Research