Skip to main content

Analyzing Interaction Patterns

Learn how to use conversation data strategically to improve the chatbot, identify trends, and make data-driven decisions about cache management and content.

Why Analyze Conversations?​

Beyond Individual Reviews​

Individual conversation review helps you fix specific problems.

Pattern analysis helps you:

  • 📊 Identify systemic issues
  • 🎯 Prioritize improvements strategically
  • 📈 Measure impact of changes
  • 🔮 Predict user needs
  • 💡 Discover optimization opportunities

Example:

  • Individual: "This conversation has negative feedback"
  • Pattern: "15 conversations about HFIM 3000 have negative feedback this week - there's a systemic problem"

Types of Patterns to Look For​

1. Topic Patterns​

What to look for:

  • Questions about the same topic asked repeatedly
  • Topics with consistently negative feedback
  • Topics with long response times
  • New topics not yet cached

Why it matters: Identifies caching opportunities and content gaps

Example Analysis:

Week of 1/12/26:
- "HFIM 3000 prerequisites": Asked 8 times, 3 negative feedback
- "Internship opportunities": Asked 6 times, 2 positive feedback
- "Career paths": Asked 5 times, average response time 10s

Actions:
1. Create cache entry for "HFIM 3000 prerequisites" (high volume + negative feedback)
2. Convert best "Internship opportunities" conversation to cache
3. Cache "Career paths" to reduce response time

2. Temporal Patterns​

What to look for:

  • Seasonal spikes (application periods, start of semester)
  • Time-of-day patterns (evening study hours)
  • Day-of-week trends (weekdays vs. weekends)
  • Event-driven spikes (deadlines, registration)

Why it matters: Helps predict future demand and prepare cache entries in advance

Example Analysis:

Pattern: Week before spring semester start (Jan 5-12):
- 3x increase in "course schedule" questions
- 5x increase in "add/drop" questions
- 2x increase in "parking" questions

Action: Create cache entries for these topics BEFORE the next semester starts

3. Feedback Patterns​

What to look for:

  • Topics with high negative feedback rates
  • Questions that never get positive feedback
  • Correlation between response time and feedback
  • Feedback trends over time (improving or declining)

Why it matters: Measures chatbot quality and identifies problem areas

Example Analysis:

Pattern: "Financial aid" questions:
- 12 conversations in January
- 8 negative feedback (67% negative rate)
- Average rating: 👎
- Common complaint: "Outdated information"

Action: Update source documents, create new cache entries with current financial aid information

4. Performance Patterns​

What to look for:

  • Questions with consistently slow response times
  • Response time trends (getting faster or slower)
  • Cache hit rate by topic
  • Sources frequently used together

Why it matters: Optimizes performance and user experience

Example Analysis:

Pattern: Questions about "faculty" take 8-12 seconds
- "Who teaches HFIM 3000?": 9.5s average
- "Professor office hours": 8.2s average
- "Faculty contact information": 11.3s average

Action: Create cache entries for faculty-related questions
Impact: Reduce response time from 10s to < 1s (10x improvement)

5. User Journey Patterns​

What to look for:

  • Common sequences of questions
  • Follow-up patterns
  • Topic transitions
  • Session complexity (single question vs. multi-question sessions)

Why it matters: Understand user needs holistically, improve context handling

Example Analysis:

Common User Journey:
1. "What is HFIM?" (introductory question)
2. "How do I apply?" (action-oriented)
3. "What are the requirements?" (details)
4. "When is the deadline?" (specifics)

Action: Create cache entries for this entire sequence
Result: Faster, more consistent experience for prospective students

How to Analyze Conversations​

Step 1: Data Collection (15 minutes)​

Weekly Data Gathering:

  1. Go to Conversations Section

  2. Collect Metrics:

    • Total conversations this week
    • Positive feedback count
    • Negative feedback count
    • No feedback count
    • Average response time

  3. Export or Document:

    • Create a spreadsheet (or use notebook)
    • Record weekly metrics

Example Spreadsheet:

Week | Total | Positive | Negative | No Feedback | Avg Response Time
-----|-------|----------|----------|-------------|------------------
1/5  | 150   | 12       | 8        | 130         | 4.2s
1/12 | 180   | 15       | 5        | 160         | 3.8s
1/19 | 210   | 18       | 3        | 189         | 3.2s

Step 2: Topic Analysis (20 minutes)​

Identify Common Topics:

  1. Review Question Column

    • Scan 50-100 recent conversations
    • Note recurring keywords or themes

  2. Group by Topic:

    • Manually categorize questions
    • Example Categories:
      • Prerequisites
      • Internships
      • Faculty
      • Career Paths
      • Application Process
      • Course Descriptions

  3. Count Occurrences:

    • How many times each topic appears
    • Prioritize topics asked 5+ times

Example Topic Analysis:

Topic                    | Count | Negative Feedback | Avg Response Time
-------------------------|-------|-------------------|------------------
HFIM 3000 Prerequisites  | 8     | 3                 | 3.5s
Internship Opportunities | 6     | 2                 | 5.2s
Career Paths             | 5     | 0                 | 10.1s
Application Process      | 4     | 1                 | 4.8s
Faculty Office Hours     | 3     | 1                 | 7.3s

Insights:

  • "Career Paths" needs caching (slow, frequent)
  • "HFIM 3000 Prerequisites" needs content fix (high negative feedback)
  • "Internship Opportunities" is working well (positive feedback)

Step 3: Feedback Analysis (15 minutes)​

Review Feedback Distribution:

  1. Calculate Feedback Ratios:

    • Positive feedback %: (Positive / Total) × 100
    • Negative feedback %: (Negative / Total) × 100
    • No feedback %: (No Feedback / Total) × 100

  2. Identify Problem Topics:

    • Filter by Negative Feedback
    • Group by topic
    • Find topics with 30%+ negative rate

  3. Read Feedback Comments:

    • Note specific complaints
    • Identify patterns ("outdated", "incomplete", "confusing")

Example Feedback Analysis:

Topic: HFIM 3000 Prerequisites
- 8 total conversations
- 3 negative (37.5% negative rate) ⚠️ HIGH
- Comments:
  - "This doesn't answer my question"
  - "Outdated information"
  - "Missing HFIM 2200 requirement"

Action: Verify current prerequisites, create cache entry with correct info

Step 4: Performance Analysis (10 minutes)​

Review Response Times:

  1. Calculate Average Response Time

    • For all conversations
    • By topic
    • By cache hit/miss

  2. Identify Slow Queries

    • List questions with > 8s response time
    • Check if they're commonly asked

  3. Correlate with Cache Status

    • Cache hits: should be < 1s
    • Cache misses: 2-8s normal, > 8s needs optimization

Example Performance Analysis:

Slow Queries (> 8s):
1. "Career paths with HFIM degree" - 10.1s avg (5 occurrences)
2. "Faculty office hours all professors" - 9.8s avg (3 occurrences)
3. "HFIM course descriptions all courses" - 11.2s avg (2 occurrences)

Action: Cache these complex queries
Expected Impact: 10s → 0.5s (20x faster)

Step 5: Source Analysis (Optional, 10 minutes)​

Review Source Quality:

  1. Open 10-15 Random Conversations

    • Check Sources section in detail modal

  2. Look for Patterns:

    • Which documents are most frequently used?
    • Are sources relevant? (relevance > 0.80)
    • Are sources current? (not from 2-3 years ago)
    • Are sources complete? (not missing pages/sections)

  3. Identify Source Issues:

    • Outdated documents frequently used
    • Low-relevance sources causing bad answers
    • Missing documents for important topics

Example Source Analysis:

Most Used Documents:
1. HFIM_Handbook_2026.pdf - 45 conversations (good, current)
2. Course_Catalog_2023.pdf - 32 conversations (⚠️ outdated)
3. Faculty_Directory_2025.pdf - 28 conversations (acceptable)

Action: Update Course_Catalog_2023.pdf to 2026 version
Impact: Improve accuracy of course-related responses

Creating Action Items​

From Analysis to Action​

After analysis, create specific action items:

Template:

Issue: [What you found]
Impact: [How many users affected, severity]
Action: [What to do]
Priority: [High/Medium/Low]
Owner: [Who will do it]
Deadline: [When]
Success Metric: [How to measure improvement]

Example Action Items​

Action Item 1:

Issue: "HFIM 3000 prerequisites" questions have 37.5% negative feedback
Impact: 8 users in one week, high severity (incorrect info)
Action: Create cache entry with verified prerequisites from 2026 handbook
Priority: HIGH
Owner: Jane Smith
Deadline: 1/20/26
Success Metric: Negative feedback rate < 10% after 2 weeks

Action Item 2:

Issue: "Career paths" questions consistently take 10+ seconds
Impact: 5 users in one week, poor user experience
Action: Create cache entry for "What career paths are available with HFIM?"
Priority: MEDIUM
Owner: John Doe
Deadline: 1/25/26
Success Metric: Response time < 1s for career path questions

Action Item 3:

Issue: Course_Catalog_2023.pdf is outdated but frequently used
Impact: 32 conversations used outdated catalog
Action: Ingest Course_Catalog_2026.pdf, remove 2023 version
Priority: HIGH
Owner: IT Support
Deadline: 1/18/26
Success Metric: All course info pulls from 2026 catalog

Weekly Analysis Workflow​

30-Minute Weekly Routine​

Every Monday (or start of week):

15 Minutes: Data Collection

  1. Open Conversations section
  2. Filter by last 7 days (if available)
  3. Record total count, feedback distribution
  4. Note 5-10 most common topics

10 Minutes: Negative Feedback Deep Dive

  1. Filter by negative feedback
  2. Read all negative feedback comments
  3. Group by topic
  4. Identify root causes

5 Minutes: Action Planning

  1. Create 2-3 action items for the week
  2. Prioritize by impact (users affected × severity)
  3. Assign owners
  4. Set deadlines

Weekly Goal: Address all negative feedback issues within 7 days

Monthly Analysis Workflow​

2-Hour Monthly Deep Dive​

Once Per Month (first Friday, for example):

30 Minutes: Trend Analysis

  1. Compare weekly metrics for the month
  2. Identify improving vs. declining trends
  3. Correlate with events (semester start, deadlines)

30 Minutes: Topic Analysis

  1. List top 20 most asked questions
  2. Check which are cached vs. not cached
  3. Calculate cache hit rate by topic

30 Minutes: Performance Review

  1. Review average response times monthly trend
  2. Identify optimization opportunities
  3. Test new cache entries created last month

30 Minutes: Strategic Planning

  1. Document findings
  2. Create 5-10 action items for next month
  3. Share report with team

Monthly Goal: Improve cache hit rate by 5-10% each month

Advanced: Cohort Analysis​

Comparing User Groups​

If you can identify user types (undergrad, grad, prospective):

Analyze by Cohort:

Undergraduate Students:
- Most asked: Course prerequisites (60%)
- Average session: 2.3 questions
- Preferred topics: Courses, schedules, requirements

Prospective Students:
- Most asked: Program overview (40%)
- Average session: 3.5 questions
- Preferred topics: Admissions, requirements, career paths

Faculty/Staff:
- Most asked: Faculty info (50%)
- Average session: 1.2 questions
- Preferred topics: Faculty, course management

Action: Create targeted cache entries for each cohort

Measuring Impact​

Before-and-After Analysis​

After implementing improvements, measure impact:

Metrics to Track:

  1. Negative Feedback Rate

    • Before: 8 negative / 150 total = 5.3%
    • After: 3 negative / 180 total = 1.7%
    • Impact: 68% reduction in negative feedback ✅

  2. Average Response Time

    • Before: 4.2s average
    • After: 3.2s average
    • Impact: 24% faster responses ✅

  3. Cache Hit Rate

    • Before: 60%
    • After: 72%
    • Impact: 20% increase in cache hits ✅

  4. Times Served (New Cache Entry)

    • Week 1: 0 (just created)
    • Week 2: 8 (users are matching!)
    • Week 4: 25 (high adoption) ✅

Common Analysis Pitfalls​

Pitfall 1: Analysis Paralysis​

Problem: Spending too much time analyzing, not enough time acting

Solution: Set time limits (30 min weekly, 2 hours monthly), focus on actionable insights

Pitfall 2: Ignoring No-Feedback Conversations​

Problem: Only focusing on explicit positive/negative feedback

Solution: Spot-check "no feedback" conversations to catch silent issues

Pitfall 3: Not Following Up​

Problem: Creating action items but not tracking completion

Solution: Use project management tool (Trello, Asana) or simple spreadsheet

Pitfall 4: Over-Caching​

Problem: Caching every question that appears twice

Solution: Cache only questions asked 5+ times or with clear quality/performance benefits

Next Steps​

Now that you can analyze interaction patterns:

  1. Review cache best practices - Optimize your cache strategy
  2. Understand cache metrics - Deep dive into statistics
  3. Troubleshoot common issues - Solve problems efficiently

Remember: Data-driven decisions are better than guesswork. Spend 30 minutes weekly analyzing patterns, and you'll dramatically improve your chatbot's performance over time!

Last updated on