Skip to main content

Conversations Overview

Learn how to review user interactions with the HFIM chatbot, analyze feedback, and use conversation data to improve the chatbot's performance.

What is the Conversations Section?​

The Conversations section lets you review every interaction between users and the chatbot. You can see:

  • 📝 Questions users asked
  • 💬 Responses the chatbot generated
  • 👍👎 User feedback (positive, negative, or none)
  • ⏱️ Response times
  • 📚 Sources used in responses
  • 🆔 Session information

Why Review Conversations?​

Reviewing conversations helps you:

1. Identify Problems Quickly​

Spot issues before they escalate:

  • ❌ Incorrect answers
  • ❌ Unhelpful responses
  • ❌ Missing information
  • ❌ Technical errors

Example: If multiple users give negative feedback about the same topic, you know there's a problem that needs fixing.

2. Improve Cache Entries​

Find opportunities to enhance the cache:

  • 💡 Common questions that aren't cached
  • 💡 Variations users actually type
  • 💡 Topics with negative feedback
  • 💡 High-quality answers worth caching

Impact: One good conversation can become a cache entry that helps hundreds of future users.

3. Understand User Needs​

Learn what users care about:

  • What questions do they ask most?
  • How do they phrase questions?
  • What information are they seeking?
  • What time of day/year do they ask?

Result: Data-driven decisions about what to cache and how to improve the chatbot.

4. Measure Chatbot Performance​

Track quality metrics:

  • Response time trends
  • Feedback ratios (positive vs. negative)
  • Source accuracy
  • Common failure patterns

Goal: Continuously improve the user experience.

Conversations Table​

The main view shows a table with all conversations:

Table Columns​

ColumnWhat It ShowsExample
IDUnique conversation number21
SessionSession ID (groups related questions)abc123...
QuestionWhat the user asked"specific"
FeedbackUser rating (👍 positive, 👎 negative, — none)👎 with comment
Response TimeHow long the response took (milliseconds)14716ms (14.7 seconds)
TimestampWhen the conversation occurredJan 11, 2026
ActionsView details or edit feedbackView, Edit buttons
Feedback Icons
  • 👍 Positive - User found the answer helpful
  • 👎 Negative - User was unsatisfied (may include comment)
  • — No feedback - User didn't rate the response

Viewing Conversation Details​

Opening the Detail Modal​

To see full conversation details:

  1. Locate the conversation in the table
  2. Click the "View" button (👁️ icon)
  3. A modal window opens with complete information

What's in the Detail Modal​

Session Information:

  • Session ID (identifies related questions from same user)
  • Conversation ID

Question:

  • Exact text the user typed
  • Shown as it was received (no processing)

Answer:

  • Full response the chatbot generated
  • Formatting preserved (markdown, bullets, etc.)

Feedback:

  • Rating (Positive/Negative/None)
  • User comment (if provided)

Performance Data:

  • Response time in milliseconds
  • Timestamp (date and time)

Sources:

  • JSON array of documents used to generate the response
  • Shows which files and pages were referenced

Understanding Feedback​

Types of Feedback​

Positive Feedback (👍)​

What it means: User found the answer helpful

What to do:

  • ✅ Review periodically to confirm quality remains high
  • ✅ Consider converting excellent answers to cache entries
  • ✅ Use as examples when creating new cache entries

Don't: Spend too much time here—focus on negative feedback

Negative Feedback (👎)​

What it means: User was dissatisfied with the response

May include: User comment explaining why

Common reasons:

  • Answer was incorrect or outdated
  • Response didn't address the question
  • Information was incomplete
  • Answer was too vague or too detailed
  • Technical error or formatting issue

What to do:

  • 🔴 HIGH PRIORITY - Review immediately
  • 🔍 Investigate the issue
  • 🛠️ Fix cache entries or system prompts
  • 📝 Create new cache entries if needed

Goal: Zero negative feedback by addressing issues proactively

No Feedback (—)​

What it means: User didn't rate the response

Why users don't give feedback:

  • They found the answer acceptable (not great, not bad)
  • They left before rating
  • They didn't notice the feedback buttons

What to do:

  • ⚠️ Lower priority than negative feedback
  • 📊 Use aggregate data to spot patterns
  • 🔍 Spot-check occasionally

Feedback Comments​

Users can add optional comments with negative feedback:

Example Comments:

  • "This doesn't answer my question"
  • "Information is outdated"
  • "Too vague, need more details"
  • "Wrong course listed"

Value: Comments provide specific, actionable feedback

Action: Always read and act on comments—they tell you exactly what to fix

Understanding Response Time​

Response Time = How long it took the chatbot to generate and return the answer

Interpreting Response Times​

Response TimePerformanceLikely Cause
50-500ms🔥 ExcellentCache hit (instant)
2,000-5,000ms✅ GoodRAG search + generation (normal)
5,000-10,000ms⚠️ AcceptableComplex query or multiple sources
10,000ms+❌ SlowPotential issue or very complex query

What Affects Response Time​

Fast responses (cache hits):

  • Question matched a cache entry
  • No document search needed
  • Instant retrieval

Medium responses (RAG):

  • Document search (vector search)
  • Source retrieval
  • AI generation
  • Normal processing

Slow responses:

  • Very long or complex questions
  • Multiple follow-up searches
  • System load or network latency
  • Potential backend issues

Taking Action​

If you see consistently slow responses (> 10 seconds):

  1. Check if specific questions are always slow
  2. Consider caching those questions
  3. Contact support if it's a systemic issue

Understanding Sources​

The Sources field shows which documents the chatbot used to generate its response.

Source Format​

[
  {
    "filename": "HFIM_Handbook_2026.pdf",
    "page": 12,
    "section": "Admission Requirements",
    "relevance": 0.95
  },
  {
    "filename": "Course_Catalog.pdf",
    "page": 34,
    "section": "HFIM 3000",
    "relevance": 0.88
  }
]

Why Sources Matter​

Transparency: Users can verify information Accuracy: Shows if chatbot used reliable sources Debugging: Helps identify wrong sources causing bad answers

Reviewing Sources​

Check for:

  • ✅ Relevant documents (related to question)
  • ✅ Current documents (not outdated)
  • ✅ Correct page numbers
  • ❌ Irrelevant sources (may cause wrong answers)
  • ❌ Old/superseded documents

Action: If sources are wrong, the answer is likely wrong too. Create a cache entry with correct information.

Common Conversation Patterns​

Pattern 1: High Negative Feedback on Specific Topic​

Observation: Multiple conversations about "HFIM 3000 prerequisites" have negative feedback

Interpretation: Current answer (cache or RAG) is inadequate

Action:

  1. Review the topic in detail
  2. Verify correct information
  3. Create or update cache entry
  4. Monitor for improvement

Pattern 2: Same Question Asked Repeatedly​

Observation: 15 conversations ask variations of "What is HFIM?"

Interpretation: Common question, high-value caching opportunity

Action:

  1. Find the best answer from these conversations
  2. Create cache entry
  3. Add variations matching how users ask
  4. Monitor "Times Served" metric

Pattern 3: Long Response Times​

Observation: Questions about "career paths" consistently take 8-12 seconds

Interpretation: Complex query requiring extensive document search

Action:

  1. Create cache entry for common career-related questions
  2. Provide comprehensive answer upfront
  3. Reduce response time from 10s to 0.5s

Pattern 4: Positive Feedback on Specific Answer​

Observation: Conversation #45 has positive feedback and excellent response

Interpretation: High-quality answer worth preserving

Action:

  1. View conversation details
  2. Click "Convert to Cache"
  3. Save as cache entry
  4. Help future users get this great answer instantly

Typical Workflow​

Weekly Conversation Review (15-30 minutes)​

Step 1: Filter by Negative Feedback (5 minutes)

  • Identify all conversations with 👎 ratings
  • Read user comments
  • Note common issues

Step 2: Investigate Issues (10 minutes)

  • Review problematic answers
  • Identify root causes (wrong info, outdated data, missing context)
  • Decide on fixes

Step 3: Take Action (10 minutes)

  • Update cache entries
  • Create new cache entries
  • Note system issues for support
  • Document findings

Step 4: Monitor (ongoing)

  • Check if negative feedback decreases
  • Verify fixes worked
  • Adjust as needed

What to Look For​

Red Flags 🚨​

Immediate attention needed:

  • Multiple negative feedbacks on same topic
  • Responses with no sources
  • Consistently slow response times (> 15 seconds)
  • Error messages in responses
  • Blank or cut-off responses

Yellow Flags ⚠️​

Investigate when time allows:

  • Questions asked repeatedly (5+ times)
  • Mixed feedback on same topic
  • Responses from outdated sources
  • Questions that seem relevant but have no answer

Green Flags ✅​

Good signs (but still monitor):

  • Positive feedback
  • Fast response times
  • Diverse questions with good answers
  • Appropriate sources cited

Privacy and Data Handling​

What Data is Collected​

Stored:

  • ✅ Questions (anonymous)
  • ✅ Responses
  • ✅ Feedback ratings
  • ✅ Response times
  • ✅ Timestamps
  • ✅ Session IDs (for context)

NOT stored:

  • ❌ User names or personal info (beyond session ID)
  • ❌ User IP addresses
  • ❌ Identifying information

Purpose: Improve chatbot quality, not track individual users

Data Retention

Conversations may be retained for analysis purposes. Check with your institution's data retention policies for specific timelines.

Next Steps​

Now that you understand conversations:

  1. Learn to view conversations - Navigate the interface
  2. Filter by feedback - Find specific conversations
  3. Edit incorrect feedback - Fix mistakes
  4. Convert good conversations to cache - Preserve excellent answers
  5. Analyze interaction patterns - Use data strategically
  6. Troubleshoot issues - Solve common problems

Remember: Conversations are a goldmine of insights. Spend 15-30 minutes weekly reviewing them, and you'll dramatically improve your chatbot's performance!

Last updated on