Performance Metrics
Learn how to interpret cache performance metrics and use them to improve your chatbot's effectiveness.
Why Metrics Matter
Performance metrics help you:
- 📊 Measure cache effectiveness - Is your cache working well?
- 🎯 Identify improvements - Which entries need attention?
- 🚀 Track progress - Are your changes making a difference?
- 💡 Make data-driven decisions - What should you cache next?
Rule of thumb: Check metrics weekly to stay on top of cache performance.
Dashboard Metrics
The Dashboard provides high-level cache performance statistics.
1. Total Cache Entries
What it means: Total number of cache entries (active + inactive)
Displayed: Count with icon (e.g., "9 Cache Entries")
Interpretation:
| Count | Status | Recommendation |
|---|---|---|
| 0-10 | Small cache | Start caching frequently asked questions |
| 11-50 | Growing cache | Good foundation, continue expanding |
| 51-100 | Mature cache | Focus on quality and maintenance |
| 100+ | Large cache | Prioritize high-impact entries, prune unused ones |
Action items:
- Too few: Review Popular Questions and create more entries
- Too many: Audit for unused or redundant entries
2. Cache Hit Rate
What it means: Percentage of user questions answered using cached responses
Formula: (Cache hits) ÷ (Total questions) × 100%
Displayed: Percentage with icon (e.g., "85.7% Hit Rate")
Interpretation:
| Hit Rate | Performance | Interpretation |
|---|---|---|
| 80%+ | ✅ Excellent | Cache is comprehensive and well-maintained |
| 60-79% | ✅ Good | Solid coverage, room for improvement |
| 40-59% | ⚠️ Fair | Many questions bypass cache |
| Below 40% | ❌ Poor | Cache needs significant expansion |
What affects hit rate:
- ✅ Number of cache entries
- ✅ Quality of question variations
- ✅ Match between cache topics and user questions
- ✅ Confidence scores (low confidence = fewer matches)
Action items:
- Low hit rate: Add more cache entries for popular topics
- High hit rate: Maintain quality, focus on niche topics
A well-maintained chatbot cache typically achieves 65-75% hit rate. Rates above 80% are exceptional!
3. Success Rate
What it means: Percentage of cache entries that successfully matched at least one user question
Formula: (Entries with Times Served > 0) ÷ (Total active entries) × 100%
Displayed: Percentage with icon (e.g., "0% Success Rate")
Interpretation:
| Success Rate | Performance | Interpretation |
|---|---|---|
| 80%+ | ✅ Excellent | Most cache entries are useful |
| 60-79% | ✅ Good | Majority of entries are matching |
| 40-59% | ⚠️ Fair | Many entries aren't matching |
| Below 40% | ❌ Poor | Too many unused entries |
Why success rate might be 0%:
- ⏰ New cache: Entries just created, haven't had time to match
- 🎯 Poor variations: Questions don't match how users ask
- 📉 Low traffic: Few users asking questions
- ❌ Inactive entries: All entries are disabled
Action items:
- 0% initially: Normal for new cache, wait 1-2 weeks
- 0% after 2+ weeks: Generate more variations, check if entries are active
- Low success rate: Review unused entries (Times Served = 0) and improve variations
4. Total Variations
What it means: Total count of all question variations across all cache entries
Displayed: Count with icon (e.g., "0 Variations")
Interpretation:
| Variations per Entry | Coverage | Recommendation |
|---|---|---|
| 0-2 | ❌ Poor | Generate variations immediately |
| 3-5 | ⚠️ Minimal | Add more variations |
| 6-10 | ✅ Good | Solid coverage |
| 11-15 | ✅ Excellent | Comprehensive coverage |
| 15+ | ⚠️ Excessive | May cause false matches |
Calculating average: Total Variations ÷ Total Cache Entries
Example: 90 variations ÷ 10 entries = 9 variations per entry (Good!)
Action items:
- 0 variations: Use bulk variation generator immediately
- Low average: Generate more variations for underperforming entries
Entry-Level Metrics
Each cache entry has individual performance metrics visible in the cache list.
1. Times Served
What it means: How many times this specific entry was returned to users
Location: Cache Management table, column "Times Served"
Interpretation:
| Times Served | Status | Action |
|---|---|---|
| 100+ | 🔥 High-value | Monitor closely, keep updated |
| 50-99 | ✅ Valuable | Maintain accuracy |
| 10-49 | ⚠️ Moderate | Consider adding variations |
| 1-9 | 📉 Low usage | Review variations and relevance |
| 0 | ❌ Unused | Improve variations or deactivate |
High-value entries (50+ times served):
- Deserve extra attention
- Small improvements have big impact
- Review monthly for accuracy
Unused entries (0 times served):
- Not matching user questions
- Poor variations or irrelevant topic
- Consider improving or removing
Action items:
- Sort by Times Served to find highest-value entries
- Focus maintenance on top 20% of entries
- Improve or remove entries with 0 after 1+ month
2. Last Served
What it means: Timestamp of when this entry was most recently used
Location: Cache Management table, column "Last Served"
Interpretation:
| Last Served | Status | Action |
|---|---|---|
| Today/Yesterday | 🔥 Active | Currently being used |
| This week | ✅ Recent | Regularly used |
| This month | ⚠️ Occasional | Check if still relevant |
| 1+ months ago | ❌ Stale | Review for relevance |
| Never | ❌ Unused | Same as "Times Served = 0" |
Why "Never"?
- Entry is new (just created)
- Variations don't match user questions
- Topic isn't relevant to users
- Status is "Inactive"
Action items:
- Old "Last Served": Verify information is still accurate
- Never served: Improve variations or remove if irrelevant
3. Last Updated
What it means: Timestamp of when this entry was last edited
Location: Cache Management table, column "Last Updated"
Interpretation:
| Last Updated | Status | Action |
|---|---|---|
| Today | ✅ Fresh | Just edited |
| This week/month | ✅ Current | Recently maintained |
| 1-3 months ago | ⚠️ Check | Review for accuracy |
| 3+ months ago | ❌ Stale | Needs update |
| 6+ months ago | 🚨 Critical | Update immediately |
Combined with TTL: If "Last Updated" exceeds TTL, entry needs review.
Example:
- TTL: 30 days
- Last Updated: 60 days ago
- Action: Review and update entry now!
Action items:
- Sort by "Last Updated" (oldest first) to find stale entries
- Set reminders based on TTL values
- Review high-traffic entries more frequently
Analyzing Trends
Weekly Review Workflow
Spend 15-30 minutes weekly reviewing metrics:
Step 1: Check Dashboard (5 minutes)
- Note Cache Hit Rate - Is it improving or declining?
- Compare to last week - What changed?
- Check Success Rate - Are new entries matching?
Record: Keep a simple log (spreadsheet or document) tracking these numbers weekly.
Example Log:
Date | Hit Rate | Success Rate | Total Entries
-----------+----------+--------------+--------------
1/1/2026 | 65% | 75% | 8
1/8/2026 | 72% | 80% | 9
1/15/2026 | 78% | 85% | 12
Step 2: Identify Top Performers (5 minutes)
- Go to Cache Management
- Sort by Times Served (High → Low)
- Review top 5-10 entries
Questions to ask:
- Are these entries still accurate?
- Do they need updated information?
- Can I create similar entries for related topics?
Step 3: Find Underperformers (10 minutes)
- Sort by Times Served (Low → High)
- Focus on entries with 0 or very low "Times Served"
Questions to ask:
- Are variations matching how users ask?
- Is the topic relevant to users?
- Should I improve or deactivate?
Action: Generate variations or deactivate if irrelevant.
Step 4: Review Negative Feedback (10 minutes)
- Go to Conversations
- Filter by Negative Feedback
- Identify if any negative feedback relates to cached responses
Questions to ask:
- Did the cache return a wrong answer?
- Is the cached response outdated?
- Do variations cause false matches?
Action: Update or fix problematic cache entries.
Monthly Deep Dive
Spend 1-2 hours monthly for comprehensive analysis:
1. Calculate Key Ratios
Variation Density: Total Variations ÷ Total Entries
- Target: 7-12 variations per entry
Utilization Rate: Entries with Times Served > 10 ÷ Total Active Entries
- Target: 60%+
Update Frequency: Entries updated this month ÷ Total Entries
- Target: 20-30% (regular maintenance)
2. Topic Analysis
Group entries by topic and compare performance:
Example:
| Topic | Entries | Avg Times Served | Hit Rate |
|---|---|---|---|
| Admissions | 5 | 45 | High |
| Courses | 8 | 12 | Medium |
| Faculty | 4 | 3 | Low |
Insights:
- Admissions cache is working well (keep maintaining)
- Courses need more variations (medium hit rate)
- Faculty cache needs improvement or removal (low hit rate)
3. Seasonal Trends
Track metrics across semesters:
Example observations:
- Fall: Higher hit rate (new students ask common questions)
- Spring: Lower hit rate (experienced students ask specific questions)
- Summer: Medium hit rate (prospective students research program)
Action: Adjust cache strategy seasonally.
Using Metrics to Prioritize Work
Priority Matrix
Use this matrix to decide which entries need attention:
| High Times Served | Low Times Served | |
|---|---|---|
| Recent Update | ✅ Maintain | ⚠️ Monitor |
| Old Update | 🚨 Update NOW | ❌ Deactivate/Improve |
Quadrant Actions:
-
High Times Served + Recent Update (✅ Maintain)
- Keep monitoring
- Make small refinements
- Ensure accuracy
-
High Times Served + Old Update (🚨 Update NOW)
- HIGHEST PRIORITY
- Many users see this entry
- Outdated info affects most users
- Update immediately!
-
Low Times Served + Recent Update (⚠️ Monitor)
- Give it time to match users
- Check again in 2-4 weeks
- Add variations if still low
-
Low Times Served + Old Update (❌ Deactivate/Improve)
- LOWEST PRIORITY
- Not matching users
- Hasn't been updated
- Consider deactivating
Setting Performance Goals
Short-Term Goals (1-3 Months)
Example Goals:
- ✅ Increase Cache Hit Rate from 60% to 70%
- ✅ Reduce entries with "Times Served = 0" by 50%
- ✅ Add 3-5 variations to all active entries
- ✅ Update all entries with "Last Updated > 60 days"
How to achieve:
- Generate variations for underperforming entries
- Review and update stale entries
- Create new entries for popular uncached topics
Long-Term Goals (6-12 Months)
Example Goals:
- 🎯 Maintain Cache Hit Rate > 75%
- 🎯 Success Rate > 85%
- 🎯 Average 8+ variations per entry
- 🎯 Update all entries at least quarterly
How to achieve:
- Establish regular maintenance schedule
- Monitor metrics consistently
- Respond quickly to negative feedback
- Expand cache based on usage patterns
Exporting Metrics (If Available)
Some admin panels may allow exporting metrics:
Useful exports:
- 📊 Cache performance over time (CSV)
- 📈 Individual entry statistics (Excel)
- 📉 Hit rate trends (graphs)
Uses:
- Track progress over time
- Share with stakeholders
- Identify long-term trends
- Justify resource allocation
How to export (if available):
- Look for "Export" or "Download" buttons
- Choose format (CSV, Excel, PDF)
- Save and analyze in spreadsheet software
Common Metric Misinterpretations
Mistake 1: "Success Rate is 0%, cache isn't working!"
Reality: If you just created entries, success rate will be 0% initially. Give it 1-2 weeks for users to ask matching questions.
Fix: Wait patiently and generate variations.
Mistake 2: "100% Hit Rate is best!"
Reality: Extremely high hit rates (95%+) may indicate:
- Cache is too aggressive (matching unrelated questions)
- Not enough unique questions being asked
- Need to allow more RAG searches for nuanced questions
Fix: Monitor user feedback for false matches.
Mistake 3: "Times Served = 0 means entry is bad"
Reality: New entries need time. Also, niche topics naturally have low "Times Served."
Fix: Wait 2-4 weeks before judging new entries. Niche entries are okay if accurate.
Mistake 4: "I should cache everything to get 100% hit rate"
Reality: Over-caching can:
- Make maintenance overwhelming
- Cause false matches
- Reduce answer flexibility
Fix: Cache strategically—focus on frequently asked questions with consistent answers.
Next Steps
Now that you understand performance metrics:
- Follow best practices for effective cache management
- Troubleshoot issues when metrics don't improve
- Review metrics weekly to track progress
Remember: Metrics are tools, not goals. Use them to improve user experience, not to chase perfect numbers!