ML Impact Scoring: Every Org Is Different

Summary

We're building an ML model that trains on YOUR organization's GitHub issue history. By analyzing how you've labeled, prioritized, and resolved past issues, the model learns which error patterns matter most to your team. SaaS companies might prioritize issues affecting many users, internal tool teams might prioritize severity, and B2B platforms might prioritize customer-facing errors. The model adapts to your domain automatically by learning from your historical GitHub data; no configuration needed. This feature is in beta testing now—reach out if you want early access.

Pattern matching tells you what an error means, but it doesn't tell you which error matters most to your organization. A TypeError affecting 5 users might be critical for a small SaaS startup where those 5 users are all enterprise customers, but the same error might be low priority for a consumer app with 100,000 users.

This is where machine learning comes in. Instead of hardcoded priority rules that work for everyone and no one, we're building a system where Cazon trains a custom ML model on your organization's GitHub issue history, learning what matters most to your team.

Every Organization Has Different Priorities

Different types of organizations prioritize errors differently:

SaaS Companies care most about affected user count and revenue impact. An error affecting 100 users is more critical than an error affecting 1 user, all else being equal. They optimize for customer satisfaction and churn prevention.

Internal Tool Teams care most about severity and blocking behavior. An error that blocks 5 employees from completing critical workflows is higher priority than an error affecting 50 employees in a non-critical feature. They optimize for productivity.

B2B Platforms care most about customer-facing errors. An error affecting paying customers is more critical than an error in internal admin tools, even if the technical severity is identical. They optimize for revenue and customer relationships.

Consumer Apps care most about error frequency and user-facing visibility. An error occurring 10,000 times but caught gracefully is lower priority than an error occurring 100 times that crashes the app. They optimize for user experience and retention.

E-commerce Sites care most about revenue-impacting errors. An error in checkout or payment flows is highest priority, even if it affects fewer users than errors in other features. They optimize for conversion.

Traditional monitoring tools treat all these organizations the same, providing generic severity levels (Critical, High, Medium, Low) based on arbitrary rules. But what's critical for one organization is medium for another. This is why manual triage exists; someone has to apply domain knowledge to translate generic severity into actual priority.

How the ML Model Will Learn Your Priorities

Our approach is different: we learn your priorities by analyzing your team's GitHub issue history. Your past labeling decisions, response times, and resolution patterns reveal what matters most to your organization.

Here's how it's designed to work:

Step 1: Connect GitHub
You connect your GitHub repository to Cazon, allowing us to import your historical issues. We look for issues labeled as bugs or errors, typically going back 6-12 months to capture enough patterns.

Step 2: Extract Priority Signals
From each closed issue, we extract signals that reveal priority: P0/P1/P2 labels, time-to-close (fast response indicates high priority), comment count and reactions (engagement indicates importance), and keywords in title and description (payment, auth, crash, etc.).

Step 3: Learn Error Features
When new errors come into Cazon, we extract similar features: affected user count from SDK tracking, error frequency and rate, severity from error message keywords (fatal, error, warning), environment (production vs staging), and critical keywords (payment, checkout, auth, etc.).

Step 4: Train Custom Model
We'll train a gradient boosted decision tree on your GitHub data, learning which error features predict high-priority issues in your codebase. The model outputs a numeric impact score (0-100) for each new error, based on patterns learned from your historical priorities.

Step 5: Continuous Improvement
As you import more issues or update labels in GitHub, the model can retrain to capture evolving priorities. If your product shifts focus (e.g., new payment system launches), importing recent issues will teach the model about your current priorities.

This is fundamentally different from rule-based systems where someone has to explicitly configure "errors in service X are high priority" or "errors affecting >100 users are critical." The ML model discovers your priorities implicitly by learning from your team's historical GitHub labeling patterns.

What Features Will Matter Most

The ML model will consider dozens of features, but the weights will vary dramatically between organizations. Here are some patterns we expect to see:

SaaS orgs typically have GitHub issues where high-priority bugs correlate with affected user count (extracted from issue descriptions), frequent updates and comments (indicating active concern), and keywords like "customer-facing" or "production outage."

Internal tool teams typically label issues with clear severity markers (P0 = broken, P3 = nice-to-have), close critical issues within hours (not days), and use keywords indicating blocking behavior ("can't deploy", "blocks workflow").

B2B platforms typically prioritize issues mentioning customer names, contract dates, or revenue impact, show faster response times for paying customer issues, and use labels like "enterprise" or "customer-reported."

E-commerce sites typically prioritize issues in checkout/payment flows with labels like "conversion-blocking", show very fast time-to-close for revenue-impacting bugs, and use keywords indicating financial impact.

The beauty of ML-based scoring is that we don't need to know these patterns upfront. The model will discover them by analyzing your GitHub issue history, then codify those patterns as numeric weights that score future errors.

Impact Scores Will Drive Categorization

The ML model will output an impact score from 0-100 for each error. This score will be mapped to familiar severity categories:

90-100: Critical - Fix immediately, likely breaking core functionality or affecting many users
70-89: High - Fix soon, significant impact on user experience or business metrics
50-69: Medium - Fix this sprint, noticeable but not blocking
0-49: Low - Fix when convenient, minimal user impact

But unlike traditional monitoring tools where these categories are assigned based on generic rules, Cazon's categories will be informed by your organization's historical priorities. What one team calls "critical" will be different from what another team calls "high" based on their actual debugging behavior.

Training Data Quality Will Matter

The model will only be as good as the training data. If your GitHub issues have inconsistent labeling or minimal metadata, the model will struggle to find patterns. But if your team has used GitHub consistently for bug tracking, even implicitly, the model will detect and learn from those patterns.

For best results, import 100+ closed issues with clear priority signals: P0/P1/P2 labels, fast vs slow time-to-close patterns, and descriptive titles/descriptions. The model looks for correlations between these historical patterns and the features we can extract from new errors.

For organizations new to Cazon without extensive GitHub history, we'll start with a baseline model trained on aggregated patterns from similar organizations, then personalize it as you import your own issues. This "warm start" approach means you'll get useful impact scores from day one, even before the model has learned your specific patterns.

Why Not Just Use Rules?

You might wonder: why use ML instead of letting teams configure rules like "errors affecting >100 users are critical"? Three reasons:

1. Rules don't capture nuance. Your priorities aren't simple thresholds. The model learns complex patterns like "payment errors are always high priority" or "warnings in dev environment are usually low priority" without you having to articulate every combination.

2. Rules require maintenance. Your priorities change as your product evolves. ML adapts automatically when you import recent issues showing your current priorities, without requiring someone to update configuration files.

3. Rules are explicit, patterns are implicit. Most teams don't document their prioritization rules, but their GitHub issue history reveals them. ML extracts those implicit patterns from historical labeling, making them explicit and codifiable without requiring teams to articulate complex decision logic.

What This Means for Your Team

With ML impact scoring in place, you'll stop guessing which error to fix first. The Cazon API, VS Code extension, and MCP server will all expose impact scores alongside errors, so developers and AI assistants always see errors sorted by learned priority rather than recency or random order.

Junior developers won't need to ask senior developers "which bug matters more?" The impact score will provide a data-driven answer based on your team's historical GitHub patterns. AI assistants won't waste time generating fixes for low-impact errors while critical issues go unfixed. Everyone will work on what matters most according to patterns learned from your issue history.

And as your product evolves, importing recent issues allows the model to adapt automatically, learning new priority patterns without requiring manual reconfiguration or lengthy priority meetings.

Behind the Scenes: Beta Testing Now

This ML impact scoring system is in active development and beta testing with select early adopters. We're validating the GitHub import pipeline, tuning model parameters, and gathering feedback on how teams want to interact with impact scores in their daily workflows.

The approach is practical: connect your GitHub repo, import 100+ historical issues, train a model on your priority patterns, and get organization-specific impact scores for new errors. The model learns from how you've labeled and triaged issues in the past, applying those patterns to errors captured in production.

Want early access? If you're interested in being part of the beta program, reach out to us at hello@cazon.dev. We're looking for teams with well-labeled GitHub issue history who can help us validate that the model learns their priority patterns accurately.

What's Next

Pattern matching gives you instant recognition of known errors. ML impact scoring will tell you which errors matter most to your organization based on learned patterns. Together, these two systems provide the intelligence layer that bridges monitoring and AI-powered debugging.

Next, we'll show you how AI coding assistants access this intelligence through the Model Context Protocol (MCP), turning Claude Desktop into a production-aware debugging assistant that knows which errors to prioritize without you telling it.

This is Post 6 of our launch series on error intelligence. Follow along as we unpack the problem, introduce solutions, and show you how Cazon is building the missing intelligence layer between production errors and AI coding assistants.

Next: Post 7 - MCP Integration: Claude Desktop Gets Production Context →