The Honest feedalyze.net Postmortem: What I Built, What I Got Wrong, and What's Next
feedalyze.net scores customer accounts 0–100 for churn risk using email, survey, and support ticket analysis. Here's what the build taught me about AI product design, and why I'm rethinking the scoring model.
feedalyze.net is a churn risk scoring tool for B2B SaaS companies. Connect your support ticketing system, NPS survey tool, and email sequences; feedalyze analyzes the signals from each channel and produces a 0–100 health score per customer account, with the contributing factors ranked by weight.
I built it in 9 days. It has real users. And I think the core scoring model is wrong.
This is the postmortem.
What I built
The product works like this:
- Integrations: Connect Intercom (support tickets), Delighted (NPS surveys), and Customer.io (email engagement). OAuth flows for all three.
- Signal collection: feedalyze pulls the last 90 days of signals per account — ticket volume, sentiment, resolution time, NPS score and trend, email open/click rates.
- Scoring: Each signal is normalized to a 0–100 sub-score. Sub-scores are combined via a weighted average into the account health score.
- Output: A dashboard showing all accounts ranked by health score, with the top 3 contributing risk factors surfaced per account.
The AI component is in the sentiment analysis step. Support ticket text is analyzed for sentiment, urgency, and complaint category using Claude. This adds a dimension that raw ticket counts miss — a single angry ticket from a power user is more signal than 10 routine how-do-I questions.
The build took 9 days and the core architecture worked on day 1. The product does what it says it does.
What the data revealed
After 6 weeks with real users, I looked at the outcomes. For accounts the model flagged as high-risk (score below 40), I asked customers to track whether those accounts actually churned, expanded, or renewed in the subsequent 90 days.
The results:
| Model prediction | Actual outcome | |---|---| | High risk (score < 40) | Churned: 34% / Renewed: 41% / Expanded: 25% | | Medium risk (40–70) | Churned: 18% / Renewed: 63% / Expanded: 19% | | Low risk (score > 70) | Churned: 8% / Renewed: 72% / Expanded: 20% |
The model is better than random — high-risk accounts churn at 4× the rate of low-risk accounts. But a 34% churn rate on high-risk accounts means 66% of accounts the model flags as high-risk actually renew or expand. That's a lot of false positives. Customer success teams acting on high-risk flags will waste significant time on accounts that were fine.
When I designed the weighting, I tuned the model to catch churns rather than to minimize false alarms. In classification terms, I maximized recall at the expense of precision. That's the wrong tradeoff for CS teams who have limited time — they need to know which accounts are actually at risk, not which accounts might possibly be at risk.
The scoring model problem
The core issue is that I'm using the wrong signals.
The signals I'm measuring — ticket volume, NPS score, email engagement — correlate with churn but they're not the drivers of churn. They're symptoms. The actual drivers of B2B SaaS churn are:
Budget cuts or procurement freeze: No amount of NPS analysis predicts when a CFO is going to freeze discretionary spend.
Champion departure: The person who bought your product leaves the company. Their replacement hasn't built the relationship yet. This is the leading cause of involuntary churn in B2B SaaS and it's invisible in your support tickets.
Product-market fit drift: The company grew, pivoted, or changed their use case. Your product no longer solves their current problem even if it solved the problem they had when they bought. Engagement signals look fine until they don't renew.
Internal politics: The decision to churn was made two quarters before the renewal date. The signals I'm measuring — support tickets, email opens — don't reflect internal procurement discussions.
None of these show up in the signals I'm measuring. The signals I am measuring reflect account health as a consequence of these drivers — by the time NPS tanks or ticket volume spikes, the churn decision has often already been made.
A churn prediction model built on symptom signals will always have high false positive rates, because symptoms appear for many reasons. A model built on driver signals — CRM activity, product usage depth, stakeholder mapping — would be substantially more predictive. The problem is that driver signals require deeper integrations that are harder to build and sell.
What I got right
Despite the scoring model's limitations, some things worked well.
The sentiment analysis adds real value. Customer success managers consistently tell me that the sentiment tagging on support tickets surfaces things they wouldn't have caught from ticket counts alone. A medium-volume account with consistently frustrated tone is a different risk profile than a high-volume account asking routine questions. The AI layer adds genuine signal here.
The dashboard design works. Users open the product, look at the sorted list, and immediately understand which accounts to call. The UX goal was to reduce the "where do I start?" friction in CS team workflows, and the sorted-by-health-score view accomplishes that.
The integrations were the right scope. Intercom + Delighted + Customer.io covers ~60% of the ICP's toolstack without requiring complex custom integrations. Most CS tools the target users have fit one of these three categories.
What I'm rebuilding
The v2 scoring model will use different signals:
Product usage depth (via Segment or Amplitude) — which features are they actually using? An account paying for an Enterprise plan but only using 2 of 10 features is high risk regardless of their NPS score.
Stakeholder activity (via CRM webhook) — when did the champion last log a meeting? Has deal activity in the account dropped? When was the last executive touchpoint? These signals require a CRM integration (HubSpot or Salesforce), which is a heavier lift but produces substantially more predictive data.
Contract and usage signals — days since last login, active seat count vs. licensed seats, feature adoption vs. onboarding checklist completion. These are internal signals that require a customer data platform integration.
The tradeoff with better signals is harder sales. Deeper integrations mean longer onboarding, more security reviews, and a higher bar for the champion to get internal approval. The current product's three-OAuth-flows onboarding takes 10 minutes. The v2 will take days.
The build-in-public lesson
Building in public means publishing failures alongside wins. This postmortem is uncomfortable to write because it's essentially "I shipped a product whose core model is demonstrably wrong."
But "demonstrably wrong" only happened because I shipped fast enough to get real outcome data. The alternative — spending 3 months building a theoretically better model before talking to users — would have taught me less while shipping later.
The 9-day build wasn't a mistake. The 9-day build produced a working product, real users, and outcome data that showed exactly what needed to change. That's precisely what an MVP is supposed to do.
feedalyze users have 6 weeks of real churn prediction outcomes with a control baseline. Most CS teams have no systematic data at all. Even a model with a 34% hit rate on high-risk accounts is meaningfully better than "we didn't know this account was at risk until they sent a cancellation email."
What's next
feedalyze v2 is in development. The scoring model will be rebuilt on product usage depth and CRM activity signals. The AI sentiment layer stays — it genuinely works. The new signal architecture requires a Segment integration and a HubSpot/Salesforce connector, both of which are underway.
Current users will be migrated automatically. The new scoring model will run in parallel with the old model for the first 30 days post-launch so users can compare predictions before fully switching.
If you're a B2B SaaS company with Segment + HubSpot and want early access to the v2 beta, reach out.
Araho Digital
We build what we write about.
Every technique in this post was used on a real client project. If you're building a SaaS product or internal tool and want it done in weeks, not months — that's what we do.
Fixed price. Fixed scope. Money-back guarantee.