The Customer Predicted Spend Tier: clear as mud

I've warned about how some analysis systems hype what they do. You've probably seen it too with those venture-based AI-everything companies that come out every month now.

A little while back Shopify released a Predicted spend tier analysis for a merchant's customers. This rated the customer as a High, Medium, or Low based on four different measurements:

• how often a customer made purchases
• how much the customer spent on average per order compared to the overall average spent per order
• how many orders the customer placed
• how recently the customer placed their last order

When it was released I noticed that was similar to RFM. It was missing the Monetary value (M, which is based on LTV) but added the customer's Average Order Value vs the overall shop's Average Order Value.

Pretty good metrics that are useful for every merchant.

But there was a bit of hype around this new metric when it launched so I was a bit skeptical. Distilling past behavior down into a single High, Medium, or Low value and calling it a prediction was a bit iffy and triggered my stock-market education.

Not as bad as the black-box AI-everything shoveled by other companies, but opaque and with a bit of "trust us" sprinkled in.

Well last month Shopify released the API for apps to access this prediction data directly. Digging into the programming docs for it reveals something completely different:

• HIGH - The customer's spend is higher than the 70th percentile of the shop's customer sales.
• LOW - The customer's spend is lower than the 10th percentile of the shop's customer sales, including customers who didn't repurchase.
• MEDIUM - The customer's spend is between the 10th and 70th percentile of the shop's customer sales.

Based on this, the only measurement used is how much the customer has spent when compared to the shop's spending. Customer spend is the same as the total Lifetime Value (LTV) and one could assume the shop's customer sales is the Average LTV. (Comparing it to the shop's total sales would be nonsense).

Notice how the merchant version doesn't mention the customer's total spend, only the average.

Something is going on here:

• The merchant docs are wrong and are making the metric seem more powerful than it really is. Which was then picked up and hyped by Shopify's content writers at launch, or
• the API docs are wrong and are describing something completely different, or
• no one knows what's actually measured and it's one of those AI black-boxes.

Given how API docs are written for a more technical audience and require engineering precision, I'm leaning toward the thought that they are the correct version and the merchant docs over-hyped what was measured.

This opaqueness happens all the time when companies mess around with metrics. From invent new names for existing metrics or play with the calculations to manufacture their own metrics, it can be a confusing mess all for "competitive advantage". There should be a burden of proof where a company has to at least explain roughly how the metric is calculated before anyone trusts and uses it.

Hopefully someone at Shopify will clarify and fix up their documentation. Segmenting and predicting customer behavior is valuable but merchants need to be able to rely and trust the measurements. Anything else leads to confusion and misuse.

Customer Predicted Spent Tier might even be something I want to pull into my app as it's in the same space as the RFM, LTV, and grading models I use.

This transparency issue is behind an addition I've included in the Focus pages for Repeat Customer Insights. Many now include "How is this measured" docs right on the page describing how the metric works, in plain English.

Now that I figured out a good design for that, I'm hoping to use them all over the place in the app instead of hoping customers know what all the industry and custom metrics mean.

Eric Davis