As AI product managers, what exactly are we optimizing? 01/15 Update SLTechnology News&Howtos

As AI product managers, what exactly are we optimizing?

2025-01-15 Update From: SLTechnology News&Howtos shulou NAV: SLTechnology News&Howtos > Internet Technology >

Shulou(Shulou.com)06/02 Report--

Https://www.toutiao.com/a6685626606284702220/

Produced by big data Digest

Author: Guy Molho

Compiled by Zhang Qiang, sleepless iris, Zhou Suyun

What is the relationship between user experience and artificial intelligence (AI)? What does this relationship mean for product managers?

These two issues are important because they affect the user experience of the product and its value proposition.

Guy Molho, the writer, is an experienced product manager. It took 15 months to develop AI-based software solutions for the customer success team of B2B companies without dabbling in too many details.

In the article, he mentioned that the product manager needs to know her customers, her work environment, mission, what she wants to achieve, and her challenges before she can design a target solution that best meets these needs.

The following is his response to "what exactly are we optimizing as AI product managers?" Answer to the question, enjoy ☟

What does AI stand for for a product manager?

Products must solve real-world problems. Technical and implementation details should serve the product and have minimal impact on usability (at least at the software level). AI is an implementation of problem solving, but its predictability has an important impact on user experience and usability.

When a company designs AI-based solutions for any market or category, it always asks itself, why do we optimize our products? Are we optimizing accuracy, positive predictive value, or hit rate? Or in other words, can accuracy or recall be optimized? Answering these questions is critical because it affects the user experience of the product and its value proposition.

As a product manager for a startup that develops AI-based solutions, I think about this every day and make our solutions based on decisions.

Recall rate and accuracy

Recall rates and accuracy sound familiar, maybe too familiar to forget what they are! Accuracy and recall are statistical terms used to measure the correlation of the results returned by the algorithm. These terms have official academic interpretations, but I would like to explain them through an example.

Weather forecast

Suppose I have a machine that predicts whether it will rain tomorrow. If it rains tomorrow, the machine returns "yes", if it doesn't rain, it returns "no". We run the machine for 100 days in a row and get the following results:

The machine predicts that it will rain 10 times for the remaining 90 days. It predicts that it will not rain.

Now let's compare the predicted results with the actual weather:

Of the 10 times it predicted rain, it did rain. The prediction is very accurate. How accurate is it? Of the 10 predictions, it rained 10 times, → 10 and 10. Our accuracy is 100%.

Does this mean that I have the ultimate rain predictor? I'm not sure. Let's look at the other 90 days.

When we calculate the total number of rainy days, we find that there are actually a total of 20 rainy days. What does that mean? The machine recalled (predicted correctly) 10 rainy days of 10 rainy days → 10 → 20 → 50%. So it predicted 50% of the rainy days, but it also missed 50% of them.

Now, do you think my machine is still great?

Source: Unsplash

Let's consider the extreme case. My machine is broken. It says it will rain every day. The results obtained are:

Rain-100 days without rain-0 days

Now let's evaluate the results again:

The machine is not so accurate now, because out of a total of 100 times it predicted, it actually rained only 20 times → 20 →.

But considering the real 20 rainy days, the machine correctly predicted all 20, that is, 20 → 20% recall rate.

Now suppose you can go to the store to buy one of the above machines, which one do you prefer? The more accurate one, that is, if it says it's going to rain, you can be sure, but you'll miss a lot of rainy days, or you won't miss any rainy days, but many other days will predict the wrong machine?

The answer is not so straightforward. This may be true for rainfall forecasting, but for many other AI-based applications, that is not the case.

This may be a bit of a twist, but it doesn't matter. I created a confusion matrix that may help you classify things and calculate accuracy and recall rates:

Accuracy = TP / (TP+FP)

Recall rate = TP / (TP+FN)

Weather forecast-machine A

Accuracy = 10 / (10 / 0) = 10 / 10 / 10 / 10 / 10 / 10 / 10 / (10 / 10 / 0) = 10 / 10 / 10 / 10 / (10 / 10 / 0) = 10 / (10 / 0) = 10 / (10 / 0) = 10 / (10 / 0) = 10 / (10 / 0) = 10 / (10 / 0)

Recall rate = 10 / (10 / 10) = 10 / 10 20 = 50%

Accuracy = 10 / (10 / 0) = 10 / 10 / 10 / 10 / 10 / 10 / 10 / (10 / 10 / 0) = 10 / 10 / 10 / 10 / (10 / 10 / 0) = 10 / (10 / 0) = 10 / (10 / 0) = 10 / (10 / 0) = 10 / (10 / 0) = 10 / (10 / 0)

Recall rate = 10 / (10 / 10) = 10 / 10 20 = 50%

Weather forecast-machine B

Accuracy = 20 / (20 / 80) = 20 / 20 / 100 = 20%

Recall rate = 20 / (20 / 0) = 20 Universe 20 = 100%

Accuracy = 20 / (20 / 80) = 20 / 20 / 100 = 20%

Recall rate = 20 / (20 / 0) = 20 Universe 20 = 100%

So, what are you optimizing?

Now, when we really understand the differences, how should we optimize our models and products? Accuracy or recall rate? Most of the time we have to choose one, and it is almost impossible to have high accuracy and recall.

The decision on what to optimize depends on many factors: psychology, economy, error cost, omission cost, reputation, time, and so on.

Let's look at three real scenarios and discuss:

Cancer detection

As a patient, would you rather be found to have cancer and start treatment, and then find no disease (false positive)? Or do you find out that you have cancer (false negative) when it is too late for treatment?

If you are a health insurance company, will your answer change? Will you fund all unnecessary treatments? Will you raise the insurance premium to beat anyone? As a doctor, will you risk your reputation to miss the test?

At least as a patient, no one wants to miss being tested. Therefore, when building products that detect cancer, it makes more sense to optimize recall rates (to avoid false negatives).

Airport security check

As a passenger, would you rather wait in a long cordon to avoid a dangerous accident through the security check? Or would you rather pass these tests quickly and take the risk that guns can be smuggled into aircraft? Regulators will certainly choose to avoid risk.

Netflix recommendation

As a user, you prefer to get highly relevant recommendations rather than general content that may be popular but not suitable for you. Therefore, in this case, the product should optimize accuracy (avoid false positives).

We can discuss dozens of examples and try to understand what the product is optimizing and how it affects the overall user experience when interacting with it.

Customer churn forecast

We have developed an AI-based product to predict customer churn in B2B companies. We enable the customer success team to focus their work on the really important customers and get better results.

Optimization accuracy means that the product will pinpoint a very targeted list of customers at risk of loss, and no one is mistakenly assigned to it. The difficulty in this direction is the lack of a bunch of customers who will lose but go undetected.

The idea to alleviate this situation is to split the list into several pages, where the first page contains the most relevant customers. If users want to explore more, they can go to the next page. Google search results provide this experience, and the home page contains the most relevant results. If you want to explore more, you can also check out other pages.

Optimizing the recall rate means that the product will be less sensitive, generate a longer list of customers at risk of loss, and make sure we don't miss anyone. The disadvantage here is that the list contains false positives, that is, customers who are not at risk of loss.

The way to mitigate this situation is to combine the list with other customer characteristics, which may provide more hints about their risks, such as priorities.

Therefore, as a product manager, I need to know from customers whether they are more tolerant of false positives or false negatives, and what kind of experience they expect from the product. Whether there are enough resources to deal with lost customers, and so on.