Shipping code is not the goal.. It turns out that absolutely nobody… | by Matt Schellhas | Nov, 2022

Dal-e – Extended edit mode ship squirrel

It turns out that nobody wants to Netflix and chill Schindler’s List.

If you hang around in tech long enough, you get to hear some great stories. One of my favorites is this recurring problem Netflix had with their recommendation engine. This engine is meant to replicate the kind of organic recommendations people use to find new movies they want to watch. The idea is that finding “hidden gem” movies on Netflix will make users value the service more, which will improve retention.

he worked for more than twenty years nowand was in almost every incarnation schindler’s list Problem. because from an algorithmic perspective, schindler’s list Checks all the boxes:

  • Many people have seen it.
  • It has been rated very highly by reviewers and users.
  • It won all kinds of awards.
  • It was successful at the box office.
  • People who have seen one Spielberg film see (and like) others.
  • People who have seen one film starring Liam Neeson, Ralph Fiennes, or Ben Kingsley tend to see (and like) others.

Based on the data alone, this is a great movie – exactly the kind of thing the engine should be recommending to users. but nobody Wants to see it. Even seeing it on the “Movies You Will ❤️” list made the whole service less trustworthy because it was so artificial. no one would recommend schindler’s list Except in very special circumstances.

eventually, schindler’s list Became a common test case for new recommendation algorithms on Netflix. the computer doesn’t know for sure Why? recommend to Inglourious Basterds well but schindler’s list Not there. It is not needed. That is not the goal.

Unfortunately, most companies miss this point when developing software. They are too busy trying to increase their velocity or meet some arbitrary deadline. Some PMs make a list of requirements. Engineers implement those requirements, then move on to the next pile of work.

But think about our example:

The idea is that finding “hidden gem” movies on Netflix will make users value the service more, which will improve retention.

The goal here isn’t to implement some flashy ML or destroy the team’s backlog in record time. It’s not a product requirement—it’s a product hypothesis,

Netflix had a few KPIs that they used to measure user retention. He theorized that a better recommendation engine would improve that metric. Then they started validating that hypothesis. Send some code, do some A/B testing, iterate as needed.

Yes, repeat as needed, It means “How long is this going to take?” Not a good answer. All of Dunn’s definitions that fixate on passing code reviews and tests and writing release notes are missing the point. If your definition of complete ends with “deployed to production”, congratulations: you’ve made it halfway through.

Because some hypotheses will be disproven. Sometimes you’ll be wrong. Sometimes that code will provide improvement in that KPI but crater three others. Not all good bets pay off.

You no longer need KPIs for everything. Please Not everything KPI. Qualitative improvement also matters. but you should some Like user feedback if your end goal is to impress users. Otherwise you are just guessing. You’re shipping code as fast as possible, and are shocked when revenue drops because your software keeps recommending schindler’s list on Saturday night.

Certainly not everything is a hypothesis. User confirmation is not required to secure user data or calculate tax rates if you have met your goal. Some requirements are actually objectively measurable requirements. Yet they are put into the same queue of user stories, estimated in the same way, deployed in the same way, even though they have very different risk profiles and require very different approaches.

For most companies, objective requirements are rarely the only valuable bits. If you know ahead of time what the result is going to be, then there is no risk; No innovation. Netflix has been successful in part because they identified a very valuable problem (user retention) and then spent more than twenty years coming up with new ways to improve it. They ship code quickly so they can get feedback quickly. That feedback turns into an improvement on the next iteration, and so on. Shipping code is just one step in that cycle. This was never the goal.

Leave a Reply