The age of AI for PMs…
Artificial intelligence (AI) is becoming the operational foundation of most companies — now at the core of their operating models — defining how tasks are actually executed.
It’s the universal engine of execution – the fourth industrial revolution. And, like all industrial revolutions, it will have a great impact not only on our economy, but all other aspects of our lives.
And you know what?
If you’re a product manager (PM), your reaction should be: 🙌📈😁
PMs are responsible for defining the why, the when and the what. Helping shepherd a cross-functional team of engineers, designers, and strategists to build a product from inception-to-launch-to-scale; one that’s valuable, feasible, and usable.
Going forward, this will remain the same — but there are a few important evolutions. Evolutions that are at the core of this post.
Now the opportunity set for PMs and the problems they’re drafted into solve has increased. In the past, PMs were focused mostly on the traditional tech products — “PM for Ads”, “PM for Mobile App X”, “PM for Marketplace Y”, etc.
Industries that were historically at the other end of the tech spectrum are now starting to understand the importance of product management in this new age of AI. Companies in manufacturing, retailing, transportation, finance, health care, law, advertising, insurance, entertainment, education, and virtually every other industry are in a race to transform their core processes, and business models, to take advantage of AI.
AI is the ‘runtime’ that is going to shape all of what we doSatya Nadella, Microsoft CEO
These new operating models can take various forms. In some cases PMs might be responsible for products that only manage flows of information (think Google or Facebook). In other cases, operating models guide how the company builds, delivers, or operates actual physical products (Amazon or Waymo). In either case, AI is at the core of the model, guiding the most critical processes and operating decisions, while humans are moved to the edge, off the critical path of value delivery.
The remainder of this post is structured as follows; first, we cover how building AI products differs from what has come before — for example, how have design sprints evolved? Next, we cover the skills a PM will need to acquire in order to actually lead an AI product team. Lastly, some practical advice for how to make the progression from PM to (AI)PM.
If you enjoy this post, please be sure to check out and pre-order my upcoming book — “10 Minute Guide to Building AI Products”. Thanks!
How building AI products differs
Take Google as an example. As soon as a user types a few letters in the search box, algorithms dynamically predict the full search term based on prior search terms and the user’s past actions. These predictions are captured in a drop down menu (the autosuggest box), which helps users zero in quickly on the desired search. Every user movement and every click are captured as data points and every data point gathered improves the prediction for future searches. The more searches, the better predictions, and the better predictions, the more the search engine is used.
An (AI)PM within ‘Search Engine Ops’ might have some new hypothesis — for example, that showing fewer ads might improve revenues on a given page, or that highlighting search results would improve click-through rates. To provide additional fodder for improvement, these hypotheses would be loaded on the experimental machinery and tested on a statistically significant sample of users.
Clearly there is no way all of this data could be analysed by a few analysts using manual tools, or even by casually assembled code. The (AI)PM solves this problem by bringing mass production methods to data processing and analytics, thus forming the core of a digital operating model.
This is a step change to the approach of PMs in the past. It’s no longer just a matter of progressing through a few interfaces and presenting output.
More subtly, there are so many ‘sub-products’ that are being called before the user has even clicked search (if they even have to click search). Compared to a decade ago, the (AI)PM now has very different thoughts, hypotheses and approaches.
Datasets have become the new wireframes. Data enables the (AI)PM to understand the problem and solution sphere. Datasets are the nuggets that let them dream up or imagine creative approaches to making their users‘ lives easier.
What was once centred around A/B testing user interfaces and completing a ‘Design Sprint’ has now evolved to also include the ‘Data Science Sprint’ — the purple flow above. Design still plays an important role — usually these two sprints run in parallel, supporting one another.
Types of AI problems and how to identify them
The (AI)PM needs domain expertise and an understanding of the basic AI/ML algorithms to identify solvable problems.
I say machine learning because most initial problems that you’ll solve as an (AI)PM will be “the low hanging fruit”, based upon supervised learning – relatively simple A to B — for example, image recognition, speech recognition, ad serving, etc.
I’m not going into the basics of machine learning here, but there are a plethora of resources and courses out there. My recommendations are at the end of this post.
With regard to the types of problems that you’re most likely to encounter, when seeking to improve workflows with AI & ML, Google have done a great job of highlighting some fairly typical use cases:
- Recommending different content to different users. Such as providing personalised suggestions for movies to watch.
- Prediction of future events. For example, showing flight prices for a trip to Denver in late November.
- Personalisation improves the user experience. Personalising automated home thermostats makes homes more comfortable and the thermostats more efficient over time.
- Natural language understanding. Dictation software requires AI to function well for different languages and speech styles.
- Recognition of an entire class of entities. It’s not possible to program every single face into a photo tagging app — it uses AI to recognise two photos as the same person.
- Detection of low occurrence events that change over time. Credit card fraud is constantly evolving and happens infrequently to individuals, but frequently across a large group. AI can learn these evolving patterns and detect new kinds of fraud as they emerge.
- An agent or bot experience for a particular domain. Booking a hotel follows a similar pattern for a large number of users and can be automated to expedite the process.
- Showing dynamic content is more efficient than a predictable interface. AI-generated suggestions from a streaming service surface new content that would be nearly impossible for a user to find otherwise.
How you approach the problem
Data science / design sprint & problem formulation
When building any product in a human-centred way, the most important decisions you’ll make are: Who are your users? What are their values? Which problem should you solve for them? How will you solve that problem? How will you know when the experience is “done”?
Gather, explore and prepare data
Data is the essential input of AI. One reason for the radical advances made by AI in recent years is that the velocity, volume and variety of data available has exploded.
Many incumbent businesses that are attempting to build out AI capabilities find that the data they possess is fragmented, incomplete and often siloed within divisions and disparate IT systems.
Data is highly fragmented, resides in various system silos with incompatible data structures, is missing common identifiers, and may not necessarily be very accurate.
I see a lot of first time (AI)PMs consistently underestimate the challenge and the urgency of the investment they face in cleaning and integrating their data across the company.
After data is gathered, a lot of work still remains in cleaning, normalising and integrating it. These steps are quite challenging. Data assets are most often plagued by all kinds of biases and even plain errors, and a significant investment needs to be made in ensuring that the data is checked carefully for inaccuracies and inconsistencies.
These things often sound simple but are not, especially as the datasets reach significant size.
And finally, you’ll need to split the data into training and test sets. The model is trained to learn from the training data, and then evaluated with the test data. Test sets are data that your model hasn’t seen before — this is how you’ll find out if, and how well, your model works. The split will depend on factors such as the number of examples in your dataset and the data distribution.
The training set needs to be large enough to successfully teach your model, and your test set should be large enough that you can adequately assess your model’s performance. This is usually the time when developers realise that adequate data can make or break the success of a model. So take the time to determine the most efficient split percentage. A typical split of your dataset could result in: 60% for training, and 40% for testing.
The success or failure of your entire AI product — depends on your ability to get the best training data in your industry.
Build, train and evaluate model
After the data is gathered and prepared, the tool that makes the data useful is the algorithm — the set of rules a machine follows to use data to make a decision, generate a prediction, or solve a particular problem.
ML is part art and part science. When you look at ML algorithms, there is no one solution or one approach that fits all. Some problems are very specific and require a unique approach. E.g. if you look at a recommender system, it’s a very common type of ML algorithm and it solves a very specific kind of problem. While some other problems are very open and need a trial & error approach. Supervised learning, classification and regression etc. are very open. They could be used in anomaly detection, or they could be used to build more general sorts of predictive models.
Hackernoon put together a useful framework for deciding upon your approach:
- Know your data
- Visualise your data
- Clean the data
- Augment your data
The next step is to categorise the problem. This is a two-step process:
First, categorise by input:
- If you have labelled data, it’s a supervised learning problem.
- If you have unlabelled data and want to find structure, it’s an unsupervised learning problem.
- If you want to optimise an objective function by interacting with an environment, it’s a reinforcement learning problem.
Next, categorise by output:
- If the output of your model is a number, it’s a regression problem.
- If the output of your model is a class, it’s a classification problem.
- If the output of your model is a set of input groups, it’s a clustering problem.
- Do you want to detect an anomaly? That’s anomaly detection
Understand your constraints
- What is your data storage capacity? Depending on the storage capacity of your system, you might not be able to store gigabytes of classification/regression models or gigabytes of data to clusterise. This is the case, for instance, for embedded systems.
- Does the prediction have to be fast? In real time applications, it is obviously very important to have a prediction as fast as possible. For instance, in autonomous driving, it’s important that the classification of road signs be as fast as possible to avoid accidents.
- Does the learning have to be fast? In some circumstances, training models quickly is necessary: sometimes, you need to rapidly update, on the fly, your model with a different dataset.
Interpreting the output
If your training data aren’t properly suited to the context, you also increase the risk of overfitting or underfitting your training set. Overfitting means the ML model is tailored too specifically to the training data, and it can stem from a variety of causes. If an ML model has overfit the training data, it can make great predictions on the training data but performs worse on the test set or when given new data.
Models can also make poor predictions due to underfitting, where a model hasn’t properly captured the complexity of the relationships among the training dataset features and therefore can’t make good predictions with training data or with new data.
There are many resources that can help the software engineers and research scientists on your team with understanding the nuances of training ML models so you can avoid overfitting and underfitting. But first, involve everyone on your product team in a conceptual discussion about the examples, features, and labels that are likely required for a good training set. Then, talk about which features are likely to be most important based on user needs.
Finally, one of the key responsibilities of a product manager is to brainstorm how your model might fail, and how to mitigate it at an early stage. Fixing a bias in the model later on might be a more difficult and costly process.
Deploying your models
If an ML model makes a prediction in Jupyter, is anyone around to hear it?
The deployment of machine learning models is the process for making your models available in production environments, where they can provide predictions to other software systems. It is only once models are deployed to production that they start adding value, making deployment a crucial step.
One of the reasons why the deployment of machine learning models is complex is because even the way the concept tends to be phrased is misleading. In truth, in a typical system for deploying machine learning models, the model part is a tiny component. This diagram above is useful for demonstrating this point.
Machine learning systems have all the challenges of traditional code, plus an additional set of machine learning-specific issues. This is well explained in the paper from Google “Hidden Technical Debt in Machine Learning Systems”.
What this means is that you cannot think about model deployment in isolation, you need to plan it at a system level. The initial deployment is not really the hard part (although it can be challenging). It’s the ongoing system maintenance, the updates and experiments, the auditing and the monitoring that are where the real technical debt starts to build up.
Carefully defining clear and secure APIs is essential for the (AI)PM. After all, APIs throttle the flow of data in and out of any AI system. Think of it as a way for the company to control all the data and functionality that it is willing to offer internal and external developers. As such, APIs control access to some of the most critical and important data.
The type of people you will be working with
Overall the ‘cross-functional’ team mostly stays the same; there’s still going to be a need for design, engineering, and QA. The additions are roles such as Data Scientist, Machine Learning Engineer and Data Engineer.
Data Scientist – these are the actual people behind the algorithms and machine learning models. They make sure the right ML approach chosen, the teaching/training process and are responsible for getting to the actual model that can be used for the purpose given.
Machine Learning Engineer – these people sit at the intersection of software engineering and data science. They leverage big data tools and programming frameworks to ensure that the raw data gathered from data pipelines are redefined as data science models that are ready to scale as needed. MLEs feed data into models defined by data scientists. They’re also responsible for taking theoretical data science models and helping scale them out to production-level models that can handle terabytes of real-time data.
Data Engineer – These people are important because they are the ones that can get you the dataset on a technical level, prep it for ML and make sure the quality of data is on point.
The impacts on frameworks
Start small and build — the lean startup principles of MVP and fast iterations still apply with ML products. Small wins generate momentum. As you learn and create momentum, the unknowns start to surface. Faster iterations and quick wins enable us to accept an agile mindset and design a culture that allows uncertainty. If you’re an (AI)PM in a large company, your playbook should be:
- Get the executive & leadership team aligned
- Envision the problem
- Build a proof of concept
- Build a minimal viable product
- Roll out to production
Does building an AI based product lend itself to Agile? First off, data scientists aren’t typically that familiar with Agile and can sometimes have difficulty harmonising their research activity with Agile. But it does pay to tie it all into development sprint cycles.
It allows both sides (engineering and the scientists) to understand what is happening and how work is progressing: product and engineering changes might be useful in informing the direction of research. Likewise, there may be data features or feedback from the scientists that could be easily engineered and would make things simpler. While research is ongoing, and often a time consuming task, having bi-weekly checkpoints where ideas can be discussed and demoed can be good for the whole team’s understanding as well as being a positive motivator.
What skills does an (AI) PM need?
Understand the underlying algorithms
There’s no getting around the fact that you’re going to have to brush up your knowledge on the maths and fundamentals behind some of the most popular ML algorithms.
Understand the problem you’re trying to solve with ML
From my experience, the problems that ML can help solve usually fall into one of these buckets:
- Could we make the user experience more tailored and personalised?
- Could we make the user experience safer?
- Could we help users to achieve their goals easier and faster?
- Could we create a new experience that was previously not possible?
Clearly communicate the vision and the people problem you’re trying to solve. People often get lost when you’re going into too many technical details, keep it simple and high-level. If ML is a completely new area for your company, share the steps in the ML development process and where you’ve got to. Then it’s easier for people to track your progress. Don’t commit to numeric goals at the start of the project: the whole is greater than the sum of its parts so you can’t evaluate the impact before you shipped your v1 model. Frame it as a learning opportunity, with clear definitions of success and failure.
Also, some of the courses that I’ve done online to help me get up to speed. And you actually have to complete them. Signing up to Coursera and doing the first two lessons by Andrew Ng does not count. It’s not going to make you an AI expert, and you’re only going to be fooling yourself. You have to get into the weeds and explore anything you don’t understand before progressing.
Know how to read scientific papers
There’s a bunch of useful guides out there on how to tackle a scientific paper — they need not be as intimidating as you first think. Scientific papers enable you to understand what trends and innovations are occurring, they’ll be the spark for your ideas and research hypotheses. You should also aim to have your team complete a scientific write-up for any of the products your building.
Account for model mistakes and biases
One of the key responsibilities of a product manager is to brainstorm how your model might fail, and how to mitigate it at an early stage. Fixing a bias in the model later on might be a more difficult and costly process.
Next steps for PMs
Nothing else remains but for me to wish you good luck on making the leap from PM to (AI)PM — investing now will be well worth the effort. Get out there and build some basic ML applications, put them on your Github, ask scientists advice whilst your working on the project — the knowledge you learn will stay with you much longer and you’ll get the respect from your colleagues that you’re actually out there building something.
Applied AI Course
Machine Learning by Andrew Ng
Machine Learning A-Z — Udemy
Intro to ML — Udacity
Introduction to Python Programming — Udacity
Introduction to Python — Kaggle Learn
Pandas — (Kaggle Learn)
Python for Developers — Jupyter Notebook Collection
Tutorials & cheetsheets
What is AI exactly? — Coldfusion
A.I. is Progressing Faster Than You Think! — Coldfusion
“I Am AI” — NVIDIA
The Rise of AI — Bloomberg
Real Difference between AI/ ML/ DS/ DL — Applied AI Course
Why Deep Learning Now? — Coldfusion
How to Study Machine Learning — Siraj Raval
Data Science/Machine Learning Project Life cycle