How to structure product-focused Machine Learning research

How to structure product-focused Machine Learning research

As more businesses start integrating machine learning into their tech stack, Product Managers are having to adapt. Not only do they need to understand basic machine learning principles, they also have to figure out how to handle technical research when planning roadmaps and sprints.

While research has always been something that engineers have done (e.g. addressing requirements such as “Speed up process X by at least 50%”), it happens far more frequently once machine learning is brought into the mix. For this reason, AI-focused Product Managers cannot get away with shoe-horning research activities into their normal product prioritisation processes.

This article sets out a basic framework you can use to keep your research activities tight and product-focused. This is based on my learnings from my time as Head of Product at JukeDeck (who used AI to generate new music) and Zyper (who use ML to identify brands’ most passionate customers on social media).

Challenges

There are four important considerations when it comes to planning product-focused machine learning research.

1) Aligning it with Product Objectives

One of the biggest challenges is to make sure the research roadmap is informed by the same priorities as the product roadmap. This often doesn’t happen, especially when research work is planned in a silo with little collaboration with Product Management.

2) Keeping it focused

A key product management responsibility is to scope projects narrowly. This ensures that the team focuses their time on activities with the highest perceived ROI.

Given the unknowns involved, scoping research projects is pretty unintuitive::

  • If you’re too prescriptive, the Researchers won’t have the flexibility to adapt to new information and use their expertise to guide the research down the most promising avenues.
  • But if you’re too vague, the research can become a series of unfocused enquiries that never make it into the end product.

3) Stopping it at the right time

Deciding to stop a line of research can be very uncomfortable: Stop it too early and you could miss out on a game-changing discovery. But let go on too long and you’ll end up pouring valuable resources into a wild goose chase.

4) Measuring progress in a meaningful way

Deciding how to measure the impact of your research can be hard. Often you’ll have to make tradeoffs between how easy your metrics are to measure and how truthfully they assess the value you’re ultimately trying to deliver.

Product-Focused Research Framework

1) Define the goals of your research

Product and Research Leadership work together to define clear and narrow goals.

2) Agree evaluation method up-front

Product and Research Leadership work together to agree how the research output should ultimately be measured.

This evaluation method should be a reliable measure of your progress toward your objective. If it is impractical for the Researchers to use this metric during the course of their research, they should work with a proxy metric.

3) List interesting ideas

The Researchers carry out light-weight exploratory research and brainstorm interesting possible areas of investigation.

4) Draw up a short list of projects

The Product Manager and Researchers agree on which projects to write proposals for.

5) Write up proposals

The Researchers write up the selected proposals.

These should include:

  • A hypothesis;
  • What will be tested;
  • A high-level idea for how the research should be structured;
  • How long the research should be timeboxed for. The timebox should allow for a few iteration cycles.

6) Agree which proposals to move forward with

Key business stakeholders review proposals and, if appropriate, give them the green light.

7) Run experiments

Carry out the experiments, as set out in the proposal.

These should run no longer than the agreed timebox. Of course they can be stopped earlier - if you get positive results sooner or conclude that it’s a waste of time.

8) Evaluate the results

The team evaluates the results in the way agreed in step 2.

9) Plan new research projects

Return to step 1.