If you can’t get Data, Trust your Gut
In tech we make lots of decisions. Sometimes our decisions are backed by data. But sometimes when you’re trying to predict the future you may not have much to go on. You have to trust your gut. You should recognize that you’re making opinion vs data based product decisions. And this is fine!
Trusting your gut is scary. As Tony Fadell wrote in Build - “Many people don’t have either a good gut instinct to follow or the faith in themselves to follow it. It takes time to develop that trust. So they try to turn an opinion driven business decision into a data-driven one. But data can’t solve an opinion based problem. So no matter how much data you get, it will always be inconclusive. This leads to analysis paralysis - death by overthinking. If you don’t have enough data to make a decision, you’ll need insights to inform your opinion. Insights can be key learnings about your customers or your market or your product space - something substantial that gives you an intuitive feeling for what you should do. You can also get outside input: talk to experts and confer with your team.”
For me, defining the vision and strategy for AI Transparency and Control at Meta was an example of the time when we didn’t have much data and I had to trust my gut. To cope with enormous ambiguity, scale and lack of data here’s how I collected insights to arrive at a plan.
Gather, read and discuss the Academic literature on AI Transparency and Control. This meant reading papers and books (i.e. The Ethical Algorithm: The Science of Socially Aware Algorithm Design by Michael Kearns & Roth is my favorite)
Work with User Research to create and run studies to understand what users, regulators and engineers wanted to see in AI with respect to Transparency.
Talked to other teams in the company who worked on similar problems. For example the Facebook Newsfeed team built ‘Why am I seeing this Post’ feature and the Ads team built a ‘Why am I seeing this Ad’
Consulted with our Privacy, Policy and Legal teams and experts outside the company.
Looked at the Competitive Landscape
After crafting the Mission, Vision and Goals, in collaboration with my eng and design counterparts, David Adkins and Christopher Reardon, I created a portfolio of product and technology investments each sitting in a clearly defined strategic pillar. This was presented to leadership. For an in-depth look at how to create an AI Product Strategy, check out this article I wrote.
How do you execute?
Balance your Product Portfolio
Problems should align with current company priorities. You should have many small projects and a few large ones with clear measurable goals. Always look out for any work that creeps in but doesn't matter.
Small projects are important because they mitigate the risk of bigger ones. Small wins will keep the team motivated and create momentum. Doing only large projects is risky and may be too slow.
2. Minimize Risks
Have a Clear Data Story - do you have enough data, labeled data if you need it, update data use terms, set guardrails around data quality and governance, timebox your projects so that you don't go off rails.
Manage Dependencies - Understand and document dependencies - on other teams, company platforms and tools, availability of data, data annotation resources, etc.
Timebox your experiments and be clear about what decisions you want to make.
3. Set Clear Expectations - with leadership, your working team, partner teams
If you do 1-3 you'll be good? Well no. Certain people still caught the case of analysis paralysis and had to be argued off the ledge. We experimented with a product idea that was extremely valuable but got stymied by fragmentation of platforms and tools. But as any hardened tech veteran knows, "the best way out is always through" and today I look back on this time as one of the most exciting opportunities of my professional career. The team is still alive, kicking and executing on Vision and Strategy two years later. If you’d like to check out some of their work see Our approach to explaining ranking and Captum.ai. And big thanks to David Adkins and Christopher Reardon for being my partners in crime.