Industry Conference 2015

Created by Emily Porta for Shift Health

Talking Points:

What is Industry Conference?

Two day conf in Cleveland, Ohio for product managers, founders, designers, developers - anyone interested in making and selling tech products.

One track: build, launch, scale

Probably ~150-200 people in attendance, mostly product managers and founders

What is Industry Conference?

Overall Impression

Good things:

• Lots of friendly people
• Several useful talks for someone new
• Variety of topics was good, some very practical talks
• Didn't miss anything: one track

Overall Impression

Not so good things:

• Didn't miss anything
• Inspirational, subjective "personal experience" talks
• Only one PM actually spoke - too focused on founders/founding

Day One: Build and Launch

Key takeaway:

Take in more data and build products based on client needs/wants both known and unknown to them

Nir Eyal - what makes some products so engaging?

"Habit forming products"

You want to make products that form use habits in customers to:

• increase customer lifetime value
• increase pricing flexibility
• increase defensibility (in tech, no rule that says the best product wins)

How do you form habits?

Habits can be manufactured - but it's hard. We need to have a good idea of customer needs they can't articulate. Look to consumer psych for answers.

The Hook

An experience designed to connect a users problem to a solution with enough frequency to cause habit.

Four parts:

Sabrina Gordon - Why Support Conversations Should Improve Your Product

Focusing too much on personas can make you miss why your users are trying to solve a problem.

Temptations to deal with scaling support:

• Triage: VIPs get great, tailored experiences
• Busy out: ignore customers until they ignore you
• Burn out: work current team to the bone. Leads to turnover, leads to organizational amnesia.
• Outsource: including marginalizing the internal team, too.

Instead, have a support strategy

One that you follow no matter your size, where the support team is constantly in conversation with the product team.

Support can give high quality, meaningful data to the product team so they can 1. Make decisions (Product Manager) and 2. Understand the importance of what they're building (team).

UX and Support

From support teams you get unsolicited feedback, which is good because: it’s workflow driven, from an open conversation, and hits at blind spots you didn’t know you had

UX and support data should both be used, the data complementing each other.

Support team should be solving the customer’s problem while also solving the company’s problem.

How to Give Feedback to Product?

This can be tricky. Avoid frequency bias: thinking whatever's just come up is the most important thing - try not to react too quickly.

Instead, look at data incoming over time: have a reporting tool where everything comes in, and tag them.

Intercom's internal tagging

• Unaware
• Confused
• Feature Request

Unaware (of feature) and Confused (UI not clear, not sure about a feature) should be looked at frequently to see how you can improve your communication.

Feature requests represent the customer voice - the PM can use these, balanced with product vision, to make product improvements.

Maggie Jan - Data-Driven Product

Build/Measure/Learn is hard to do when you don't measure or learn: enter analytics.

Analytics allows you to adjust your product fit faster in response to real feedback.

A Good Metric is:

• Explanable
• Comparative (maintains context)
• Meaningful

"A good metric is behaviour-changing"

Applying Analytics to your business

Collect event data:

• the action the person did
• state - who is the user
• time - when they did it, active users per day

Visualize data:

• internal dashboard
• website
• heatmaps

Common Mistakes

• Confirmation Bias
• Leading the Witness
• Correlation vs Causation

To avoid these, involve more than one person in data-gathering, and be disciplined about how you capture your data.

Chris Spiek - jobs-to-be-done

Discovering why people buy your product and why they fire it.

Has a mentor who always knew what to put in the product, while everyone else was feature-creeping based on customer feedback without a vision.

Jobs to be Done

A methodology inbetween the solution and the job (or the what and the why)

Products are services - customers are trying to solve something in their life by using your product.

How do we make product decisions today?

"More peanuts or less?"

"Melting point of chocolate?"

"Single or two pack?"

This is a dumb way to think about things. No one buys a chocolate bar because they literally need two chocolate items in one package, or they need something with "some" peanuts but not too many.

Milky Way vs. Snickers

The two aren't in competition: Snickers is thought of as "energizing" and solving hunger. Milky way is soft, endulgent - you eat it when you need comfort.

Once you understand the job (solve hunger, gain energy, feel better emotionally) making the product becomes obvious.

Two JTBD tools to understand why people buy your product:

The Timeline is the process a customer goes through to get to buying and its result.

The Forces are what acts on a customer whenever thye're buying something.

The Timeline

Ask your clients what they were experiencing when they decided to go with your product, and do it soon after they made the choice.

The Forces of Progress

The four forces acting on a consumer whenever they're buying a new product. Two promote choice, two block change:

When the top two are greater than the bottom two, people will make a decision to buy.

How to find out how your customers choose a solution?

Do interviews with people who have purchased your product and people who've fired it.

Ask them to tell you the STORY of how they made the decision to buy or not buy. Do not talk about the product itself.

Find the energy in what they're saying - when they're energy spikes, go after that information.

Day Two: Scale

Key takeaway:

Moar data

Richard White - Customer Feedback: The Other Half of the Data Story

Uservoice surveyed 300 Product Managers on how they do things.

Product management today: how do we decide what the right product roadmap is? We get in a room and make it up.

Who has the most influence over product roadmaps? PM's, C-levels, and then everyone else.

Data "revolution"

It's pretty slow. But gradually we're using customer data to make more decisions. Customers have a lower and lower tolerance for bad digital experiences, and executives know this. PMs get pressure from them to change.

But how do we get the customer feedback to learn why what is happening is happening?

Direct from the customer

• 52% of PM's said they don't currently have a way of getting direct feedback from customers :|
• Lots use excel - this isn't scalable
• surveys - but response rates are low
• Text analytics - from all incomming qualitative data streams. Advanced, you get trending keywords, etc. Basically just stuff.

Result: "some customers are talking about billing a lot."

What does work?

Online services - feedback out in the open (like uservoice - surprise!)

Indirect, through sales and support - support is an excellent source of feedback, they have a huge interest in improving the product.

But tagging/categorization, or just conversations randomly occuring aren't good enough.

How can we do this better?

• link issues to the route cause - this is what we’re doing at salesforce. This is a billing issue, but what kind of billing issue is it? link the tag to the actual issue. eg. “billing issue > didn’t have mastercard” instead of just "billing issue”
• sales guy loses deal because we didn’t have a feature, every time that happens it goes in and we can see how many people request things

Then what?

Now that you have contextualized user feedback, what do you do with it?

We can start integrating it with other data: revenue, spend, longevity of customer, happiness, etc.

Result: "500 customers have asked for this, they represent 15% of revenue they’re 10 of my top 100 accounts, etc. etc. etc."

Product Management as Portfolio Management

With this feedback, you can start to make decisions like:

"This quarter we’re going to do 60% slam dunk stuff, 10% totally innovative, 30% inbetween, etc."

Because you know who the customers are, what the most important and numerous ones care about, and where the potential growth is.

Aviran Mordo - Advanced A/B Testing

Experiment Driven Development: Wix did over 700 experiments on their users just in August 2015

What is A/B testing?

Having two versions of the same thing you want to test out, and see which one works better. A is the existing version, B is the new change.

Collect this data, then compare the two options, once you reach statistical significance you stop the experiment and scrap what doesn’t work

WixPETRI - manages their AB testing: github.com/wix/petri

How to do it?

When you start an AB test, start with 10% of the users, measure how they react to this new feature

Then slowly start to increase the percentage if numbers look good for the B

This can take a long time to reach statistical significance.

Final Thoughts

Product Management has a long way to come.

Most are either founders/co-founders, or more senior people from other areas who moved over.

PMs who focus on collecting, analyzing customer data and responding by building products with that data in mind are very valuable.