Marketing Goal: Support Improvements

[joethreepwood]

Context

I've been asked to (and want to) get involved with the support process, to see if there are some improvements we can make here and any lessons/connections we can bring over from other work, such as onboarding. It's not an overt marketing team goal for the quarter, but as #170 is winding down a little it feels like I can invest some time here.

I'm a planner, so I wanted to put a transparent issue out to plan the work and invite feedback/ideas. Especially important because this work doesn't typically happen on GitHub, but needs to be documented somehow.

Things I'm going to do

A bunch of ideas I've had, or which came up in context-gathering calls.

• Have a context-gathering meeting with Simon and Marcus
• Standardize existing macros for smoothers comms
• Clarified support ticket status posthog.com#7975
• Start a bi-weekly one-touch ticket update - in Content Ideas
• Introduce monthly/weekly reporting on key metrics
• Find out how (and start) tracking time to resolution, per team - Zendesk dashboard
• Investigate macro incident comms or alerting processes
• Investigate methods for telling users what support prio they are on
• Investigate methods for upselling low prio users into Teams plan
• Investigate frequency ranking for teams (more tickets in short time = higher prio)
• Community: Automatically move support questions from Zendesk to forum posthog.com#7961
• Investigate if we can use Yarn to improve the support exp ruled out, too big a lift for small a gain
• Collect frequent requests and turn them into docs improvements/new content
• Do some competitors analysis to see how we can specifically match Amplitude, Heap, etc.
• Auto out-of-hours responses Discussed and ruled out for now
• Set a no weekend support policy
• Setup business hours for 5PM to 9AM CET
• One dashboard to rule them all
• Shadow how Prod Analytics solves tickets
• Shadow how Feature Success solves tickets
• Shadow how Marcus assigns tickets
• Investigate one-touch %age by agent (see if Marcus is biasing any results)
• Improve the CS/Support Hero handbook to give better guidance
• https://github.com/PostHog/company-internal/issues/1332#issuecomment-2049197912
• Improve the escalation macros to set better user expectations
• Create a new macro for feature requests and on-hold statuses
• Request: Improving the support modal posthog#21073
• Clarify what the support plan for Surveys is (Support is covering FS now)
• Create a Support training process as part of employee onboarding posthog.com#8284
• Support: New response targets and SLAs posthog.com#8345
• (Re)activate support for free users, with new routing
• (Re)define SLAs in Zendesk after Support: New response targets and SLAs posthog.com#8345 merges
• feat: Update SLAs, remove GitHub buttons posthog#21818 (review)
• The Zendesk & Email Domain Problem #201
• Edit views to give everyone SLA visibility

Next steps and feedback

All feedback appreciated. For now, I'm going to try and churn through this plan and identify + add new ideas as I go.

[joethreepwood]

I've created a new dashboard for tracking time-to-resolution per team.

Median first resolution time over last 30 days, by team: 

Everyone: 42 hours
CS: 12 hours
Growth: 16 hours
Analytics: 25 hours
Infra: 31 hours
Replay: 53 hours
Pipeline: 53 hours
Feature success: 169 hours
Warehouse: 261 hours

[charlescook-ph]

Honestly these are all great ideas to try out! Really excited to see this.

Introduce monthly/weekly reporting on key metrics

I think an important part of this will be getting everyone aligned on what the actual definitions are we are using - ie. what does 'closed' mean. I think our reporting historically has actually made things seem worse than they actually are sometimes because (for example) a bunch of tickets that are open is because of a feature request, not because we've 'failed' to get back to someone.

[joethreepwood]

In that instance, I think the metrics that matter most are:

First reply time: The duration between ticket creation and the first public agent reply on the ticket.
Time to first resolution: Duration between ticket creation and its first resolution.
Time to final resolution: Duration between ticket creation and its most recent resolution.

These should be showing us how fast we are at getting back to people and solving problems, our accuracy in solving problems, and the overall user wait time.

Zendesk documents these mostly here.

'Closed' doesn't matter as much to me/users as 'Solved', so really we want to be focused on this metric. I've got a dashboard up to monitor it in Zendesk now and will start reporting on it into Slack.

[minekansu]

Median first resolution time over last 30 days, by team:

Tracking/reporting 90th percentile for these metrics could be more beneficial here. People at the tail end of the support experience are the ones who are upset/at risk of churning/wouldn't engage with CS. And by trying to improve the 90th percentile will automatically improve the median.

[minekansu]

In line with the "analyze the extremes" theme, it could also be useful to look at top 1% support experiences to see what went well. This approach is more qualitative but can provide clues about what works (this is how Zappos did support reviews)

[joethreepwood]

I've created a new bi-weekly dashboard to track the most essential data I think we need to focus on now. The goals I think we should focus on are...

• Increasing the % of one-touch tickets
• Improving the first reply time across all teams
• Hitting our SLAs for first replies

This would ensure we basically communicate faster and more efficiently, and that we keep our promises. This is the minimum we should be holding ourselves too. Image below covers the last 14 days.

A couple of things immediately jump out on this dashboard that I think we can take action on, and which I'd love to get feedback on from the relevant people. I'll break them apart as follows.

We breach SLAs a lot

We can't trust this figure currently, however, because it's likely skewed by our business hours setup. We don't have, as far as I can tell, clear business hours setup and no triggers for telling users that their request is out of hours. If someone raises a ticket on late Friday, Saturday, or Sunday and we don't respond until early Monday? That's an SLA breach. 18% of tickets are created on a Friday, Saturday, or Sunday.

We're hiring for another support engineer already, which should help. However, we should also consider clarifying OOH rules in Zendesk, communicating this to users, and an escalation path for critical issues. Can we push critical issues to someone, even at the weekend and evenings? Probably good to hear from @charlescook-ph @simfish85 @MarconLP here.

Product analytics gets the most work

This team gets more than 2x tickets than anyone else, but has the same level of resource to solve them. They also manage to solve more than 60% of tickets in one reply, which indicates these should be easily solvable tickets. They have a low breach SLA rate, and a fast response rate. What this team does, works. I'm going to shadow someone on this team to see how it's done.

I'd love to know from @mariusandra and @MarconLP if this team works differently to other teams in support, or if it would be worth assigning additional resource here. I'll also dig into the tickets this team gets to see if we can make better use of macros, or content, to solve issues.

Feature success isn't doing as well

Excluding the (somewhat exceptional) Web and Infra teams, Feature Success takes the longest to send a first reply, the longest to reach any resolution, and breaches the most SLAs. But they also have a high one-touch %age, so the issues can't be that complex?

My suspicion is that there's a process problem at play and that this team either isn't prioritizing support, or isn't treating tickets correctly. Would be good to chat about this with @MarconLP and @neilkakkar to see if there's some quick training needed? I'm going to shadow someone on this team too to see how it's done.

[charlescook-ph]

On the SLA breach stuff, is there a smart way to track this in Zendesk so it's only counting working day response times in terms of when the targets are met?

Given the vast majority of our paying customers are using PostHog in a business context, I don't think there is a need for us to worry about weekend/OOH support at this stage - if we were a consumer product, this would be different, but there probably isn't that expectation in B2B SaaS land.

[joethreepwood]

On the SLA breach stuff, is there a smart way to track this in Zendesk so it's only counting working day response times in terms of when the targets are met?

Going to look at this next, but that'll basically mean what I allude to: setting business hours. We can probably do a simple version of this (e.g. No weekends), but that won't solve the whole issue. If a ticket gets created at 6PM GMT then it still may not get looked at within the SLA timeframe.

Given the vast majority of our paying customers are using PostHog in a business context, I don't think there is a need for us to worry about weekend/OOH support at this stage - if we were a consumer product, this would be different, but there probably isn't that expectation in B2B SaaS land.

Agree we can cut weekends as it's a low number of tickets, but OOH is still worth considering I think. Timezones make a big difference if it's just Marcus looking at many tickets.

(Not suggesting anything certain here until I get stuck into the numbers more though).

[joethreepwood]

Standard hours now set: https://posthog.slack.com/archives/C01MGUHFH6G/p1710330676365379

[joethreepwood]

@MarconLP Super rough at the moment, but I've started speccing out some ideas for macros I could add to speed things up for you. If you get chance, please take a look and add (or remove) any further ideas!

https://docs.google.com/document/d/1HRgTvi7oqGL5oOVI6NXY6E_anMbOc_h31yIQRKdwjTk/edit?usp=sharing

[MarconLP]

Can we push critical issues to someone, even at the weekend and evenings? Probably good to hear from @charlescook-ph @simfish85 @MarconLP here.

Currently, the entire CS team is responding to 1-hour response tickets through the #support-high-priority channel. That means we can cover the entire US and EU day.

[joethreepwood]

I shadowed some support heroes and found some new ideas.

Discussion of shadowing findings: #189
New actions: added above

[joethreepwood]

• Clarify what the support plan for Surveys is

Discussed this with Neil. The plan is that Surveys support is now only in 'maintenance mode'. FS team will fix any outstanding issues, but we're saying no to all feature requests.

[joethreepwood]

I've had some back and forth with Zendesk over the last few days. Unfortunately, they say it's not possible to use the number of tickets a user as a trigger. That means we can't do any sort of frequency ranking for tickets/users in terms of either priority or upselling them to a Teams plan, or just entering a card. Sad times, but also probably for the best given the feedback on the latter idea.

Crossed them off the list, on to the next thing.

[joethreepwood]

Investigate methods for telling users what support prio they are on

Improve the escalation macros to set better user expectations

One thing I'm considered to better hit both of these points is to update the automations which assign users to a priority.

• For users who are assigned not paying or Low priority: Automatically tell users they can ask these questions in the community and that we can't provide support for non-paying users due to volumes. If you're a paying user and you're contacting us with a personal address, use your work one. Include to SLAs in case a bill is unpaid.
• For users who are assigned Medium priority: Automatically thank users for the report and confirm that we'll aim to respond within our SLA, but that we do not offer weekend support.
• For users who are assigned High or Critical priority: Automatically thank users for the report, confirm that we've tagged this as a High/Critical priority ticket and will attempt to respond within our SLA.
• For billing issues: Thanks, we will get back to you ASAP.

This would be an automation and so would go into effect immediately, so eager to get your thoughts before I set it up @slshults @MarconLP !

[joethreepwood]

I did some research into how Amplitude, Mixpanel, Heap, LaunchDarkly, Matomo, Plausible, and Sprig compare to us in terms of their support channels, coverage, SLAs and free user support. Sheet here: https://docs.google.com/spreadsheets/d/1k4-j8lAA7PigkeLQBPZpomP20epjyZY0T9TT_vuSVg8/edit?usp=sharing

Idea here was to help us think about, if we treat support as it's own product with a revenue line dependant on the Teams add-on (as @jamesefhawkins recently suggested), then what opportunities are their to differentiate our offering.

Here's what I concluded:

• Who has the best support offering?
  LaunchDarkly. They have the most comprehensive set of features and training materials available, the best SLAs, and the best operating hours. Mixpanel is a close second. Amplitude trails in third.

• Should support generate revenue directly?
  Both Mixpanel and LaunchDarkly are able to do more because they treat support as a revenue generator, with explicit paid add-ons just for premium levels of support. Amplitude also does this. There's definitely a strong precedent and case for treating support as a line-item (regardless of whether it makes profit).

• Should we focus on 24 hour or out of hours support?
  Nobody offers 24/7 support for free, or 24/5 support without generating revenue from support. Based on this, I don't think there's a strong need to hire more engineers with out-of-hours support or broader timezone coverage in mind. This is further supported by analysis of when tickets get made for us.

• How do our SLAs compare?
  With the exception of Heap, Mixpanel, and LaunchDarkly in a few cases we are on-par or promising above most other competitors. Given our SLA performance there's an argument here that we could arguably ease up on our SLAs a little and still remain competitive. If we wanted to settle for less.

• Should we support free users?
  There's no strong consensus here. Some competitors do, some don't. Very few have SLAs for free users though. Based on everything I've learned about support over Q2 so far, I think we're fully justified in not offering support to free users given the level of support they require and how competitors deal with similar situations.

[joethreepwood]

Support auto-responses now set up. https://posthog.slack.com/archives/C01MGUHFH6G/p1713870998047369

[joethreepwood]

Proposing new response targets: PostHog/posthog.com#8345

[joethreepwood]

New modal changes merging in: PostHog/posthog#21818
New SLAs agreed: PostHog/posthog.com#8345
New dashboard updates: https://posthoghelp.zendesk.com/explore#/new-dashboard-builder/50115001/

This closes out most of the work I had planned for support now, so I'll stop meddling there and shift over to #193 by having a kick-off call with Mine tonight.