Working with tech companies to build better developer experiences. Specializing in building and renovating developer communities. MBA, certified in change management. Expertise in developer education, documentation, developer relations, product management, and developer community strategy and management.

The Scientific Method and Community Metrics

Photo by Isaac Smith on Unsplash

One of the recurring topics in the community manager network is how to measure success and “prove value.” As always, I have Thoughts™️ on this topic.

Here’s the TL;DR version:

1) Align executive expectations on outcomes. These form your “north star” objectives.

2) Write your hypothesis on how best to measure and support progress against those objectives.

3) Define a limited set of measurable and actionable key performance indicators (KPIs) that will help you course correct, if needed.

4) Develop an action plan for when things go unexpectedly.

The Grand Experiment of Community Building

Successful community building is, in many ways, a scientific experiment. While communities of humans share many broad characteristics, each community has its own inner culture of acceptable behaviors; shared interests, needs, and desires; and expectations. As a result, community building is a new experiment each time you join or start a new community. With experience, community builders develop a personal framework of community building that we are familiar with. We may know what generally works and what doesn’t for a particular type of community, but we can never operate with 100% certainty that a given program or action will impact our community in the way we expect.

Setting Up the Experiment

With your executive team’s input, start by developing a hypothesis about the impact of a successful community on the business. This often takes the form of a vision or mission statement. Here are a few examples:

  • “Our community will facilitate peer-to-peer knowledge sharing that will reduce our support costs.”
  • “Our community content will be valuable and SEO optimized to improve our brand reach and recognition amongst developers.”
  • “Our community will host customer-only exclusive events that connect our customers with our internal experts, increasing product adoption amongst our customers.
  • “Our community will help our end users find answers quickly, increasing overall customer satisfaction and renewal rates.”

Note that these hypotheses focus on the community value add that will impact a particular business metric that is important to the company and that the leadership team expects to see to consider the community a success. This hypothesis should be a result of discussions with that same leadership team in which you ask them to answer the questions: “Why do we want a community?” “Why did we choose to pursue this?” “What do we expect to see from a successful community?”

🛑 Do not move forward until you have reached consensus amongst your executive stakeholders on the purpose of the community. Trying to develop goals and plans without the agreement amongst your leadership team on what the intent of the community is will set you up for failure and frustration in the future.

💡 If you are struggling to achieve consensus, this is also a good time to identify your executive sponsor or the person who you can go to to rally their peers and subordinates in support of your project.

Document this consensus via your hypothesis statement. This is a good time to start a centralized repository of information about your project that is visible publicly within your company.

Designing Your Methodology

Once you have your hypothesis, this will serve as your Objective (the “O” in “OKR”). Think about how you will quantify your progress towards, or achievement of, this objective. The “how will you quantify this?” component is the Key Result (“KR” in “OKR”) that you will use to communicate to your leadership team on the success of your program. The Key Result should align with the business outcomes you identified in your hypothesis, these might look like:

  • “Decrease support ticket volume by X%” (aka “ticket deflection”)
  • “Increase new developer members in our community by X%”
  • “Increase feature adoption by X% of existing customers” or “increase user licenses by X% at existing customers”
  • “Increase NPS by X% year-over-year” or “Increase customer satisfaction survey ratings by X% month-over-month”

From your Key Results, you can start to develop your working metrics. These are the core key performance indicators (KPIs) that you will build a reporting dashboard around and that will help you monitor the pulse of your community program. Your KPIs must give you a specific course of action, should they trend in the wrong direction or should the community respond to your actions unexpectedly. KPIs are focused on establishing the cause-and-effect of explicit actions that you’ve taken to stimulate a certain result from your community. This might look something like:

  • Time to first response, percentage of questions with answers, rate of return by new members
  • New member growth, increased mentions of the company/product in tracked channels, increased positive sentiment amongst developer channels, increased referrals to trial downloads
  • Increased number of new questions, increased percentage of questions answered by community members/non-employees, increased percentage of product ideas submitted by community members, increased attendance at product-focused activities (events, meetups, talks), increased participation in beta testing/collaboration opportunities between the community and the product team(s)
  • Increased rate of return for new members, increased ratio of daily active users:monthly active users, increased positive user sentiment

Of course, we all know that the community often does several or even all of these things, so how do you prioritize? Again, focus back to your hypothesis. What is the priority for the business leadership team? What is the priority for your executive sponsor? You should look to your leadership team to help you determine what the priorities are for the business.

❗️ Select no more than ten KPIs to track and report on.

Remember that your KPIs should be actionable and should give you direction. This means that you should be able to directly influence those measurements by taking action. And, if you have too many signals, they become overwhelming noise that is impossible to parse into real, actionable next steps. So, hone in on your hypothesis and keep your scope tight.

💡 You should always be able to answer the question: “Why are we tracking this and what will we do with the information we gather?”

Run Your Experiment

Any community builder will tell you that community building is a long game. Set up your executive team and stakeholders with the appropriate expectations for the amount of time it will take to see results. This should be addressed early on in defining and getting buy-in for your hypothesis. It could take up to a year to establish the foundational elements of a new community. You’ll need to give yourself and your community time to respond to any initiatives that you implement. Setting these expectations early will help to keep you from feeling stressed, pressured to deliver results prematurely, and making bad decisions down the road.

Your roadmap should directly tie to the KPIs you’ve established, with each action and each task designed to influence them. One way to ensure that this happens is by using a project management tool like Atlassian’s Jira, creating Epics, or themes, for each KPI and assigning your tasks and initiatives to those themes.

Telling the Story of Your Success

Check in on the community program’s performance against your KPIs regularly – but not too often. Remember that community building is a long game and we’re looking for trends over time, not transactional and short-term blips. I’d recommend reporting on your KPIs once a month. More often than that and you’re likely to make deductions and assumptions based on limited data. Less often and you may end up wasting time and resources on tactics that simply do not work for your community.

Plot your events and activities across your trend lines to see how what you’ve been working on impacts your KPIs. This will help you write the narrative of your successes (where the trends and events intersect) and your learnings (where the trends and events don’t seem to have any correlation).

Wherever your activities correspond to a spike in your trend lines, you may choose to dig in deeper (if you want to be extra fancy, you could do a regression analysis if you have a skeptical leadership team or you just want to understand more of the factors that might influence your outcomes for better or worse. But at a minimum, do more of the activities that make your trend lines go the right direction.)

Interpreting Unexpected Results

So, what do you do when you see an outcome in your trend line that you don’t expect? Maybe the trend goes opposite the direction than you expected or maybe there is no measurable impact at all. The chart above could represent a fairly standard first-year trend for a brand new community for an existing customer base. What if we added a few anomalous events to these metrics?

💭 The anomalies here occur in January, April, August, and September (and possibly December). What do you think might be happening here?

If you see an unexpected result from a planned activity or event, there are a couple of possible causes that you should explore:

  • Not enough time for the impact to be clear. If your KPI is actually a lagging indicator, there may not be a linear correlation or immediate impact from your activities. For example, is it reasonable to expect immediate increases in your SEO ranking compared to a well-established site, like Stack Overflow, if you’ve made some changes to your own site? Probably not. So, you shouldn’t expect to see a major increase in referrals from search results for particular keywords in the same month that you’ve made the UI changes designed to impact SEO ranking.
  • Your tactics are not aligned with your community’s motivations. I explain a bit about intrinsic vs. extrinsic motivators in my talk about psychological safety, but this topic is worth exploring more deeply if you’re unclear about what motivates your community members’ behavior. A lack of understanding here will create bigger problems for you in the long run and should be explored as soon as possible in your planning of any new initiative.
  • Other factors influencing your outcomes. Whether your activity or event is drowned out by an even bigger news story or announcement, your members are dissatisfied with your company for reasons outside of your control – like a bad customer support experience, or something as simple as a regional holiday or cultural event (e.g. summer vacation in western Europe) could cause a disconnect between a tactic that would otherwise resonate with your members and the results you’re looking for. When researching anomalies, be sure to rule out any external factors that could have skewed your results.

💭 In the chart above, the easiest anomalies to eliminate would be the launch (January), European summer months, and end-of-year holiday season. You could validate this by checking the year-over-year trends during that same time period. And this, of course, would depend on the regional location breakdown of your community members. The next step would be to compare the trends against any known external factors, such as company or industry events, breaking news stories, new feature releases, etc. which may have brought more attention to your community than usual. Lastly, check against your own initiatives to determine which tactics had the most desired (or most undesirable) impact on your KPIs, such as webinars, meetups, email newsletters, UI changes, discussion prompts, and more.

💡Remember that your KPIs should be providing you with actionable intelligence, meaning that there should be clear and explicit actions to take whether the metrics prove or disprove your hypothesis.

Adapting the Roadmap as a Living Document

The most important element of the scientific approach to community strategy planning is the willingness to adapt based on new information. Your strategy should evolve as you begin to compile more and more data to ensure that you are not wasting time and resources on tactics that are not strategically beneficial to your community and your company. Keep going back to your hypothesis to check and see if:

  1. it still reflects the expectations and needs of the business
  2. it is supported by your evidence

If not, it’s time to revisit and revise with your stakeholders. Admitting that your hypothesis was wrong, your plans didn’t work out, or your assumptions were inaccurate can be a tough conversation to have, especially with your management team, but ultimately you are working in the best interests of your company and your community by leaning on data to make smart, educated decisions about how to best serve your members.

Why aren’t more community managers talking about data privacy?

Photo by Joshua Sortino on Unsplash

It continues to surprise me how many of my fellow community managers are either resistant to, or just plain disinterested in, topics surrounding personal data protection and privacy. We own the most visible and vulnerable centers of non-compliance when it comes to legislation like GDPR, California’s CCPA and CPRA, and similar laws enacted in Colorado and Virginia (and active legislation moving through the U.S. state governments in New York, Massachusetts, North Carolina, Minnesota, Ohio, and Pennsylvania). Non-compliance with data privacy legislation, especially GDPR, is a very expensive prospect if the offenses are egregious and numerous. GDPR violations can cost a company up to 20 million euros or 4% of their global revenue per year per violation! (The biggest fine doled out so far has been Amazon’s whopping 746 million euros in 2021.) Yet, many community manager groups and professionals are hesitant to talk about data privacy as a core aspect of their jobs.

When I was at SAP, I led the team responsible for implementing GDPR compliance for a community of just under 3 million users, many of whom are European Union citizens and thus, fall under the protections of GDPR. As a tech giant in Germany, SAP is held to the highest standards of both good and ethical business practices by the European business and tech communities, as well as by the governing bodies of the country and the EU (not to mention its own internal compliance and ethics teams). What this means is that under intense scrutiny, the team and I, in partnership with our legal and data protection and privacy offices, had to figure out how to navigate an intensely complex technical ecosystem to implement both policies and platform features that were compliant and also user-friendly. As non-engineers and non-lawyers ourselves, this was a very daunting task and it took a lot of research, learning on the job, and collaboration. That said, when it was all said and done, I realized that I had developed a deeper understanding of what it truly meant to build a user experience through Privacy by Design.

I took this experience with me as I ventured out into the broader world of developer community work, annoyingly becoming that person in every conversation who questioned how the user data was handled, how it was anonymized and protected, how the platform handled compliance with user requests. What I found as I started thinking about getting back into more content creation for the community manager audience is that there are surprisingly very few of us who have had this experience. I’m not sure if it’s because most of my fellow community managers have worked solely in U.S.-based tech companies (I know this isn’t true…) or because it’s never been put on their shoulders to manage, but it’s concerning to me as the community profession develops rapidly in the wake of the pandemic.

If you are utilizing a vendor-built platform (think: Khoros, Salesforce, inSided, Vanilla, Discourse, etc.) many of the features and functionality to ensure GDPR compliance are built into the platform. Yay you! But you still need to handle the process side of the equation. Who handles user requests? Where do they come in? How are they processed? What are your policies regarding data retention? Do you know how to manage requests for data that goes beyond just your platform? What about your other data processors? Does your user data flow into a CRM? A digital marketing platform? An analytics dashboard? Is that data anonymized? How do you handle Right to be Forgotten requests in a public-facing community? What are the limitations to your responsibilities? Is your privacy statement updated to cover community engagement? These are all questions that community managers should be equipped to address.

There are lots of resources out there to help you learn about these various legislative requirements, but the deeper question I have is: why do we think this is not our responsibility? I consider myself a caretaker of my members’ data. It’s why I generally don’t share it with marketing, sales, or any other function of the business that my users did not consent to (and that creates plenty of tension with those teams!). Have many community leaders shelved this issue in the “future problems” bin? Are organizations putting these compliance measures in the hands of “Trust and Safety” professionals instead? How does that work into ownership of the community member experience? Where does the community leader fit into that model?

Would love to hear your thoughts in the comments.

How to Hire Community Managers

Microphone in focus with colorful abstract image out of focus in background
Photo by Michal Czyz on Unsplash

Tech industry community managers are hot on the market right now, with an exploding number of new job postings – particularly with companies that have never invested in community building in the past. There’s a need for empathetic people-people who can help build strong relationships with a business’ customers in a pandemic world where in-person interactions are severely limited. Tech companies are starting to realize that community managers neatly fill this need.

That said, I’ve heard from many hiring managers that they do not know how to hire a community manager. They lack the understanding of the role to effectively write a job description or conduct an interview process that will ensure that they end up with a candidate who can succeed at achieving the company’s needs. With many years of experience working in communities from enterprise customers and business users to individual developers, here are my insights into how to effectively recruit community managers in today’s market:

  1. Understand WHY you are hiring them. Community managers do a lot of emotional labor when it comes to educating and building buy-in into their day jobs. In a single day, they may go from executive presentation to playground monitor and back again. It’s exponentially more challenging to stay energized and productive if they are also having to justify their job role to an executive team that is unclear on their expectations for the job. This is setting your community manager up for failure.
  2. Be clear about the job level and how it will meet your company’s needs. Once you’ve identified the needs you’re trying to meet, be sure to hire someone at the right seniority level to meet those needs. If you’re building an entirely new community from scratch, hiring a junior level community manager, or someone for whom this is their first community role, will also set them and the community up for struggles.
  3. Understand the demands of the role and pay them accordingly. Give your new community manager the appropriate title and pay that matches their role and level. It will help if you are up front about this in your job description and allow prospective candidates to self-select out of the recruitment process if the role is too junior or senior for their experience. Understand that community managers are not just social media managers by another name. They are not digital marketing managers. They are not content strategists. Community manager is a broadly defined role that encompasses many skill sets. Especially on a small or one-person team, this person may be expected to know: customer relations, corporate communications, user experience design, product management, project management, program management, event planning, email marketing, social media marketing, crisis communications, release communications, data analytics and business intelligence, and more.
  4. Remove friction in your interview process. Don’t make community managers perform arbitrary tasks or jump through the proverbial hoops to navigate your recruitment process. Be up front about your recruitment process. Don’t change mid-stream. Communicate regularly. Talk about requirements for salary, priorities for their next role, and how they see this role evolving over the next 5+ years. Don’t ghost them. The DevRel/Community Manager community is small and interconnected. People will talk.
  5. Don’t ask them to work for free. Don’t ask community managers to do homework – especially if this takes the form of writing up an entire program proposal and launch plan. Community strategy should be built around the needs of each individual community. There is no one-size-fits-all solution that can or should be blanket applied to every community. Your new community manager should spend their first 30-60 days getting to know your community, your company, and their stakeholders, integrating their needs and expectations into their community program plan, and researching the resource needs and availability to execute on that plan. They should be prioritizing tasks to build a roadmap for the future, based on the limitations and priorities of their stakeholders and their organization. Only then can they give you what you’re asking them to do with extremely limited information.
  6. Look for growth potential. If you are hiring a junior or mid-level candidate that will work with a more senior community manager, recognize the potential for growth. Many characteristics that make community managers unique are specific to their characters. Presentation skills, data analysis, technical knowledge, and other tactical skills are teachable. Empathy and intuition are not. Hire the right person, then train them.
  7. Move quickly. As mentioned, community managers are a hot commodity right now. If you delay and drag your feet through the decision process, you’ll lose your shot at the best candidates.
  8. You have limited chances to get community right. Your community is a highly visible investment in your public image, your relationships with your users, and the long-term success of your business. The more times you trip up in this process, the less faith and loyalty your community will have for you. The risk is high. The rewards are higher and longer lasting.
  9. Give your community manager time to succeed. Community is a long game. If you want to succeed, you need to understand that success for community is not measured in leads, conversions, or sales. We measure success in the long-term relationships we build with our users, who then go on to bring us with them throughout their careers. They become our advocates, our partners, our cheerleaders, and our best critics. This isn’t something that will happen within one month or one year even. Remember: trust is hard earned and easily lost.

In short, be decisive, know what you’re looking for and why, and know what you can offer, so that when the right candidate falls into your lap, you don’t miss out on the chance to build something impactful and amazing for your organization.