Collier’s Learning Report, 2018

Between Christmas and New Years Eve, I took a few hours to look back on my 2018. I reviewed my time sheets, my projects, my business development tracker, my calendar, and my journal. I reflected on my wins, my challenges, my values, and my goals both for my business and for my personal life.

As we are settling into the new year, I’d like to share some highlights from my Annual Review with you. My hope is that this essay encourages you to take the time to reflect on your last 12 months, and gives you an honest look into my life at the moment.

This year, I’m structuring my annual review as answers to the following three questions:

1.       What is the story of my life last year?

2.       What did I learn from pursuing my goals?

3.       What am I working towards this year?

What was the story in my life last year?

From all that review, if I could put my 2018 into a few words, those would be “All In”.

From mile 14 on a hike in the Gore range this summer, near Silverthorne Colorado

From mile 14 on a hike in the Gore range this summer, near Silverthorne Colorado

There were no monumental changes in my life in 2018 – no international travel or cross-country road trips, no moving or switching jobs for my wife or myself.  The year was filled with long work-weeks, broken up by skiing, hiking, and construction projects on weekends. In short, making progress on the projects I started in 2017.

My home office is less quiet now, as my wife and I welcomed a rescued puppy – a border collie named Lucy – into our home last October. I’m also commuting to client meetings in a new truck, a huge improvement from the beat-up Dodge 1500 van that served double duty as a camper and an office-on-the-road.    

My business has been successful, and has pushed me to grow in great ways. Being sound financially as an independent consultant is no small feat! I worked with 20 nonprofit clients in 2018, and gained traction with some new types of projects. I’ve also struggled mightily to keep up with my workload, and making quality time to spend with my wife and friends has been a challenge I haven’t yet solved.

What did I learn from pursuing my goals?

At the beginning of 2018, I committed myself to several goals. I learned so much from pursuing them, and I’ll share those learnings here. For each goal, I also gave myself a rating from 1 (completely failed) to 10 (hit this goal out of the park).

Goal 1: Become as professional about developing my business as executing my client work.

Score: 9/10. The first few months of 2018 were incredibly stressful, as I had little work and no pipeline of new consulting clients. I realize now that I under-invested in networking and other business development activities in the fall of 2017 due to several major overlapping projects. In 2018 I began tracking all of my business development activities, and set a monthly target of 8 one-on-one conversations with potential clients. I also began pitching training topics to relevant conferences and associations. My sales habits and pipeline are significantly stronger now – there’s still so much to do, but I’ve made some progress.

Goal 2: Win and execute 150+ hours of analytics and evaluation consulting work.

Score: 10/10. I thought this goal was in the bag in January 2018, when I was a finalist for an evaluation contract with an education organization. That didn’t pan out, and it took months until I had a conversation with Banks Benitez at Uncharted about revamping their impact measurement strategy. That eventually matured into my biggest and most rewarding consulting project of the 2018. 

Goal 3: Significantly increase my statistics knowledge.

Score: 3/10. I work in a field where most professionals have graduate degrees and formal training in statistical analysis, so I know I need to fill this gap in my knowledge. In 2018 I decided to transition from completing online courses (which I would invariably stall on) to reading through books to increase my knowledge. My intent was to get through three books – I made it through one during the year. The problem here is that I intend to allocate an hour each morning (6 – 7 am) for personal learning. However, I wasn’t disciplined in my sleep schedule over the summer so getting up for this was a struggle. In the fall, I was better about getting up but had so much billable work that I needed to use my mornings on client projects instead of learning. Reclaiming my mornings for learning is one of my priorities for 2019.

Goal 4: Take up a weekly, social hobby for at least 3 months:

Score: 6/10. Technically, I fully completed this when I joined one of Mile High Ultimate Frisbee’s spring leagues. Participating in that league was a blast, but with summer travel and then a very busy fall of work, I just didn’t have time to participate in another league. Moreover, I realized that this goal didn’t move me closer to my ultimate aim of building a better community of friends in Denver. I didn’t have any prior connections with my teammates, and I realized a once-a-week sports league for three months isn’t enough time to build meaningful friendships. This next year, I need to focus more of my time with existing friends and acquaintances here in Denver.

Goal 5: Build muscle (target weight = 170 lbs.)

Score: 7/10. I actually lowered my sights for this goal over the year, from a target of 170 lbs. to a target of 165 lbs. (I started the year at 159). Despite that, I feel good about my progress. I’ve struggled to gain strength for years. It took a lot of discipline to make progress in 2018 – sticking to a workout routine, recovering from injuries, trying to get enough sleep to adequately recover, and trying to eat enough while keeping my diet healthy. In this process, I came to terms with what I would need to do to gain more weight: I would need to up my sleep to closer to 9 hours a night, eliminate alcohol from my diet, and increase the length of my workouts to ~90 minutes three times per week. It would be very difficult for me to fit these into my current lifestyle, so I’m comfortable with just maintaining the progress I have made.

Goal 6: Take 5 digital-detoxes throughout the year (defined as 4 consecutive days offline)

Score: 9/10. My digital-detoxes were: March – skiing in Tahoe (4 days); July – trip to Chicago (5 days); September – College friends’ reunion in Arizona (4 days); November – Thanksgiving (4 days); December – Christmas through New Year’s (10 days). I felt so much better about my work, and did better work after each digital detox. However, they still didn’t feel like quite enough. I had a habit of overworking myself to the point of burnout prior to these days off, so I was recovered but not necessarily energized when I returned. Next year, my goal is to space this time off better throughout the year, and try to add one more day to these digital detoxes so I have more time to recover.

What am I working toward this year?

In 2019, I want to build off of what I’ve learned and adjust. When I was thinking about my goals for 2019, they fell into five categories, and I’m focusing on 1-2 goals in each per quarter:

Category: Prepare my business for growth

  • Q1 Goal: Align my rates for my long-term clients to those I’m charging new clients

Category: Model health and effective habits

  • Q1 Goal: Begin my morning routine at 6 am daily

  • Q1 Goal: Consume 5 or fewer alcoholic drinks per week

Category: Take four digitally-free vacations (5+ days)

  • Q1 Goal: Book June and September vacations

Category: Transform my home

  • Q1 Goal: Find and hire a landscape designer

Category: Have a social life

  • Q1 Goal: Take Melissa on one truly interesting / unique date night per month (this means each week I should try to make a date night fun and creative, knowing that some things I try won’t pan out)


2019 – Let’s Do This

I know I’ll need to continue being “all in” to make these goals a reality. What are you working towards? If you have important goals you are focused on this year I’d love to learn about them, and hear how I can help you stay on track. If nothing else, I want you to know I’m here struggling with you, and I’m grateful for your continued support.

With gratitude,


7 Strategies for asking effective questions

This post originally appeared on Medium

Flashback to Mrs. Eckhart’s sixth grade classroom in small-town Iowa. The lesson for the day: Where do storms come from? The teacher opened the floor for questions and a small, pale boy with brown hair raised his hand: “Why does the sky look green when a storm is coming? And what makes the hail? Is that the same thing that causes wall clouds?”

“Paul, why don’t we give someone else a chance to ask questions,” Mrs. E replied with a slight bite in her voice. I must have driven her crazy.

I’ve always loved asking questions. Maybe that’s why I gravitated towards the field of evaluation and impact analytics. But, I was never really trained in asking questions effectively. And you probably weren’t either.

A few weeks ago, I published a post about how to ask better data analysis questions. Now, I’d like to share a few tips for asking better questions of your coworkers and clients to solve problems and understand impact.

1. Choose the Right Time and Place. 
If your question would take five minutes or less to answer, give someone a call or stop by their desk. For longer discussions, set up an appointment. For executives, ask their assistant or direct reports about their communication preferences, so you can choose the best time and place to get your question answered.

2. Prepare.
 Don’t hold a conversation with a client or colleague until you’re crystal clear about your intended outcome. Then, create a discussion guide that you can follow when talking about the problem you’re trying to solve or question you’re trying to understand. The questions in your guide should start general and then become more specific. You may want to mark the three most critical questions to be sure you cover them if the conversation wanders.

3. Provide Context
 You will naturally want to dive right into the nitty-gritty details. For example, your real question may be, “Why is it that 5% of the case notes we record indicate the client “attended” a session, but the session length was only 1 minute?” Don’t lead with this. Make sure the person you’re talking to understands the purpose of the conversation before asking detailed questions.

One tip I learned the hard way is to respect power dynamics when setting context at the start of a meeting. If you’re not the most senior person in a group discussion, give that person the opportunity to kick things off. Once everyone at the table knows the purpose of the meeting and why you’re asking questions, then you can dive into the detail.

4. Balance Macro and Micro Questions. 
 Chances are you’ve been aggregating questions about a particular subject. When presenting these questions to clients or colleagues, start high level. I find a basic opening question along the lines of:

  • “Walk me through how you do…?”
  • “Tell me about your experience with…?”; or
  • “What do you want to see happen when…?”

…often gets me 80% of what I need to know. And sometimes these broad, open-ended questions have led me to learn things I didn’t even know to ask!

5. Ask Follow-up Questions.
 Surely there was something interesting or unexpected in your interviewee’s answer to the initial broad questions. Trust your gut and follow up on it, especially if you suspect there might be more to the story than they’re letting on. Don’t just take my word on this — author and consultant Peter Blockprovides some great tips on how to behave authentically in critical conversations.

6. Acknowledge Feelings.
 Some of the most useful follow-up questions I ask are around exploring others’ motivations and reactions to their circumstances. Presumably, this challenge or question has existed quite a while. Consider the following questions:

  • Why is now is the right time to tackle this question or problem?
  • How does not knowing the answer make other staff and clients feel?
  • What are other people in the organization (or even your interviewees) doing to perpetuate the problem?
  • Despite the problems, what’s been working well?

7. Manage Cadence.
 Pay attention to the cadence of the interview. Some people ask and answer questions aggressively, talking over others like they’re placing a trade on Wall Street. Others treat an conversation like a therapy session, requiring a lot of silence before opening up. Learn to read the cadence of an interview quickly and match your style to your interviewees to maximize the amount and quality of information you get.

Finally, remember that it’s up to you to engineer a good ending to conversations exploring important questions or problems. Regardless of what happened, give genuine thanks to those who shared their knowledge and experiences. Acknowledge the fact they had other priorities, and share with them what to expect from you or others next.

For me, asking good questions is a critical skill for those trying to build data-literate and impact-literate teams. Do you have other questions about how to promote data literacy in your organization? I’ll respond directly to any comments and also include your thoughts in my next post.

Questions you should never ask if you're in the business of making an impact

This post originally appeared on Medium.

In my last job, I can’t remember the number of times I was asked questions like this:

“Our nonprofit is collecting all this data about our client outcomes. How do these relate to our demographics?”
“What is our funding mix? Is it good?”
“How do we know what donors are probably going to give again next year?”

I’ve worked in and around many nonprofits and social enterprises, enough to know questions that can be answered with data come up all the time. Much of my work over the last several years has been helping social sector leaders answer questions like these. This work can be tricky — over five years ago, David Henderson pointed out, “Data helps answer questions, it does not determine what questions should be answered.”

Asking good questions with the data at hand is an important skill for teams and organizations making the world a better place. But all this questioning and analysis can have a shadow side — asking the wrong questions can be a waste of time and distract your team from your most important goals. Here are 5 common traps leaders in the social sector make when asking questions, and how to avoid them.

1. Questions without an objective: Those of us who work in the nonprofit or social enterprise space often give up financial gain to pursue work we find interesting, challenging, and meaningful. But when we’re thinking about investing resources to answer questions about the work we do, “interesting” is not good enough. Is it clear to your staff why your question is important? For example:

OBSCURE OBJECTIVE: “I just saw one of my friends share a really interesting article from the Huffington Post about the link between family stressors and early childhood health outcomes. Can you tell me how these are related in our clients?”

CLEAR OBJECTIVE: “As we talked about last year, one of our top learning is to understand and promote positive health outcomes for our clients. Can you help us analyze whether our children’s health indicators have any relationship to their parent’s reported stress levels?

2. Questions that have low strategic value: A second criteria for considering questions to answer through data analysis is, “Does the answer help us improve our ability to fulfill our mission?” For example, questions that test your assumptions or logic of how change happens are often strategically important, even if they make you or your staff uncomfortable. So too are questions that have clear implications for your budget or resource allocation.

LOW VALUE: “How many clients expressed satisfaction in our financial counseling program last year? Is there a better number that demonstrates how well our program is working?”

HIGH VALUE: “Our program assumes that because clients set unique financial goals and have unique circumstances, the amount of support each will receive must vary widely. However, are there any cutoffs where additional support from us is unlikely to help a client increase his or her financial well-being?”

3. Questions where you don’t have expectations: A third important aspect to a good analysis question is: Is the answer testable? If you don’t have any internal expectations or external points of comparison, interpreting whether the answer you get is good or bad isn’t possible.

NOT TESTABLE: “What was the effect of our program to increase parental resilience among the at-risk families we serve?”

TESTABLE: “Other programs like ours using the same assessment tools find they have a small positive effect on parental resilience, as measured by an effect size (Cohen’s D) of 0.3 or greater. What’s our effect size?”

4. Questions that have been answered before: In the world of nonprofits and social enterprises, there is a commonly held belief that each organization is unique beyond the point of comparison. While this is true to a certain extent, chances are the issue you’re addressing has been around for a while. Please, before you start mining your own data spend a few hours on Google Scholar or at your local college or university checking out the research literature on the issue you’re addressing. You may not find the answer you need, but may find out how to ask a better question.

ALREADY ANSWERED: “Is the head start approach to early childhood education really effective?”

NOT YET ANSWERED: “As a startup early childhood education provider, how does the social and emotional development of our children compare to that seen in similar head start programs?”

5. Questions where the opportunities for action are unclear: Perhaps the most important aspect of a good analysis question is whether the results are actionable. Will the results potentially cause you to change your strategy, execution, or culture? If not, the question may not be worth exploring.

NOT ACTIONABLE: “Can you give me a summary of client outcomes for each of my direct reports? I’m not sure we have enough data to talk to them about the results, but I think looking at this could be interesting.”

ACTIONABLE: “Can you tell me which workshops last year were most engaging for our clients based on their feedback? We only have budget this year for three workshops and I need to decide which ones we should focus on.”

One final thought — as a changemaker in the social sector, timing is everything. Sometimes it might seem that you can’t make a decision without more analysis; other times you don’t have time for any analysis; often you’ll have these thoughts at the same time! One strategy is to set aside time for answering critical questions about your finances, programs, and operations every year. As questions come up, save them for your Annual Analysis project. Prepare your staff to be engaged, and if you need outside experts or volunteers, get them lined up. Then focus your attention on your most important, timely, and strategic questions.

Do you have other tips to help social sector leaders ask smarter questions? Send them to me and I’ll share your thoughts in an upcoming post.

Why nonprofit leaders need habits for measuring performance

This post originally appeared on the American Evaluation Association's blog, and can be found here. 

Over the last two years I worked as the Data and Evaluation Manager at the San Francisco Child Abuse Prevention Center (SFCAPC), a mid-size nonprofit focused on ending child abuse in San Francisco. This was a fantastic learning experience — I worked with 50+ staff members ranging from policy advocates to social workers, helping them use our data to serve clients better.

About two months into this job, I read a book that changed how I approach this work entirely:

I believe great people to be those who know how they got to where they are, and what they need to do to go where they’re going. They go to work on their lives, not just in their lives… They compare what they’ve done with what they intend to do. And when there’s disparity between the two, they don’t wait very long to make up the difference. — Michael E Gerber, The E-Myth Revisited

Reading Gerber’s book convinced me on the importance of systems and habits to help people succeed at their jobs. This particular nonprofit had a database that held client information — Efforts to Outcomes — but had few habits to improve that system and use that data to serve our clients better. So, I set out to create habits for how we do our “data” work.

I read through books, blogs, and web sites, and talked to mentors and friends to get a sense for what other organizations do. I found many resources that shaped my approach to creating these new “data habits”. Some of the best are:

  • The books, whitepapers, and newsletter from the Leap of Reason Institute — The free e-books on this site by Mario Moreno and David Hunter, and the institute’s recent whitepaper, The Performance Imperative, are the resources I recommend most often to others.
  • Sheri Cheney Jones’ book Impact and Excellence — Jones’ book contains many useful strategies she uses in her consulting practice to help clients generating insights from their data when resources are scarce.
  • The Root Cause Blog — Root Cause is another consulting firm that supports nonprofits in creating effective data and evaluation habits; they share some of their tricks on their blog.
  • The Data Analysts for Social Good professional association — Members get access to 25+ webinars ranging on topics from justifying the return on investment of data analysis to introductory analytical techniques using Excel, R, and other platforms.

After reading these I was convinced that developing regular habits would be critical to my organization’s ability to use it’s data to improve the lives of clients. But exactly what these habits should be remained unclear.

So with the support of our programs staff, I began experimenting. Over the last 18 months, we arrived at six specific habits that helped me provide consistent value to our staff.

  1. Keep a reporting calendar: Organizations are often required to submit detailed program participant and activity uploads for certain government contracts. I created a comprehensive calendar of when to submit these uploads and who needed to review the data before it uploaded.
  2. Define data integrity controls: Data integrity controls minimize the risk that information in a database is incorrect. For example, we scrubbed our database of test data monthly and audited a sample of new families to verify the accuracy of data entry each quarter. We summarized data integrity controls in a spreadsheet outlining each procedure, information source, performer, reviewer, and results.
  3. Review dashboards: The first week of each month, I sent a performance dashboard to each of the program managers. Managers discussed these metrics with their teams. Then they shared explanations for variances and any action items they were going to take at the manager’s meeting the following week.
  4. Schedule time for troubleshooting and report development: To build staff buy-in I needed to be responsive to database troubleshooting and report development needs. I tracked time for these tasks and blocked out time for them weekly. In an average week, I spent 2–10 hours troubleshooting and training, and 5–15 hours developing self-service reports staff could use to access program data themselves.
  5. Automate annual development data pulls: A significant “data” responsibility was pulling data for our development team, including demographics and unduplicated client counts. Working with our development team, I developed a self-service report designed to answer 80% of their “stats” asks, saving everyone time.
  6. Have a data analysis process: Stakeholders across our organization came to me with many good questions to explore in our data which I just didn’t have time to answer. I created a master tracker of these questions and set aside several weeks each year to explore those that were most critical. This “Annual Data Analysis” process set expectations and created a process to focus our limited analysis time.

These habits helped us save time, set expectations, and create lasting systems. They’re far from perfect, but were a huge step forward for us. Now that I’m no longer working for this organization, I realize these habits not only helped organize the work, but enabled continuity between me and my successor.

If you’ve read this and are working in the social sector, I’d love to know — What are the data & evaluation habits you’ve found valuable at your organization?