(The Importance of Failure in Data Analysis)
I recently had the chance to participate in the Transform the Sector event in Toronto — and it was the highlight of my year so far.
The conference was designed to highlight the possibilities (and responsibilities) that come with using digital data in the social sector. I was asked to co-lead a panel called Behind Closed Doors: Tales of Data Analysis with conference founder Andrew Means. We are both longtime advocates of digital data use, and this was an opportunity to give our audience a backstage pass into our world: the highs and lows, the successes and failures.
It may seem strange that I was excited to get up and talk about my failures in data analysis, but I was. I frequently participate in these types of sessions at conferences — the Joint Statistical Meeting, the American Evaluation Association Conference, and others — but this felt different. It was important — both because it would be challenging to do but also because it would give my audience a new view into my experience as a data expert.
It’s easy for us as experts to get up on stage and talk about the ideals of digital data: the wonderful things it can be used to accomplish, the beautiful tools that have been built, even the care we take in getting all the ethics right.
But that can leave the audience with the idea that, if you want it badly enough or work hard enough, all your data dreams will come true — that success is inevitable for those with enough will and perseverance — when, in fact, failure in data analysis is all too common. We don’t often hear the stories of failure because, well… nobody wants to stand up in front of their peers and admit things went wrong. This was a moment to do something new.
Sharing Our Data Horror Stories
Andrew and I shared some of our worst failures in data analysis at the event.
Andrew spoke about a time when, very early in his career, his enthusiasm for robust data collection badly disrupted an after-school YMCA program as he insisted that all programs measure the height and weight of each child every week. I shared an incident where I failed to correctly train enumerators in a national survey and wound up with unlinkable data for a sample of 250,000 people. (In my defense, we developed a machine learning matching algorithm to correct my error, which was a silver lining!)
The point was that, in field work and grassroots efforts, failures in data analysis are at least as common as successes. And beyond being common, our data failure stories — especially in the social sector — are valuable. It’s vital that we share and discuss them, openly and often, because we all stand to learn just as much from what went wrong as from what went right.
The Value of Failures in Data Analysis
Some of you may be looking at me a little funny right now. Surely successes are what we learn from — what is the point of examining and sharing projects that didn’t make the grade?
Let me give you some examples:
Data collection methods that don’t work well.
Our instinct here is to view failed collection methods as a roadblock; an impediment to uncovering the information we need. In reality, examining data collection methods that aren’t working can give us greater insight into what is occurring in the field. Why is this particular method not working? What does that mean to our project? Can we use this insight to learn more about the groups we’re working with?
Impact evaluations that show our intervention isn’t effective.
This is not a failure in data analysis at all, but rather, an extremely valuable tool that helps us redirect our efforts and avoid wasting resources. The understanding we gain from an impact evaluation that shows we are not achieving our goal brings us one step closer to discovering what might work instead.
Data visualizations people can’t make heads or tails of.
It’s frustrating to pour your heart and soul into what you believe is a work of art, only to have your audience reject — or worse, misinterpret — it because they can’t understand what it’s trying to say. But if we are truly passionate about communicating the results of our efforts, this feedback is vital — the most beautiful visualizations in the world are useless if they fail to accurately communicate with our audiences.
Share Failures and Successes
At Datassist, we combine one part science, one part art, and a whole lot of passion to help you tell a data story that will capture both the hearts and minds of your audience. Whether you’re struggling to learn from your failures in data analysis or need help sharing your success, our team is at your service. Get in touch with us today to see why the most meaningful change begins with data.