Numbers are only as useful as the questions you ask of them

Errol Morris recently made a point about filmmaking that expresses precisely how I feel about interpreting data:

"There is no mode of expression, no technique of production that will instantly produce truth or falsehood."

Data, like film or photography is a representation of "the real world". We study it in the hopes of finding enlightenment and understanding. However, there is no "technique of [data representation] that will instantly produce truth or falsehood." If someone is disillusioned with data it is often because they expect too much from it. Data can't "tell us" anything. It can only take something that is hard to grasp and offer up otherwise submerged surface area for examination, inquiry and analysis.An important part of analyzing data is doubt and questioning, yet most data reported by the mainstream media is doubt-resistant. Some magic number is reported to the public with no real data to pick apart and study. Where do these magic numbers come from?I wrote a while back about Yale Professor Don Green saying casually that he never believes tidy data. Beware of dumbed down data! When presented with data that conveniently boils down to "one number" that can explain it all, raise an eyebrow and dig deeper.Responsible reporting of new data findings should probe and challenge the data. Where did the data come from? How reliable are these sources? What data is missing? How might what's missing change the results? (This is the hardest to pull-off because it requires us to imagine what we don't know.) How is the way the data is presented inadvertently influencing how it will be interpreted? What assumptions will each person bring to their interpretation of this data? Are they valid? Who's in a position to make that judgment? Without at least asking these questions, eye-catching "magic number" headlines are a disservice to the public, designed to catch eyeballs with false clarity rather than expose the confusing uncertainty of reality.More often than not, analyzing one data set simply propels you to collect and question more data. Now that I have this data, what other data do I need? Now that I have answers to these questions, what other questions do I now know I need to ask?This is not to say that data never yields answers. Generally speaking, however, every hard-won answer simply opens the door to 5 more questions you couldn't have imagined at the outset.