What an Atlanta Newspaper Has Taught Us About Data Reliability

I am done with my public bashing of the Atlanta newspaper that spread misleading research about standardized test scores around the nation.  However, what we have seen as a result of this debacle is that as part of "Reinventing Public Education," we need to do some serious reinventing of data analysis generated by our public school system.  

The Georgia Department of Education released a statement yesterday that disclosed a new formula for calculating the state's graduation rate.  Amazingly, but not surprisingly, the restatement brought last year's graduation rate down nearly 14 points, from nearly 81 percent to just over 67 percent.  No one really thought the 81 percent figure was a reliable figure.   So what does this all mean?

This indicates the great challenge in researchers trying to compare data across states and localities.   It is common knowledge that the public school system is the last bastion of society mired in the analog age, and for public education to be able to be reinvented to succeed in a digital world, the entire infrastructure needs to be rebuilt from the ground up.  This is EXACTLY why you can't "cram" technology into schools, as my friend Michael Horn (co-author of Disrupting Class) has pointed out so eloquently.  The challenge of obtaining data that is comparable across states is laborious and tedious at best, if not impossible to complete.   As you saw in Georgia, what appears simple as asking for a "graduate rate" is clearly not so simple.

This is symptomatic of the inherent conflict between Hamiltonian Federalism and Jeffersonian Republicanism.  We need the federal resources to compel states to report data using the same formula, but even in this case, Georgia obtain a waiver to calculate the graduation rate using a 5-year, not 4-year cohort.   I am not lobbying for one methodology over another, but think about all of the disclaimers a researcher will have to annotate because states are not calculating a graduation rate the same way.

What this example has shown this blogger is that we need to be VERY skeptical of education data in the public domain.   How un-nerving for citizens to know that reported data may not be what it seems.   This is one of the many challenges faced in education reform efforts. 

We should all be worried about this one.