When test scores seem too good to believe

Read the original story here.

MILFORD, Ohio — Scott Mueller seemed to have an uncanny sense about what his students should study to prepare for upcoming state skills tests.

By 2010, the teacher had spent his 16-year career entirely at Charles Seipelt Elementary School. Like other Seipelt teachers, Mueller regularly wrote study guides for his classes ahead of state tests.

On test day last April, several fifth-graders immediately recognized some of the questions on their math tests. The questions were the same as those on the study guide Mueller had given out the day before. Some numbers on the actual tests were identical to those in the study guide and the questions were in the same order, the kids told other Seipelt teachers. 

The report of possible cheating quickly reached district officials, who put Mueller on paid leave. He initially denied any wrongdoing. Ultimately, investigators concluded that Mueller had looked at questions for both fifth-grade math and science tests in advance — a violation of testing rules — and then copied them, sometimes word for word, into a school computer to develop his study guides. 

The 50-year-old teacher resigned. He signed a consent agreement with the Ohio State Board of Education admitting that, by looking at the 2010 tests in advance to prepare study guides, he had "engaged in conduct unbecoming a licensed educator." His teaching license was suspended for three months.

At Seipelt, as in other schools nationwide, young students tipped off officials that something was amiss. Yet if anyone had taken a closer look at the past few years' scores, they might have noticed other testing irregularities at Seipelt.  

In several grades, standardized test scores at Seipelt fluctuated year to year, sometimes rising sharply, then falling, according to data USA TODAY obtained from the Ohio Department of Education. 

In 2005, for example, the school's third-graders tested in the 67th percentile statewide in math. As fourth-graders a year later, when Mueller was one of their teachers, their scores jumped to the 97th percentile, among the best in the state. As fifth-graders in 2007, the scores plunged to the 49th percentile. Then, in 2008, when they were in sixth grade, their scores climbed again to the 90th percentile. 

Seipelt's gains and losses are typical of a pattern uncovered by a USA TODAY investigation of the standardized tests of millions of students in six states and the District of Columbia. The newspaper identified 1,610 examples of anomalies in which public school classes — a school's entire fifth grade, for example — boasted what analysts regard as statistically rare, perhaps suspect, gains on state tests. 

Such anomalies surfaced in Washington, D.C., and each of the states — Arizona, California, Colorado, Florida, Michigan and Ohio — where USA TODAY analyzed test scores. For each state, the newspaper obtained three to seven years' worth of scores. There were another 317 examples of equally large, year-to-year declines in an entire grade's scores. 

USA TODAY used a methodology widely recognized by mathematicians, psychometricians and testing companies. It compared year-to-year changes in test scores and singled out grades within schools for which gains were 3 standard deviations or more from the average statewide gain on that test. In layman's language, that means the students in that grade showed greater improvement than 99.9% of their classmates statewide.

The higher the standard deviation, the rarer that improvement is. In dozens of cases, USA TODAY found 5, 6 and even 7 standard deviations, making those gains even more exceptional.

Large year-to-year jumps in test scores by an entire grade should raise red flags, especially if scores drop in later grades, says Brian Jacob, director of the Center on Local, State and Urban Policy at the University of Michigan. Such fluctuations by themselves do not prove there was cheating, but Jacob says they offer "a reasonable way to identify suspicious things" that should be investigated.

Education reformers say a surge in scores is possible without cheating. Mike Feinberg, the Houston-based founder of the Knowledge Is Power Program (KIPP), a 99-school chain of charter schools widely recognized for raising test scores, says "remarkable growth" is possible with "great teaching and more of it." Where you have "just an amazing teacher who can motivate his kids to really work hard," classes can see gains that might seem "unbelievable" at first glance. 

If big gains were followed by deep losses, Feinberg cautioned, you'd have to ask, "What were the adults doing that might not have been … ethical?" 

"You know something is profoundly different" at schools with spikes in scores, says John Tanner of Test Sense, a San Antonio consulting firm that works with schools nationwide. But, if an investigation shows the school is making "profound changes" commensurate with the gains, "I would give them the benefit of the doubt," he adds.

Others are more skeptical. "An individual student can exceed beyond their wildest dreams in any given year, but when a whole group shifts its position dramatically, you have to worry," says John Fremer, president of Caveon Test Security, a Utah company hired by states and school districts to investigate test irregularities.

Thomas Haladyna, a professor emeritus at Arizona State University, says test gains of 3 standard deviations or more for an entire grade are "so incredible that you have to ask yourself, 'How can this be real?' " 

Haladyna says such a spike in scores would be like finding "a weight-loss clinic where you lose 100 pounds a day."

In the past decade, similar score spikes uncovered by The Dallas Morning News and the Atlanta Journal-Constitution, using the same methods as USA TODAY, led state officials in Texas and Georgia to conduct major probes of hundreds of schools. Most recently, Atlanta Public Schools Superintendent Beverly Hall annouced she will step down in June, following inquries by federal and state investigators of alleged cheating at 58 Atlanta schools. 

In Ohio, data show that two of Seipelt Elementary's score fluctuations — the fifth-graders' 48-percentile decline in 2007 and the 41-percentile climb in sixth grade a year later — registered more than 3 standard deviations. School Principal Melissa Borger attributed the fifth-grade decline to inexperienced teachers, the sixth-grade jump to a teacher with 25 years' experience teaching math.

School and district officials here said they saw no reason to suspect that any of their teachers cheated before 2010. Robert Farrell, superintendent of the Milford school district, said in an interview with The Cincinnati Enquirer, a partner of USA TODAY on this project, that he considered Mueller's transgression a one-time event. Mueller did not respond to requests for comment from the Enquirer.

The 66 fifth-graders who used Mueller's study guides had to retake their math and science tests, at a total cost of $3,300. They passed with high marks.

Cause for celebration

Dramatic test improvements are usually causes for celebration.

That's because of the increasingly high stakes attached to the tests required under the federal No Child Left Behind (NCLB) law. Although most school districts retain the power to hire and fire teachers, 10 states now require that student scores be the main criterion in teacher evaluations. Some states and districts reward educators for raising scores; a teacher may earn a bonus of as much as $25,000 in Washington, D.C., if his or her students' scores climb. NCLB also puts principals' jobs on the line if students' scores don't improve. Most of the 130 Detroit public schools closed since 2005 were cited for having low test scores. 

The Obama administration has begun doling out extra money to the states that tie teacher evaluations to test scores. At the same time, NCLB's harshest penalties for underachieving schools and teachers are about to kick in: By 2014, the law dictates, 100% of public school students must be "proficient" in math and reading. If not, a school can face replacement of its entire staff. 

Given the mounting pressure on teachers, principals and superintendents to produce high scores, "no one has incentives to vigorously pursue" testing irregularities, says Gregory Cizek, a professor at the University of North Carolina-Chapel Hill who studies cheating. "In fact, there's a strong disincentive."

Fremer, the Caveon executive, says the idea of investigating high scores seems silly to most educators. 

"Scores are going up? Good. Scores are going down? Bad. And you want me to investigate when scores are going up? What's the matter with you?"

Investigations can be time-consuming and expensive. In Michigan, the average cost for an investigation ranges from "several hundred to a few thousand dollars," says Joseph Martineau, director of the Office of Educational Assessment & Accountability. He told the Detroit Free Press, another USA TODAY partner, that the costs include hiring "independent investigators who are former school administrators and have been trained in police investigation methods."

John Boivin, administrator of California's standardized testing program, says his state once conducted random test audits at 150 to 200 schools a year. California dropped the audits two years ago because of record budget deficits. And the state no longer collects data on which schools show unusually high rates of erasures on answer sheets — sometimes a clue, experts say, that either students or school officials might be cheating. Total savings: $105,000.

Even when suspicious scores are investigated, it can be hard to identify a culprit unless someone confesses. Ed Roeber, who headed Michigan's testing program for nearly 20 years, says evidence of cheating is almost always circumstantial. "It's very difficult to prove," Roeber says. "Often what you end up with is the feeling that something might have happened, but I don't know for sure and, even if I did, I couldn't prove it in a court of law."

That result frustrated him, Roeber says. "It made me angry because you were cheating kids. You're not finding out if they need help. You're painting this picture that is incorrect."