K-12 State Education Performance Rankings – The Real Truth

US News and World Report publishes annual State rankings that are widely used to assess performance in K-12. No one has ever challenged them…until now. In a major new report:  Everything You Know About State Education Rankings Is Wrong: Minds and dollars are a terrible thing to waste. the authors have gone back to the drawing board and revisited the assumptions used by US News, and, in so doing, have come up with completely different data and results that turn the K-12 education world on its ear! You can read the article in the link above or you can read our summary of the major takeaways below:

There’s just one problem with the current US News narrative: Their rankings are riddled with methodological flaws.

Here are the highlights for the New Ranking System:

1.   They built a new set of rankings based on students’ performance on the National Assessment of Education Progress (NAEP), a battery of standardized tests sometimes called “The Nation’s Report Card.” These tests are given to fourth and eighth graders as well as some high school seniors.
2.   They fixed two serious problems common to traditional rankings. First, they removed factors that do not measure K–12 student performance or teaching effectiveness, such as spending per student (intentions to raise performance are not the same as raising performance), graduation rates (which often indicate nothing about learning, since 38 states do not have graduation proficiency exams), and pre-K enrollment.
3.   The second and more important change was to disaggregate student performance data so that they could compare likes with likes. Traditional rankings effectively reward states for not having many minority students. States do well simply because they are populated by families from more socioeconomically successful ethnic categories—not because they are actually doing a good job educating their various categories of students.
4.   By looking at test scores for students in fourth and eighth grade in math, reading, and science, and by separating students by racial category, they get 24 different possible basis of comparison. This allowed them to measure how well states do for each specific student type—Asian fourth-grade math students, for instance. (They adjusted their rankings to compensate for the fact that not all states report scores for every student group.)
5.   In order to examine the relationship between expenditures and quality more precisely, they ran multiple regression analyses on the data, which included several other variables. The regression results support the view that expenditures are not linked to student performance.
6.   The regression results revealed other findings as well. The most interesting is that union strength has a powerful negative effect on student performance. It’s well-known that teachers unions aim to increase wages, which might lead to better teachers and increased test scores. But apparently, other union goals that are harmful to student performance—such as protecting poor teachers from being fired or blocking merit-based pay—have a greater impact. This may come as a shock to those who think teachers unions are a recipe for educational success.

Let’s look at the data. Click this link if you want the full data: Table   Here is a screenshot of the table so we can look at the West Coast States and then at changes nationally to rankings.

What do we see when we look at the chart?

1.   The right column is the U.S. News rankings vs the rankings correcting the methodology errors and assumption changes. Massachusetts went from #1 to #2 in Quality and #10 in Efficiency. Why? Because they have a largely white (homogenous) population and spend a lot of money per student.
2.   The new #1 in quality is Virginia vs. being #12 in the USN report
3.   The new #1 in efficiency is Florida vs. the USN ranking of….#40. Why? Because Florida has a more diverse student mix and spends less money per student while getting better academic results

Where does Washington State come out?

•   Under the old/flawed USN report Washington State was #26
•   Under the revised and updated report Washington State ranks #17 in Quality and #17 in Efficiency
•   Why did Washington State move up? They get better results across all ethnic groups while spending less money and getting more from it.

Where does Oregon come out?

•   The old USN ranking was #37
•   The correct ranking puts Oregon #44 in Quality and #29 in Efficiency.
•   Why? Because their results fall below their expenditures. They don’t do as well across ethnic groups.

How about the giant California?

•   The USN ranking was #44
•   The new results show California #34 in Quality and #14 in Efficiency
•   Why the differences? For the money they spend across the ethnic mix minorities do better and given the money they spend they get better results

How about Alaska?

•   The USN ranking was #46
•   The new rankings show Alaska #42 in Quality and #47 in Efficiency
•   Why the differences? Poor results and poor use of money

Now let’s take a look at the scatterplot they shared in the report: Scatterplot

Let’s look at the Spend vs Results (Quality):

•    The “X” axiss (the horizontal along the bottom) shows the amount per student each State spends per student at the time of this report.
•   The “Y” axis (the vertical along the left side) shows the “Quality” score. The larger the number the better
•   For the amount of money spent per student Florida blows every other State away
•   In the highest Quality range (3.25 to 3.75) Florida (#40 USN), Texas (#33 USN), Virginia (#12 USN), Massachusetts (#1 USN) and New Jersey (#3 USN) lead the way.
• The MOST Effective and Cost Efficient are: Florida #1; Texas #2; Georgia #3; Indiana #4′ Maryland #5 and Montana #6
• It’s worth noting that Texas is often maligned as producing poor results when, in reality, it is #2 in the nation for results vs. expense

Summary:

• Are there takeaways?
  • Yes. Spending more money has almost no relationship to results
  • Teachers Unions lead to increased expenditures but have no relationship to improvement in results

They conclude: The time is ripe to re-evaluate education policy in this country. After all, minds and dollars are terrible things to waste.