Some measures of intelligence cannot be executed through testing alone, Daniel Koretz says.

Question: Is testing an adequate measurement of student achievement?


Daniel Koretz:    Well, I think, standardized tests as we currently have them are enormously useful.  So, for example, we all know anyone who follows education that there are enormous inequities in American schooling.  One of the ways we know this is because of standardized tests.  That’s what tells us, not only that the inequities are there but how big they are.  And I personally think that it push for accountability in schools is not going away anytime soon.  The public clearly wants schools to be held accountable for increasing the level of their productivity, which means teaching kids more.  So I see no way to have a sensible accountability system without testing.  I just think we have to use it more rationally.  It’s very much like… you know, this is perhaps a weak analogy, but it’s like overprescribing a very strong medication.  The problem is not that the testing shouldn’t be used.  I wouldn’t argue that testing shouldn’t be used, we’re just using it in the wrong way, putting far too much pressure on increases in scores.      

Question: What other standards could we use to measure intelligence?

Daniel Koretz:   

Well, there are new ideas about how to measure students’ achievement.  Some of them are very hard to use on a large scale.  That one of the difficulties we have is that there’s a lot of pressure now and should be a lot of pressure to test high [order] thinking skills, complex problem solving what people like to call 21st century skills.  Some of that is very hard to do with the test that you drop in from a state capital.  I won’t give you a long explanation of the reasons why but one of them is that to measure whether a student has good problem solving skills, you have to give the student a test that somewhat [novel].  If you give the student something that’s routine for them, they don’t have to use real problem solving skills, they can just redo what they’ve done before.  It’s very hard sitting in the state capital to know what things are [novel] for you and what things are [novel] for me, what things are [novel] for children in this school over here and for children in another school.  So to some extent, we have to rely on more local measures of performance, which is what testing experts long ago, two generations ago, told educators to do.  They said over and over use test as a supplementary source of information.  Never use them alone to measure the quality of education. 
   

Daniel Koretz:    There are old techniques but they are not well evaluated.  For example, in countries such as the Netherlands, it’s routine for schools to be visited by outside inspectors who look at far more than test scores.  And, in fact, the Dutch public often hears about test scores in the context of an inspector’s report.  The problem is that we don’t have a real research and development approach to this.  We don’t do what engineers would call systems engineering.  The logical thing to do would be to say, well, we have a complicated problem.  Teachers have a very complex role.  So, therefore, we have a complicated problem in designing an accountability system.  As a result, we’ll try different things.  We’ll measure constantly how they affect schooling and will tweak the system as engineers would say we’ll bury the parameters.  We don’t do that.  We drop these things in from the sky.  We get a tiny bit of evaluation some years later and say, oh that one didn’t work.  But now we know and we drew up another one in.  And the result is that kids are basically left behind.  So what we have is a lot of interesting ideas about better ways of holding schools accountable and very little hard research.  And I would say that that’s really an ethical problem, not just a political problem.  It’s a political problem because we lack information that we could use to better serve children.  It’s an ethical problem because children are not consenting adults.  When we drop into schools these very high powered policies that clearly change teacher’s behavior in dramatic ways, we have an obligation, in my view, to monitor what happens.