Is the ACT a Valid Test? (Spoiler Alert: No.)

Is the ACT a Valid Test? (Spoiler Alert: No.)

ACT, Inc. released the results of its 2016 National Curriculum Survey earlier this year. The Survey goes out every three or four years to elementary, middle school, high school, and college teachers, as well as to workforce professionals. It collects information about what respondents are teaching, how they teach it, what they care about, and so forth. It serves as the basis upon which ACT builds its tests.

Because the Survey provides a look into both what pre-college students are being taught, and what they need to know to be prepared for college, it is a useful tool for examining the serious and persistent problem of college readiness—why it is that the majority of high school graduates are underprepared for college-level academic work. ACT itself reports that 72% of its test-takers fall short of at least one of its college-readiness benchmarks, which confirms the widespread underpreparedness reported by other sources. And indeed, ACT’s National Curriculum Survey reveals wide disjunctures between high school teaching and college expectations, which may have something to do with why students aren’t better prepared.

But ironically, by pointing out these disjunctures, the Survey raises questions about the validity of the ACT exam itself. The ACT is a test that straddles the space between high school and college, claiming to be both reflective of high school curricula and a measure of college readiness. But if ACT’s own Survey reveals that high school curricula do not align with college expectations, how can the ACT validly claim to measure both?

Tests are all about validity. Their value and utility depend upon them actually measuring what they purport to measure. If a test does not actually assess what it purports to, then it’s not a valid test, and any inferences made based on its results are faulty—inferences such as “this kid has been taught the skills needed for college success but hasn’t learned them very well.”

The two claims ACT, Inc. makes about the ACT test are at odds with each other, which calls into question the test’s validity. The claim that the test is “curriculum based” rests on Survey results, which ACT says serve as empirical evidence upon which it decides how to build the test. In this way, according to ACT, the test reflects what is being taught in high schools—an important claim, since testing kids on things they haven’t been taught doesn’t tell anyone much about their abilities.

ACT also, of course, claims that the test is a measure of college readiness. Through the Survey, it gathers an understanding of what college instructors expect from entering students. This understanding is reflected in ACT’s College and Career Readiness Standards, a set of “descriptions of the essential skills and knowledge students need to become ready for college and career.”

According to ACT, the Standards are validated by the Survey in a process that “ensures that our assessments always measure not only what is being taught in schools around the country, but also what demonstrably matters most for college and career readiness.”

But can the ACT test both what is taught in high school and what is expected in college if those two things don’t square up, as is suggested by their National Curriculum Survey and other research?

Perhaps there’s a significant degree of overlap. Perhaps ACT can identify and test students on those things that fall into both the learned-it-in-high-school category and the better-know-it-for-college category. ACT says indeed there is overlap, and that they have a way of figuring out what’s in it.

How do they do it? According to a 2015 white paper, “ACT first identifies what postsecondary faculty, including instructors of entry-level college and workforce training courses, expect of their entering students—that is, the knowledge and skills students need to be ready for entry-level postsecondary courses and jobs. ACT then compares these expectations to what is really happening in elementary, middle, and high school classrooms. ACT uses the results of these comparisons to determine the skills and knowledge that should be measured on ACT assessments and to guide its test blueprints.”

The company does not explain how this process of comparison works, but it implies that they identify a subset of knowledge and skills that fall into both camps, then simply test kids on that.

To feel confident in this process, we would need to be certain that the subset is sufficient in size and scope to support the dual claims. That is, we would need to know what lies outside the overlap slice, as well as what lies within. What is being taught in high school that does not appear on the test because it is not a college-ready expectation? Likewise, what college-ready expectations do not appear on the test because they are not being taught in high school?

Once we knew those things, then we could validate the ACT by answering this question: Is the overlap slice sufficient to support both the claim that the test measures what is being taught in high school and the claim that it measures college readiness?

In other words, is there enough of the high school curriculum on the test to justify calling it a valid measure of high school achievement? And are there, at the same time, enough college expectations on the test to justify calling it a valid measure of college readiness?

ACT doesn’t attempt to answer these questions. As far as the ACT is concerned, if you demonstrate proficiency on the test, then ipso facto you’ve both mastered your high school curriculum and are ready for college, because the claims they make for the test require that the two constructs be identical.

What if you don’t do so well on the test? Is it because you haven’t learned well enough what you’ve been taught? Or because you haven’t been taught what you’re being tested on?

The ACT simply doesn’t allow for the second possibility.

In point of fact, if high schools were teaching certain essential college-ready skills – how to revise your work in response to feedback, for example—a conventional standardized test like the ACT would never be able to detect it, because it cannot provide for test-takers opportunities to do the kind of authentic, extended, or collaborative intellectual work that will be required of them in college.

But alas, as mentioned already, plenty of research demonstrates that there is a significant difference between high school learning and college expectations, suggesting that any overlap might not be very robust.  According to a six-year national study on college readiness from Stanford, “coursework between high school and college is not connected; students graduate from high school under one set of standards and, three months later, are required to meet a whole new set of standards in college.”

ACT’s own research confirms this. Two things jump out from the National Curriculum Survey results. First, as we can see from the table below, in many cases the Survey does not ask high school teachers and college instructors the same questions, so there is not much opportunity for determining where high school teaching lines up, or not, with college expectations. The Survey doesn’t look like a very good tool for comparing high school teaching to college expectations in Writing, for example.

The second thing is, where the Survey does provide opportunities for comparing high school with college, it finds that high school teaching does not align with college expectations. The Survey report points out, for example, that high school Writing teachers and college instructors are not emphasizing the same skills. Further, high school math teachers do not agree with college math instructors about what skills are important for success in STEM courses. Less than half of high school teachers believe that the Common Core math standards (which ACT stresses are in line with its own College Readiness Standards) match college instructors’ expectations for college readiness.

In other words, ACT’s own Survey shows that, to a significant extent, the knowledge and skills high school teachers are teaching are not the knowledge and skills college instructors are expecting of entering students.

Hence the college-readiness gap.

But if those two bodies of knowledge and skills aren’t the same, how can ACT support the claim that its test measures both what students actually learn and what ACT says they should learn for college readiness? The test doesn’t distinguish a “high-school-learning” part from a “college-requirements” part. As far as the test is concerned, it’s all the same.

In fact, ACT can’t really support both claims at the same time. But they make them anyway because they want to sell the test to two distinct markets. They want to sell it to students who are trying to get into college, so they call it a college-readiness test. And they want to sell it to states and districts for accountability purposes. These entities want to know whether their students are learning what they’re being taught; thus ACT calls the test curriculum-based.

But, we might wonder, don’t standards take care of all this? Standards, after all, both reflect the skills needed for college readiness and guide high school curriculum, right? Therefore, if the test aligns with the standards, then it’s both curriculum-based and a college-readiness indicator, because those are the same thing.

Most states have adopted the Common Core State Standards. Those that haven’t have concocted their own state standards, which are pretty much in line with the CCSS. In addition, ACT has its own College and Career Readiness Standards, which, it says, line up with both the CCSS and any non-CCSS state standards you care to throw at it. (As ACT says, “If a state’s standards represent the knowledge and skills that prepare students for college and career, then ACT Aspire and the ACT measure that knowledge and those skills”—a statement that manages to be both a non sequitur and a tautology.)

Again, however, ACT’s own research shows that neither high school teachers nor college instructors are much convinced that the CCSS reflect college-level expectations anyway. Asked by the Survey, “To what extent do you feel that the Common Core State Standards are aligned with college instructors’ expectations regarding college readiness,” the majority of both high school and college teachers responded little or slightly, rather than a great deal or completely.

In other words, according to its own data, ACT shouldn’t really get away with equating standards-based “curriculum achievement” with “college readiness.”

So what’s the cost of the ACT’s tricky claim-game? The cost is that we get farther away from understanding and addressing the college-readiness gap, so long as everyone believes that the ACT is really measuring what it says it does.

Wherever high school curricula lack significant overlap with the skills and knowledge ACT identifies as necessary for college-readiness, the test measures not what students have learned but what they haven’t been taught. This, then, contrary to ACT’s claims, is not an indicator of student readiness or achievement, but a measure of the distance between high school teaching and college expectations (or at least those ACT identifies and can test for).

But this is not how the interpretation of test results falls out for either student or state customers. Rather, the inescapable inference for both is that the majority of students have been taught what they need to know but simply haven’t learned it well enough—student’s fault, or teacher’s fault, but not the test’s fault for leading everyone to a lousy inference.

The faulty inference that issues from the ACT doesn’t help matters where students’ future opportunities are at stake; prospective colleges have no way of knowing that a kid was tested on things she was never actually taught. And it doesn’t help where states are trying to figure out how to improve their education systems. Rather, it makes matters worse by misdirecting both states and students away from the problem of how better to connect high school learning to necessary college skills, and toward the problem of how to get kids to score better on the test.

We do indeed want an education system in which high school curricula are focused securely on the skills and knowledge we confidently know are needed for success upon entry into college.  Demonstrably, that’s not what we have now, so we don’t need a test that falsely suggests otherwise.

© 2016 BetterRhetor Resources LLC


College As Culture Shock

College As Culture Shock

The standards-based education reform movement, kicked off by A Nation At Risk in 1983, has been around long enough now to start showing results, if it’s going to. Unfortunately, there is not much evidence that this path is leading anywhere good. The latest Nation’s Report Card, based on National Assessment of Educational Progress (NAEP) testing, shows that twelfth graders in 2015 weren’t any more ready for college than were twelfth-graders in 2012.  Likewise, students in 2012 didn’t make progress over the kids tested in 2007, or 2003, or 1998.

Generally, the NAEP assessments show that over an extended period only about a quarter of high school students have graduated ready for college-level academics. For low-income and minority groups, the numbers are even worse, and aren’t budging. (For a more extended discussion of the College Readiness Gap, click here.)

Why, given all the efforts made in recent years toward defining and teaching the skills and knowledge needed for college success, are we still not seeing rising numbers of high school graduates ready for college?

Because standards, important as they may be, are not enough. It is increasingly evident that learning entails much more than the acquisition of cognitive skills by themselves. For example, researchers now emphasize students’ dispositions and behaviors—e.g. their “grit”—as necessary components of academic success. Education reformer Diane Ravitch identifies poverty and race-based inequalities as the unaddressed culprits in our failure to make progress in the standards-based education era.

Where college readiness is concerned, we must consider that most students, when they first enter a college classroom, are encountering what amounts to a new and strange culture, with its own embedded behavioral expectations, its own language, its own values and assumptions. Many entering students, regardless of the standards-based cognitive skills they may have acquired in high school, are essentially strangers in a strange land, largely unacquainted with the world they’ve entered into. They don’t speak the language, they’re unfamiliar with the underlying values that structure rules for meaning-making, they don’t quite know how to successfully make use of the skills and knowledge they do bring with them from high school.

They are not prepared for the culture of college academics.

Moving to a foreign country, most high schoolers might be able to order a meal and buy a bus ticket from the start, but they would not find themselves truly understanding and thriving within their new habitat until they became oriented to how the natives think, what motivates them, their belief structures, why they behave as they do. The same is true of students entering the foreign world of postsecondary academics.

And, of course, where higher education is concerned, the goal ultimately is to integrate these new denizens into the culture to such a degree that they become wholly familiar, conversant, and accepted members. That’s what being educated means.

The norms of college academic culture are unfamiliar to most matriculating students, but especially those who are among the first in their families to go to college. For the most part, how academic culture operates goes unexplained, yet understanding and negotiating it is critical to every student’s college success.

To do well in a college classroom, students must acquire a high-level map of the governing values, shared assumptions, accepted and expected ways of communicating and thinking and behaving—all the elements that comprise the webs of significance constitutive of any culture.

The cultural elements endemic to the intellectual enterprise of college academics are only thoroughly acquired over time, through experience and exposure, through a kind of osmosis, as students absorb the language and concepts employed by their instructors, begin to read and respond to an array of academic texts, participate in collaborative academic work, sink deeper into their major field of study, write papers, do research, and gradually begin to grasp the kinds of performances that are considered successful.

At a fundamental level, acquiring a college education entails enculturation into the practices and discourses that structure meaning-making in higher academics.

Every student brings to school particular linguistic resources, background experiences, assumptions, values, and expectations from their own native culture. The degrees of variance between native and academic culture differ widely, however. Some students are better able to function within academic culture from the start because it is already relatively familiar to them. The vocabulary and values characteristic of college academics are not so far from what they’ve grown up with. They enter college already knowing pretty much how things work, how to interpret what’s being said, how to discern and meet the behavioral and performance expectations.

For other students, the culture they come from bears little resemblance to the one they’re entering, and it takes a lot more than mastery of basic cognitive skills to be successful.

To be clear, we’re not talking here strictly about immigrants from distant lands, who may be struggling with English or American cultural norms more broadly. Nor are we talking about students grappling with the unfamiliarity of college life per se—class schedules, dorms, social activities, and so forth.

Rather, we’re talking about anyone for whom the norms of the intellectual culture of college are unfamiliar—anyone for whom, for example, the primacy and procedures of academic, evidence-centered argumentation are not already woven into the fabric of how meaning gets made in their world. There are many ethnicity- and income-defined subgroups within American culture that fall into this category.

Wide disjunctures between native and academic cultural norms, absent some targeted intervention, are how socio-economic disadvantages transmute into educational disadvantages. All the more reason, then, for all students, regardless of income or background, to begin the process of enculturation as they prepare for college, rather than after they get there.

There is no exhaustive taxonomy of academic cultural norms, but we’ve identified eight key concepts—interdependent and mutually reinforcing—that every student would do well to grasp prior to entering the intellectual culture of academia.

Our FREE .pdf—A High Schooler’s Guide to the Culture of College Academics: 8 Key Concepts—defining and exploring them is available here. For a white paper surveying the literature and laying out a theoretical basis for this guide visit here.

These concepts can be difficult to get across, but they are graspable by high schoolers and, we believe, are in keeping with the level of rigor needed in high school instruction if it is to drive college success.

Part of what’s so hard about the transition to college is precisely the need to learn the operative norms of the intellectual culture when they are not at all self-evident, and not familiar to students from their previous school experience or upbringing. It makes sense, then, to begin providing some enculturation into the norms of college-level academics as part of college prep instruction in high school.

© 2016 BetterRhetor Resources LLC


Function Follows Form

Function Follows Form

The problems with standardized tests lie less with the content they cover than with their very form—which drives their content and everything else about them.

The tests have looked pretty much the way they do ever since the fifties—a bunch of kids all in the same place, bubbling in answers to the same questions, under the same strict time limits, under the watchful gaze of a roving proctor. Replicate across district, state, nation, world.

The tests took this form not because it is good at measuring what kids know, but because it is efficient. The form is the product of 20th-century industrialization, with speed, uniformity, quantifiability, and mass production being the governing virtues.

Large-scale testing of this type was made possible by two industrial-era inventions: 1) the multiple choice question, which allowed for the super-quick recording and evaluation of responses; and 2) Scantron technology, which turned the job of checking answers over to an electronic scanning machine that could read bubble sheets at blinding speed, achieving a quantum leap in scalability and cost efficiency.

A third thing created the large-scale standardized test, as well: the science of psychological measurement, itself born of the quintessentially 20th-century project aimed at bringing the rigor and precision of the hard sciences to the messy business of human thought and behavior.

“If only we could control the variables well enough,” supposed the mid-century psychometricians, presumably adjusting their spectacles and checking their clipboards, “why, we could reliably measure even intelligence itself, quantifying each subject’s relative value within the collective!” Backs were slapped. Huzzahs exchanged.

It sounds kind of scary now, in the way that Taylorism and Skinner Boxes sound scary. And some of this science was indeed put to nefarious ends, such as justifying the exclusion of whole groups of Americans from college admission, as The College Board did in its early years.

But mostly the impulse to quantify was sincerely pointed toward improving and democratizing education. After all, if there were a basis for comparing students under standardized conditions, we might be able to glean some reliable insights into how our education system treats different subgroups, how geographic regions differ and why, whether our efforts to educate are improving over time, etc. Maybe we could figure out who needs more help, and how to teach better, and where best to put our resources.

Or we might even be able simply to look at a number to determine who is ready for college and who is perhaps, ahem, not quite Hahvahd material, so sorry.

Notice, however, how many suspect assumptions underlie the whole project. Are all of these students equally prepared for the experience of taking this test? It is, after all, pretty weird and stressful and artificial. Should all kids be expected to have the same knowledge and abilities? Is there really only one type of academic success? Are we confident that the tasks we’re putting in front of students are yielding the information we want? Given how contrived the test format and testing experience are, do we really even know what we’re measuring?

The big question, of course—the one that forever dogs standardized testing—is this: are the tests measuring what they need to or only what they can? And if only what they can, is that good enough to support the kind of test-results-based inferences we’re making about kids and teachers and schools?

The constraints that format standardization places on content, time, and space drive what gets tested. That is, the validity and reliability of the test require the standardization of conditions—which means corralling groups of kids into the same kinds of spaces, for the same amount of time, and asking them the same questions. The multiple choice format historically makes all this possible, but is only good for eliciting certain kinds of knowledge and skills. The main skill it elicits, of course, is the dubious skill of test-taking itself. It can also do pretty well at testing a kid on basic skills, such as the rules of grammar, but cannot prompt from a student her capacity for “higher order” academic abilities, such as generating original ideas for an essay.

In fact, even including some number of constructed response questions and technology-enhanced items, conventional standardized tests cannot elicit the kinds of things essential for authentic academic work in college:

  • Analyzing conflicting source documents
  • Supporting arguments with evidence
  • Solving complex problems that have no obvious answer
  • Drawing conclusions
  • Offering explanations
  • Conducting research
  • Thinking deeply about what you’re being taught.

They can’t do it given test-taking time constraints, and they can’t do it because the cost of evaluating this kind of student work on a large scale would blow-up the whole enterprise.

And this short list from the National Research Council leaves a lot out, including the many social, dispositional, and behavioral skills students need for success.

When so much in our education system is determined by scores on large-scale standardized tests—especially school funding and teacher evaluations —it is not surprising that many schools, however frustrated in their own efforts, resort to training kids to perform on the tests. Otherwise, they’re out of business. But that means the kids are not learning what they need most for actual academic success, only what they need for test-taking.

To extend the tragedy, our students aren’t even doing well on the tests they’re being trained to take. How do we know this? The tests!

In other words, we’re operating within a strange, Escher-like world, in which standardized tests serve as the instruments used to monitor how much they themselves are screwing up education.

When we consider all the essential knowledge, skills, and abilities that these tests, because of their very form, cannot elicit and measure, it’s clear that we really need to start reevaluating their usefulness.

One further thing to consider: even now, as more and more tests migrate to computer, as computer adaptive testing and tech-enhanced items and automated scoring et cetera become more common, large-scale standardized tests are still overwhelmingly reliant on the multiple choice question.

That is, even as a world of immensely powerful and networked digital technologies has grown up around them, the tests, in their basic form, are still rooted in the mid-20th-century paradigm of the bubble-in Scantron sheet.

This actually gives rise to hope, however. It raises the possibility that we might already have the tools to create different kinds of instruments for education measurement, but that we’re just not using them. These would be instruments that share the original goals of providing insights for improving and democratizing education, but which also overcome the limitations on content, time, and space that have always made old-school standardization such a poor governing principle for assessing students.

© 2016 BetterRhetor Resources LLC


Welcome to the Sausage Factory

Welcome to the Sausage Factory

I used to work for ACT, Inc., designing and developing student assessments. In my final years there, I was Director of the Writing and Communications Literacies group. In one of my last major projects, I headed the team responsible for the revamped ACT Writing Test, which rolled out in 2015.

That roll-out was famously botched. ACT Test Development neglected to tend to the basics of scoring the new test; it’s partner, Pearson, screwed up the reporting. Students and their families had paid and trusted ACT to send scores in time for college application deadlines. There were serious life consequences for missing those deadlines. As a remedy, ACT ham-handedly told students to take a picture of their paper score reports and send that to their prospective colleges. Needless to say, it was all a big mess.

Let it be known that I had nothing to do with that debacle: I was gone months before. (Changes in leadership. Peter principle. ‘Nuff said.)

My team and I improved the form and content of the writing test significantly, moving it away as best we could from the old binary prompt: “Some people think X about this bland made-up issue that you care nothing about; others think Y. Now pick a side and write something intelligible about it in 30 minutes.”

As hard as I worked to improve things, however, I came to realize that the job was hopeless. ACT would never present test-takers with an authentic writing task, one that would engage them, teach them, prepare them for college-level work, or give them a chance to show what they can really do. An exercise in writing that asks students to respond to a topic they have no interest in; that they’ve never even thought about before; that constrains them under a strict, arbitrary time limit; that resembles in no way the work they’ll be asked to do in any other context—well, it’s a pointless exercise at best. Worse than pointless, it’s damaging to a student’s understanding of what competent writing is, and why they should care about it.

The whole (supposed) reason for taking the ACT test is to predict college success. But in fact, largely because of the contrived and constrained form of the test itself, ACT is incapable of presenting to colleges an accurate portrait of a student’s abilities and potential.

The longer I worked at ACT, the more disillusioned I became with the organization—and with conventional standardized testing as a force in education. The tests my group was charged with creating were clearly driving how writing was being taught in schools, and it was a very limited model of writing indeed—certainly not one that encouraged the kind of thinking and communication skills students really need. The thousands upon thousands of student responses we saw each year made this apparent with depressing consistency.

My experience at ACT prompted me to think critically about test development, with two key realizations shaking out:

1. For most kids, there is a huge gap between what they learn in high school and what they are expected to do when they get to college.

This is no stunning revelation. There are plenty of studies and statistics that bear out the fact that a huge percentage of high school graduates are underprepared for college. The ramifications are no secret either: not enough kids go to college; too many need remediation once they get there; too few graduate; too few graduate within four years.

Despite all the emphasis on “college readiness” in education circles, something is obviously not working properly. Actually, many things, but the crucial one that jumped out to my eyes was this:

2. Standardized tests, including—perhaps especially including—the ACT and SAT, are part of the problem.

These tests are highly contrived instruments that do not elicit or assess the kind of student performances required in authentic educational environments. Yet teaching and learning must conform to these tests because of their outsized role in key educational decisions. Little wonder then that so many students are unprepared for the demands of real academic work.

My intention for this blog is not merely to complain about testing, but to give it some serious critique from the point of view of someone with hands-on experience. Maybe this kind of discussion can be valuable to educators or policymakers, or parents, or even test makers, who likewise are thinking hard about how standardized tests are affecting education.

My larger goal in writing this blog, as for BetterRhetor, is to help address the readiness gap between high school and college, and contribute to solutions that lead to more success for more kids.

We want to see college prep instruction and readiness assessment move to a higher level of efficacy; we want to see every student move up a level in education and life opportunities.

At the same time, we want to see the college-admissions playing field leveled up as well, so that students aren’t disadvantaged in their access to education and their readiness for college academics because of their income or background.

The current system is not working. Not enough high schoolers are developing the skills they need for success upon entry into college, despite the rise of standards-based education. The 60-year ACT/SAT admissions testing duopoly, which serves as a gatekeeper for so many students wanting into college, disadvantages and distracts instead of helping kids transition from high school to college academics. We need an alternative. (Click here for a discussion of the duopoly and the college readiness gap.)

We need to make available to colleges not faceless collections of scores and data, but rich, textured portraits of students that show their social, personal, and cognitive abilities, and their promise for academic success. Ultimately, we need a better way to connect students with colleges that believe in them. That’s BetterRhetor’s goal.

© 2016 BetterRhetor Resources LLC


Pin It on Pinterest

Promote college-ready writing for your students

Share with your tribe!