The Future of Testing

The Future of Testing

Given how much the rest of education has changed since the middle of the 20th century, it’s remarkable that the model of large-scale student assessment we have today still looks pretty much the way it did back in the nineteen-fifties: a group of kids under careful watch, lined up in rows of seats in a rigidly controlled space, all asked the same questions, each silently bubbling in answer sheets under the same strict time limits.

To be sure, new technologies have been incorporated into standardized testing over the decades: machine scoring, computerized authoring and delivery, computer adaptive testing, technology-enhanced items, automated essay scoring, automated item generation. But these innovations—not all of them widespread; it’s still very much a paper-and-pencil world in most US schools—haven’t really changed the underlying testing paradigm. Whether computer- or paper-based, the tests still are comprised mostly of multiple-choice questions. They still require highly contrived and regimented conditions for administration. They still make use of the same measurement principles and techniques. They still are governed by the values of 20th-century industrialization: speed, uniformity, cost efficiency, quantifiability, and mass production.

This model of testing persists precisely because it so effectively delivers on machine-age promises of reliability and efficiency at a large scale. But these benefits come at a cost: the fixed content and rigid testing conditions severely constrain the skills and knowledge that can be assessed. A battery of multiple-choice and short-answer questions on a timed test may do a pretty good job of evaluating basic skills, but it falls short of measuring many of the most essential academic competencies: sustained engagement, invention, planning, collaboration, experimenting and revising, spit-shining a finished project—the kind of skills developed through authentic, substantive educational experiences.

Standardized testing has not kept up with advances in learning science. It ignores, for example, the non-cognitive skills that research today tells us are integral to learning—personal resilience, for example, or a willingness to cooperate. What’s more, we acknowledge today that students develop their academic competencies, cognitive and non-cognitive, in particular educational contexts in which their own varied interests, backgrounds, identities, and languages are brought to bear as valued resources. Conventional standardized tests work to neutralize the impact of these variables, rather than incorporate them.

We do need, and will continue to need, large-scale assessments, despite the many dissatisfactions we may have with them at present. Classroom assessment by itself doesn’t tell us what we need to know about student performance at the state or national level. Without large-scale assessment, we’re blind to differences among subgroups and regions, and thus cannot make fully informed decisions about who needs more help, where best to put resources, which efforts are working and which aren’t.

The central problem to address, then, is how to get an accurate assessment of a fuller range of authentic academic competencies in a way that is educative, timely, affordable, and scalable—a tall order indeed. Recognizing the limitations of the existing testing paradigm, the Every Student Succeeds Act (ESSA) of 2015 opened the door for a limited number of states to try out alternative models that might eventually replace existing accountability tests. Thanks in part to this opportunity, plus ever-advancing technologies, new ideas are in the works.

Here are some directions in which the future of testing may be headed:

Classroom-Based Evidence. The assessment of authentic classroom work can provide a fuller and more genuine portrait of student abilities than we get from the snapshot view afforded by timed multiple-choice-based tests. Indeed, portfolio assessments are widely used in a variety of contexts, from individual courses to district-level graduation requirements. Historically, however, they haven’t worked well at scale. Experiments with large-scale portfolio assessment in the 1990s were abandoned as they proved cumbersome and expensive, and as states found it difficult to establish comparability across schools and districts.

Hopes for using collections of authentic student evidence in large-scale assessments are being revived, however, as ESSA creates new opportunities for state-level change. The anti-standardized testing group, FairTest, has developed a model to help guide state system innovations toward local assessment of classroom-based evidence. The model folds teacher-evaluated, student-focused extended projects into a statewide accountability system with built-in checks for quality and comparability. FairTest cites programs already underway in New Hampshire and elsewhere as evidence of where this approach might lead.

The FairTest model doesn’t rely on new technologies, but large-scale portfolio assessment potentially becomes more feasible today, compared with the low-tech version in the nineties, thanks to easier digitization, cheaper storage, and ubiquitous connectivity. More than mere repositories for uploaded student work, platforms today can combine creation and social interaction spaces with advanced data analytics. This creates opportunities for assessing new constructs (research, or collaborative problem-solving, for example), gaining new insights into student competencies (e.g. social skills), and even automating some dimensions of portfolio assessment to make it faster and more affordable. Scholar, a social knowledge platform currently in use in higher education, provides a glimpse into the kind of environment in which large-scale e-portfolio assessment might someday take root.

Real-World Fidelity. Another shortcoming of multiple-choice based standardized tests is that that they do not present students with authentic contexts in which to demonstrate their knowledge and skills. More authentic tasks, critics argue, better elicit the actual skills associated with the constructs measured, and thus lead to more valid test score interpretations.

Computer-based tests create opportunities for item types that more closely resemble real-world activities, compared with traditional multiple-choice questions. Technology-enhanced items (TEIs) can, for example, allow students to manipulate digital objects, highlight text, show their calculations, or respond to multimedia sources. While such items fall short of replicating real-world activities, they do represent a step beyond selecting an answer from a list and filling in a bubble sheet.

Many computer-based versions of standardized tests now add TEIs to the mix of conventional items in hopes of measuring a broader range of skills and improving test validity. In truth, however, TEIs bring their own set of test development challenges. Though eager to use them, test makers at this point do not know very much about what a given TEI might measure beyond a conventional multiple-choice question, if anything. Additionally, in their quest for greater real-world fidelity, TEIs can at the same time introduce a new layer of measurement interference, requiring examinees not only to demonstrate their academic ability, but also to master novel test item formats and response actions.

Despite their current limitations, however, technology-enhanced items will likely continue pushing standardized testing toward greater real-world fidelity, particularly as they grow more adept at simulating authentic problems and interactions, and better at providing test takers with opportunities to devise and exercise their own problem-solving strategies. The latest iteration of the PISA test, a large-scale international assessment, simulates student-to-student interaction to gauge test takers’ collaborative problem-solving skills. Future versions will connect real students with one another in real time.

Continuous Assessment. As tests evolve toward truer representations of real-world tasks, they will likely pick up a trick or two from computer-based games, such as Mars Generation One: Argubot Academy or Physics Playground. These games, like many others, immerse students in complex problem-solving activities. To the extent that conventional test-makers learn likewise to engage students in absorbing tasks, they will better succeed at eliciting the kinds of performances that accurately reflect students’ capabilities. When tasks lack relevance and authenticity they work against students’ ability to demonstrate their best work.

In addition to engaging their interest, computer-based educational games can continuously assess students’ performances without interrupting their learning. The games register a student’s success at accomplishing a task; but more than that, they can capture behind-the-scenes data that reveal, for example, how persistent or creative the student was in finding a solution.

As they develop, platforms delivering academic instruction might also automatically assess some dimensions of authentic student performance as it happens, without interrupting learning activities. The Assessment and Teaching of 21st Century Skills project, from the University of Melbourne, provides an example of how an academic platform can capture log stream and chat stream data to model and evaluate student activity. This kind of “stealth assessment” creates opportunities for including non-cognitive competencies—e.g., level of effort, willingness to contribute—in the overall picture of a student’s abilities.

Inclusion. To achieve statistical reliability, conventional standardized tests demand rigorously uniform test-taker experiences. Accordingly, the tests have always had a hard time accommodating examinees with special needs. Education today, however, persistently leads away from uniformity, towards greater inclusion and accommodation of the whole community of learners, including those with various physical, learning, and language differences.

Computer-based testing presents both opportunities and challenges for accessibility. On one hand, special tools, such as magnifiers and glosses, can be built into standard items. On the other, TEI formats using color, interactivity, response actions requiring fine motor skills, and other features can be difficult or impossible for some test takers. Nevertheless, research suggests that, overall, the digital testing environment can improve access to testing for students with disabilities.

Among the challenges to inclusivity in US testing is the problem of evaluating students who are learning English against standards that assume they already have English language skills. According to Professor Alida Anderson of American University, this problem highlights the need for future assessment systems to be more flexible, not only in the design and delivery of test content, but also in the interpretation and use of standards. Towards that end, programs such as the New York State Bilingual Common Core Initiative are developing bilingual standards and learning progressions that align with English language-based standards frameworks. These efforts promise a fairer and more accurate interpretation of test results for more students.

My own company, BetterRhetor, is combining some of the innovations discussed above in an effort to overcome the limitations of conventional testing (see our long-term vision here). Our web-based platform, for use in classrooms, will deliver five-week instructional modules in Writing and STEM. Assessment of student performance is facilitated by the platform and integrated into instruction. The modules will teach, elicit, capture, and assess not only cognitive skills, but also social and personal competencies. Because students engage over an extended period, we’ll be able to supply actionable feedback, as well as indications of progress. Our overall goal is to provide teachers and schools with a highly effective instructional resource that generates a rich portrait of their students’ authentic abilities.

These kinds of innovation will likely require parallel innovations in measurement science if they are to take hold in large-scale assessment. Test reliability, for instance, might be reframed in terms of negotiated interpretations by panels of local stakeholders, instead of statistical correlations among test scores. Determinations of validity may need to consider how well a test elicits fair and authentic performances from the full complement of learners in an educational community. Comparability across schools and districts may need to take into account the degree to which an assessment supports not just institutional needs but also student learning.

Ideally, future forms of large-scale assessment will function as integral dimensions of learning itself, rather than interruptions or intrusions. They’ll both evaluate and reinforce the full array of knowledge and skills required for the successful completion of real academic work in real educational contexts.

Many thanks to Professor Alida Anderson, School of Education, American University, for her insights into inclusive testing.

© 2018 BetterRhetor Resources LLC

(This post was originally written for and published on Getting Smart.)

BACK TO BLOG HOME

The School Choice Paradox: Competition vs. Monopoly

The School Choice Paradox: Competition vs. Monopoly

Secretary of Education Betsy DeVos recently released a list of proposed priorities for her department’s competitive grants program. Number one is “Empowering Families to Choose a High-Quality Education that Meets Their Child’s Unique Needs.” In other words—no surprise—school choice initiatives remain at the top of the Secretary’s agenda.

DeVos’s argument for school choice, like most everyone else’s, hinges on a faith in competition to improve education. In an open marketplace full of choices, according to advocates, schools compete with one another to attract students. This competition drives up the quality of education all around, as parents choose only the best performing options for their kids. As in any competitive environment, if a school wants to survive, it had better stand and deliver.

The hardest of hardcore school choicers want to minimize government involvement to the greatest extent possible, leaving it entirely up to the competitive market’s invisible hand to ensure that every child has access to a good education. School choice advocates, including DeVos, who characterize the public school system as a dead-end monopoly, believe that government regulations, such as open admissions requirements, price controls, accountability mandates, and reporting requirements, only impede a dynamic market’s ability to improve efficiency and quality.

But these school choicers ignore the basic truth that unregulated markets tend not toward greater competition but toward monopoly. If we turn our education system into an unregulated market, where government has no role to play in ensuring the public good, then we risk trading one type of monopoly for another. School-choice models counting on competition to improve education undercut themselves if they eliminate the government’s role in ensuring competition through regulation.

It’s not difficult to make the case that a laissez-faire market breeds monopolies and oligarchies. Just look at the robber barons in the days before antitrust legislation. Better yet, consider the current status of the US economy: after decades of deregulation, only a handful of companies now dominate media, telecommunications, finance, tech industries, oil, transportation, defense contracting, agriculture, retail, and other key sectors.

Companies don’t want competition: they want to eliminate competitors and have the market all to themselves. It takes government policy to keep that from happening. From the New Deal to the 1970s—the decades of the U.S.’s greatest economic growth and income equality—the federal government developed and enforced policies to limit the consolidation of competitors, enforce fair trade practices, protect consumers, and otherwise regulate markets to reduce barriers to entry, ensure competition, and discourage the rise of monopolies. Systematic deregulation, which began under President Carter and has continued largely without pause through every administration since, has resulted quite demonstrably not in more competition but in less.

It therefore doesn’t take a great leap of imagination to see the same thing happening in education, if school choice policies premised on the benefits of a free market don’t ensure competition through regulation. School choice champions such as DeVos, who rail against the public monopolization of schools, should be just as adamant about guarding education against the private monopolization so evident in other deregulated markets. If education were to go the way of, say, finance or agriculture, it would be dominated by a handful of private, too-big-to-fail companies with proprietary curricula and an incentive to increase returns to shareholders rather than safeguard the public good. Imagine Pearson owning 80 percent of all K-12 schools, free to decide for themselves what to teach, to whom, and toward what end.

What sorts of policies are required to ensure that the market-based education system we are surely headed towards stays competitive? At the minimum, school choice policies must:

Limit bigness. The consolidation of competitors must be constrained, even at the expense of greater efficiency. Unchecked mergers and acquisitions, justified on the basis of increased efficiency, have been the signal route by which a small number of players have eliminated competition and achieved too-big-to-fail status in under-regulated industries.

Keep public schools competitive. As more private players enter education, traditional public schools will serve as a bulwark against the concentration of private power. Currently, however, district schools often are not allowed to compete on equal footing. For example, in many cases charter schools are free to weed out students who lower test scores or cost more to educate. In contrast, traditional public schools must educate everyone who comes through the door, including the students expelled from charter schools. Little wonder then when district schools look worse both financially and academically in comparison to their competitors. School choice policy designed to ensure competition must make sure that traditional public schools aren’t disadvantaged, with their funding siphoned away to help their competitors, and rivals playing by more favorable rules.

Straighten out accountability. As it stands, both the presence and absence of accountability requirements distort incentives, resulting in a market that is neither fair nor efficient. In states such as Michigan, where there is no strong performance-based accountability for charter schools, the incentive is simply to attract as many students as possible. Charters compete fiercely to fill seats, even in poor urban and rural areas, because each enrollee garners around $7,600 in public funds. Schools are not required to show results, however, and so they are not incentivized to actually deliver a high quality education to their students. As a result, the state has lots of charters, but they rank near the bottom of all schools nationwide.

In contrast, in states where all schools are held accountable by student performance measures, charters have a disincentive to serve populations least likely to generate high test scores—namely, the rural and urban poor. Evidence indicates that “for-profit charters are less likely than other types of schools to locate in low-income neighborhoods and educate low-income students.” In other words, there’s no percentage in the market for poor kids.

Public accountability for publicly-funded schools is an obvious tenet of good stewardship. But in a market-based system, policymakers must recognize and remedy the incentive distortions accountability requirements can create.

Require reporting. The competitive market model depends on parents making informed choices. Policy aimed at ensuring competition must define the information essential for fairly comparing competitors and require that schools report it regularly and truthfully. Parents should be able to compare options based on consistent and reliable criteria.

Pitting schools against one another in a competitive marketplace may indeed be a way to improve education. But if we decide that a market-based model is the answer, then we must also be wise about how that market operates. We have no lack of examples of what happens in markets absent government-enforced restrictions on the concentration of power. If we’re committed to reforming our education system through market-based competition, then we must also commit to government regulations that ensure the market works for the public good.

© 2017 BetterRhetor Resources LLC

(This post has also been published in Age of Awareness.)

BACK TO BLOG HOME

When Is School Choice a Bad Choice?

When Is School Choice a Bad Choice?

With Betsy DeVos as head of the Department of Education, it is a sure bet that school choice initiatives will be at the top of national school reform efforts in the coming years. DeVos, a longtime  advocate for alternatives to traditional public schools—including charter schools, vouchers, education savings accounts, online schools, and other ideas—will no doubt support President Trump’s plan to direct $20 billion a year away from traditional public schools and towards school choice programs.

School choice is one of the rare political initiatives that has fans on both the right and left. Unfortunately, there is still no definitive answer as to what kinds of alternatives work best, or even whether policies promoting school choice can improve education nationally. After 25 years of experimentation with charter schools, the results are mixed. Some charters have outperformed traditional public schools, while others have offered no improvement at all. In some cases, charter schools have merely lined the pockets of their founders while failing students utterly. There are similarly mixed results for school voucher programs.

Underlying school choice is a faith in the workings of a competitive market to generate better outcomes. Competition pressures all schools to improve by giving parents, through the exercise of their choice, the power to hold their kids’ schools accountable, so the thinking goes.

The special problem the free market presents in the case of education, however, is that when competitors and experiments fail, children suffer. At stake in a marketplace of education alternatives is not the viability of some consumer gadget, but the life chances of real kids who may never recover from falling behind.

Of course, as school choice champions will be quick to note, children suffer no less tragically when their “monopolistic” traditional public schools fail them, and they have nowhere else to turn.

Education policy based on market principles by definition expects some schools to fail—and, though this point often goes unvoiced, it therefore expects some children to be casualties of those failures. Recognizing this risk, and seeking to mitigate it, is therefore incumbent upon parents, educators, and others, who might reasonably ask: When does policy make school choice a bad choice?

We have enough experience at this point to consider at least a few guidelines:

School choice is a bad choice when it encourages profiteering.

Over $600 billion in federal, state, and local funds go to K-12 public education each year. That’s a tempting pot of public money that many private interests would like to lay hands on. These interests have not been shy about lobbying for models and mechanisms that would siphon some of that money their way, nor in fighting against regulations that would require them to meet accountability standards or make financial disclosures—all in the name of enhancing school choice.

In Secretary DeVos’s home state of Michigan, for example, The Detroit Free Press found widespread abuse, a lack of accountability, and poor academic performance throughout the state’s charter schools, two-thirds of which are run by for-profit management companies.

Online charter schools, dominated by for-profit companies such as Pearson and K12 Inc., have had an “overwhelming negative impact,” according to recent studies, with the majority of students showing “far weaker academic growth in both math and reading compared to their traditional public school peers.”

As with any business, the for-profit entities that contract to run public charter schools have an incentive to maximize revenue while minimizing costs. It’s no surprise then that many seek to enroll all the students they can, while getting by with as few teachers, facilities, and instructional resources as possible. For-profit charters are also motivated to avoid accommodating students with problems or disabilities, since these students can bring down both financial and academic performance measures. When it comes to educating a community’s children with public funds, a for-profit model can conflict with the best interests of students, parents, teachers, and taxpayers.

School choice is a bad choice when it is a tool for advancing ideology rather than education.

In some cases, school choice is promoted by people and entities who see it as but one element in a larger ideological vision for society—a vision in which public services are privatized to the largest extent possible while government is reduced to the bare minimum. When this ideology is the engine behind school reform measures, the goal becomes less about improving education than about devaluing and defunding traditional public schools.

The Mackinac Center for Public Policy, for example, is a Michigan think tank that pushes for deregulation and privatization in many areas of society, including education. The center champions school choice in its rhetoric, but its stated endpoint for education policy is “no government schools,” a stance that favors the elimination of choices rather than their expansion.

Belief in the market is so strong, and enmity towards regulation so fierce in this ideology, that its evangelists resist any sort of government-enforced accountability; for them, “choice and competition engender accountability.” There is no body of evidence supporting this laissez-faire model, only a ferocious faith that it alone can solve the nation’s education problems. There is evidence, however, that school choice without government regulation does not improve education and can harm disadvantaged students

When education policy is set by the belief that any nontraditional school is a better choice than a traditional public school simply because it is not operated by a government entity, then ideology has overtaken evidence-based reasoning. That possibility is worth paying attention to in the current school reform climate, since Secretary DeVos has been a major funder of the Mackinac Center and her husband has served on its board of directors.

School choice is a bad choice when it promotes racial and economic segregation.

In many areas of the country, traditional public school districts replicate entrenched racial and economic divides. Wealthy neighborhoods tend to have well-funded public schools stocked with wealthy kids, while schools in poorer locations are starved for funds and populated mostly by poor kids.

Charter schools and vouchers can help relieve some of the problems of traditional public schools for disadvantaged people, as they have in Oakland and elsewhere. Studies show, however, that charter schools and other school choice programs, whatever the intention behind their authorization, can also reinforce segregation.

A study by the National Bureau of Economic Research, for instance, concluded that “charter schools in North Carolina are increasingly serving the interests of relatively able white students in racially imbalanced schools.” Meanwhile, a judge found that Nevada’s recent experiment with a universal school voucher program mostly benefited wealthy families who already had access to good public schools.

Policies that push for school choice as a means for improving education opportunities need to ensure that mechanisms are in place to make the programs and schools truly accessible to the people who would benefit most. Without an accompanying means of transportation, for example, the choice of a private school on the other side of town may in practice be no choice at all.

—–

While it remains to be seen whether school choice will be a good choice while Betsy DeVos is Secretary of Education, it is worth noting that of the roughly $620 billion spent annually on K-12 public education, less than $60 billion comes from the federal government; the lion’s share of school funding is state and local. Thus, while the proposed $20 billion in federal funds for school choice programs is nothing to sneeze at, its overall impact will be comparatively modest.

The most meaningful choices about school choice will continue to be made at the local and state level. So for anyone interested in making sure that school choice policy in fact leads to good choices, efforts will probably have the most impact closest to home.

© 2017 BetterRhetor Resources LLC

(This post has also been published in BRIGHT Magazine.)

BACK TO BLOG HOME

Pin It on Pinterest

Promote college-ready writing for your students

Share with your tribe!