A Probability Exam posing as an Algebra Exam

August 19, 2022 9:25 PM

Guessing C For Every Answer Is Now Enough To Pass The New York State Algebra Exam "The state exam scores also matter to school districts because a school with terrible test scores will be under NYSED’s spotlight — they call it a “focus” school — and come in for mountains of additional paperwork, meetings, and ultimately a humiliating loss of local control. NYSED also prides itself on “data-driven” decisions about what research to pursue, what grants to award, and what programs to fix or update or terminate. Care to guess what data is in the driver’s seat in that data-driven car? You guessed it, test scores. Picture Toonces the Driving Cat, except Toonces is named Test Scores. The bottom line is that because the Regents exams are important, NYSED is very careful to produce high quality tests that challenge the students and accurately measure and report student progress. Ha, just kidding. That cat is driving you right off a cliff."
posted by storybored (29 comments total)

Everyone knows you guess B. That way you have a chance at the true/false ones.
posted by ctmf at 10:26 PM on August 19, 2022 [6 favorites]

In South Africa:
30% is a pass for math lit of you get at least 40% for three other subjects.

Emperor's new clothes.

*maths 'literacy': subject intended as an applied version of maths
posted by BrStekker at 10:37 PM on August 19, 2022

There's more to this though - this is the result of a decision made decades ago, and the ramifications of doing so. The problem is that a lot of those students probably shouldn't be in those courses. This isn't said to be mean - the reality is that the Regents tests were originally intended for the upper level students, the ones who were planning on going on to college. Back when I went through the system (and I'm old enough that I went through the Sequential Course in Mathematics, a.k.a. Sequential Math), only around half of the student body would take Regents examinations - the rest would take local examinations with a set of state diagnostic exams called the RCTs. All that passing the Regents did was add a certification to your diploma - mine has the higher tier, since I was basically what you'd call a "gunner" when I was in HS, and tested well on Regents exams, even acing two.

So what happened?

Simple - the state, under the usual pressures, decided to make Regents certification mandatory. All students would go through a process that was not designed for that, that had mainly been built around testing students who were intending to go to higher education. And with that, the die was cast, because under those pressures, something had to give - you either had to accept you were going to have a lot of fails, or you were going to need to make the test easier.

Guess what happened.
posted by NoxAeternum at 12:03 AM on August 20, 2022 [31 favorites]

Test Scores would make a pretty good cat name, yes
posted by DoctorFedora at 12:19 AM on August 20, 2022 [3 favorites]

I think this teacher places way too much value on standardized test scores, tbh, though I think he'd agree that the test as it stands is a waste of everyone's time.
posted by Aleyn at 1:54 AM on August 20, 2022 [1 favorite]

The teacher understands that standardized tests are rightly understood as gatekeeping mechanisms - for schools and for students. Tests purport to quantify something that can't be quantified accurately (the teacher makes some good points about the low-level content of the test). School isn't a manufacturing plant and students aren't products rolling off an assembly line, but the whole testing industry is predicated on that metaphor. However, a factory operates by reducing costs and maximizing sales, and often double-entry accounting results in ignoring what happens in between. (There's a reason why schools so often falsify results.). Just because you can assign a number to something doesn't make it either more real or more useful, and that's the core mistake.
posted by Peach at 4:45 AM on August 20, 2022 [18 favorites]

Just because you can assign a number to something doesn't make it either more real or more useful

HERESY

It's that kind of attitude that's been responsible for impeding the open-ended productivity improvements that could have cut this corporation's wages expenditure to zero decades ago.

Get with the program. These C-suite bonuses are not going to pay for themselves!
posted by flabdablet at 5:21 AM on August 20, 2022 [5 favorites]

Is this person a math teacher? They're sorely underestimating the ability of people (never mind kids) to botch that free response question, especially if it requires writing a coherent sentence to get points.
posted by hoyland at 5:29 AM on August 20, 2022 [1 favorite]

They have to make it easy because they're not actually teaching how to solve the problems. When I was in school I sat there in math class like it was in a foreign language. No fucking clue what was going on. It was all random words. My mom hired a friend of a friend who was a math teacher who just... told me how to solve the problems. There were rules, and patterns, and it made sense. Wow! I was like Jesus Christ this whole thing is a scam.
posted by bleep at 5:36 AM on August 20, 2022 [15 favorites]

Maybe they should stop having multiple guess tests, and pay people to mark exam scripts with short form answers. It's maths, it's not that hard to mark.

(As an aside, I would expect the standards, teaching and tests to be pitched such that around 80% of students nowadays were capable of passing at the usual age of 15 or 16, and 95% plus by age 18. If we're talking about the level of maths a school leaver should reach.)
posted by plonkee at 6:29 AM on August 20, 2022

“This is dumb. There is a pattern here.”

Heh.

Although questions 19 and 31 in the first set of sample questions the author provides do look reasonable to me. 31 was a short answer question that, yeah, was testing whether students knew the definition of rational/irrational numbers, but in a manner that requires them to actually use the definition in practice, not just be able to pick it out of four options. And if the Regents were an authentic or reasonable assessment, then the policy he complains about of allowing a lower pass rate in the year where students’ schooling was significantly disrupted by the pandemic is reasonable. And the author seems to not give the student described sufficient credit for the knowledge that they do have, as hoyland obliquely alludes to - eg. being able to read a graph, although a basic skill in the grand scheme of university-preparedness-level math, is still a skill that must be learned (sample question 33 in the article), and is generally part of an algebra I class, and the student did in fact need to have that skill to answer the free response question. I suspect the author and I would have some broader disagreements on various aspects of math education; he sounds very old school and traditional in his approach.

But the whole notion of a standardized test being as important as the Regents are - in particular, being sufficient and accurate as the sole data used in making the sort of decisions that the Regents are used for - is really dumb. In other words, I definitely agree that there is a problem here, though I don’t think the author has quite correctly identified it.
posted by eviemath at 6:34 AM on August 20, 2022 [1 favorite]

Re: marking math tests.

So I worked as one of the markers for the College Board AP Calculus tests a couple years in a row a while back. Numbers of students taking that test had been increasing before my stint as a reader, but really took off shortly afterwards. The AP Calculus test is short answer (aka free response). The questions are extremely carefully designed, with common misconceptions and computational mistakes (eg. transpositions or ways that people commonly miswrite numbers when copying them) kept in mind and the specific question parameters chosen so that students aren’t going to accidentally get the right answer through any of the common incorrect reasoning or computational methods. Marking it is still a pretty big task, which keeps the test development team busy for weeks before the full test marking session checking their solution templates against random samples of student tests so that they can develop complete and clear “here’s what mark to assign if the student does this” trainings. Test markers are assigned to teams marking all the same questions for 7 hours a day for I believe it’s now around two weeks. That was after the initial two days of training on the marking rubric, to ensure that the marking would be as consistent as possible despite having tens of (I’m sure now well over a hundred) people working on each question. Because consistency in marking is really important when the standardized test has such big consequences for students and for schools and school districts (as well as, sometimes, for teachers’ career progress), and you might perhaps be surprised at the variety of ways math teachers can mark the same answer to the same question, depending on what they interpret as the important points in the question and solution.

In short, it’s a big task. College Board makes a ton of money and so they can throw money at test development, benchmarking, and marking student responses to do a reasonable job at the AP exams. I imagine the level of state funding for the NY Regents exams isn’t quite as generous.
posted by eviemath at 6:55 AM on August 20, 2022 [16 favorites]

> Is this person a math teacher?

i hope they're not a lit-comp teacher cuz i noped right off of their writing style
posted by glonous keming at 7:44 AM on August 20, 2022 [2 favorites]

I'm imagining right now what would happen if graduation diplomas in my state reflected HOW the student had met the testing requirements... Because most of our students don't pass the common core Pearson developed test (algebra 1 is the easiest and has the highest pass rate, algebra 2 is single-digits except at the district STEM school) and instead meet the requirement in alternate ways like having acceptable ACCUPLACER or ASVAB or SAT or ACT scores. Which is a whole other discussion, since SAT/ACT scores are normed - meaning 500 is always approximately the average score and a certain percentage of students will always score below the threshold no matter what.

The students who have major gaps in math will pass via a portfolio process which has a lot more opportunities for teachers to step in and help them. Imagine if those students graduated with "portfolio" marked on their diplomas...

Anyway the state tests here have also been getting easier, but this is good. They were too hard before. The questions are becoming more straight forward / more visual and the wording is becoming less convoluted. For many students the issue with the original version of the test was mostly a reading comprehension problem more than a problem with the math content. Many students in this district do not speak English at home.
posted by subdee at 8:55 AM on August 20, 2022 [1 favorite]

Worrying about the New York State Algebra Regents exam is my job. I teach Algebra II at a charter school in NYC, and I also chair the math department, so I think about these tests a lot. The tests are high-stakes for my students. We use the Regents as their final exam, and if they don't pass the final exam, they go to summer school and retake the test in August. If they don't pass the August test, they repeat the course. While I try to avoid "teaching to the test," it's what I have to prepare them for, so we do spend a lot of time going over old Regents problems in the context of the current unit, and I think the last 3 weeks of class was all final review / test prep.

The result? Even with the Very Generous Curve that the author identifies (needing only 30% raw score to pass the test with a 65) only 42% of my students passed in June. Well, 48% if you only look at the students who showed up, or whose scores weren't thrown out for cheating. After a second pass in August, that number goes up to 55%. Which, you know, isn't very heartening, as a teacher. The last year for city-wide data for was 2019 (no Regents in '20 or '21), when 70% of all NYC students got a passing score on the Algebra 2 Regents.

So yes, I agree with the author that the curve on these tests is very generous. (Although, shame on him, he has cherry-picked his "look-these-are-so-easy" questions from different Algebra 1 exams.) But if it is really "too easy" then why are only half my students passing? Is it:

1) Post-COVID academic slump
2) This year was the first return to the Regents exams in two years -- most of my students have not taken a Regents before, where in years past this would have been their third years taking Regents exams
3) My own failing as a teacher
4) All Of The Above

I think NoxAeternum is correct -- the Regents wasn't designed for this, but here we are using it. I don't have a coherent point to make here, alas, I'm still just grappling with my results and what it means for me and my students for this year.
posted by Ishbadiddle at 9:04 AM on August 20, 2022 [28 favorites]

Regents tests are a trap and I'm fine with traps that are easy to trick. Although as Ishbadiddle points out, they aren't actually that easy to trick. If you're gonna set kids up to fail, don't complain about the workarounds that develop to keep kids from failing. Complain about the dismal funding/support/resources dedicated to public education. This article was mad at a symptom and pretty shitty and condescending about kids who can't pass this easy, gimme test.
posted by Mavri at 9:38 AM on August 20, 2022 [5 favorites]

The sad reality is low income districts spend so much more time on test prep than
high-income districts and still get worse results. Some of the test questions are good and I have no problem giving the kids one of the good open-ended questions once a month and spending time discussing and comparing answers afterward but giving a lot of multiple choice questions for practice doesn't tend to help much when the big issue is gaps from early schooling (bc of new / inexperienced or absent teachers - these districts have high turnover), lack of student motivation, and not covering enough of the curriculum bc of all the time we spend on test prep and filling in gaps. You kind of need a coherent sequence for math where each topic leads to the next, seeing the questions out of order can work for test prep IN ADDITION to the regular curriculum but it can't replace it. If the students don't have the mental framework for learning new math concepts and aren't motivated to cram the random questions will leave their brains as fast as they entered.
posted by subdee at 10:10 AM on August 20, 2022 [4 favorites]

Like Ishbadiddle I teach in NY and I pretty much came here to comment the same points they and NoxAeternum made. The test is curved so much because it looks bad when too many kids fail. Too many kids fail for many different reasons, but part of it is the fact that many (most?) students will never use the math they learn for the exam ever again. This gets into broader debates in education about the value of math and how/what we choose to teach.

Even the author's cherry-picked examples of "easy" questions illustrate this. Sure the standard form of polynomials or the definition of an irrational number is perhaps a relatively easy fact to memorize for a test, but how many adults know them off the top of their head? How many people use that knowledge in their lives?

I also think highly educated people tend to lack a clear awareness of the lived experience of large portions of the US population. Measuring literacy is notoriously tricky because there are so many different ways to approach measuring it depending on what you focus on, assessing it (especially outside of school-aged kids) is messy, and the need to control for adults for whom English is a second language, but about half of US adults have only basic English literacy (nerds can dig into what the literacy levels the PIAAC, an international comparison, uses look like here and numeracy here). It's hard to summarize what this means in a sentence but basically around half of US adults can read the directions to get their kids to school on time but won't be able to understand their homework past about 5th grade. Approximately 25% of adults struggle to use a browser or email. 1 in 3 US adults are comfortable only with very basic arithmetic, counting, sorting, and similar tasks.

For some more perspective, remember the point made up-thread that the Regents were originally intended for those headed to college. In 1970 10% of Americans over 25 were college grads. In 1990 it was about 20%. In 2010 it was 30% and today we're edging towards 40%. You don't get that change in such a relatively short timespan without changes to the expectations around what it means to be "college-ready." And if only 40% of the kids in your high school pass the Regents exams for English or Algebra 1, your superintendent will crucify you. Rock and a hard place.
posted by Wretch729 at 11:21 AM on August 20, 2022 [17 favorites]

eg. being able to read a graph, although a basic skill in the grand scheme of university-preparedness-level math, is still a skill that must be learned (sample question 33 in the article), and is generally part of an algebra I class, and the student did in fact need to have that skill to answer the free response question.

Seriously, I teach a popular gen-ed college STEM course, and the average level of graph-reading skill is abysmal. It's really easy to miss the fact that students lack this basic skill if you never ask students simple questions about graphs; if you jump right to teaching the subject material, you can easily miss that they haven't read the axis labels, or can't to tell the basic story that is communicated by the trendline—much less understanding the phenomena that created the graph, the causes of maxima or minima, the repercussions of rates of change, or whatever else you're trying to get at.

There is all sorts of conceptual math learning that you can probe with direct questions, rather than requiring complex problem solving.

And, if I could make a possibly-unwarranted generalization, the folks who push for the "rigor" of complex problem solving as the measure of all things tend to be just fine with leaving behind the students who need more support to master core skills.
posted by BrashTech at 11:29 AM on August 20, 2022 [14 favorites]

A few other things, after re-reading the article:

1) The author writes as if the decision to lower the passing grade from a 65 to a 50 has come out of nowhere, or is just part of the inevitable erosion of standards etc. Where has he been for the past 2 years? If you only read this article you might not know that the tests were cancelled in 2020 and 2021, or that very little math learning took place for about a year and a half. The Board of Regents is reacting to that reality.

2) A passing grade is considered “proficient.” It is not considered “college ready.” To get a “college ready” score you need an 81% raw score, not 30%. I tell my students that it’s relatively easy to pass the Regents, but it’s hard to do really well.

3) Part of his complaint is that for some problems you only need to know some definitions. Well, yeah? If my students can actually explain, in a sentence, why 9^(5/2)=243 , using the properties of rational exponents, that’s a win in my book.

4) Similarly, he seems upset that many problems can be solved by just having some mathematical reasoning and knowing how to use a graphing calculator. (They are — gasp! — required!) And, again, I think that’s the point. Putting a formula into a TI-84, plotting points from the table with precision, comparing your pencil graph to the one on your calculator — this may not be the highest level of thinking and problem solving, but it’s a foundational skill that must be learned.
posted by Ishbadiddle at 1:13 PM on August 20, 2022 [6 favorites]

So the graduating-from-high-school standard is really "are you in the upper n-th of your cohort", rather than "can you do all these kinds of problems"? I see why the school system feels so stuck, but wow, that only sort of fixes the students' problems.

I have more faith in people being able to learn stuff later in life than I used to, but also you can get in a lot of trouble misreading graphs until then.
posted by clew at 2:13 PM on August 20, 2022 [1 favorite]

Reading graphs is really important but we still havent taken much away from the curriculum - there's still imaginary numbers (disclaimer: I love imaginary numbers and you do use them in physics and some engineering classes) and factoring cubic polynomials and all sorts of other stuff on there. Well, I guess matrix math and geometry proofs are slowly being taken out and replaced with more graph-reading and context problems. You can't just add things without taking things away, something's gotta give.

Gonna repeat the commenters above who are pointing out that the last 3 years of math education were not normal - especially not in NYC. In my district just outside the state line we were remote from March 2020 to September 2022. A year and a half of virtual with all the testing cancelled, last year was students first back in class in person in nearly two years. You think all those students doing virtual learning were learning math in those two years? Ahahaha no. Some of them were working full time with their cellphones on and logged into zoom. A lot of them were watching younger kids at home. A bunch more were sleeping, playing video games, watching TV, getting high ...

Some kids learned REALLY well virtually but a larger number learned very little during that time.
posted by subdee at 2:37 PM on August 20, 2022 [2 favorites]

Tangentially related to math testing, I regularly hire for finance analyst roles in my team paying $110k minimum from a pool of supposedly "experienced" commercial analysts with years of experience, but when I ask them a simple question around exchange rates it turns out 9 out of 10 people come up with completely bonkers answers that have no logic behind it. We give them 25 minutes to go away and do anything they like (Google, ask a friend, whatever) and answer something that should take any student 5 minutes, or they can stay and ask us any questions they like about it and we'll happily answer it and guide them to the answer, because it's the 21st century and even doctors use Google to help them diagnose patients...

I initially thought this would be a "gimme" question intending to break the ice and get the candidates comfortable with a question they could nail easily, to then spark some interesting discussion around how different companies treat exchange rate variances in their profit reporting - we have different methodologies ourselves depending on what stage the projects are at depending on the objective of the reporting / analysis and I'm open to learning how their company might approach it.

But I did not expect 9 out of 10 shortlisted candidates to be completely unable to answer it. The ones that we DID hire all went on to have really good careers here. As a test, of course, this turned out ideal for us, because you want to fail 90% of people right away so you can get to the one you want to hire, but it's kind of the opposite for a school test I guess...
posted by xdvesper at 4:05 PM on August 20, 2022 [10 favorites]

when I ask them a simple question around exchange rates it turns out 9 out of 10 people come up with completely bonkers answers that have no logic behind it.

Fascinating. Sounds like the finance version of Fizz Buzz:

>Most good programmers should be able to write out on paper a program which does this in a under a couple of minutes.

Similarly I think you can probably weed out ML engineers with a simple question about the central limit theorem. For fairly obvious reasons, the same dynamic at play in Algebra I is still a factor in the popular Masters in Machine Learning degree mills.
posted by pwnguin at 8:31 PM on August 20, 2022

If you're gonna set kids up to fail, don't complain about the workarounds that develop to keep kids from failing. Complain about the dismal funding/support/resources dedicated to public education. This article was mad at a symptom and pretty shitty and condescending about kids who can't pass this easy, gimme test.

That seems unfair. The author was really clearly mad at the test designers, not the kids ("Shame on the NYED and the Regents".) The entire framing is him trying to help a student pass, despite the student not knowing the material (through no fault of their own), because he thinks the test is bullshit.
posted by mark k at 9:32 PM on August 20, 2022 [1 favorite]

pwnguin well it took me just a few minutes to write a short nested if-statement in Excel that does the Fizz-Buzz thing for a column of numbers. Hah, maybe I should include this test this as well... writing custom automation is a pretty significant part of our work too. I can vouch for what the blogger is saying - when you break down a person's inability to build a big thing, it actually comes down to multiple tiny bricks in the wall that they don't know how to fit into place.

One thing I've noticed is that most tertiary education has become application / job-training focused, and in many jobs these analysts basically become machine operators - they perform functions on a black box, follow the instructions, and information comes out the other end.

Few universities have their main focus on fundamentals / research. Those that do even get criticized for teaching / drilling students in fundamental theory that isn't really applicable for most jobs out there...
posted by xdvesper at 10:01 PM on August 20, 2022 [1 favorite]

Mathsemantics by Edward MacNeal [me-prev + NL-prev] is a whole book about better ways than exam scores to identify useful quants-for-hire.
posted by BobTheScientist at 11:09 PM on August 20, 2022

I definitely agree that there is a problem here, though I don’t think the author has quite correctly identified it.

Perhaps he just guessed C.
posted by flabdablet at 3:01 AM on August 21, 2022 [3 favorites]

So yes, I agree with the author that the curve on these tests is very generous. (Although, shame on him, he has cherry-picked his "look-these-are-so-easy" questions from different Algebra 1 exams.) But if it is really "too easy" then why are only half my students passing?

As someone who once trained in test construction at the graduate level I'd just like to point out that any properly constructed test will have easy questions by design. It's how you calibrate and validate the test to ensure it has the proper distribution to effectively discriminate along the desired dimensions. A good test will also have some questions that are almost impossible. When people complain about easy or hard items in a test the only thing you are really learning is that they don't understand tests or testing. Also grading on a curve is post-facto repair of a test that didn't produce the desired distribution. This may be by design or by accident. These are all conscious and deliberate decisions rather than the test producing some accurate measure of an underlying reality independent of the tester's goals.

My opposition to this type of cutoff testing isn't based on the technical aspects of testing which can be improved, adjusted or degraded at will but instead the underlying system that drives the testing. We pretend people are worthy of future investment or not based tests. The problem isn't the tests. The problem is that we constrain our investments in people. There is no wheat and chaff to separated here. There is only people. Teach them all as much as they all want and need and the world is better off. It is the very least we should do. Refining the systems of discarding people is like deliberately not putting enough lifeboats on the titanic and coming up with an accurate "fair" system for deciding who drowns instead of just providing enough lifeboats so nobody does.

So people who subvert the cutoff agenda by making the test "too easy" are effectively my heroes. They are subverting the disinvestment-in-human-beings agenda from within and I salute them for giving more kids more chances to learn.
posted by srboisvert at 4:51 AM on August 21, 2022 [13 favorites]