What about the First-Term ACS Exam?

For longer than the 14 years I’ve been at Elon University, we’ve been administering the full-year ACS final exam in organic chemistry at the end of spring semester. It’s a valuable tool to assess our effectiveness in teaching the fundamental material that students are expected to know, and it also lets us see how our students stack up against students from across the country. In fact, the yearlong ACS exam was one of the assessments I used to assure myself that teaching under a mechanistic organization was better for students (see my previous post). That exam remained relevant for my course after I switched because, after all, the content I was teaching didn’t change; only the order in which it is presented did. But what about the first-term ACS organic exam? How does it fit with a mechanistically organized course? Now that I’ve been administering the first-term ACS exam in my organic classes for a few years, I have some answers to a few specific questions that tend to come up.

1. How well does the coverage on the first-term ACS exam align with the material that’s taught in the first semester under your mechanistic organization?

It’s no secret that the first-term organic ACS exam was written assuming the course follows a traditional functional-group organization. Unlike the full-year exam, which lays down expectations only about the total content at the end of the two-semester sequence, the first-term exam assumes a midpoint. So the exam is essentially defining the path. The good news, though, is that the first half of my textbook (which is how far I get by the end of the first semester) overlaps pretty well with the first-term ACS exam, though obviously it does not do so perfectly. By my count, students are equipped to answer 52 problems out of the 70 on the 2010 exam, or nearly 75%. It’s not 100% because some material in the first half of my book is typically found in the second semester under a traditional functional-group organization. Examples include alpha halogenations and alkylations, the alkylation of amines, the Hofmann elimination, and some biochemical topics such as the general classification of biomolecules and electrophoresis.

2. How do students learning under this mechanistic organization fare on the first-term exam?

You might be surprised. Teaching out of my new textbook, my classes averaged a raw score of 43.8 on the 2010 version of the exam, which corresponds to roughly the 65^th percentile. So my students significantly outperformed those students who made up the national norm. Those students who made up the norm were predominantly (or perhaps entirely) taught under a traditional functional-group organization, so, in theory, they should have been equipped to answer most of the 70 problems. This tells me that the things my students learn, they learn much better. And it’s not just my students. This semester I was on sabbatical and my new colleague, Jamie Ludwig, stepped in to teach all three sections using my textbook for the first time. Her students did even better, averaging a raw score of 45, which is the 68^th percentile!

3. How can the first-term exam be used to assess students who learn under this mechanistic organization?

The full-year exam helps me assess the strength of individual students, as well as the strength of an entire class. With my students on a level playing field with everyone else taking that exam, the percentile scores can be taken essentially at face value. But on the first-term exam, my students haven’t seen as much of the content as students in traditionally organized courses, so the 65^th percentile that they score isn’t the most accurate reflection of how good they truly are at the content they learned. Had they seen more of the content, they presumably would have had significantly higher raw scores. Is there a way to get an idea of what their scores would have been if they were exposed to the same content as traditionally-taught students? I devised a way that you might find useful. It is based on the idea that a student’s raw score, R, is the sum of the number of problems to which they knew the answer, K, and the number of problems on which they guessed, which is the remainder of the 70. If I know R, I can back out K by assuming that students have a 25% chance of guessing correctly. Once I have R, I can scale it based on a reasonable number of ACS exam problems students should typically be equipped to answer if they were in a course organized by functional group, which I’ll call F. The adjusted raw score, R_adj, would then be computed as follows:

R_adj = (3F/156)(R – 17.5) + 17.5

If I think students in a traditional functional-group-organized course would have been equipped to answer 62 problems, then, plugging in for F = 62 and my class average of R = 43.8, the new R_adj comes to 48.9. According to the national norm, this translates into 78^th percentile, which is similar to the 75^th percentile that my classes have been averaging on the full-year exam.

-Joel Karty