What you’re about to read should not be taken as advice to vigorously pursue or resolutely avoid certain papers available at the University of Otago because of cold, hard statistics. In fact, that’s the absolute opposite of what this feature intends to promote. Yes, it will serve as a reminder of how your time at university is one of assessment and grading, with figures constantly attached to our work. And that’s something that we should think about more critically.
However, just as important are the less measurable, yet incredibly useful functions of higher education. The Education Act mandates that universities accept a “role as critic and conscience of society.” So while it’s possible to measure your successes (or lack of), it’s also imperative to consider the less easily measured outputs of tertiary study: creating smart, engaged, critical thinkers who can apply themselves to various situations. So don’t treat what you learn in a vacuum. Take your knowledge and build upon it and actually use it for the greater good. This leads us back to the data, and just how limiting it is. Confused? Read on to understand why.
Disparate
The hypocrisy of writing an article to bring transparency to grading, without publishing the tabulated data, appears to ring loud and clear. However, the report slowly crumbled into ambiguity after various Deans and HODs provided robust contextual analysis for the discrepancies.All 100-, 200-, 300- and 400-level undergraduate papers are assessed as “focus papers.” Papers with fewer than 30 students are disregarded. A student’s average grade for other papers at the same level in the same year is compared against their grade for the focus paper.
We’ll engage in a hypothetical scenario to make it more readily understandable. Say we have Mike, a second-year student, who studied MIKE201, MIKE202, MIKE203 and MIKE107 in 2012 (Mike’s a bit of a narcissist). MIKE201 is our focus paper, and Mike received a grade of 82. MIKE105 is disregarded, because it’s at a different level. Our fictitious character received 77 for MIKE202 and 73 for MIKE203. The average of these other 200-level papers is thus 75.
If the grade for the focus paper is five or more marks lower than the average, it falls under the “lower” category; within five marks, it falls under the “same” category; five or more marks higher than the average, it falls under the “higher” category. Here, Mike would fall under the “higher” category where MIKE201 is the focus paper. Once all students in the focus paper have been coded, we are left with a table, looking something similar to this (purely as an example):
The important figure from here is that listed under “difference.” This can be a positive or negative figure. A positive figures indicates that students generally do better in this paper than their averages at the same level in that year, and a negative figure indicates the contrary. When these figures go beyond (-)25, and particularly beyond (-)40, they are listed at Standout Papers. So MIKE201 would trigger a listing as a Standout Paper, coming in with a figure of 46.
From this, it might seem a logical assumption that papers with figures well over 25 are comparatively “easy,” and those with figures heading in the other direction beyond -25 are comparatively “hard.” From year to year there are around 30 courses in the 40 per cent standout range, and around 70 courses in the 25 per cent standout range.
So, does this mean we have around 100 courses taught every year that don’t stack up against the rest of the papers on offer at the University of Otago? We clearly do when numerically analysed under this system. But such a literal approach fails to recognises the nuances of teaching styles, various assessment structures, subject specialisation and student cohorts from year to year. Critic thought it was necessary to unravel the factors that lead to these disparities, so what follows is an account of some of the conversations had with various HODs and Deans.
Bear in mind that while these Departments had Standout Papers on this list, we have elected not to publish the specific papers, how many papers appeared, and whether their figures were positive, negative or a combination of both. Nor has every single paper, of the 222 that appeared in 2012 (many of which had not appeared before), been deeply scrutinised; instead, we spoke with some, but not all, of the departments that appeared more than once. By the time you get to the end of the article, you’ll understand why.
Various Perspectives
“I think you need a more sophisticated analysis than … assuming that some papers are just easier than others,” says Ian McAndrew, HOD of Management. He agrees that assessment structure may attract different students and subsequently influence overall grades, but teaching and learning styles are also relevant.Doug Booth, HOD of Physical Education, Sport and Exercise Sciences, told Critic that any paper average that falls outside a rough 60 to 75 per cent band is closely scrutinised, both within the Major and in the wider School end-of-semester meetings. But why the Standout Papers? Student enjoyment and high levels of motivation, strong engagement with practical content, or a paper that has been newly introduced may trigger a listing on the report.
In the Dentistry and Oral Health field, courses are heavily prescribed. Alison Rich, Associate Dean for Undergraduate Studies, notes that “academic” theoretical papers often see higher grades; by comparison, the “same students may do less well in the clinical papers where clinical skills which are very new … may need to be honed and developed over time.”
Jacques van der Meer, Associate Dean (Academic) of the College of Education, highlights further reasons for varying grades. “In papers where students probably have a higher interest themselves in the subject, or feel more engaged, they tend to do better.” This is reinforced by Rob Aitkin, HOD of Marketing, who says that his “view of education, particularly, is [that] my job [is] to introduce you to some ideas that you may not have thought about, in relation to the experiences you’ve actually had. So we reward people’s critical reflections on their own experiences.”
Christina Hulbe, Dean of Surveying, considers that students may be more “highly motivated in papers that fit the traditional ‘land surveying’ image” compared to other areas. Ultimately, though, it’s hard to say with conviction what the definite causes of Standout Papers might be; indeed, causes may vary from year to year.
The BSurv is a professional degree, so the School is aware of the need to “certify a certain level of achievement in related papers. We are not setting students up for success or for continued growth as professionals if we waver on the standards relative to the statistics of cohort performance.” Broader aims of the School include assisting students in “building specific skills at a more advanced level than provided by secondary education or technical training.”
HOD of Music Graeme Downes remarks that some students enter the subject without having read music notation in its NCEA counterpart. So the Department is clearly focused on providing some fundamental skills to its students; the flipside is that those who are already familiar with this practice will naturally do well in these papers.
Dorian Owen, HOD of Economics, considers that certain papers are targeted at students who will go on to postgraduate study. Such papers have more rigorous content, and students will tend to find them harder. He also noted that some papers are more “challenging and technically demanding … with less subjectivity, with more clearly ‘right’ or ‘wrong’ answers.” So a switch from “verbal and graphical analysis” to more abstract content may see some struggle. Though the department wisely advises students at course approval of the differences between papers, “ultimately, the only way for students to see if it suits them is to try it.”
Despite one particular paper standing out as hard, recent figures show that nearly half the class (from semester one this year) are in the A range. So while it stands out by the criteria of the grade report, it also attracts bright students who still get some very good grades. They might not be within five per cent of their average, but it shows that under scrutiny, the report may not be as useful as initially thought.
Owen even alludes to this himself, asking why the differences between the average percentage grades of the focus paper versus the average of other papers at that level are not calculated. For whatever reason, it still is worth considering and may well result in data that allows “real outliers” to be identified. It’s a valid point given that many of the Standout Papers, from various departments, actually had quite “normal” bell-curves of grades within papers.
Steve Duffull, Dean of Pharmacy, highlighted the difficulty in arriving at specific reasons for Standout Papers. Given that Pharmacy is a competitive-entry course, students will already be achieving well to be admitted into the programme. Likewise, Michael Winikoff, HOD of Information Technology, was unable to provide a comprehensive analysis, though he does point out that the report “is only one piece of information in the bigger picture.”
Even so, the report can indicate genuinely difficult papers. Accountancy and Finance HOD, Colin Campbell-Hunt, noted out that one of the department’s Standout Papers was a new paper that had merged content from two separate papers. This resulted in too high a workload and was therefore “just too ambitious.” Since then, the Department has culled some of the material.
What is appearing is a pattern; though nobody was reluctant to answer these admittedly complex questions, the data doesn’t explain itself in isolation. It’s not necessarily the case that these Standout Papers are easy or hard, but perhaps related to way the lecturer encourages students to think, and how deeply those students respond.
Let's Get Critical
We should be asking ourselves not what the “right” answer is, but why we arrived at a certain result. Downes says critical thinking is something that the Music department does “a lot of. You can’t be a performer or composer (or both) without it, nor can more traditionally ‘written’ areas of music research (musicology and ethnomusicology) do without it … it always spirals out to critical reflection and the nature of the artwork that eventually results.”This kind of critical engagement is the antithesis of techniques such as rote learning. Rote learning does occur, and may help you “get by,” but it won’t stretch you to your best ability. Academic theories and concepts are not to be simply “learnt,” but convincingly understood, applied and critiqued.
Mark Henaghan, the Dean of Law, discusses what critical thinking means to him. “It means you’re able to read something, or assess something, with your own mind, rather than just receive information and pass it on somewhere else.” It’s a skill we take beyond our university years. “When you go out to the world, all you’ve got is your mind, the knowledge you have. Some dissipates very quickly, but the ability to be able to adapt and think for yourself as new situations arise – that’s the most valuable thing you have. If you’ve got that you can adapt to any job, any position, and you can be a major contributor to society.”
Van der Heer says that at the College of Education “everything is framed in contestation,” to promote critical thinking. He explains that “students come into university still with [the] high school assumption that there are definites; that things are right or wrong.” So he provides “provocative statements” to force the students to think deeply about their assumptions and start a “cognitive conflict” in their own heads, and with their peers.
The College of Education aims to make its graduates “develop into critical, reflective teacher-practitioners – so they don’t just go into classrooms thinking they’re going to do technical stuff, but constantly reflecting on what they’re doing, how that’s impacting on student achievement … to constantly reassess, constantly reflect on what they’re doing, and be aware that the social context determines a lot of what happens in a classroom.”
For some, a degree is not always going to be the key to a specific career, and learning itself is the reward. Tony Harland admits that this, however, is a “luxury” for most students, given intensive assessment throughout the semester and the pressures of finding a job. It can be difficult to balance incredibly deep engagement with the content presented in one paper, when you have assessments piling up for two, three, four or five other papers, not to mention commitments independent of your tertiary study. Many students do manage it, and they come out with exceptionally good grades. It is worth asking yourself, however, how much that A is really worth. Think about how many papers you’ve taken and whether the grade you achieved really reflected how much you engaged with the paper. How much knowledge has crystallised itself in your consciousness? What has stayed with you over the years, or over the semester, and what has vanished?
Catherine Fowler, Acting HOD of Media, Film and Communication, states that there are fundamental attributes that certain papers will try to develop, such as “communication, critical thinking and research skills,” but that discrepancies between grading may arise because of students’ strengths and weaknesses, as well as the various assessments. She notes that critical thinking can be tested to some degree, but there is always an aspect of it that “escapes such measurement.” Their successful students will be equipped to “engage critically with issues and problems in a world that increasingly organises itself around media and communication technologies.”
Campbell-Hunt points out that the Accountancy and Finance Department attracts some incredibly bright students. But, he says, there are also those “who are in here for the money. [They] see these subjects as a way of earning a good level of income. They’re not attracted by the ideas.” This certainly touches on a very interesting motivator for university study, and why people not only choose to take up tertiary study, but also pick particular subjects. And let’s be fair; this is not exactly a cheap endeavour, so knowing what your job prospects are before you’ve got a degree in hand isn’t exactly rocket science. (Now there’s something to think about as a career path.) Campbell-Hunt does point out that “some people fall in love with accountancy. They get captured by the beauty of this interlocking system.” But where does the critical thinking come in?
300-level papers are incredibly focussed on it: teasing out the context, understanding managerial decision-making, reporting to shareholders and understanding the role to play in the wider scheme of things. It’s at 300-level where the more “mechanical approach,” despite its necessity, is foregone for deeper thinking. But in order to engage with the material on a deeper, critical level, students need to have a robust understanding of accountancy as “a machine with a set of structures and procedures.” This is what happens in the earlier levels, providing solid foundations for the necessary challenges of 300-level papers.
Harland says critical thinking is the “foundation of university education.” Do you want to glide through your university days and get a comfortable GPA, understanding concepts well, but leaving that thinking behind you as you progress into the “real world”? Or do you want to truly engage with what you learn, build upon your knowledge, and, in the words of Harland, “have the time of your life, a good education, [then] go out and make a difference?”
The Complexities of Assessment
What is taken very seriously across all departments consulted are the markers’ meetings, which allow for thorough scrutiny of grades awarded: the distribution of various grades, the average grade for the paper, whether that average grade sits within an acceptable range, moderation and cross-marking.Monitoring assessment criteria is no easy feat, but to at least mitigate inherent subjectivity, van der Meer explains that “really clear marking schedules” help, as does “clearly align[ing] our learning objectives with what we do in workshops and classes, and then the assessments … it never will be perfect. But we constantly have to strive towards it. That’s why there is so much research and writing about [assessment] in higher education.”
Real Motivation
Downes says that Bachelor of Music students “are often highly motivated and gain entry by audition if they are doing performance, so they are often very good before they even get here.” Music is an extreme example of targeted student excellence, but it’s certainly a university-wide phenomenon. Aitkin is clear on what he wants from his Marketing students: “at the end of three years doing a BCom, or any degree, I’m not so concerned whether they understand business inside out, I want them to be able to think about anything, anything they come up against.”While particular papers might sound like an easy route, if you’re not a naturally talented musician, or passionate about biochemistry, or a dancer with an instinctive affinity for fluid movement, what’s the point in electing – or avoiding – a particular paper because comparative figures suggest they might be result in a higher or lower mark than your average? And possibly a difference of only five percentage points?
For this reason, Critic has decided not to publish the raw data that we have – not yet. If we’re following the mandate of what university education is really about, you should select papers (if you have that luxury, at least) because you want to stretch your mind, not because it’s an easy pass. Of course, with the shift to publishing GPAs on transcripts, it’s advice that sits uneasily with the harsh reality of a competitive job market. But maybe that’s the fundamental problem; maybe we’re thinking about university as a means to an end, when really, as Harland says, they should be some of the best years of our life.
