{"id":470,"date":"2011-10-31T10:01:18","date_gmt":"2011-10-30T21:01:18","guid":{"rendered":"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the_curious_problem_of_assessi\/"},"modified":"2011-10-31T10:01:18","modified_gmt":"2011-10-30T21:01:18","slug":"the-curious-problem-of-assessi","status":"publish","type":"post","link":"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/","title":{"rendered":"The curious problem of assessing physics"},"content":{"rendered":"

Here’s a bit more on the NCEA<\/a> Physics assessments that I heard about at the NZ Institute of Physics Conference last week. I alluded to it very briefly in a previous post<\/a>.  This comes from my notes of the presentation given by David Lillis, a statistician at the NZ Qualifications Authority<\/a>.<\/p>\n

Unsurprisingly, NZQA throw lots of statistical analyses at the various exams. That’s quite reassuring – they are looking at whether each exam is doing its job properly. That is, does it provide a fair and reasonable assessment of a candidate’s performance in a particular area? They do things like look for gender\/culture bias in particular questions, whether there is redundancy in questions (i.e. two questions on the exam ask basically the same thing – this can be checked through correlations between candidates’ scores on different questions), whether the question\/exam succeeds in distinguishing ‘not achieved’, ‘achieved’, ‘merit’ and ‘excellence’, what is the relative ‘difficulty’ of an exam (e.g. by looking at how candidates in one exam fared in other exams) and so on.<\/p>\n

But the analysis that was most interesting for me was the Principal Component Analysis<\/a> on the students’ responses to the  physics assessment standards.<\/p>\n

Principal Component Analysis is a pretty mathematical thing to work through, but, explaining in non-mathematical terms, it’s about finding out how many and what ‘dimensions’ are required to explain the variation in the results. So, an example – if you look at how the contestants do on ‘Masterchef’ or some equivalent competition, you might find that most of their performance is down to how well they can cook (the first, most important dimension), but part of their performance (the second dimension) might be attributable to their ability to select combinations of flavours that work, and a little bit (a third dimension) on their ability to present their dishes. PCA on an exam or assessment helps to draw out what the assessment really is doing. One would hope that it is really assessing what it is meant to be assessing.<\/p>\n

Now, here’s the thing with physics. For nearly all NCEA assessment standards EXCEPT physics, there is only ever one important dimension. That is, students who do well on (say) question 1, will also do well on question 2, and well on question 3, because, fundamentally, all aspects of all questions are addressing the dimension.  However, for physics assessments, there are usually two important dimensions.  So, for example, if we take the standard ‘Understanding waves’, we see that the first dimension, the one that explains most of the variation in the students’ scores, is how well do the students understand waves? And that’s good. But there is another dimension that’s apparent, and that’s to do with the quantitative or qualitative nature of questions, in other words, how much mathematical work is in a question. To get really good marks in the standard, you have to not only understand about waves, but be able to undertake quantitative calculations too. So really there are two different aspects being examined at the same time.<\/p>\n

My understanding from the talk was that nearly all physics assessments show this, but it is rare in another subject (even a science subject).<\/p>\n

So, the conclusion that can be made from that is that physics exams don’t just assess physics; they also assess to some extent maths.  This probably won’t come as a surprise to most physics teachers, but it came as a surprise to me to see it so clearly evidenced through a careful statistical analysis. Is this two-dimensional nature a good or bad thing? I’m not sure on that one. My gut feeling is that physics should be about physics, not mathematics, and our assessments should reflect it, but on the other hand there’s no getting away from the fact that physics is a quantitative subject. It is good, though, to see that in NCEA at least the ‘Understanding the Physics’ is the primary dimension, NOT the ‘doing maths’. I’d love to analyze some of the exams we set here to see if that remains the case at university.<\/p>\n

 <\/p>\n

 <\/p>\n

 <\/p>\n","protected":false},"excerpt":{"rendered":"

Here’s a bit more on the NCEA Physics assessments that I heard about at the NZ Institute of Physics Conference last week. I alluded to it very briefly in a previous post.  This comes from my notes of the presentation given by David Lillis, a statistician at the NZ Qualifications Authority. Unsurprisingly, NZQA throw lots […]<\/p>\n","protected":false},"author":20,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[19,13,18],"yoast_head":"\nThe curious problem of assessing physics : Physics Stop<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"The curious problem of assessing physics : Physics Stop\" \/>\n<meta property=\"og:description\" content=\"Here’s a bit more on the NCEA Physics assessments that I heard about at the NZ Institute of Physics Conference last week. I alluded to it very briefly in a previous post.  This comes from my notes of the presentation given by David Lillis, a statistician at the NZ Qualifications Authority. Unsurprisingly, NZQA throw lots […]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/\" \/>\n<meta property=\"og:site_name\" content=\"Physics Stop\" \/>\n<meta property=\"article:published_time\" content=\"2011-10-30T21:01:18+00:00\" \/>\n<meta name=\"author\" content=\"Marcus\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Marcus\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"3 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\/\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/\",\"url\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/\",\"name\":\"The curious problem of assessing physics : Physics Stop\",\"isPartOf\":{\"@id\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/#website\"},\"datePublished\":\"2011-10-30T21:01:18+00:00\",\"dateModified\":\"2011-10-30T21:01:18+00:00\",\"author\":{\"@id\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/#\/schema\/person\/34d359280e44f926f2df12eb3e50aa22\"},\"breadcrumb\":{\"@id\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/#breadcrumb\"},\"inLanguage\":\"en-NZ\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"The curious problem of assessing physics\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/#website\",\"url\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/\",\"name\":\"Physics Stop\",\"description\":\"A light-hearted look at everyday physics going on in the world, written by a physicist but intended for a general audience.\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/?s={search_term_string}\"},\"query-input\":\"required name=search_term_string\"}],\"inLanguage\":\"en-NZ\"},{\"@type\":\"Person\",\"@id\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/#\/schema\/person\/34d359280e44f926f2df12eb3e50aa22\",\"name\":\"Marcus\",\"url\":\"https:\/\/blog.waikato.ac.nz\/physicsstop\/author\/mtwilson\/\"}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"The curious problem of assessing physics : Physics Stop","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/","og_locale":"en_US","og_type":"article","og_title":"The curious problem of assessing physics : Physics Stop","og_description":"Here’s a bit more on the NCEA Physics assessments that I heard about at the NZ Institute of Physics Conference last week. I alluded to it very briefly in a previous post.  This comes from my notes of the presentation given by David Lillis, a statistician at the NZ Qualifications Authority. Unsurprisingly, NZQA throw lots […]","og_url":"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/","og_site_name":"Physics Stop","article_published_time":"2011-10-30T21:01:18+00:00","author":"Marcus","twitter_card":"summary_large_image","twitter_misc":{"Written by":"Marcus","Est. reading time":"3 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/","url":"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/","name":"The curious problem of assessing physics : Physics Stop","isPartOf":{"@id":"https:\/\/blog.waikato.ac.nz\/physicsstop\/#website"},"datePublished":"2011-10-30T21:01:18+00:00","dateModified":"2011-10-30T21:01:18+00:00","author":{"@id":"https:\/\/blog.waikato.ac.nz\/physicsstop\/#\/schema\/person\/34d359280e44f926f2df12eb3e50aa22"},"breadcrumb":{"@id":"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/#breadcrumb"},"inLanguage":"en-NZ","potentialAction":[{"@type":"ReadAction","target":["https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/blog.waikato.ac.nz\/physicsstop\/2011\/10\/31\/the-curious-problem-of-assessi\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/blog.waikato.ac.nz\/physicsstop\/"},{"@type":"ListItem","position":2,"name":"The curious problem of assessing physics"}]},{"@type":"WebSite","@id":"https:\/\/blog.waikato.ac.nz\/physicsstop\/#website","url":"https:\/\/blog.waikato.ac.nz\/physicsstop\/","name":"Physics Stop","description":"A light-hearted look at everyday physics going on in the world, written by a physicist but intended for a general audience.","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/blog.waikato.ac.nz\/physicsstop\/?s={search_term_string}"},"query-input":"required name=search_term_string"}],"inLanguage":"en-NZ"},{"@type":"Person","@id":"https:\/\/blog.waikato.ac.nz\/physicsstop\/#\/schema\/person\/34d359280e44f926f2df12eb3e50aa22","name":"Marcus","url":"https:\/\/blog.waikato.ac.nz\/physicsstop\/author\/mtwilson\/"}]}},"_links":{"self":[{"href":"https:\/\/blog.waikato.ac.nz\/physicsstop\/wp-json\/wp\/v2\/posts\/470"}],"collection":[{"href":"https:\/\/blog.waikato.ac.nz\/physicsstop\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/blog.waikato.ac.nz\/physicsstop\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/blog.waikato.ac.nz\/physicsstop\/wp-json\/wp\/v2\/users\/20"}],"replies":[{"embeddable":true,"href":"https:\/\/blog.waikato.ac.nz\/physicsstop\/wp-json\/wp\/v2\/comments?post=470"}],"version-history":[{"count":0,"href":"https:\/\/blog.waikato.ac.nz\/physicsstop\/wp-json\/wp\/v2\/posts\/470\/revisions"}],"wp:attachment":[{"href":"https:\/\/blog.waikato.ac.nz\/physicsstop\/wp-json\/wp\/v2\/media?parent=470"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/blog.waikato.ac.nz\/physicsstop\/wp-json\/wp\/v2\/categories?post=470"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/blog.waikato.ac.nz\/physicsstop\/wp-json\/wp\/v2\/tags?post=470"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}