Learning in the Black Box

The recent changes in Swedish schools have, among other things, meant an emphasis on a new form of course objectives. Students will develop their ability to reflect on their own learning in different subjects, develop their ability to use ways of working to support their own learning and develop an ability to take responsibility for their own learning, and so on. These objectives can be linked to the globally growing movement The 21st Century Skills Movement, which focuses on the question of what it takes to be a highly educated and skilled citizen in the 2000s. In addition to the traditional core skills of reading, writing and arithmetic, it finds also that initiative, ability to take responsibility for their own learning and the ability to choose learning strategies is necessary.

Another change in recent years is the increased amount of assessments, not least in the form of national tests, of which there are currently two in grade three, and five each in grades six and nine. The national tests represent a significant device, and many teachers testify to how much of their time as they take.

But the changes and investments – which are aimed at a school where knowledge is safeguarded – lack coordination. They combine the knowledge and skills of the 2000s with the testing and evaluation styles of the 1900s – a confusion that is doomed to fail. Why invest in new learning goals if we fail to invest in methods to evaluate how well they have been achieved?

Consider the tests common in schools today, with national tests as typical examples. They are based on a basic design used throughout the 1900s. Contrasting them with schools’ new, emerging goals, it is obvious that this kind of test – the evaluation of knowledge – has not followed the trend. First, they are too narrow in what they can evaluate. One cannot assess a student’s ability to take responsibility for and manage their own learning by studying how they behave between two given points in time, when they have also received very clear directives for what they should do. One cannot evaluate a student’s ability to select appropriate learning- or problem-solving strategies, if the situation being evaluated contains degrees of freedom and choice to take on tasks in different ways. One cannot evaluate a student’s willingness and motivation to learn and develop, if the foundation does not run over a longer period of time and include choices for students between different activities.

Secondly, these 1900s tests are generally summative, that is, they focus on the (final) station the student has reached. These tests have certainly been preceded by learning processes and development, but the learning as such remains in the ‘black box’. Such retrospective evaluations compete with the educational work of time and focus. Formative assessment is, however, similar to educational work, present- and forward-looking rather than backward-looking, and focuses on how individuals can improve their performance and develop.

These two shortcomings of today’s 1900s tests have similarities. Information on which ultimate goals students reach does not say anything about how they reach them. But in society, the ability and motivation to learn and develop is highly relevant, and therefore we also want to evaluate this in education. Not that the level of competence in a given situation is uninteresting, but that current evaluation forms focus disproportionately on given static modes. And above all, we want evaluation that is integrated with – rather than competes with – learning and educational work.

Among pedagogic people, the consensus is often that formative assessment is preferable to summative. Yet we see a major investment in summative, standardised evaluation in schools. Why? The answer is that formative assessment is vastly more time-consuming than summative – as long as you stick with the 1900s’ evaluation methods and tools.

This is where technology comes onto the scene. Well-designed digital learning programs can provide powerful support for formative assessment. How to build such programs is an important area of ??research in cognitive science, with both theoretically intriguing questions and direct, practical implications for decisions taken by today’s school politicians for tomorrow’s schools.

First, a clarification of the concept of learning programs in this context. By learning programs, we refer to pedagogical-dedicated digital games in school domains: social studies, natural sciences, Swedish, English, history, mathematics, and so on. These games contain stories or narratives; missions to solve that require the collecting, processing and application of knowledge; feedback and support of various kinds; progression opportunities of various kinds.

Playing a learning game means constantly making choices, and its suitability as an evaluation instrument lies in this fact. A well-formed learning game provides a rich environment in terms of educationally related options: choosing ways of tackling a task, choosing between sources of information, choosing to move on to a more difficult level or not, choosing to deal with a challenge (perhaps in different ways), choosing to rehearse or not, choosing to earnestly work on or take a break at intervals, and so on. A computer can log all the choices students make, and in particular the choices that are of interest to a teacher or evaluator. It can gather information about such things as relevant as the result of a 1900s-style test: which choices students make in a learning situation when no one talks about what they should choose, how students react to change, how they respond to feedback of various kinds, the degree to which students take responsibility and manage their own work, the degree to which students use and choose different methods that are suitable for the task and themselves.

It must be noted that digital learning games cannot be used straight off for evaluation. Assessment of learning is a possible feature of digital learning games, alongside their main function of supporting learning. A prerequisite for using a learning game as an evaluation tool is that it has a module that logs game play, but first and foremost that it also filters the logged information in an appropriate way and produces a clear and readable summary for the teacher.

Take the example of a digital learning game used during a series of lessons in a subject in primary schools. There should be documentation after each lesson in the first step in the form of a simple graphical overview of pupils’ progress, along with information about things that stand out: where have many become stuck? Where have some become stuck? Are there students who differ in what they worked on and how they progressed? Thereafter, the teacher can go in and get more detailed information on individual students about such things as development over time (the progression in all classes), when each student passes easily or has difficulty, which strategies the student prefers for certain tasks, what the students spend the least- and most time on, where each student eventually gets stuck, how the students act at critical moments and handle challenges, how the students respond to various forms of feedback, how focused the students work and, where applicable, how the students interact with other students in the game.

Such a report and such information provides a sound basis for the teacher so that the next lesson has formative educational conversations with a number of selected students – much less haphazardly than when they receive such formative evaluation interactions by the teacher going around and seeing what students are doing, or where the students raise their hands or look like they need help. The usual starting question to students about how it is going and if they are experiencing difficulties, can be skipped – the teacher already has enough information to begin an educational talk, including some information about things that the student may not have thought of on their own.

It also means that for each individual student there is specific information that the teacher and student can use to discuss metacognition, i.e., student thinking about their own thinking and learning. For example, ”When in the game you come to the task where you write captions, you often do it like this, but sometimes you do it like this. What do you think about the two ways?” Or, ”When you solved the task of finding out how the inventions are related to each other to show it to your digital students in the game, you worked very quickly; you then chose to remove some of the sources, how did you think about it?” Or ”This is what you find difficult and takes time, you choose to solve it piece-by-piece, a little each lesson, does this way of working suit you?” Research clearly shows that metacognitive skills – knowledge of learning strategies and the ability to plan and regulate one’s own learning – are critical to a student’s success. It has also been shown that successful teachers use metacognitive information and instruct the learning strategies as part of teaching in a concrete and detailed way. Unfortunately, in Swedish schools, we see a strongly increase in metacognitive tasks of an abstract and general nature, such as in the following ‘reflection task’ for a 12 to 13-year-old: ”Reflect on how satisfied you are with your work and with materials and techniques, with your creativity and expression, and with your work process overall”, or in a system where 9 to 11-year-olds once per semester are asked to write a general comment about ”how I manage to plan my work time.”

It is even possible to use a test that is part of a digital learning game in a more classical manner as a final assessment. Then one can use it advantageously in a similar way as in the U.S. state of Oregon, which administers the national test digitally. There, students can do each test up to three times per year during the period from October to May, with only the best result counting. Immediately after the test is done, both the students and the teacher get the result and can use it to identify what the students need to work on. (For a digital learning game one can correspondingly let students choose three test dates during the time they are working in the subject area and the game.)

Such an approach, with the emphasis on formative assessment, reduces two problems of 1900s-style assessment, namely: 1) the excessive loading of a single test, something we know can create great stress and completely disable some students, and 2) the artificiality, in that at one defined point between two given times is their only chance to produce and deliver for example a text or the solution to a number of tasks – something that is a relatively uncommon occurrence in most professions.

 A further advantage of Oregon’s assessment system – even if the data are traditional test questions, with answers in multiple-choice format, and not the richer form of data offered in a digital learning game – is that the difficulty of the tests is adapted for the student in question. A student answers many questions correctly will continue to get difficult questions, while a student who fails the questions will get easier questions. This provides more detailed information about where a student is than results from a test that has a specific and limited number of questions on a given level of difficulty.

 So far, Swedish schools as an arena for digital media have unfortunately been a strange environment. These media have undoubtedly made inroads in schools, often at high speed and accompanied by grandiose promises, but we have seen extremely little of digital products that actually support teachers in their professional mission, namely to teach, support and assess students in the best and most development-friendly manner. Instead, the heavy betting has been on the surrounding features various administrative platforms – for example, for absence reporting or to fill in level matrices in educators’ language, that sometimes neither the students nor the parents, nor even teachers themselves understand.

 Even the digital platforms that are said to be educational and aimed at formative evaluation use only the digital format without digital functionality, i.e., the computer’s capacity to process, analyse, connect, and compile information. The shortcomings are often serious in their concreteness. Here are some authentic examples of formative evaluative reviews targeted to students in grades six and seven: ”The test results show that you should work on developing the ability to link laboratory results to the issue and write conclusions related to physical models and theories”; ”Your piece about democracy is approved, but to develop you should in future tasks work to define your terms better and bring more nuanced and problematised reasoning”; ”You have achieved your work process and developed a simple sketch. In order to move forward and achieve a higher grade, you should deepen your future work with images and a presentation of it and show a more well-developed analysis of your work.”

 Who seriously believes that these have a positive effect on student learning and development?

Yet those ‘formative’ and often metacognitively orientated reviews are becoming more common in Swedish schools, even though they are in fact ineffective. Formative support, including metacognitive support, must be given at a far more concrete level, as well as in direct connection to what the students are working with and can improve, to be pedagogically effective.

In addition to the focus on administrative platforms (including those called ‘pedagogical’), is a naive belief that the digitisation of schools involves supplying computers or tablet PCs and internet connections – so that teachers and students can use Wikipedia, Microsoft Office and social media, or possibly some of the numerous ‘book-in-a-box’ applications available via the internet.

But the fact is that no business is getting better and stronger with the help of digital hardware and standard programs – whether commercial, dental, architectural or some other activity. Schools require digital tools that are for both teachers and students.