[cross posted from CACM blog]
You have probably come across some viewpoint articles about
computational thinking in CACM in recent years (Guzdial, 2008, Wing, 2006).
Computational thinking is the seductive notion that ways of solving problems
like a computer scientist should be taught to kids early on, like reading,
writing and ‘rithmetic. What’s not to
like about that idea? No doubt every discipline is fond of its methods and approaches
and feels everyone in the general population who know about them.
It’s a potentially useful idea for my current project Making
Games in Schools, which is training teachers to work with game making software
in their classes. The idea is that kids learn important computing concepts
while making their games, and so that participation in the project should make
learners more positive to CS and improve their understanding of such concepts.
But how do you measure this? At the start of the project, I thought: “Great.
I’ll just borrow from the material which is amassing in the CS Education
community about computational thinking and use it as a pre/post test
measurement.” It turns out that the community isn’t quite at that stage yet. It
seems we’re still discussing what the concept of computational thinking might
include in detail, and therefore haven’t got as far as defining an assessment
(if I’m wrong about this, I would be delighted if you would let me know!) I did
have to find some way of measuring
the kids’ knowledge of CS concepts, so here’s what I did.
Our colleagues at Sussex University (http://www.flipproject.org.uk/) have
been working on a sister project to ours, and started thinking about how
computational thinking and a visual programming language for game making might
be related (Howland et al., 2009). Keiron Nicholson had the
brain wave that perhaps you could assess some aspects of understanding of CS
concepts by asking children to play a specially prepared game and then answer
questions about it. Their team developed a pilot game which could be used as a
design tool to find out how children expressed programming constructs in natural
language. In our project, we needed a test which could be administered to
hundreds of children and easily marked, so we developed another version of this
game and matching multiple choice questions. We now have a pre and post test of
matching difficulty (I hope!) and have piloted it for a couple of iterations
with children and teachers. The test is aimed at 11-12 year olds.
Based on the curriculum guidelines in Scotland, and what I
have read about computational thinking, the test covers:
- Identifying rules
describing complete sequences of actions in the right order
- Identifying conditions and
consequences (IF and ELSE)
- Simple operators such as
AND, OR and NOT
- Tracking variable state:
reasoning about what events previously in the game have led to the current
outcome
- Abstraction/
categorisation (see below)
Below is a screen shot from an example encounter.
The player
meets The Guardian of the Well, who asks her to investigate why some of her
animals are being poisoned when they drink from the well. A sequence of
creatures are seen to come and drink at the well, and either choke and die or
continue to gambol around happily. The chickens and imps die, whereas the
spiders and the pigs survive. A feature
of the creature’s morphology determines its fate which you can see from
watching the video (I am not giving away the answer here!). A number of questions relate to this
encounter, an example of which is below.
Essentially, they are intended to
assess whether the viewer can identify rules which govern the behaviour of
members of different classes and then extrapolate from these rules. It could be
taken as a measure of ability to identify inheritance relationships, and to
abstract away from the concrete to the more general.
After pilot testing, we decided to create a video of the
game for the class to view rather than getting individual kids to play the game
for themselves. There are a number of reasons for this, but mostly it is
because you can’t guarantee that the kids will play the game in the fashion
intended and this means that everyone could have a different experience on
which to base their test answers. For example, if a kid skipped a conversation
or took a wrong turning, it could affect their score unfairly. We also found that the test was far too easy (average
score 75%!) so I spent a contorted weekend trying to think of the most devious distracters
I could which would match with different sorts of misconceptions. Our latest
panel of teachers reckon that their classes should find the current version
challenging.
There are of course limitations to this approach. Are we
really measuring computational thinking? This is such a broad concept that it
is almost impossible to say. It is measuring some aspects of it which are
easiest to define. Does it rely too heavily on English language skills such as
reading comprehension? Perhaps, but we hope that enabling the learners to view
a visual representation of the encounters should cut reliance on reading.
Further, their teachers will read the questions out loud and answer vocabulary queries
for the class. Is the multiple choice style of the questions appropriate? It
could measure learners’ abilities to identify rules but not their ability to
produce them. The problem is, of course, that learners at the beginning of
their computing careers have no representation system for expressing rules
apart from natural language, and in the Sussex group’s pilot project they found
it tremendously hard. They are not used to using ordinary language to express
concepts concisely and comprehensively but they have no alternative formalism
at this stage.
You can see videos of the games and the question here and here.
If you are interested in this area, please do take a look
and let me know what you think. I expect another couple of iterations will be
required to get it right so expert opinions and very valuable. If you can think
of more questions or encounters (particularly harder ones) I would be very grateful.
My head is spinning from trying to think of them!
GUZDIAL, M.
2008. Education<br />Paving the way for computational thinking. Commun. ACM, 51, 25-27.
HOWLAND, K., GOOD, J. & NICHOLSON, K. 2009.
Language-based support for computational thinking. Proceedings of the 2009 IEEE Symposium on Visual Languages and
Human-Centric Computing (VL/HCC). IEEE Computer Society.
WING, J. M. 2006. Computational thinking. Commun. ACM, 49, 33-35.