Student Evaluation: The day one method

tumblr_mluyz16Nk01qzi5bfo1_1280This post is for teachers, especially new PBL teachers. Its about getting student evaluations of your practice. If you’re new to PBL or new to teaching there’s a natural concern about how well you are teaching for understanding not just to be interesting, fun or charismatic.

Inspired by a lengthy explanation of how to go about “student evaluations” (related to higher education) I’ve found over several years, there is often an assumption that this is an end-of-course diagnostic. I won’t go back over the issues students have with evaluation of courses — and teaching, there is a ton of literature about that — yet we’re still seeing lots of discussion about the most effective ‘survey’ which is related to what kind of survey tool to use. Again, not interested in that debate either.

What Project Based Learning taught me, though observation, experience and theory is that all PBL course designs begin with the end in mind. So how can you create an effective student evaluation at the beginning, if in actual fact you’re not too sure how each session will go during the project? In higher education, courses are often required to submit exams in say week 5 of a 16 week course, meaning that teachers are often locked into things they can’t easily change. In schools, the hallowed “scope and sequence” is a compliance document and often only certain people can change it, and many teachers see it as bondage. Do not move off the path!

Of course PBL is all about reducing the distance you can see down the path. You know where the end is, you have plan of how to get there and the kind of resources to take on the trip … but can you really design an effective ‘student evaluation’ at the outset?

Yes. It’s remarkably simple if you’re using technology. Just create a blank Google Document and share the link with your students. That document is something you get to comment on, but they get to author. Personally, I like to have students create ‘anon’ gmail accounts from day one too. but I understand some people will have some regulation which might prevent this method. For you, just share a OneNote or a bank document via Dropbox, some internal drive and so on. There is a way you can do this I assure you. Also, share this document with your colleagues … because they too can give you advice and tips just at the time you might be struggling or unsure.

It’s a critical document because it creates trust between you and them — and shows that throughout the coursework/project you are actively thinking about teaching, and what it means to be an effective teacher. As you change, your students change and society changes around you — this approach lets you revisit your values, technologies and what is important to you and your students THIS time not everytime.

In short, the most important document you can share with a student is a blank one.

Now who’s too busy to do that?

A-W of Useful PBL Words

PBL is known for its open ended questions which can’t be easily answered. Cynically, some will say that PBL can’t be assessed “properly” because of it. Not true. In fact, one reason to become a PBL teacher is because you want to ask questions which demand focused and specific responses. Here is a quick start list of words to build questions and measurable tasks.

Useful objective words for PBL

adjust, alter, analyse, amend, answer, approve, assemble, assess, audit, build, calculate, call, carry out, categorise, check, cross, close, complete, decide, describe, develop, diagnose, divide, draft, draw, eliminate, explain, estimate, extract, file, find, fit, generate, hire, hold, identify, implement, inform, interview, justify, label, lift, list, locate, lower, make, mark, map, monitor, name, negotiate, obtain, operate, perform, pitch, prepare, place, plan, prove, question, read, recommend, remove, report, research, review, schedule, select, sell, solve, spend, state, supervise, spell, test, train, translate, turn, update, use, verify, weigh, write.

How to select great topics for PBL in Australia

The driving question is always a sticking point for teachers new to PBL. Writing a few powerful words in a sentence or two, powerful enough to charge curiosity and enthusiasm is a skill. This is why great copyrighters get paid vast sums for writing relatively little.

The driving question, I always found to be an awkward and misleading term. What PBL is trying to do is drive a topic, not a reply to a philosophic question. Kids are not tested on their philosophic ponderings by the machine. This to me is the biggest reason teachers struggle with PBL early on – it seems too arty or foggy to hit those content standards hard enough. This is not a reflection on BIE, more a reality of the vast cultural differences between how Australia goes about teaching and how America does. America, great at lots of things, drives on the other side of the road and can’t make a car that goes around a corner anything like the Europeans. My point is – Australian teachers need to adapt all US-Import PBL models – and that is hard work.

I prefer to think about topics. I’m not sure crafting one kick-ass question is a brilliant strategy, as learning for kids is all about the extremes of experience and the limits of reality. A kid won’t discover these using the BIE framework though it’s better than the relentless lecture/exercise regime.

PBL in Australia is significantly different to the US (warning to those gazing at US consultancy networks for the answer) – our and their frameworks are significantly different as is the culture and side of the road we drive on. While I respect the hard work and success of groups such saw New Tech Foundation, they are selling a product that was built for America, it still needs heavy adaptation for Australian culture, methods, environments and approaches. It’s not a “one click head-shot” to get better performance or outcomes. Worse still it assumes one method supplants another, and at the really really rubbish end are those who are proposing that PBL combined with business development models are somehow going to improve critical thinking – with no evidence at all to back it up.

PBL is better (in Australian contexts) to be thought of as topic based. Being able to identify quality topics requires using a criteria that can be sustained and justified. It’s easy to be too vague and philosophic when scratching down ‘the driving question’. Don’t do that, it’s a really bad idea.

Not every aspect of the Australian curriculum (or a topic in it) is suitable or needs it! This relates to the idea that PBL is not a full-time requirement either (but if you sell cars, you don’t talk about bikes much), so I wince when people say “we’re a PBL school” – if they are, they are doing the students an injustice in my view.

I prefer the idea that teachers use the best strategy for the job – and the job is to create wonder and curiosity such that students explore the limits and extremes of the world, not the prescriptive view. Even though they undoubtedly benefit from puking up the ‘model answer’ in the big test, and high-stakes HSC teacher (PBL or otherwise) will coach the last term to get those grades (for the students and the school), as least with PBL you can be honest and say, this the answer they want, what would be the opposite, what would be the biggest mistake, the smallest use and so on. It’s better than pretending, and everyone knows how the game is being played – they do anyway, we just elephant in the room it I guess.

So what is the criteria for topics? In my experience, this is something the PBL-lead group establish and help the rest of the staff identify – ahead of trying to actually do it. PBL requires BOTH teacher and exec training of course … it’s dan hard for a teacher to drive on the right when the rest of the school drives on the left.

  • sufficient width
  • sufficient depth
  • sufficient connections with the self—cultural, imaginative, and emotional ties
  • not too constrainedly technical
  • not too general or too unconstrained (e.g. animal is too general, tiger is maybe OK, but cats is optimal)
  • not focused on the more degrading features of human existence or common phobias
  • each topic must provide an equivalently rich experience for all students.

So once you have identified your topics (not based on the fact you HAVE to teach them) – then you can start to think about the kinds of questions that will get kids emotionally involved – and that to my mind is also going to be quite different to much of what I’ve seen in the BIE handbook too.

The primary goal of GBL and PBL is to restore scholarship

What if the primary goal of PBL or GBL for that matter wasn’t improved academic performance?

What if I’m yelling about them because they are essential components of experience, but have been removed by small minds who find thinking somewhat challenging? Well, they are not completely banned, they have just been reduced or rebadged to make them easier to deal with. For example, school camps and college field trips are allowed, as is a weekly dose of competitive gaming outside in shorts. How did these things sneak inside the hallowed halls of academic improvement. They didn’t, the improved academic performance line is simply a an effective way to avoid improved academic performance improvement. It’s sort of brilliant. Like using the weight of the attacker against themselves to win.

What people actually want is stable academic results which show organisational improvement over time, thus reflecting the billiance of those who are running it all. Has nothing to do with well-being, social inclusion, self-efficacy and so on – those are things which the factory worker’s mind never needed.

To all those ‘getting to it’ or ‘getting around to it’, please pay particular attention to research that has been around for over 25 years.

Motivational and self-regulated learning components are essential to the classroom.

This means of course classrooms without them (those lecturing students then giving them a task being the opposite) will never improve under the direction of the regime – yet will of course make easy targets (poor teachers, crap students and so on). The entire ‘learning’ experience is still diffective and incompatible with the world as it exists today – yet this is normal and tolerated. PBL and GBL are ways to take up Civil Rights if nothing else, and I’m not sure what could matter more to kids or parents right now.

Even worse, proven methods (not ideas) such as PBL and GBL are set aside in favour of ‘bacon-thinking’ and other unproven (yet social-media popular) tales and personalities – receiving endless attention and money. Why? Because there is no risk, no compulsion and no data to prove one way or the other.

1. Self-regulation of cognition and behavior is an important aspect of student learning and academic performance in the classroom context (Corno & Mandinach, 1983; Corno & Rohrkemper, 1985).

2. self-regulated learning includes students’ metacognitive strategies for planning, monitoring, and modifying their cognition (e.g., Brown, Bransford, Campione, & Ferrara, 1983; Corno, 1986; Zimmerman & Pons, 1986, 1988).

3. Capable students who persist at a difficult task or block out distractors (i.e., noisy classmates) maintain their cognitive engagement in the task, enabling them to perform better (Corno, 1986; Corno & Rohrkemper, 1985). Different cognitive strategies such as rehearsal, elaboration, and organizational strategies have been found to foster active cognitive engagement in learning and result in higher levels of achievement (Weinstein & Mayer, 1986).

The value component of student motivation involves students’ goals for the task and their beliefs about the importance and interest of the task. Although this component has been conceptualized in a variety of ways (e.g., learning vs. performance goals, intrinsic vs. extrinsic orientation, task value, and intrinsic interest), this motivational component essentially concerns students’ reasons for doing a task. In other words, what are students’ individual answers to the question, “Why am I doing this task?” The research suggests that students with a motivational orientation involving goals of mastery, learning, and challenge, as well as beliefs that the task is interesting and important, will engage in more metacognitive activity, more cognitive strategy use, and more effective effort management (e.g., Ames & Archer, 1988; Dweck & Elliott, 1983; Eccles, 1983; Meece, Blumenfeld, & Hoyle, 1988; Nolen, 1988; Paris & Oka, 1986).

All these references are 20 years old.

What exactly is Capatian Obvious telling us from the stage in 2012 that we don’t know?

My view is they should be challenged to:

1. Present some new evidence what what they say has the impact they claim

2. Present a method, framework and resources required to model/copy/implement in the everyday classroom

3. And be paid no more than a casual lecturer per hour – as that is what they are no matter how entertaining the are.

4. Not talk about or infer experience and expertise in things they have not themselves done/taught or made

In the mean time, PBL and GBL do work. I am pretty sure the last decade of conference lectures has not.

(A post celebrating Mr 11s end of primary school, 6 years of no technological improvement or benefit what-so-ever).

Are you a Cyborg? – Kicking off PBL ideas

More than one person at ISTE commented how hard it was to start off a PBL project – and found that crafting the essential question and providing the ‘entry event’ challenging.

Wikipedia comments “When students use technology as a tool to communicate with others, they take on an active role vs. a passive role of transmitting the information by a teacher, a book, or broadcast. The student is constantly making choices on how to obtain, display, or manipulate information. Technology makes it possible for students to think actively about the choices they make and execute. Every student has the opportunity to get involved either individually or as a group.”

This post, illustrates how to develop these two elements using an example.

We could simply ask are you a cyborg?and ask students to ‘discuss’. Chances are you’ll get a binary response low order response, and so makes a poor PBL question.

If we want to stop kids Googling the answer, then pose questions they can’t Google.

The question becomes something like “How can we create cyborgs to …” or “Can cyborgs help us …” – add you own ending here to make the question open ended, not be easily answered. It should be short, but lead to more granular questions students ask to direct their own learning.

In the age of ‘the internet’, assume students will Google the keywords.

It is a waste of time asking them to do low order response activities that they can Google. Push them into asking their own questions as researchers.

This is where the entry event to a PBL project is critical.

The entry event is the first encounter the students have with the unit of work – and immediately after, they receive sufficient project mapping, information and assessment rubrics to let them know what they ultimately have to do.

So in the entry event, you want to give them core information and ideas that they can explore, without giving explicit direction. If they need it – then they need to learn to ask for it.

If we asked ‘are you a cyborg?‘ – the immediate literal response is no, as a cyborg is a fictional hybrid of human and machine depicted in movies, television and literature.

We might instead give them information that allows them to think about movies, television, literature and more. The first thing I encourage teachers to do is try to write a short response to the essential question themselves – and use that to create the entry event.

So if we begin with the lowest order response “a cyborg is a fictional hybrid of human and machine depicted in movies, television and literature”, we can then provide some minimal information appropriate to the subject, and expand possible avenues for experiential learning. For example, provide a back-story that they can easily relate to.

James Cameron’s “The Terminator” (1984) features a cyborg. In it, the central anti-hero is a manufactured, industrial futuristic robot called the T-800. It was created by machines to perform a human-like assassination with ruthless efficiency, devoid of humanistic emotion or empathy. The film is set against a post apocalyptic background where artificial intelligent agents wage war against their human inventors. This theme, exploring nature’s conflict with technology had been widely explored by writers for decades in several genres including horror (Shelly’s Frankenstien), steam-punk (Jules Verne 20,000 Leagues Under the Sea). Technology in the printing and publishing industry itself enabled the ‘paperback’ to be widely distributed, triggered from the 1950’s ‘B-Movie’ and ‘Pulp-Fiction’ paperbacks.

This might be all you give students. It is packed with thought-triggers – and even as a text, there are multiple ways to ask students to un-pack it.

In visual arts, you might set the end product to be a movie-poster. In drama you might want a performance, in english you might want to explore distopia … so whatever you include … it is there to trigger a goal-orientated reaction, but not provide ‘the answer’.

In literature, author Martin Caidin wrote the sci-fi fiction novel ‘Cyborg’ in 1978, later adapted in the television series “The six million dollar man”. A decade earlier, Caiden explored the concept of ‘bionics’ and ‘artificial intelligence’ in “The God Machine”, describing his characters as having human body parts, replaced by machinery.

This is a smaller example, perhaps something to explore in Studies of Religion or biology … perhaps both.

The cyborg theme is also present in animation (Astro Boy, Battle of Planets) at this same time together with emerging consumer technology game consoles and 1980s arcade (RoboCop,1987) and micro-computer games (Cyborg, 1986 on the Commodore 64) together with seeing a cross-over in traditional ‘dice’ fantasy role-playing games (‘Space Marine’ in the role playing Warhammer 40,000, 1987) where cyborgs fought against mythical fantasy creatures such as Orcs and Trolls from earlier fictions such as Lord of the Rings.

For games based learning twists … let students find old games and explore playing them.

The cyborg was widely represented in popular culture though film, art, literature, animation, games and music where the human is often transformed as the hero or pitched against artificial humanoid representations, exploring what the technological future might become as man attempts to create machine in his own augmented image.

In history, we might look at the role technology as played in war and how it impacted society as a result. The above is a very simple paragraph which might start a class discussion.

Robotics, Artificial Intelligence (AI) and bio-mechanics are however very real area of academic research and development, along with the wider field of ‘cybernetics’ – in which machines and systems combine to produce systems. Arguably, the use of many technological advancements from heart pace-makers to sunglasses – or computers as assistive devices, would in fact make us ‘cyborgs’, participating as part of a wider cybernetic organism – such as the internet.

This might be a topic for design and technology … asking students to create some adaptive technology.

The term ‘cyborg’ figuratively describes a human with machine adaptations, or an entirely technological humanoid created in such a way that it represents a human with artificial intelligence – in either case, the representation to the audience is fictional. The differentiation lies in the context, real or imagined. We are clearly not a literary ‘cyborg’, though figurative perspective though humans do use a range of devices that allow un-natural performance, tools and systems – not least the internet.

The essential question has to be open enough that more than one ‘what if’ scenario can be explored in multiple ways. The entry event must provide sufficient ‘clues’ and stimulus about where students might go, but it does not have to be epic or exhaustive, but it has links with the ‘real world’.

For teachers only just beginning to use technology. Take any of these plain texts and turn them into hyperlinked text, or take them to your librarian and ask for help linking resources to elements of the text. The over all aim is NOT to spend time producing more information for students, but to think more critically about how you present triggers and resources for them to take a more exhilarating role as researchers.

*Note: this post is NOT a road map to PBL, but I hope answers and gives some ideas to those teachers who asked me about these two components at ISTE. Project Based Learning is not GROUP WORK either … so don’t take this a literal road map, but a sign-post.