Wednesday, 11 June 2014

Assessment tool validity (more from #EUMETCAL )

Hi again from lovely Poland,

The discussion this morning concentrates on assessment of forecasters in the content of the CAeM AMP competencies.

Many of us are thinking about what resources and training we need to fulfil this assessment task competently.

The presentations so far so what it happening in a couple of places (Ireland and Russia) in their assessment of AMP. It is really impressive to see the leaps that are being made to prove that the AMP are genuinely competent.

Some of the discussion revolves around what are the right tools for the assessment and how do we make sure we have the right/skilled people doing the assessment.

There is some information on the validity of different ways of assessing from the world of recruitment. There is some research available on how good different methods such as interview, observation etc are at predicting if people will be able to perform in a job.  This resource may be useful: OPP validity of assessment. It comes from the world of recruitment but makes an interesting comparison for our work.

( This reminds me of the Dunning–Kruger effect documented in this paper (unskilled and unaware) which talks about the way that people in the lower levels of skill may grossly over-estimate their skill. The paper starts with a funny story about a thief and some lemon juice.  )

Other questions raised have been:
  • What can be / is the role of peer assessment and what are the issues that need to be managed to make this useful
  • What is the "frame of the assessment" do we describe people who are incompetent in terms of AMP competencies or people who need to develop their competency. 
In implementing the AMP competencies successful organisations seem to have taken in to account the feelings of everyone impacted and had a change process that respects those feelings.

Assessor training is really important, some organisations are in countries that have vocational assessor training practice / training to draw on.

I am curious how the implementation of competencies will play out outside of aviation. This is a question of change management. Outside of aviation, the decision to mandate the competencies will be national rather than come from an external entity. It may well be fine as the process to adopt AMP competencies has broken the taboo about competencies for all of us.

The discussion continues ....



Tuesday, 10 June 2014

Asynchronous online teaching - discussion at #EUMETCAL 2014

Hello from sunny Poland!

EUMETCAL 2014 has just started here are some notes from the discussion on asynchronous (online learning).

EUMETSAT (satellite applications training with IMTR in Kenya) and WMO (training of trainers) recently ran online courses that had no component that required everyone to be in the same "event" at the same time (so asynchronous).

I was involved in both of those (and so an not a neutral observer - if that is possible).

EUMETSAT presented their online course for Africa that as run joint with IMTR. It takes parts of a revised version of the ASMET1 COMET module and makes lessons for each section by having tasks for the participants to compete.

We then discussed what it is like for participants and students in this kind of course and what some of things people might take into consideration in running this kind of event.

Some of the things people mentioned during the discussion were:


  • Time to develop the materials and get the first course in place will be more than you think, even when you take this into account. 
  • Connection with students is different on line, not harder than classroom, just different - it is mediated though all the contact in forums and assignments.
  • Just like in a classroom it is important to welcome and orient people at the beginning of the course
  • Classroom, synchronous online, asynchronous online - there is no best there is only good instructional choice
  • The participants really value the experience particularly being able to work on something real,  around and in their job, over time, and getting good feedback from peers and instructors. 
I am really proud to have been involved in both of these. With hindsight there are many ways they could have been better, but these courses were good. They were good in a significant part because they were designed with participation in mind. 

I think in a lot of discussion were are bumping into many of the ideas expressed in the Edinburgh University Manifesto for Teaching Online (see http://onlineteachingmanifesto.wordpress.com/).  In particular the role of the teacher and what it means to engage with students. 

Pat also noticed that we are also bumping into ideas of instructors presence. Which is an area of interest in the ODL (Online Distance Learning) community. 

And

some of us tweeting from the conference will use the tag #EUMETCAL

And 

Thanks to Team Poland for their work here - it is great!