A living guidance to prepare a simulator exercise

During CALMet 2015 we had a workshop on needs for an Assessment Simulator Exercise seen from
different perspectives.  We worked in groups. The question to answer for each group was: “What do you need to feel yourself safe in this position in the simulator environment.” We have collected all opinions and have put these together into this guidance. In November we have added opinions from the WMO/Eumetcal Train the Trainer participants to this list.

This is a living document which contains a minimum needs list from what we hope that it will expand in the future.  We hope that this document will help you. We have the feeling that you may want to use this document as a guidance to prepare your simulator exercises. In case you don’t have any assessment goals for your simulator exercise please realise that the assessment part of this document most probably still contains some useful points to use in your simulator exercise.

Good luck by organising a simulator exercise and we hope to hear your simulator experiences so that we can learn from them too.

A big thanks to my partners in crime, Mark, Ian and Vesa, for the help with doing these workshops and all the colleagues that gave input on the various occasions.

Heleen

What are the minimum needs you have to think about when organising a simulator exercise?

From forecaster point of view:

• A thorough Briefing before a simulator exercise starts. This includes:

ü  Program of the simulation day (What will be happening during the day; program related)

ü  Be clear what the consequences are in case they fail (think this is most probably also something for the invitation)

ü  Be clear on what you are expecting the forecasters (what products need to be created for instance; what part of the shift you are going to focus on, what the boundaries are)

ü  Assessment criteria you are using

ü  Briefing of the weather situation

• Give the participant time to adapt to the simulated weather situation and the used tools
• Have a schedule with clear tasks during the simulator time for the participant
• Have enough materials available for a proper decision making process.
• Use realistic cases inline with the needs of the participant.
• Before the forecaster is being accessed for the first time please do a practice one first to give the forecaster a feel how it all works.
• Safe and pseudo real environment
• Objectives shall be clear and achievable
• The simulator session should be fair, measurable and authentic
• Keep simulator sessions surprising. Sometimes a simulations need to be “Ohh nothing happened”. Otherwise the forecaster will stop thinking.
• The simulator should not be the only assessment tool in an assessment process.
• Give the participant time to unwind/step out of the simulator exercise
• A good debriefing is very important.

From assessors point of view:

• Pretty much the same desires as the forecasters: It has to be authentic, fair and measurable.
• It should add the right amount of stress: the simulator environment should not stress everyone out. The stress should be the regular amount of stress and pressure to handle that job. We are testing whether people can handle the pressure of doing that job and not whether people can handle the pressure of being tested.
• It should be consisted and not biased: the outcome should not be based on personal, political, racial, and gender things. Like KNMI having two assessors and different way to make sure there is not making sure that there is not a biased process.
• Making sure that the assessors themselves are competent to do the job

ü  They should have some knowledge about assessment. They should be trained in assessment.

ü  They should know what the job is they are assessing.  The assessment team must have some forecasting skills. In order to understand what is going on you need that knowledge

ü  You do need some form of assessment of the assessment process. So the assessors should occasionally be observed. To check that they are doing a good job. The quality of their output should be checked time from time

• The whole thing should be evidence based.
• There should be documentation produced so you can then say: “There is the evidence. This is what we observed. This is what we decided.” This will add to the fairness.
• It should be focussed. So not trying to do aviation forecasting, marine forecasting and now casting and long range forecasting at the same time and every process you have in your organisation.
•  Authentic means:

ü  That assessment shall be connected to the job description

ü  You should be following the right guidelines

ü  Under the right amount of pressure,

ü  Using the right tools

ü  Using it efficiently

ü  You should be following consistently organisations processes

ü  You should be making good decisions as a forecaster following the organisations guidelines and communicating in the right way

So the right person at the right time in the right manner

• How can you assess the soft thing like communication (example phone call)? Suggestions that came from the working group on gathering evidence:

ü  These things should be recorded

ü  Look at the length

ü  Look at the clarity; was the persons language understood

ü  Used the forecaster used the correct language communicating in

ü  Using the right language

ü  Was the forecaster ambiguous or not ambiguous

ü  Was it focused on what was meant to happen

ü  Was it following the standard operational procedure

ü  Did they check whether their message was actually understood

ü  If they were stressed; did they keep that stress to themselves or did they stress out the customers or pilot

ü  Was the communication relevant

ü  Fundamentally you can test this communication question: “Can de decision be made as a result of this communication?” If the answer is yes, then it has been a good form of communication.

From response cell point of view:

Definition Response cell: People in the Response Cell are a kind of middle ground; these are people drawn from the pool of forecasters. That means that they have some insides, but also some unique perspective and often have experience in doing this. It is a strange perspective and sometimes a scary place to be because they are assessing amongst their peers. They are assessing their own skills and these people can reflect back on how they might do their job themselves. There are multiple things they can get out of this position. The key role here is that Response Cell people are the enabler/medium for seeing how the people tested behave in various situations.

• Having good knowledge of the people your are playing

ü  Who are the customers

ü  What are their needs

ü  What language they are using

• Develop a script so you have a realistic scenario; a plan. Not too scripted to prevent you are being lost in case things go of track but a few key points you can go that are testing the things you need to be testing.
• Use a realistic scenario. A real life situation. There is no rule that says that you can’t be creative around a real life situation, but the realer the better, because they will feel it.
• Pre planning:

ü  Make sure that it does work in the way you think is going to work

ü  Make a rubric: lets be clear what we are looking for.

-       So pin down the forecasters skills, communication and behaviours you are looking for.

-       The is not necessary a correct answers to a questions but what you need to pin down is where do you drawn the line of what is acceptable and what is not acceptable.

• Don’t set a simulator exercise set up to fail
• Project the right emotions into the simulator exercise. Play realistic scenarios.
• Prepare something for the people in the response cell where you can easily tick boxes and write your observations. Pressure can also be very high in the response cell.
• Don’t be afraid to make it really extreme, but do keep it realistic in what people can handle in the given time. Remember: very tired people don’t learn!

Tips and tricks:

• You might want to stop the exercise for whatever reason. Show a time out sign and say/shout “NO PLAY”.
• Keep it simple!
• Don’t overload people!
• Use the materials/products they normally use
• Can be good to involve people of the work floor with preparing the simulators
• People who will experience an assessment for the first time might be afraid because they don’t know what to expect. Training them first is advised.
• You can also use simulators for an introduction to a course.
• You can use a simulator to learn participants the limitations of the used products
• You can play with speed to increase the pressure
• In some cases it is very useful to use limited resources for your simulator exercise.  Don’t worry participants will always complain over the amount and the kind of data available.
• Safe the data for your simulator exercise as soon as early as possible. That will safe you time. If possible it is very handy to have a database with weather information at least for a few days.
• Know the screen sizes of the machines you are going to use for the simulator exercise before you start programming.