Thursday, 29 October 2015

What Determines Quality in Training?

A workshop on identifying quality criteria was offered at the CALMet Conference, September, 2015, in Seoul, Korea. The participants were presented with this scenario: You are reviewing a training announcement from an unknown provider in a topic area you are interested in, but you would like to know about the potential quality from a reading of the announcement before investigating further.
Participants were asked to work in small groups to develop criteria within three areas: Organizational, Processes, Product. They were also asked to assign priorities to the criteria as time allowed.

The resulting draft criteria lists compiled from their decisions was then compared to a previously prepared draft list of criteria, which was developed from a review of several existing quality management systems for educational resources. These included ISO/IEC 19796, ISO 29990, Open ECBCheck (for eLearning courses), EFQUEL (for eLearning programmes), Commonwealth of Learning (UNESCO), and the WMO Competency Requirements for Education and Training Providers. Approximately 6 additional systems were examined.

New suggestions for criteria from the workshop were added or merged with the existing ones.

What came out on the top? See the list below, as well as the more complete report. Many more important criteria are contained there, as well as a discussion of the results.

Organizational

•How trainers are hired, trained, and evaluated
•Existing accreditation that covers vocational training, or existing quality assurance processes
•Numbers, roles, and qualifications of training staff members (including access to content expertise)
•Location/country, accessibility (including online presence)
•Facilities, resources, and technology infrastructure

Processes

•how learning outcomes are specified and how they inform the training content and activities
•Good development and delivery processes, documentation of processes, ISO certification (if used)
•How learning needs are determined
•how training is evaluated and improved

Products (including both courses and resources)

•currency of the content (up to date)
•learning activities match learning outcomes, opportunities for students to apply what they learn
•the relevance of course content to job competencies
•assessment methods, matched to learning outcomes
•performance-based learning outcomes
•Timing of courses (Duration and scheduling)
•the results of past evaluations of this course or resource (or related ones), student recommendations


Conclusion

Thanks to those who participated in developing these draft criteria!

As work continues to decide how quality information might be shared, in a WMO Global Campus Shared Calendar and future learning resource Registry, for example, the discussions to date should provide some useful input. But they also raise several issues for further debate.

•How does accreditation relate to quality assurance of vocation training and individual courses and resources? Is additional QA still useful?
•Does ISO 29990 certification cover the required criteria? Is more quality information required for specific resources and courses?
•How can we apply quality standards that are inclusive of many types of training centers and units?
•Is QA information voluntary, or mandatory? How will it be communicated?

Sunday, 4 October 2015

CALMet Online World Café

Roro Yuliana Purwanti and Vesa Nietosvaara

Inspired by the coffee and café tradition in the Korean culture, CALMet Online planning workshop was conducted using the World Café Method. Our goal was to gain new ideas and approaches for successful CALMet Online events in the future.

We started with a 5 minute presentation about previous CALMet Online events, shared a warm welcome and introduced participants to the World Café process and rules (called “café etiquette”). According to the etiquette, each participant chooses the table they would love to sit for the first round. Four different tables were available for participants to sit and discuss the given topic. After 15 minutes, the bell rings and participants move to the other table, change groups and discuss the next topic. Result of the previous topic are left visible on the table, as a reference for the new group.

In the first round, we asked the participant “Do we need CALMet Online? Why? To what purpose”. As a result of the discussion the participants agreed that CALMet Online is still needed, as a forum to meet and share ideas and innovations, as well as to keep communication and collaboration within community. 


In the second round,the participants worked further on “What components in CALMet Online need improvement”. They explored the existing components of each CALMet Online, and identified which components should be improved.

In the third round, the participants were encouraged to imagine “How the future CALMet Online could look like? What other approach can be used?” In response with this topic, beside provided some new approaches on schedule and activities, participants also came up with ideas in technology and communication, such as using more social media and possibility of using mobile device applications. 

In the last round, participants were still sitting in a group, but they now needed to work individually on topic “What can you do to make the next CALMet Online a success?” Their task was to fill in the blank individual worksheet with their advice, feedback and suggestions for success for the next event. They discussed also within the groups.

After the 4th round finished, we asked participants to voluntarily tell us about what had been happening in their tables. They made nice summary that we captured and noted as a reference for further CALMet Online event planning.  Some of them also shared their commitment to contribute in the next events from planning to implementation and evaluation, as response to the last topic. We also had some fresh ideas and initiatives written in their A3 group worksheets.

From facilitator’s perspective, we are happy that the workshop reached its goal of having contributions from CALMet community for the early stages of planning. Fifteen minutes was ideal for each round, because participants needed the first 5 minutes to adapt with the new group and another 10 minutes was suitable to produce some discussion results.

Overall, indicated by some contributions we had for the next events, we can say that the workshop conducted smoothly and helpful for us when planning CALMet Online 2016. More about the World Café method, see http://www.theworldcafe.com/


Tuesday, 29 September 2015

Workshop “Success Profiles for Trainers”

Wilfried Jacobs (DWD), wilfried.jacobs@dwd.de

1.    Background

For successful training not only software or training material of high standard are necessary but also eligible trainers. It is a big challenge to find trainers who are prepared and able to deliver training accordingly. The WMO-Paper No 1114 “Guidelines for Trainers in Meteorological, Hydrological and Climate Service” is already very helpful. However, checklists for assessing trainers and persons who intend to work as trainer are strongly needed. The goal of this workshop was to prepare a list of challenges, required competencies and attitudes that can be used as an input for a success profile for trainers. Different roles (teaching, training management, both) and different training situations (distant training, blended) were discussed.






2.    Results


a)    Challenges

Although many well-known statements were found it made sense to consider them before discussing the personal attributes and competencies.
  • For self-study clear instructions, aims and objectives are necessary and the trainer should avoid the feeling of isolation;
  • During live presentations a compromise between the schedule and flexibility is necessary. A further challenge is to have a plan B in the case of technical problems;
  •  Because online-training (self-study and online presentation) is in principle possible with hundreds of students a big challenge for trainers is to accommodate with many participants, especially if contributions in fora or home work has to be considered. To give all support is one of the biggest challenge and very time consuming.
  • For blended courses an according link/bridge between the online and the classroom part should be achieved. At the beginning some resistance between trainers and students may happen.
  • For all (self-study, online presentation and blended) the challenge is to have enough personal and budget in order to develop and to monitor the material that has to be also logical and consistent. The challenges in relation to pedagogic are manifold: Despite the presentation style the motivation, listening to students, pace, giving feedback also the knowledge of the audience and the different cultural backgrounds are crucial. A further challenge is to be aware in recent developments of media. Additionally, a training support officer is needed for overcoming the technical issues. In relation to the training organization the challenges exist to find experts for teaching and supporting, to get the required budget, to advocate for needs and to provide marketing by announcing a training event or by finding sponsors.

      b)  Competencies and personal attributes

No training roles have been distinguished.
  • Pedagogic / personality: Important attributes are competence in communication, and in engaging people (ability to make the training attractive, narrative and rhetoric skills). Trainers should be empathic, tactful and respectful, emotional intelligent, honest and reliable. Also creativity, curiosity, flexibility, agility of thoughts and a good understanding of their role in combination with availability and responsivity are important. Trainers and managers have to be able to work in a team and to work under pressure.
  •  Even if a training support officer is available the trainer and the training manager should have technical competencies (Learning Management Systems, media in order to select the according teaching method).
  • Expertise (good and current knowledge) is very important for trainers. However, also the manager should have a good overview.
  • A high competence in organizing is required for both trainers and training managers. Whereas the trainer has mainly to organize the training event the training manager needs also competencies in planning training from a strategic point of view, in organizing the budget and the personal and in handling service-orientated.
c)    Conclusions

During this workshop many challenges and required attributes for trainers and training managers have been described. The results can be seen as a promising first step. However, further discussions will be necessary in order to complete the list of attributes and to improve the logical structure. As soon as this has been realized a check list should be prepared that can be used for training of trainers and for self-assessment of trainers. It might help in the selection of trainers according to the challenges in relation to particular training events. Such a list can also be helpful for assessing applicants who are interested to work as a trainer or as a manager of training.

Thursday, 17 September 2015


Strategies for Enhancing Learner Retention (Brad Snyder, Meteorological Service of Canada)

This is designed as a follow-up from the workshop on “Strategies for Enhancing Learner Retention” at CALMet XI, as a way to summarise the findings and hopefully inspire other summaries and further exploration.

It is common for learners who participate in training to forget much of the material in weeks -or even days- after it is given. It is in everyone’s best interest to maximize learner retention and minimize what learners forget. This workshop was designed to give participants an understanding of various types of learner retention strategies and to consider how they can be incorporated into different learning solutions. An ignite presentation introduced some of these strategies (in addition to what was presented in the Booster Question plenary talk).

The following is a summary of learner retention strategies based on Brown et al (2014)

  1. Pre-course reading – spreading out learning by incorporating elements of instruction before formal training.

  1. Flipped classroom - Lectures on line followed by Exercises (Homework) in the classroom

  1. Times for reflection during class – forcing student to retrieve and reflect on information, often in the form of writing or oral exercises.

  1. Quizzes and feedback - quizzes are more effective than restudying material. Feedback helps correct or guide students in the right direction.

  1. Interleaving topics – studies have shown that interleaving or mixing related topics together (as opposed to topics given sequentially) helps build retention.

  1. Post-course reflection – similar to during the class, refection (in the form of exercises or webinars) helps the retrieval process and hence retention.

  1. Booster questions - Questions sent to students after training – in the form of quizzing to practice retrieval and increase retention.

The short workshop had participants assess the effectiveness and/or potential use of these seven strategies for the following modes of delivery.

  1. Online (asynchronous)
  2. Online (synchronous)
  3. Classroom
  4. Blended Course

Participants assigned ratings for each strategy/mode of delivery; 1 being least effective, and 3 being most effective. The table below summarises the findings from 4 break-out groups (according to mode of delivery) at the conference:





Mode of Delivery

Strategy
Online
Asynchronous
Online
Synchronous
Classroom
Blended
Pre-course reading
3
3
1
3
Flipped Classroom
2
2
3
3
Reflection -during
3
1
2
3
Quizzes/feedback
3
2
3
3
Interleaving
1
3
3
3
Reflection - post
2
1
2
3
Booster questions
3
3
2
3



In terms of those strategies which were considered effective, Quizzes/feedback during training and Booster questions following training, ranked highest. The strategy considered least effective was probably post-course reflection.

In terms of mode of delivery considered most effective for these strategies, it was a blended approach (not too surprisingly). Online synchronous was believed to be least effective (slightly) in adopting these strategies.

Had there been more time, it would have been worth discussing the relative merits amongst all participants to get consensus and ensure clarity. What is clear is that there are strategies that we can adopt now to help improve learner retention. It is hoped that this information will inspire trainers to incorporate these strategies in future courses.

Reference

Brown, Peter C., Henry L. Roediger, Mark A. McDaniel. Make It Stick: The Science of Successful Learning. Belknap Press. 2014. Print.




Wednesday, 17 June 2015

Competency assessment: a resource on quality in assessment

At the CALMET conference the presentation from Team ROMATSA on competency assessment got me thinking. A presenter mentioned that the process must be authentic and of course fair. They also mentioned that is can't cost too much. I wondered what other quality criteria or idicators are there for competency assessment processes.

This paper link presents some nice dimensions, and includes authenticity and fairness. I found it a useful read to understand how one group things about quality in assessment. 

Tuesday, 5 May 2015

The Worksheet for Designing Learning Activities

Are you considering including a new, more complex learning activity in your course or training session? One that requires more careful thought and planning? Using the Worksheet for Designing Learning Activities may be of help as you consider all the design factors you need to make it a success. 

This worksheet was developed for use in the 2015 WMO/Eumetcal Online Course for Trainers, but I wanted to make it more widely available through CALMet Commons. We have had great success with it in the course, with participants coming up with some excellent applications for the worksheet.

Please find the worksheet template here among the CALMet Commons resources, as well as this excellent application from Grahame Niles (CIMH).

Tuesday, 31 March 2015

CALMet XI Call for Proposals

It's time to get serious about planning for CALMet XI! The Call for Proposals will remain open through 24 April, and you can now also register for the conference at the KMA registration site and also find information about the venue, hotel, and transportation. We hope you can join us in Seoul for this bi-annual event. We're hoping for a large and diverse group of attendees excited to share best practice and meet new friends and colleagues. Don't miss this rare opportunity to meet other training professionals working in your discipline.

- The CALMet Planning Group

More details can be found at: http://www.calmet.org/p/calmet-xi.html

Monday, 23 March 2015


CALMet Online 2014 – a quarter year after…

With the support of CALMet Working Group, the CALMet Online 2014 was successfully conducted from 6 October to 30 November 2014, with Vesa Nietosvaara (EUMETSAT) and Roro Yuliana Purwanti (BMKG) as the co-chairs. Attended by 185 participants including facilitators, represented organizations from 62 countries and 5 continents, we gathered together old colleagues and welcomed new ones into the community.

What happened during the event…

We worked together, posted questions, provided answers, discussed issues, problems, challenges and alternative solutions, formed team and playing games, collecting materials, sharing resources, practiced forecasting skills using simulators, designed and produced some useful outputs. Those handy outputs vary from a series of learning solution and activities, a webinar checklist, material and resources of Conceptual Models, practical guidance for preparing the new satellite generation to lessons-learned from distance learning and online activities.

What the statistic evaluation result said…

In the frame of region, Europe had the largest number of participants of 63, Latin America in the second place with 45 participants and Africa was in the third rank with 39 participants as shown in figure.



A big number of participants from Argentina probably was thanks to two bilingual sessions in English and Spanish during the event, which made it comfortable to participate the discussion.


Participants from Africa participated actively especially in the synchronous session. In the session, Sarah Kimani, a trainer from IMTR Kenya, introduced weather chat as a tool for training or weather briefing, that can support very low bandwidth webinars. Some African participants actively participated not only because of the weather briefing topic on African weather that week, but also because of a low bandwidth solution for joining the chat. It was was possible to attend this activity while doing the usual daily tasks, as this session had a slower pace than a usual synchronous session. This solution is a good opportunity for countries with have challenges with access to internet to participate in online activities.
 

A big number of interaction during the session of “Webinar Checklist” was probably caused by the topic that was familiar to most of the participants, who have their own experiences of attending the webinar sessions. Participants from novice to the expert joined the, sharing their own experiences.

The session “Teaching with the Conceptual Model” also had the big number of interaction. This was a new topic - but as Conceptual Models are applied and developed in many regions, the topic proved to be interesting.

What we learned as co-chairs and what to suggest for the next CALMet Online of 2016…


Technically we managed really well, there was really a minimum amount of technical challenges, and we thank Maja Kuna for her great assistance before and during the event. The topics were various, as co-chairs we were also happy to see many kinds of training activities presented. On participatory side we really would like to find ways to engage the audience more. However, keeping in mind this is effectively an online conference, we were not expecting everyone participate the same way they would do on an online course.

We also see how this event could be a way to enhance the community. Newcomers have a better opportunity to be recognized by their active participation. Some new resource persons and experts can be identified, this in turn makes it possible to establish new cooperation among experts in the field.
 

To consider for the next CALMet Online: we would suggest the next co-chairs to find out how to best deliver the activities with a maximum input from everyone. We saw excitement that was high during some group activities; maybe study groups could be established in the future event? We think some smaller size communication could be useful also for CALMet Online, and to shift it towards an online workshop rather than an online conference. We also think that the wide spectrum of topics helps to get wider participation and potentially help participants to get what they need.
 

As the event was in a conference format, the certificate provided in the conference was a prove for the attendance and not for measuring competency. We expect that the knowledge, insight and networks the participants gained during the event, can inspire them to develop their training activities in their institution and community.