PREPARING TEACHERS TO TEACH WITH ICT

Cyprus Pedagogical PREPARING TEACHERS TO TEACH WITH ICT LEONARDO EUPT3 PROJECT Deliverable D9: Trials Evaluation Plan and Instruments WP7: Pilot Testing and Implementation of the training in each partner country WP Leader: Cyprus Pedagogical Project No: 2009-1-CY1-LEO05-00465

Cyprus Pedagogical Contact person for Report Anastasia Economou Cyprus Pedagogical anasta@cyearn.pi.ac.cy Version control V 1.0 V 2.0 Final Version First draft Second draft Final August 18, 2010, August 30, 2010 September 30, 2010

Abbreviations Cyprus Pedagogical CARDET OU UNIPI CPI EUPT3 TTLLF LD OULDI ICT Center for the Advancement of Research and Development in Educational Technology Ltd, Cyprus The UK Open University University of Piraeus Research Center, Greece Cyprus Pedagogical Leonardo Project on Preparing Teachers to Teach with ICT Teacher Training and Life Long Learning Framework Learning Design Open University Learning Design Initiative Information and Communication Technologies

Cyprus Pedagogical Table of Contents 1. Introduction... 1 1.1. Executive Summary... 1 1.1.1. Training interventions... 1 1.1.2. Evaluation Plan... 1 1.2. Purpose of this document:... 2 1.3. Introduction to the sections in this report... 2 2. Evaluation framework... 3 2.1. Scope of the evaluation... 3 2.2. Evaluation framework... 3 3. Research tools... 5 4. Methodology... 6 4.1. Data Collection... 6 4.2. Data Analysis... 6 5. References... 7 6. Appendices... 8 Appendix 1: Questionnaire... 9 Appendix 2: Interview protocol... 19 Appendix 3: Participants Usage pattern checklist... 20 Appendix 4: Expert Review Rubric... 21

Cyprus Pedagogical 1. Introduction 1.1. Executive Summary The aims of Work Package 7 as described in the Description of Work document of EUPT3 are to: develop instruments for the evaluation of the modules pilot test the curriculum and online environment in partner countries further implement training with more teacher-participants collect and analyze data prepare a report for improving the full implementations During Work Package 7, there will be three training interventions. In each training intervention the two training modules will be implemented (please also refer to D4 and D5 for more detailed description). 1.1.1. Training interventions The first training intervention will target teacher trainers/coaches and will require 6 hours to complete in a one day workshop. The second intervention will target the broader base of school teachers and will require 15 hours of centralized training, as well as school-based support and implementation, as follows: two face-to-face meetings for a total of 10 hours one online meeting for a total of 5 hours asynchronous communication and individual work using the online learning environment and various tools school-based support and visits 1.1.2. Evaluation Plan The evaluation is comprised of three major activities as follows: 1. Core intervention testing. The training modules will be implemented and tested by at least 30 teachers/trainers and a facilitator in each partner country to assess their effectiveness in order to revise and improve them. Data collection will take place concurrently and extend after completion of testing the modules. The following instruments will be designed to be used for data collection purposes: 1) 2 questionnaires: demographics, and participants perceptions/satisfaction that will include a combination of open and closed-ended questions to discover teachers usage and preferences of all components of the training modules 1

Cyprus Pedagogical 2) a semi-structured interview protocol to be used in interviews with teacher-participants to uncover in more depth their usage patterns, preferences, and perceptions in regards to the training 3) participants usage patterns 4) expert review rubrics for content and tools Data will be analyzed in two ways: quantitative analysis will provide statistics on usage patterns (e.g. number of downloads of instructional material), participation rates, and learner satisfaction with the training, while qualitative analysis will be conducted through the constant comparative method to discover common themes in regards to the success and limitations of the training modules. 2. Broader teacher implementation of training. Following the first activity testing, the training will be implemented with two groups of 20 teachers and trainers each in Greece and 2 groups in the UK. In Cyprus, a total of 100 teachers/trainers will be trained using the material, while 100 more teachers will be trained on a school-based support programme. The same data collection and analysis measures will be used as described in the first phase above. Teachers who complete the training both during initial testing and implementation phases will receive a certificate. 3. Continuous professional development. Following the training, teachers will be part of the online community and continue to use the tools and resources provided. Feedback will be gathered through interviews by school-based coaches as well as through data collection from online access on the Cloudworks space. Results from this evaluation will be communicated through a report (D10), which will also include suggested further improvements for the teacher training. Based on the evaluation report, modules will be revised accordingly. 1.2. Purpose of this document: The purpose of this document is to: describe a framework on which the evaluation plan will be based provide the instruments to be used for the data collection describe the methodology of data collection and analysis 1.3. Introduction to the sections in this report This report consists of the following sections: Section 1: gives a short introduction to Work Package 7 and Deliverable 9 Section 2: describes the framework developed for the evaluation Section 3: describes the instruments to be used for the data collection Section 4: describes the methodology of data collection and analysis 2

2. Evaluation framework Cyprus Pedagogical 2.1. Scope of the evaluation The goal of the evaluation is to generate useful information on challenges and successes of the EUPT3 training content and implementation, in respects of successful preparation of teachers in teaching and learning with ICT. The training involves two major modules: Module 1: Selecting tools, Creating, and Implementing ICT-based Learning Activities The first module will focus on training teachers on how to select appropriate activities, ICT tools, and resources so as to integrate them in their lesson design. To do this they will use visualization tools. Upon completion of the module, teachers will be able to: 1. create learning designs that integrate ICT in the classroom by selecting appropriate ICT-related activities, strategies and tools to address a lesson s objectives 2. select and adapt existing learning designs from a repository (e.g. www.eepimorfosi.ac.cy, www.skoool.gr) that are appropriate based on a lesson s objectives 3. implement learning designs and ICT-related activities in the classroom Module 2: Sharing, Reflecting, Discussing, and Improving Lesson Plans The aim of the second module is to advance teachers lifelong learning skills in the area of ICT by sharing, reflecting, discussing/debating, improving, and reusing learning designs. In order to accomplish this aim, we will first train teachers to use the tools, and then provide them with multiple hands-on opportunities to share, and learn from each other. To do this they will use a social networking environment. These two modules comprise the major challenges for the exploitability of the project that will also construct the framework of the evaluation. 2.2. Evaluation framework A framework was developed in order to facilitate the achievement of a common structure and content, directly related to the aims and objectives of the evaluation, throughout the various research tools. The evaluation thus will investigate the stakeholders views on the framework shown in Diagram 1. It will also gather data from observations during the school-based implementation, as well as date from the online participation. 3

Diagram 1: Evaluation framework Cyprus Pedagogical Stakeholders views Facilitators Trainers/ coaches Teachers Content o Modules o Interventions Tools o Usability o Usefulness Resources o Usability o Usefulness Activities o Pedagogical value o Curriculum relevance o Added value to learning Support o Pedagogical support o Technical support o Facilitator support School-based observations Online participation Learning design process Learning design outcomes Sharing, reflecting, adapting 4

3. Research tools Cyprus Pedagogical Data collection will take place concurrently and extend after completion of testing the modules. The following instruments are designed to be used for data collection purposes: Two questionnaires: 1) Demographics questionnaire 2) Participants perceptions/satisfaction questionnaire that will include a combination of open and closed-ended questions to discover teachers usage and preferences of all components of the training modules Semi-structured interview protocol to be used in interviews with teacherparticipants to uncover in more depth their usage patterns, preferences, and perceptions in regards to the training Participants usage patterns Expert review rubrics for content and tools The instruments listed above are included in the Appendices of this document. 5

4. Methodology Cyprus Pedagogical 4.1. Data Collection The Data collection process will follow a mixed methods approach. Data will be collected using both qualitative and quantitative procedures. Data will be collected via observations, semi-structured group interviews, questionnaires and through copies of the groups Learning Designs (artifacts). Statistics will also derive from Moodle and Cloudworks in order to have a descriptive account of participants logins and contributions to the online communities. Additionally, teacher-trainers will be asked to complete a reflective diary of each meeting right after it is held, in order to get their view on each meeting and to compare it with the researchers observation reports. Furthermore, the artifacts from groups Learning Designs will also be collected and analysed, both during and after the process. Lastly, reflective diaries will be kept right after each classroom implementation. Apart from ongoing observation, data will also be collected through semi-structured group interviews conducted right after the training groups finish their designs and classroom implementations. The questions included in the semi-structured group interviews will be designed according to the project s research questions. In addition, through the interviews the researchers will have the opportunity to clarify misconceptions and answer further questions regarding the LD methodology. The group interviews will be audio-taped and then transcribed in detail for further analysis. 4.2. Data Analysis Data will be analyzed in two ways: quantitative analysis will provide statistics on usage patterns (e.g. number of downloads of instructional material), participation rates, and learner satisfaction with the training, while qualitative analysis will be conducted through the constant comparative method to discover common themes in regards to the success and limitations of the training modules. 6

5. References Cyprus Pedagogical Leonardo EUPT3 Project (2010). Deliverable 2 - Report on State of the Art on Preparing teachers and trainers to teach with ICT. Leonardo EUPT3 Project (2010). Deliverable 3 - Report on the type of, and process for, the transfer of innovation from other projects. Leonardo EUPT3 Project (2010). Deliverable 4 - Teacher training and lifelong learning framework. Leonardo EUPT3 Project (2010). Deliverable 5 - Curriculum for training teachers and trainers. 7

6. Appendices Cyprus Pedagogical Appendix 1: Questionnaire Appendix 2: Interview protocol Appendix 3: Checklist Appendix 4: Rubric 8

Cyprus Pedagogical Appendix 1: Questionnaire Ερωτηματολόγιο Learning Design Παιδαγωγικό Ινςτιτοφτο Μζροσ Α Δημογραφικά ςτοιχεία a. Επιλζξτε Βακμίδα Εκπαίδευςθσ: Πρωτοβάκμια / Δευτεροβάκμια b. Επιλζξτε Σπουδζσ: Πτυχίο / Μεταπτυχιακό / Διδακτορικό c. Χρόνια Υπθρεςίασ: d. Είςτε Πυρινασ; Ναι / Όχι e. Πόςο ςυχνά κα λζγατε ότι χρθςιμοποιείτε τισ ΤΠΕ: Α) Για προςωπικι χριςθ Ποτζ / 1 φορά τθ βδομάδα / 2-3 φορζσ τθ βδομάδα / κάκε μζρα Β) Για επαγγελματικι χριςθ Ποτζ / 1 φορά τθ βδομάδα / 2-3 φορζσ τθ βδομάδα / κάκε μζρα e. Νιϊκετε ικανόσ/ι να χρθςιμοποιείτε τισ ΤΠΕ για: Α) Για προςωπικοφσ λόγουσ; Όχι / κάπωσ ικανόσ/ι / Ικανόσ/ι / Πολφ ικανόσ/ι Β) Για επαγγελματικοφσ λόγουσ; Όχι / κάπωσ ικανόσ/ι / Ικανόσ/ι / Πολφ ικανόσ/ι g. Ζχετε παρακολουκιςει ςεμινάρια για τισ ΤΠΕ; Αν ναι, ποια ιταν αυτά; Μζροσ Β Επιμόρφωςη Learning Design α) Δομή και Περιεχόμενο Εργαςτηρίου Learning Design Παρακαλϊ δθλϊςτε κατά πόςο ςυμφωνείτε με τισ πιο κάτω δθλϊςεισ. 1 = Διαφωνϊ απόλυτα 3=Οφτε ςυμφωνϊ οφτε διαφωνϊ 4=Συμφωνϊ κάπωσ 2=Διαφωνϊ κάπωσ 5=Συμφωνϊ απόλυτα 1. Οι ςτόχοι του εργαςτθρίου ιταν ξεκάκαροι για εςάσ 2. Οι δραςτθριότθτεσ του εργαςτθρίου ιταν καλά οργανωμζνεσ 3. Οι δραςτθριότθτεσ του εργαςτθρίου επζτρεπαν διαδραςτικι ςυμμετοχι 4. Οι δραςτθριότθτεσ του εργαςτθρίου ενίςχυςαν τισ γνϊςεισ ςασ ςχετικά με τθν ενςωμάτωςθ τθσ τεχνολογίασ ςτο μάκθμα 5. Οι δραςτθριότθτεσ του εργαςτθρίου ενίςχυςαν τισ γνϊςεισ ςασ ςχετικά με το μακθςιακό ςχεδιαςμό 6. Οι δραςτθριότθτεσ του εργαςτθρίου ιταν εφκολεσ να γίνουν 9

4 Web2.0 εργαλεία 10 τρόποι καταςτροφισ μακιματοσ Macro View Course Map Μeso View Learning Outcomes Micro View Activity View Cyprus Pedagogical 7. Υπιρχε αρκετόσ χρόνοσ για τθν εκπλιρωςθ των δραςτθριοτιτων 8. Οι δραςτθριότθτεσ ιταν κατάλλθλεσ για το κζμα του εργαςτθρίου 9. Το περιεχόμενο του εργαςτθρίου ιταν ςχετικό με τθν πρακτικι ςασ 10. Το εργαςτιριο ανταποκρίκθκε ςτισ προςδοκίεσ ςασ 11. Ζχετε κατανοιςει τουσ ςτόχουσ του εργαςτθρίου β) ΔΡΑΣΗΡΙΟΣΗΣΕ ΕΡΓΑΣΗΡΙΟΤ Παρακαλϊ αξιολογιςτε τισ δραςτθριότθτεσ του εργαςτθρίου Learning Design 1 = Διαφωνϊ απόλυτα 3=Οφτε ςυμφωνϊ οφτε διαφωνϊ 4=Συμφωνϊ κάπωσ 2=Διαφωνϊ κάπωσ 5=Συμφωνϊ απόλυτα Δραςτθριότθτα Σε ποιο βακμό πιςτεφετε ότι: 12. ζχετε κατανοιςει τουσ ςτόχουσ τθσ δραςτθριότθτασ 13. είναι χριςιμθ δραςτθριότθτα 14. είναι ενδιαφζρουςα δραςτθριότθτα 15. είναι εφκολθ δραςτθριότθτα 16. νιϊςατε ικανοποιθμζνοσ/θ από τθ δραςτθριότθτα 17. Είχατε καλι υποςτιριξθ κατά τθ διάρκεια τθσ δραςτθριότθτασ 10

Cyprus Pedagogical γ) ΔΙΔΑΚΟΝΣΕ (Παιδαγωγικό Ινςτιτοφτο / Πυρήνεσ) Παρακαλϊ δθλϊςτε κατά πόςο ςυμφωνείτε με τισ πιο κάτω δθλϊςεισ ςχετικά με τουσ διδάςκοντεσ 1 = Διαφωνϊ απόλυτα 3=Οφτε ςυμφωνϊ οφτε διαφωνϊ 4=Συμφωνϊ κάπωσ 2=Διαφωνϊ κάπωσ 5=Συμφωνϊ απόλυτα 18. είχαν επαρκείσ γνϊςεισ για το ςυγκεκριμζνο κζμα 19. είχαν καλι επικοινωνία μαηί ςασ 20. ιταν καλά προετοιμαςμζνοι 21. αφιζρωναν τον απαραίτθτο χρόνο για επιπλζον επεξθγιςεισ που ηθτοφςατε 22. ςασ ζδωςαν τθν κακοδιγθςθ που χρειαηόςαςταν κατά τθ διαδικαςία του μακθςιακοφ ςασ ςχεδιαςμοφ δ) ΙΚΑΝΟΠΟΙΗΗ Παρακαλϊ δθλϊςτε κατά πόςο ςυμφωνείτε με τισ πιο κάτω δθλϊςεισ. 1 = Διαφωνϊ απόλυτα 3=Οφτε ςυμφωνϊ οφτε διαφωνϊ 4=Συμφωνϊ κάπωσ 2=Διαφωνϊ κάπωσ 5=Συμφωνϊ απόλυτα Νιϊκω ικανοποιθμζνοσ/θ από: 23. τθ γενικι οργάνωςθ του προγράμματοσ 24. το περιεχόμενο του προγράμματοσ 25. τισ δραςτθριότθτεσ που ζγιναν 26. το υλικό που δόκθκε από το Παιδαγωγικό Ινςτιτοφτο 27. το θλεκτρονικό υλικό που υπιρχε ςτο moodle του Παιδαγωγικοφ Ινςτιτοφτου 28. τισ μορφζσ επικοινωνίασ (προςωπικι, τθλεφωνικι και θλεκτρονικι) 11

Visualized Learning Design Cyprus Pedagogical Μζροσ Γ Μεθοδολογία Η μεκοδολογία που ειςθγείται το Ανοικτό Πανεπιςτιμιο του Η.Β. ςτθρίηεται πάνω ςτθν οπτικοποίθςθ του Μακθςιακοφ Σχεδιαςμοφ (Visualized Learning Design). Ζχοντασ ζρκει ςε επαφι με τθ μεκοδολογία αυτι, ςε ποιο βακμό ςυμφωνείτε ότι θ μεκοδολογία αυτι: 1 = Διαφωνϊ απόλυτα 3=Οφτε ςυμφωνϊ οφτε διαφωνϊ 4=Συμφωνϊ κάπωσ 2=Διαφωνϊ κάπωσ 5=Συμφωνϊ απόλυτα 29. διευκόλυνε τθ διαδικαςία του ςχεδιαςμοφ ςασ 30. βοικθςε ϊςτε να οργανϊςετε τθ ςκζψθ ςασ καλφτερα 31. ζκανε τον τρόπο που ςχεδιάηετε τα μακιματά μου πιο εμφανι ςε εςάσ 32. ζκανε τον τρόπο που ςχεδιάηετε τα μακιματά ςασ πιο εμφανι ςτουσ ςυναδζλφουσ ςασ 33. διευκόλυνε τθ ςυνεργαςία ςασ με τουσ ςυναδζλφουσ ςασ 34. ςασ βοικθςε να εκφράςετε πιο εφκολα τισ απόψεισ ςασ κακϊσ ςχεδιάηατε με τουσ ςυναδζλφουσ ςασ 35. ςασ βοικθςε να περιλάβετε ςτο μάκθμά ςασ παράγοντεσ που ςυνικωσ δεν ςυνυπολογίηατε όταν ςχεδιάηατε τα μακιματά ςασ 36. ςασ βοικθςε να οργανϊςετε καλφτερα το μάκθμά ςασ 37. ςασ βοικθςε να κατανοιςετε καλφτερα τον τρόπο που ςχεδιάηετε το μάκθμά ςασ 38. ςασ βοικθςε να προβλζψετε κινδφνουσ που ίςωσ επθρζαηαν το μάκθμά ςασ 39. ωφζλθςε γενικά τθν πρακτικι ςασ 40. ςασ βοικθςε να ςχεδιάςετε ζνα πιο αποτελεςματικό μάκθμα για τουσ μακθτζσ ςασ Μζροσ Δ Γενική Ικανοποίηςη και Επίδραςη Παρακαλϊ δθλϊςτε κατά πόςο ςυμφωνείτε με τισ πιο κάτω δθλϊςεισ. 1 = Διαφωνϊ απόλυτα 3=Οφτε ςυμφωνϊ οφτε διαφωνϊ 4=Συμφωνϊ κάπωσ 2=Διαφωνϊ κάπωσ 5=Συμφωνϊ απόλυτα 41. Νιϊκω ευχαριςτθμζνοσ/θ από αυτά που πιρα από το εργαςτιριο Learning Design 42. Είχα ικανοποιθτικι υποςτιριξθ κατά τθ διάρκεια τθσ εφαρμογισ τθσ μεκοδολογίασ του Learning Design 43. Στο μζλλον, κα ξαναχρθςιμοποιιςω τθ μεκοδολογία αυτι για το ςχεδιαςμό των μακθμάτων μου 44. Νιϊκω ότι θ μεκοδολογία αυτι είναι χριςιμθ ςτο επάγγελμά μου 45. Η μεκοδολογία αυτι είναι εφκολθ ςτθν εφαρμογι τθσ 12

Cyprus Pedagogical 46. Η μεκοδολογία αυτι δεν είναι χρονοβόρα ςτθν εφαρμογι τθσ 47. Θα ιταν καλό να μάκουν κι άλλοι ςυνάδελφοι τθ μεκοδολογία αυτι 48. Τι ςασ άρεςε περιςςότερο από τθ μεκοδολογία αυτι; 49. Τι δεν ςασ άρεςε από τθ μεκοδολογία αυτι; 50. Άλλα ςχόλια Ευχαριςτοφμε για τθ ςυμμετοχι ςασ! 13

Part Α Demographics Cyprus Pedagogical Learning Design Questionnaire Cyprus Pedagogical a. Teaching: Primary / Secondary b. Studies: Bachelor Degree / Master / Research Degree / c. Years Teaching: d. Are you a Teacher/Coach; Yes / No e. How often would you say you use ICT for: Α) Personal reasons Never / once a week / 2-3 times per week / every day Β) Professional reasons Never / once a week / 2-3 times per week / every day f. Do you feel confident using ICT for: Α) Personal purposes No / Somewhat confident / Confident / Very confident Β) Professional purposes No / Somewhat confident / Confident / Very confident g. Have you attended any seminars regarding ICT? If yes, which were they? Part Β Learning Design Training a) Content and structure of Learning Design workshops Q To what extent do you agree with the following statements? Please use the following scale: 1 = Completely disagree 3=Neither agree nor disagree 4= Partially agree 2=Partially disagree 5=Completely agree 1. The goals of the workshop were clear to you 2. Workshop activities were well organized 3. Workshop activities allowed interactive participation 4. 5. Workshop activities enhanced your knowledge regarding integration of ICT in your lesson Workshop activities enhanced your knowledge regarding learning design 6. Workshop activities were easy to follow 7. There was enough time to complete the workshop activities 14

4 Web2.0 tools 10 ways to ruin a lesson with ICT Macro View Course Map Μeso View Learning Outcomes Micro View Activity View Cyprus Pedagogical 8. Workshop activities were suitable for the subject of the workshop 9. Workshop content was relevant to your practice 10. The workshop matched your expectations 11. You have understood the goals of the workshop b) WORKSHOP ACTIVITIES To what extent do you agree with the following statements regarding the activities of the workshop? Please use the following scale: 1 = Completely disagree 3=Neither agree nor disagree 4= Partially agree 2=Partially disagree 5=Completely agree ACTIVITY 12. you have understood activity s goal 13. it is a useful activity 14. it is an interesting activity 15. it is an easy-to-complete activity 16. you were satisfied by the activity 17. you were supported during the activity c) COACHES (CPI / Teacher-Coach) To what extent do you agree with the following statements regarding your coaches? Please use the following scale: 1 = Completely disagree 3=Neither agree nor disagree 4= Partially agree 2=Partially disagree 5=Completely agree 18. they had adequate knowledge on the subject 15

Cyprus Pedagogical 19. they communicated well with you 20. they were well prepared 21. they devoted enough time to answer to your questions and issues 22. they guided you well, throughout your learning design d) SATISFACTION To what extent do you agree with the following statements regarding your satisfaction? Please use the following scale: 1 = Completely disagree 3=Neither agree nor disagree 4= Partially agree 2=Partially disagree 5=Completely agree I feel satisfied by 23. the overall organization of the program 24. the content of the program 25. the workshop activities 26. the material given by Cyprus Pedagogical 27. the e-material of Learning Design that I found in CPI s Moodle 28. the communication (face-to-face, telephone and online) 16

Design Approach Cyprus Pedagogical Part C Visualized Learning The Learning Design approach suggested by OU is based on visualization of Learning Design. Using the following scale, to what extent do you agree that this approach has: 1 = Completely disagree 3=Neither agree nor disagree 4= Partially agree 2=Partially disagree 5=Completely agree 29. facilitated your learning design process? 30. helped you to better organize your thoughts? 31. made the way you design your lessons more visible to you? 32. made the way you design your lessons more visible to your colleagues? 33. facilitated collaboration with your colleagues? 34. helped you to share your thoughts with your colleagues in an easy way? 35. helped you to include factors in your lessons, that you would not otherwise have counted if you designed a lesson in your usual way? 36. helped you organize your lesson better? 37. helped you to better understand the way you design your lesson? 38. helped you predict risks that might have affected your lesson? 39. benefitted your practice? 40. helped you design a more effective lesson for your students? Part D Overall satisfaction and Impact To what extent do you agree with the following statements? Please use the following scale: 1 = Completely disagree 3=Neither agree nor disagree 4= Partially agree 2=Partially disagree 5=Completely agree 41. I am pleased I attended the Learning Design workshop 42. I was provided with satisfactory support while applying the Learning Design approach 43. I will use this Learning Design approach again in the future 44. I feel that this approach is useful for my practice 45. This approach is easy to apply 17

Cyprus Pedagogical 46. The application of this approach is not time-consuming 47. It would be good if other colleagues learnt more about this approach 48. What did you enjoy most from this approach? 49. What did you enjoy least from this approach? 50. Other comments Thank you for your participation! 18

Cyprus Pedagogical Appendix 2: Interview protocol Interview to Trainees Introductory Questions 1. Describe your participation in the project. How were you selected? 2. Which other ICT related trainings did you participate in? 3. What were the workshops like? The LD Approach 4. What do you think of the LD approach? 5. What are some of the strengths of the LD approach? 6. What are some of the weaknesses of the LD approach? 7. Do you think that visualization of LD helps? How? 8. In what ways is the LD approach helping? Cloudworks & Compendium 9. What do you think of Cloudworks? 10. What do you think of the CompendiumLD notation? 11. How useful was the information you found in Cloudworks? The Workshop 12. Which activity did you find most interesting? Why? 13. Which activity did you find less interesting? Why? 14. What difficulties did you face during the workshop? How did you deal with them? Collaboration with Colleagues 15. How did your team work to complete each activity? 16. How good was the collaboration with your team? Describe. (1-10 & description) 17. How did you share your designs during the workshop? 18. (If applicable) How did you feel about sharing your designs with the classroom? 19. How could the collaboration amongst the team be improved? Trainer Collaboration 20. How was the collaboration with the trainer? Describe. 21. Did you have sufficient support by the trainer during the workshop? 22. How could the collaboration with the trainer be improved? Overall Workshop Evaluation 23. What worked well in the workshops? 24. How can the workshops and trainings be improved? 25. To what level were your expectations out of this workshop met? (1-10) 26. Do you think that participating in this workshop will help you achieve better learning outcomes in your classroom? Explain. After the trainings 27. [Did you / Will you] use the LD approach after the training? Describe. 28. [Did you / Will you] create any additional designs based on the LD approach after the trainings? 29. [Did you / Will you] share any additional designs with other colleagues? How? 30. [Did you / Will you] visit Cloudworks again? 31. [Did you / Will you] participate in any Cloudworks discussions after the trainings? 32. [Did you / Will you] participate in the Cloudworks community again? Why? 19

Cyprus Pedagogical Appendix 3: Participants Usage pattern checklist The following Cloudworks statistics are available either through the site itself or through Google Analytics. These statistics will be monitored throughout the project in order to extract important information on the usage patterns of workshop participants and perform improvements when needed. 1. Available from site Cloudscape statistics (no specific dates) Content o Number of clouds o Number of comments o Number of embeds o Number of items of extra content o Number of links People o Number of followers o Number of attendees o Number of distinct people commenting Views o Number of views of cloudscape o Number of distinct people logged in and viewing cloudscape clouds o Number of distinct guests (i.e. distinct IP addresses) viewing cloudscape clouds Cloudscape statistics (between specific dates) o Number of distinct people logged in and viewing cloudscape clouds o Number of distinct guests (i.e. distinct IP addresses) viewing cloudscape clouds 2. Available from Google Analytics Using Content overview functionality against a specific URL (i.e. Cloudscape page OR Cloud page) o Pageviews o Unique pageviews o Average time on page o Traffic source (i.e. the route taken to access the page) o Geographic (language, continent, sub-continent, country, region, city) Using Content by title functionality against a keyword (i.e. Design-practice ) o Pageviews o Unique pageviews o Average time on page o Traffic source (i.e. the route taken to access the page) o Geographic (language, continent, sub-continent, country, region, city) 20

Cyprus Pedagogical Appendix 4: Expert Review Rubric Rubric for the Evaluation of Learning Design (process and product) Grade the items below on a scale from 1-5. To what degree does the process / product demonstrate the following? (Use the questions below each criterion to help your evaluation) Criteria Excellent Very Good Average Low Very low 5 4 3 2 1 PROCESS (The Learning Design Process) Better sharing and discussion Do participants discuss amongst themselves and the instructor? Do participants share ideas and designs with their colleagues? Does sharing and discussion improve during the process? Using theory more explicitly (learning theory and learning design theory) Does the instructor describe how theory is to be used in the process? Is it clear which part of the learning theory is used in each step of the process? Is it clear which part of the LD theory is used in each step of the process? Encouraging reflections for improvement Do the participants look back and discuss their work and teaching experiences in order to learn from them? Are participants encouraged to consider ideas on how to improve their practice? Do participants reflect on how to improve their practice using Learning Design? Critical reflection During the process, do the participants describe experiences and make observations? Do they reconsider and question experiences and methods they used? Do they ask questions to each other and elaborate on their decisions? 21

The value of the approach Does the process clearly demonstrate the value of the LD methodology? Are the benefits of following this approach made clear to the participants? Do the participants comprehend and embrace the value of the approach? Change of practice Does the process encourage change of practice? Do the participants find the LD process useful in their practice from now on? Do the participants describe ways they intend to integrate LD in their teaching? Engaging the teacher designer Do the activities provide opportunities for the teacher to be creative? Do participants get involved in the design process actively? PRODUCT (Final Lesson Plan) Evidence of good pedagogy Does the lesson plan include a specific methodology? Are the learning objectives clearly defined? Is there a logical sequence of activities? Is the timing of the activities appropriate? Are there specific learning outcomes? Interaction Cyprus Pedagogical Does the lesson plan include activities that involve interaction amongst students? How about activities that involve interaction between students and teacher? How about activities that involve any other form of interaction (including the use of ICT tools, Web 2.0 technologies etc.)? Critical reflection Does the lesson plan include opportunities for students to review, question and assess their work in any form? Do any of the activities involve analysis, critique, comparing and contrasting? Authentic Learning Are the activities designed to include practical application of knowledge? Do any of the activities involve real-life scenarios and problems? 22

Cyprus Pedagogical Do any of the activities occur in environments outside the classroom? (E.g. labs, field work, video, audio, simulations etc.) Conversation Do the activities in the lesson plan involve discussion amongst teacher & students? Do they involve discussion amongst students? Experience Do the activities in the lesson plan involve practice and application of knowledge for the students? Do they involve any form of exploration / investigation on behalf of the students? Does the lesson plan use any experiential techniques? (E.g. Case studies, experiments, field trips, games, role-playing, simulations) Activity Are the activities in the lesson plan designed to encourage active or passive learning? Do the activities involve multiple learning environments (Computer-based, Lab-based, Field-based, Audio-based, Video, Lecture-based, Work-based, Seminar-based)? Collaboration Do the activities in the lesson plan involve team work? If yes, are the activities designed so that all group members will participate actively? (e.g. specific roles in each team, moderation and facilitation, presentation of the work in the class) Representing knowledge in multiple ways Are there multiple types of activities in the lesson plan? Does the lesson plan involve the use of a variety of teaching material (e.g. books, maps, charts, audio, video, internet etc.)? Application Are the students expected to produce tangible outputs during the lesson? Do the expected outputs demonstrate whether the learning outcomes have been achieved? Added value of ICT Does the use of ICT add any value to the lesson plan? Are there any disadvantages because of the use of ICT? Would the lesson be affected if the parts involving ICT were skipped? 23

Alignment of outcomes, activities,tools and assessment Is there a clear relationship between activities and learning outcomes? Cyprus Pedagogical Is the lesson designed so that there is use of the appropriate tools for each activity? Acknowledges context Does the lesson plan acknowledge the context in which learning will occur? (E.g. aims, pre-requisites, subject, environment, time, difficulty, required skills) Affordances of tools mapped to activities Are the tools used in the activities related to the context of each activity? Do the tools bring any value in each activity? Evaluator: Date: 24