EUROPEAN QUALITY OBSERVATORY DATA COLLECTION QUESTIONNAIRES AND GREEK CASE STUDΙΕS VERSION: 0.1 (DRAFT VERSION) LAST EDITOR: ITI -CERTH TEAM DATE: 2004-03-04.
A.1 Version Version Date / Contributor Summary of Changes 0.1 2004-03-01 ITI-CERTH Initial Version Team
Contents A.1 Version...ii 1 Introduction... 4 1.1 Purpose and Scope... 4 1.2 Overview... 4 1.3 Definitions, Abbreviations, Acronyms... 4 2 Questionnaire Design... 5 2.1 Questionnaires for data collection... 5 2.1.1 Aim- Description... 5 2.2 Description of the Questionnaires... 5 2.2.1 General... 5 2.2.2 Questionnaire A: Description of a Quality Approach... 6 2.2.3 Questionnaire B: Description of an Experience from a Quality Approach. 19 3 Greek Pilot Case Studies... 31 3.1 Go-Online Quality Approach... 31 3.1.1 Short description of the pilot case study... 31 3.2 Doukas School Quality Approach... 35 3.2.1 Short description of the case study... 35 4 Conclusions, Proposals and Future Directions... 40 ANNEX: Questionnaires in Greek... 41 3
1 Introduction 1.1 Purpose and Scope The purpose of this document is to present the instruments developed by ITI-CERTH in order to facilitate data collection about the quality approaches that are developed or / and used around Europe in the field of e- learning. The proposed instruments (questionnaires) have been developed according to the structure of the EQO Conceptual Model in order to support directly mapping of collected information into the elements of the EQO Model and consequently in the EQO repository. To demonstrate the application of the questionnaires for QAs and QA Experiences data collection, we present how we applied it for a set of Greek Case Studies. This version of the document contains two case studies; the Quality Control and Assurance Procedures of the Greek Go-Online initiative and the QA that a primary and secondary educational organisation (Doukas Schools) applies during the selection of e-learning software. 1.2 Overview This document consists on the following parts : In Section 1 a presentation of the purpose and scope of this document is made. In Section 2 a presentation of the design and a description of the questionnaires is made. More specifically, a description of the aim and strusture and secondly the actual presentation of the questionnaires that can support data collection for EQO portal is made. The application of the questionnaires for a Greek case study is presented in Section 3. After a short description of the QA the results of the application of the two quesionnaires on it are presented in two tables. Finally, Section 4 presents some conclusions and proposals for future work. Annex 1 presents the questionnaires in their Greek version, as they have been used for the Greek case studies. 1.3 Definitions, Abbreviations, Acronyms QA : Quality Approach EQO : European Quality Observatory ITI-CERTH : Informatics & Telematics Institute Centre for Research and Technology Hellas 4
2 Questionnaire Design 2.1 Questionnaires for data collection 2.1.1 Aim- Description Firstly, purpose of this section is the definition of the aim and structure and secondly the presentation of the questionnaires that can support the collection of data about the quality approaches that are developed or/ and used around Europe in the field of e-learning. Purpose of use of these questionnaires is the facilitation of the data s collection from individuals or organisations that are not familiarised with the EQO Conceptual Model. 2.2 Description of the Questionnaires 2.2.1 General Collection of data about the development and use of formal quality approaches or internal quality assurance processes in the field of e-learning can be facilitated with the use of specially designed questionnaires, the structure of which should be based on the structure of the EQO Model. These questionnaires will support actors who are not familiar with the EQO Conceptual Model to provide information about quality approaches in a structured form, that can be directly stored in the EQO Repository. After studying the four sections of the EQO Conceptual Model (the first three of which are theoretical and deal with the development of a quality approach, whereas the fourth is based on the assessment of the experience from its application) it seemed more appropriate to develop two separate questionnaires. Each of the two questionnaires aims at the collection of different kind of data and is are addressed to different types of organisations and individuals. The aim of the first questionnaire is to collect data about the description of quality approaches that have been developed around Europe and is addressed to individuals or organisations that have developed or contributed to the development of an approach. The second questionnaire, which aims at the collection of experimental data from the application of a formal quality approach or internal quality control and assurance processes in the field of e-learning, is addressed to organisations that have applied some kind of quality processes to the design and development of the e-learning services that they offer. That means that the second questionnaire is addressed to educational organisations and / or their quality experts. The first questionnaire reflects a mapping of the three first categories of the EQO Conceptual Model, whereas the second reflects a mapping of the fourth category of elements. The collection of data by means of the two questionnaires is rather flexible and it can be achieved in several ways. At first, there is the possibility of them to be filled in by the stakeholders in hard copy. Secondly data can be collected by means of an interview (either personal or telephone) with the relative stakeholders. Both methods require afterwards a manual transformation of the collected results in the format required by the EQO Conceptual Model, so that they can be stored in the repository. They are useful though for verifying the elements of the Conceptual Model and exploring their limitations. These methods are therefore due to be applied during the first phases of data collection, where a set of initial quality approaches will be used. Finally, after the questionnaires have been tested and the elements of the Conceptual Model verified and finalised, the web - based version of the two questionnaires 5
can be implemented. In this way, appropriately formatted data will be directly stored in the EQO repository. 2.2.2 Questionnaire A: Description of a Quality Approach 2.2.2.1. Aim description The specific questionnaire (Questionnaire A), aims to facilitate the collection of data that describe quality approaches that have been developed around Europe in the field of e-learning, either from individuals or from organisations. 2.2.2.2 Structure description At this point the structure of the first questionnaire will be described, while the figure A.1 shows an outline of it. As it can be seen, the questionnaire has four sections of questions (which are based on the EQO Model): Section 0: The first category of questions is introductory and aims at the collection of information about the person who completes the questionnaire and the organisation which he is affiliated with (full name, telephone, name and type of organisation). Section 1: The second category of questions aims at the collection of general information about the quality approach (title, edition, location, cost of use). Section 2: The third category aims at the collection of data relative to the context of the quality approach (specific topics, cultural or geographical specialties etc ). Section 3: The fourth category of questions deals with questions regarding the goals, philosophy and methods of the quality approach (goals, description of the approach s aims, methods). 2.2.2.3 Description of the questions per section Here we will shortly present the questions that are included in each of the four sections of the questionnaire. Section 0: Information about person completing the questionnaire This questionnaire after some general guidelines asks from the interviewee to fill in his personal elements and the elements of the organisation that he is affiliated with. More specifically, the contributing person is asked to fill in the date that the questionnaire is completed, his full name and some elements useful for the achievement of contact with him in case that the collection of data is not made by means of an interview, such as his telephone number and his e-mail address. Additionally, the completion of elements that have to do with the organisation that has developed the quality approach is asked, such as its name, type, size and its web site. As a last question, the person completing the questionnaire is asked to specify his role in the organisation that he is affiliated with. 6
SECTION 0 INFORMATION ABOUT THE PERSON COMPLETING THE QUESTIONNAIRE Full name, (e-mail) Address, Name of organization, Organization type SECTION 1 GENERAL INFORMATION ABOUT THE QUALITY APPROACH Title, Edition, Description,Cost of use SECTION 2 CONTEXT FOR WHICH THE QUALITY APPROACH HAS BEEN DEVELOPED Educational sector, Target group, Cultural or Geographical elements and specialties, Specific topics SECTION 3 GOALS, PHILOSOPHY AND METHOD OF QUALITY APPROACH APPLICATION Goals, Focus, Methods that the approach uses Figure 2.1: Outline of the structure of the first questionnaire about the collection of data regarding the description of quality approaches. Section 1: General information about the quality approach In Section 1 of the questionnaire an attempt is made to collect general information about the quality approach, such as its title, the edition being described and the date of publication of the specific version. At this point a question has been added that is not included in the EQO Model asking whether the quality approach has been applied in a specific context and if the answer is positive, whether there are any experimental results that will be taken into consideration in future improved editions. Afterwards, a short description of the quality approach is asked to be provided, so that people can understand if it s appropriate for the specific target groups that exist in their organisation. Also the location (e.g. internet address), where the quality approach can be obtained from is asked to be specified. 7
Finally, the interviewee is asked to complete elements that have to do with the rights of the quality approach and the conditions of its usage. More specifically he is asked to answer whether there is a cost for somebody to obtain / use the quality approach and if the answer is yes, he is asked to specify it. Additionally, the interviewee is asked to shortly describe the restrictions of use for the specific approach (if there are any). Section 2: Context for which the quality approach has been developed Section 2 of the questionnaire is aiming at the collection of data related to the context for which the quality approach has been developed as well as at the collection of data about the cultural elements and specialties that have been taken into consideration when the specific approach was developed, if there were any. So the interviewee is asked to answer whether the quality approach is related with specific educational sectors and if yes he is asked to specify them, as well as whether the educational institutions that will apply the specific approach are related to a specific industry sector (such as art, education, services). Following that the target group in which the approach can be used is asked to be specified. After that the contributing person is asked to specify whether the approach was developed for application in a specific country/ ies and if the answer is positive he is asked to identify it / them. Afterwards, the interviewee is asked to give by means of a short description a picture of the cultural or geographical elements and specialties that have been taken into consideration when the approach was developed (if there were any). The interviewee is also asked to specify the topics for which the approach has been developed, e.g. mathematics, history, computers etc Finally, the interviewee is asked to mention the educational processes with which the approach is related as well as to provide a short description of the way of their relation. Section 3: Goals, philosophy and method of quality approach application In Section 3 an attempt is made to record the goals of the quality approach in a measurable way. That s why the interviewee is asked to specify the importance of each of the nine below mentioned aspects of e-learning quality: 1. Quality of institutional and executive organisation 2. Quality of technological infrastructure 3. Quality of student services 4. Quality of instructional design and course development 5. Quality of instruction and instructors 6. Quality of conditions and processed for the e-learning program delivery 7. Quality of financial health / management 8. Quality of regulatory and legal compliance 9. Quality of evaluation and assessment procedures 8
After that the interviewee is asked to provide a short description regarding the way the quality approach covers the above goals. Consecutively, he is asked to specify the focus / orientation of the quality approach, that means whether the approach is product, process or competency oriented; and if none of these, he is asked to specify its focus. The next question regards the methods that the specific approach uses. The interviewee is asked to check as many from a list containing the values: benchmarking, evaluation, management approaches, quality criteria, standards, specification, accreditation process, policy and / or mention any methods that are not included in the above list. Afterwards the interviewee is asked to provide an assessment of the expected contribution of the specific approach in each of the nine quality aspects identified before by means of a five - graded scale with values ranging from no contribution to great contribution. At the end of the questionnaire some questions are included that aim at the collection of data about the status of e-learning quality in Europe / in the country of the person interviewed. The interviewee is asked to give his estimation about the status of e- learning quality in his country and the extent that formal quality approaches are applied to the e-learning services offered in the specific geographical area. The person completing the questionnaire is also asked to provide an estimation for the next 5-10 years. He / she is also asked to describe the extent to which the quality approach described has been applied in the specific country and whether it has also been applied outside the specific country. Finally, he / she is asked whether receipt of information about e- learning quality and the EQO would be interesting for the organisation he / she is affiliated with. 9
EUROPEAN QUALITY OBSERVATORY. QUESTIONNAIRE #1 Description of a Quality Approach The aim of this questionnaire is to collect information about an e-learning quality approach. Completing it will greatly help us in the collection of data about the quality approaches developed around Europe, which are applied in the field of e-learning. The estimated time you will need for the questionnaire completion is about 15-20 minutes. After its completion, please send the questionnaire either in electronic form in the address nikosm@iti.gr, or by fax in 210-6896082 (Mr Manouselis). Thank you in advance for your help! 10
General Guidelines There are two types of questions, closed and open. In closed questions please check (with X) the choice or choices appropriate. In open questions please provide a short text with your answer (usually not more than 2 or 3 sentences are required). Section 0. INFORMATION ABOUT PERSON COMPLETING THE QUESTIONNAIRE 0.1 Date that this questionnaire is completed e.g.. 2003-07-16 0.2 Full name of the person completing the questionnaire e.g. Nikos Manouselis 0.3 Telephone number & Fax number of the person completing the questionnaire e.g. +30 210 6839916 & +30 210 6839917 0.4 E-mail address of person completing the questionnaire e.g. nikosm@iti.gr 0.5 Name of organisation/ institution that the person is affiliated with e.g. Informatics and Telematics Institute Center for Research and Technology Hellas 0.6 Short description of organisation/ institution type e.g. Research and Development Institute 0.7 Size of the organisation/ institution (by approximate number of employees) 0-10 11-100 101-500 > 500 0.8 Address of the organisation/institution e.g. 42 Arkadias & Taygetou St., Halandri, Athens, Greece, GR-15234 0.9 Web site of organisation/ institution e.g. http://www.ask.iti.gr 0.10 Please describe your role in the organisation/institution (choose one on the following): CEO/ Manager/ Official Quality manager Consultant/ Evaluator Learner 11
Teacher Developers (Software developer, Content broker, Authors etc.) Policy Maker Level Other (please specify)... Section 1. GENERAL INFORMATION ABOUT THE QUALITY APPROACH 1.1 Please provide the official title of the quality approach (& its acronym if existing) e.g. EFQM Excellence Model 1.2 Please provide the version / edition that you describe e.g. Version 1.0 1.3 Which is the date of publication of this particular version / edition? e.g. 2003-07-16 1.4 Has the quality approach been applied in a specific context? If yes, are you aware if there are any experimental/application results that will be taken into consideration for future editions? 1.5 Please provide a short description of the quality approach (3-4 sentences) appropriate for people who consider applying this particular QA in their organisation. e.g. This quality approach defines a set of quality criteria useful in evaluation e-learning services and programs of higher education institutes. 1.6 Where can this quality approach be obtained from? e.g. Linday & Evans, Total Quality Control and Management, or http://www.host.org 1.7 In which language/s is this quality approach available? 1.8 Is there a cost for obtaining / using this quality approach? Yes No 1.9 If yes, which is this cost? (in euros) 1.10 If there is such a cost, which is the charging scheme? Charge per use Subscription for specific time period 12
Please define the time period, i.e. 1 year:. Purchased once Other (please specify) 1.11 Does this quality approach have associated copyrights or restrictions of use? Yes No 1.12 If yes, can you please shortly describe (in 1-2 sentences) the restrictions of use for the specific QA? e.g. This quality approach can be freely used in higher education institutes, but it is not allowed to be used by profit organisations without obtaining written permission of the copyright holder. Section 2. CONTEXT FOR WHICH THE QUALITY APPROACH HAS BEEN DEVELOPED 2.1 Does this quality approach refer to one or more educational sectors, and to which? General / basic education Higher education Vocational Education Continuing Education 2.2 Does this quality approach refer to one or more industrial / production sectors, and to which? Agriculture Arts Industry Services 2.3 In which target group can this quality approach be applied? Author Counselor Learner Manager Parent Teacher 13
Developers (Software developer, Content broker, Authors etc.) Content providers (companies and institutions) Evaluators Policy makers Technology/ Infrastructure Providers Other(please specify)... 2.4 Are there any cultural or geographical elements and specialties that have been taken into consideration when this quality approach was developed? If yes which are they? (please describe in 1-2 sentences) e.g. in specific country, region or target group 2.5 Has this quality approach been developed for application in specific topics? If yes, please identify which. e.g. Technology, Finance, Law 2.6 To which educational process/ es can this quality approach be applied? Institutional Support Course Development Teaching/Learning Course Structure Student Support Faculty Support Evaluation and Assessment Other (please specify)... 2.7 Can you please provide a short description (2-3 sentences) on the way this quality approach is related with the specific educational process/ es? e.g. The guidelines from the University of Essen are engaged during the design and development of online courses. Section 3. GOALS, PHILOSOPHY, AND METHOD OF QUALITY APPROACH APPLICATION 3.1 Which of the following goals (aspects of e-learning quality) does the specific quality approach aims to satisfy, and to which degree each one of them? Please next to each goal identify how important is the specific aspect for the quality approach, from 1 to 100 e.g. 60 14
Goals Quality of institutional and executive organisation, thus it aims to ensure the commitment of the organisation to correspond to the requirements of the educational program. Quality of technological infrastructure, thus it aims to ensure the existence of the appropriate organisation s infrastructure for the provision of the educational program. Quality of student services, thus it aims to ensure that the services offered before, during, and after the completion of the educational program are of high standards. Quality of instructional design and course development, thus it aims to ensure that the design and development processes of the educational material are of high quality. Importance Quality of instruction and instructors, thus it aims to ensure that the quality level of instruction is high. Quality of conditions and processes for the e- learning program delivery, thus it aims to ensure that the conditions of access to the e-learning services are easy and transparent to the users. Quality of financial health / management, thus it aims to ensure the viability of the e-learning product/ service. Quality of regulatory and legal compliance, thus it aims to ensure that the e-learning product/ service follows the regulations and laws under which it operates. Quality of evaluation and assessment procedures, thus it aims to contribute in the quality improvement of the quality assessment processes that the e-learning product/ service contains. 3.2 Do you believe that there is some additional goal that is not covered by the list above? Please provide a short description of each additional goal (1-2 sentences). 3.3 Please provide any additional comments or remarks that you have regarding the way the quality approach covers the above goals. e.g The quality of institutional and executive organisation can be improved, but it is not possible to have measurable results of the degree. 3.4 What is the general focus/orientation of the quality approach? (select from the following options) Product-oriented 15
Process-oriented Competency-oriented Other (please specify)... 3.5 Please identify the methods this quality approach uses (check as many from the following list as necessary): Benchmarking Evaluation Management-approaches Quality Criteria Standards Certification Accreditation process Policy Other (please specify)... 3.6 Please provide an assessment of the expected contribution of the specific quality approach in each of the quality aspects identified before (mark one empty cell in each row with X): Quality of institutional and executive organisation Quality of technological infrastructure Quality of student services Quality of instructional design and course development Quality of instruction and instructors Quality of conditions and processes for the e-learning program delivery Quality of financial health / management Quality of regulatory and legal compliance Quality of evaluation and assessment procedures No Contribution Little Contribution Medium Contribution Good Contribution Great Contribution 16
Section 4. GENERAL OVERVIEW QUESTIONS 4.1 How would you define the status of e-learning quality in Greece? To which degree do the e- learning services, programs and products focus on quality? (please select one of the following choices) Maximum degree Large degree Medium degree Small degree Not at all 4.2 To which extent do you believe that formal quality approaches are applied in the processes of designing, developing and deploying e-learning services, programs and products in your country? (please select one of the following choices) Maximum extent Great extent Medium extent Small extent Not at all 4.3 And what is your prediction for the next 5-10 years? To which extent will they be applied? (please select one of the following choices) Maximum extent Great extent Medium extent Small extent Not at all 4.4 To which extent has the quality approach that you described been applied in your country? Is it also used outside your country? (please describe in 1-2 sentences) 4.5 Do you have something to comment regarding this questionnaire or to add to this description? 4.6 Are you interested in receiving more information about e-learning quality and the European Quality Observatory at your e-mail address? Yes No 17
Thank you very much for your time! 18
2.2.3 Questionnaire B: Description of an Experience from a Quality Approach 2.2.3.1 Aim description The second questionnaire is addressed to organisations that provide e-learning services or products around Europe. The specific questionnaire has as a target the collection of data regarding whether the above mentioned organisations apply internal quality control and assurance processes to the services that they provide, or whether they apply some formal quality approaches (such as EFQM, ISO etc). In case that none of the above happens, the recording of possible reasons for the nonapplication of quality processes as well as the possible interest for a future application by the organisations is interesting. The specific questions, as somebody could notice have nothing to do with the collection of data regarding quality processes, such as the rest of them, but they deal with another dimension, that of the supply of relative information to the organisations that will express interest for the quality assurance processes of the services that they offer. 2.2.3.2 Structure description At this point the questionnaire structure will be presented, which is pictured in the figure 2.2. The questionnaire has as introduction some questions that aim at the collection of information about the person completing the questionnaire (full name, e- mail address as well as date of questionnaire completion). After that information about the organisation is asked to be provided (name, type, size etc) After the general introductory questions one is following that determines the further completion of the questionnaire. The interviewee is asked to answer whether the organisation he is affiliated with, applies some quality control processes to the e- learning services that it offers. In case the answer is negative the interviewee is asked to answer a restricted number of questions that have as general aim the identification of reasons for which the organisation doesn t apply quality processes. In case the answer is positive the interviewee is asked to specify whether the organisation applies a formal quality approach or internal quality processes and describe it shortly. After that the questions have been expressed in such a way that they allow the collection of data for both cases, either the case that the organisation applies internal quality processes or the one that a formal quality approach is applied (e.g. ISO, EFQM etc). The questions aim at the collection of data about the application of quality processes and are divided in three sections. The first one aims at the collection of general information about the application of quality processes of e-learning services that are offered, the second aims at the collection of data regarding the context in which the specific processes are applied and the third the assessment of their application. In the end, general questions are included that aim at the collection of general statistical data and estimations of the interviewee regarding the status of e-learning services around Europe. 19
INFORMATION ABOUT PERSON COMPLETING THE QUESTIONNAIRE Full name, E-mail address, Date of questionnaire completion ORGANIZATION Name of organisation, Organization type, Size, (E-mail) address GENERAL QUESTIONS ABOUT POSSIBLE FUTURE APPLICATION NO DO YOU APPLY QUALITY ASSURANCE PROCESSES TO THE E-LEARNING SERVICES THAT YOU OFFER? YES Reasons for non-application, Familiarisation with the concept of quality, Interest for future application DO YOU APPLY A FORMAL QUALITY APPROACH? YES INFORMATION ABOUT THE FORMAL QUALITY APPROACH NO Title, Location, Short description EXPERIENCE FROM QUALITY ASSURANCE / CONTROL PROCESSES APPLICATION GENERAL INFORMATION Function of responsible actor, Cost of implementation, Number of people involved in implementation CONTEXT Educational processes, Target group ASSESSMENT OF THE EXPERIENCE Assessment value, Success / Failure factors Figure 2.2: Outline of the structure of the second questionnaire about the collection of data regarding the description of an experience from an e-learning quality approach application. 20
2.2.3.3 Description of the questions per section Here we will shortly present the questions included in each of the four questionnaire sections. Section 0: Information about person completing the questionnaire The second questionnaire like the first one, after some general guidelines that provides, asks the interviewee to complete his personal elements as well as data about the organisation he is affiliated with. More specifically, the contributing person is asked to provide the date of the questionnaire completion, his full name as well as some elements useful for the achievement of contact with him in case that the collection of data is not made by means of an interview, such as his telephone number and his e-mail address. After that, the completion of elements that have to do with the organisation is asked, such as its name, type, size and its web site. In the end, the person completing the questionnaire is asked to specify his role in the organisation that he is affiliated with. He has the possibility to choose from an available list or to specify it himself in case that his role is not included in the list. In contrast with the first questionnaire, in the introduction of the specific one some additional questions are included. So, the question comes next whether some quality assurance processes are applied to the e-learning services that the organisation offers. In case the answer is negative some questions come next that have as target the identification of the reasons for which the organisation hasn t and whether there is enough knowledge within the organisation around the concept of quality. In the end, the interviewee is asked whether the performance of it is judged as satisfactory, as well as whether there is some interest for possible future application of quality processes. At this point the questionnaire completion for this type of cases comes to an end. If the answer in the above question is positive, the interviewee is asked to specify whether the organisation applies a formal quality approach and if yes, is asked to provide a short description of it and specify the location where the further search of information is possible. The questions that follow have as target the collection of data regarding the experience from the application of any kind of quality assurance processes in the organisation and are divided in the three following sections. Section 1: General information about the application of the quality approach in your organisation In Section 1 the interviewee is asked to specify the function of the person who was responsible for the application of the quality assurance processes within the organisation, whether the organisation has had prior experience with processes like these in the past and what was the cost of these processes implementation. He is also asked to specify the number of people who were involves in the implementation process of the quality processes and what was their role. In the end the contributor is asked to specify the start and the end - 21