Mission report of Activity: 4.5 Pilot Evaluations

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)
AZ/14/ENP/OT/31

EU Short Term Expert Mission Report

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.5 Pilot Evaluations

Name of the Expert: Ms Eve Eisenschmidt, Mr Hannu Apajalahti, Ms Tiia Bach

Dates of the Mission: 10 – 14 April 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

International cooperation in quality assurance has been an essential element of the Bologna process aiming to create a European Higher Education Area. A central tool in this work has been the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (also known as the ESG).

The Twinning project offers an opportunity for applying the ESG in Azerbaijani higher education. One of the mandatory results of the project is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan (AzSG) in line with the ESG and to test them with three higher education institutions.

In April 2016 a Drafting Group was appointed by the Ministry of Education to work on a proposal for AzSG. A draft manual for the pilot evaluations was discussed with the Advisory Group in a seminar in June 2016. The draft was published on the Twinning project’s website in order to get feedback on the assessment areas and criteria. Amendments were made to the manual based on the feedback. The capacity of the pilot institutions to conduct a self-evaluation was supported through several trainings, which took place starting in September 2016. The self-evaluation reports were submitted by the three pilot universities in January-February 2017 and were translated into English. In February 2017 informative visits were conducted to all three higher education institutions participating in the pilot. Practical details of the upcoming site visits were discussed during these visits.
The main aims of the pilot evaluations have been set as: supporting the strategic management of institutions, providing external feedback to the institutions’ own internal quality assurance procedures as well as informing the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG. The pilot evaluations will have an institutional approach with the focus on teaching and learning. The evaluation report will provide the pilot institutions with information regarding their strengths and good practices as well as recommendations for the institutions’ further development.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG). The objective of Activity 4.5 is to coordinate a trial/test run of the Standards and Guidelines at three higher education institutions in Azerbaijan. The first pilot was conducted at the Azerbaijan State Economic University during the STE mission of 3-7 April 2017.

The aim of the current mission was to conduct a pilot evaluation at Azerbaijan State Pedagogical University during the week of 10-14 April 2017. The first day of the mission focused on preparing the members of the evaluation group for the site visit. That followed by a three-day visit to the university, during which the evaluation team held interviews with the management, staff, students, alumni and external stakeholders of the institution. The outcome of the visit will be an evaluation report, which in addition to recommendations to the university under evaluation will also provide suggestions to the Ministry of Education regarding regulations governing the Azerbaijani higher education.

4. Time Schedule of the Mission

A detailed schedule of the mission is attached to the report (see Appendix 1).

5. Achievement of the Expected Results

The expected results for the mission were achieved: a practical testing of the Standards and Guidelines was carried out at Azerbaijan State Pedagogical University.

A long and thorough discussion about challenges and opportunities of Azerbaijan’s higher education was held among all participants during the kick-off meeting at the Ministry of Education on the first day of the mission. During the second half of the day the team had its meeting, which included a training for the team members by STEs Ms. Eisenschmidt (chair of the evaluation group) and Ms Bach (Project Coordinator) who reviewed the aims of the evaluation, the code of ethics guiding the evaluation team, and how to conduct the interviews and present questions. The team members then focused on formulating detailed questions for the interviews that were going to be held during the site visit with various stakeholders and developing the interview strategy.

The site visit was carried out successfully over three days and the team interviewed altogether nearly 80 members of the university on one of the campuses in Baku. Interview sessions were held with the management, deans, teaching staff, support and administrative staff, students, alumni and employers. During the interviews, the evaluation team was able to ask questions to verify and supplement observations that had been made on the basis of the self-evaluation report and other material submitted by the university. The interviews were conducted in English and translated into Azerbaijani. The interviews were held in a good and open atmosphere.
Three observers – one from Baku State University and two from the Accreditation and Nostrification Office (ANO) – accompanied the evaluation team in order to learn from such evaluation process.
The evaluation team had a good cooperation with the contact person and other representatives of the university who were extremely helpful during the entire site visit by providing everything that was necessary for the team to successfully complete its work.

6. Unexpected Results

None.

7. Issues Left Open After the Mission

None.

8. Recommendations for Future Missions

As recommendations for the next pilot evaluation, the STEs suggest the following:

- A detailed schedule of the site visit was sent only a day before arrival at the university. Thus, the evaluation group did not have a clear idea beforehand who exactly (people and their positions) they were going to meet. If possible, ensure that the team receives it earlier to better prepare for the interviews.

- It is highly recommended that even if the evaluation process is new for the local team members, all members do the preparation work on the Self-Evaluation Report of the institution under evaluation and give their initial input by sending to the Project Manager (STE from FINEEC/EKKA) questions that arise while reading the report, and list the topics that should be further discussed during the site visit. This would allow the Project Manager to do considerable amount of preparatory work by the first team meeting on Monday (collect the questions, sort them, etc.).

- It is important to allow ample time before the site visit for the team members to get to know each other better, and for the STEs take that opportunity to get to know the local context, culture, what kind of behaviour might be appropriate during the visit and what is not.

- The team leader should pay enough attention that everyone from the team has had enough time to share his/her own understanding of the situation before the site visit. Some extra time might be needed to figure out the best possible approach to asking the questions and dividing the roles among the team members. It is important to secure that each team member is actively involved in the whole process.

- The project manager (STE from FINEEC/EKKA) should use enough time to establish a shared understanding of the practical details (especially number of interviewees, interview settings, room arrangements, name labels for the interviews, and rules during the interviews) with the contact person of the university. If necessary, make re-arrangements during the site visit (e.g., if the rectorate has been prepared for the interviews, ask for a more appropriate, separate room where discussions can be held). In that sense the meeting with the university’s contact person was extremely helpful on the first day of the mission; the team could go through the practical details of the visit (some of which seemed to come as a surprise, although this had been shared by email earlier), clarify some issues and questions, request for additional materials, etc.

9. Conclusions and General Remarks Concerning the Project

The pilot evaluations are a pivotal point in the Twinning project. Two pilot evaluations have now been conducted. The pilot evaluation in Azerbaijan State Pedagogical University was conducted in good cooperation with the participating higher education institution. ANO has been actively involved in the process and these evaluations have been hopefully an interesting experience for the local observers. The evaluation team felt very well supported and taken care of by the Resident Twinning Adviser and other support staff during the whole week. We are equally thankful for excellent interpretation that was provided for the interviews during the site visit.

The Pilot Analysis has been planned for early June. Although the timeline for the pilots is quite short and the teams are pressed for time, they will likely provide plenty of useful feedback on the criteria and evaluation model of the Standards and Guidelines for Quality Assurance of Higher Education in Azerbaijan.

Mission report of Activity: 4.5 Pilot Evaluations

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.5 Pilot Evaluations

Name of the Expert: Ms. Kristiina Tõnnisson, Mr. Mikko Vieltojärvi, Ms. Hilla Aurén

Dates of the Mission: 3 – 7 April 2017 (Ms. Kristiina Tõnnisson), 5 – 7 April 2017 (Ms. Hilla Aurén and Mr. Mikko Vieltojärvi)

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

International cooperation in quality assurance has been an essential element of the Bologna process aiming to create a European Higher Education Area. A central tool in this work has been the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (also known as the ESG). The Twinning project offers an opportunity for applying the ESG in Azerbaijani higher education. One of the mandatory results of the project is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan (AzSG) in line with the ESG and to test them with three higher education institutions.

In April 2016 a Drafting Group was appointed by the Ministry of Education to work on a proposal for AzSG. A draft manual for the pilot evaluations was discussed with the Advisory Group in a seminar in June 2016. The draft was published on the Twinning project’s website in order to get feedback on the assessment areas and criteria. Amendments were made to the manual based on the feedback. The capacity of the pilot institutions to conduct a self-evaluation was supported through several trainings, which took place starting in September 2016. The self-evaluation reports were submitted by the three pilot universities in January-February 2017 and were translated to English. In February 2017 informative visits were conducted to all three higher education institutions participating in the pilot. Practical details of the upcoming site visits were discussed during these visits.

The main aims of the pilot evaluations have been set as: supporting the strategic management of institutions, providing external feedback to the institutions’ own internal quality assurance procedures as well as informing the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG. The pilot evaluations will have an institutional approach with the focus on teaching and learning. The evaluation report will provide the pilot institutions with information regarding their strengths and good practices as well as recommendations for the institutions’ further development.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG) and test them with three higher education institutions.

The objective of Activity 4.5 is to coordinate a trial/test run of the Standards and Guidelines at three higher education institutions in Azerbaijan. The first pilot was conducted at the Azerbaijan State Economic University during the STE mission of 3-7 April 2017. The first two days of the mission focused on preparing the experts of the evaluation group for the site visit. During the last three days of the mission, the evaluation group conducted a site visit to the University, interviewing management, staff, and students of the University.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results for the mission were achieved: A practical testing of the Standards and Guidelines was carried out at the Azerbaijan State Economic University.

During the training, STE Ms. Tõnnisson reviewed the aims of the evaluation, the code of ethics guiding the evaluation team, and how to conduct the interviews and present questions. A long and thorough discussion about challenges and opportunities of Azerbaijan’s higher education was held among all participants during the kick-off meeting. Later the discussion focused on challenges of the site visits and possibilities of writing evaluation report. The second day focused more concretely on preparing the meeting with various stakeholders and on formulating the questions. The team worked the full day on forming and reforming, on structuring and restructuring, on focusing and refocusing the questions for all sessions for the meetings during the 3 days long site visit.

The site visit was carried out successfully over three days and the team interviewed altogether nearly 100 members of the university on three different campuses. Interview sessions were held with the management, teaching staff, support and administrative staff as well as students from different faculties. During the interviews, the evaluation team was able to ask questions to verify and supplement observations made based on the self-evaluation report and other material submitted by the university.

Observers from the Accreditation and Nostrification Office (ANO) accompanied the evaluation team. The interviews were conducted in English, translated into Azerbaijani.

The University contact person had expressed beforehand concerns about external stakeholders being unable to attend the interviews. The evaluation team used the time that was allocated for interviewing external stakeholders by interviewing personnel from the Career Centre, the Internship Centre and the Communication and Marketing Department. The ability of the evaluation team to quickly adapt to changing conditions during the interview process was a key to making the site visit successful. The cooperation with the University was carried out in a good atmosphere.

6. Unexpected Results

The site visit had to be rescheduled with very short notice, and as a result the original mission plan was adjusted. The evaluation team ended up having one additional day for an all-day meeting among the evaluation team, which provided very useful. Under the guidance and instruction of STE Kristiina Tõnnisson, the team was able to create a common understanding of the most important issues for the interviews and to prepare the final list of interview questions. Although this day was not part of the original mission plan, due to the other two STEs’ difficulties in obtaining visas during the Nowroz holiday, the site visit was moved toward the end of the week (Wednesday-Friday) and this opened up one more day for the team to prepare.

7. Issues Left Open After the Mission

The request of compensation for local participants of the evaluation team is being processed at the Ministry of Education. In addition, the participation of the evaluation team members in the Pilot Analysis has to be confirmed. 

8. Recommendations for Future Missions

As recommendations for the next two pilot evaluations, the STEs suggest the following:

• Reserve considerable amount of time for creating a common understanding among the evaluation team before the interviews. It is important that team members are prepared and they have thought about what are the main issues to find out during interviews, since time in-country before the site visit will be limited.
• The project manager (STE from FINEEC/EKKA) should use enough time to establish a shared understanding of the practical details (especially number of interviewees, interview settings, room arrangements and rules during the interviews) with the contact person of the university.
• There are several terms which the STEs realized have different meanings or lack meaning in the local context, and it is crucial to establish shared understanding of such concepts (e.g. course, syllabus, curricula, study programme, subject, learning outcome etc.); It would be useful to have some short written introduction to national higher education system before the visit.
• The team leader should pay enough attention that everyone from the team has had enough time to share his/her own understanding of the situation before the site visit. Some extra time might be needed to figure out the best possible approach to asking the questions and dividing the roles among the team members. It is important to secure that each team member is actively involved in the whole process.

9. Conclusions and General Remarks Concerning the Project

The Pilot Evaluations are a pivotal point in the Twinning project. One Pilot Evaluation has now been conducted and two more are to take place within the month of April. The Pilot was conducted in good cooperation with the participating higher education institution. ANO has also been actively involved and will be sending observers to all three site visits.

The Pilot Analysis is set to take place in early June. Although the timeline for the Pilots is quite short and the teams are pressed for time, the Pilots will likely provide plenty of useful feedback on the criteria and evaluation model of the Standards and Guidelines for Quality Assurance of Higher Education in Azerbaijan.

Mission report of Activity: 4.5 Pilot Evaluations

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.5 Pilot Evaluations

Name of the Experts: Mr. Heikki Malinen, Mr. Tauno Otto, Mr. Touko Apajalahti

Dates of the Mission: 24 – 28 April 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

International cooperation in quality assurance has been an essential element of the Bologna process aiming to create a European Higher Education Area. A central tool in this work has been the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (also known as the ESG). The Twinning project offers an opportunity for applying the ESG in Azerbaijani higher education. One of the mandatory results of the project is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan (AzSG) in line with the ESG and to test them with three higher education institutions.

In April 2016 a Drafting Group was appointed by the Ministry of Education to work on a proposal for AzSG. A draft manual for the pilot evaluations was discussed with the Advisory Group in a seminar in June 2016. The draft was published on the Twinning project’s website in order to get feedback on the assessment areas and criteria. Amendments were made to the manual based on the feedback. The capacity of the pilot institutions to conduct a self-evaluation was supported through several trainings, which took place starting in September 2016. The self-evaluation reports were submitted by the three pilot universities in January-February 2017 and were translated to English. In February 2017 informative visits were conducted to all three higher education institutions participating in the pilot. Practical details of the upcoming site visits were discussed during these visits.

The main aims of the pilot evaluations have been set as: supporting the strategic management of institutions, providing external feedback to the institutions’ own internal quality assurance procedures as well as informing the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG. The pilot evaluations will have an institutional approach with the focus on teaching and learning. The evaluation report will provide the pilot institutions with information regarding their strengths and good practices as well as recommendations for the institutions’ further development.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG) and test them with three higher education institutions.

The objective of Activity 4.5 is to coordinate a trial/test run of the Standards and Guidelines at three higher education institutions in Azerbaijan. The objective of this mission was to conduct the third pilot at the Azerbaijan Technical University during the STE mission of 24-28 April 2017.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results for the mission were achieved: A practical testing of the Standards and Guidelines was carried out at the Azerbaijan Technical University.

The first day of the mission focused on preparing the experts of the evaluation group for the site visit. During the following three days of the mission, the evaluation group conducted a site visit to the University, interviewing management, staff, and students of the University. On the last day, the evaluation team began the drafting of the review report.

During the first day, first a kick-off meeting was organised at the Ministry of Education in order to discuss the Azerbaijani higher education system with the evaluation team.

In the afternoon’s meeting of the evaluation team, STEs reviewed together with the rest of the team  the aims of the evaluation, the code of ethics guiding the evaluation team, and how to conduct the interviews and present questions. Also, during the meeting the team finalised preparations for the meetings with various stakeholders during the site visit.

The site visit was carried out successfully over three days and the team interviewed altogether nearly 100 members of the university community. Interview sessions were held with the management, teaching staff, support and administrative staff as well as students from different faculties, graduates and external stakeholders. During the interviews, the evaluation team discussed various topics with the interviewees to verify and supplement observations made based on the self-evaluation report and other material submitted by the university.

Observers from the Accreditation and Nostrification Office (ANO) accompanied the evaluation team. The interviews were conducted in English, translated into Azerbaijani.

6. Unexpected Results

No unexpected results were encountered.

7. Issues Left Open After the Mission

No issues were left open.

8. Recommendations for Future Missions

9. Conclusions and General Remarks Concerning the Project

All pilot evaluations have now been conducted. Also the third pilot was conducted in good cooperation with the participating university. ANO has also been actively involved in all three pilot evaluations.

It is recommended that ANO and other project participants will take advantage of the experience and of the upcoming analysis week in order to further building of the capacity of the universities to undertake such evaluations. Special attention should be paid into the self-evaluation phase, in order to have a comprehensive self-evaluation and to include all the necessary annexes. Also, it would be good to have more systematic training for the evaluators in an early phase of the evaluation process: both for those new team members for whom it is the first experience of such process, and for international members who do not have deep knowledge of the local higher education system before the process. However, in conclusion, based on the third pilot, the university and the local experts are fully capable of carrying out such processes in a fruitful and beneficial manner.

Mission report of Activity: 1.2 Revisions of the Legislative Framework

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 1. Legal and Regulatory

Activity: 1.2 Revisions of the Legislative Framework

Name of the Experts: Ms Sille Uusna, Ms Heli Mattisen

Dates of the Mission: 1-5 May 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

During the first missions of Activity 1.2, in 2016 the STEs gave to MoE some concrete recommendations regarding the new State standard and program for higher education (State standard) draft, in order to ensure its compatibility with the principles of the European higher education area and create consistency between different regulations concerning the HE subject area. The next 1.2. mission was focused on the Statute of higher education institutions (Statute of HEIs) and led to recommendations. The third mission of 1.2. concentrated to accreditation rules.

Recently, the MoE introduced to STE’s updated drafts of:

• the State Standard for HE;
• the Accreditation standards for evaluating the activities of HEIs;
• the Accreditation rules of HEIs and study programs.

3. Objectives and Tasks of the Mission

This mission purpose was to evaluate the progress that have been made by implementing STEs recommendations and to provide concrete input into legislative drafts based on the recommendations and to support the elaboration process of the legislation relevant to QA and the AzQF sections relevant for higher education, and adherence to the EHEA.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results of the mission were achieved. In workshops, the recommendations were comprehensively discussed.

6. Unexpected Results

There were no unexpected results.

7. Issues Left Open After the Mission

According to the representatives of the MoE, the new draft of the Statues of HEIs is under preparation. The new draft should consider most of the recommendations made previously by the STEs. The MoE continues work with the draft of State standard.

8. Recommendations for Future Missions

The cooperation between experts from different departments in MoE should be fully recognized and strongly encouraged also during future Missions. State standards about doctoral education and learning outcomes and general quality requirements.

9. Conclusions and General Remarks Concerning the Project

Considerable efforts have been made by the Ministry of Education (MoE) in adopting the principles of European higher education area. The draft of the new State standard takes into account several recommendations that STEs have made during previous missions. However, the STEs still have recommendations which include some unevitable ones regarding the overall drafting process and further developments of the Statute of HEIs and the State standard, and accreditation rules and standards.

We were impressed by improvements made in both accreditation documents developed by ANO - Accreditation standards for evaluating the activities of higher education institutions and Accreditation rules of higher education institutions and study programs. The document providing the standards for accreditation is in sound accordance with ESG while taking into account national context. Pilote evaluations conducted in the framework of Twinning project will certainly give an evidence based input for further development of both documents.

URGENT RECOMMENDATIONS

Cooperation

1. During the mission, STEs have met with highly competent and motivated experts from HE Department, ANO and Legal Department. We strongly recommend promoting cooperation between MoE experts to combine competence in different fields. It is also advisable to use the format of joint brainstorming, working groups and also more informal communication during working process for discussing future initiatives of HE issues the MoE. Joint effort and promoted cooperation, also in other manner than formal feedback in official consulting process, could lead to an excellent result.

2. It would be most advisable to promote cooperation with stakeholders in HE field. For example, supporting the establishment of Rectors Conference or form a higher education advisory board, led by the MoE.

Hierarchy of legislation

3. At the moment there seems to be indistinctness about the hierarchy of different legal acts and provisions. We recommend that MoE experts from different departments could in a form of joint brainstorming draft a possible structure and hierarchy of reasonable HE legislation system, aiming to avoid overlapping while assuring coherence.

4. Some efforts have been made in reviewing the existing legislation and repealing the provisions not needed anymore (consolidation of accreditation regulations, a new draft for Scientific Board etc). We strongly recommend continuing with that kind of activities. For example, the separate regulations for each study level (BA, MA, PhD) should be consolidated in the State standard.

5. According to the representatives of MoE, a new draft of Statutes of HEIs will be prepared. The STEs recommend to develop the Statutes of HEIs to comprehensive higher education (HE) framework regulation which includes all basic principles of HE.

Quality requirements

6. At the moment, the quality requirements for establishing new HEIs/study programs have not been included in any general framework regulation yet. The link between opening a new study program and accreditation seems to be not clear.

In order to ensure the continuity, the quality requirements for opening new study program should be in accordance with the requirements later taken into account for accreditation. There may be a need for more detailed regulation how to initially evaluate the quality of a new curricula. The STEs recommend to use as much as possible the competence and recourses of ANO to provide continuity between initial and following evaluations.

We recommend:

Add to the Statute for HEI (framework act)

• Basic principles for opening a new curriculum, including initial quality control;
• Basic principles for accreditation, including the object of accreditation (institutions or study programs or both); accreditation period; accrediting body and it’s general formation principles, composition (incl. limitations); consequences of a negative accreditation.

Add to the State Standard of HE

• Basic quality standards describing in general terms the societal expectations towards the quality of HE in Azerbaijan. Based on the general quality standards ANO will specify the quality requirements and assessment criteria at the next level.

An example of general quality standards which are in line with ESG and could be the bases for more detailed requirements and criteria in the framework of external assessment:

educational values

• The higher education institution shall develop students’ ability to adapt to modern requirements and conditions, to be competitive, independent and creative individuals and responsible citizens, to live and act in information society and to build communication.
• The institution shall support academic integrity and is vigilant against academic fraud; it supports guarding against intolerance of any kind or discrimination against the students and staff.

learning outcomes and ECTS

• The design and development of study plans is in line with areas of activity and educational objectives of the institution, expectations of the society, labour market needs, and the latest research.
• Objectives, intended learning outcomes, admission and graduation requirements of the programs are clearly defined.

• Expected student workload is defined in credit point system at all levels of higher education.
• Qualification resulting from the program is referred to the appropriate level of the European qualifications framework in order to assure the comparability of qualifications.

academic staff

• Academic staff with adequate qualifications ensure the achievement of the objectives and learning outcomes of the study program, as well as the quality and sustainability of teaching and learning.
• The distribution of full-time teaching staff by age and qualifications facilitates the sustainability of studies.

teaching and learning process

• Teaching and learning takes into account the educational needs of students, provides flexible learning opportunities and encourages active participation of students in creating the learning process.
• Practical work/internship supports the achievement of the learning outcomes of the program.
• Student assessment is objective, consistent, transparent and supports the achievement of learning outcomes.
• Up-to-date teaching materials, innovative, creative and interactive learning methods and educational technology are used in teaching and learning process.

resources and support services

• Resources (teaching and learning environments, teaching materials, teaching aids and equipment, premises, financial resources) support the achievement of objectives in the study program.
• Students are provided academic, career and psychological counselling by the higher education institution. Students’ academic progress is monitored and supported; the institution has processes and tools to support special educational needs.
• Information technology tools for studies and research and development activities and connections to domestic and international information networks are accessible and necessary data bases are available.

Learning outcomes

7. The STEs recommend to include the HE level descriptors (levels 6-8) from the draft of NQF in an Annex to the State Standards for HE and to refer to their correspondence with a respective level in EQF. It gives a clear reference point to curriculum developers and supports Azeri HE to meet the objectives of the Bologna process.

8. As there is no differentiation of learning outcomes at level 8 between PhD and Doctor of Sciences, and the compulsory credit point system is not applicable for the program of Doctor of Sciences, therefore we recommend not to describe the DSc program in the State standards for higher education.

9. The principles of the student assessment are very well described (e.g., transparency, objectivity, fairness, self-evaluation, diverse methods and tools of assessment). In addition, the main aim of assessment should also be clearly stated: to measure whether the learning outcomes have been achieved as well as to support students’ learning. The assessment methods and criteria should be in full accordance with the learning outcome(s).

Accreditation rules and standards

10. According to clause 3.3 of the ESG, agencies should be independent and act autonomously. This includes organizational and operational independence and independence of formal outcomes. Currently, ANO enjoys a fair amount of operational independence. However, as ANO is a part of the MoE and the final decisions are made by the Accreditation Council chaired by the minister of education, organizational independence is clearly lacking.

As soon as the full trust of the Ministry has been gained, we recommend giving ANO full organizational and operational autonomy regarding its procedures and formal outcomes of the quality assurance processes. Implementing this recommendation definitely requires some fundamental changes on legislative level, but is essential.

11. The rules for the composition of the Accreditation Council have been removed from the last draft of the Accreditation Rules and Procedures, as they will be defined in another document Regulation of the Accreditation Council approved by the Ministry of Education. However, it is important to clarify the rules and principles (incl the number of Council members and the election period) for selection of the member of the Accreditation Council and make the inclusion of a student member in the Council obligatory. An example of composition rules for the council applied by EKKA can be found here.

12. According to clause 3.5. of ESG agencies should have adequate and appropriate resources, both human and financial, to carry out their work. It is in the public interest that agencies are adequately and appropriately funded, given higher education’s important impact on the development of societies and individuals. Therefore, it is essential to provide financial resources needed for conducting quality assurance activities in an objective, effective and efficient manner and for involving best experts.

There are different practices regarding financing of external quality assurance activities. The most common practice is, that the basic funding of agencies (covering permanent staff, premises and other permanent costs) is offered by the state and institutions are paying fees covering variable costs connected to the single accreditation procedures (expert fees, transport, accommodation). While ANO’s permanent costs are fairly covered by the MoE sources for covering variable costs depending on the scope of accreditations are clearly lacking. We recommend to develop and approve the accreditation fees for higher education institutions depending on the scope (size of the institution, number of programs etc) of the accreditation procedure.

13. Highly qualified and trained experts are the key actors in the accreditation procedure. In addition to their academic competence they have to be aware of their role in the procedure, follow the code of ethics of experts and have a common understanding of the quality standards (see also ESG 2.4). Therefore, it is essential to add to the requirements for a member of an expert group the participation in a training offered by ANO.

14. According to the clause 2.7. of ESG complaints and appeals processes should be clearly defined as part of the design of external quality assurance processes and communicated to the institutions.

A complaints procedure allows an institution to state its dissatisfaction about the conduct of the process or those carrying it out. In an appeals procedure, the institution questions the formal outcomes of the process, where it can demonstrate that the outcome is not based on sound evidence, that criteria have not been correctly applied or that the processes have not been consistently implemented. We recommend to include the description of the complaints and appeal procedures in the Accreditation Rules of Higher Education Institutions and Study Programs.

ADDITIONALLY, TO CONSIDER OTHER RECOMMENDATIONS PROVIDED IN PREVIOUS REPORTS

15. As there is still no legal framework regarding the possible use of foreign agencies in the accreditation process, no further developments can be detected regarding the use of foreign agencies in the accreditation process in ANO regulations. Presumably, changes should first be made on legislative level. Thus, we strongly recommend including the conditions and procedure for the use of foreign agencies in the Statute of HEIs and mention this possibility also in the Accreditation Rules. Whether or not to officially recognize the results of any kind of cross-border evaluation is obviously a policy decision for the MoE, i.e. Accreditation Council headed by the Minister. However, we believe that allowing these kinds of activities would create a positive incentive for further internationalization of the HEIs and also ANO. It would also help to disseminate best practices of other agencies.

16. The STEs, taking into account the recommendations of previous missions, point out the need to clarify in framework act the main division of responsibilities between Rector and Scientific Board. The STEs also recommend to consider some performance indicators to universities and to regulate the structure of self-administration of student unions in a centralized manner. Also, the STEs recommend to include the basic criteria and characteristics of a research university in either the Education Law or the Statute of HEIs. 

Mission report of Activity: 3.4 Further Implementation of the AzQF

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 3. Developing AzQF

Activity: 3.4 Further Implementation of the AzQF

Name of the Expert: Mr Olav Aarna, Ms Maiki Udam, Ms Aune Valk

Dates of the Mission: 8-12 May 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

The development of Azerbaijani National Qualifications Framework (AzQF) for lifelong learning started already in 2011 supported by the World Bank and the European Training Foundation (ETF) projects. By 2012, the draft Decree on establishing the AzQF was developed, stipulating the main principles, institutions involved, level descriptors, placement of types of qualifications, and quality assurance of qualifications. In 2012 the draft AzQF was submitted to the Council of Europe for expertise. Recommendations proposed by the Council of Europe expert have been taken into account while designing the AzQF final draft. Seminars introducing the AzQF to stakeholders and international partners were held in 2012 and 2014. Until recently, the Decree has still been a draft form.

In 2014, ETF prepared a comprehensive document “Analysis of existing qualification standards in Azerbaijan”, containing the descriptions of different qualifications' levels as well as concrete recommendations for further actions for 2015-2017 with division of roles and responsibilities in development, revision, approval and application of AzQF. A draft of the implementation plan for the AzQF has also been prepared by the ETF and Azerbaijani counterparts.

In November 2015 the Minister of Education introduced the draft AzQF document to the line ministries and other stakeholders. Feedback from different ministries included several recommendations for amendments in the text.

During the last two years the Ministry of Education (MoE) staff has been renewed fundamentally. Only one person, Mr. Azad Ahundov, has been involved in the development of AzQF from the very beginning. As a result, the new staff members, incl. the heads of departments and units of the MoE have limited understanding about the AzQF purpose, objectives, guiding principles, and the way to implement it. More importantly, until recently there has been no clear ownership of the AzQF in the MoE.

The current state of affairs is the following. In the first week of May 2017 the Minister of Education submitted the draft Decree on establishing the AzQF with minor changes to the Cabinet of Ministers for approval. Following the decisions taken at the meeting with the minister on 22 July 2016, a secretariat headed by the deputy head of Apparatus Mr. Farzali Gadirov has been established at the MoE to administer further development and implementation of the AzQF. National expert Mr. Ilkin Nazarov has also prepared a draft “Action Plan on Implementation of National Qualifications Framework for Lifelong Learning of the Republic of Azerbaijan (2018-2025)”, and a “Concept of establishment of National Commission and Secretariat for implementation of AzQF” commissioned by the MoE.

3. Objectives and Tasks of the Mission

Objective:

The elaboration of a roadmap for the further implementation of the AzQF sections relevant for higher education (leading towards a successful self-certification).

Tasks (methods):

• An one-day workshop on further implementation of the AzQF HE sections for the representatives of MoE and universities has been conducted for 10-20 people;
• A roadmap with objectives, responsibilities, timelines and an indication of required resources for the self-certification process has been drafted.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

All expected results were achieved.

6. Unexpected Results

There were no unexpected results.

7. Issues Left Open After the Mission

There are no issues that have been left open after this mission.

8. Recommendations for Future Missions

As it has been the final mission week of the Component 3 then there are no more missions coming.

For future projects we would recommend to deal especially with the following issues concerning the implementation of the AzQF:

• Developing the system for recognition of prior (including non-formal and informal) learning;
• Creating databases/IT system(s) for managing the information about students, staff, study progress, admission, graduation etc;
• Training teachers in innovative teaching and assessment methods.

9. Conclusions and General Remarks Concerning the Project

The mission has proved to be successful. The AzQF development and implementation process has gained new momentum. The draft “Action Plan on Implementation of National Qualifications Framework for Lifelong Learning of the Republic of Azerbaijan (2018-2025)”, and the “Concept of establishment of National Commission and Secretariat for implementation of AzQF” are a good starting point for launching the AzQF implementation.

A general recommendation would be to add also wider impact indicators to the action plan, e.g., increased mobility of students and staff, increased participation of adult people in lifelong learning, employment rate of graduates of VET and higher education, etc.

The recommendations concerning further development of the Action Plan are incorporated into the text of the document and in the column under the title Note (cf. Annex 2).

Discussions with the employees of the MoE and participants of the seminar on 11 May 2017 revealed the following overarching problem areas requiring special attention for the success of the AzQF implementation process:

• Timeframe of implementation

Implementation of the AzQF means developing a new and contemporary National Qualifications System, with the AzQF serving as a nucleus of this system. The Action Plan encompasses the period 2018-2025. It foresees actions in five groups:

1. Improving legislative framework in connection with implementation of AzQF.
2. Creating institutional system coordinating AzQF implementation.
3. Bringing content of qualifications into conformity with the labor market requirements.
4. Improving and strengthening quality assurance system.
5. Promoting implementation of result-based qualification system.

Many activities cover very long period, e.g. 4.1. Creating National Qualifications Register (2018-2025).  Obviously the creation of the Register does not take seven years. More important is to know by which time the Regiser will become functional.

We recommend to define intermediate milestones of the AzQF implementation, e.g. self-certification of the AzQF in reference to the Qualifications Framework of the European Higher Education Area (and European Qualifications Framework) by 2020 (p.5.8).

• Institutional structure for the AzQF implementation

The Concept foresees establishing two new bodies for coordinating the implementation process:

• National Commission,

The Secretariat is envisaged as an independent structural unit of the MoE, performing three functions:

• Management of qualifications;
• Development of qualification standards;
• Development of the National Qualifications Register.

The functions and structure of the Secretariat seem appropriate. However, performing these functions is a full-time job for substantial number of employees. Therefore, the experts strongly recommend:

• Establish the Secretariat as an independent department of the MoE with permanent staff (gradually growing in number);
• Consider changing the name for more appropriate one, because the present one is misleading. It rather refers to the secretarial support for the National Commission.
• Formalise the roles of the institutional partners involved in the different aspects of the AzQF implementation, incl. functions performed for different types of qualifications, on an appropriate level.

It is of utmost importance to engage all partners and stakeholders (ministries and agencies, educational institutions, student unions, employers’ organisations) into the implementation process of the AzQF. It requires regular communication not only in formal setting of meetings of the National Commission and other decision making bodies, but also different less formal means of informing each other about the developments and problems with respect to the AzQF implementation.

• Qualifications standards

In the draft Action Plan on Implementation of National Qualifications Framework for Lifelong Learning of the Republic of Azerbaijan (2018-2025) development of different qualification standards is foreseen (see 3.2, 3.5, 3.6). We recommend to have two sets of standards:

• Level standards for eight levels of the AzQF;
• Sectoral standards for specific occupations/professions, including tasks and competences of a particular occupation/profession. These standards could also include assessment standards for assessing, validating and certification persons’ competence.

Based on the level standards and sectoral standards both the national and institutional curricula can be developed. There is no need for separate qualification standards on formal education (3.6.1) and qualification standards for non-formal and informal education (3.6.2). 

• Human resource development: training and networking

It is obvious that engagement of people, training and supporting them is crucial in the successful application of any new initiative.

Besides classical training that is necessary for several groups and topics, we recommend to create and support networks of practitioners. Professional networks have proved to be better teacher training method than traditional face to face training (see additional Measure 2.7).

For developing and applying outcomes-based curricula, introducing new teaching and assessment methods, developing systems for recognition of prior (informal and non-formal) learning

(counselling, assessment and accreditation), supporting mobility and academic recognition, developing lifelong learning opportunities at universities etc, there must be people responsible for these topics in the HEIs. In the starting phase, there is often just one person in the institution who shall introduce it. For him/her it is very difficult if there is no support. The best supporters are people with similar tasks and experience in other institutions. Helping these people to get together and create a network is very important if you want that the best practices are shared. Centralised leading and support system may be not enough in these cases.

As the largest target group in applying the changes are university teachers, it would be useful to set up centre(s)/units that are responsible for teaching academic staff (see additional Measure 2.6).

• Recognition of prior learning

Recognition of prior (informal and non-formal) learning (RPL) is a corner stone of lifelong learning mindset. Currently this topic is covered at two levels in the action plan: legislation and promotion to end-users. The most difficult target group is, however, universities. Preparing their systems and training their people (counselling and assessment of RPL) are even more crucial and recommended to add to the action plan. 

• Quality Assurance

Quality assurance, both internal and external, is an important part of national qualifications systems. One objective in the action plan is Improving internal and external quality assurance system and linking it with ESG and EQAVET (4.3).

The topics that need further consideration with respect to the ESG are among others:

• Independence of ANO (especially in regard to decision-making process, i.e the accreditation/assessment decisions should not be influenced by any third body, including MoE);
• Follow-up procedures and activities during review cycles.

In addition, it is necessary to train institutions on internal quality assurance, including self-evaluation, and review experts on external quality assurance.

Creation of databases and IT systems would help to collect and monitor data about study programmes, staff, students and alumni.  

EVALUATION REPORT FOR INSTITUTIONAL PILOT EVALUATION of Azerbaijan State University of Economics

EVALUATION REPORT FOR INSTITUTIONAL PILOT EVALUATION of Azerbaijan State Pedagogical University

EVALUATION REPORT FOR INSTITUTIONAL PILOT EVALUATION of Azerbaijan Technical University

Mission report of the Activity: 4.6 Analysis of the Pilots

TWINNING AZERBAIJAN

Support to the Ministry of Education of the Republic of Azerbaijan for Further Adherence of the Higher Education System to the European Higher Education Area (AZ-ad-EHEA)

AZ/14/ENP/OT/31

EU Short Term Expert Mission Report 

1. Basic Information

Component and Activity:

Component: 4. Standards and Guidelines for QA in HE

Activity: 4.6 Analysis of the Pilots

Names of the Experts: Ms Helka Kekäläinen, Ms Heli Mattisen, Ms Kirsi Hiltunen, Ms Kati Isoaho, Mr Hannu Apajalahti, Mr Tauno Otto, Mr Heikki Malinen, Mr Touko Apajalahti, Ms Tiia Bach, Ms Lagle Zobel

Dates of the Mission: 5 – 9 June 2017

Contractor: Finnish Education Evaluation Centre (FINEEC) /

   Estonian Quality Agency for Higher and Vocational Education (EKKA)

2. Relevant Background Information/State of Affairs

The objective of Activity 4.5 was to coordinate a trial/test run of the Standards and Guidelines at three higher education institutions in Azerbaijan. The main aims of the pilot evaluations was to support the strategic management of institutions, provide external feedback to the institutions’ own internal quality assurance procedures, as well as inform the internal and external stakeholders of the compliance of the institutions’ quality assurance with the ESG.

The three pilot universities (Azerbaijan State Pedagogical University, Azerbaijan State Economic University and Azerbaijan Technical University) submitted their self-evaluation reports in January – February, 2017. The site visits of the pilot universities took place in April, 2017 and the assessment reports were finalized in May – June, 2017.

The pilot evaluations had an institutional approach with the focus on teaching and learning. The evaluation reports provide the pilot institutions with information regarding their strengths and good practices, as well as recommendations for the institutions’ further development and suggestions to the MoE regarding the regulations governing higher education in Azerbaijan. The summary of main strengths and areas of improvement of UNEC, ASPU and AZTU is set out in Annex I of this report.

3. Objectives and Tasks of the Mission

The overall aim of the Component 4 is to develop Standards and Guidelines for Quality Assurance in Higher Education in Azerbaijan, in line with the European Standards and Guidelines for Quality Assurance (ESG). The main objectives of Missions of Activity 4.6 were:

• to analyze the results of the pilot evaluations and to integrate remarks into the revised proposal for the Standards and Guidelines for Quality Assurance, and
• to propose a roadmap for the implementation of an EHEA compatible QA system (including an ENQA compatible QA Agency for HE), and ensure appropriate dissemination.

4. Time Schedule of the Mission

5. Achievement of the Expected Results

The expected results for the mission were achieved: the results of the pilot evaluations were analysed and recommendations were prepared for the future activities of the Accreditation and Nostrification Office (ANO) in Azerbaijan.

From Monday, June 5, to Tuesday, June 6, feedback meetings were conducted with the Azerbaijan State Economic University, the Azerbaijan State Pedagogical University and the Azerbaijan Technical University. During the meetings, the members of evaluation panels gave a presentation outlining the main strengths and areas of improvement of each institution, followed by a discussion about possible follow-up activities by the HEIs. The HEIs also had an opportunity to give a short presentation on the impact of the evaluation and to describe some good practices identified in their university during the evaluation process.

On Wednesday, June 7, an analysis of the pilot evaluations took place in the format of a Learning Café workshop, where the representatives of the HEIs and the panel members had an opportunity to share during four respective sessions their opinions and suggestions on the topics of 1) self-evaluation, 2) criteria and manual for pilot evaluations, 3) site visit end the evaluation process, and 4) the evaluation report. The project managers of the panels served as rapporteurs to these sessions and compiled a summary of the key findings under each topic set out in Annex I to this report.

On Thursday, June 8, a discussion took place between the STEs, panel members and the representative of ANO resulting in recommendations for ANO’s further activities in the areas of assessment criteria, self-evaluation, composition of panels and training, the site visit and writing of the evaluation report.  These recommendations support the implementation of an EHEA compatible QA system in Azerbaijan and are set out in Annex II to this report.

6. Unexpected Results

On June 2, 2017, an Action Plan on the implementation of the Order №2199 of the President of the Republic of Azerbaijan on the “Additional measures with regard to increasing the efficiency of business environment and to further improving the standing of our country in international ratings” was approved by the Order of the President of Republic of Azerbaijan[1]. In Section 9.1.2.3 of this Action Plan (Quality of education system), it is stated that standards and guidelines for quality assurance in Azerbaijani higher education system are developed in line with the relevant European Standards and Guidelines within the TWINNING programme of the European Union​.

7. Issues Left Open After the Mission

No issues were left open after the mission

8. Recommendations for Future Missions

-

9. Conclusions and General Remarks Concerning the Project

The main findings of the Mission are set out in Annexes I and II to this report as follows:

ANNEX I: The summary of main strengths and areas of improvement of UNEC, ASPU and AzTU and summary of the key findings of the analysis of pilot evaluations

ANNEX II: Recommendations for ANO’s further activities supporting the implementation of an EHEA compatible QA system in Azerbaijan

Annex I to the Mission report of Activity: 4.6 Analysis of the Pilots

ANNEX I  

Main strengths and areas of improvement in the area of higher education in the Republic of Azerbaijan based on pilot evaluations

SUMMARY OF THE PILOT EVALUATION IN UNEC, ASPU AND AzTU (mark with ‘X’):

OVERALL STRENGTHS

• The roles and focuses of the universities – pedagogy, economy, technology –  are clear and well justified
• Willingness to improve, awareness of the need for change
• Orientation towards improving the management structure and create conditions for the senior management staff to take more responsibility and ownership in fulfilling their role
• Awareness of the need to improve management and leadership competences

OVERALL AREAS OF IMPROVEMENT

• The universities should work on its mission and vision statements and general objectives, which could be more academic in content and which lay down the general framework for the strategic choices.
• All the relevant stakeholders should be taken along to the strategy process in the future. The universities should pay more attention to inform all stakeholders about the strategic development plan and the objectives.
• Even if the management and staff are aware of the needs to monitor changes in the labour market, the needs of the society, and the need to involve more stakeholders, addressing those needs requires a more systematic and efficient approach.
• Management of lower level is very fragmented – very many positions and duties. There is too much focus on rector’s level – middle management level does not work properly.
• Reviewing of study programmes and syllabuses should be conducted regularly. Students and external stakeholders should be involved in these reviews.
• The improvement of teaching staff’s competencies as well as the monitoring of the improvement should be systematic.
• More emphasise should be given to the relevance of up-to-date teaching materials, technical resources should be improved and the efficient use of technical equipment in the teaching and learning process assured.
• Learning outcome based and student-centred approach in teaching and learning need to be developed, e.g. praising the first one and supporting the last ones, as the chain is as strong as the weakest link; more real elective courses students; assessment of achieved learning outcomes instead of multiple choice tests etc.
• Establishing systematic mechanisms that support the staff in conducting research activities. The change from teaching mind set to research mind set should be prioritized. Access to international research databases is integral to improving research activities. Courses and training on research methodologies and skills should be offered to students.
• Internationalization is a challenge at all levels – better English language skills, at least one full course in English, participation in exchange programmes (both, academic staff as well as students), international research projects etc.
• The distribution of full-time teaching staff by age is uneven. More younger, qualified and motivated teachers and researchers are needed in order to facilitate the improvements in the area of higher education.

RECOMMENDATIONS AT THE STATE LEVEL

• The  legal and normative framework which regulates governance of higher education institutions should be reconsidered by the Ministry of Education to allow universities to implement a more effective management system, e.g., have two different governing bodies – academic council and operational management body.
• Higher education institutions should be granted more freedom and autonomy to continuously modify and update the curricula of their study programmes according to needs of labour market and based on recent research results.
• Doctoral studies should be reorganised based on the Bologna system, as the 2-level system   of doctoral degrees is demotivating and may cause losing the interest of young generation to start their academic career.
• Access to up-to-date academic research through electronic databases should be ensured at all Azerbaijani higher education institutions.
• For internationalisation of higher education and in order to become part of international research community, the English language competences should be raised already at the level of secondary education. National curriculum should be reconsidered having stronger emphasis on acquiring English language skills.

ANNEX I

Component 4: Standards and Guidelines for QA in Higher Education

Activity 4.6: Analysis of the Pilots 5-9 June 2017

Summary of the key findings of the analysis of pilot evaluations

The Learning café workshop took place on June 7, 2017 in the Azerbaijan Technical Universities in four stations.

1. The topic of the first station was Self-evaluation and the Secretary and Rapporteur was Kati Isoaho.

1. The following themes of discussion were proposed to the pilot university representatives:

• Assess the Guidelines for Self-Evaluation document. What was useful and how could the document be improved? How do you consider the number and contents of claims included in each category? What kind of appendices would be most beneficial (e.g. statistics, organisation chart, other figures etc.) to be included in the self-evaluation report?
• Asses the process of conducting the Self-Evaluation: Was it useful? How was the cooperation with your colleagues? Did you learn anything new? How would you organize it next time? Would you need external trainers for it? Who should at least be involved in the self-evaluation process? How would you assess your university´s capacity to write evaluative text and how would you enhance/improve it?

2. The members of the evaluation groups were asked to discuss the following questions:

• Asses the Self-Evaluation documents as the guiding documents for the evaluation work: What worked well? What created most misunderstandings or difficulty? What was missing? How could the Guidelines for Self-Evaluation be improved?
• How would you develop the self-evaluation capacity of your Azerbaijani colleagues? For instance, is there need for increasing capacity in group work, stakeholder involvement, student involvement or evaluative and reflective writing, etc.

3. As a result of the discussions, three main issues were discussed: the practical side of conducting SER, the usefulness of the criteria in the Accreditation Manual and the quality of the SERs. As a result, the following key findings can be outlined:

• From a practical point of view, a lot universities felt that there were some challenges regarding the scheduling issues (which are common on institutional level) and reaching the relevant persons inside the institutions due to these scheduling issues – an examination period was taking place at the same time as the report-writing.
• It was also mentioned in the discussion that to carry out SER process, institutions need to build up a team of more than 10 people involved with someone in charge. It should be a team effort and never the responsibility of solely one person to take care of the process.
• One of the advantages of the SER process is that it creates a possibility for different chairs in the HEIs to get to know each other and each others’ activities better than before.
• One point of concern was that the use of the final report was not fully clear for all the university representatives.
• There were different opinions on the usefulness of the SER criteria: on the one hand, they cover all the key operations of the university, but on the other hand, they were not considered to be measurable enough – more figures and equals would have been needed.
• It was highlighted that because Azerbaijan cooperates internationally, the SER criteria should be applicable globally, not only in the European context.
• The quality of level and information provided in SERs was also discussed. The evaluation team members would have needed more descriptive information on the universities’ operations, and foreign experts were hoping for more information on the local HE system. Also, the general opinion seemed to be that there should have been more appendixes to the SERs – there was too much factual information in the report that made it challenging to read.
• There were some gaps in SERs, as some areas that were not discussed e.g: how the strategic plan was developed, what tools were used and who participated in the process, etc. Also, it was felt that the reports were always not analytical/informative enough. It has to be decided how to avoid gaps in the future – whether to provide more guidelines or to raise the capacity of the HEIs how to conduct SER on an institutional level.
• The local experts were commended by the foreign colleagues, as they served a necessary purpose of providing information on local context and Azerbaijani HE system.
• In conclusion, it can be said that the SER is crucial for the whole evaluation process. If the quality of SER is good, it makes the whole process successful. It is the task of ANO to stress this in the future and raise the capacity of HE institutions in this matter.

1. The topic of the second station was Criteria and Manual for Pilot Evaluations and the Secretary and Rapporteur was Kirsi Hiltunen.

1. The following discussion themes were proposed to the pilot university representatives:

• Assess the Manual for Pilot Evaluations document. What was useful and how could the document be improved?
• Assess the assessment areas and criteria: What worked well and what should be improved? Please, discuss with your group each criteria individually. Which areas were most difficult to self-assess? Where did the criteria overlap? Where were the biggest differences between the self-evaluation and the external evaluation? What would you leave out? Anything you would like to include?
• What other feedback would you like to give to develop Standards and Guidelines for Quality Assurance and future QA system in Azerbaijan?

2. The members of the evaluation groups were asked to discuss the following questions:

• Assess the functionality of the assessment areas and criteria: How could they be improved?
• Please, discuss with your group each criteria individually. Which areas were most difficult to assess? Where did the criteria overlap? Where were the biggest differences between the self-evaluation and the external evaluation? What would you leave out? Anything you would like to include?
• What other feedback would you like to give to develop Standards and Guidelines for Quality Assurance and future QA system in Azerbaijan?

3. As a result of the discussion, the following key findings can be outlined:

• The Manual for Pilot Evaluations was considered to be very good – systematic and providing good guidelines for self-evaluation. The manual provides an outline of how the learning environment and innovative practices could be applied in a university. The guidelines explain in detail how to develop management, strategic planning and research in a university. According to the university representatives, the manual outlined ways in which a quality assurance system could be established in a university. If was also found that the manual opens the way to interactive discussions both within university and with the external stakeholders. The manual also fosters the culture of reporting, establishing quality as an integral asset, and encourages communication between the students and the faculty.
• It was concluded there was not too much overlapping between different criteria – some was found in research and human resources (e.g, a question of involvement of staff in research projects and information about quality management system), but not much. The universities wouldn’t leave out anything from the criteria and believe that all of them are needed.
• It was also discussed which areas were the most difficult to self-assess/assess, and the following criteria were pointed out:
• the assessment area of management was mentioned twice;
• study programmes and their preparation (many people do not have skills yet to transfer from the traditional way to the new way);
• the prioritization of research areas and how research activities support teaching
• the human resources: it was not considered fit for purpose to have a requirement that selection procedures for administrative staff should be open for public (selected usually from amongst the graduates and university staff).
• the infrastructure (for the evaluation panel) because there was not enough data, i.e., the panel could only visit 5-10 classrooms.
• The biggest difference between self-evaluation and external evaluation was considered to be that the SER was more focused on quantitative aspects and the qualitative aspects were missing.

However, it was also mentioned that there were no so much differences between SER and the evaluation report. According to some HEIs, the experts seemed to have studied the activities so well ”as if they had had informants from the universities”.

• Also, a discussion took place wheteher something should be added to the criteria. Professional development of staff was mentioned as a separate assessment are, as well as cooperation between university and the industry and the academic ethics. It was also mentioned that there should be criteria in place how to assess study programme groups. The universities expressed an opinion that the sub-criteria included in the table should also be graded and the grading should be more measurable. They also found that the grading should be measurable (including percentage).The evaluation teams, in turn, considered there to be enough assessment areas but far too much sub-criteria.
• A question was raised by the teams wheter the data collected by MER could also be used in evaluations.

• The topic of the third station was Site-visit and the evaluation process and the Secretary and Rapporteur was Tiia Bach.

1. The following discussion themes were provided for the pilot university representatives:

• Assess the timetable of the evaluation: How could the timetable be improved?
• Assess the implementation of the site visit: What worked well and how could the site visit be improved?
• Assess the performance of the FINEEC/EKKA project manager: Where did she/he perform well? How could the activities of the project manager be improved?
• Assess the cooperation between FINEEC/EKKA and your university: What worked well and how could the cooperation be improved?
• Assess the work of the evaluation group: Where did it perform well, what could it have done better?
• Assess the whole audit process: What was the best for your university and what could be improved?

2. The members of the evaluation groups were asked to discuss the following themes:

• Assess the overall scheduling of the evaluation: How could the scheduling be improved?
• Assess the team’s preparation for the site visit: What worked well, and how could the preparation be improved?
• Assess the implementation of the site visit: What worked well, and how could the visit be improved?
• Evaluate the activities of the FINEEC/EKKA project manager: What worked well, and could the activities of the project manager be improved?
• Assess the training provided prior to the evaluation: What worked well, and how the training be improved?
• Assess the activities of your evaluation group: Where did the team succeed, what could have been done better?
• What kind of feedback would you like to give to FINEEC/EKKA in general? What did you learn from the evaluation process?

3. As a result of the discussion, the following key findings can be outlined:

• All the panel members were found to be well-prepared, professional and constructive.
• The university representatives found that the timetable for the visit was appropriate and there was a sufficient mount of time for the site-visit (three days). They had a clear overview before the visit how the visit would look like. However, some inferviewees were invited to the discussions at the last minute, which was challenging. All the important units in the HEIs were covered.
• The experts found that the preparation for the visit went very well and the tasks were clearly divided. However, more supportive documents should have been presented by the institutions.
• The experts appreciated that all the interviews were conducted in an open and transparent atmosphere. Regarding the site visit – more time was needed to see the facilities needed. This should be taken into account in the future.
• It was brouht out that there was too little time for some expert teams to prepare the evaluation reports. In the future, HEIs need more time to submit their comments to the report.
• According to the university representatives, the questions asked at the interviews were relevant, but sometimes not specific enough and not enough tailored to the profile of the university. Sometimes it was felt that the length of interviews should have been longer.
• The collaboration between universities and project management was good. HEIs appreciated the possibility to present addidional materials during the process.
• Regarding the whole process, it was mentioned that analysing all the assessment areas gave the universities an input for drafting study development plan, identify their weaknesses, etc.
• It was considered to be very important that more local experts are trained and the teams include labour market representatives in the future.

1. The topic of the fourth station was Report and the Secretary and Rapporteur was Touko Apajalahti.

1. The following themes were discussed with the pilot university representatives:

• Assess the content of the evaluation report: What was useful, what was unclear or unnecessary?
• How would you improve reporting?
• What would be the best way to disseminate the information of the pilots? Who should read the report?

2. The members of the evaluation groups discussed on the following topics:

• Assess the process of producing the evaluation report: What worked well, and how could the process be improved? You may also comment on the quality of the report.
• What would be the best way to disseminate the information of the pilots? Who should read the report?

3. As a result of the discussion, the following key findings can be outlined:

• Both the teams and universities were satisfied with the quality of the report. The HEIs were especially satisfied with the practical recommendations they can use in their development activities.
• The teams got good support from the chairs and Project Managers.
• To improve the process, there could be more guidelines for the team members to write the report (regarding the style, level of detail, special tools like Google Docs, etc.).
• It was stressed that enough time is needed for writing the report and the translation should be of high-quality to avoid any misundestandings. The responsible persons for each topic should also check the facts in the report.
• Regarding how to disseminate the results: both univesities and the panel members found that the reports should be also public in the future and published both on universities’ and ANO’s website.
• The report should be read by basically all stakeholders: the university management (who can make a change), student representatives, external partners (who can influence change), and everyone in the university (who should be committed to change.) Also, the Ministry, because sometimes system-wide changes are needed.