7.3.3Recommendations for the Flemish Government

KPIs and output parameters must be set in a smart way. The government must formulate clear objectives and KPIs need to be linked to these objectives. VARIO formulates recommendations on how to achieve this.

Recommendation 1: Apply the proposed conceptual framework for setting up KPIs

A conceptual framework for setting up KPIs has already partly been drawn up in Flanders, but there are important gaps and coherence is lacking. The first important recommendation, as part of the measures to remedy this, is to apply the conceptual framework for setting up KPIs in function of policy objectives proposed by VARIO (Chapter 2).

Recommendation 2: Translate the long-term strategic objectives in clear objectives for the EWI policy domain

VARIO notes that the feedback between the strategic long-term objectives and objectives per policy domain is insufficient (steps 2 and 5 in the conceptual framework).

The policy objectives for economy, science and innovation are not always clearly defined and the link with the overarching long-term objectives is not clearly articulated. However, VARIO considers it important that the overarching long-term strategic objectives are translated into clear and concrete objectives for the EWI policy domain. The following principles should be applied:

  • Transversal and crossdomain thinking. The translation of long-term strategic objectives into objectives per policy domain must not and cannot be a linear one-to-one relationship. Several policy domains contribute - in different ways - to certain long-term objectives. Science and innovation can contribute pre-eminently to objectives in many other policy domains.
  • Tuning and synchronizing. The objectives formulated at the level of the various policy domains must not contain any contradictions.
  • The whole must form a dynamic process in which there is interaction between the policy objectives of the Flemish Government on the one hand and those of the different policy domains on the other. 

Recommendation 3: Make an extensive exercise for the EWI policy domain – how do instruments/ structures/actors/programs fit in with the objectives set?

It is important to clearly identify the individual objectives of the different instruments/structures/ actors/programmes within the policy domain EWI and how they contribute together, at systemic level, to the objectives of the entire policy domain (steps 3 and 4 in the conceptual framework). When designing new initiatives, it is important to consider how they will contribute to the objectives of the policy domain and how they will interact with existing initiatives.

VARIO notes that such exercises have not yet been carried out in a structured manner in Flanders. Therefore, VARIO advises to conduct a thorough analysis of the existing initiatives and actors, including the following:

  • The (priority) objectives of each individual instrument/structure/actor/programme.
  • The objectives of the policy domain to which they must contribute (as a matter of priority)
  • An assessment of how they reinforce or counteract each other's objectives.

Based on the results of this analysis, the existing system can then be streamlined.

Recommendation 4: Link KPIs to the objectives at the different levels

It is important that the right KPIs are linked to the objectives, at the long-term strategic level (step 1), at the level of the policy domain (step 2) and at the level of the individual structures/actors/ instruments/programmes (step 3). Objectives and KPIs should be attached to new initiatives when they are launched. Transparency on the KPIs is needed at all levels. KPIs should be complementary and create synergies.

Recommendation 4.1: Draw up high-quality KPIs for the policy domain EWI

VARIO notes that although there is frequent monitoring of activities in the EWI policy domain (see VRWI indicator set and Flemish Indicatorbook), no clear KPIs, including targets, are linked to the policy objectives for science and innovation (step 2). VARIO therefore recommends that, in addition to clear objectives, the corresponding KPIs for the EWI policy domain should also be formulated clearly. These should have a longer-term perspective and be more impact-oriented than the KPIs at a lower level (the individual instruments/structures/actors/programmes). For this purpose, one can look for example at the EU's 'Research and Innovation Observatory' (RIO) includes (macro) impact indicators such as added value for services broken down by knowledge intensity, employment in knowledge-intensive activities as a percentage of total employment...

Recommendation 4.2: Establish high-quality KPIs for the EWI instruments/structures/actors/ programmes

At the level of individual instruments/structures/actors/programmes, KPIs and output indicators are common and are mostly directly linked to funding (step 3). At this level, it is normal to focus on short term (output) and medium term (outcome). It is more difficult to link KPIs to impact, as there are many factors that are beyond the control of the initiatives themselves, and isolating their specific impact from that of other sources of funding and initiatives is often difficult. VARIO believes that impact is very important (cf. recommendation 5.2), but asks to be cautious about impact indicators at this level. They are not very common at the moment, but for example the cluster pacts do include impact indicators. Especially for such indicators it is essential to remain alert for as to whether the correct measurement is being made and to adjust them if necessary.

Recommendation 4.3: Look beyond indicators, output parameters and other quantitative data

Quality aspects are already (partly) included in the KPIs and output parameters. In addition, VARIO believes that we should not restrict ourselves to quantifiable information, and that more qualitative data should be included, certainly for impact analyses (see also recommendation 5.2), e.g. on the basis of surveys and (in-depth) interviews. Wide surveys, outside of the initiatives to which the KPIs and output parameters apply, can also help to detect the undesirable side effects of KPIs and output parameters.

Recommendation 5: Assess at appropriate times whether objectives have been achieved

At appropriate times one needs to monitor to what extent objectives have been achieved. It should be checked whether and to what extent the existing KPIs contribute to (not) achieving the objectives (steps 4 and 5). It is important to remember that KPIs are not a goal in themselves but a tool for realizing higher goals. This evaluation, both ex-post and ex-ante, should be carried out at different levels:

  1. To what extent are the long-term objectives of the Flemish Government achieved?
    • How do the various policy domains jointly contribute to this?
  2. To what extent are the objectives of the EWI policy domain achieved?
    • How do the various instruments/structures/actors/programmes jointly contribute to this?

The focus of recommendations 5.1 and 5.2 below is at the level of the EWI policy domain. The KPIs need to be adapted if the objectives set are not achieved, or if the KPIs of one objective counteracts the achievement of another. For this purpose, the government must make use of data analysis and monitoring systems, which are becoming increasingly effective (see recommendation 6).

Recommendation 5.1: Need for more systemic evaluations

VARIO is in favour of adjusting the current, strongly programme- and structure-driven evaluation culture towards more and broader system evaluations. VARIO strongly recommends including how the initiatives jointly contribute to the achievement of objectives. An evaluation of portfolios of instruments/structures/actors/programmes within the EWI policy domain and their interrelationships should be carried out on a more regular basis. These can provide insights that are relevant for future policy decisions regarding the EWI policy mix.

Recommendation 5.2: Need for more standardised and systemic impact analyses

In order to gain a better understanding of how the tools/structures/actors/programmes contribute to the stated objectives of the EWI policy domain, it is necessary to look beyond their individual outputs (short term) and results (medium term), and to carry out impact analyses. VARIO recommends that more emphasis be placed on impact analyses and on standardising their methodology, so that they can be better compared and interpreted. In addition, VARIO advocates that impact analyses should be more systematic. What is the combined impact of the various initiatives and to what extent do they allow to achieve the objectives formulated at policy domain level?

Recommendation 6: Develop monitoring tools further

In order to carry out evaluations and impact analyses, the necessary data must be available. That is why monitoring, and therefore the creation and measurement of good indicators, is extremely important.

At present, monitoring is still often carried out at output level. In order to monitor the long-term strategies of policy, a longer-term perspective must also be included in the monitoring. Thus, it is necessary to monitor not only outputs but also results and impacts.

There is currently a problem in the accessibility of existing data. VARIO considers it important to improve and to simplify access to (administrative) data for the evaluation of EWI policy. In its memorandum 2019-2024, VARIO already called for an open data platform where the necessary data for monitoring and evaluation should be made available to policy makers.

  • Although data are regularly collected, they are often not used on an aggregated level for analysis, the reason being a problem of linking different information sources. This can be overcome by a better (IT) data infrastructure. VARIO recommends to work on this and sees a role for artificial intelligence in making maximum use of the available data.
  • The extent to which and how FRIS (Flanders Research Information Space) can contribute to the evaluation of the objectives of the EWI policy domain and its instruments/structures/ actors/programmes must be examined. The use of the FRIS portal for this purpose is in any case in line with the objectives of this initiative. The possibility of linking additional data sources to the FRIS portal should also be examined.
  • It should be examined how anonymised data can be made available for analysis and evaluation without compromising the confidentiality of (sensitive) data.

Recommendation 7: Make more in-house capacity for policy preparation and evaluation available

As in its memorandum 2019-2024, VARIO advocates that the public administration itself should have enough substantive expertise. In-house knowledge and capacity building are crucial to correctly interpret (monitoring) data and evaluation reports provided by external parties and to translate them into policy. Knowledge, and especially continuity in knowledge, is often at the level of administration, and this needs to be called upon more frequently by policy makers.