This is the summary of conversation between Asmae and Ricardo as part of the pre-workshop interaction. The responses depict views on issues related to evaluation of ICTs.
What constitutes credible evaluation and acceptable evidence in your context?
I find the question a bit vague. Evaluation should be conducted in a scientific, objective manner. It is meant to give us insight about how the project is advancing (or not advancing) in accomplishing its pre-defined objectives (outputs), and to learn what changes (outcomes) (at the organizational, social, regulatory, and policy levels) have taken place on the ground after the project implementation .
How do you currently do M&E? What methods, approaches, etc. are you required / do you choose to use?
E-Fez is an e-government project with 2 components: technology and action research. On the technology side, we have worked on building ICT outputs (a context-sensitive e-Gov platform) and their related on-site implementation & deployment. Action-research focuses on systematically assessing the project actions/outputs and their related changes/outcomes. Specifically, we have worked on identifying, tracking, and assessing indicators (or “progress markers”) before and after the project implementation (in effort to identify, highlight, and demonstrate results/changes (outcomes) produced/ generated by the project: automating citizen-centric service delivery). Outcome Mapping is our guiding approach. We combine qualitative and quantitative methods. Our methods rely on fieldwork and include, among other things, observation, in-depth interviews, focus-groups, and formal & informal on-site discussions. We use (quantitative) surveys in order to learn how citizens are experiencing/ perceiving/ assessing the newly deployed automated service delivery; and hence, in order to quantify some of our empirical results.
What are the strengths and weaknesses of the way you currently do M&E?
Doing M&E in itself is good. In Morocco ICT4D research projects are not yet a top-priority. When they do take place, they tend to be conducted with a technology only focus. However, in our e-Fez project we are combining ICT implementation with M&E, this is new in our Moroccan developing country context. We monitor on a systematic basis to track progress, to verify that we are on the right track. We use indicators to document the environment before and after deployment. Feedback is very important. A strength is that we have scientific and objective feedback to disseminate the initiative, and show policy makers that it is working, that is it feasible and to effectively communicate the project concrete/measurable benefits; in other words it is also a strategic tool.
Weakness: we need to find out if we are doing it right. We need to know best practices and others’ experiences.
How familiar you are with the following evaluation approaches?
GEM (Gender Evaluation Methodology)
Not familiar with it.
I know about gender, but not about a Gender Evaluation Methodology. This is very exciting. As part of the e-Fez project we will be launching an additional dimension: a wireless network between government offices, and the gender dimension is a priority. You want to be gender sensitive but a challenge is to translate it into specific actions. That is very important, it will facilitate the work
What is your level of understanding and experience conducting gender analysis?
I have a Bachelor Degree in International Development with focus on gender studies.
OM (Outcome Mapping)
I have attended workshops and I use it in my work. Attended an OM workshop in South Africa -Jan 2007 plus another in Beirut in April 2007 to further localize it. I was very pleased to meet Sarah Earl at the workshop!
If you are interested in using parts of OM, do you want to use it for ongoing monitoring during the project or for summative evaluation?
I have been using it since 2004 via IDRC project. IDRC said it would be hard to assess impact, hence focus on outcomes.
How would you differentiate between outputs and outcomes?
This was a big assignment in e-Fez project: a challenge to differentiate, in ICT projects often the technical output is often the focus, but for us we go beyond that to outcomes: how is the system generating changes? Outcomes are changes directly linked to outputs.
MSC (Most Significant Change)
Not familiar at all.
What other approaches do you use?
The focus is on OM and we use surveys to quantify.
How much attention do you want us to give to LogFrames?
Provide brief review and reference materials. I am not very familiar and therefore it would be nice to learn about it. In Cape Town there was a lot of debate and waste of time discussing this!
How much attention do you want us to give to ICT indicators?
Provide brief review and reference materials. This is maybe more important than LFA, we can use the Wiki after the workshop.
Preference for the reference materials to take home:
Both hard copies and electronic.
Provide your three main expectations for this workshop.
- To evaluate ICT projects – this is an opportunity to meet people with the same interests.
- To learn about GEM and MSC methodologies and receive reference materials.
- To internalize these approach and learn to effectively evaluate our project situation.