0.9 Dr. Concha: Hi everyone and welcome to the ElevateEvaluate podcast. I am your host Maritza Concha and I'm super excited to be here with you. My colleague and good friend Maria Elena Villar is here to talk about the episodes that were submitted by my graduate students. How you doing Maria Elena:

​

0.24 Dr. Villar: I'm doing well, thank you for inviting me again. I really enjoyed listening to the episodes.

​

0.31 Dr. Concha: Great, for every semester I had this idea of submitting three or four episodes. The best episodes after each semester right, but I could not help at this time. I had about 9 excellent episodes and I have to be honest with you for a six-week class, my graduate students exceeded all my expectations.

​

.054 Dr. Villar: I was also very pleased. I really liked the diversity of topics and the different approaches that they took to explaining the concepts and telling the story and linking it to real life. I really think that was one of the strengths of the series this time.

​

1.10 Dr. Concha: I think so and they also have some experience working in the nonprofit sector, so it also that combination that helped to illustrate their point they were trying to make. So this is great!  We have nine episodes organized in general topics. These topics are the role of stakeholders improving evaluation program evaluation, general techniques in service delivery, cultural competency evaluation, an ethics. So, let's talk more about each of these. Let’s begin with the role of stakeholders’ involvement. What I notice was that my students were able to understand what stakeholders are, who they are, how important their input is, in supporting evaluation. So one question that I have, Maria Elena is what has been your experience in working with stakeholders in evaluation?  What are some of the successes or the challenges you encountered during your career?

​

2.10 Dr. Villar: I think it's impossible to overstate how important it is. Not to do evaluation in isolation because you know they were there. If there's a grant or a logic model, there are specific outcomes. An if you only look at measuring those outcomes, \you're probably missing a big part of the picture if you don't involve the people that were part of making those outcomes come to fruition. So, the staff, the clients, just really understanding the context is such an important part of evaluation. We've had many experiences where we go in with an evaluation plan, and after speaking to the stakeholders, you realize that's the wrong approach. That's not the right way to reach the population. These kinds of questions could be considered threatening. Or even just misunderstanding how the services were delivered or part of the context. So, an evaluator should never just go in with their own tools without first consulting everyone else.

​

3.22 Dr. Concha:  I agree completely with you. I think one of the lessons learned for me is that if it we want to collect a survey or doing interviews, how you're able to write appropriate questions and make sure that the content is being respectful for the community itself and try to understand who are our participants.  If we don't get the buy in from the stakeholders, if you do not talk to them, how are we going to know? It is a difficult process, but it's a necessary one to ensure that we are making things right and correct.

​

4.03 Dr. Villar: I was happy to see that the concept that value of respect for the recipients of services throughout that which  is relevant to cultural competence to ethics and to just putting together in evaluation tools and bringing together stakeholders. It was a theme that went through all of the episodes

​

4.25 Dr. Concha: I think, so definitely, and so the second topic has to do with general techniques and program evaluation. Just importance of doing program evaluation in service delivery. So, one of the topics that was addressed in one of the episodes have to do with passion that you have to have. You need to have passion because what you are doing as an evaluator is to become an agent for social change. But how difficult can it be to be passionate and ethical at the same time? That is one of the things I always struggled with. I don’t know if you have found the same issues.

​

4.58 Dr. Villar:  You know the situation, an ethical and objective, right? And because the rationale for bringing in external evaluation evaluators usually is the expectation that if you're part of the program, you're going to be naturally biased towards thinking it's a great program, right? Because you're part of it. I agree with the student. That said, it helps to be passionate in the larger sense in the sense of the big picture. These kinds of community services are going to work or so important to help people reach their full potential, and they're going to work to the extent that they achieve what they're trying to achieve that they are run in an effective and efficient manner so that they can continue right and. And to me it does help because you know evaluation can also be tedious and boring, and there's a you know, kind of lonely part to it. But but being passionate about that big picture and why doing good program evaluation that provides actionable feedback for the programs to improve and to demonstrate to funders that good work is being done, I find that passion is helpful.

​

6.16 Dr. Concha: I agree with you and sometimes you even doing things differently as an evaluator trying to break the cycle of traditional evaluation techniques and skills like even if you're doing the analysis, you can take a look at that data with different eyes, and so you can get information that other people cannot see, and that's because you're passionate about it and you're still being ethical, so I think that's where is really important the  fact that you have to be passionate about. And the other thing that they it was mentioned in one of the episodes is how do we write evaluation questions? The importance of diction when is technical too technical, right? So what criteria we need to take into consideration when we write evaluation questions?

​

7.04 Dr. Villar: That was a very interesting episode for me. It was very well researched and provided great examples of how wording does the word of choice in your data collection tools can make a big difference if somebody is reading something that's way above their comfort level in terms of literacy level or uses you start reading something with a lot of words that you don't understand, you immediately lose interest and feel and can feel inadequate like this is not for you and disengage, right? So it's. It's extremely important and that ties into the stakeholders and also to the cultural competence. Making sure you get you get input from folks. I mean a very common scale used, you know Likert type scale that's used in questionnaires for some populations. That's not a very easy thing to complete. I mean what's the difference between strongly agree and just agree? And if you're not used to thinking of things on a scale of 1 to 5, suddenly putting a paper in front of you that has that isn't going to be helpful. So you kind of have to understand what what's the best way to get the feedback that you need from the people that you're trying to approach.

​

8.23 Dr. Concha: And I think that for me, as an evaluator, one of the things that I notice is that sometimes it's difficult to even distinguish between the types of evaluation questions you ask between process and outcome evaluation questions or cost effective evaluations or related to needs assessment. So those things can be sometimes problematic because they're too technical or to vague, and you don't have criteria or some sense of measurement in the question to help you measure what you're supposed to be measuring, so that's when the technicality  comes to being an evaluator and how do you communicate. Because communication is key here with the stakeholders and with the community in general in terms of other seeing or justifying the evaluation questions that you have asked. So I think that was also very interesting episode.

​

9.18 Dr. Villar: I have any experiences of having to change. A question or completely delete it. Reluctantly, but because I'm convinced that it's not going to end up being valid because it's not coming across the way I intended, right?

​

9.37 Dr. Concha: Yes, you learn as you grow, I guess through experience. So, the next topic is cultural competency evaluation. So, and this is very, very important in specifically in the current times that we're living in right now.  How do we advocate for social justice? When doing program evaluation right and some topics addressed this issue about understanding your program using cultural competence lessons, understanding your target audience, and even your staff. So,  I know you have a very good input here Maria Elena from one of the episodes that you want to share.

10:16 Dr. Villar: Well, I was commenting to you before we started talking that. I thought that there were a couple of episodes that really illustrated cultural competence and what it means in practice through anecdotes, right? So one of the one of the episodes described an evaluation situation where young students were asked to a Deans office and just the cultural factors related to place and identity and power, right? Where it has nothing to do with race and ethnicity, which is usually what most people think of when they think of state culture and just the way that. They told that story. Really helped to illustrate that aspect of Culture. And also the episode about LGBTQ+ cultural competence, besides giving a lot of great tips and advice on how you can try to be more culturally competent on these issues. I also spoke about a personal example of how they became by observation and by interest, and by finding out things on their own. Knowledgeable about a culture that was different from theirs and encouraging listeners to do the same with LGBTQ people and culture. So, I just thought that using those anecdotes was a very powerful way to explain the concept.

​

11:54 Dr. Concha: Definitely, I think that's one of the key lessons here that the students use their own

anecdotes or the things that have happened through work experiences as well may help understand a little bit more well how to apply the topic right on cultural competence.  And in my particular experience-I mean, this is something that you also know that we both have been working with the Latino community for several years now.  At least I started with the farmworkers population and doing evaluations for domestic violence prevention services in the area of homestead.  Most of the work that I have done in my in the beginning of my career has to focus with the farmworkers population and it was so difficult for me to even if we translated a tool in Spanish may be challenging.  How do you approach these population? Some people have low literacy levels, so even if I use very simple Spanish words, they still feel not engaged It may seem that I'm really not collaborative, they don't want to collaborate with me because they didn't feel that they trust me or they didn't so comfortable. So couple of things that I learned throughout time that even you know, translating a document or evaluation tool into the primary language of the target population will not work - that you also need to understand their values and attitudes and behaviors and which is key to get their buy in and to get information that you're looking for. The ultimate purpose of obviously is helping them out.

​

13:38 Dr. Villar: And that example ties together with the stakeholders, right? because you know you not being a farm worker doesn't mean you can't evaluate a farmworker project, but. Having other farm workers involved in the process and really talking to stakeholders and maybe cultural brokers that will introduce kind of what you're trying to do to folks will increase that trust, and that is crucial. I mean, I think everybody, no matter what their situation is. If you don't trust why someone is asking you questions, you're not going to answer them honestly,

 

14:15 Dr. Concha: Exactly exactly, and then the last topic has to do with ethics in evaluation, so it was very interesting to see some of my students’ work.  Some decided to compare the ethical guidelines from the American Evaluation Association and the Canadian Evaluations Association, and they were able to see how they different from each other depending on the cultural aspects as well at the country Level I will say.  Also some other students they talk about in the importance of using informed consents. The issue of confidentiality and safety. So here is my question that I have when we talk about ethics, where do we draw the line between research and evaluation when it comes to ethics? Is it clear? Or is a line that does not exist and we talking about the same thing? What do you think about it?

​

15:05 Dr. Villar: What an interesting question. Um? I think that as evaluators we should try to use the same rigorous standards of ethics as we are mandated to use for research when we do evaluation with the understanding that sometimes when things are taken to the field and you know they can be adapted and that perhaps not everything needs to be the same rules of IRB, for example, apply. But the principles of respect for persons informed consent, minimizing harm increasing benefit. Those things should be present at all times and respecting people, and so you know. And that includes paying people for their time. If you're asking them for a lot to spend a lot of time on your evaluation, the program staff are getting paid. The evaluators are getting paid. You know why not the participants that are. Taking their time right? Yeah, you know. And things not causing harm. Obviously. Not asking things that could incriminate them or put them in any bad light, but even. Things like one of the episodes was about evaluating domestic violence. If your programs, if you're talking to survivors of domestic violence, you know it might be great to hear their story, but do you really need to kind of re traumatize them or put them in an uncomfortable situation again for only for purposes of evaluation, right, right? So minimum, you know minimizing harm, making sure that that the burden on the respondents is minimal, and then of course, the ethical. The kind of research ethics of it, of being honest and reporting accurately, and, being transparent about limitations or errors right now, I feel very strongly about ethics, and I think there's very few things that are ethical dilemmas. I think most things, if you really look at them, there's an ethical way to do it. And then and then the other way to do it. So that's why it's important to keep talking about it.

​

17:20 Dr. Concha: Yeah, definitely. And I think particularly attention must be paid with sensitive or vulnerable populations. That's where we need to be very, very careful with everybody but with particular populations such as  domestic violence victims and also victims of human trafficking, for instance, to make sure that we are not re traumatizing them and for the purpose of getting information that will meet our needs as evaluators or evaluation questions that we are trying to answer, so that's one of the key issues that I think I'm taking away.

​

17:46 Dr. Concha: Thank you Maria Elena for your time to talk about the reflections that we just did on my students work, and I'm very thankful for having such a good group of students, so this is just a notification for future students or potential students. If you're interested in taking a class with me and program evaluations, you have to register to any of the program evaluation classes that you see if I teach undergraduate and graduate students at UCF. Another thing that I want to share with all of you, is that starting  in the fall,  I will be interviewing program evaluation consultants to ask them about what recommendations they have for students who may be thinking about program evaluation as a career. So, what are their success stories and also their failures? Because I think you learn most of the time from your failures, not your successes. And please do not forget to subscribe to the ElevateEvaluate podcast. And again you can find it at the Apple podcast or Google podcasts or Spotify. Anything else, Maria Elena:

 

19:00 Dr. Villar: Thank you so much for having me and for finding such a great way to talk about evaluation.

 

19:07 Dr. Concha: You're welcome, and this is exciting. Thank you so much. Keep in touch,