Monday
September 25, 2017
Quality Matters Webinar
Operational Quality Assurance ‑ Ensuring Quality Across an Organization
	
	CART Disclaimer:  This rough edit transcript, which may contain missing, misspelled or paraphrased words, is only provided for your immediate review and is not certified as verbatim and is not to be cited in any way. 
	
	[Captioner standing by] 
	
	>> All right, guys, and then some people [audio fading] [indiscernible], okay?  How about instruction?  And for those of you that are on the webinar, oops, for those of you that are on the webinar that could not hear anything I was just saying because I wasn't in front of the microphone, Patrick Intravichit, who is our Manager for the Virtual School is manning the chat box and there to support you in any discussion.  
	Again, my name is Crystal Guiler and I'm the National Curriculum Manager at Florida Virtual School for those of you who could not hear me on the Zoom meeting.  
	Okay so our agenda for today, we're going to be looking at decisions in leadership and some things that, you know, might be a little bit benign, which is how they are made, how they are done, and how they relate to the mission, but I have a reason, so hang in there with me.  We are going to talk about who and what we ultimately serve.  Who are our customers when creating courses or programs or developing those programs for Online Education?  And then what is our product?  And how do we serve that product when it comes to Quality Assurance as well as Quality Matters standards associated?  We're going to talk a little bit about decision mapping, so how you can map your mission, your vision, and your commitment for your institution or organization back down to a statement which then maps to Quality Matters standards to best support decision‑making and data gathering.  And then we're going to talk about the proof of concept and how even though it's something that has kind of gotten lost a little bit as Online Education has been moving forward in full development, it has not gotten lost in IT a lot of the time, but we'll talk a little bit about that and why it's so important. 
	Okay, so decisions in leadership. I love this little cartoon. So what can we do about the leadership course?  I don't know. What do you think?  And isn't there anyone we could ask?  [Laughter]. So a lot of the time when we're making decisions or supporting decisions with leadership and in an educational institution, we are bringing challenges to the table, right?  And we are bringing challenges to best support students.  But sometimes it's really hard for that person to make an informed decision that is in turn going to support the students if we have not really thought through how does it impact our mission, our customer, the student, and how will it impact faculty and support resources?  
	So thinking about decision‑making. So we have a lot of different components that go into decision‑making for leadership.  We have, of course, the monetary. You know, what's our budget?  And can we afford it?  We have the time it's going to take and whether we're somehow limited on that time, if there's some kind of deadline.  And of course, we have the people.  Who do we need to accomplish it?  Can anyone else think of any other areas which might impact leadership decisions?  Things that leaders ‑‑ my leaders in the room, what else do you need to think about when you are making a decision that impacts your customer base, your students, and your faculty?  
	>> Clear, strategic vision. 
	>> CRYSTAL GUILER: Clear, strategic vision. 
	>> [Off microphone] 
	>> CRYSTAL GUILER: Exactly.  Does it line up with where you want to be in three months, six months in serving your customers?  Anyone else?  
	>> [Off microphone]. 
	>> CRYSTAL GUILER: I'm sorry?  
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Federal and state guidelines, the law. Accessibility. [Laughter]. Absolutely. Anyone else?  
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Absolutely. That's a lot of them. Jim has the chat back there, by the way. He just came up with all of those himself. [Laughter]. Okay, wonderful. 
	Okay, so the thing that I truly believe is it all comes down to our vision, our mission, and our values, right?  How does it impact our strategic plan?  Of course, all of those components are necessary in good decision making, but that's where we start. So whether you are making the decisions for your organization or whether you are making the decision to feed the decision to somebody who will ultimately be making that decision, if it does not align up to your strategic goals, if it does not align up to your vision and you cannot show that, then you cannot guarantee quality in that decision.  You cannot guarantee you are going to get the results you want. And a lot of the times we don't even know the result we want, we just know we need to solve a problem. 
	So and I'll get to Quality Matters here in just a second, but first, after yourself the questions when you have a problem that you're trying to solve. What is our goal?  What are we trying to accomplish here?  Trying to solve a problem, trying to bring strides into the future when it comes to impacts students. How does it impact your mission/vision/and values?  And who does it serve?  Does the decision serve the right people?  These are the three questions that if somebody has not asked themselves this and can answer them, I send them back to the drawing board when they bring me a challenge they want to talk about. 
	So what does this have to do with Quality Assurance?  Everything. It has everything to do with Quality Assurance.  The way we define our quality is completely within our strategic alignment for our strategic goals for organization. It is absolutely living within our vision.  The way we make decisions lives within our vision if we have a good vision statement. 
	So let's take a look at the definition of quality. So I like to move around when I present, but I have to stay with this, so forgive me if you see me going like this [laughter] over and over.  So defining qualify, of course, by Google, is the standard of something measured against something else, but defining quality for an educational institution is not the same thing.  Defining quality for an educational institution is not just about your product, it's about your customer.  It's about who is actually going to benefit from this work. So let's take a look at Florida Virtual for a second. So Florida Virtual, our mission is to provide a high quality technology‑based education that provides the skills and knowledge students need for success.  Our vision is to transform education worldwide one student at a time.  And our commitment is the student is the center of every decision we make.  That sounds like any mission statement. We sat down and figured out where we want to be in a few years, figured out our customer base, but all of this drives what we consider quality and what we consider quality and what you consider quality, of course serving students is going to be similar, but it's not the exact same thing.  It's what it means to us.  It what it means to our organization, exactly how we function, the different instructional models that we have, and the different staff that we employ.  So when we really take the Florida Virtual mission, vision, and commitment and we take out the key words, quality as defined by the FLVS mission is to ensure that all decisions are made with the student at the center. So go back to our commitment, if I ever make a decision and I cannot tell you it's in the best interest of students, take everything else out of the equation, if I cannot answer that question, then I am not matching our organizational commitment. So how am I going to bring that to the board [laughter] and tell them I'd like to do this if I'm not matching that commitment?  They have to have some kind of impact on transforming education. That's our strategy. Where do we want to be?  Do we want to make a difference or just solve a problem?  Can we do both?  And last but not least, that they lead the student to success. That we provide them with what they need in order to be successful.  So with our definition of quality, I want to take you through a little journey, okay?  And that little journey was figuring out how we could implement Quality Assurance in multiple departments within our organization to best serve our mission, our vision, and our commitment. 
	So what we figured out is the Quality Matters standards can help.  They can. You know, you would think these are standards to develop a course, right?  But it's more than that. These are standards to ensure quality, to make sure that we're meeting the standards this are necessary as deemed by the experts in the field, as deemed by all of the people with the Standing Committee and ensuring that all departments are ensuring this quality that we need.  
	So going through ‑‑ I'm having clicking challenges, okay.  So going through our different departments, so we have our instructional department. We employ over 1300 teachers in the State of Florida. And these teachers, which we employ are delivering content that needs to meet these Quality Matters standards. Well, how do we know they are?  And how do we know what they need in order to do that successfully?  So what that comes down to is we created something called Quality Matters Instructors.  And I'm sure you have something similar in each of your institutions, somebody who goes through the courses, looks at the data, looks at the feedback, generates opportunities for professional development, for improvement, and I'll go through this a little bit more. 
	So why do we have QAI?  Back‑to‑back quality statement. Once you have your quality statement, you can justify anything. You can justify exactly why you need it as long as it feeds up to that quality statement. So guaranteed review and assurances of Quality Matters standards.  Those Quality Matters standards are front and center on rubrics when reviewing student/teacher interactions within our courses. Data to drive students’ achievement and policy decisions. So how do we want to spend that PD money?  Do we want to spend it on feedback?  Do we want to spend it on enhancing collaboration?  Do we want to spend it on both?  And do we need to?  Because we need the data in order to tell us and these people give us that data. And quantitative and qualitative data to inform performance reviews.  
	As for course design which takes up the majority of the Quality Matters standards for us, we actually have two different positions here. So one person is not responsible for everything. And there's always a second set of eyes. We have our Quality Assurance Analyst who ensure that it is accessible and usable. These Quality Assurance Analysts are not content people, although they are educated and probably could, you know, make sure that the learning objectives align, they are not content people. They are there to make sure that the content is accessible and usable based on not only the Quality Matters standards here but Accessibility Guidelines and 508. 
	And then we have the curriculum specialist and these are a mix between your [indiscernible] and Instructional Designers making sure that the learning objectives align and all of the other associated standards.  
	Why do we have them?  First of all, we have to guarantee we are building the courses appropriately for students.  We have to guarantee we are meeting quality expectations and we need to make sure we are doing it not because of rubric says we have to or because we want to be the best out there, we want to do it because our commitment is to our students. So once we drive back up to that quality statement, we make sure we have these positions in place for that reason, then we're able to feed into our mission and, therefore, we are accomplishing what we set out to do in serving students.  
	Okay, so last we have our IT people, right?  Everyone has an IT department no matter how big or how small, you have somebody who is responsible for the Learning Management System, someone who is responsible for the technology within the course and how we're using it, and of course, they are responsible for certain standards as well. Is why do we have them?  Again, guaranteed review and assurances of QM standards, but also the opportunity to gather realtime data on student use of technology barriers and solutions.  
	So how does this all come together?  Why am I telling you all about this and giving you Florida Virtual School as an example?  And that is that no one can make good decisions when it comes to especially education without a few things in mind.  Number one, Quality Assurance. Okay?  So quality does not just live in one department. We all own Quality Assurance in within way, shape or form.  Your Instructor own the quality. Your IT staff owns the quality.  Your Instructional Designers own the quality. Your testers own the quality.  Peer Reviewers own the quality.  So in the end, if you don't have one defined quality statement, how are all of those people guaranteeing quality?  If we cannot define quality as an organization at the high level so everyone knows exactly what they are saying when they say that something is quality and it's owned by everyone, then we don't have quality.  And the QM standards will help you drive that.  They will help you get that buy‑in. They will help you lead up to that strategic vision.  So a little simple example, so say you have $25,000 to spend. So what are you going to spend it on?  You know you ‑‑ it's in instruction, so that money is not going to making a new course. That money is not going to, you know, buying this new little widget in the LMS or anything like that. This money is absolutely going to instruction. So the first thing people think about is, oh, well, we'll get teacher PD, right?  Okay. You can do that. Maybe we can lower the student loads. You know, maybe put a few less kids in each person's class with that money. Maybe we don't know what we need and why don't we know what we need?  Because we don't have data. We can ‑‑ think for a moment, just think for a quick moment and if you do have someone who is next to you, feel free to pass and talk, think about a problem that you know exists, that you know needs to be solved in your organization at this exact moment, okay?  No matter how big or how small that problem is think about it. And then I want you to think, what is our mission?  And do I have the data to show this is really a problem?  Okay?  And if you do think you have the data, call me over because I want to talk to you. Okay?  So just take just a couple minutes, talk about your biggest challenge, and talk about if you don't have the data how you can get the data and what data you need and how does it match up with your mission and your strategic goals. Take just a couple minutes.  
	And Pat, if you could support in the chat there. 
	>> CRYSTAL GUILER: Can I give you all about, probably, another minute to wrap up?  And then we're going to talk a little bit more.  
	Okay, well, I'm glad you all had the opportunity to have some really great conversations.  I heard some really good conversations about, you know, kind of looking at some challenges and determining, wait, what is the data we need?  How do we even define that data?  And how can we possibly be able to make a decision and know what the outcome is going to be if we can't even define how we would set the stage to reach that outcome with the decision?  So do we have any brave souls who, and you have to be really brave because the microphone is here for the people online, so do we have any brave souls that would be willing to kind of share their conversation?  Please.  You have to be really brave. Outgoing. Extroverts in the room. Anyone?  What about anyone online?  Would somebody be willing to make the mic and if they're not ‑‑ can we hear them, Jim?  
	>> [Off microphone]. 
	>> CRYSTAL GUILER: [Laughter]. Okay, how about somebody give me an example from the audience and we'll talk through it. 
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Okay. I'll ask you. 
	>> [Off microphone]. 
	>> CRYSTAL GUILER: So let's talk through that one. So more E‑tech individuals to come online, make sure they engage with the students, I think I heard within the first few days, to ensure student success.  How do we know that will ensure student success?  How do we define that?  I'm not challenging the question, I'm challenging the way we get there. And that's different for every single one of us.  And that is what is so important here is that every single one of us are going to have a little bit different online student focused definition of quality for our organization.  And when we have that definition of quality, then we're going to be able to answer those questions.  We cannot ensure student success if we don't know how we're going to get there. And the way we get there is we need to know if we have this person, do we have data?  Maybe we do. Do we have data on teachers checking in with students within the first few days versus not?  And whether or not the completion rates were better. Or on time work assignments were better. We can gather data, but it's a matter of backtracking and not getting ahead of ourselves because if so, then what do we have?  We have a whole bunch of people doing something and in the might we be scratching our heads and saying, why aren't the students successful?  [Laughter]. Go ahead.  
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Mm‑hmm. 
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Right. So you have plenty of data, but you don't necessarily have the right data. 
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Sure. Yeah, absolutely. So what she is saying is that they have the resources and they have plenty of data from the Learning Management System, correct me if I say anything off, but what the data is giving them is the number of clicks, which doesn't necessarily give you information on student and faculty engagement. And the engagement factor and the actual conversation which is taking place and the work that's taking place and the meaningful aspect is missing. Right?  Because what you are getting from the LMS is you are getting quantitative data. You are not getting qualitative data and qualitative data is really hard.  So one thing that you might do is embed, and these are just ideas, right, and anyone here could come up with plenty and some might work for you or not, but maybe embed some qualitative service. 
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Mm‑hmm. 
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Yeah. 
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Right. 
	>> [Off microphone]. 
	>> CRYSTAL GUILER: Yeah and so what she was saying was [laughter] so the, so the thing is that they have the survey data. They have the qualitative data, but they are not necessarily getting the results that they need from it. The student loves it or they hate it, they are maybe not detailed data in there. I think we all strive for that. You know, how can we get there?  And if we're not asking the questions and having the tough conversations, we never will. Okay, great. 
	Well, I'm going to go ahead and move on to the next piece here.  So I left this one on here just to beat a dead horse.  Data, we need data. It doesn't matter what your question is or how you think you're going to solve if you don't have the data, you can't do it. So how can we get that data?  Right?  What can we do to get the data?  And that's the big question.  And we don't always have the perfect answer, but that brings me to investing. Invest in the data. If you have $25,000 to spend on instruction, maybe ask yourself, what data do I need to know what instruction needs?  How much is that going to cost?  How am I going to get the data?  When I have the data, next time I have a pot of money, I can serve our program, I can make sure we are meeting Quality Matters standards to a tee. 
	So I put a question on here. Do you need more roles?  This doesn't always mean more people, right?  Sometimes that can just mean that somebody is going to wear another hat. Peer Reviewer, that sort of thing.  Do you need a review process in place?  You know, what is your process?  Are other people looking at it?  Or do you need documentation?  Once you have those things in place, that's where the data comes from.  I mean, you can put a quality analyst in the class room, but if they don't have a rubric, how are they going to get the data?  Right?  Sometimes you have to start from the beginning. 
	So don't get caught up in the tennis match. Decision‑making with quality in mind. Quick decisions can only be made with data. A long‑term decision cannot. Let's talk about what a long‑term decision might look like, okay?  Prove it before you bruise it. [Laughter]. I love this one. Okay, so POC, okay?  We do pilots with students sometimes, right?  I think we've all been there, we have gathered data from it. Purposeful pilots are more successful in a lot of ways.  But this is something in the IT world that has really stuck over time.  Our proof of concept is of course something you are going to do if you're going to implement a brand new, huge, Learning Management System.  You are not going to put that out to every faculty and every student, I hope you aren't, without some kind of pilot stage, some kind of beta to make sure you capture all the bugs.  Why can't online courses and instructional models be the same?  If we're going to make a decision that is going to solve a problem, unless it's an immediate decision that will critically impact students if we don't make the decision right then, maybe we don't make it. And think about, if I were the one on the hot seat and it was absolutely my responsibility in two years to make sure that we met our strategic goals for those years, what information would I want to make that decision?  To make a good decision?  And what do you need?  You need data. Data, data, data. [Laughter]. So proof of concept will provide you with the data that you need.  A proof of concept will allow to take a step back and figure out, what do I need to achieve it?  And gather five or six things to start answering the questions.  And look at the data, bring it back and figure out what model actually helped you the most. That's the best advice I can give.  Remember Quality Matters is always going to inform that quality conversation. So Jeff asked a really good question when I was over there and he said, so how do you get them to buy‑in?  Right?  How do you get IT to buy into Quality Matters?  And the answer is, you don't have to. Because if you have a well‑defined mission and well‑defined commitment and a well‑defined definition of quality that extends over all departments, Quality Matters standards can just be a check box for them. You're Instructional Designers, your Instructors, those are the people that you have to get to buy‑in. Right?  But IT doesn't have to buy‑in. They need to meet the Quality Matters standards.  Give them a checklist.  Don't even tell them it's a standard.  [Laughter]. That's my advice to you. 
	So I want to thank you for coming. I left some time at the end just in case anyone had questions or wanted to have further conversations. I'm also here if you need any support. And if you have not figured it out, I love data [laughter], so if you are having any conversations in the future, you are starting a POC and you just want to talk through, you know, here's where we are hoping to be and we're not quite sure where to go here, I'm happy to support you. That's what I do every day. So thank you so much. 
	>> [Applause].  
	
	CART Disclaimer:  This rough edit transcript, which may contain missing, misspelled or paraphrased words, is only provided for your immediate review and is not certified as verbatim and is not to be cited in any way.