CART Provided by A LaCARTe Connections.

This rough edit transcript, which may contain missing, misspelled or paraphrased words, is only provided for your immediate review and is not certified as verbatim and is not to be cited in any way.

Quality Matter
September 26, 2017

>> I'll start with Aheley.  She's a graduate assistant for student’s services and intern for New York thrives.  New York out of time network.  She's going to represent the student voice here on the panel.  Next to Aheley to her right is Paul gas ton.  The trustee professor at Kent State University a long-time consultant.  The author of higher accreditation.  At the end is Dr. Andria.  She's also a program coordinator for Masters of Science and education program.  She's representing ‑‑ I think you're representing the faculty and the institution view in my m ways on this.  Dr. Greg Von Lehmen currently serves special assistant for cyber security.  Greg is going to represent administrator’s viewpoint given his long experience in that area.  So I want to kick this off and I think I'll start the first questions are for Andria and Greg you can answer in any order.  What does rigger mean to you at your organization and what is it not.  You might consider talking about things like it's use in determining transfer credit or substitution for prerequisite.  What do you think it means to students at your institution and how your institution evaluates whether or not a learning experience is rigorous?  You can take those many pieces of this what is rigor and what is not.  In any order.
>> Okay.  Here's my 20‑page answer.  You ready.  [Laughing]
Whenever I was coming up with ideas to talk about today this is not just me.  I invited my colleagues in the college to hey some of them took up on offer.  I took notes from e‑mails and conversations.  I tried to synthesize a little bit of what came out in the discussion.  What I found was this is a really nice conversation to have with your faculty.  Because we all think we know what rigor is when we actually try to put in words it's very complex and multifaceted.  I'll give a quick rundown of a couple things my colleagued said.  One is it's planning.  It's planning at the course level and program level.  You can't build rigor in a course of isolation you need to develop across the program.  It fits in a degree plan and what students are doing.  Appropriately level learning objective.  That was a very common theme.  What that means students need content mastery.  Not just general intelligence to get through it.  That was something important.  Accountability for teachers and that boil down to having standards but then enforcing those standards.  It's not enough to put something on syllabus you have to live by and enforce that.  Accountability for students was also important.  One colleague said in a great way.  She said in hybrid course she gets in front of graduate student do not believe anything I say.  Now this is a bit unnerving for students to hear.  Especially coming from traditional perspective.  I'm not going to be with you.  You have to be accountable for that literature, that best practice so I can guide you.  It's not now have say because the teacher said so.  So accountability there.  Activity student engage.  We are training professionals.  Especially graduate programs Myer Ed leadership and counseling.  Students need to do in the course work what they are going to do in their career.  Having the meaningful product of learning and not just test taking.  This is also true at the program level.  I have required thesis in the program I coordinate.  It's not enough for students to go through all the classes and get ready for that without doing research project.  That's a completely different step.  Connections are very important.  Connecting your prior knowledge to the current course to what you're going to do in the profession and I think that's something that it speaks to prior learning experience as well.  Students cannot explain that connection we don't need to credit that.  They need to see how it fits in the degree plan.  Of course things that come out of quality ‑‑ alignment and assessment.  You're going to have threads in the program that aren't just on your assessment plans and you need to account for those things and that's when that planning comes in.  Timely ‑‑ I'll take a moment and pause, is art.  As I'm going through all of this.  I'm a teacher at heart.  A seasoned teacher knows what is you mean by the art of teaching.  There's something where you can curate materials and stage them in such a way that students have a learning experience that goes beyond the objectives.  I don't think question mandate that.  We certainly can't require that.  But what we can do is get teachers who are looking for that experience R it's meaningful.  Students get in a flow state at that they are working hard but doesn't feel hard.  Those things are rigor.
>> Wow that's great.  Greg we'd like to hear your thoughts.
>> Well I think I would agree with just absolutely everything Andria has said.  I think I would add something to it.  Rigor has two components in my view.  There are means and there are ends.  And I think everything that Andria has spoken to are all things pertains to the mean.  Planning, accountability of faculty and students.  Student engagement, I heard active learning kind of an echo of checkering and game son there.  And so forth.  I think these are all things certainly important to rigor but I think there's more to be said.  I think what's more to be said is about the outcomes that you're aiming for in all of those activities that Andria described.  I think it's the outcomes that really anchor the rigor of a program.  At UMC to reflect institutional view you all are very sophisticated you'll say all much the same thing.  How do you define the appropriate outcomes?  It's a combination of kind of faculty intersubjective agreement based on academic preparation.  It's referenced to appropriate discipline based accreditation standards, that are common in business or computing, and other fields.  And also informed by industry board or appropriate boards of experts.  At UMC we are very large university, large military population, large online population, close to about 90,000 students now.  And most of them are part time learners.  We have very few first-time freshmen.  Average age 30, 31.  Under graduate level.  Most working, most married.  These are people in school really to improve the employment prospect.  So having well informed program objectives that kind of echo or reflect what the market is looking for, is very important.  And in those outcomes, include not simply the discipline content.  Talking about management or cyber security or some other field, but also other outcomes that employers are looking for and we all know what these are.  Ability to think critically, to be able to communicate clearly, both in writing and orally, to be quantitatively literate at some level to be technology fluent, these are all also important outcomes that really cross all discipline boundaries in our cover dent and appreciated by all employers.  From an ‑‑ so I would say all those things Andria said are very important and make sense that you would say those things because it reflects faculty point of view.  I say rigor really starts with the outcomes of a program because otherwise all those activities are really misguided.  They don't have the appropriate aimer and for what h you're trying to gender mass mastery.  There are other vehicles used to promote those objectives.  I'm sure at your university and the other universities represented here rubrics are used to communicate, the expectations to students, and to kind of reflect what is expected on the institution in terms of rigor.  Not only in terms of the academic content but also the other qualities that I mentioned.  Very important is a feedback loop.  And all of us are very familiar with learning outcomes assessment at UMC we experimented with I think called academic profile, then EPE ‑‑ I forget what the acronym stands for.  It presented meth logical inferential challenges.  So now all of our learning outcomes assessments are embedded assessments.  As students go through the work we're collecting data on how they are doing that. lead feeds back into conversation about whether or not what we're doing in our particular courses really is in gendering the mastery of those outcomes that we're aiming for at the course and at the program level.  So let me stop there.
>> Okay.  Thanks Greg.  So Aheley, we'd like to hear from you to ‑‑ not that you can represent all students everywhere.  Your definition of rigor whether you think do students talk about or even consider whether the relative rigor of courses when making decisions and do they act on the idea that something's rigorous or not so what do you think?
>> Those are really good questions.  I guess I'd start by saying from the student perspective I think rigor is contractual.  So are you teaching me what you say ‑‑ am I learning what you say you're going to teach in this syllabus.  The syllabi is contractual.  I think a lot of time my background is higher Ed I think sometimes we underestimate student about what it is they think they feel and that they are conveying they are receiving in a class.  From myself I'm a doctoral candidate right at this point I'm trying to gain mastery.  I'm looking for whether or not this syllabus or sill by across courses actually represents the ‑‑ what the teacher says I'm going to learn.  That then goes into an issue of pedagogy.  Students are ‑‑ you will not get engagement unless you give engagement is basically what I'm saying.  So if a student feels that you have set a limit and culture that means that they will be challenged, they will have fun, fun still belongs in the classroom no matter what level students this l then engages.  I think that's important.  I'm also sort of speaking from the adjunct faculty side because I teach as well.  It's interesting that I can go in between these two ‑‑ these two identities.  Right.  The other ‑‑ your other question about do students talk about or consider the relative ‑‑ consider rigor.  Well, I think it's important to recognize that no college students don't go around oh no that class was not rigorous.  It offered absolutely no rigor.  What they do say though is instead students have a remarkable way of speaking to what rigor is not.  And they will say things like this is was a waste of time.  I did not learn anything in this course.  I brought the book home and had to teach myself.  Why does the ‑‑ I don't like lecture.  Does the professor take no opportunity to answer my questions.  These are the types of things that students say and what they are alludes to as rigor and essentially what they are alluding to is if they've learned.  If they've learned anything or if they even have the will to learn anything.  If they are coming into the classroom ready to give you their attention.  You know what that's like.  This is a cell phone generation.  Students have cell phone in their hand and that says something that. says something when they their head is down and constantly texting or tweeting these days.  I think that's extremely important that the academic success will follow after you set the cultural success which is a rigorous environment.  Which means that faculty very much concerned about the pedagogy and how they are teaching and what they are teaching.  That would be ‑‑ I have more to say.  But I think that would be sort of the biggest take away is engage them, they will engage you.
>> I think we're all going to want to hear what you have to say.  Let's go to Paul.  The question for you is a little bit different.  So ‑‑ it's why would a foundation have an interest in rigor, what's the foundation stake in this and what is at stake in this?
>> Yeah why did ‑‑ [Indiscernible] foundation put down the noun send know Fort Burkes.
>> Exactly.
>> The understanding is rigor if we properly understand the word in terms of educational quality, really offers the most inviting and supporting path way to the foundations big goal which is also a national big goal.  Which is a significant increase in the percent of Americans who hold quality credentials.  If that goal is going to be obtained many of those individuals will eastern them through distance learning but for any of these goals to be meaningful, you can't just make credentials easier to obtain that would be one path.  But it would be a meaning less path quality needs to be assured and that's really where rigor comes in.  But I do say properly understood because we have really rehabilitated that word.  And we use it in a positive sense that a lot of the people outside this room are not going to understand.  When we use the word we talk about really what we want for students that they become more critical thinkers.  They become more creative.  They exercise a healthy skepticism with regard to preconception and assumption and with b alert to bridges that connect different sectors of learning.  Like your example of disbelief everything I say.  Great preparation.  So I think one way to do that is echo what some of my colleagues have said.  We emphasize the equivalent importance of clarity with regards to outcomes, processes, with regard to the demands that we're going to make on students in terms of time on task and how much it's going to cost and what resources do they need.  And we need to be clear about what the opportunities are that await them once they receive the credential.  So I think we share as educators in this regard a practical and in some ways political challenge.  One of educating the sectors who's understanding of rigor may be very different from our own and by that I mean the federal government, department of education, and if you need examples of that, just think of the recent opinion of the inspector general with regard to western govern university or think back to the same office and opinion on solidifying making more concrete the credit hour.  Talk about a major step back.  Looking at the area of accreditation critically and making sure that understanding we share is understood by them, shared by them and I would ask there that you compare a recent statement from the counselor of regional accrediting commissioners which seems really to get it in terms of distance education and importance of outcomes what we would call rigor.  Compare that to the American bar association which has still the most stringent limitation of distance learning as a component of educational program reflecting a profound misunderstanding of distance education.  Then finally political leader view of outcomes may be very reductive and short sided.  I'm going to end with suggestions about how we do it as educators as institutions, as foundations, as people who have bought into this importance of rigor.  I think one of the things is to tell stories of success of our students to show how our students succeed but then on the other hand to be vocal in pushing back against ill informed positions I don't mean being defensive I mean being in defense of your students and our teachers.  I think we need to be pushing information to legislative staffers at every possible occasion.  In an attempt sometimes admittedly a feudal attempt but a well attempt to inform the legislatives about the values that drive us and success we're trying to achieve.  I think we seek support for programs that are genuinely innovative from organization such as lieu minute foundation or feds or other sources that give support and prom answer in to our best effort.  I think we can do more to celebrate internal victories within our institutions so we all become spokes person for jobs well done.  I think we maintain a commitment to this organization quality matters.  It provides one of the few reliable bridges across the casteism of higher education and K through 12.  That's a success story that needs to be celebrated.
>> So I'm going to do one follow up given ‑‑ trying to manage time.  I'm going to follow up Paul's sort of charge that we you know we make this case to the external stakeholders who make the policy decisions and who enact the laws and I think the challenge there is we've heard everybody's perspective on this and it raises to me the issue of measurability of the things we're talking about and I say this because I had read a ‑‑ the senate committee on health and education.  Was taking ‑‑ actually in a response about accreditation was taking higher task pointing to academic adrift and saying higher Ed was not mean meeting rigorous standards.  Students weren't writing long enough papers, they weren't reading enough books and therefore wasn't a rigorous learning experience and defining it that way and then later in that same paragraph pointing to alternative learning, outside of higher education, as high-quality learning.  And so I ask the pan list to ‑‑ for your reaction to the idea that rigor can be measured by things like the number of papers that you write and the number of books that you read and maybe I'll start with Aheley because I do ‑‑ she teased us a little bit about having more to say.  I'll start with you.
>> That's a good question when you think about measurable because that's how right ‑‑ we want to be able to measure it.  One of the first thing that comes to state of mind inclusion.  I think that the way that we measure sometimes is counter intuitive to the students that we serve.  So one thing is for example if you take something like Afrocentric pedagogy.  The way in which an afternoon American student or a woman or a someone ‑‑ some ‑‑ we're of all diverse identity our epistemology are very different.  The way in which I know I learn and I measure whether or not I've learned something is quite different and it may not align with how many papers I write, and how many books I read.  So I think we kind of need to start there.  I guess what I'm really advocating for is culture shifts in the way that we are thinking about what measurement even looks like.  One thing that we talk a lot about is G PA or high achieving ‑‑ we talk about high scores.  The literature says many students just feeling collective uplift is a measure of them learning in the classroom.  It may not always look like a high test score.  We want it to look that way but it may not always look that way.  I think we need to sort of maybe reconceptualize how we're thinking of measuring or what the outcome needs to look like especially because we're moving in a direction where cultural diversity is huge.  We're serving students who are from every part of the universe and we need to be able to account for that.
>> Thank you Aheley.  I'd like to ask other pan list if they want to address the issue about the difficulty and defining in measuring these things.
>> I'd like to weigh in.  Struggled with this issue for about three years in the terms of degree ‑‑ [Indiscernible] I think one of our conclusion was just as you say ways in which a student will demonstrate learning will vary widely.  And all deserve respect.  But that also means that statement of learning outcome needs to be accessible in terms of a student's ability to demonstrate the acquisition or the having realized the benefits of that program.  Or that course.
And I believe if that is the case, if we can draw the attention of our policy makers to what students can do, as a result of their education, we'll be there or at least part of the way there.
>> Go ahead.
>> I think this l be a never-ending conversation.  I'll say first of all I think that state legislatures and parents have every right to ask the question of accountability of institutions for their outcomes.  And I think that institution's need to be able to demonstrate to those audiences that they are achieving the outcomes that they advertise for their degree programs.  The real challenge is that institutions are strap for resources and they are challenged kind of by the inherent difficulty of formulating appropriate outcome measures and then executing on those particular measures.  I say this as someone who still participates on accreditation teams and you go I understand into institution after institution I've been on middle state ‑‑ I actually live in New Hampshire and participate on NHASK team.  It's not unusual to go in where they haven't completely implemented the learning programs or they have had a program in place where the data is not very high quality, where you're learning outcome assessment office may be one person who may have other responsibilities, and so I think it's a ‑‑ the questions of accountability are just questions.  And I think they ‑‑ for the reasons that I mentioned they are difficult for institutions to answer.  And I think that once institutions are able to show legislatures in the case of public institutions these are the data that show our successes and where we're challenged and this is what we're going concretely to address these challenges and oh by the way, if you want tows accelerate the progress we're making in this area we'll be happy to talk to you about budget request.  I think ‑‑ so that's my perspective on this particular issue.
>> Andria I'll give you the last word.
>> One-minute last word.  We are coming up on another accreditation site visit.  Assessment plans and what we're doing and getting project explicit.  When I was asking about how do you know you have v rigor we had resources I know because of classes I have rubric and provide feedback and students engages in things they are going to do in profession.  When we look at program level.  I take student artifact and we evaluate by a committee instead of just relying on class grade.  And then we also hold in the curriculum process which underwent a large revision last year.  We have a set of people start at the program through the department, through the college into counsels at the university level and curriculum where they review things and looking for alignment and things like that. I think we're generating documentation for assessment plans so we have something in place and we go at it knowing that it's never going to be perfect and it's not static.  It's very dynamic.  We work at this continuous improvement process.  We have this now and it's providing data maybe we missed something.  Next time we do differently and access something else.  It's on going process of continuous feedback.  That would be an assessment of what students are doing and how well they are prepared.
>> So now what we're going to do is ‑‑ and this is sort of aspirational list for you to consider.  Given the amount of time we have, I think we'd like you to group up whether you know ‑‑ you need to do that's convenient.  What we'd like is hear from you.  We want you to talk a few minutes amongst yourself.  Really what we like to hear back can you reach a common understanding of academic rigor amongst even the folks you talk to and to report back to us what that is and then to follow that up with any observations or questions based on what you just heard and what the panelist have shared so we can engage back about this.  Let's take time ‑‑ five, ten minutes.  To have ‑‑ I don't think you'll get too far down this list.  If you start with the definition academic rigor what it means from your perspective that would be great.  Thank you.
>> Okay.  So this is always the hard part, right.  You get folks engaging then we ask you not to engage any more.  Could I have the ‑‑ will the panelist come back up quickly.  Greg, calling you.  We'll get the panelist back up here quickly.  And what we ‑‑ I'll ‑‑ I think what I'm going to do and I'll get up from my seat in a minute so s to ‑‑ we'll release you for a quick break and so that you know we can reset up.  But I want to give you a chance to share back a little bit about the conversations and I don't know whether you actually could come to a common definition of rigor based on conversations I was hearing but I would like you to ‑‑ if you have any observations you want to share, and if you're too shy we'll ask the panelist to reflect on what they were hearing.  I want to give you an opportunity to reflect back based on questions you had and ‑‑ or the conversations you were having and any observations for the panelist.  So anybody want to give us a little report back on the conversation?
Please.
All right.  We have some remote folks we are trying to be inclusive.
>> I'm rigorous because only five percent of the pass the class so taking it to extreme.  It was ‑‑ [Indiscernible] in terms of defining it, measuring it and being ‑‑ [Indiscernible] come to an agreement.
>> All right.  Did anybody ‑‑ any of you have a conversation or observation about the point is rigor sometimes used as a barrier to new ideas or invitation or innovation do you hear that?  Is that a ‑‑ Leah?
>> My discussion partner Karen and I talked about this a lot.  What we're concerned about is idea of open admission access and opportunity balanced against academic rigor and sometimes a mission of institution or mission of alternative learning opportunity is to give students an opportunity to try.  Let me try this online course.  Maybe very low cost to see if I can get foot in the door without providing that opportunity, you know we could stand in the way of somebody meeting academic goals.  So academic rigor gets more complicated when you think about mission of institution or mission of provider and what is the goal to allow students to get into a learning experience and they use that as a building block or a barrier entry.
>> So context matters is that really the ‑‑
>> And mission.
>> And mission.  Right.
All right.  Cheryl.
>> Well I think we came to the conclusion there's no definition.  But one of the things I'm also ‑‑ at an institution where we have no entrance requirement at 2 years and some of our faculty feel like quantity ‑‑ I think you asked that question and it kind of got lost in something the number of assignment submitted or number of things they have to read determines rigor and that's not the case.  If they're the same level of content, there's no building, then there's no rigor.  I think we have to dispel the myth quantity equals rigor.  We also talked about the amount of time spent does that equal rigor.  Amount of time a student has to spend, again there's no rigor.
>> There is no answer.  For the panel list, you were out there having the conversation.  You want to reflect on anything.  Anything strike you as ‑‑ in terms of resonating with how you felt about it or actually seeing something you hasn't considered?
>> If I could make an observation I was listening to a group that was talking about academic rigor in relation to alternative learning.  And I think that's a very important question.  Particularly for institution that are open admission and willing to allow students to give it a try and are inherently because of admission open to recognizing learning that their students may bring to the institution for academic credit.  I think very important in that conversation the roll of intermediaries.  I kind of conceptualize as a supply chain issue.  So you have various forms of alternative learns that can be training in the military, training in a corporate setting, it can be a Mook it can be learning an individual picks up on the job and so I think there is a roll for what I call market intermediaries in validating that learning so for example, we have the American council on education that evaluates training in the military and outside the military for academic credit recommendations.  We have learning counts .org, which is an initiative of KL which is kind of gold standard portfolio evaluation of experiential learning and so the evaluations that are made through KL's process is also is helpful to institution if that student gone through the KL process come to us learning counts .org would give me so many hours for this that and I think there's proms develops.  I think the decree ‑‑ is that a‑‑ [Indiscernible] giving you as much credit as possible.
>> Yes.  I still have to get back.
>> [Laughing] I'll do my best.  You know also is a very promising development in this area because there are many and more offering of alternative education whether it's coding boot camps whether it's industry certification in IT and I think to ‑‑ I think also the equip pilot which is an attempt to kind of get on the inside of alternative learning enterprising and to evaluate their products and to validate the learning that happens there.  So I think in this whole conversation about rigor, the roll of what I call market intermediaries who are looking at alternative learning and evaluating it from an academic point of view is very helpful.
>> Well, I'm ‑‑ we're actually impinging on your break time.  Let me just say this.  I don't have to say anything else.  Greg just queued up the next panel because that's what you're going to hear about.  What I'd like to do is let you take a quick break.  Hopefully you can come back.  If you need to go to another session you can do that.  We're going to take a quick break.  And then we'll be done.
>> I want to give a 20 second last word that which rigor is current slang for really, really cool which I discovered in a sports story about signature curry that his three-point shot in a particular instance was rigor and listening to these discussions around the room this gives me my first opportunity to use the word they were rich, they were diverse, they were rigor.
>> All right.  I love that.  That's got to be tweeted out for sure.
[Applause]

QM Meeting 2

>> One thing I'm particularly interested in is if the current movement to Al con assessment does that make evaluation of rigor if it's based on student achieve.  I think that's probably enough to start with.  So ‑‑ unless one of you want to volunteer I'm going to start down there with Lisa.
>> Okay.  The first, I think question ‑‑
>> Just tell us about what you do.  Tell you what I do.  So basically, we conduct academic reviews of noncollegiate grant learning.
We are looking at the comparability of that course work to the college work.  We're looking at instruction and assessment.  Through that function what we're trying to do is accomplish that conferrable credit equivalency.  We set forward ‑‑ when I say we I don't mean we or my staff but the academic review team we bring together.  Faculty from all over the world participate in the review process.  Ultimately, we hope with a positive outcome we can set forth college credit recommendation that's recognized by institutions throughout the con try.  We do have a list of cooperating colleges and universities and again it's not a guarantee that the college or university will recognize the credit but it's a starting point and so I think you'll hear from my colleagues that they are also involved in very similar works so ‑‑ [Indiscernible]
>> Yeah why don't we do that.  Why don't we go through this and ‑‑ describe what you do and then I'll ask you the very pointed questions I want to ask.  Leah.
>> Hi everyone I'm Leah Matthews.  My daughter asks me mom what do you do.  To be honest I'm on the phone or answering e‑mail most of the day.  Beside from that DEAC is in the 91st year an accrediting organization for distance education.  We have 102 accredited institutions DEAC recognized by department of education and counsel for higher education accreditation.  What's unique about DEAC is that most of our institutions do not participate in title 4.  Accreditation is sought for other purposes to have entry into some of the profession accreditations reimbursement from employer’s other forms of tuition assistance gained through accreditation.  I think one of the other driving factor of not being in title 4 is two little words that last week kind of landed with a grenade regular and substitutive interaction.  Anybody that knows those two words know some pretty big news was released last week about the interpretation of those two words.  It's because many DEAC schools offer programs with flexible time frame, competency base student pace work, lots of different schedule of interaction between faculty and student.  There's lots of question about how title 4 fit in that model.  Although that's not our topic today.  It's great interest in distance education and DEAC looks forward to teaching learns outside the traditional academic calendar of learning.
>> I'm Mary Slusse.  Unlike my colleagues I'm not in Washington, D.C.  I'm actually in Chicago.  Within CAL our mission is to focus on helping adult’s earn credentials for their life learning and work and when we define credential we move beyond just higher education degrees, we also include things like training, volunteering, on the job training, certification, things like that.  So we ‑‑ very similar to the foundation we really try to help learners around the country figure out how to gain movement in their lives and earning more credentials to help them.  Specifically I'm on the higher education within KALE and I manage the learning could wants program.  Our program is focused on experiential learning or prior learning assessment specifically portfolio development and assessment processes.  I work very closely with students we offer a four-credit course on portfolio development so I'm responsible for the curriculum faculty associated with the course we provide students.  We partner with colleges and universities around the country.  Where with this service with that university almost as an adjunct arm of the university where we will teach students how to develop a portfolio and then I have faculty that report in through me all over the country who are subject matter and credential experts who then access those portfolios against stated learned outcomes for potential college credits.  When a student comes to us they are actually basing the stories of their learning or what we call learning narrative on a specific syllabus with stated learning outcome from any regionally accredited institution and then we are basing our assessment on that particular syllabus.  So that's the crocks of what I do.
>> Okay.  I'm from the American counsel and educations college credit recommendation service which is such a mouth full we have shortened to credit.  We don't want to confuse big credit with little credit.  So back in the mid 1900ACE started evaluating military training for service members in order to help them go back to school and acclimate back into society.  About 30 years after that launch military started looking at occupations and MOS for credit at that time some smart person said wait a minute anemia their jobs learned how to do things that were important.  Why not evaluate workforce training and AC credit was born?  Very similar to Lisa we do a lot of same thing.  Contrary to popular belief we're not going to have a boxing match we're actually friends and colleagues and do a lot of things together.  And then we also do a little bit of what DHEC does where we look at organization and determine the types of quality and rigor they offer in their programs.  We have about 150 active organizations currently with 50 historically.
>> Okay.  So you know the big question is then picking up on what you just said how do you do that?  Like what does academic rigor look like.  What do you look for ‑‑ I realize it's not you personally doing this.  But you know what's the charge here for the teams in how do they determine what is and what isn't rigorous and how ‑‑ does that change depending on what kind of learning they are looking at?
>> So I would say that we determine rigor based on what higher education accomplishes as rigor.  Basically we use faculty evaluator to evaluate courses they are the ones measuring rigor.  If what they are seeing in these nontraditional courses is align with rigor in classroom that's when they are able to make credit recommendation.  I think it was probably back in 2012 when there was a big shift from the creditors about assessment it took a time to trickle down for some of sacks faculty saying this assessment is up to par.  Some other faculty were saying we have to do something different.  We went through a big shift how we communicated through organization about what rigor means.  We really look to guidance from faculty members to set the tone.  We want these credits to be accepted at institution as equally as the courses you teach in the classroom.
>> I would add peer review and engagement is ‑‑ all accreditors reach out to faculty and experts in the fields whether it's teaching, or problem dissing in a profession, to engage on what the curriculum content is.  And another piece of it has to involve technology and the use of that technology is it appropriate, effective insuring academic integrity are we verifying who the learning is.
>> Assessments aren't going to do any good if we can't accomplish academe ‑‑ I think rigor has to fold all these aspects in ‑‑
>> I would piggy back off what both Leah and Kara have mentioned in terms of definition of rigor and how it's subject to change in keeping with higher education landscape.  We look at it from a very granular level and we use tools that are available to us for example, the QM rubric is something that we would provide to or at least mention as a resource to noncollege gait providers sayings you need to look at these tools available to self‑access your level of rigor before coming to us for an evaluation because we like higher education are also challenged by that definition of rigor.  I heard a lot of conversation about what is rigor.  Well in some cases rigor means more.  More books to read, more tests to take.  That's how we get to rigor.  In our world in the work we do and evaluation process we're looking at things that are just far more detail than more.  More doesn't equate to rigor.  We're looking at the structured formal alignment of the assessments.  We're looking at the very evaluation methods that are used.  We're looking at the multiple concepts that are being drawn together to construct new knowledge.  When we think about rigor we're thinking more of something that is not just a linear scaffolding to learn more of a spiral.  When we're looking more college gait ‑‑ we're sharing the resources we find to exemplify rigor and we're also looking at formative and summative assessment and sophistication of topic.  We're mirroring the process you would in your own institutions and it did say it is a changing evolving world when it comes to rigor.  I'm really excited we're having this conversation and being here to glean more information from you in terms of what your definition of rigor is.  We need our process to actually demonstrate your viewpoints of rigor.  I was excited to hear from some of the early panelist about re-conceptualizing the way learn as good measured.  And I know that I'll learn some more from the other panelist here today.  From.
>> From my perspective ‑‑ we only use sill by that come from regional accredited colleges and universities so there's an assumption there that a certain level of rigor has gone into the curricula develop process.  Secondly from our own perspective.  Assessment methodology we operate off a comprehensive rubric that takes into account primarily whether or not a student has fully demonstrated competence related to those course outcomes stated on syllabus.  In addition outcomes here also looking at things like self‑reflection.  Does a student ‑‑ is a student able to reflect on academic theory and broader concept and ideas.  Does that student provide compelling Ed that further demonstrates learning?  Have they shown have they applied that learning over time and continue to apply that learning even after a specific event maybe that they are siting as part of that learning narrative.  In addition to the outcome base part of the assessment we're also focusing on things like writing ability.  Does that student demonstrate because this is a written portfolio that they're able to write at a college level?  So there is an expectation that just because a student comes and says I have knowledge about this I have learning in my background, our biggest challenge back to a student is that learning at the college level and can you demonstrate it compellingly at that college level.  So all of our assessors come from colleges and universities.  They are actively involved in the ongoing development and maintenance of the are you bricks we utilize.  In addition to quantitative score a student might receive on rubric they are also receiving the qualitative feedback that talks about this is exactly where you nail this content.  This is where you have opportunity for further learning, further growth and then we're hoping that that feedback then ‑‑ if it does translate into a credit recommendation for the student, that that student is still going to continue to learn and grow as a result of that.
>> Okay.  So I'll push a little further on the idea of outcomes assessment and clearly Mary that's ‑‑ we see a lot of pressure now ash ‑‑ a lot of talk and political circumstantial particularly about a push towards outcomes assessment and ‑‑ and I guess taken to the extreme can we evaluate rigor on the basis of what ‑‑ is that a ligament way to consider rigor when you're look at a course of study or academic institution program.  I'll ‑‑ maybe ask Leah to take that first.
>> One of the panelist immediately before us I think our student said learning is contract yule and I see academic rigor as going hand in hand with outcome assessment.  Outcome's assessment is in fact measuring that rigor and measuring how well students have accomplished that educational objective.  What the actuality assessment of teaching around willing shown.  How that outcome is mapped at the course level, at the program level and institutional goals level.  And when institutions do that well, when they mapped their curriculum and their assessments effectively, then I think you're going to see academic rigor threaded through the entire learning experience for the students and I think accreditation has a model of engaging peer review, institutional improvement and outcome assessment practice as a way to tie assessment of rigor into all those aspect of the institution.  I think it goes hand in hand.  One doesn't replace the other.  Kara do you want to respond.
>> Yeah I was thinking about rigor being different.  And the way you measure it has to be different as well back to what Aheley said in the first panel about we have to think about if we're measuring the rigor and outcomes in a way counter intuitive to the students or a way that's going to show what they are learning.  If we fake MUMC in fire program I would not want a firefighter in that program writing a paper about how to use the hose.  Right.  I want them out on the training ground showing me how they do it.  It's a very low level of learning you know but it's what's appropriate the to measure the rigor of that type of an outcome.
>> Outcomes are not necessarily consistent.  I look at syllabi from colleges and universities all week long.  Hundreds and hundreds of outcomes on a weekly basis.  And no two syllabi or ever the same in terms of consistency.  One of the folks I was talking too what ‑‑ the biggest challenge I see outcomes are not always complete.  I think even though we have a accreditation guideline and ACE and NCCRS are sofa nominal in working with institutions in terms of credit recommendations, the biggest challenge that we have with outcomes based access system that outcomes may do a great job at defining a verb in a subject but if there's no criterion referencing, that then makes that outcome not quite enough to really fully define whether that outcome is considered rigorous and a student doesn't know what level he or she needs to demonstrate that learning.  That's probably from my perspective.  I could get on a soap box about this all day.  From my perspective that's the biggest challenge in higher education we have, first of all we don't have common definition for anything.  Outcomes, competencies, CBE, none of those things do we have a common definition for.  We aren't even consistent in terms of how we develop syllabi how we develop the outcomes so that students know consistently from course to course what that expectation is in terms of what they must be able to demonstrate as a result of that course.
>> In my view I think that the shift towards outcomes assessment really underlies the importance of rigor and especially in terms of the promise to the students.  So Aheley mentioned the syllabus that's the contract and you're stating to the student what they will be able to do after completing this course a promise is made there and it goes beyond an expectation when you start to drill down in the specific objectives and the way it will be accessed.  I think the shift towards ‑‑ particularly if it's done well is access to rigor and can actually be a little bit reverse engineering to get and arrive to that promise to the student so I think that the shifts will stay and I think that there will be more intention.  It's something NCRS really challenged by in terms of educating noncollegiate providers.  It's still being provided ‑‑ [Indiscernible] in terms of quality assessment.
>> I have one last question then I want to be able to solicit questions for either panels.  But if you had any guidance to give academic institutions ‑‑ I guess particularly higher Ed in this case for how they might meaningfully and productively and rigorously be able to integrate and accommodate the different learning path ways that exists and we see that this is really a growing trend that students are looking outside of higher education but what would you recommend to help higher education to be able to be a part of that path way and help students have an onramp to accredit ‑‑ programs what would you suggest.
>> I would say first this is the new normal.  Some of the commentary around accreditors how the first-time full time student really isn't the case anymore.  We still have fixation on iPad.  We're not capturing 75 percent of the student enrollment because students aren't in that category any more.  They are bringing alternative learning bringing courses they've taken online.  They want credit for experiences they've had as volunteers or as professionals not yet with the degree credentials.  I think we have a real responsibility to try to find ways to give access to teaching and learning that gives people credit for these types of learns experiences and it's the kind of experts that are up here that I think contribute to that discussion.  So going back to rigor, and having detailed curriculum maps helps align these alternative forms of learning.  So if a student brings a transcript that has a breakdown of that syllabus and can show you maybe some of their assessments, then you can map that to the curriculum at the institution and see, yeah, maybe this credit can transfer from this alternative site.  Maybe it can be a meaningful ‑‑ being awarded by this institution.  I think the sometime has come to give learners credit for what they've accomplished.  If it is in fact a meaningful match with what an institution has to offer in their curriculum, it should be considered.
>> I totally agree with you.
>> I would also add I would invite all of you if you've never done a peer review with any organization up here.  Join us.  It's highly rigorous.  There's a lot of collegial conversation that happens.  It's a team base atmosphere.  There's rubrics and structures.  I think the more institutional faculty understand what happens in these third-party reviews, the better it will be for the student path ways.  I think that ‑‑ you'll be hard press to find a student who's gone outside the traditional path way to do work on his or her own I just want to do the bare minimum to get by to get this degree.  Those are not the students that you're going to find that's training this path way.  They are the ones that shopped all the different alternative providers because they have to pay money for them they are going to pick the best one with the best outcomes.  So these are the folks that already going a little beyond in order to insure they have an opportunity.
>> Not only that, but KALE worked with limited foundation.  That was a longitudinal study based on 48 institutions over six years 62,000 student.  What we found as a result of that study students who had some form of alternative credit or what we call prior learning assessment or credit for prior learning CPL learning they were ‑‑ in addition to that we found these students took up to ‑‑ as much as 8 more courses on their campuses that they didn't need to take they spent more time and more money on campus because they were recognized for the learning they had coming into their degree plans.  So it's ‑‑ it really is a matter that alternative learning is here it's going to stay.  It's not going anywhere in fact it's probably that definition is growing more than anything.  Recognizing that and becoming part of the national conversation.  Familiarizing yourself with the terms and helping to create a common lexicon I think is probably the first thing I would recommend but become familiar with that and then do get involved and certainly if you go to our KALE.org you can have access to research information.
>> So let's ‑‑ I want to leave a little time here at the end for your questions.  Those of you here for both sessions.  You had time to talk to each other about this.  If you have questions or issues that came up that we went it didn't address if cow want to raise those that would be ‑‑ I'm sure the panelist either set would be happy to take it.
>> Will you talk a little bit about what you see or vision of the future uses of badging ‑‑ [Indiscernible] maybe ways to demonstrate is what you're talking about so much going on behind the scene.  Maybe I'm not as familiar with what's already available to student’s employers other institutions it's more visual what the students are getting credit for related to what they do.
>> I'll take that one.  So at ACE we work with credit I will who is initially Lumina funded now they've gone off on their own.  We've just implemented badged for credit recommended courses.  As ‑‑ Hoover over that badge it will tell them the learning outcome, credit recommendations that are being made.  We are in our next phase transforming our transcript to make them interactive.  You'll be able to Hoover over the information and see what the competency the students have learned there.  The credential engine huge.  We been a part of that from the beginning.  I think that's going to be a really great place to go to see how everything connected in the open badge platform has been really a game changer I think.  The open banal platform.
>> I wonder if you want to give a definition of credential ‑‑ [Indiscernible] actually is.
>> Sure.  [Indiscernible] enable a user to gather information about programs appropriate to that student's objectives and to begin to make a reasonable comparison among them.  It will be not a ratings service or you know a plus creditor but it will provide the user with the tools necessary to make some judgments and to reach of decisions.  As someone said earlier, it is completely inclusive in terms of available credential.  You'll find ‑‑ you'll find associate degrees, bachelor degrees, you'll find certifications, you'll find certificate and the two are different of course.  And so it ‑‑ it is an attempt to bring together this wild west of credentials and enable people to make sense out of it.
>> Just two comments.  One about the credential engine Paul and I were talking.  I think just as ‑‑ [Indiscernible] .org will say if a student comes and goes through their portfolio process and three or six semester hours awarded for whatever communication skills or writing or basic accounting whatever it might be, all the schools who are member school of learning count .org will accept that credit recommendation.  I was thinking for credential engine just looking at some of the partners yesterday at the credential engine, there are schools that will accept industry certification for so many semester hour accreditation.  Network administration, or other IT fields.  Would be wonderful if those were announced or you know on the credit engine because basically they become a path way in the same way you're credit recommendations become a path way to an institution.  The other comment I want to make not so much a question it's a comment.  So what we have here are market intermediary that translate training or spacious learning into academic ‑‑ credit recommendations so it's recognizable by accredited institutions to be accepted or imported into a degree program and accelerate a student path way to graduation.  I think it's interesting that academic institutions are going in the other direction offering various kinds of certification that are not accredited or recognized for academic credit but are valued by ‑‑ I'm sorry valued by employers.  And I'm thinking of this phenomenon of say micro Masters on Ed X or I think course called unanimous know degree or that's ‑‑ I get moots mixed up.  But I think that's an interesting development.  I don't think that will threaten the business models of ace and so on.  But it is an interesting development where in this case the market is making a judgment about what is valuable and what is not.  And therefore professionals are going through those programs or those courses to achieve that credential.  And one good example is John Hopkins offers a ten course ‑‑ I guess they call it a unanimous know degree.  In data science and I noticed that the NSA in a job announcement for data science position indicated that possession of that credential was equivalent to one-year job experience for the purpose of applying for the position and if you were hired the level at which you would be paid.  So any way, not a question just an observation that it seems that accredited institutions themselves are getting into the business of not accredited or alternative education.
>> I think that's a ‑‑ that's interesting to me as well.  I think what we're seeing from NCRS some of those mooks earlier accomplished they had intention to bring students into to give students to self‑access their own strength and weaknesses and be able to prioritize where they need to go in terms of developing the necessary skills to be success informal college around then over time what we've been finding some of those earlier mooks then also wanted to add the option to be able to have a review that would lead to academic credit.  So we're really in an interesting place in higher emaciation to see emerge of so many opportunities.  So appreciate the observation and comments.
>> I'm giving voice to somebody in the previous session.  This is not my original idea.  In talking about in small group activity.  I heard the comment for academics to talk about what we think rigor is a bit self‑indulgent.  We're saying we're great and we're doing this.  What we need to include also in the stakeholder social security those ‑‑ [Indiscernible] those external folks.  What do they have to say about this.  Are we actually getting students to what they need to see.
>> That's a great point.  We had a question back here.
>> [Indiscernible]
>> What happens when financial goals of institution collide with academic rigor.
>> Who would like to take this [Laughing]
>> Well in 30 words or less, I think that again the answer really concentration on student success and the documentation of that success in ways that are substitutively appealing to legislators that are a part of their core interests and we don't often do a very good job of that in terms of how we describe student success.  I think we do need to describe in terms they appreciate and then go beyond that.  We talk about the limitations of the surrogates of success such as earnings and student loan pay back but I think we also need to recognize those are in some ways legitimate measures then we need to go beyond that and provide other measures.
>> My question is about rigor.  And academics at ‑‑ not always the outcome where the rigor is focused it's the process to getting to the outcome.  So in accessing syllabi or accessing the rigor of a training courses or courses offered in ‑‑ by employers, how do you measure process?
>> From my perspective, the whole purpose of prior learning assessment is that a student should be able to demonstrate in a compelling way the outcomes of a course without the pen fit of taking the course.  The learning activities, the resources, the instructor, any of that.  So we don't validate the process.  We don't even really look for that.  Where it does come into play a little bit is that the assessor who's reviewing a portfolio and reviewing outcome in the student story is looking sometimes at what are some of the sample activities that maybe the person that wrote the syllabus drafts the outcome might have done in the classroom and does the student reflect some of that.  But it's not a requirement.  So we're really focused on the ends not the means from our perspective any way.
>> I would say something similar for us.  We don't have a requirement of you must complete this many activities and many n many readings in order to receive a credit recommendation because we're working with work place.  I'm not going ‑‑ those who had jet Ed courses.  When we look at workforce that corporation is training to that mission.  Get them train and on the floor.  If they don't perform and can't do outcomes well they are fire, right.  So the process of getting there for us is not as important as can they do it.  Now do we see more rigor in training course where they actually have to demonstrate as on the job training, typically, yes.  Similarly ‑‑ it doesn't matter how they got there but they got there and can do it.
>> I actually think my small group ‑‑ we talked a lot about the process.  So one thing that we were able to arrive at collectively was that rigor involves challenge.  And whatever that might be.  Someone throughout an example if there's a course on how to tie the shoe for most of us in the room that's not rigor unless you didn't know how to tie your shoe.  So there was a certain level of challenge.  I think that we talked a lot about that because one person mentioned also that maybe a way to kind of feel this process is do pre and post like valuations.  Like we always evaluate students right after the course where it's fresh in their mind or jaded and they hate what they done or they really, really liked it.  I think it would be interesting to do some survey three years out, four years out.  Right, because rigor doesn't ‑‑ [Indiscernible] It doesn't necessarily menu have to enjoy your course for it to be rigorous.  In fact, courses rigorous for students are the ones that are pain in the butt and end up giving the skills that you need down the road.  Skills like confidence, skills like resilience.  Skills like staying up all night to study.  I think the process the very important.  I think ‑‑ that was a great question.  I think the process is so important to that conversation.
>> When we're looking at noncollegiate learning we're looking at instructor qualification.  We have a certain standard that must be met.  When evaluation team on site construct conducting review. noncollegiate are challenges by the evaluation process.  It may take two, three, four, five years and more attempts in order to pass our standard for accredit recommendation because we are ‑‑ evaluators are looking at that process.  How are you measuring that learning and what process are you using?  We're doing a lot of academic consultant in some ways.  So process is very important.  And I know with corporate training as Kara said there's a body not performing they are no longer there.  There's proof in the pudding for that process as well.  So good question.
>> Is there another question?
>> So this is ‑‑ my world is really that ‑‑ I'm not an adversity.  I'm really interested in alternative past.  However not in the United States, so I want to ask anybody that would have an answer actually sent an e‑mail to credential engine through their info to find out what are the trends possible in going to a more globalize view of this things for Spanish and other languages connecting con tries is there anything?
>> I can speak to that a little bit.  You know going international is really, really challenging.  At DEAC we have number of institutions expand internationally.  We have found government outside the United States are very interested in what's going on with distance education enrollment.  We've met with many ministry of education and higher said is much more control as a government function and quality assurance is carried by government not these independent private organization.  So you know there's ‑‑ I guess a bit of skepticism about it.  Still a lot of preconceived notion about distance Ed and lack of quality just because distance education.  I attended a conference about six months ago.  It was attended by 70 international organizations on quality assurance and all of them were still ‑‑ where the United States were 15 years.  How anything could possibly be quality in distance learning education setting.
>> I'll just mention you can Google credential engine and get a lot of information about it.  Google credential engine it will be fully and open to anyone.  It's be in sort of a pilot form so far it may already be open.  Certainly, about this time it's on the brick of being open and accessible and then you can begin to navigate it.  But you could be just by Google it.
>> Okay.  Well I want to say thank you to everyone.  I feel like there are probably questions that we didn't get to.  We definitely would like to continue the conversation in a number of different ways.  One easy way perhaps is via twitter is hash tag academic rigor.  I know we had a lot of great information from the panelist today.  And we didn't solve the question of academic rigor obviously.  So we're going to be moving forward on this.  I do thinks the well-established that this is a very important conversation.  I want to say thank you so much to the panelist.
>> I just – yeah, I want to thank everybody for ‑‑ especially those who have been through both sessions late in the day but you know I think that's why we're all here.  Quality matters and ‑‑ rigor matters so thank you.
>> Thank you so much everybody.  
[Applause] 
 
CART Provided by A LaCARTe Connections.

This rough edit transcript, which may contain missing, misspelled or paraphrased words, is only provided for your immediate review and is not certified as verbatim and is not to be cited in any way.