There is likely to be two strands to the OER evaluation strategy- the more quantitative KPIs required by the HEA, and the more qualitative ‘lessons learned’ and ‘what next?’ wanted by JISC.

YK and MS have had a meeting with Helen Beetham, Subject Strand E&S consultant with the OER Programme. Helen is hoping to develop a set of templates for case studies etc. to help with E&S.

Suggestions/thoughts for further discussion:
1. Take a lighter touch on following all E&S ‘leads’, esp. in terms of working with institutions (individual and institutional strands will be looking at this as well)- what are our unique investigative questions for this project? (I have some ideas… what are yours?)
2. Concentrate and track a few of these specific issues, rather than having too wide a spread of investigation.
3. the SC strand will probably reveal more than originally thought- and potentially more than other two strands.
4. What is quality? Main accessibility measure?
5. need strategies for capturing evidence during process- to drive interim reports etc. blog entry perfectly acceptable!
6. Links to CoP strands
7. Does the theme or discipline affect engagement?
8. Does ‘ESD’ within HE loosen ‘business hold/branding’?
9. why are some communities better at sharing than others?

Any more?

To add from OER consortium discussions online- where were the sticking points for each partner? Maybe send a quick email/phone the insititutional links before their memory fades?

well it looks like the legal stuff may finally be coming to and end – what a massive pain in the backside … if it wasnt for some dedicated people who can actually read and understand this kind of stuff – i’d be sunk. Cant wait for the time when everybody will have to put their stuff onto university database jorum what nots – we will need to hire more lawyers and contract specilalist then academic staff and students combined…perhaps that’s the idea? reduce academic staff numbers (from non STEM non REF facing departments) and replace them with administrators to create one big world of FEC, forward facing IPR backdrop convergence material….! I can see now why so much resource is required for the  OU modules.

Attending: Michael Sanders, Yolande Knight, Neil Witt.

Apologies: Brian Chalkley, Sharon Gedye

1. Mike provided updates on the following:

Consortium Agreements

Having initially been given advice at OER start up meeting not to use the JISC Consortium Agreement template, Mike wrote a ‘plain english’ version based on other successful consortium bids at Plymouth. A couple of the partners’ legal contacts had issues with the draft that needed ironing out. Following consultation with other Subject Strand projects (C-SAP and Bioscience), Mike decided to go back towards the JISC template, as others have done. That second draft has gone out for consultation with partners.

There may be an issue with all partners being physically present to sign the consortium agreement by the necessary deadline (due to the time of year). Mike has contacted Programme Management to ask what consequences there may be if this occurs.

Use of the Blog and Sharepoint

Mike is hoping to encourage all partners to use the blog to keep a record of their project’s life cycle: successes, challenges etc.. Users are encouraged to be upfront with issues, as this blog acts as part of monitoring and evaluation of the project.

Mike has also agreed to investigate the use of Sharepoint on behalf of the SC strand (along with other projects). Several members of the project team have not had good experiences with Sharepoint, but are willing to give it another go.

Repositories

Neil has just set up another repository for an external organisation and therefore is keen to develop a similar repository for C-change, separate to UPlaCe. Although Neil is taking all project responsibilities with him to his new post (E-Learning Director), he still wants to work with C-change in sharing a learning technologist (0.6 FTE C-change, 0.4 FTE other). Yolie to liaise with HELP CETL Manager to sort this out. This post would work in collaboration with HELP CETL LT, Neil and Mike on the development of repository support material for academics, and suggested workflows. These would be some of the projects outputs, linked to reuse and further contribution to OER after the end of the project. It also fits in neatly with the CETL’s ideas for sustained repositories, which allows shared working and added value.

Steering Group Meeting

The first C-Change Steering Group Meeting has occurred, with attendees from 3 of the project partners.

IPR Workshop

This is set to occur sometime in September, but Mike needs to encourage partners to give dates available for attendance. Venue to be in London.

Evaluation

Due to changes in Neil’s position, Anne McDermott will now not work as evaluator. Yolande is currently taking over this role, liaising with Helen Beetham as the Subject Strand consultant. Mike will ensure that previous discussions and feedback from community (from Christmas onwards) is ‘retrospectively blogged’.

Professional Bodies

Mike will be talking to all professional bodies about possible future roles in the project.

Quality Assurance

As part of original project bid, we know that all materials being provided has already been validated as part of an accredited programme. Will there be issues with ‘comparative qualities’? It is likely that this will only occur in terms of ‘technical’ quality: are there three types of quality to be considered, namely content/knowledge, pedagogic context, and technical format? Neil suggested that the main quality measure to be concerned about is ‘can it be re-used easily’? There is a possibility that if a resource is too ‘technical’, that this might affect ease of re-use. It might therefore be better to produce resources at a lower technical spec to allow more accessible re-use.

How do we/ can we quality assure re-use of materials? One way might be to ensure this is covered by attributable licensing- terms of license could be that if you use something, let us know what you do with it, and let us have it back- i.e. feed it back in to the system.  That way we can track how things are used/developed. If this occurred, there would be a QA issue however, as altered materials would not have been through accreditation.

Branding

Common sense needed here: keep low key and subtle, and again, relate directly to licensing. Need to ensure have a handle URI for each resource.

Community of Practice

Following first Steering Meeting, the Communities of Practice have been combined to one, with two strands (pedagogic and OER). Neil will set up an ELG from the C-change webpage to allow easy communication for groups.

Having tried to create a consortium agreement that was human readable I’ve now reverted to one based on the JISC template (with a little dash of C-SAPs CA). After some concern from one of our partners about the wording of certain clauses in the original it seems that our new one, based on the JISC template, sits more comfortably with legal colleagues. Having originally been steered away from using the JISC template it might be that it ultimately proves to be more than adequate in getting that all important sign off by all 11 of our partners by the end of July.

Mike and Yolie ‘attended’ the first online Elluminate session, which introduced the evaluation and synthesis part of the OER programme and raised some interesting questions for future discussion. The E&S team will synthesize the discussion at a later date, but what follows is a brief summary of the discussion from our point of view, with appropriate links to other blogs/wikis/Powerpoints etc..

The link for accompanying Powerpoints, and a live recording of this Elluminate session can be found here: http://www.caledonianacademy.net/spaces/oer/index.php?n=Main.Presentations

The JISC OER ‘Synthesis and Evaluation Function’ aims to build a shared framework for the evaluation and synthesis of the OER piulot programme- so does what it says on the tin! There is a project wiki: http://www.caledonianacademy.net/spaces/oer/index.php?n=Main.HomePage, which will hold information for programme participants, and, hopefully, allow continuing dialouge with the S&E consultants. The three OER strands (individual, subject and institutional) have specific consultants working on each: the Subject Strand has Helen Beetham as contact, and Mike and I are already in discussion with her with a (very early) draft/notes on possible evaluation strategies (document to follow). The current draft evaluation and synthesis framework can be found here: http://www.caledonianacademy.net/spaces/oer/index.php?n=Main.GenericFramework.

The first Elluminate discussion session covered the following topics: marketing and branding; possible benefits and support for staff; pedagogy of OER; managing databases and metadata (this will be the topic for the next Elluminate session, on the second Tuesday of August, 2pm); reuse of OER and accompanying quality issues; general E&S queries. Due to the nature of Elluminate as a forum, the discussion was bitty and a little confused, but some interesting questions could be extracted, which may be worth highlighting here for further consideration by the team and partners.

Firstly, however, here is a brief intro into the E&S framework development:

How should we develop our evaluation strategy?

To allow our evaluation strategy to map to the generic E&S framework, we need to identify our project’s key outcomes, especially in relation to the following: findings, impact, benefits and lessons learned.

We need to begin to define ‘measures of openness’ (not quite sure what this means at this stage!).

The E&S team will try to help us identify appropriate factors, methods, timings and measures that will help us achieve the above: but this relies on us being proactive in seeking advice.

Once we have these methods/timings/measures agreed, it is up to us as a project to apply these through our evaluation processes.

The rationale for working in this way is that such a structure allows for the development of a common language for collation of data, challenges, solutions and outputs, encourages the sharing of questions/issues and allows the identification of key areas of interest and useful approaches for the future.

Problems/issues/questions might fit in to three categories:

1. those that can be answered by JISC/HEA Programme Managers (and tend to apply to all projects)- and an answer/solution is available.

2. those that can be answered by programme support (the OU team) and need expert advice from those already working on OER

3. those that are issues for evaluation- that is, questions that don’t have a clear answer, but the programme is investigating. Thies issues/questions can also be seperated in to:

• organisational and IPR
• social and cultural
• technical

Issues that were raised by Elluminate participants during discussion:

Marketing and Branding

• Our University’s reputation is at stake with this project, how do we approach marketing and branding? Reputational benefits may depend on identifying authorship.
• Do we need to look at business models?
• What is the difference between ‘branding’ and author/institutional acknowledgement?
• Should this just sit in the accompanying metadata?
• Is anyone thinking of using licensing that doesn’t mandate attribution?
• If an OER is branded, the branding should not impede re-use.
• Could we just use a UKOER logo (in addition to UKOER tag)? How about logo alongside originator (project partner logo?) details?
• Will a lack of branding/acknowledgement prevent reassurance of already reticent academic staff?
• Creative Commons licences wouldn’t mandate retention of UKOER logo

Benefits and support for staff

• Links investigating benefits for staff (individuals and institutions): http://ie-repository.jisc.ac.uk/265/ (Good Intentions Report); CETIS OER Briefing Paper
• Who are the stakeholders and how can we support them, and evaluate their experience? Academic, teaching staff, technical staff, learners
• Do we need to consider evaluation of end-user communities? Who are they/likely to be? What do they need?
• How do we investigate benefits of OER whilst, at the same time, encouraging engagement?
• What motivates enthusiasts? Ask E&S to investigate participants with OpenLearn at the OU- a very enthusiastic and aware group.
• Do we need to look at how discipline cultures work across institutions? How do different disciplines share research, for example? Might this have some effect on how a discipline engages with OER?
• Colleagues at Caledonian University can cite ‘reuse of resources they have authored’ in their application for promotion on the basis of learning and teaching.
• Is anyone doing a formal survey on academics’ preferences for release of material under creative commons?

Pedagogy

• Fragmentation of resources makes it difficult for the background pedagogy (good or bad) to be transmitted wholesale to the user
• do we even want to put such caveats (e.g. ‘must be used in this pedagogic context’) against material?
• Ope Spires thinks that should not wrap material up with unnecessary contextual material
• Making resources easier to use overcomes barriers
• For the purposes of this programme, cannot evaluate learning and teaching quality (above and beyond initial accredited material) in terms of reuse, but maybe we can explore changes in practice and perceptions around quality?
• If we are not looking at pedagogic value, then surely this brings the programme’s sustainability credibility in to question?
• the JISC Mosaic Report has a good overview of this issue

Managing Metadata and Databases

• Need advice on subject schemas vs. subject tags?
• Tracking: how do we track across both OpenJorum and other repositories?
• Can track visits using Google Analytics.
• Should not just track the ‘what’ but also the ‘why’
• Another suggestion is to use javascript to record downloads- you can add a click event handler to document links, and use this to track using e.g. google analytics. An example can be found here.
• p.s. from Yolie- I’m happy with metadata discussions, but can’t guarantee that I’ve necessarily got the right words or ‘ideas’ down with respect to e.g. tracking. Far too technical for me!

Reuse and Quality

• Evaluating reuse of OERs may well be out of scope of programme: programme is focussed on creation/repurposing and release of OERs rather than on their reuse
• However, reuse will influence quality of OER, and a couple of the institutional projects will be looking at reuse, so should keep an eye on these.
• However, we may well get hints at reuse, the hows and wherefores- so make sure manage to pick these up as and when they appear.
• Should ‘intention to reuse’ be seen as part of the release? Open release implies re-use and so is part of the change of mind-set
• The programme is starting on the road of ‘building an OER culture into course design and delivery’
• SC credibility for OER project is affected if the actual resources are not valuable for reuse. Is difficult to evaluate the potential for the OERs in context until someone else has tried to embedd them. How do we maximise component reuse?
• For Biosciences, each project partner is finding a ‘client’ as a user for their OER, to act as a critical friend, rather than simply rely on the repository and reactions/use of the latter.
• Is there a difference between quality in terms of original resource (accredited programmes, therefore quality assured) vs. quality within new context (reuse)? (we think yes to this)
• We have no control over context of reuse?

Other Queries

• Institutional programme blogs can be found at http://www.netvibves.com/hwilliamson#oer-institutional_projects
• If anyone is interested in the Oxford project on Audio Visual enterprise level infrastructure, you can find more information at http://steeple.oucs.ox.ac.uk
• Could projects use Elluminate with their partners? (programme leaders investigating licensing for this)
• In terms of evaluation framework, could we have a ‘phase 1′ and a ‘phase 2′, so that we can get the project and partners started (evaluting as we go), without necessarily knowing all the questions we want to ask overall?
• Projects must gather evidence throughout lifecycle, but mapping this to the framework may well take a couple of attempts. The E&S team will help with this.

As the project continues, we will pick out particular issues raised here (and at other meetings) and ask partners for their thoughts.

I have made contact with Debra Morris who works with Hugh Davis, nominated by DVC (Education) as carrying an institutional brief.

She tells me that material in our university repository (Edshare) can be assigned open access now if the authors so wish.

Debra knows of 3 OED groups elsewhere in the University but has not so far identified the final two (of 6). It looks like when all are teased out there will be an attempt to get everyone together.
Debra regards the OER project in the LLAS subject centre here (HumBox) as taking a lead, trying to do much the same job as GEES is doing.

Started project – modules in place and ready to begin re-purposing. But already hit snag number 1 – member of the team may be leaving. Raises interesting questions as to what you do with material submitted into OER from one institution when the author moves? Do you simply change the name and resubmit? or does the material remain the property of the original institution. Change the name – add new logo?? Material could be resubmitted from new institution – same material two sources….. arr the world of OER….

At the recent steering group meeting the issue of having a common look and feel or house style was raised. It was generally felt that this should be kept light touch with just a front page for each resource. As a starting point for designing a common template for a ‘cover-sheet’ for each C-Change OER resource I have generated the following list of potential items. This list is presented in no particular order and is shared with you to stimulate debate. It is too long and needs refining. Have a think about how it might work for your potential OER offerings – clearly It will have to work equally well for all potential outputs. Your thoughts are welcomed.

1. Authors
2. Institution Name
3. Keywords
4. Level
5. Learning outcomes
6. Material Type e.g. assignment,
7. Media type e.g. Text/HTML
8. Language
9. Conditions of use
10. Copyright holder
11. Context – description of the context from which the resource came, how it is used with students (teaching methods), relevant ‘design’ history, relevant previous experience of students
12. Use – how is it used with students
13. Benefits / Evidence of success – how does this resource benefit the student learning experience?
14. Advice and tips – preparation required for staff, students, learning environment, common issues and barriers faced by students and how to overcome them.
15. Links to other relevant C-Change resources – For example, you may be offering numerous separate resources that all emanate from a single module (e.g. 10 lectures and 2 assignments) and it will be necessary to cross-reference these.