PUMR - Self-assessment, Formative and Summative Evaluation, Coaching Support

Self-assessment, evaluation and coaching recognises the different positions in which participating institutions will find themselves, and the consequently different configurations of needs that institutions will have. The PUMR Core Team and those drawn from its 60 plus PASCAL Associates from around the world, are all experts in their fields.

We recognise that there is no one formula that fits all. However, we offer as a starting point a well-validated tool (UPBEAT) for self-assessment and to facilitate development. It has been developed by the PUMR Academic Director, Professor James Powell and used in a range of international contexts. It is designed to capture common ideals, strategic innovations, ideas and paradigms that can be shared to the mutual benefit of universities across the globe. The assessment, evaluation and coaching support are at two levels.

 

Self-Assessment Process

Level 1 offers basic access to networking and knowledge sharing through the dedicated PUMR programme sections on the PASCAL website (this is private to participating organisations) and the opportunity for self-assessment as a potential PUMR - using the Self-Assessment Schedule, which will be sent to those agreeing to a Level 1 contract with PASCAL. Once completed, it is returned to the PUMR Core team. We will either - for those already engaged at a relevant quality and level - fast track to validation as an obvious PUMR or more usually, recommend the support of our team of experts to identify the university’s development needs within the UPBEAT framework.

 

Evaluation - Formative and Summative

Level 1 also offers access to understanding how the UPBEAT tool can be used effectively for evaluation purposes. The PUMR Core team will help the institution to select ten case studies, which represent the highest levels of current external engagement. In addition, the team will assist the institution in evaluating one of the case studies. Through these processes the client receives an introduction to the use of UPBEAT for its own formative evaluation.

Level 2 offers support from the PUMR Core team with a full evaluation and developmental programme. It is intended for those institutions not yet ready for validation but wishing to develop outreach activities from their current position .Again, the UPBEAT tool is used for formative evaluation - at Level 2 this is carried out across all 10 selected case studies of outreach work - as the university develops an extended programme to attain higher levels of engagement in preparation for becoming a PUMR. The tool is then used for summative evaluation when the formal assessment is made.

 

Coaching

Coaching is offered in conjunction with one case study at Level 1 and at Level 2 with the full formative and summative evaluation of all ten areas of work. It is provided for participating people, as required, with the purpose of raising the standard of current practices to PUMR standards, using the UPBEAT tool to evaluate the progress. For the coaching and final summative evaluation, PASCAL pairs one university academic, who has also held a senior position in university leadership of academic enterprise or outreach, together with one senior person from an external agency with skills appropriate to the university being evaluated to become a PUMR. 

 

PUMR Programme experts

Every member of the PUMR programme staff, including the Core Team, is an expert in his or her field. They are drawn from all over the world, from all levels of government, from business, from industry, from voluntary organisations and of course, from every sector of education. Their collective expertise is exhaustive because where specialists are required who are not already Associates, our network is such that we can locate them.

The PUMR Programme experts:

• conduct formative (Level 1 and 2) and summative (Level 2 only) evaluations according to the processes developed specifically to support PUMR evaluation and support (ie  UPBEAT)
• provide coaching support (Level 2 only) to help those universities who want to become PUMRs but are not presently ready
• provide coaching (Level 2 only) to all those in the PUMR network to attain higher levels of engagement, discuss their work through the PUMR programme’s virtual social networking and showcase their work on the PUMR programme website.

At Level 1, the role of the experts is to check university policy and strategy documents to see if they conform to the principles of the PUMR process. They help in the choice of the appropriate 10 case studies which demonstrate best practices across the university. They suggest the level of support and evaluation the university needs and with reference to one of the case studies in the context of UPBEAT, coach university staff in the production of an example of what needs to be undertaken. Where necessary the experts carry out, with the client, a formative evaluation of one of the cases, including working with the client to produce a report.

At Level 2, the experts discuss the depth required of the 10 case studies with the client and work with them to identify and provide the coaching needed to meet the PUMR programme UPBEAT aims. The experts work with the client to develop full formative evaluations of the case studies. These formative evaluation write-ups/reports of activity/outcomes for each case study are then used to guide the university towards best practices. The coaching and the project final report of activities will of course be different in each place, as there is no particular end point for the programme, but we have a simple PUMR format which takes this into account. Participants can join at any time, and the length of time over which they are involved will be variable, depending on needs and level of participation. Once the university feels it is ready to be formerly assessed, it submits its report to PASCAL (based on its write up of Policy, Vision, Mission and Strategy) and its ten cases conforming to PUMR principles. PASCAL then undertakes the summative evaluation

 

UPBEAT Evaluation

The PUMR programme’s preferred evaluation procedure is UPBEAT which is written up fully elsewhere in this web-site, but PASCAL are not precious about this. Should a university have its own processes of assessment against the PUMR principles then PASCAL is more than happy for this to be used instead. However, it is important that, as part of a university’s vision, mission and strategy, that the principles underlying its engagement with business and the community reflect the PUMR programme’s aims and objectives, namely:

Universities which harness local, regional and global talents through deep and meaningful collaborations to identify worthy problems, create innovative and cost effective solutions and co-produce with external partners from business, industry, civil and voluntary services and the community, successful systemic deliverables enabling real improvement for lasting impact

Underpinning PASCAL’s view of the detailed need for PUMRs is that academics should demonstrate key enterprise skills, namely: Solution Enabling, Individual Creative Talent, Intelligent Partnering and New Business Acumen.

Solution Enabling addresses the issues of ideas, need, creativity and solutions; Individual Creative Talent addresses concepts of leadership, motivation, skills and capabilities; Intelligent Partnering looks at team development, networks and the governance and operational principles that make these relationships work; New Business Acumen focuses on making the ideas applicable in the real world, considering not only the “product” but also the mechanisms by which the product is delivered to the end user together with the resources and processes which are marshalled to ensure that the idea is effectively delivered. Each of these skills can be linked to different aspects of innovation. Table 1 below shows the broad definitions for UPBEAT that are included in the PUMR Handbook given to those who enrol in PUMR.

 

Validation

Those universities who have an appropriate vision, mission and strategy reflecting PUMR principles and at least ten cases supporting best practices reflecting this will be known at ‘PASCAL Universities for a Modern Renaissance’.

Table 1 – Descriptions for the Fourfold of Enterprise Sub-Skills

Skill	Theme	Description
Solution Enabling	Understanding Need	This describes how the team or individual has developed their understanding of the need for the solving of a particular real world problem. They recognise deep needs, but also how to turn the need into a solution demanded by a significant audience. This can range from basic recognition to a deep understanding of the problem based on multiple iterations of a possible range of solutions. ‘The real problem is to know what the real problem is….and then keep extending and refining it so that solutions reach a wider audience, to a higher level, with greater impact
	Developing Solutions	This describes how the team or individual has developed ideas, created solutions and tested them in the real world to refine and further develop newer and improved solutions with more generic and far reaching impact. Higher levels of enterprise engagement suggest increasing levels of innovation and creativity applied to the problem domain with greater effect and impact.
Individual talent	Leadership	This looks at the role of leadership within the project, how it is developed and articulated with regards to the project.
	Skills and Capabilities	The skills and capabilities to deliver the ideas or projects must be considered. This looks at the necessary functional components of the skills necessary to cope with the best solutions satisfying both the needs and demands of the project
	Coaching and Learning	Coaching and learning looks to how the talent of the individuals chosen to be in the creative team are being developed and the processes that are in place to manage these; this is both functional, managerial, leadership  and creative team working skills
	Motivation	Motivation considers the understanding and meeting of the drivers for individuals to be engaged within the project and how the leaders of the academic enterprise project build on these
Intelligent partnering	Teamwork	This considers how the team is understood and managed. This considers issues such as team roles and responsibilities, shared objectives and processes to manage these interrelationships; how the team moves from a recognition of the need for collaboration, through creative team-working, to innovative partnering and to forming strategic alliances.
	Networks	This addresses how the team works with and communicates with appropriate networks related to the problem domain at both an academic and non-academic level.
	Communication >> Conversations	This addresses how the team communicates with each other and their wider stakeholder groups to make them aware of the activity that they are engaging with; then onto the development of an appropriate discourse and useful conversations
	Co- identification of the final problem, Co-creation, and Co-Production	This looks at how teams work together in terms of jointly determining the development of solutions and ideas. This can range from very loose association from industry participation on steering groups, joint development of solutions and strategic partnerships.
New business acumen	Demand	This is concerned with the process of understanding and designing solutions that meet demand of the target group. This means solutions that not only meet the need, but are also understood and appropriable by the end users.
	Impact	This addresses the issue of understanding, measuring and maximising the level of impact of the solution. This requires robust approaches that feedback into the solution.
	Resources	Recognition of cost in use and development. The difference between cost and price is critical. Furthermore resources must be effectively acquired and managed to ensure that solution delivery is possible. Entrepreneurial academic leaders must therefore learn ways of acquiring funding from a pluralism of sources, not just the traditional ones.
	(Business) Management	These are the processes that are concerned with effective management of (business) process issues which support the effective performance of the team. This may include basic issues such as invoicing, or more complex issues such as spin out companies