Evaluation & Insight

ORS seeks to provide considered evaluation and insight that delivers real added-value for clients. Our experienced research team can offer a range of mixed methodology approaches to address complex research objectives. The participants range from the general public and service users to senior stakeholders and policymakers.

We design bespoke projects incorporating the most appropriate qualitative and quantitative methods to meet your needs and deliver actionable insight.

This project played an important role in understanding views from residents on the Rebuilding a Greener Hackney programme, which was one of the most ambitious of its kind in the country. The research outputs exceeded expectations and were incredibly insightful, particularly those relating to behavioural change. The final report was excellent and well received within the Council.

Customer Service and Performance Monitoring

ORS regularly conducts customer satisfaction and performance monitoring. We work with clients on either an ad-hoc or continuous basis.

Annual satisfaction surveys are common amongst our local authority and housing association clients and we can include standardised questions which can be benchmarked against national datasets as well as making comparisons with previous years.

We also manage a portfolio of continuous satisfaction surveys for numerous clients, with regular monthly or quarterly reporting. Typically these projects are based on a quantitative survey of users undertaken through face-to-face or telephone interviews. We provide dashboards to monitor feedback in real-time alongside regular reporting to highlight long-term trends. We also have systems in place to provide clients with daily feedback about specific problems that respondents identify, so that these can be addressed without delay.

Our continuous satisfaction surveys are often accompanied by more detailed evaluation and insight. This often includes data analytics and analysis to identify the underlying factors that influence key metrics. Survey results can also be explored through flexible qualitative research, including focus groups and workshops as well as targeted in-depth interviews. These seek to explain what might be driving any changes and provide practical insights into how to improve satisfaction levels.

From the outset ORS were the stand-out candidates to take over our customer satisfaction survey programme. It was evident from their pitch that they quickly grasped the complexity and challenges of our business and customers and have proven to be flexible and agile when dealing with an ever-changing landscape. ORS have taken time to listen and have provided sound advice as to how we can better create actionable insights from research and have delivered quality standards throughout. On a personal level, I enjoy working with ORS and view them as colleagues rather than having a customer/supplier relationship.

Evaluations and Impact Assessments

ORS undertakes process and impact evaluations of the success of policies, interventions, and initiatives. Such evaluation and insight projects are often complex, involving multi-phased research and a range of techniques including:

  • Literature, documentation, and data reviews
  • Theory of Change
  • Secondary data analysis
  • Stakeholder research
  • Evaluability Assessment
  • Case studies and vignettes
  • Mapping and exploration of user journeys and outcomes

Our integrated teams can offer qualitative research amongst service users and senior policymakers alongside large-scale data collection and complex analysis. We aim to provide practical insights to support future policy and service improvements.

These projects are often conducted for national government bodies and, in addition to our membership of professional bodies, we work in accordance with the Magenta Book.

We have been discussing how engaging and impressive your presentation was yesterday, so thank you so much for all the work you clearly put in to it. I’m sure it was also really useful for the Youth Offending Team managers and SPs to see some of the product of their efforts to support the evaluation and critically reflect on points such as the timing of ECM and number of reviews.

Cognitive Testing

Getting the questions right is key any for any research project. It is therefore important that participants understand and interpret the questions in the way that was intended. The meaning of questions can also be different for participants from different backgrounds. This may be related to their age, ethnicity or education, or simply where they live.

A cognitive testing exercise can clarify that questions are being interpreted consistently and that any guidance or instruction is clearly understood. Several forms of wording can be contrasted and compared, and the process of evaluation and insight can also identify further alternatives through feedback and discussion.

ORS has specific expertise and experience of conducting cognitive testing of Welsh language materials to ensure that bilingual surveys retain consistency.

You've done a really good job. I think it is a really well put together report, and is a firm base for exploration of the questions in the cognitive interviews.

Data Analytics and Insight

ORS has the knowledge, skills and resources to work with ‘big data’. One of our key areas of expertise is delivering maximum value from the data or information available by undertaking extended quantitative analysis and modelling. Our evaluation and insight work for clients routinely integrates survey data with a wide range of secondary data sources. These can provide real added insight to the survey results.

We regularly work with data such as the Index of Multiple Deprivation (IMD) and other lifestyle information. These can add an important additional dimension when interpreting the findings from resident surveys. Accredited researchers are also able to access safeguarded data via the Office for National Statistics (ONS) Secure Research Service. This includes access to detailed flow data from the Census (such as commuting and migration) and a range of other ONS datasets that are not in the public domain.

In helping our clients to interpret our findings we routinely use comparators and benchmarking. We also use graphical representation techniques, such as geocoding primary data and spatially mapping the findings. The team can also produce infographics to summarise key findings in an easily accessible format.

The ORS project manager was highly professional in terms of knowledge and use of data, research methods, client responsiveness as well as dealing very effectively with issues that arose during the project. We are very satisfied that the final report was completed to a high standard and reflected our specific requirement.