Skip to main content Accessibility Statement

We work with students and staff from across the Scottish higher education sector to deliver a student-led project in each year of the Enhancement Theme. A Steering Group, which has a student chair and involves input from a range of staff and students as well as sparqs, provide direction for the project. Each project aims to develop policy, practice, resources and networks around common issues and areas of interest.


In 2018-19, the Students using evidence project explored the role of data and evidence in the work of student representatives. The project involved students and staff from a range of Scottish institutions and students associations/unions, as well as staff from QAA Scotland and sparqs.

Sector-wide activity

Guide to using evidence

Guide to Using Evidence in Higher Education

Publication Date: 11 Sep 2019


Section 7: Evidence of evidence use how do you show that you have used evidence in your decision-making

Publication Date: 11 Sep 2019



ANALYSIS

Analysis is a process of sorting and organising data in a systematic way in order to make sense of it. The nature of the data, not the method, will prescribe the process of analysis.


Quantitative data analysis uses numbers as the unit of analysis. It usually involves the application of statistical reasoning to describe average responses, the spread of the data (dispersion), and patterns and relationships in the data.


For example, the relationship between gender and evaluations of teaching quality explored via evaluation questionnaires.


Qualitative data analysis involves preparing the data via transcription of audio or cataloguing of visual and applying an analytical process - usually coding component parts and collating similar codes into larger themes.


Secondary data analysis involves the analysis of data which already exists (called secondary sources) rather than generated by the researcher.



ANONYMITY

Researchers should recognise the entitlement of both institutions and individual participants to privacy. Anonymity refers to the extent to which participants can be individually identified during and after data collection.


Remember to think about the anonymity of groups such as courses, or a whole institution. Guarantees of anonymity can increase likelihood to participate and share honest experiences and opinions.


Surveys can be completed anonymously by not asking for personal details and not tracking to personal records.

There are risks to anonymity when collecting descriptive data (such as job titles) and asking questions which provoke answers which are very specific to a person or group of people.


You cannot guarantee anonymity during an interview a focus group, but you can anonymise the data - maybe using a pseudonym or fictionalising the participants - and instead discuss confidentiality.



CAUSALITY

Causality can be determined by using quantitative analysis. It refers to a causal relationship between two variables where a change in one is caused by the extent of another (cause-effect).


For example, peer mentoring is the cause of the change in the extent of student attainment; the change in student attainment is said to be dependent on peer mentoring.


Experimental methods and some statistical tests are good ways to explore causality. Testing for causality should apply random sampling and controls for other variables which may be influencing the relationship. Without this, conclusions should limited to either inferences or statistical tests of association and difference - one variable is connected to another but cause and effect cannot be determined (see Correlation).


Remember to check the language used in any reporting of your findings. Qualitative analysis would provide a rich, detailed description of a change, but would not make statistical claims of causality.



CONFIDENTIALITY

Confidentiality and anonymity are terms which are often used interchangeably. They are not the same, but they are often discussed together.


Ensuring confidentiality is an activity of the researcher to hold data in confidence and within the boundaries of the research process. If participants are guaranteed anonymity, this activity involves keeping their identity confidential.


There are some situations where confidentiality needs specific consideration - consider a focus group where participants are visible and topics are discussed as a group.


The boundaries of confidentiality regarding wellbeing and misconduct also need outlining, including the circumstances in which confidentiality may be breached and why.



CONFIRMATION BIAS

Confirmation bias is the tendency to interpret and search for information consistent with your prior beliefs, assumptions or targets.


This could involve searching for literature which confirms your own thinking, asking questions in an interview which lead the respondents to confirm your own thinking, and selecting illustrative quotes which do the same.


For example, if a university has heavily invested funding into online self-help tools for students, there may be a tendency to look for information which only highlights the benefits of this initiative.


A robust research design and process of analysis, acknowledging any potential bias, limitations or conflicts of interest is essential.



CORRELATION

This statistical measure of quantitative data is concerned with how closely two variables (questions) are related. You can only assess correlation when using data which is numerical, presented in internals, or in an order.


Findings can be shown visually on a graph and with a correlation coefficient and can lead to conclusions such as: “There is a positive correlation between NSS Q10 Feedback on my work has been timely and NSS Q15 The course is well organised and is running smoothly - levels of agreement one goes up in one variable and up in the other,” OR “there is a negative correlation between length of lecture and number of students attending - levels of agreement one goes up in one variable and down in the other.”


Qualitative analysis would provide a rich, detailed description of the data, but would not make statistical claims of correlation.



CRITICAL THINKING

Critical thinking is a slippery and highly contested process.


For some it is about problem solving whereas others see it as an ongoing and challenging social process.


This latter view is reflected in the definition (adapted from Jones-Devitt and Smith, 2007) in which critical thinking is defined as:


making sense of the world through a collaborative process of questioning questions, challenging assumptions, recognising that knowledge can evolve chaotically; ultimately with the aim of continually improving thinking.



DATA

Data is information collected for a specific purpose, including research and evaluation.


A method is required to generate data.


These methods produce quantitative (numbers) or qualitative data (words / visuals).


Analysis is necessary to make sense of data or data only exists as numbers or words / visuals.


Data plus analysis create evidence.


Primary data collection refers to a process of designing a new project and collecting new data. It is different to the analysis of secondary data sources - this is data which already exists.



EVALUATION

There are some basic characteristics of evaluation:

  • it is structured and planned
  • it is objective and goal focused
  • it gathers and analyses evidence to help make decisions about things. These decisions may be about interventions, activities and initiatives, and provide actions in response to questions such as “are they working in the way we had hoped?” or “are they value for money?”

In a university, researchers and students may be involved in evaluation, for example projects in local communities which aim to raise awareness and facilitate access into higher education.


Remember that evaluative evidence will be only one type of evidence used to create an evidence base and inform decision making - also watch the entry for evidence-informed decision making.



EVIDENCE

Data gathered through structured research and evaluation is only one component of evidence. Evidence can also include stakeholder values and perspectives, organisational context and practitioner reflections which are collected more informally.


This triangulation of sources can provide a robust rationale for change and can help to eliminate bias which may appear in a single source.


There is a known difference between evidence-based and evidence-informed decision making. The use of the term evidence-based decision making assumes a privilege of quantitative research and evaluation collected by experts.


Evidence-informed decision making takes a more critical and flexible appraisal of the context in which the evidence is being applied.


You may prefer to use evidence informed to acknowledge the importance of the sometimes messy and unsystematic data gathering that can occur within higher education.



GATEKEEPER

A gatekeeper is a person who stands between the researcher and a potential participant. Gatekeepers are able to control who has access, and when, to the participant.


There are numerous gatekeepers in a university. If you are looking to collect data from students on your course you may want to use the first five minutes of a lecture to advertise the opportunity to participate. The lecturer would need to agree to this - this is known as granting access - and they would be known as a gatekeeper.


Those who administer and analyse institutional surveys or collate notes from Student Rep meetings also act as gatekeepers of that data who you will need to influence them in order to gain access.


Gatekeepers can also help recruit participants, but this can bias the research and should be carefully considered.



GDPR

The General Data Protection Regulation (GDPR) is a European Regulation which formed a new framework for regulating personal data in the UK from 25th May 2018. This replaces the Data Protection Act 1998.


All data controllers and organisations collecting or in any way “processing” personal data must now comply with this Regulation.


All UK Universities work within this Regulation, and this applies to anyone collecting and storing personal data, including students. Check your institution website for more details, and in particular the Student Privacy Notice which outlines the legal basis for processing personal student data.



GENERALISATION

Generalisation describes the extent to which research findings can be applied to settings other than that in which they were originally tested.


Large surveys which employ random sampling techniques are able to generalise findings from the sample to the wider population.


Qualitative data does not claim to produce findings which can be generalised as it does not collect sample data which is representative of the wider population.


Rather, it places merit in the depth of understanding gleaned about the specific setting being researched.



HYPOTHESIS

A hypothesis is a specific statement which relates to a research problem.


It is a statement framed as a suggested answer to a research question and would use an evidence base to support this assumption. Your research findings are then used to empirically test whether your hypothesis was correct.


An example could be: Students are more likely to complete a survey when the email request comes from a known and trusted contact.


A hypothesis is most commonly used in quantitative research and involves statistical testing. In evaluation you may hear reference to a theory of change, which is an approach used to rationalise an intervention and its intended outcomes.



INFORMED CONSENT

The British Association of Educational Research suggests that voluntary informed and ongoing consent is the condition by which participants understand and agree to their participation, and the terms and practicalities of it, without any duress, prior to the research getting underway.


It should be made clear to participants that they can withdraw at any point without needing to provide an explanation.


Consent forms can be handed out to interview and focus group participants at the start of a session with an information sheet which clearly outlines the task and requirement of the participants.


Informed consent to participate in a survey is assumed once the participant clicks ‘submit’. This should be outlined at the start and end of the survey. If the online survey does not ask for any personal details and is anonymous, participants will not be able to withdraw their data - this should also be made clear.



METHODOLOGY AND METHOD

A methodology is the justification for the methods used to carry out the research. This will include a theoretical justification of your approach including the overall design, how you will recruit a sample, and how your data will be analysed.


The methods refer to the practical steps taken to collect your data.


A survey and a focus group are examples of a method.



PILOT STUDY

A pilot study is often carried out before the main data collection to test the feasibility of the method.


They help to develop and refine research instruments and procedures, including the skills of the researcher.


As students are becoming over-researched, it is important that any study works well and makes the best use of their time.


A pilot study can often highlight practical issues, such as timing and suitability of the research environment, which you cannot accurately estimate.



QUANTITATIVE AND QUALITATIVE

Quantitative data is expressed numerically and has been generated using a structured and rigid data collection method. This means that the focus of the questions and the units for analysis have been prescribed by the researcher (e.g. closed questions in a survey) or an information management system (e.g. official student records data).


The aim of quantitative data is to quantify variability in a large sample and look for patterns, trends over time, correlations and sometimes causality and generalisability to a population through statistical analysis.


Qualitative data relies on the interpretation of the data by the researcher. The data collection is more flexible and allows participants to add value to the data by directing the content. Qualitative data can be words (e.g. from an interview, focus group or a written document) or visuals (e.g. a photograph or artwork).


The intention is to create a rich interpretation of emotions and perceptions, often including reflections over a period of time.



QUESTIONS - OPEN AND CLOSED

Closed questions ask the respondent to choose from a number of predetermined options. An open question allows the respondent to write in their own answer.


Don’t underestimate how long it takes to design good questions! Questions should avoid ambiguous, inappropriate or prejudicial language, and should control bias by avoiding leading questions.



RESEARCH

Research is defined as activity which seeks to contribute new insights to a body of knowledge.


The research process would include the identification of a research problem and a research question, and the selection of the most appropriate methodology to help answer it.


The tools used within a methodology are often referred to as research instruments.


Those who are involved in the data generation are often referred to as research participants. Research also includes the dissemination of findings and consideration of impact.


For example, institutional research in a university may be conducted to better understand why some students withdraw from their studies.



RESPONSE RATE

Response rates are most often applied to survey data collection and refer to the number of surveys completed as a proportion of those that were eligible to complete it.


It is only possible to calculate a response rate when you know the total number of eligible respondents.


For example, an institutional response rate for the National Student Survey may be set a 70% target. This means that it is hoped that 70% of eligible students will complete the survey.



SAMPLE

A sample is a selected target group for participation in your research. A sample is drawn from a wider population (all possible respondents).


You should have a rationale for your sample and think carefully about how you will access them. Your choice of sample relates to your research problem and how you intend to explore it.


You may choose to sample students on a specific course, by a demographic characteristics such as gender or ethnicity, or randomly by selecting every other student sat in a lecture hall.



SURVEY FATIGUE

Survey fatigue, sometimes called respondent fatigue, refers to the deterioration of the quality of survey data as the participant tires of the process.


Surveys fatigue can occur in-survey and can have an effect on the answers are given - selecting answers without consideration (all B’s) or repeating a ‘Don’t Know’ answer will have an impact on your overall findings.


Survey fatigue can also occur across a survey population, when multiple requests are sent to the same potential participants.


Without an oversight or survey strategy, important student surveys may incur low response rate or answers which are influenced by agitation.



SYNTHESIS

Synthesis is a process which follows analysis and moves towards a more comprehensive critical evaluation.


This stage of thinking recognises the limits of existing knowledge upon which to build new explanations.


This could include the need to examine gaps in the evidence and a discussion about what new evidence is needed in the future.



TRIANGULATION

There are four types of triangulation - methodological triangulation, data triangulation, theoretical triangulation, and researcher triangulation.


The aim of triangulation is to view evidence from more than one perspective.


Methodological triangulation involves using more than one method (for example, a student survey AND a student focus group) or the same method more than once, to compare and contrast findings collected from the same group of participants.


Data triangulation involves the use of different sources of data, for example, from different groups of participants or data collected within a different time or space.


Theoretical triangulation involves using more than one perspective. For example, applying feminist theory to a proposal would guide the data collection and analysis.


Finally, researcher triangulation would use more than one researcher to compare interpretation and provide a check for any bias that might be apparent.



VALIDITY / RELIABILITY / TRUSTWORTHY / AUTHENTICITY

During the analysis of data and consideration of the emerging evidence base it is necessary to verify the quality and credibility of the sources and the process which was used to generate them.


Validity refers to the accuracy of the data and whether it is the most appropriate for answering the research question.


Reliability refers to the design of the research instrument and the extent to which the same results would be generated by the instrument if the data collection was to happen again. These terms are much easier to apply to quantitative data.


Qualitative data uses an assessment of authenticity - what biases may have affected the data collection? Consider who the researcher is and their relationship to the participant.


And trustworthiness - to what extent can you trust the data you have collected? Keeping a researcher diary or reading transcripts with participants are two strategies which can be applied to mitigate.


Some of these verifications are much more difficult to secure with data from secondary sources.



Examples of practice

Webinar: Students using students' data


Data is an important tool. It provides us with the evidence we need to enhance the student learning experience, but how much do we involve our students in data processes? What are the barriers, and what are the potential solutions? Sorcha Kirker (Vice-President for Higher Education, Highlands and Islands Students Association) talked us through student interaction, engagement with, and use of data and the problems faced in engaging students. It also looked at the positive elements of student involvement, and explored a case study of good practice. The webinar also drew on the work of the Enhancement Theme Student-led project, by exploring the different ways that academic representatives systems generate, use and share data and evidence to improve the student experience.

Students using students' data

Publication Date: 19 Jun 2019

Students Using Students’ Data Engagement Activities

Publication Date: 19 Jun 2019




We have been gathering examples of innovative and effective practice from students. We will add more case studies as the project progresses.



Enhancing learning and teaching through Student-led Teaching Awards

Publication Date: 11 Sep 2019

Using nomination data from Student-led Teaching Awards to identify and share good practice

Publication Date: 11 Sep 2019

A Look into the minds of Year 1 Students

Publication Date: 25 Apr 2019

Supporting a step change in the use of MyAberdeen (the university’s VLE) from a students’ perspective

Publication Date: 25 Apr 2019

School-specific careers survey

Publication Date: 25 Apr 2019

Student Course Evaluation Form (SCEF) What do staff and students really think about SCEF

Publication Date: 25 Apr 2019

Understanding what gets students to engage

Publication Date: 25 Apr 2019




Curated resources

Developing practice using evidence and data


We have selected some examples of other resources that students and students association staff might find useful in developing their practice and policy around the use of evidence and data.


  • School of Data | School of Data is a global network committed to advancing data literacy in civil society
  • UK Data Service Data Skills Modules | These introductory level interactive modules are designed for users who want to get to grips with keys aspects of survey, longitudinal and aggregate data
  • Analysing Qualitative Data | A short guide to analysing qualitative data by AdvanceHE, such as focus group transcripts or free-text survey responses, for practitioners working in universities and colleges

  • Seeing Data | Resources to help non-experts develop their ability to make sense of data visualisations
  • Visualising Data | Wide range of resources and links to visualisation tools and information  
  • Data Stories podcast | A podcast on data visualization