Analysis is a process of sorting and organising data in a systematic way in order to make sense of it. The nature of the data, not the method, will prescribe the process of analysis.

Quantitative data analysis uses numbers as the unit of analysis. It usually involves the application of statistical reasoning to describe average responses, the spread of the data (dispersion), and patterns and relationships in the data.

For example, the relationship between gender and evaluations of teaching quality explored via evaluation questionnaires.

Qualitative data analysis involves preparing the data via transcription of audio or cataloguing of visual and applying an analytical process - usually coding component parts and collating similar codes into larger themes.

Secondary data analysis involves the analysis of data which already exists (called secondary sources) rather than generated by the researcher.

Researchers should recognise the entitlement of both institutions and individual participants to privacy. Anonymity refers to the extent to which participants can be individually identified during and after data collection.

Remember to think about the anonymity of groups such as courses, or a whole institution. Guarantees of anonymity can increase likelihood to participate and share honest experiences and opinions.

Surveys can be completed anonymously by not asking for personal details and not tracking to personal records.

There are risks to anonymity when collecting descriptive data (such as job titles) and asking questions which provoke answers which are very specific to a person or group of people.

You cannot guarantee anonymity during an interview a focus group, but you can anonymise the data - maybe using a pseudonym or fictionalising the participants - and instead discuss confidentiality.

Causality can be determined by using quantitative analysis. It refers to a causal relationship between two variables where a change in one is caused by the extent of another (cause-effect).

For example, peer mentoring is the cause of the change in the extent of student attainment; the change in student attainment is said to be dependent on peer mentoring.

Experimental methods and some statistical tests are good ways to explore causality. Testing for causality should apply random sampling and controls for other variables which may be influencing the relationship. Without this, conclusions should limited to either inferences or statistical tests of association and difference - one variable is connected to another but cause and effect cannot be determined (see Correlation).

Remember to check the language used in any reporting of your findings. Qualitative analysis would provide a rich, detailed description of a change, but would not make statistical claims of causality.

Confidentiality and anonymity are terms which are often used interchangeably. They are not the same, but they are often discussed together.

Ensuring confidentiality is an activity of the researcher to hold data in confidence and within the boundaries of the research process. If participants are guaranteed anonymity, this activity involves keeping their identity confidential.

There are some situations where confidentiality needs specific consideration - consider a focus group where participants are visible and topics are discussed as a group.

The boundaries of confidentiality regarding wellbeing and misconduct also need outlining, including the circumstances in which confidentiality may be breached and why.

Confirmation bias is the tendency to interpret and search for information consistent with your prior beliefs, assumptions or targets.

This could involve searching for literature which confirms your own thinking, asking questions in an interview which lead the respondents to confirm your own thinking, and selecting illustrative quotes which do the same.

For example, if a university has heavily invested funding into online self-help tools for students, there may be a tendency to look for information which only highlights the benefits of this initiative.

A robust research design and process of analysis, acknowledging any potential bias, limitations or conflicts of interest is essential.

This statistical measure of quantitative data is concerned with how closely two variables (questions) are related. You can only assess correlation when using data which is numerical, presented in internals, or in an order.

Findings can be shown visually on a graph and with a correlation coefficient and can lead to conclusions such as: “There is a positive correlation between NSS Q10 Feedback on my work has been timely and NSS Q15 The course is well organised and is running smoothly - levels of agreement one goes up in one variable and up in the other,” OR “there is a negative correlation between length of lecture and number of students attending - levels of agreement one goes up in one variable and down in the other.”

Qualitative analysis would provide a rich, detailed description of the data, but would not make statistical claims of correlation.

Critical thinking is a slippery and highly contested process.

For some it is about problem solving whereas others see it as an ongoing and challenging social process.

This latter view is reflected in the definition (adapted from Jones-Devitt and Smith, 2007) in which critical thinking is defined as:

making sense of the world through a collaborative process of questioning questions, challenging assumptions, recognising that knowledge can evolve chaotically; ultimately with the aim of continually improving thinking.

Data is information collected for a specific purpose, including research and evaluation.

A method is required to generate data.

These methods produce quantitative (numbers) or qualitative data (words / visuals).

Analysis is necessary to make sense of data or data only exists as numbers or words / visuals.

Data plus analysis create evidence.

Primary data collection refers to a process of designing a new project and collecting new data. It is different to the analysis of secondary data sources - this is data which already exists.

There are some basic characteristics of evaluation:

• it is structured and planned
• it is objective and goal focused
• it gathers and analyses evidence to help make decisions about things. These decisions may be about interventions, activities and initiatives, and provide actions in response to questions such as “are they working in the way we had hoped?” or “are they value for money?”

In a university, researchers and students may be involved in evaluation, for example projects in local communities which aim to raise awareness and facilitate access into higher education.

Remember that evaluative evidence will be only one type of evidence used to create an evidence base and inform decision making - also watch the entry for evidence-informed decision making.

Data gathered through structured research and evaluation is only one component of evidence. Evidence can also include stakeholder values and perspectives, organisational context and practitioner reflections which are collected more informally.

This triangulation of sources can provide a robust rationale for change and can help to eliminate bias which may appear in a single source.

There is a known difference between evidence-based and evidence-informed decision making. The use of the term evidence-based decision making assumes a privilege of quantitative research and evaluation collected by experts.

Evidence-informed decision making takes a more critical and flexible appraisal of the context in which the evidence is being applied.

You may prefer to use evidence informed to acknowledge the importance of the sometimes messy and unsystematic data gathering that can occur within higher education.

A gatekeeper is a person who stands between the researcher and a potential participant. Gatekeepers are able to control who has access, and when, to the participant.

There are numerous gatekeepers in a university. If you are looking to collect data from students on your course you may want to use the first five minutes of a lecture to advertise the opportunity to participate. The lecturer would need to agree to this - this is known as granting access - and they would be known as a gatekeeper.

Those who administer and analyse institutional surveys or collate notes from Student Rep meetings also act as gatekeepers of that data who you will need to influence them in order to gain access.

Gatekeepers can also help recruit participants, but this can bias the research and should be carefully considered.

The General Data Protection Regulation (GDPR) is a European Regulation which formed a new framework for regulating personal data in the UK from 25th May 2018. This replaces the Data Protection Act 1998.

All data controllers and organisations collecting or in any way “processing” personal data must now comply with this Regulation.

All UK Universities work within this Regulation, and this applies to anyone collecting and storing personal data, including students. Check your institution website for more details, and in particular the Student Privacy Notice which outlines the legal basis for processing personal student data.

Generalisation describes the extent to which research findings can be applied to settings other than that in which they were originally tested.

Large surveys which employ random sampling techniques are able to generalise findings from the sample to the wider population.

Qualitative data does not claim to produce findings which can be generalised as it does not collect sample data which is representative of the wider population.

Rather, it places merit in the depth of understanding gleaned about the specific setting being researched.

A hypothesis is a specific statement which relates to a research problem.

It is a statement framed as a suggested answer to a research question and would use an evidence base to support this assumption. Your research findings are then used to empirically test whether your hypothesis was correct.

An example could be: Students are more likely to complete a survey when the email request comes from a known and trusted contact.

A hypothesis is most commonly used in quantitative research and involves statistical testing. In evaluation you may hear reference to a theory of change, which is an approach used to rationalise an intervention and its intended outcomes.

The British Association of Educational Research suggests that voluntary informed and ongoing consent is the condition by which participants understand and agree to their participation, and the terms and practicalities of it, without any duress, prior to the research getting underway.

It should be made clear to participants that they can withdraw at any point without needing to provide an explanation.

Consent forms can be handed out to interview and focus group participants at the start of a session with an information sheet which clearly outlines the task and requirement of the participants.

Informed consent to participate in a survey is assumed once the participant clicks ‘submit’. This should be outlined at the start and end of the survey. If the online survey does not ask for any personal details and is anonymous, participants will not be able to withdraw their data - this should also be made clear.

A methodology is the justification for the methods used to carry out the research. This will include a theoretical justification of your approach including the overall design, how you will recruit a sample, and how your data will be analysed.

The methods refer to the practical steps taken to collect your data.

A survey and a focus group are examples of a method.

A pilot study is often carried out before the main data collection to test the feasibility of the method.

They help to develop and refine research instruments and procedures, including the skills of the researcher.

As students are becoming over-researched, it is important that any study works well and makes the best use of their time.

A pilot study can often highlight practical issues, such as timing and suitability of the research environment, which you cannot accurately estimate.

Proportionality is making choices about the design of evaluation or research which are appropriate to, for example, the scale, cost, risk and potential impact of an intervention: ie according to Parsons, ‘whether and how use of sources and design choices are ‘proportionate’ to the circumstances being evaluated and the decision-making needs’.

In practice, those collating evidence should try to find the most robust method possible for the context and try not to be driven by budget and timetable. In reality, this means managing expectations of key stakeholders and re-negotiating resourcing if the work is compromised.

For example, a £5 million spend on a Student Hardship Fund may require an evaluation with a range of quantitative outcome measures and qualitative interpretations to explore impact. This may need to be carried out by an independent evaluator (rather than self-evaluated by the Finance Dept) and may run longitudinally over a number of years to include more than one student cohort.

In comparison, a series of bi-monthly outreach workshops to encourage routes into teaching may be evaluated using pre and post-workshop questionnaires, designed, delivered and analysed by the Outreach Team.

Quantitative data is expressed numerically and has been generated using a structured and rigid data collection method. This means that the focus of the questions and the units for analysis have been prescribed by the researcher (e.g. closed questions in a survey) or an information management system (e.g. official student records data).

The aim of quantitative data is to quantify variability in a large sample and look for patterns, trends over time, correlations and sometimes causality and generalisability to a population through statistical analysis.

Qualitative data relies on the interpretation of the data by the researcher. The data collection is more flexible and allows participants to add value to the data by directing the content. Qualitative data can be words (e.g. from an interview, focus group or a written document) or visuals (e.g. a photograph or artwork).

The intention is to create a rich interpretation of emotions and perceptions, often including reflections over a period of time.

Closed questions ask the respondent to choose from a number of predetermined options. An open question allows the respondent to write in their own answer.

Don’t underestimate how long it takes to design good questions! Questions should avoid ambiguous, inappropriate or prejudicial language, and should control bias by avoiding leading questions.

There are various methods you can employ to evaluate an activity, intervention or initiative. They can utilise either quantitative or qualitative data. Qualitative methods can describe and interpret the context and outcomes, quantitative methods focus on measurement and prediction. Experimental evaluation methods can use quantitative data to attempt to establish cause and effect. In a randomised control trial (RCT), effects of one or more interventions are assessed by randomly dividing a research population into an experimental group and a control group. The experimental group and control group should be similar, and only access to the intervention by the experimental group should differ. This allows any impact to be attributed to the intervention. For example, implementing a flipped classroom approach to half of a module cohort, and a traditional lecture approach for the other. Impact can be measured using statistical analysis of assessment scores. Although championed as good evaluative methods, robust RCTs are difficult to achieve in educational settings due to complexity and the ability to ensure comparable groups. There are also associated ethical implications of withholding access to beneficial interventions for the control group. RCTs require a great deal of thought, design and planning, and if done well can illuminate what works in a given context.  Evaluators may also want to consider the range of other possibilities for evaluations including realistic notions of what works, and for whom.

Research is defined as activity which seeks to contribute new insights to a body of knowledge.

The research process would include the identification of a research problem and a research question, and the selection of the most appropriate methodology to help answer it.

The tools used within a methodology are often referred to as research instruments.

Those who are involved in the data generation are often referred to as research participants. Research also includes the dissemination of findings and consideration of impact.

For example, institutional research in a university may be conducted to better understand why some students withdraw from their studies.

Response rates are most often applied to survey data collection and refer to the number of surveys completed as a proportion of those that were eligible to complete it.

It is only possible to calculate a response rate when you know the total number of eligible respondents.

For example, an institutional response rate for the National Student Survey may be set a 70% target. This means that it is hoped that 70% of eligible students will complete the survey.

A sample is a selected target group for participation in your research. A sample is drawn from a wider population (all possible respondents).

You should have a rationale for your sample and think carefully about how you will access them. Your choice of sample relates to your research problem and how you intend to explore it.

You may choose to sample students on a specific course, by a demographic characteristics such as gender or ethnicity, or randomly by selecting every other student sat in a lecture hall.

Survey fatigue, sometimes called respondent fatigue, refers to the deterioration of the quality of survey data as the participant tires of the process.

Surveys fatigue can occur in-survey and can have an effect on the answers are given - selecting answers without consideration (all B’s) or repeating a ‘Don’t Know’ answer will have an impact on your overall findings.

Survey fatigue can also occur across a survey population, when multiple requests are sent to the same potential participants.

Without an oversight or survey strategy, important student surveys may incur low response rate or answers which are influenced by agitation.

Synthesis is a process which follows analysis and moves towards a more comprehensive critical evaluation.

This stage of thinking recognises the limits of existing knowledge upon which to build new explanations.

This could include the need to examine gaps in the evidence and a discussion about what new evidence is needed in the future.

There are four types of triangulation - methodological triangulation, data triangulation, theoretical triangulation, and researcher triangulation.

The aim of triangulation is to view evidence from more than one perspective.

Methodological triangulation involves using more than one method (for example, a student survey AND a student focus group) or the same method more than once, to compare and contrast findings collected from the same group of participants.

Data triangulation involves the use of different sources of data, for example, from different groups of participants or data collected within a different time or space.

Theoretical triangulation involves using more than one perspective. For example, applying feminist theory to a proposal would guide the data collection and analysis.

Finally, researcher triangulation would use more than one researcher to compare interpretation and provide a check for any bias that might be apparent.

During the analysis of data and consideration of the emerging evidence base it is necessary to verify the quality and credibility of the sources and the process which was used to generate them.

Validity refers to the accuracy of the data and whether it is the most appropriate for answering the research question.

Reliability refers to the design of the research instrument and the extent to which the same results would be generated by the instrument if the data collection was to happen again. These terms are much easier to apply to quantitative data.

Qualitative data uses an assessment of authenticity - what biases may have affected the data collection? Consider who the researcher is and their relationship to the participant.

And trustworthiness - to what extent can you trust the data you have collected? Keeping a researcher diary or reading transcripts with participants are two strategies which can be applied to mitigate.

Some of these verifications are much more difficult to secure with data from secondary sources.