Chapter contents
Chapter summary
This chapter will help you understand outputs, outcomes and impact, how to measure them and how to manage the data. We include some insights and examples from existing parent-infant relationship teams. At the end of the chapter there is a table of measurement tools describing their properties and utility for clinical assessment and outcome measurement.
You can download this chapter as a PDF here, or simply scroll down to read on screen.
Good feedback is the key to improvement.
Bill Gates [1]
The purpose of measuring outcomes
There are several good reasons to invest in measuring outcomes:
1. Being sure the intervention is safe and works
Many sensible ideas to improve the world turn out to be unexpectedly harmful when their outcomes are measured. The most notable example is the Scared Straight programme, an American programme to deter at-risk young people from committing crime, but which led to increased recidivism [2].
There is increasing attention being paid to the “dark logic” of interventions [3] , whereby well-intentioned programmes have unintended negative consequences.
Outcome measurement is therefore essential to ensuring safe, effective practice.
2. To assure funders that the work delivers the desired outcomes
In most circumstances, the clear reporting of outcome measures is crucial to the maintenance of funding. Some interventions do not achieve all their expected outcomes but may achieve some unexpected ones. Funders are usually keen to understand what outcomes their money is achieving and so may want clear explanations/ training on what the measures tell them.
3. Quality improvement
Collecting, reviewing and understanding outcomes is an essential part of the quality improvement cycle.
4. Understanding what works for whom
Outcome measures help teams to better understand which interventions work for which groups of families.
The Early Intervention Foundation has produced “10 steps to Evaluation Success“ [4] which helpfully covers a range of evaluation topics including Theories of Change, logic models and testing for effectiveness.
Defining outputs, outcomes and impact
There are lots of different definitions of outputs, outcomes, impacts and the relationships between them. These terms are unhelpfully used interchangeably. The following information is not presented as a definitive text, but as a helpful guide with references to further sources of useful information.
Outputs are what your service produces [5] as a result of your activities. Activities are the things you do (e.g. individual work or groups) and outputs are what those activities generate. They are usually easy to measure because they are described in volume terms, e.g. six families attending a group; one family attended four individual sessions; two training courses, etc. You might say “We ran four Mellow Babies groups (activities) which were attended by thirty parents in total (outputs)”.
Outcomes are the effect, value or achievements that result from of your work. They are usually described in change terms, e.g. 30% improvement in parental sensitivity; 15 people have now qualified; five points reduction in anxiety, etc. Outcomes should not just be “a sandwich of good intentions” [6] ; they should be what your work is focussed on. Some commissioners adopt the Outcomes-Based Accountability (OBA) framework to ensure providers are focussed on delivering outcomes. If this is the case with your commissioners, we would recommend the National Children’s Bureau report on OBA [7] as a starting point and that you enquire about whether there is local OBA training available.
Short-, medium- and long-term outcomes
In our example Theories of Change, we use “short-term outcomes” to describe the outcomes that come about during the intervention, such that they can be seen or measured by the end of the intervention.
We use “medium-term” to mean after the intervention is finished (exactly how long depends on a number of factors including the nature of the intervention and what length of follow-up is planned). “Long-term outcomes” are different in that they are outcomes for a population, community or society and so this is the same as “impact”. Hence, impact (long-term outcomes) tends to be the cumulative result of your short- and medium-term outcomes having been sustained over the long term.
Long-term outcomes (“impact”) are the hardest to measure since they are what we hope our efforts will accomplish but are often uncertain, unpredictable or too long-term to measure.
Outcomes and impact should be presented with your audience’s priorities in mind.
Long-term outcomes are affected by multiple factors so it is rarely possible to say your intervention definitely, and solely, caused the long-term impact, more that the work contributed to it. For example, a crime prevention initiative in 2019 may have contributed to, but not been the sole cause of, a reduction in offences in 2020: the appalling bad weather was another contributory factor (persistent rain is a known factor in reducing offence rates).
Medium- or long-term outcomes might be used as Key Performance Indicators (KPIs) by your funder/commissioner. Theories of Change lay out your evidence-based theoretical arguments as to why your short- and medium-term outcomes can feasibly be thought to contribute to KPIs, so ensure you think about local strategic priorities when creating your Theories of Change.
Distinguishing between short and medium outcomes helps practitioners and evaluators better understand when they should measure outcomes appropriately.
Failure to think carefully about when outcomes are likely to come about runs the risk of measuring outcomes too early or too late which can lead to ill-informed conclusions about the effectiveness of a service.
Activities, reach, outputs, outcomes and impact might be described like this:
We have delivered 286 individual sessions for referred families.
These sessions were delivered to 28 different families living in the CV34 and CV35 postcodes.
68% of the sessions were ‘attended as planned’, 20% of sessions were ‘cancelled in advance’, 12% of sessions were ‘not attended and no prior cancellation’.
20 (71%) families were assessed as having an improved parent-infant interaction between their first and last session, using the KIPS scale; 22 (79%) parents reported a clinically-significant improvement in mental health as measured by the HADS.
At six-month follow-up, 24 (86%) infants were assessed as demonstrating improved initiation of interaction with their parent; of 8 infants who had been on a Child Protection Plan at time of referral, 6 had come off their plan with no further social care involvement.
All 28 families seen have benefitted in some way; 86% of parent-infant dyads show improved interaction, an early precursor of improved emotional regulation which is proven to contribute towards school readiness; six infants (75% of those on a Child Protection Plan at time of referral) came off plan within six months of their last session. This contributes towards the local priority of reducing the number of children requiring long term social care input.
Sourcing evidence about outcomes and impact
Research in Practice (rip.org.uk) is a charity which helps organisations and individuals in England and Wales to access, understand and apply evidence in their work with children, young people and families. They bring together findings from academic research, the expertise and insights of practitioners, and the expertise and experiences of children and families.
RiP have created a model of Evidence-Informed Practice [8] to represent these three sources of evidence. Their members can access learning resources and opportunities via RiP’s national Partnership network.
The Research in Practice model of Evidence-Informed Practice relates to individual practice but we also recommend it as a useful blueprint for the collation of outcomes evidence: one third of the information should come from research and academic evidence, one third information from practice expertise (i.e. practitioners’ views of the work) and one third information from service users/beneficiaries and other stakeholders.
These three components can be translated into relevant questions, such as:
- Is the team delivering work that is based on the latest research and evidence?
Chapter 2 The Case for Change and Chapter 4 Clinical Interventions and Evidence-Informed Practice can help you answer this. - Do expert practitioners consider this to be valid and effective work?
The responsibility to ensure clinically valid and effective work falls to the team’s clinical lead. The Parent-Infant Foundation is currently developing a set of Quality Standards for teams which will be co-created with practitioners. - Does the data from a range of stakeholders including service users/beneficiaries show the work to be effective?
The information below focusses on collecting and analysing data from a range of sources.
Evidence relating to clinical-level vs system-level outcomes
Outputs, outcomes and impact can be measured at different levels to provide insights into different aspects of the team’s work.
Clinical-level refers to changes in individual families, system-level refers to changes in the wider system around the team, including at a community or local population-level. This is not a rigid distinction, simply a suggested way for you to approach evidence gathering and reporting methodically.
Clinical-level evidence includes the number and types of sessions delivered to individual families and changes in pre- and post- intervention clinical scores e.g. the percentage increase of parental sensitivity over time. System-level evidence includes the number of local workers trained or offered consultations and how that work has been rated or created change locally in the system, for example by increasing professional skills in identifying children at risk.
Your clinical-level and system-level Theories of Change should map onto the same long-term impact. Theories of Change help clarify the team’s purpose and clinical objectives and how they lead to the desired impact. They can also help with decisions such as which interventions to use and which training to invest in. We strongly recommend parent-infant relationship teams develop their own Theories of Change, ideally with local stakeholders, or use our templates (system-level in Chapter 3, clinical-level in Chapter 4).
An example of a system-level Theory of Change: the impacts of specialised parent-infant relationship teams on a local system
- At least 15% of new babies experience complex or persistent relationship difficulties with their parent/carer(s). Without specialised help these unresolved problems can undermine a range of life outcomes and families may require future specialist interventions including in the most severe cases, a child being taken into care
- Unresolved parent-infant relationship difficulties can be passed on to future generations of parents leading to inter-generational distress and additional high costs to the public purse
- The complex and persistent nature of some parent-infant relationship difficulties are beyond the scope of universal or typical early help support, and need specialised, multi-disciplinary intervention
- Frontline practitioners may lack confidence or awareness to identify early relationship problems and provide or refer families to appropriate support
- The right kind of specialised help may not be available locally
- Local leaders, including commissioners, may be unaware of the importance of parent-infant relationships or face a lack of local strategic co-ordination in supporting the work
- A variety of direct therapeutic work to address and improve the difficulties in the parent-infant relationship
- Training, consultancy and campaigning to raise public and professional awareness and improve workforce capacity to protect and promote the parent-infant relationship
- Act as “systems champions” by facilitating local networks and working with local leaders and organisations to improve awareness, co-ordination and decision-making
- Improved parent-child attunement and interaction (a direct outcome of work with families and an indirect outcome of work with other professionals)
- Improved capacity for the public and professionals to identify and support babies and their parents
- Improvements in how organisations work separately and together, so that babies can receive timely and appropriate support
- More children benefit from a sufficiently secure and nurturing relationship with at least one parent/carer
- Local cost savings as fewer children need to be referred to speech therapy, early help, children’s services, CAMHS, paediatrics, or special educational needs services for problems rooted in parent-infant relationships
- More children experience better social, economic, physical and mental health outcomes across the lifecourse
- Fewer children move into the Looked After system
- Fewer children need mental health support as older children or adults for attachment-related difficulties
- Fewer families experience the transmission of parent-infant relationship difficulties into the next generation
An example of a clinical-level Theory of Change
- Not every child has access to a sufficiently secure relationship with at least one permanent adult carer
- Unresolved parental traumas from the past (“ghosts in the nursery”) or present can be translated into parental states of mind that get played out in maladaptive ways and these damage the interactions with the baby
- Aspects of the parent’s behaviour can lack sensitivity or capacity for appropriate responsiveness leading to distress in the baby
- Aspects of the baby’s behaviour can trigger unresolved traumas in the parent, leading to stress or lack of pleasure from parenting
- Address the states of mind and interactional behaviour of the parent that negatively impact the baby. Give meaning to why these occur and how they can be changed
- Improve reflective functioning and parental capacity to provide emotional regulation for their infant
- Improve infants’ capacity to engage confidently and feel secure with parent
- Offer families a variety of direct therapeutic approaches (with the parent-infant dyad but sometimes also with the family triad, the parental couple without the baby and/or with parents individually) which:
- Address parental unresolved traumas, current stressors, anxieties and risk factors
- Support parents’ strengths to improve parental sensitivity, mentalisation and reflective functioning
- Signpost and facilitate contact with a range of other services which can address current stressors (such as housing, financial stress, substance misuse, parental conflict/relationship strain)
- Decreased traumatising behaviour by the parent towards the baby, reduced sense of stress with the baby, improved parental empathy, consistency and motivation
- Parent and infant feel safe with each other, improved warmth in the interaction, improved attunement and more developmentally appropriate interactions
- Improved infant invitation and initiation of interaction with adults including parents
- Improved assessment and support of the family’s needs, child protection issues and the parent’s capacity to change
- Improvements in parent’s capacity to sustain emotional and behavioural self-regulation
- Quality of parent-child relationships for indicated child and siblings is improved
- Child is more relaxed, with improved social and emotional development
- Improvements in parents’ openness to trusting relationships with helping professionals and in the effectiveness of professional assessment and support
- Improved likelihood of child securing better physical and mental health, social, emotional, cognitive and language development
- Reduced risk of child needing referral to speech therapy, early help, children’s services, CAMHS, paediatrics, or special educational needs services for problems rooted in parent-infant relationships
- Reduced risk of transmission of parent-infant relationship difficulties into the next generation
Gathering local data and evidence
There is a range of ways to gather objective local evidence about outcomes, including:
- Setting clinical goals and reviewing progress against them, e.g. Parent and Baby Outcomes Star™ [9] or the Goals-Based Outcomes (GBO) from Children and Young People’s IAPT dataset
- Using self-report or observational measurement tools and tracking progress over time, e.g. HADS or ASQ:SE2
- Gathering quantitative participant feedback e.g. training evaluation feedback forms
Setting and monitoring clinical goals
Many practitioners set goals with the families they work with as a way of mutually agreeing the terms and focus of the work. How well the goals have been achieved by the end can be measured as an outcome. This can be done effectively in an informal way but there are some helpful tools and information available if required from sources such as Child Outcomes Research Consortium (CORC) [10] .
The current CORC advice around setting and monitoring clinical goals is:
- Set the goals over the first three sessions of the intervention / assessment
Some clients come with very clear ideas of the goals they want to achieve, others take a little longer to decide. It may not take three sessions to agree goals with clients but all goals, if they are going to used as measures for outcomes, should be fixed in the first three sessions. - Record up to three goals
Three is probably a good number of goals to be getting on with but it’s not a limit. Record how close the family feels they are to reaching the goal at the outset of the work on a scale from zero to ten where ‘zero’ means the goal is not met in any way, ‘ten’ means the goal is met completely and a rating of ‘five’ means they are half way to reaching the goal. - Review regularly and reflect
Reviewing the goals in order to discuss progress can be done at every session, or frequently throughout the work. At the end of the work, record how close the family now feels they are to their goal, on a scale 0-10. The difference in scores between the start and end of the work provides a numerical measure of progress. Whilst not specific to parent-infant work, the goal setting worksheet by Choices in Recovery [11] demonstrates how a simple goal setting sheet can be user-friendly.
The ABCPiP practitioners in Ballygowan, Northern Ireland are trained in the Parent and Baby Outcomes Star™, a licensed tool to support early conversations with families about what their hopes are for the work, the issues they want to work on and to track progress against these issues over time.
Roberta Marshall and Janine Dougan, clinical co-managers of the team, told us “we do find the Parent and Baby Outcomes Star a useful tool, not just for measuring progress but also for having a helpful conversation about what the family want from our work together”.
More details can be found at: http://www.outcomesstar.org.uk/usingthe-star/see-the-stars/parent-andbaby-star/
It is worth noting that the Early Intervention Foundation views outcome stars as useful tools for engaging parents and discussing progress, but not as a validated way of assessing outcomes [12] .
Using self-report or observational measurement tools to track progress over time
Some measurement tools are suitable for both clinical assessment and pre- and post-outcomes measurement, and this can save time and effort on the part of practitioners. Some clinical assessment tools are not statistically validated for test-retest situations and therefore, strictly speaking, should not be used for outcomes measurement. However, the parent-infant relationship field is not replete with validated, low-cost measures which are quick and easy to administer and score, so some of the most clinically-useful tools are put to use locally as outcome measures. The alternative would be practitioners using additional academic outcome measures on top of their clinical assessment tools which would be burdensome and impractical, but one should bear in mind these statistical limitations.
At the end of this chapter, you will find tables of information about self-report, interview and observational measurement tools relevant to the work of specialised parent-infant relationship teams. Where available, we have included weblinks for further information about evidence, how to acquire the tool and where to receive training.
The PIP teams’ approach to assessment and outcome measures
The description below may help you to think about how to construct your own suite of assessment and outcome measures.
The information here is not intended as a prescriptive approach to outcome measurement: the parent-infant relationships sector does not have one standard set of recommended tools.
Where available, the gold standard outcome measure for parent-infant work is formal assessment of attachment security. This provides a reliable and clear indication that change has occurred in the parent-infant relationship. However, this is often impractical for routine clinical use, as it can be time-consuming and require special training which is often expensive.
The vast majority of specialised parent-infant relationship teams use quicker, cheaper methods such as video observation, questionnaires and information-gathering from other sources.
The nine Parent Infant Partnership teams (PIPs) used a collection of measures chosen both for their clinical application and for the way they ‘triangulate’ the infant and the caregiving relationship. These demonstrated to potential partners and commissioners that there were good, evidenced, measures in place for service evaluation.
All scores are collected on the Parent-Infant Foundation Data Portal in a way that removes “personally-identifying details”.
1. Changes to the family’s levels of risk and stress
The Risks and Stresses checklist developed by Gloucestershire Infant Mental Health Team [13] is used by those referring into a PIP team, and is later updated by clinicians, to profile the details of those families who are engaged with the service. These are factors in parents’ lives that can have a negative impact on the caregiving relationship.
2. The quality of the caregiving relationship
This is assessed using the ‘Levels of Adaptive Functioning’ (LOAF) section from Zero to Three’s DC:0-5. This assesses the prime caregiving relationship as well as the wider caregiving environment. It also provides detailed guidance for diagnosing a relationship-specific disorder in infancy.
3. Child’s social and emotional development
The Ages and Stages Questionnaire (Social and Emotional; 0-2) produces a score which can be compared to the benchmark cut-off for each age; above this indicates a serious difficulty.
The ASQ:SE2 can be used to demonstrate that the infant has attained, or remained on, an acceptable pathway of social and emotional development in a situation when this might be jeopardized. A reduction in the mean score, whether below or above the cutoff, indicates an improvement in social and emotional functioning.
4. Parental Mental Health
The Hospital Anxiety and Depression Scale (HADS) has seven questions each for anxiety and depression and takes about 5 minutes to complete. It enables early identification of both, each of which can leave less space in the caregiver’s mind for the baby.
5. Parenting Interactivity
The Keys to Interactive Parenting Scale (KIPS) is a coded video observation and gives a way of assessing twelve different aspects of parenting behavior from about 10 -15 minutes of interaction. KIPS produces clinically-useful information that may be fed back to caregiver using interaction guidance since it can pinpoint
clearly defined strengths. Here too the mean score is significant, with any increase showing an observable improvement in the interaction between caregiver and child.
6. Parent satisfaction questionnaire
A final parent-completed satisfaction questionnaire covers the parent’s experience and observations. They can also be used with a change of tense from ‘was’ to ‘is’ to monitor the therapeutic contract while treatment is in progress. Graded answers on a Likert rating scale means that responses can be recorded quickly and easily and there should also be space for free text.
Gathering quantitative participant feedback
Traditional participant feedback/parent satisfaction questionnaires tend to lean towards factors that affect ‘acceptability’ rather than measuring outcomes, although outcome questions can also be included. For example, training feedback forms tend to ask about the venue, the agenda or the speaker (satisfaction/acceptability) rather than whether the delegate has learned anything new (short-term outcome).
Without outcomes questions, these forms tell us little about the effectiveness of a service in bringing about identified outcomes/changes but can be valuable in gauging the temperature of a person’s experience or getting feedback about specific aspects of service provision.
‘Participants’ might include families who have participated in therapy, colleagues who have participated in consultation or delegates that have participated in training. There are lots of examples on the internet of participant feedback/parent satisfaction questionnaires which generate numerical data, we have provided some examples of parent evaluation feedback forms and training evaluation forms in the Network area of the Parent-Infant Foundation website. Evaluators may also find the Kirkpatrick Model [14] helpful.
There are more examples at https://www.sampleforms.com/parentfeedback-form.html.
A note about electronic administration
Survey Monkey, MS Forms and other free, specialised software can make the administration, scoring and analysis of simple forms quicker and easier via computer or tablet.
A word of caution: some published questionnaires are only free to use in paper format and require a licence to be acquired from the author before electronic administration can be used.
Electronic administration may assist you to better support parents who experience language, literacy, or sensory barriers.
Gathering the expertise and insights of stakeholders
The remaining two-thirds of the Research in Practice model of Evidence-Informed Practice relate to gathering the expertise and insights of two important groups of stakeholders: practitioners and families. Obviously, teams might also like to extend their evaluation reach to other stakeholders such as the local children’s workforce, commissioners and other teams such as CAMHS.
This type of evidence can be collected through quantitative methods such as questionnaires or surveys, but this can miss the richness and nuance of qualitative feedback.
Using the same training evaluation feedback form for every training course, irrespective of the topic, makes data collection and comparison easier.
In all data collection activity, it is crucial that the method and tools are sensitively designed to be appropriate to the audience and to the questions you are asking. There are numerous ways to collect information from practitioners and service users/beneficiaries, here are just a few:
- Ask practitioners to agree a closing statement with families which reflects the important aspects of their work together, the critical ingredients of the work that led to progress or the reasons for lack of progress, how the family and practitioner will remember the work and what it has meant to each of them. These vignettes can be anonymised and collated
- Ask families to draw or write on sticky-notes their experiences of the work. In groups, this can be a collective activity to include photographs and messages to one another, to form an album or poster
- Invite families to provide written or photographic accounts of their experience of working with the parent-infant relationship team
- Invite families to an informal and carefully facilitated participation forum, where they can feedback to leaders about their experience of the service
- Invite families to participate in telephone feedback sessions
- Ask practitioners to identify key areas of focus and mechanisms of change in a particular piece of therapeutic work. Map these against the Theories of Change to see if they align [15]
- Collate anonymised parent letters and notes, of both thanks and of complaints
- Ask stakeholders to complete a timeline of their recent experiences of the parent-infant team. This can easily be done with lining paper and felt tips. It is a useful activity
- to understand the temporal links between what has been delivered and the outcomes it has led to
- Ask stakeholders collectively to create an image of the system as it exists now (or before the parent-infant team existed) and how they would like it to look (or how it does look, now that the parent-infant team has been working for some time). This works well with small groups, such as a strategic board or local health visiting team
- Families sometimes agree to be filmed talking about the service they received
- Families can be invited to coffee mornings at the team’s base or places where they work, such as the Neonatal Intensive Care Unit (NICU) to talk about service
- Some families could be invited to attend professionals’ planning meetings to share their experience of the service
We acknowledge that this toolkit does not cover service user participation in any detail. The Parent-Infant Foundation is keen to gather experiences from across the Network and is engaging with other partners to support the development of resources on this topic.
We hope to have more detailed content for the review in a years’ time.
Statistical analysis of quantitative data
Local reporting requirements do not usually extend beyond having to show pre- and post-intervention changes in questionnaire scores. The clinicians in the parent-infant team are usually able to provide commentary about the clinical significance of such changes. However, for more formal purposes, such as planning to publish data in a journal or present it at an academic conference, some teams seek the reassurance of statistical analysis.
Psychologists are typically trained in statistical analysis although do not always have easy access to statistical analysis software. The Royal Statistical Society may be able to advise teams and, for those located in charities, may be able to offer some support from one of their pro-bono statisticians (www.rss.org.uk).
Data management
Outputs data
Output data is relatively straightforward to collect as it is simply counting activities and numbers. Most specialised parent-infant relationship teams use either data management software, such as the Parent-Infant Foundation’s data portal, or spreadsheets that keep track of outputs as they occur. This is typically data provided by practitioners to the administrator for input, simple analysis and periodic reporting.
In our experience, teams often want to analyse their data by certain categories, for example how many referrals were antenatal vs postnatal, how many referrals related to children on a child protection plan, etc. This helps teams answer questions such as “are we doing enough to raise awareness with our midwifery colleagues?” and “should we start a dialogue with children’s services commissioners about the increasing demand for work with babies?”. There is a balance to be struck between the burden of recording and analysing many data fields versus the utility of the data. We recommend that every field of data you collect is ‘actionable’ i.e. that it can be and is used to inform improvements during regular review.
Outcomes data
Some data management systems integrate clinical record keeping functions with the ability to collate and report clinical scores. System 1 is an example of a widely-used, large data management system (in the NHS) which can be adapted locally to collect pre- and post-intervention scores. Essex Partnership University NHS Foundation Trust have done this for their new Together with Baby parent-infant relationship team to assist outcome measure data collection. Where data management systems can’t do this, teams may be left to create their own spreadsheets which link families’ clinical records to separate databases of scores through the use of a unique identifying code number.
The Parent-Infant Foundation’s data portal is a free software offer being developed for early 2020. It will not offer a clinical record-keeping function but will allow teams to upload and draw reports on outcome measure scores easily, and to compare their own data to an anonymised, aggregated data set from other teams. This will save time in that teams will not have to design their own spreadsheets, and it will help teams to benchmark their own data against that of others.
For more information about how you can access this free software, please contact us directly.
Working with disadvantaged and vulnerable parents may alter engagement and attendance data.
We recommend the Early Intervention Foundation’s 2019 Engaging Disadvantaged and Vulnerable Families [16] for an evidence review of this area.
Data-linkages to systems outcomes and long-term outcomes
Some specialised parent-infant relationship teams can access data from other local services and organisations which can be tracked back to the families they have worked with. For example, in some areas of Scotland there is comprehensive collection of SDQ scores for all three-year-olds and this could facilitate interesting follow-up analyses of parent-infant relationship work.
Other examples might include the linking of a child’s parent-infant work with the team to their school readiness scores (EYFS or the new standardised reception assessments from September 2020), parental mental health screening scores as collected by health visitors, or standardised child development scores during mandated child health surveillance visits.
This kind of data sharing normally requires a formal information sharing agreement between all relevant partners which covers consent issues and GDPR responsibilities.
A word of caution about long-term or distal outcome measures: these are influenced by a range of factors such as the quality of childcare during pre-school years, or the socio-economic experiences of the family and are therefore not generally used by clinical teams to evidence outcomes of one specific early-life intervention, unless already linked through their local authority. This echoes the points made above that Theories of Change are necessary to show how your interventions are believed to be linked to the intended long-term outcomes but you may not be able to prove causality through outcome measurement.
Cost-benefits data
At a local level, generating valid and reliable cost-benefits data about a team or intervention is cost-prohibitive for funders and we have yet to find a team which has been funded to complete such an analysis.
Globally, the parent-infant relationship research base is not sophisticated enough to generate meaningful calculations about the cost-benefits of how specific interventions map onto outcomes which would be relevant to UK commissioners. This does not mean that we cannot describe potential cost-savings or the general principle that prevention saves money, but it does make it difficult to attribute exact figures.
Even well-established universal measures, such as the Early Years Foundation Stage assessment cannot yet be confidently used to assess the cost-benefits of interventions in the first 1001 days. The Parent-Infant Foundation is currently working with economists to think about how to strengthen the research and data in order to begin to address this area.
The Nobel Prize-winning work of James Heckman embeds the principle that effective interventions pay the greatest returns on investment the earlier in the life course they are applied. There is ample neuroscientific evidence linking the quality of parenting and parent-child interaction to child development outcomes, and the Adverse Childhood Experiences studies demonstrate a clear life-long impact of childhood adversity. See Chapter 2 The Case for Change for further information.
At a local team level, qualitative information about individual families, referral patterns and working practices which demonstrates positive change will support the principle if not the detail of cost-savings. For example: where social workers have been able to remove a child from a Child Protection Plan following the family’s work with the team; where, following training, health visitors feel more confident to work with parent-infant dyads without referral to CAMHS; where a parent who has had a previous child removed into care is able to keep a subsequent child following work with the team; where specialised consultation has helped a social worker craft a more effective family support offer.
Reporting outcomes
Commissioners give us some consistent messages about how they prefer impact to be reported.
1. Co-creation
The format of how you report your outputs, outcomes and impacts is best co-created with the audience, in this case your commissioners or funders (some have standardised templates and reporting schedules).
Commissioners and funders want to understand how your work contributes to their local strategic priorities so this point needs to be very clear in your reports.
For example, Leeds wants all children to be in safe, supportive families and to reduce the need for children to be taken into care. These are key impact areas that the Infant Mental Health Services’ outputs and outcomes fit within, alongside a range of other services and programmes.
If a local priority is ‘preventing children going into care’ or ‘improving school readiness’ you should explain in your report how the team’s work contributes to this.
Commissioners and funders usually want reporting requirements to be proportionate and not a huge burden on clinical resource, whilst answering key questions on impact. They are often willing to negotiate how reporting best achieves that balance.
2. Style and content
Funders and commissioners welcome concise, clearly-presented quantitative and qualitative data with clear, brief explanations of what the data means and what the measures can tell us. Visual devices and infographics may be helpful.
We would recommend the LivPIP/Parent Support Service Social Impact report [17] as an excellent example of a periodic impact statement. We have other examples of more regular reporting formats in the Network area of our website.
Remembering the importance of the work to families
This excerpt is from a mother who was first seen while pregnant and at the end of the work with the parent-infant team many months later, she was asked to comment on what aspects of the intervention she had found helpful. (All identifying details have been changed.)
Specifically, this mother felt she had been helped by considering her own childhood, having a time to think about her baby and their burgeoning relationship, being given information on development and simply the sense of being listened to and understood. At a six month follow up things were still going well. In her own words:
"Before meeting K, I was extremely anxious about carrying a baby successfully and also worried about how my fears would impact on our relationship once she was born. K was able to talk me though those worries and give reassurance when I started to lose my confidence and give into the anxiety.
Once Annie was born, meeting with her helped to set aside worries and assure me of Annie’s normal development. She also gave insight into how Annie might be experiencing the situation and how to help her cope with the newness of being in the outside world. K’s kind and patient manner and ability to convey practical and clinically-based rationale helped me transition from a fearful to positive parent.
The support I received was exceptional.
I felt very lucky to be able to access the service. It was a huge transition going back to work with its own challenges.
If I hadn’t been given the chance to get a sound foundation I’m not sure I would have been able to handle the increased pressure of being a full-time working mum. If only you could give me more sleep!"
Downloads
Footnotes
[1] Bill Gates. https://www.inc.com/jana-kasperkevic/bill-gates-proper-feedback-is-key-to-improvement.html
[2] National Institute of Justice. Practice profile: Juvenile Awareness Programs (Scared Straight). https://www.crimesolutions.gov/PracticeDetails.aspx?ID=4
[3] Bonell C, Jamal F, Melendez-Torres GJ, et al. ‘Dark logic’: theorising the harmful consequences of public health interventions. J Epidemiol Community Health 2015;69:95-98.
[4] Asmussen, K., Brims, L., & McBride, T (2019). 10 steps for evaluation success. The Early Intervention Foundation, London. https://www.eif.org.uk/resource/10-steps-for-evaluation-success
[5] Fritz, J. (2019). How to Talk About Non-profit Impact from Inputs to Outcomes. Inputs, Outputs, Outcomes, Impact - How are they Different? https://www.thebalancesmb.com/inputs-outputs-outcomes-impact-what-s-the-difference-2502227
[6] Drucker, P (2011). Chapter 1 – The Commitment. The World According to Cheever Blog. https://theworldaccordingtocheever.blogspot.com/2011/03/peter-drucker-chapter-1-commitment.html
[7] National Children’s Bureau. Outcomes Based Accountability (OBA): Guidance. What is OBA and how can NCB help you with it? https://www.ncb.org.uk/resources-publications/outcomes-based-accountability-oba-guidance
[8] Research in Practice (2011). Evidenced informed Practice updated 2018.
[9] Triangle Consulting Social Enterprise Ltd. Parent and baby Outcome StarTM The Outcomes Star for perinatal mental health and well-being. http://www.outcomesstar.org.uk/using-the-star/see-the-stars/parent-and-baby-star/
[10] Child Outcome Research Consortium. https://www.corc.uk.net/
[11] Janssen Pharmaceuticals, Inc. (2015). Choosing your Goals. Choices in Recovery. http://www.choicesinrecovery.com/sites/default/files/Worksheet_1_ChoosingYourGoals.pdf
[12] Personal correspondence, 4 Sep 2019.
[13] The Wave Trust (2013). Conception to 2: Age of Opportunity. Appendix 2C p.89. https://www.wavetrust.org/conception-to-age-2-the-age-of-opportunity
[14] Kirkpatrick Partners (2019) https://www.kirkpatrickpartners.com/Our-Philosophy/The-Kirkpatrick-Model
[15] With sincere thanks to Anna Freud Centre PIP for their insights into this exercise.
[16]. Pote, I., Doubell, L., Brims, L., Larbie, J., Stock, L. & Lewing, B. (2019). Engaging disadvantaged and vulnerable parents An evidence review. The Early Intervention Foundation, London. https://www.eif.org.uk/reports/engaging-disadvantaged-and-vulnerable-parents-an-evidence-review
[17] Person Shaped Support (2017). Social Impact Report: Parent and Baby Wellness. PND (Postnatal Depression) and LivPIP (Liverpool Parent-Infant Partnership) http://www.psspeople.com/wp-content/uploads/2017/04/Parent-and-Baby-Wellness-Impact-Report-v2-27032017.pdf
This chapter is subject to the legal disclaimers as outlined here.
This chapter is correct at the time of publication, 1st October 2019, and is due for review on 1st January 2021.
