Australian Government: Attorney-General's Department
Australian Government: Attorney-General's DepartmentAchieving a Just and Secure Society

Framework for evaluation and longitudinal research - March 2007

INTRODUCTION

INDICATORS, OBJECTIVES AND EVALUATION QUESTIONS

EVALUATING INDIVIDUAL COMPONENTS OF THE REFORM PACKAGE - OVERVIEW AND DATA SOURCES

EVALUATING INDIVIDUAL COMPONENTS OF THE REFORM PACKAGE - FACSIA RESPONSIBILITIES

EVALUATING INDIVIDUAL COMPONENTS OF THE REFORM PACKAGE - AGD RESPONSIBILITIES

EVALUATING THE WHOLE PACKAGE OF REFORMS

BIBLIOGRAPHY

 

Introduction

This document outlines a proposed framework for the evaluation of the Australian Government's new family law reform package. This framework has been developed by the Australian Institute of Family Studies at therequest of the Attorney-General’s Department (AGD) and the Department ofFamilies, Community Services and Indigenous Affairs (FaCSIA).

The Government implementation plan identifies the separate components of the reform package as:

  • Changes to the Law (Family Law Act 1975 and consequential changes to the Federal Magistrates Act 1999, the Income Assessment Act 1997 and the Marriage Act 1961)
  • Family Relationship Centres
  • The Family Relationship Advice Line
  • Family Relationships Online
  • Expanded Early Intervention Services1
  • Expanded Post-Separation Services2
  • Combined Court Registries, and
  • Community Education Campaign.

The Institute’s brief excludes the Combined Court Registry, the Community Education Campaign and the 30% increase in funding for established FRSPservices. However, to the extent to which these components of the reform package contribute to the overall impact of the reforms, they will be captured in the wider evaluation.

The proposed framework has been designed to provide information for the following:

  • Initial evaluation of the operation of the first 15 Family Relationship Centres, Family Relationship Advice Line and Family Relationships Online (July – December 2007)
  • Contribution to administrative reviews of the new and expanded servicesas required
  • Full program evaluation (by approximately 2013), and
  • Ongoing longer-term evaluation (10 year time horizon).

Back to top

Background: The family law reform package

The Parliamentary Report Every Picture Tells a Story suggested that community dissatisfaction with family law process was widespread and thatthe wider family law system was in need of reform. The Report identified a number of features of the existing family law system that were problematic. These included unclear entry points into the family law system, a bewildering and largely uncoordinated range of related services for separating parents (previously likened to a ‘maze’), as well as a perceived systemic bias againstparents sharing the care of children after separation.

A number of the Report’s recommendations addressed the need for the enhancement of services to assist former partners to develop child-focussed post-separation parenting arrangements. In particular, the Report drew attention to the fact that children do poorly when they are subjected to ongoing entrenched parental conflict and linked the continuation of conflict in a percentage of cases with adversarial styles of dispute resolution. It noted that a significant number of children whose parents had separated were growing up without the involvement of one of their parents. At the same time, the Report noted that not all parenting relationships are beneficial to children and pointed to the need to make early identifications of violence andchild abuse and to deal with these expeditiously.

In June 2005, the Australian Government accepted the bulk of the Report’s recommendations and consequently announced the largest ever investment in reforming the family law system: $397 million over four years.

The reform package can be categorised into three strands of measures:

  1. Changes to the law – the introduction new legislative principles which promote shared parental responsibility and the continuing involvement of both parents in children’s lives (subject to the child’s best interests); phased-in compulsory use of alternative dispute resolution mechanismsbefore access to the Family Court is permitted, with exceptions; and less adversarial processes for the courts in the determination of matters involving children. The reforms also modify the context, requirements and environment in which existing and new services will be delivered.
  2. New services – the establishment of new service types, including 65 Family Relationship Centres (15 having commenced operation in July 2006; and 25 more being introduced in each of the two subsequent years), the Family Relationship Advice Line (which will provide information, referral and advice to individuals over the phone), and Family Relationships Online (a web-site complementing these services).
  3. Integrated and expanded services – expanded early intervention services and post-separation services. The expansion of services includes the establishment of services in areas where they had previously not existed, as well as an increase in the level of existing services, but not the 30% increase in funding given to all existing FRSP services.

AGD and FaCSIA have joint responsibility for the implementation of the reform package. AGD has responsibility for the Family Relationship Centres, the Family Relationships Advice Line, Family Relationships Online, the expanded post-separation services and the changes to the law, while FaCSIA has responsibility for the expanded early intervention services. This evaluation framework has been developed for both the AGD and FaCSIAareas of responsibility in order to produce a whole of Government outlook. A time line for the implementation of the reform package is shown in Figure 1.

Given the pervasive nature and potentially far-reaching effects of the reforms, as well as the significant investment of Government expenditure, it isimportant that there is a thorough evaluation of the legislative reforms, thenew and expanded services funded under the package and the effect of these initiatives on Australian families.

In addition, the Community Education Program about the new family law system is expected to increase the general awareness of the new and expanded services available in the community, and thereby increase the levelof demand for, and use of, these services. The ability of the new system to respond to this increased pressure will also need to be monitored closely.

Figure 1. Time line of implementation of the Family Law Reform Package
(click on image to open the full size version )

Fugure 1. Time line of implementation of the Family Law Reform Package

Back to top

Indicators, objectives and evaluation questions

A critical part of any evaluation of the new reform package will involvedemonstrating that the new programs and policies are (a) operating inaccordance with their policy and legislative intent; and (b) achievingparticular outcomes. This section proposes a set of big picture indicators to help assess the reform’s impacts, and offers a number of questions to help guide the direction of the evaluation.3

The breadth of the reforms, and the culture shift they seek, require differentassessment timelines – short-term, medium-term, and long-term. ‘Cultural’ change is rarely achieved quickly. Indeed such change may be generational. Any evaluation strategy needs to be mindful of what to look for, and when to realistically start looking for it.

Back to top

Policy objectives

The Government’s core policy objectives of the family reform package are to4:

  1. help prevent separation and build strong, healthy family relationships
  2. encourage greater involvement by both parents in their children’s lives after separation, and also protect children from violence and abuse
  3. in the case of separation, to provide information, advice and dispute resolution services to help parents agree on what is best for their children rather than contesting parenting proposals in the courtroom, and
  4. have a new entry-point that provides a doorway to other services that families need and facilitates access to those services.

To facilitate the achievement of these objectives, the Government is putting inplace mechanisms and arrangements that:

  • support processes designed to reduce the amount of acrimony and conflict in the resolution of family disputes over children
  • require separating parents to attempt to resolve their disputes with the help of non-court dispute resolution processes and services, and
  • introduce less adversarial court processes for those cases that do require adjudication.

Back to top

“Big picture” indicators

At the broadest level, the policy and legislative intent of the reforms can be best articulated as a set of big picture indicators – both individual and systemic.

  • A shift from the idea that relationship problems necessarily imply the need for separation, to the idea that some relationship problems are capable of negotiation and resolution [Policy objective 1]
  • A shift away from discourses of separation, and disputation over children, as predominantly a legal problem towards discourses of separation, and disputation over children, as predominantly a relationship problem [Policy objectives 1 & 2]

Individual-level indicators

  • An increased knowledge of the availability of relationship support services, and an increased willingness to seek help with family relationship problems [Policy objective 1]
  • Better handling of conflict within ongoing relationships and post-separation [Policy objectives 1 & 3]
  • Increased use of non-adversarial processes, in the event of separation [Policy objectives 3]
    • in negotiations over parenting arrangements
    • within the Court systems (including hearings)
  • An increase in the exercise of “shared parental responsibility” in circumstances where this benefits the children [Policy objective 2]
  • Parenting time [Policy objective 2] – in circumstances where this benefits the children:
    • A reduction in the “little or no contact” category
    • An increase in “substantial time” category

System indicators

  • Improved accessibility, coordination and quality of services [Policyobjective 4]
    • Simplified access to family and relationship support services and dispute resolution services
    • Better coordination within the system of family and relationship support services and dispute resolution services
    • Greater capacity to recognise and respond to need
  • Increased referrals by service providers to non court-based processes in appropriate cases [Policy objective 3]
  • Decreased use of adversarial processes within the Court system [Policy objective 3]
  • Reduced proportion of parenting disputes filed in the courts [Policy objective 3]

Back to top

“Big picture” evaluation questions

The preceding indicators can also be framed as specific evaluation questions. However, not all of these questions are readily amenable to measurement.

Policy objective 1: Helping to build strong, healthy relationships and preventseparation

  • Are parents more inclined than in the past to seek help when faced with relationship, parenting or other difficulties?
  • Of couples and families who require help in building stronger and healthier relationships, has the proportion receiving help increased?
  • Did the help received contribute to building stronger/healthier relationships?
  • Have the reforms helped members of separating families to improve their relationships with each other?
  • Have the reforms helped family members improve their negotiation and conflict resolution skills with each other?

Policy objective 2: To encourage greater involvement by both parents in their children’s lives after separation, and also protect children from violence and abuse

  • Is there a reduction in the proportion of non-resident parents who report “little or no contact”?
  • Is there an increase in the proportion of parents who exercise shared parental responsibility (including shared decision-making on long-term issues relating to the children)?
  • How durable and workable are the agreements that parents reach about parenting arrangements?
  • Are violence and child protection issues better identified and more quickly and more appropriately addressed?

Policy objective 3: Helping separating parents agree on what is best for their children (rather than litigating) through the provision of useful information and advice, and effective dispute resolution services

  • Have the reforms helped family members improve their negotiation and conflict resolution skills? (Also related to Policy objective 1)
  • Have the reforms resulted in a reduced rate of entrenched conflict in separating families and an increase in sustainable agreements about post-separation arrangements?
  • Is there a decrease in the medium to long term in the proportion of separating parents filing disputes in Court? (It is possible that increased expectations around parenting after separation might -at least for a time - increase the proportion of disputes between separating parents.)
  • Has the proportion of separating parents who establish, maintain or improve their parenting skills and relationships with their children,increased since the introduction of the reforms?
  • Are separated parents more positively disposed towards each others’ parenting roles than prior to introduction of the reforms?
  • Has a higher proportion of separating parents focused on the best interests of their children?
  • When circumstances change and new opportunities arise, is there an increase in the extent to which decisions about parenting are renegotiated in a child-focused and non-adversarial manner?
  • Has the quality of relationships with grandparents and significant extended family members remained as strong, or improved, after separation?
  • Have the reforms provided timely and adequate services for those needing a service without unnecessarily involving those capable of and wanting to make their own responsible parenting arrangements?
  • Are services effective in reducing conflict and helping parents to agree on child-focused post-separation parenting arrangements?
  • Have the reforms helped separated parents to resolve disagreements about child support matters in a child-focused way?

Policy objective 4: The family law system should have a highly visible entry-point which operates as a doorway to other services, and helps families to access these other services.

  • Do a higher proportion of families across Australia have recognisable, easy and timely access to the range of services that will meet their specific needs? (that is, ‘the right service at the right time’)
  • Are the various components of the reform package operating as part of an integrated family law system – with actively facilitated and appropriate cross-referrals?
  • Have the reforms led to a greater proportion of child-focused agreements and a reduced proportion of filing in courts?
  • Are litigated cases involving children dealt with in a less adversarial, child-focused manner?
  • Have the reforms improved people’s experience of family law services?
  • Has case law continued to evolve in the direction of more individually tailored child-focused arrangements?

More broadly:

  • To what extent do the family courts function in accordance with the legislative principles relating to:
    • the benefits to children of having a safe and meaningful relationship with both parents
    • the benefits to children of having a safe and meaningful relationship with grandparents and other people who are significant to the children
    • protection of children from violence and abuse
    • the parties attempting to resolve the dispute before filingapplications
    • less adversarial proceedings in children’s cases (Division 12 A)
  • To what extent, and in what ways, have the changes affected the wellbeing of children and their parents?
  • To what extent have the principles that encourage parties to resolve disputes before filing applications, led to other important consequences – whether positive or negative?
  • Have there been any other unintended consequences (beneficial orotherwise), and what are they?
  • How durable are any of the observed effects?

In relation to the ‘culture’ shift aim noted earlier:

  • What were the public expectations of the reforms, and has there been a better understanding of the child-focused aims of family law?
  • Did public acceptance and knowledge of non-adversarial post-separation dispute resolution processes increase over time?
  • In the long term, have more child-focused approaches to the resolution offamily stress and family conflict occurred?

It is important that measures that allow each specific evaluation question to be answered are either readily available from existing data or able be collected without too great an investment of resources.

Back to top

Evaluating individual components of the reform package – overview and data sources

The next three sections provide a framework for evaluating the individual components of the reform package. This section provides an overview of the components of the family law reform package that are to be evaluated and details the sources of data that would potentially be available. Section 4 then considers in more detail how the components that FaCSIA have responsibility for will be evaluated, while Section 5 does the same for the components for which AGD have responsibility.

The type of evaluation is likely to vary between the different components. In particular, the evaluation methodology for the various service delivery components will differ markedly from the evaluation methodology with respect to changes to the law. Also, the target population of interest captured by the evaluation of each component will differ, as different services are directed at different groups.

The key components to be examined here are as follows:

  • changes to the law
  • Family Relationship Centres
  • the Family Relationship Advice Line
  • Family Relationships Online
  • expanded early intervention services, and
  • expanded post-separation services.

The evaluation of each component may make use of one or more of three types of data:

  • program (throughput; service use etc)
  • in-house service provider client feedback data, and
  • talking with people (surveys).

Of these, surveys are likely to provide some of the most powerful insights because they can be tailored to particular evaluation questions. Nonetheless all are useful, especially if a set of common questions and response options are included withinand across these different domains.

The evaluation of the reforms will need to draw on data from a number of sources. The data that are needed fall into two types: (i) program data (also termed system data), and (ii) survey data. The program data are important because they allow for speedy assessment of client profiles, service use, referral flows, and so forth.6

Back to top

Program data

Important sources of program data include:

  • FRSP Online data (including new data to be collected by the FamilyRelationship Centres)
  • Call Management System (AGD) and Centrelink telephony data
  • AGD data, including Legal Aid, National Association of Community Legal Centres and Indigenous legal and family violence services
  • Casetrack and court files (Family Court of Australia (FCoA) and Federal Magistrate Court (FMC))
  • CSA data (including the Child Support Evaluation Dataset)
  • Centrelink data
  • National Legal Aid data, and
  • data from relevant State/Territory services.

While these data sources should provide valuable information for evaluation of the family law reform package, a number of issues would need to be addressed including those relating to privacy.

Back to top

Survey data

Potential sources of survey data include:

  • ABS surveys – Family Characteristics Survey; General Social Science Survey; Personal Safety Survey
  • the Household, Income and Labour Dynamics in Australia (HILDA) survey
  • the Longitudinal Study of Australian Children (LSAC)
  • datasets held by the Australian Institute of Health and Welfare
  • allegations of violence in family law matters (AIFS)
  • various other AIFS surveys, including Caring for Children after Separation (Waves 1–3) and the Fertility Decision Making Project, and
  • surveys of legal professionals and other service providers (e.g., Family Law Section members, Family Court personnel and Family Dispute Resolution practitioners).

Back to top

New baseline data

Given that the roll-out of the first 15 Family Relationship Centres began on 1 July 2006, and the Family Law Amendment (Shared Parental Responsibility) Act 2006 came into effect at the same time, it was crucial that any baseline data collection exercise occur prior (or as close as possible) to that date. Three studies have recently been conducted as part of a baseline data collection program conductedby the Institute: a survey of parents in the general population; a survey ofseparated/divorced parents; and a survey of family lawyers.

General Population of Parents Survey

The General Population of Parents Survey (GPPS) was a nationally representative survey of 5,000 parents living in private dwellings, across Australia, selected randomly and stratified by gender and geographical location. This sample included a large proportion of separated parents. The sampling unit was a household containing an English-speaking parent with at least one child aged less than 18 years. The 15-minute telephone interview examined, among other things: attitudes towards seeking professional help for parenting and relationship difficulties; post-separation parenting issues; and grandparents’ relationships with the children.

Caring for Children after Parental Separation (Wave 3)

This Institute survey comprises approximately 500 separated parents with at least one child under 18. The survey was initially conducted in 2003, and containsdetailed information on patterns of parent–child contact. A second wave of data, focusing on attitudes to child support, was collected in 2005. A third wave of data is currently being collected. Wave 3 draws largely on the GPPS described above but contains more detailed information about post-separation parenting (including the occurrence of major disputes about post-separation parenting arrangements).

Survey of lawyers

The effectiveness of the reforms will, in part, be dependent on the awareness and understanding of family law service providers, in particular family lawyers, and the willingness of this group to practice in the new family law environment. However, there is little information in Australia about the practices of family law specialists and the attitudes of this important group toward the reforms, particularly those relating to non-adversarial dispute resolution.

An internet-based survey has recently been conducted of around 400 family lawyers who are registered with the Family Law Section of the Law Council of Australia. This survey focused on lawyers’ views of various key aspects of the new family law system.

The utility of baseline data

These surveys, and their replication at subsequent points in time, are likely to be a key component of the evaluation strategy.7 They will help assess the extent to which the new policy measures individually, and in concert, are achieving their intended objectives in terms of appropriateness, effectiveness and efficiency.Information from this baseline data collection can be used to provide feedback to policy makers, service managers and practitioners for further development of thenew family law system, including policy, systems and operational positions for both AGD and FaCSIA.

Back to top

Supplementary data

Smaller-scale, more qualitative, data may also be useful, although more care is needed because they are unlikely to be generalisable. These data may nonetheless provide useful clues with respect to the manner in which the post-reform changes are developing.

Back to top

Summary and conclusion

A considerable amount of family law data (program and survey) exist, a great deal of which may be useful as a benchmark in order to assess the effectiveness of the new family law system. However, the evaluation of the reform package cannot be achieved without the collection of new data that will emerge as the system unfolds. The next three sections set out some proposals for the collection of these new data and how they can be combined with the existing benchmark data to evaluate the reform package.

Back to top

Evaluating individual components of the reform package – FaCSIA responsibilities

Within the new family law system, FaCSIA has responsibility for early intervention services funded under the Family Relationships Services Program(FRSP). Early intervention services aim to strengthen family relationships by providing all family members with the relationship skills to effectively manage change and develop problem-solving skills to prevent family breakdown. Services assist family members across the full continuum of the family lifecycle by providing relationship education, parenting skills (with children of all ages), advice on issues arising when families blend and a range of other day-to-day services to sustain relationships over time.

The individual services for which FaCSIA has responsibility are:

  • Mensline Australia
  • Men and Family Relationships Services
  • Family Relationships Counselling (joint responsibility with AGD)
  • Family Relationships Education and Skills Training, and
  • Specialised Family Violence Services.

The expanded early intervention services will be primarily evaluated against the first of the policy objectives set out in Section 2, of helping families to build strong, healthy relationships and so help prevent the separation of parents in the first instance.

A key element of the evaluation of the early intervention services will be anassessment of how effective the service has been in strengthening relationships and preventing separation (Policy Objective 1). For those parents who go on to separate despite the intervention, the effectiveness of the early intervention services can be assessed in terms of the skills they provide separated parents to subsequently develop sustainable parenting.

To evaluate the effectiveness of the early intervention services against Policy Objective 1, it would be important to have a similar group with which to compare, who had not accessed the intervention services and so identify the differences between the groups in terms of relationship strength and prevalence of relationship break-down. In reality, locating such a comparison group (that is, people with similar types and degrees of relationship strain prior to separation thathave not accessed early intervention services) will be highly problematic and to be undertaken in a comprehensive manner would require significant resources.

For parents that do go on to separate, the impact of accessing, or not accessing, early intervention services prior to separation, on parents’ ability to deal with the consequences of separating on themselves and their children (Policy Objectives 2 and 3), can be assessed more readily because the population of separated parents is more readily identifiable. Once identified, separated parents can simply be asked which, if any, early intervention services they have used in the past, and reasons for non-use (where relevant).

Strategies that are relevant to evaluating early intervention services are:

Survey data – General Population of Parents Survey (GPPS)

The General Population of Parents Survey (GPPS) was conducted in July 2006 with a sample across Australia of 5,000 parents with a child under-18 years. The data provide an opportunity to examine help-seeking behaviour among parents that are experiencing difficulties within the family unit. In particular, the 2006 survey collected information about any help that was sought in relation to behavioural problems of children within the family, or in relation to separating from a partner. It also asked whether parents had ever felt their relationship with their partner was in serious trouble, although it did not go on to ask about any help parents may have sought to overcome these difficulties (except in the case of an actual separation).

A proposed repeat of the GPPS in mid-2008 will provide population level data to monitor help-seeking attitudes and behaviours. This data will be compared with the data from the 2006 GPPS. However, as noted in Section 2, cultural change is rarely achieved quickly. Indeed, it may take years for changes to be apparent(across different cohorts of parents) concerning such matters as help-seeking behaviour, the extent to which there is ready acceptance of a child-focused approach to family conflict and to post-separation parenting arrangements, the extent to which relationships between the children and both sets of grandparents are fostered, and so on.

Back to top

Service provider manager qualitative interviews

Face-to-face interviews with managers of Family Relationship Centres and early intervention and post-separation services will be undertaken. These interviews will identify managers’ expectations regarding the usefulness and effectiveness of the service, its relationship with other services, difficulties faced, areas requiring improvement and possible ways of improving the service.

These interviews will provide qualitative information from service managers on changes in attitudes, expectations and views about overall service effectiveness. It will also provide information on the interaction between family relationship services and other services, including the appropriateness and effectives of referrals as perceived by the managers. This information will improve understanding of the overall impact of the family law reform package.

Back to top

Staff surveys

Three online surveys of staff of the new and expanded services will be undertaken in 2007, 2008 and 2009 to ascertain staff perspectives on the same issues discussed by managers.

Back to top

Client surveys

A sample of 2000 clients for each of the Family Relationship Centres, early intervention services and post-separation services in each of the 2007-08 and 2008—09 financial years will be surveyed, subject to their agreement.

Back to top

Further information

A selection of organisational audits, status reports and program data (FRSP Online) held by FaCSIA will also be available to inform the evaluation.

Back to top

Evaluating individual components of the reform package – AGD responsibilities

AGD is responsible for all remaining components of the New Family Law System, as set out in Section 3. With the exception of Family Relationship Centres (whose target population includes couples entering relationships, intact families as well as separating families), the target population for AGD is separated families.8 The components to be evaluated are:

  • Family Relationship Centres
  • the Family Relationship Advice Line
  • Family Relationships Online
  • expanded Post-Separation Services, and
  • changes to the law.

The post-separation services refer to:

  • Family Dispute Resolution
  • Children’s Contact Services, and
  • Parenting Orders Program.

As with the early intervention services, these post-separation services already exist as part of the FRSP but under the family law reforms have been expanded both into new locations and with respect to the level of services inexisting locations.

The following relate more specifically to possible data sources, processes and questions that are germane to the individual components listed earlier.

Back to top

Family Relationship Centres

Assessing the effectiveness of Family Relationship Centres (the Centres) will follow a similar approach to that of the early intervention services, although in the case of Centres there is not the complication of assessing an expansion of an existing service as the Centres are being set up for the first time.

Action research

The funding model encourages Centres to build on their strengths, and to engage in a diverse range of practice delivery models. This, along with each Centre needing to evolve and refine its service delivery methods, points to the utility of a separate program of action research at the Centres. Action research is a tool typically used by services to improve service delivery. As a re-iterative process, strategies are implemented, and their effectiveness is assessed, which may then lead to further refinements that are evaluated.

Back to top

Family Relationship Advice Line

Data that are relevant to evaluating the Family Relationship Advice Line are telephony data, call management system data, the use of test calls, and referral behaviour from other components. Staff and client surveys are also important; for example, a module on the use of the Advice Line could be incorporated into a survey of FRC users.

Back to top

Family Relationships Online

As for the Advice Line, system data can be used for performance monitoring and implementation (process evaluation). In addition, a web-based survey of users could be undertaken. Users could also be invited to participate in a telephone survey. This could form the basis of action research (e.g., improvement in navigating the website).

Back to top

Expanded Post-Separation Services

Assessing the effectiveness of the services themselves is relatively straightforward in that a similar approach to that outlined in Section 4 for the early intervention services can be adopted. However, as previously stated, evaluating the effectiveness of the expansion of services is more complex as it is unlikely that we will be able to see if any pre-and post-expansion differences are a function of the expansion itself.

Back to top

Changes to the law

The legislation provides an important basis for identifying what questions should be addressed in the evaluation of the reforms.

Firstly, it contains numerous specific directions as to how different categories of people should behave: parents, litigants, lawyers, counsellors9, and judicial officers. The evaluation could usefully assess the extent to which those directions are complied with.

Secondly, the legislation also implicitly indicates a range of behaviours that it wishes to encourage. These behaviours are in some cases implicit in the specific directions. (For example, the Act explicitly requires courts to consider certain things when making parenting orders, and it is reasonable to say that it implicitly directs litigants and their legal representatives to address those matters in their evidence and submissions.) They can also be identified by reference to the certain provisions of the Act that state objectives and principles.

From these legislative indications, one can derive a set of questions the answers to which should reveal the extent to which the legislation has been effective. The questions reflect the view that in evaluating any reform, it is useful to consider both whether the objectives were achieved, and also whether the change produced unintended or unexpected outcomes that compromise (or, perhaps, enhance) the benefits of the reform.

The questions deal in turn with what appear to be the main legislative themes: parental involvement; protection of children from violence and abuse; and two aspects of dispute resolution: the encouragement of people to resolve disputes before litigating, and the new ‘less adversarial’ court procedures contained in Division 12A of Part VII of the Act. Examples of these questions are as follows:

Questions about parental involvement

To what extent do the family courts function in accordance with the principles relating to the benefits to children that derive from having a relationship with both parents?

Questions about protecting children from violence and abuse

To what extent do the family courts function in accordance with the principles that relate to protection of children from violence and abuse?

Questions relating to the resolution of disputes without requiring litigation

To what extent do the family courts function in accordance with the principles that encourage parties to attempt to resolve the dispute before filing applications?

Questions relating to the “less adversarial” proceedings (Div 12A)

To what extent do the family courts function in accordance with the principles that promote “less adversarial” proceedings in children’s cases (Division12A)?

Possible research methods in relation to these questions overlap considerably. They include surveys/interviews with the relevant categories of people, such as parents, litigants, lawyers, counsellors, registrars and judicial officers, as well as examination of court and other records. In relation to the questions about unexpected consequences, it is recommended that there should be open-ended components of the other research that would encourage disclosure of relevant information.

Back to top

Evaluating the whole package of reforms

While evaluating each component of the reform package separately is crucial, it will also be important to evaluate the reform package as a whole. This section sets out a proposed framework for conducting this broader, more integrative evaluation, a great deal of which is based on the evaluation components described in previous sections.

In order to provide measures of the broad indicators and evaluation questions outlined in Section 2, it will be necessary to collect data on:

  • the outcomes for children and parents themselves (e.g., health and wellbeing, developmental progress)
  • the nature and quality of parental and parent-child relationships and interactions (e.g., contact patterns, level of violence or abuse)
  • preparedness to use, and ability to access, early intervention and post-separation relationship service
  • the effectiveness of services
  • the extent to which the services inter-relate and support each other, and
  • overall pathways parents take following separation.

This information can only be obtained by surveys of parents who are separating (or at risk of separating) and surveys of service providers.10 It may also be possible to conduct surveys of children and their experiences of and views about the separation of their parents.

Evaluation of family law reform has in the past tended to be patchy and piecemeal, with little assessment of inter-relationships between the different parts of the system and the outcomes. As discussed above, the challenge will be in identifying the impact of the reforms separately from the impacts of factors other than the family law reforms.

Back to top

Longitudinal study of separating parents

While cross-sectional surveys (that is, surveys at a particular point in time) can be useful, it would be very difficult to identify the impacts of the familylaw reform package using this kind of data. Longitudinal surveys (that is, surveys that interview the same group of people on a number of occasions over a period of time) are needed. A longitudinal approach allows the following of people as they make choices about using (or not using) different components of the family law system, including any use (or not) of early intervention services that parents may have accessed prior to separation. These data can help us understand reasons behind the use or non-use of services (e.g. financial constraints, the desire to avoid conflict, revenge and soon), and related outcomes for children and parents.

It is therefore proposed that a longitudinal survey of Australian families with dependent children experiencing relationship breakdown be undertaken. In order to gain insight into the effects of the reforms it will be necessary to include at least two groups:

  • parents who entered and exited the family law system before the reforms were introduced (1 July 2006).11 This is the pre-reform cohort, and
  • parents who entered the family law system after the reforms were introduced (1 July 2006). This is the post-reform cohort.12

The post-reform cohort could be sampled from: (i) parents living in a geographic area with a Family Relationship Centre at the time of separation;and (ii) parents living in a geographic area without a Family Relationship Centre at the time of separation.

Comparison of the pathways and experiences for the post-reform cohort parents who lived in an area with a Family Relationship Centre at the time of separation with those of parents living in areas without a Centre will provide the opportunity, under certain assumptions, to evaluate the effects of the Centres separately from the impact of other aspects of the family law reform package.13

An important issue is how to recruit the samples of separating parents for the respective cohorts. The options are either through screening questions of the general population (either through cold-calling, mail-out or door-knocking) or selecting a sample from a suitable program data set. Because the proportion of the Australian population who are parents in the process of separating at any point in time is quite small, screening a random sample of households would be extremely expensive and is likely to have a high non-response rate. This approach is thus not recommended. The recommended approach is to obtain pre-and post-reform samples of separated parents from one or more program datasets.

The two most promising sources of program data for the purposes of identifying separating parents appear to be Centrelink data on Family TaxBenefit (FTB) recipients or the Child Support Agency (CSA). Both the FTB and CSA program data would provide a readily accessible sampling frame that could offer representative samples of separating parents. Using program data would allow contact details to be continually updated as the CSA orCentrelink are notified of address changes. The possibility of using these program databases would need to be carefully explored.14

Perhaps the most significant advantage of the program data is that they allow parents who separated prior to the introduction of the family law package to be identified in a relatively easy manner. However, it is important to note that the information collected from parents who separated before the reforms were introduced will be largely retrospective, focusing on a period of some two to three years earlier. Differences observed in the pattern of results apparent for the pre- and post-reform samples may therefore arise not only through the reforms themselves, but also through differences in the nature of the data collected.

Ideally both parents from separating couples would be interviewed. Achieving this is likely to be challenging and so this option would need to be carefully explored (including pilot testing) before a decision was made to adopt such an approach.15 An advantage of the CSA client database as a sampling frame is that it will generally provide contact information for both parents. It is worth noting that that a sample derived from FTB program data would contain an under-representation of non-resident parents who are not living with dependent stepchildren. On the other hand, this sample would be more representative that the CSA program data of resident parents with household incomes around $100,000 and below.

Given that one of the purposes of the proposed design is to identify pathways through the family law system it will be important to conduct fairly regular interviews with the post-reform cohort, particularly soon after separation. Issues relating to cost and respondent burden would need to be considered, but ideally interviews could be conducted twice a year for the first two years following separation and once every two years thereafter. The cohorts should be followed for as long as possible.

In our view, identifying the effects of the reform package on children will be very difficult without a longitudinal design such as the one proposed. Until longitudinal research that specifically focuses on parents’ pathways, and the reasons behind their decisions, is conducted, it is difficult to gauge how the family law system is operating as a whole, and how families can be helped to choose the best pathway for their particular needs.

Back to top

Evaluation of family law system from the service provider perspective

In addition to evaluating the reform package with the use of information from families, it will be important to evaluate it from the perspective of service providers and to assess the extent to which the system is operating as intended. It will also be crucial to identify any areas of the system requiring attention.

The proposed framework for the evaluation of the individual components of service provision involves conducting surveys of service providers (with the exception of the evaluation of Family Relationships Online) and their clients.

These surveys need to be designed to allow the results from service providers and clients at different points in the system to be comparable on some key dimensions, including the extent to which there is coordination between service providers and client satisfaction with key elements (e.g. accessibility, respect received – see Section 2). It will also be important to produce a system-wide picture of the extent to which families are moving efficiently through the system while still receiving the assistance required and with the interests of the children being protected.

While there should be questions asked of service providers and their clients that are specific to the services being evaluated, it would also be crucial to have a core of identical questions that are asked of service providers at different points in the system in order to allow comparability. This is important for identifying any areas of the system that might need refinement.

Back to top

Changes in attitudes and behaviour (“Cultural” change)

Changes in community attitudes towards the family law system, the seeking of professional help for parenting and relationship difficulties, post-separation parenting issues, and grandparents’ relationships with the children can be measured by repeating the baseline General Population of Parents Survey conducted by the Institute in mid-2006. Repeating the survey in mid-2008 with a different sample would allow information on any changes in attitudes to be available. At sometime in the future, it would be useful to repeat this survey at regular intervals to continue to monitor changes in attitudes, although as has been pointed out previously, cultural change is something that is likely to change only slowly over time.

Baseline data about these attitudes for the target population of separated parents was collected in 2006 using Wave 3 of the Institute’s Caring for Children after Parental Separation survey. While it is not possible to undertake further waves of this study, in the monitoring of attitudes for separated parents are able to be measured as a significant proportion of any repeat of the GPPS can be expected to be separated parents, and their attitudes compared to those of separated parents from the 2006 GPPS.

Back to top

Changes in the views and behaviours of family lawyers

Given the importance of family lawyers to the success of the reform package, changes in the views and behaviours of family lawyers will need to be monitored. The Survey of Family Lawyers, conducted by the Institute in mid-2006 will provide the baseline data on this issue.16 It will be important to repeat this survey at regular intervals to continue to monitor changes in attitudes and behaviours and to provide data for administrative reviews, as required.

Back to top

Information milestones

As outlined in the introduction, the framework has been designed to provide information for each evaluation point.

Operation of 1st 15 Family Relationship Centres and National Advice Line (July – December 2007)

Some information will be available from the process evaluation.

Information provision for any interim evaluation after full roll-out of services (October-December 2008)

It is envisaged that this framework could provide the following information:

  • approximately 12 months of data from the longitudinal study of families (both the pre-and post-reform cohorts)
  • data from the process evaluations of the individual components of the reforms, including:
    • service user studies,
    • service provider studies, and
    • analysis of program data.
  • the monitoring of help seeking behaviour via replication of the General Population of Parents Survey, and
  • the monitoring of the attitudes of lawyers via replication of the lawyers survey conducted by the Institute in mid-2006.

Full program evaluation (approximately 2013) subject to available funding.

Back to top

Bibliography

Australian Government. (2003). Every picture tells a story: Report on the inquiry into child custody arrangements in the event of family separation. Canberra: Commonwealth of Australia.

Australian Government. (2005). A new family law system: Government response to 'Every picture tells a story': Response to the report of the house of representatives standing committee on family and community affairs inquiry into child-custody arrangements in the event of family separation. Canberra: Commonwealth of Australia.

Australian Government (2005). A new family law system: Government response to 'Every picture tells a story': Implementation plan for package of reforms - Confidential paper.

Family Law Pathways Advisory Group. (2001). Out of the maze: Pathways to the future for families experiencing separation. Canberra: Commonwealth of Australia.

Back to top

FOOT NOTES

1These include: Mensline Australia, Men and Family Relationships, Family Relationships Counselling, Family Relationships Education and Skills Training, and enhanced responses to family violence.
2These are the Parenting Orders Program (formerly Contact Orders Program (COP)), Children’s Contact Services Program and Family Dispute Resolution services (previously known as Family Relationship Mediation, Conciliation Services and Regional PrimaryDispute Resolution).
3 These indicators, objectives, and evaluation questions are based on a set of indicators developed in discussions between AIFS, AGD and FaCSIA, and subsequently proposed to the Family Relationship Centre Taskforce. In developing the framework, the original set have also been subsequently refined and added to.
4 See Government implementation plan (2005).
6 The Institute’s Family Law Data Mapping Project, commissioned by AGD, is likely to provide a useful summary of these data, and identify the key fields for evaluating the operational aspects of the new family law system.
7 The Institute’s Caring for Children after Parental Separation is unlikely to be followed up after the third wave of data collection.
8 In some cases, the target population will include separated families with children who are full-time university students or who have disabled adult children.
9 For convenience, the term “counsellors” will be used in this summary as a generic term including all relevant professionals other than lawyers.
10 For example, without asking separating parents directly about why they chose a certain pathway, it would remain unclear whether one or both of them saw legal processes and possibly court as a first option; or whether they chose a lawyer for their negotiation skills, or reputation as a litigator, or sympathetic stance with respect to mediation; or whether they were even aware of the existence of services such as mediation.
11 This group would also include parents experiencing separation who have no formal contact with the family law system.
12 This design can be described as a cross-sequential longitudinal design. Cross-sequential designs involve a comparison of at least two separate cohorts that are followed over time – in this instance, pre-reform versus post-reform. However, the pre-reform data will collected will be retrospective.
13 This comparison assumes that the decisions about the geographic locations for the first tranche of Family Relationship Centres is unrelated to other factors that impact on pathways through the family law system and outcomes for separating parents and their children or that if there is a relationship that it can be controlled for using statistical techniques.
14 The Centrelink data on FTB recipients has been used as a sampling frame for the evaluation of the Stronger Families and Community Strategy.
15 This option is likely to be quite expensive.
16 Assistance in the design of this survey was provided by Professor Richard Chisholm.

Back to top