Open data in your organisation
This section of the toolkit looks at how your organisation can make useful and practical plans for improving its level of open data. Start by taking the open data self-assessment, or use the creating open data strategies and case studies sections to prepare your organisation’s own open data strategy. If you’ve already got your approach figured out, jump straight in to choosing data sets to publish.
Open data self-assessment
Consider your organisation or team’s capability in each of the categories below to identify your team’s open data achievements to date and opportunities for improvement.
|Awareness||Key senior staff understand what open data is and how it can be used in their organisation||All staff understand the relevance of open data and use it for core agency business||Staff have access to an analysis of the organisation’s data and how it could benefit the government, economy and society||Staff understanding of the implementation and use of government data is reviewed regularly|
|Data management||Individual staff use documented data management processes||Data management policies and procedures are known and used organisation-wide||Business processes are regularly reviewed to ensure compliance with the organisation’s data management procedures||Data management across the organisation is tracked against metrics, including through continuous external benchmarking|
|Creation||Key staff are aware of documented guidelines for making data available in open formats||Staff across the organisation follow the open data guidelines as part of everyday work||Open data guidelines are aligned with the whole-of-government approach||Formal, documented processes are in place to ensure continuous improvement of open data guidelines|
|Distribution||Internal and external stakeholders know where to find the organisation’s open data, including through data portals, clearing houses and peer networks||Formal arrangements for data access and use exist between organisations. Staff have a mandate to engage in peer networks||Regular engagement is held with user groups and stakeholder reference panels to get feedback on data assets, check the organisation’s systems are accessible and align open data strategies with business drivers across government||All sections of the organisation make a continuous effort to improve the discoverability, accessibility and timeliness of data|
|Raising capability||All information management projects are recognised by the organisation and managed in a systematic manner. Data management standards with links to business processes are implemented, and resources for training are allocated to relevant parts of the organisation||All data-related agency processes are managed through a central point and are linked to defined user needs. Internal benchmarking compares data management and release practices across the organisation.||Data release rates are quantified and compared across the agency. Processes and training are universally available and quantified.||A continuous improvement process is in place, based on quantified measures of process efficiency and a range of management processes.|
|Working across government||IT and technical teams provide services using data to other people in the organisation and to other organisations through peer networks||There is official permission for collaboration between teams and formalised agreements for working with other organisations||Performance and review processes to support formal data provision are in place internally and externally||Service agreements are benchmarked and combined agreements for collaborative information use are in place|
Creating an open data strategy
An organisation’s open data strategy is a practical document that identifies how the government’s commitment to open data applies within an individual organisation.
Organisations can use their open data strategies to identify the value they put on open data, how they will act on open data commitments, the data that will and won’t be made available (for example, data may be withheld for privacy or national security purposes), how they will interact with open data stakeholders such as contractors and data users, and an indicative order and timetable for the data to be made available. If you are creating an open data strategy for your own organisation, the examples below and the open data self-assessment will help to identify the best strategic open data goals for your organisation based on the level of progress to date. The page on choosing data sets to publish may also be useful for prioritising the publication of your organisation’s data.
- The strategy’s purpose and audience
- Open data achievements so far
- Scope of the open data project
- Definition of data
- Goals and objectives
- Types of data held
- Data evaluation
- Release strategy
- Data for release
- Future open data program
Examples of organisations’ open data strategies
The Queensland Government has published its open data strategies for others to learn from. See the website for strategies you can use as a guide. If your organisation has published an open data strategy that you would like to share, please send a link to firstname.lastname@example.org and we will add your strategy to this page.
Choosing data sets to publish
What data should be published?
Open data is about giving open access to data that is:
- Structured – the data is recorded in a uniform way based on a common model. This can be as simple as a spreadsheet with rules around how the data is recorded. For example, if one of the inputs is the state or territory that a program is run in, there could be a rule that the state is recorded as ‘WA’ rather than ‘Western Australia’ or ‘Wstn Aus’. See creating data sets for more information about structuring data.
- Machine readable – the data is digitised and available in a format that can be read automatically by machines for retrieval, downloading, indexing and searching. See data formats for more information about machine readable formats.
- Licenced to minimise the limits on how people can use and redistribute the data. See licencing your data for more information.
There are three broad data types found in Government Agencies:
- Raw data generated out of business as usual activities – such as spatial data from a program, energy ratings data, crime statistics, administration data, etc. Often this data is stored in databases and primarily used in business applications.
- Processed data – new data that results from a process such as tables from annual reports, FOI logs, other data generated for the functions and running of agencies. This could also be an aggregate view of a raw data set, fit for public access.
- System data – data that is automatically generated from other processes such as web analytics, project management, access logs and other systems.
Identifying different data across your organisation means getting out of the traditional data teams and looks at other datasets that exist and how you can leverage them to improve services, policies and efficiencies.
How should I prioritise data?
If your organisation is looking for data to start publishing, you can identify popular or easily opened data sets by:
- Analysing FOI, Parliamentary and external helpdesk requests for common requests. This could help to reduce time and resources spent on providing the same data to individual requests, as well as identifying popular data sets for publication.
- Looking at information the agency already publishes, either in data form or PDF form, for some quick results. Publishing the data on data.gov.au will improve accessibility, reuse, discoverability and the ability for your agency to reuse the data. For example, the tables from your annual report, budget, grants, administrative data or mandatory reporting are all useful.
- Assessing what data sets exist across your agency and identifying a top 10 that would provide greater economic, transparency or policy benefits if made publicly available.
- Identify where you need data to deliver a new service or application and consider making the data feed publicly available to cut costs in your service delivery and to enable external services based on your data.
- In all new systems consider the data that is created and how you can best reuse and appropriate expose the data from the start.
Your entity’s priorities for opening data should be identified as part of a broader open data strategy.
Sourcing data for publication
A core theme of the current policies that address the management of public sector information is that government information should be managed as a national resource and made available for reuse unless there are good reasons not to (eg the data could identify individuals). To achieve this, your organisation can:
- Consider whether upcoming publications contain information that is suitable for release in an open data format. Many government reports include agency data about financial, economic, social or regulatory activity or trends, but present the data in formats unsuited to reuse (eg publishing spreadsheets in a PDF file rather than a reusable spread sheet format). To fix this, consider publishing the underlying raw data alongside the report in a reusable open data format.
- For example, as part of the budget process the Government now publishes financial data from the portfolio budget statements in a reusable format on data.gov.au in addition to the statements themselves which are available on departmental websites.
- Consider whether agency websites and mobile apps that present information to the public could also be released in data form. For example, the Department of Human Services’ (DHS) Service Locator tool allows people to find their nearest Centrelink, Medicare or Child Support location. DHS released the raw Service Locator geospatial data on data.gov.au. This makes the data available to the public to build their own apps using the data.
Another example is the Department of Finance’s release of historical government contract data from the AusTender website on data.gov.au.
- Assess whether any material they have already published online in publications, websites or apps is suitable for release as open data. This could include material published as part of the FOI Act Information Publication Scheme or in an agency disclosure log.
The collection of links below demonstrate the breadth and value of the ways that data.gov.au and other open data resources are currently used. The Government Data Use Cases on data.gov.au are also an excellent resource for understanding how data from data.gov.au can be used.
The National Map is a website for map-based access to Australian spatial data from government agencies. It is designed to:
- Provide easy access to authoritative and other spatial data to government, business and the public
- Facilitate the opening of data by federal, state and local government bodies
- Provide an open framework of geospatial data services that supports commercial and community innovation
Trove brings together content from libraries, museums, archives and other research organisations with tools to explore and build. Trove is a growing repository of full text digital resources, but it is also an aggregation of metadata for users to build upon, a set of services, and a community in its own right.
Energy Rating Tools
A joint initiative of Australian Commonwealth, State and Territory and New Zealand governments has produced an energy efficiency dataset for over seven million major household appliances sold in Australia and New Zealand every year. The Department of Industry has used this data to create a mobile app to help households choose energy efficient appliances. The dataset has also been built upon for private projects such as the energy calculator created by Jonathon Moffatt and Wai Liu for GovHack 2014.
Opening the energy ratings data has:
- Created an automatically-created API (Application Program Interface) for the dataset from data.gov.au, saving the cost of commissioning an API and purchasing the servers to host the dataset
- Helped retailers to automatically update the Energy Rating Labels of appliances for sale on their website and reduce the risk of a Trade Practices Act breach for displaying inaccurate or misleading rating information
- Empowered the community to build on the data. A range of GovHack 2014 projects demonstrated the potential to mix energy rating data with other datasets for great outcomes, including the prototype for the Energy Calculator and Comparison Tool, which demonstrated the potential to use the energy rating dataset to help households crunch the costs of their appliances.
The Australian Public Service Big Data Strategy
As indicated in the APS ICT Strategy, the Department of Finance has developed a Big Data Strategy which sets out the actions that the Government is taking to harness the opportunities afforded by big data without compromising the privacy of individuals.
The Strategy, which has been endorsed by the Secretaries' ICT Governance Board, provides a whole-of-government (WofG) approach to big data in order to enhance services, deliver new services and provide better policy advice, while incorporating best practice privacy protections and leveraging existing ICT investments.
Australian Public Service Better Practice Guide for Big Data
The APS Better Practice Guide for Big Data aims to support the adoption of big data analytics by Australian Government agencies. The Guide provides advice to agencies on key considerations for adopting and using big data. It also aims to assist agencies make better use of their data assets whilst ensuring that the Government continues to protect the privacy rights of individuals and security of information.
Whole-of-Government Data Analytics Centre of Excellence
The WofG Data Analytics Centre of Excellence (CoE) was established by the Australian Taxation Office as a space to build analytics capability across government. The purpose of the CoE is to enable a common capability framework for analytics, as well as an opportunity to share technical knowledge, skills and tools. The CoE also helps build collaborative arrangements with tertiary institutions to aid the development of analytics professionals. The CoE has developed a website which provides information about data analytics at: http://analyticsspace.net.au/