Publishing Data
Recommended Data Formats
Recommended Data Formats
Open data is contingent upon the use of accessible data formats. The data.gov.au platform will host files of nearly any type and currently allows entities to publish multiple formats where appropriate or useful to users, though we strongly discourage the use of PDFs as they are not easily accessible. We recommend that you post tabular data in CSV if you want API access to be automatically generated for your dataset.
Supported Filetypes
Currently supported vector filetypes are: MapInfo TAB, KML, KMZ and SHP. Currently supported raster filetypes are: GeoTIFF and ESRI ArcGRID (ASCII and binary)
For the moment, other formats can be converted using the Feature Manipulation Engine (FME) or Geospatial Data Abstraction Library (GDAL). There is currently no limit for the size of a file uploaded, but whenever possible, please make the data.gov.au team aware should you need to upload a large file.
Clean Data
Data.gov.au will only enable API or data visualisation for ‘clean’ CSV files. Make sure at least one of the data files you upload to your dataset is as clean as possible. This will make it easier for others to reuse.
Make sure the CSV file has as few mistakes or bad data entries as possible. This means making sure all the dates are a common format and that there are no missing entries and no non-text data embedded.
For tabular data, you just want a spreadsheet with a single header row and then multiple fields. KML data has its own format that must be followed.
CKAN specifically understands tabular and spatial data types and generates API access to such datasets, if the data files are clean. Clean means the datasets are raw and appropriately structured data. More information on clean data can be found here
For example:
Important Note: Due to the nature of our database if you would like to allow API access to your dataset you will need to keep your column headings under 63 characters.
Removing Author Metadata: It is possible that some filetypes will contain additional author metadata. It is possible to remove this metadata using the Windows interface.
Compressing a CSV into a Zip
Compressing CSV's reduces the amount of time it takes to upload and download a file. This is particularly useful when a CSV has a large amount of data.
For windows users:
- Browse to the relevant CSV
- Select the file you wish to compress
- Right click on the CSV
- Hover over “send to”
- Click "Compressed (zipped) file"
You will now have a ZIP file in the same folder as your original CSV
- Please note that you should only compress one csv at a time if you are uploading it to data.gov.au.
To upload your newly created ZIP file follow the instructions on Manual Publishing
Spatial Data
At the end of 2013, we expanded the geospatial functionality of data.gov.au. These changes established additional and improved support specifically for spatial data services. This means data.gov.au can now present a spatial API endpoint for certain types of spatial files. An example of the new functionality can be found seen in the Geelong Roofprints dataset.
When any of the supported filetypes are uploaded through the CKAN platform, they will automatically be added to the GeoServer. There should be no need for user intervention, but it can take up to 24 hours for the file to be ingested. We are looking at ways to make ingestion more immediate. (GeoJSON, WMS and WFS resources will be automatically added to spatial datasets if ingested correctly)
Spatial data on data.gov.au is also viewable through the NationalMap. To access the visualisation, click the Data button on the left, then click Data Providers and then click data.gov.au. It may take up to 24 hours for newly added datasets to become available on the National Map.
Uploading Spatial Data
At this time, data.gov.au only supports a single spatial file per dataset. If a dataset contains multiple spatial resources it will be ignored by the geoserver ingestor. Once the spatial file is successfully ingested into the geoserver data.gov.au will automatically generate a number of additional resources for the dataset.
KML / KMZ files
Data.gov.au will accept either KML and KMZ files. Simply upload the file using the normal process as outlined in Manual Publishing.
Shape files
Before uploading a shape file you should zip all the relevant files into a single archive. When adding the zip file as a resource set the format to ‘shp’.