top of page

Federated Climate Data Initiative (FCDI)

The FCDI is a collaboration between various Universities, government and research organisations across Australia to make data and models available at scale. The Eratos Platform has enabled researchers to connect and access vast sources of climate data across Australia using Nodes installed on-premise. These Nodes are located across the Australian Research Data Commons NECTAR infrastructure, as well as deployed at the University of Tasmania, University of New South Wales, University of Queensland and Murdoch University.

Image by Appolinary Kalashnikova

Challenge

Australia’s climate research community has an enormous amount of data (10+ petabytes). Sharing this data between researchers is challenging, if not sometimes impossible, particularly for ensemble modelling as this data is difficult to move and share. 

 

In addition to the barriers regarding physical access to this data, it is also challenging to interrogate these disparate datasets together due to a lack of compatibility and interoperability. Institutions and fields of research may use different formats, structures and systems, making it difficult to integrate large datasets seamlessly. Incompatibility issues can arise when attempting to combine data from various sources, hindering collaboration and analysis.
 

Solution

 

Eratos is a Data Infrastructure Platform solving the way organisations and individuals make informed decisions about the world we live in. Eratos makes finding and accessing disparate data at scale effortless, with interoperability and reusability built into its DNA.

 

Eratos does this through three products:

  • a data infrastructure platform to make any dataset available in situ (PaaS);

  • a developer environment for Creators to build models, workflows and applications (SaaS); and,

  • a marketplace to publish and share scalable solutions built on top of the platform.

 

On Eratos, you can access, build, collaborate and share data, models and solutions. Particular focus to date has been on research, education, federal and state government, agriculture, financial services and smart cities.

 

High-resolution climate data has been federated from multiple sources (CSIRO, Bureau of Meteorology, NSW/VIC/QLD Govts, various universities), leaving the data in-situ, while making data findable and accessible. Downloading subsets of data is fast and scalable. The sophisticated Eratos Data Fabric allows these disparate datasets to be interrogated together, allowing the untapped wealth of these important data archives to be unlocked for their effective utilisation.

 

Eratos offers its customers the ability to deploy self-managed gateway nodes within their own infrastructure. This includes AWS customers who are looking to unlock geo-spatial data within their tenant in a secure and controlled way. Datasets become discoverable via Eratos' semantic metadata catalogue and allows data scientists to drill down directly and perform queries in place without having to move large volumes of data before it can be used.
 

Results and Benefits

 

Open, federated access to all data means more accurate scenario modelling of future climate across all of Australia.  New product development pathways enable bespoke modelling by climate scientists for government and industry application.

 

Analytical workflows can be 10-100 times faster, with certain tasks now capable within hours that used to take months. This is the first time all available data can be visible through a single access point. Now the true data curation tasks can begin. 
 

Leveraging AWS to scale solutions

 

The Eratos Platform utilises Amazon Web Services (AWS) to leverage best in class cloud scaling architecture, including Amazon Elastic Container Service (Amazon ECS), Amazon Elastic Kubernetes Service (AWS EKS), Amazon Elastic File System (Amazon EFS) and Amazon Simple Storage Service (S3).

Epilogue​

 

“The FCDI project could not have been achieved without the collaboration with Eratos.
They have built and designed the software architecture that allows for public-good (i.e. open-source) and commercial players to seamlessly sit alongside each other and collaborate.
The datasets being produced by Australian researchers is critical to the future planning activities of many industries. Without a mechanism to overcome the research-commerce divide, the latent value held within these high-resolution future climate projections cannot be unlocked.
The system Eratos have created overcomes the key barriers to collaboration both within the research space, and across the research-commerce divide.”

 

Dr Tom Remenyi, Chief Investigator & Project Lead - Federated Climate Data Initiative, University of Tasmania
 

bottom of page