I hope you’ve had a great start to 2022. We are excited about what is in store for the Federated Climate Data Initiative. This will be the year when the project is rolled out completely and climate scientists will reap the benefits. This, of course, will have significant flow-on effects for other sectors.
In this edition, I’d like to give you some insight into what this project actually is. If we were to compare it to other climate-related projects currently underway, the FCDI is a small fish in a pond full of sea monsters. But, its significance is not related to its comparative size. It’s a hugely significant first step in allowing our entire climate science and services sector to collaborate on a national scale.
To help us get our heads around exactly how it all fits together, we’re going to call on the man at the coal face of the construction, Eratos CTO and co-founder, Paul Stegeman.
Existing platforms to be enhanced by the FCDI?
One of the most important jobs of the FCDI is to help existing data delivery platforms reach their full potential. By making large quantities of data available to them very quickly they can do what they do with far more efficiency. Some of the programs planned to be getting an FCDI upgrade are:
National Environmental Science Program 2 (NESP2)
This program essentially helps on the ground users deliver real results on the goals of their environmental projects. It’s a one-stop shop for scientific knowledge, decision tools and options for practical management to support. It basically allows environmental scientists to ask more specific questions about the data. Having quick and easy access to all of the country’s climate data is a huge advantage for this program. The FCDI could dramatically enhance the NESP2’s performance and the quality of the product it delivers to its users.
Climate Science for Agriculture (CSA)
Farmers are among the most reliant on climate forecasts and seasonal predictions. They can mean the difference between a bumper crop and a bust year. The CSA is a tool designed specifically for farmers and the agricultural industry. Having quick and easy access to the latest climate data as soon as it’s available, makes the CSA’s forecasts and predictions more accurate. The FCDI could make that data available to our farmers so they can make the best decision for their crops and their environment.
Australian Climate Service (ACS)
The ACS is like Australia’s climate early warning system. It helps people to understand the threats posed by a changing climate and how to limit the damage it could cause; threats posed by natural hazards like floods, bushfires, drought or severe storms. The service uses the most up to date climate data to help its customers prevent, prepare and respond to incoming threats. Obviously, having large amounts of data available as quickly as possible is a top priority for the ACS. This is where the FCDI would come in, providing public-facing natural hazard information as it becomes available.
Climate Change in Australia (CCiA)
CCiA is aimed at top-level decision-makers, providing them with the climate information, projections tools and data to help them make better decisions for Australia’s future. For example, the CCiA is working with Australia's Electricity sector which is being increasingly impacted by the weather. The CCiA provides the sector with better climate and weather information to help build resilience in the power grid. The resources available through the CCiA website are currently being integrated into the FCDI. Essentially, we are helping Australians make better decisions by making more data available to them.
Eratos CTO Paul Stegeman in the spotlight
Paul Stegeman is one of the co-founders of Eratos and is the brains behind the build. Paul has an academic background in aerospace and computational physics from Monash University. He’s currently the Chief Technology Officer and one of the four co-founders of Eratos along with John Chung, Sasha McMonagle-Ihasz and Steve Taitoko (who we met in the last newsletter). He’s also the guy who’s overseeing the build of the FCDI.
“The FCDI is all about providing a single source of truth for all climate data in Australia,” he says. “Actually, not just climate data, but climate models. It allows scientists to onboard these data and the models that produce them, regardless of where it's sitting in the country as well as sharing access to that with other scientists or even industry.”
Tackling a big problem
Paul says that one of the biggest issues with climate data is the sheer volume of it. “You don't generally want to have to move it… it can be quite difficult to give access to external scientists.”
“We have about one and a half petabytes worth of climate data, in total across just Australia, In terms of providing access to that data [one way] is to get a scientist to copy it. Moving petabytes of data takes a seriously long time, it's often generally easier to ship it. Amazon has this service which is literally a truck that allows them to ship petabytes of data sitting on the back of a Mack truck. Then they drive it to the next location, and they dump the data there. We solve that by not moving the data.”
Paul and his team allow scientists to access the subset of the data they need without moving the data. “Ninety-nine per cent of requests aren't going to be accessing the entire data set, but just a subset of that data. So why don't we pull out that subset? Or why don't we run the models alongside the data? So the [FCDI] allows scientists to run scripts where the data is and not have to worry about moving the data, and then they get the result back.”
The basic building blocks
“We've got these concepts called a primary node, which you can think of as a piece of cloud-managed software. Then we have the gateway nodes which form a peer to peer network. They can talk to each other but can also sit wherever the data is. So it doesn't matter where it's sitting on your desktop, whether it's sitting in a compute cluster, in the cloud, or even on an embedded device. The Gateway node sits there to act as a translator to the data sitting on that system regardless of the format.”
How will it be used
The FCDI sounds like a system that most of us won’t be affected by. You may think it’ll mostly be scientists and engineers in white lab coats, which to a certain degree is true, but Paul says the information produced as a direct result of the FCDI will affect almost everyone.
“Say we want to calculate over the next century, the number of days that a particular location is going to be above 40 degrees for a year on average; the scientists would access the Eratos system and connect to the FCDI [which] will provide them access to the data and any available underlying models.
“They basically go and ask ‘Okay give me all the climate data for surface temperature for a given location.’ The system then basically goes off and finds all the available data that fits that criteria”
Working with existing systems
There are dozens of existing data platforms all around the world holding on to high value information. The problem is, there is no easy way to search for the data you need that may exist in separate data platforms.
“If you're looking for climate data in the US, you might go to NOAA,” says Paul. “If you're looking for the weather data globally over the last year, you may go to use data from European agencies, or if your interest is historically over Australia you'd go to the Bureau of Meteorology. Then there are various other platforms like the Earth System Grid infrastructure. However, those platforms, more or less, have non-uniform ways of accessing the data or even querying the data. So you have to do the implementation for each of them separately."
“There are some standards out there, however, those standards aren't uniformly accessible. That data is just sitting just on maybe, someone's desktop or on their computer infrastructure at their university. We provide a tool that allows them to effectively spin up the gateway node there, and then make all the data sitting on those systems available in a uniform manner.”
Connecting the Climate Science Dots - a podcast from the FCDI
Get the inside scoop on the latest FCDI developments, here how our amazing team is overcoming obstacles and learn about some of the key players in our new podcast, Connecting the Climate Science Dots.
Hosted by former SBS and ABC journalist, Sam Ikin, Connecting the Climate Science Dots is available wherever you get podcasts.
Share | |
Authors | Dr Tomas Remenyi, Paul Stegeman |
Cover Image |