Steve Taitoko

Dr Tomas Remenyi

Paul Stegeman

John Chung

Data61

Monica Liu

Sacha McMonagle-Ihasz

A new data platform will help industries better manage and use insights from sensing and spatial data for a more competitive edge in the domestic and international markets.

A collaboration between Australia’s national science agency 

 and Eratos will help software engineers and researchers develop more informed, effective and scalable digital solutions for owners and managers of natural and built world assets.

Steve Taitoko, Co-Founder and CEO of Eratos said the partnership responded to a growing market demand.

“More and more industries across Australia including agriculture, water supply and energy now rely on the constant analysis of large sets of data to not only pinpoint current trends, but to also develop a future-gazing outlook for potential issues in the years ahead to inform strategy,” Mr Taitoko said.

“This will be crucial to future-proofing a number of industries globally, and this new partnership will see CSIRO developed Senaps fully integrated into Eratos, and both organisations working together to make this a reality.”

Under the partnership, CSIRO’s data analytics platform 

 will be incorporated with Eratos’ platform to create a distributed data management technology with cloud based IoT and scientific workflow management.

The platform will integrate huge volumes of data from Internet of Things (IoT) sensors, satellite images, and weather projections to help create highly accurate predictions and simulations for better decision making. 

Application specific APIs, databases and authorisation models do not need to be built by the user, and the platform will be equipped with secure hosting and custom analytics automation.

This level of oversight for weather-dependent activities such as crop yields and anticipated rainfall patterns will aid industries like agriculture and water supply forecasting.

Chris Sharman, principal engineer at CSIRO, said with data-driven technologies fundamentally changing the economic landscape and extreme weather becoming more common, these industries must utilise data-led predictive insights to future-proof themselves.

“Seeing research and technology lead to the development of real-world solutions that can save businesses critical resources and enhance their productivity and longevity is something we strive to accomplish at CSIRO, and Eratos is the perfect partner for us to ensure Senaps reaches that point,” Mr Sharman said.

Senaps is already a sophisticated platform, having been in development for over five years and applied across industries including environmental governance, smart cities, and livestock management.

A collaboration between Australia’s national science agency CSIRO and Eratos will help software engineers and researchers develop more informed, effective and scalable digital solutions for owners and managers of natural and built world assets.

Cover

Reference

CSIRO, Eratos partner to help the world turn data into opportunity

I hope you’ve had a great start to 2022. We are excited about what is in store for the Federated Climate Data Initiative. This will be the year when the project is rolled out completely and climate scientists will reap the benefits. This, of course, will have significant flow-on effects for other sectors.

In this edition, I’d like to give you some insight into what this project actually is. If we were to compare it to other climate-related projects currently underway, the FCDI is a small fish in a pond full of sea monsters. But, its significance is not related to its comparative size. It’s a hugely significant first step in allowing our entire climate science and services sector to collaborate on a national scale.

To help us get our heads around exactly how it all fits together, we’re going to call on the man at the coal face of the construction, Eratos CTO and co-founder, Paul Stegeman.

Existing platforms to be enhanced by the FCDI?

One of the most important jobs of the FCDI is to help existing data delivery platforms reach their full potential. By making large quantities of data available to them very quickly they can do what they do with far more efficiency. Some of the programs planned to be getting an FCDI upgrade are:


National Environmental Science Program 2 (NESP2)

This program essentially helps on the ground users deliver real results on the goals of their environmental projects. It’s a one-stop shop for scientific knowledge, decision tools and options for practical management to support. It basically allows environmental scientists to ask more specific questions about the data. Having quick and easy access to all of the country’s climate data is a huge advantage for this program. The FCDI could dramatically enhance the NESP2’s performance and the quality of the product it delivers to its users.


Farmers are among the most reliant on climate forecasts and seasonal predictions. They can mean the difference between a bumper crop and a bust year. The CSA is a tool designed specifically for farmers and the agricultural industry. Having quick and easy access to the latest climate data as soon as it’s available, makes the CSA’s forecasts and predictions more accurate. The FCDI could make that data available to our farmers so they can make the best decision for their crops and their environment.


The ACS is like Australia’s climate early warning system. It helps people to understand the threats posed by a changing climate and how to limit the damage it could cause; threats posed by natural hazards like floods, bushfires, drought or severe storms. The service uses the most up to date climate data to help its customers prevent, prepare and respond to incoming threats. Obviously, having large amounts of data available as quickly as possible is a top priority for the ACS. This is where the FCDI would come in, providing public-facing natural hazard information as it becomes available.

CCiA is aimed at top-level decision-makers, providing them with the climate information, projections tools and data to help them make better decisions for Australia’s future. For example, the CCiA is working with Australia's Electricity sector which is being increasingly impacted by the weather. The CCiA provides the sector with better climate and weather information to help build resilience in the power grid. The resources available through the CCiA website are currently being integrated into the FCDI. Essentially, we are helping Australians make better decisions by making more data available to them.

Eratos CTO Paul Stegeman in the spotlight

Paul Stegeman is one of the co-founders of Eratos and is the brains behind the build. Paul has an academic background in aerospace and computational physics from Monash University. He’s currently the Chief Technology Officer and one of the four co-founders of Eratos along with John Chung, Sasha McMonagle-Ihasz and Steve Taitoko (who we met in the last newsletter). He’s also the guy who’s overseeing the build of the FCDI.

“The FCDI is all about providing a single source of truth for all climate data in Australia,” he says. “Actually, not just climate data, but climate models. It allows scientists to onboard these data and the models that produce them, regardless of where it's sitting in the country as well as sharing access to that with other scientists or even industry.”

Paul says that one of the biggest issues with climate data is the sheer volume of it. “You don't generally want to have to move it… it can be quite difficult to give access to external scientists.”

“We have about one and a half petabytes worth of climate data, in total across just Australia, In terms of providing access to that data [one way] is to get a scientist to copy it. Moving petabytes of data takes a seriously long time, it's often generally easier to ship it. Amazon has this service which is literally a truck that allows them to ship petabytes of data sitting on the back of a Mack truck. Then they drive it to the next location, and they dump the data there. We solve that by not moving the data.”

Paul and his team allow scientists to access the subset of the data they need without moving the data. 

“Ninety-nine per cent of requests aren't going to be accessing the entire data set, but just a subset of that data. So why don't we pull out that subset? Or why don't we run the models alongside the data? So the [FCDI] allows scientists to run scripts where the data is and not have to worry about moving the data, and then they get the result back.”

“We've got these concepts called a primary node, which you can think of as a piece of cloud-managed software. Then we have the gateway nodes which form a peer to peer network. They can talk to each other but can also sit wherever the data is. So it doesn't matter where it's sitting on your desktop, whether it's sitting in a compute cluster, in the cloud, or even on an embedded device. The Gateway node sits there to act as a translator to the data sitting on that system regardless of the format.”

The FCDI sounds like a system that most of us won’t be affected by. You may think it’ll mostly be scientists and engineers in white lab coats, which to a certain degree is true, but Paul says the information produced as a direct result of the FCDI will affect almost everyone.

“Say we want to calculate over the next century, the number of days that a particular location is going to be above 40 degrees for a year on average; the scientists would access the Eratos system and connect to the FCDI [which] will provide them access to the data and any available underlying models.

“They basically go and ask ‘Okay give me all the climate data for surface temperature for a given location.’ The system then basically goes off and finds all the available data that fits that criteria”

There are dozens of existing data platforms all around the world holding on to high value information. The problem is, there is no easy way to search for the data you need that may exist in separate data platforms.


“If you're looking for climate data in the US, you might go to NOAA,”

“If you're looking for the weather data globally over the last year, you may go to use data from European agencies, or if your interest is historically over Australia you'd go to the Bureau of Meteorology. Then there are various other platforms like the Earth System Grid infrastructure. However, those platforms, more or less, have non-uniform ways of accessing the data or even querying the data. So you have to do the implementation for each of them separately."

“There are some standards out there, however, those standards aren't uniformly accessible. That data is just sitting just on maybe, someone's desktop or on their computer infrastructure at their university. We provide a tool that allows them to effectively spin up the gateway node there, and then make all the data sitting on those systems available in a uniform manner.”

Connecting the Climate Science Dots - a podcast from the FCDI

Get the inside scoop on the latest FCDI developments, here how our amazing team is overcoming obstacles and learn about some of the key players in our new podcast, Connecting the Climate Science Dots. Hosted by former SBS and ABC journalist, Sam Ikin, Connecting the Climate Science Dots is available wherever you get podcasts.

We are excited about what is in store for the Federated Climate Data Initiative. This will be the year when the project is rolled out completely and climate scientists will reap the benefits. This, of course, will have significant flow-on effects for other sectors.

The New Year of the Federated Climate Data Initiative

As you probably already know, we are solving a complex problem. Around the country, hundreds of scientists produce petabytes (PB) of the most up-to-date data about the world’s climate. Our scientists and researchers are creating a wealth of high-quality, high-resolution climate model outputs. This data can help researchers, government, industry, or the general public make better decisions about Australia’s future.

The problem is that the quantity of information they produce is hard to share between researchers or other people who need it. Until now, the supply of this information has not been keeping up with its growing demand.



The solution is the Federated Climate Data Initiative (FCDI).



We are creating a single, united virtual collection from the enormous amounts of information currently stored around the country. The aim is to make the data much easier to find, access and use, dramatically enhancing the quality of future research and the services built this new knowledge, particularly around future risk.



Australia’s many regional climate projections are vital for understanding what impacts we must adapt to, for example, an increased bushfire risk. We can then determine which risks we can reduce, which could involve changing how we do things, such as what we grow and where we grow it. And, importantly, we can work out which impacts we want to avoid, for example, losing precious ecosystems if we allow climate change to increase by 2°C. By accelerating how we create products and services from these projections, change-makers will be able to act on the findings sooner.

Although the Federated Climate Data Initiative has been in the planning since 2018, as of April 2021, the Department of Industry, Science, Energy and Resources Disaster Management and Risk Reduction Fund has officially funded it with initial seed funding of $500,000. Development of contractual arrangements with partners is ongoing.

Since April, we have been busy setting up data access nodes across the country. We have installed nodes in Tasmania, Western Australia, Queensland and Victoria. They are steadily adding to the data available through the network.

We are building nodes in South Australia and within the National Computing Infrastructure are well underway. We are also working with the Australian Climate Service, the Climate Services for Agriculture Program and the National Environmental Science Program on how the FCDI can support them best. We will keep you updated on these developments through future newsletters.

All state and territory governments have now joined the project. The CSIRO, the Bureau of Meteorology and the Department of Home Affairs are also contributors. We’ve also welcomed some of Australia’s leading universities to the team including:

And welcoming our new implementation partner, Eratos, who is helping the project come to life.

If you’d like to part of this historic project, we would love to hear from you. To get involved or if you want to promote complimentary activities through our newsletter, please contact the Chief Investigator, 

In each issue of the newsletter, we will put the spotlight on each team contributing to this important collaboration.

The first spotlight is on the newest player in this field, Eratos, with CEO Steve Taitoko Eratos is one of the key players in the FCDI. The Melbourne based startup is responsible for building the infrastructure that the FCDI is based on. Tom Remenyi met with Steve Taitoko for the first time about two years ago, and they immediately knew they needed to work together to make the project happen.

What does Eratos hope to achieve through the FCDI?

Steve describes Eratos as a company “that’s looking to change how we make decisions about the world we live in.” They do that by creating the technology to allow people to make decisions from data that hasn’t been available before, data based on real-time information about the world we live in.



“Our purpose is how we can help humans survive on this ever increasingly heating rock that’s hurtling through space,” Steve says. “We can use a lot more information to create shared and collaborative knowledge to make better decisions about the world we live in. Decisions like what we plant in our soils, how we build our cities, how we can ensure that humans and our ecosystems survive within a changing climate, how we adapt to more extreme climate events that are going to be happening.”

Steve says that the Federated Climate Data Initiative and Eratos fit together like two pieces of a puzzle. “I was going to Hobart for a couple of days holiday, and I organised some meetings while I was down there. One of those meetings happened to be with Dr Tom Remenyi.”



“He showed me a diagram of an ARC linkage proposal… and I said to him, that diagram you’ve just drawn, that’s what we’re building, so don’t worry about that part; we will take care of that.” And through that meeting, a serendipitous partnership was born. Tom would focus on climate science, and Eratos would support him with the platform build. “That’s how it all started,” Steve says. “It was really random, just a random meeting.”



To hear more from Steve Taitoko and other key players in the project; keep an eye out for our new podcast. We will keep you updated on that. 

We are solving a complex problem. Our scientists and researchers are creating a wealth of high-quality, high-resolution climate model outputs. This data can help researchers, government, industry, or the general public make better decisions about the future.

Federated Climate Data Initiative Newsletter

Throughout human history there have been significant innovations that have defined our civilisation in the way we think and act. For several million years our actions and thoughts were defined in the paradigm of Hunting and Gathering. Following this period through multigenerational innovations we entered the Agricultural Age which extended for thousands of years. Subsequent inventions of the printing press and reading glasses generated a foundation for our collective civilisational consciousness to expand. This gave way to exponential innovations that allowed us to arrive more rapidly into the Industrial Age and Information Age.

Today, we're at the precipice of our next significant era - the “Augmented Age". History indicates that this exponential innovation horizon will usher a new age resulting in more rapid and radical changes in the way we solve problems - further accelerating the expansion of our collective consciousness and innovations.

The previous asymmetry between our digital and physical worlds is changing, impacting our shared reality. We are begging to see the coupling of our digital world and physical worlds. Manufacturing and utilities are measuring vital performance increases using the Internet of Things (IoT). Surgeons are experimenting with holographic anatomic models during surgical procedures. Governments are generating digital twins of their cities as virtual test environments to help improve decision-making and scenario-planning.

Still, today's technological applications are just snippets of what could be accomplished. To accelerate and maximise our potential we must design systems in which humans and machines function collaboratively in ways that complement each other's strengths and counterbalance each other's limitations. Our inherent human abilities are going to be augmented by computational systems transporting us into the Augmented Age.

"Men will set the goals, formulate the hypothesis, determine the criteria, and perform the evaluations. Computing machines will do the routinizable work that must be done to prepare the way for insights and decisions in technical and scientific thinking. . . . The symbiotic partnership will perform intellectual operations much more effectively than man alone can perform them."  

To expand human and computer intelligence, we must explore the relationship by understanding and transforming our perspective on how we interact with technology:


We must shift perspectives from seeing machines as separate entities that fundamentally are technologies that automate tasks, to exploring the idea that machines' capabilities will be used 

 expanding our collective consciousness and improving the abilities of humans,

Consider the full potential of both humans and machines evolving mutually creating more 

rather than the approach of machines as a tool for reducing 

We need to develop a more general outlook of 

 for human-computer collaboration, rather than a human-dictated view of deploying technologies.

We need to change our focus to a more cohesive cyber-physical world rather than individual physical or virtual worlds.

Through changing our perspective towards working together and augmenting each other’s work, we will start to occupy a new space of advancement and produce unique ideas and solve problems in new and beautiful ways.

What is this new space, and how will this human and machine collaboration work?

Exploring this question is a project called 

 conducted by Autodesk. The project’s objective was a prototype partnership of algorithms, robots and humans, working collectively to solve a remarkably complicated design problem; the task was to build a pavilion.

The humans navigated around the construction site, shaping and forming the elements. They carried out the labourers tasks, which is very difficult for robots to accomplish because of their non-isomorphic body. Conversely, the machines did the fibre winding, which is virtually impossible for a human to do. And all of this being directed by an AI system, telling the humans and robots what to do and managing and tracking the numerous individual components. Most excitingly the construction of this pavilion was not possible without humans, machines and algorithms collaborating and augmenting each other’s work.

Humans, machines and a better process for operating algorithms will side-by-side yield more reliable outcomes than both the most gifted humans or the most advanced algorithms working in isolation. The need to design these systems for human-computer collaboration demands more attention.

The late Steve Jobs explained augmentation and the concept of human-machine collaboration very well.

"I think one of the things that really separates us from the high primates is that we're tool builders. I read a study that measured the efficiency of locomotion for various species on the planet. . . . Humans came in with a rather unimpressive showing, about a third of the way down the list, but somebody at Scientific American had the insight to test the efficiency of locomotion for a man on a bicycle. . . . A human on a bicycle blew the condor away, completely off the top of the charts. And that's what a computer is to me . . . it's the most remarkable tool that we've ever come up with; it's the equivalent of a bicycle for our minds." 

Steve Jobs is recognised for introducing human-centric design thinking into the personal computer. For us to achieve human-machine symbiosis and the "Augmented Age" it will require a similar foundation of design thinking into the domains of, robotics, data science and AI.

The state of things to come will be unlike anything we've ever seen before. We're going to have a world with more variation, more connectedness, more dynamism, and, of course, more beauty. Why? Because we will be redesigning the relationship between technology, nature and humanity.

Today, we're at the precipice of our next significant era - the “Augmented Age". History indicates that this exponential innovation horizon will usher a new age resulting in more rapid and radical changes in the way we solve problems.

Cover Image

Lesh, Neal & Marks, Joe & Rich, Charles & Sidner, Candace. (2004). "Man-Computer Symbiosis" Revisited: Achieving Natural Communication and Collaboration with Computers.. IEICE Transactions. 87-D. 1290-1298.

J. C. R. Licklider, “Man-computer symbiosis,” IRE Transactions on Human Factors in Electronics, March 1960, http://worrydream.com/refs/Licklider%20 -%20Man-Computer%20Symbiosis.pdf. In this essay, Licklider analogized the relationship of humans and computers with that of the fig wasp and the fig tree.

Jobs made this comment in the 1990 documentary film Memory & Imagination: New Pathways to the Library of Congress. Clip available at “Steve Jobs, ‘Computers are like a bicycle for our minds.’ - Michael Lawrence Films,” YouTube video, 1:39, posted by Michael Lawrence, June 1, 2006, https:// www.youtube.com/watch?v=ob_GX50Za6c.

References

Design For The Augmented Age

People regularly ask what the meaning of Eratos is. In short, we named the company after 

, the Greek philosopher, mathematician, and scholar who lived around 200 BC. Among his many important achievements, he was the chief librarian at the Great Library of Alexandria, and it was at that institution that the earliest recorded examples of metadata can be found.

Metadata simply put, is data about data. It was applied to scrolls at the library by attaching small tags containing the title, subject and author, allowing library users to assume the content, without having to unroll each scroll and so they could be returned to their rightful location after use. Metadata today still apply that simple principle, but with a little more detail.

The use of metadata is not just a convenient ‘nice to have’, it is a ‘must-have’, particularly as we design more advanced analytics relying upon machine-to-machine communications. Mixing structured and unstructured data into automated analytic processes, mandates rapid access to data about the data, to retrieve precise, relevant and quality data.

, we discussed the power of the FAIR principles and how they are foundational in the design of Eratos. The key to achieving a data system based on FAIR principles is robust metadata. In this context:

Metadata makes data findable by allowing search on the elements of metadata that tell the story of the data.

Metadata makes data accessible by creating elements within metadata that provide access control and permissions management.

Metadata makes data interoperable through the standardisation of the metadata’s vocabularies and formats.

Metadata makes data reusable by providing rich contextual provenance about the data’s origins, use, licence, and governance.

In making data findable and accessible, the task is relatively straight forward, relying on the addition of descriptive elements to metadata. However, making data interoperable and reusable requires a high level of coordination to ensure metadata is created consistently, regardless of how it was generated or by who. To ensure standardisation, frameworks called schemas are required.

A schema provides the structure of a metadata collection to a specific domain, such as published research. They establish a set of rules that define required information, describing the semantics and relationships of the elements, for example 

 (DOIs) which are persistent interoperable identifiers for use on digital networks, adopted by scholars as a means to describe published research.

Much effort has been made to create standard schemas through international validation by organisations such as the International Standards Organisation. One of the most commonly used standards is the 

 (DCMI). Used often on webpages, it is simple and generic with just 15 elements, containing information such as title, date, type, format, creator, and rights.

Eratos utilises standard schemas to ensure metadata across all fields of cyber-physical data is interoperable and reusable. As they are machine-readable, schemas provide scalability, particularly as we move to more data generated and stored at the edge and are a vital component as we build the world’s semantic web of cyber-physical data.

Without standardised schemas and metadata, it will be impossible to extract needles from a rapidly expanding data haystack at speed and scale. Thankfully, the groundwork laid by our academic community allows us to bring robust and uniform metadata into the mainstream and for that, we have better data.

Metadata simply put, is data about data. The use of metadata is not just a convenient ‘nice to have’, it is a ‘must-have’, particularly as we design more advanced analytics relying upon machine-to-machine communications.

Better Data With Metadata

Data is generated and captured in every aspect of our daily lives these days. As individuals, the phones in our pockets, watches on our wrists, our televisions, and even the cars we drive. On the industrial scale, we have the burgeoning Internet of Things (IoT), with myriad sensors deployed on everything from 

This data is being generated from what is broadly called, cyber-physical systems (CPS), the intersection of the physical world with digital devices, sensors, computation, control and networking.

Like the internet connected people to information, these CPS are connecting information to the physical world. CPS are what underpins our transition to the 

Founding executive editor, Kevin Kelly wrote in 

 of a world where “someday soon, every place and thing in the real world—every street, lamppost, building, and room—will have its full-size digital twin in the mirrorworld.”

This mirrorworld “will reflect not just what something looks like but its context, meaning, and function. We will interact with it, manipulate it, and experience it like we do the real world.”

Technologies such as digital twins will form the basis of the mirrorworld, creating a living 3D metaverse that mimic the real-world environment. Loaded with sensors streaming data from everywhere, these digital avatars will provide infinite analytical possibilities, allowing humans and machines to diagnose, optimise, and predict in ways never before dreamt of.

Much of the data generated by CPS has never existed before, providing many exciting opportunities to gain fresh insights about our natural and built worlds. However, transforming data at the scale required comes with many challenges.

Of most concern is our inability to cope with the data flowing from CPS. A typical data scientist will lay awake at night worried about the three V’s: Volume (amount of data); Velocity (the speed at which data is generated); and, Variability (differences in data types, quality, programming languages). With the onslaught of data coming down the pipe from CPS, the size of problems caused by the three V’s are beyond most data scientists’ capability to control.

It seems incongruent to say that the existence of CPS is the culmination of centuries worth of research and technological innovation, yet they appear to have snuck up on us, given our unpreparedness to cope with the data explosion they have caused .

Fortunately, there are many researchers, data scientists, and organisations collaborating to harness CPS to generate quantum leaps in the way in which we interact with our natural and built worlds.

Take for example the groundbreaking work undertaken by Australia’s preeminent research organisation the CSIRO. Within their cyber-physical systems unit, they are 

 to better understand what factors are affecting colonies and pollination to reverse the unprecedented decline in bee populations globally.

One of the largest scale CPS programs in the world is 

. Along with digitising all government services, the program is focussed on creating interconnected systems for urban living, mobility, and health. For example, town planning is linked to transport design to plan for the seamless integration of autonomous vehicles into their society.

We believe in the potential that CPS has to positively reshape our world, and are committed to solving the many data problems inherent in translating raw sensor data into meaningful, actionable insights about our natural and built worlds.

The idea of a mirrorworld is interesting, but right now, we’re not quite sure what’s looking back at us. If we have anything to do with it, that’s about to change.

Like the internet connected people to information, Cyber-physical systems are connecting information to the physical world. They underpin our transition to the 4th Industrial Revolution.

The Rise of Cyber-Physical Systems

There exists a well-worn phrase that ‘data is the next oil’, the irony of which appears lost given the technologically-driven shift to renewable energy. However, as the inventor of the internet, Tim Berners-Lee said, “data is a precious thing and will last longer than the systems themselves”, suggesting that data has more than a single useful life.

The idea of data as reusable or temporal in nature isn’t a new concept, however, even amongst the research community, there is a growing realisation that outdated systems and lack of data governance, have resulted in significant amounts of data locked-up inside institutions. The result is some of our most highly prized data, created by some of our brightest minds, are gathering virtual dust at the bottom of digital filing cabinets.

In recognition of this problem, in 2016 a group of scientists developed the 

 to establish guidelines that support the reusability of scholarly data, making data findable, accessible, interoperable, and reusable. The catalyst for this initiative was driven by artificial intelligence’s insatiable appetite for data with minimal human intervention, or what the group called 

Clearly, data practices of the past were standing in the way of progress for the machine-to-machine future. So important is the need to create more accessible and reusable data, that at the 2016 G20 Hangzhou Summit, G20 leaders 

 endorsing the application of FAIR principles as a vital component of science and technology innovation-driven growth.

 in Data Intelligence, researchers set out how FAIR principles could be implemented to achieve machine-actionability. It noted under each principle that:

Data should be easy to find for both humans and computers, where machine-actionable metadata are essential for process automation.

 Access control is essential for human and machine interaction with data, with strong governance.

Resources attributed to data should be structured so that machines can aggregate them into a richer, unified view of the data and its resources. To enable this, the semantics of each resource must be clearly defined.

 Descriptions of data through its metadata and resources are sufficient for humans and machines to decide whether the data is relevant for reuse, and if so, under what conditions should it be used.

This matters because we have an arms race of sorts, to integrate artificial intelligent systems into all parts of our lives. A prerequisite for this to happen is that data must be AI-ready.

The US government’s biomedical and health research organisation, the National Institutes of Health (NIH) are moving into 

, and have adopted FAIR principles to help guide their transition. One of the core themes of their plan, AI-ready data, means that the “creation and stewardship of data sets to enable machine learning may be the NIH’s single greatest lever to accelerate the application of AI within biomedicine.”

It is no surprise that organisations like the NIH are looking to leverage artificial intelligence to transform their operations. McKinsey & Company 

 to add US$5.8 trillion in global value annually by 2035. They estimate that only 20% of AI-aware companies are using the technology in a core business process or at scale. Four out of the five challenges causing this low adoption relate to data that is not AI-ready.

It is for this reason, the FAIR principles guide the development of Eratos. In applying these to cyber-physical systems, there is a clear path toward unlocking the vast treasure trove of data that already exists about our natural and built worlds. With the right framework, this also provides a significant opportunity to new data streams, given they will plug into robust data infrastructure, leveraging integrated systems and automation, to continually build deep, rich, contextual data.

Eratos is taking the tracks laid by academia and building a bridge to apply it to similar problems in the commercial world. Using the FAIR data principles provides an elegant and considered solution, to address a broad and complex problem.

A group of scientists developed the FAIR data principles to establish guidelines that support the reusability of scholarly data, making data findable, accessible, interoperable, and reusable.

Making Data Fair

As a term, ‘data science’ is a dominant feature within our information systems lexicon. It is referred to often, but do we truly understand what it is and how best to unleash its power as a force for good?

The concept was first used in the early 1970s, however, data science as a term wasn’t codified until 

’s 2001 paper.  Since then, it has been interchangeably used in emerging areas such as big data, artificial intelligence, and IoT, with little regard to how to apply and extract the greatest value from it. As a result, significant barriers exist to the mainstream adoption and scalability of data science.

At its core, data science encompasses data strategy, engineering, analytics and visualisation. All of this must then be wrapped within the context of particular domain expertise to be useful. Our typical response has been to wrap responsibility for all of this into one tidy role description.

Finding one, let alone the hundreds of thousands of individuals required who have the skill, experience, and wherewithal to meet the expectations of that job description, is like finding the proverbial unicorn. Welcome to the world of the data scientist, and thus the pinch point of our data science supply chain.

Kaggle State of Data Science and Machine Learning

 survey of over 16,000 data professionals, a number of recurring themes were identified as challenges faced by data scientists, including data curation, management support and budget approval.

Given the vast scope of responsibility we are asking our data scientists to take on and the structural barriers in their way, we have a mountain to climb to reach the full potential of advanced analytics.

To resolve the issue of the vast scope of a data scientist role, we must look at ways to treat data science as a collaborative workflow and not a single data scientist role. Breaking out the core functions of strategy, engineering, analytics and visualisation into distinct roles, will be a good start to build cross-functional teams with domain expertise in each function.

To resolve the second challenge of breaking down barriers to better data management, we must address some of the following core issues:

It is estimated that a data scientist spends 

80% of their time finding and cleaning suitable data

 before they can even use it. This creates an even wider gap between the application of their domain expertise and the scale of problems they are solving.

The way in which we are interacting with data is changing rapidly. For instance, it is estimated that 

75% of data will be generated and processed on the edge by 2035

, a big shift from the current 10%. Combine this with 

, and we have constructed the perfect barriers to full data flow and collaboration.

Data privacy, sovereignty, and shareability are significant barriers to access and collaboration. Without coherent governance and policy management built into data management tools, interoperability within and external to organisations is almost impossible.

Often referred to as data velocity, incoming data is growing exponentially. It is estimated that we currently have over 

. To put that in context, if you took an average smartphone’s storage capacity, 44ZB would fill 188 trillion phones, which if laid end-to-end would circumnavigate the earth about 670 times.

. A threefold increase in data storage at the scale of data that already exists presents a big problem given the way in which our data is currently architectured and governed.

These hardwired and structural barriers have resulted in the burden of ingesting and processing data at scale falling on the few. To resolve this, we must shift ownership to the highest levels of our institutions. This will involve: redesigning our systems; building strong governance settings; establishing data teams; creating better tools; and, providing pathways to training.

Ultimately, to truly become AI-ready we must build a data culture capable of allowing science to flourish, unlocking new value, and solving the wicked problems faced by humanity.

Scientific advances, real world benefits

Journal entries