Australia’s public sector has spent years building the digital foundations for more connected, responsive and secure services. But as agencies work towards the ambitions of the 2030 Data and Digital Government Strategy, one of the biggest barriers is not a lack of data. It is the inability to turn that data into timely, operational intelligence.
For Anna Mascarello, Elastic’s A/NZ regional vice president of public sector and education, the challenge is clear: government has no shortage of data, but too much of it remains trapped across systems, tools, teams and jurisdictions, inhibiting access to information.
“Data is everywhere, but intelligence is nowhere,” Ms Mascarello told the Commercial Disco(very) podcast.
The problem is not unique to government. It is a legacy of how organisations have evolved over decades, often buying or building separate tools to solve specific problems.
Security platforms, observability platforms, reporting platforms and data platforms where each is designed for a particular purpose, creating valuable but disconnected silos of information.
For public sector agencies, the consequences from inability to access data insights can be serious. During major crises such as the Black Summer bushfires, the pandemic or large-scale cyber security incidents, the ability to understand what is happening in real time can shape the speed and quality of a response.
“Imagine an analyst arriving at work on a Monday morning,” Ms Mascarello said.
“They could be a security analyst investigating a threat that occurred over the weekend. They could be someone in service delivery trying to understand why a citizen portal went down at 2am on Sunday morning. Or they could be part of a policy team trying to answer Senate Estimates questions about the outcomes of a particular program.
“The use case doesn’t matter. The experience is the same – they’re trying to find a needle in a haystack.”
In many agencies, the data needed to answer those questions already exists. But it may sit across different systems, with different teams, different retention policies and different levels of accessibility. This creates a complex environment where being able to easily search for information across sources becomes very difficult and time consuming.
Ms Mascarello said this creates a data architecture gap rather than a pure technology gap.
“Agencies have spent 20 years buying tools that solve narrow problems… but they’ve never had the ability to look across all those systems and derive unified insights.”
Historically, attempts to solve this have often meant large-scale transformation programs, expensive migration projects and attempts to consolidate data into centralised platforms. But Ms Mascarello said this approach can be too slow, too risky and too costly for agencies already grappling with ageing infrastructure, security requirements and budget pressure.
This model relies on separating the layers: how data is collected, where it is stored and how it is analysed. This is sometimes described as a data mesh or a decoupled data strategy.
As Ms Mascarello put it, the strategy shifts toward bringing “the analytics to the data, not the data to the analytics”.
That means data can be collected close to where it resides, including across different classification levels or hosting environments, while agencies gain the ability to search and interrogate it for insights. Elastic’s Cross-Cluster Search enables this kind of distributed visibility, allowing organisations to query data across different environments without forcing everything into a single monolithic store.
For agencies managing complex estates, this can be a significant shift.
A civilian agency, for example, may have separate teams responsible for IT operations, cyber security, insider threat detection and compliance. Traditionally, each may use its own tools and data pipelines. But the same underlying information – network logs, endpoint telemetry or user activity – may be relevant to all of them.
With a shared data foundation, operations teams can use the information for performance monitoring, security teams for threat hunting, and compliance teams for audit trails. When incidents occur, agencies are not scrambling to manually piece together evidence from disconnected systems.
For agencies managing separate tools across SaaS, legacy, and air-gapped environments creates significant operational complexity.
To solve this, Ms Mascarello said organisation need flexibility and choice in how they deploy technology because missions, classification levels and operational constraints vary significantly.
However, they need this flexibility without fragmenting their operations. This requires a high interoperable technology platform which can be deployed in a standardised way across these different environments to provide a unified operational view.
For example, a recent partnership embedding elastic into Google Distributed Cloud allows agencies to maintain an identical technical foundation across both connected and disconnected environments. This consistency reduces training burdens and ensures teams have a single, coherent view across their entire estate.
This approach has already proven successful globally.
Elastic’s work with the US Cybersecurity and Infrastructure Security Agency avoided a costly rep-and-replace program, instead standardising data collection to enable whole-of-government threat detection.
Ms Mascarello said these lessons are directly relevant to Australia as agencies strengthen sovereign capability through the Australian Signals Directorate’s (ASD) REDSPICE program, where the vital starting point remains visibility.
“Can Australian agencies answer what’s happening in their network right now? Can ASD see across the entire government estate when a threat moves laterally between departments?
“In many cases, that’s still a challenge.”
The pathway to more proactive, AI-scaled security begins with consistent logging, normalised telemetry and a shared capability that allows agencies to see across their environments. This lays the foundation for security use cases such as threat hunting, detection engineering and automated response, which are built on top of the data infrastructure layer.
“You don’t jump straight to sophisticated capabilities until the foundation is in place,” Ms Mascarello said.
For agencies facing budget pressure, workforce constraints and large legacy estates, this is the critical point. Better visibility does not have to begin with a massive transformation program.
Ms Mascarello said that this visibility is rooted in a strong search foundation. Elastic began as a search company, which brings a unique perspective to its observability and security solutions. The ability to leverage modern search techniques integrated with agentic AI solutions natively on the platform, tightly alongside system telemetry, allows organisations to realise the benefits of AI in responsible setting.
“When you think about it, observability and security are fundamentally search problems. The experience and what you’re looking for – performance bugs and malicious actors – is just a little different to what we typically think of for search as a capability.” she said.
This is the misconception she most wants to challenge; that unified visibility requires agencies to rip-and-replace everything they already have.
“It doesn’t,” she said.
“Elastic is able to interoperate with existing systems as a unified data and search layer. Legacy systems can remain in place. Existing data stores can stay where they are. This approach allows agencies to start deriving value for their data in weeks rather than years.”
For a public sector under pressure to deliver better services, strengthen cyber resilience and modernise without losing control of mission-critical systems, that may be the real opportunity: not more data, but a better way to act on it.
This article was produced in partnership with Elastic.
Do you know more? Contact James Riley via Email.