Survey wars: A murky tale
Scott Handsaker: Some of the results don't pass the pub test. That's a problem
Three significant surveys of the Australian startup sector have been released in the last week, with each painting a very different picture of the small industry.
The contrasting results have led to calls for the various surveys to work together to enable more accurate data to be used to guide policy and funding decisions.
Startup Muster, the largest survey of the Australian startup sector, outlined its first ever negative results, finding a decline in the number of active startups in the country, as well as a decline in the number of companies being launched.
The next day, Victorian government body LaunchVic released findings from its own survey of the state’s startup sector. In stark contrast to Startup Muster’s report, the survey found Victoria’s startup sector to be growing rapidly and evolving positively.
And this week another significant survey was unveiled, with the EY and Fintech Australia FinTech census finding that FinTech startups specifically are growing in terms of revenue and employees.
These contrasting and sometimes confusing surveys need to be taken with a grain of salt, a number of leading figures have said.
The studies are important, but they’re not necessarily an accurate summary of the sector as a whole, according to CyRise founder Scott Handsaker.
“Surveys of the Australian startup ecosystem are useful and we should support them as much as possible. But we should also exercise caution in interpretation,” Mr Handsaker told InnovationAus.com.
“Extrapolating trends from incomplete data sets is likely to give erroneous results.”
“It’s unfortunate because it would be incredibly beneficial from a public policy perspective to have accurate data on the underlying startup community here in Australia.”
In isolation, none of the surveys provide an accurate and comprehensive overview of the sector, Victorian-based entrepreneur and founder of Schedugram Hugh Stephens said.
“The degree of sample bias and unbounded extrapolation we’re seeing from startup community surveys is insane,” Mr Stephens told InnovationAus.com.
“The conclusions that are claimed and we’re seeing reported drive great clicks but do little to represent the reality or diversity of the ecosystem they aim to represent.”
“While we should celebrate and embrace efforts, any kind of survey targeting a small sector should be done in collaboration instead of competition. I am fearful that the results of any of these surveys might be used to guide funding or policy decisions in isolation.”
Startup Muster’s report, in collaboration with the CSIRO’s Data61, found that there were 1,675 active startups in Australia in 2017, that number dropped to 1,465 in 2018.
It also found that there were 1,291 startups founded between the 2016 and 2017 report, but only 712 launched in the last year.
It also found startup activity to be particularly on the decline in Victoria, with the state hosting 18 per cent of all startups in 2015 and only 13 per cent in the latest survey.
In contrast, the LaunchVic report found that there are up to 3,200 startups and scale-ups just in Victoria. A report by Startup Genome last year estimated there were between 1,300 and 2,100 startups in Sydney alone.
These differences could be down to different definitions of a startup, the inclusion of larger scale-ups, methodology and bias, and it’s important to create more cohesive data on the sector, Mr Handsaker said.
“If LaunchVic are right, then Startup Muster are out by a factor of 10 and Startup Genome by a factor of three. If Startup Muster are correct, then LaunchVic’s definition of a startup is massively inflated. Both are probably wrong,” he said.
“Some of the high level statistics in Startup Muster don’t pass the pub test and are unlikely to be true.
"Who really believes that in a growing economy and expanding population, that Australia went backwards in the number of startups?"
“Is there another developed economy on the planet where the number of startups is running downhill? Maybe then, the sky is not falling and journalists should be more skeptical of underlying data sources before running their stories.”
“So, is LaunchVic correct with almost 3,000 startups in Victoria alone? I suspect their trend is at least in the right direction, but 3,000 sounds high and doesn’t pass the pub test either. It is hard to tell without seeing the underlying data, but the most likely cause here is an overly generous definition of what constitutes a startup.”
Startup Muster’s definition of a startup is an “early stage business that has a large addressable market that utilises technology to capture that market quickly. It received 140,000 answers in total from 3500 people or organisations.
After an “exhaustive post-survey validation process”, the survey includes answers from 777 founders, 321 future founders and 654 supporters.
LaunchVic’s study focused both on startups and “scale-ups”, and included data collected on over 2,770 Victorian companies. More than 1,600 startups and scale-ups completed an online survey over the last two years.
The EY FinTech census was the result of 151 FinTechs completing an online survey and 12 qualitative interviews with “FinTech leaders”. It found that one in five of the surveyed FinTechs are now profitable, up from one in seven last year.
“It also found that year-on-year median revenue growth has increased by 125 per cent.”