The robodebt scheme was an example of a government “beta testing” algorithms on its most vulnerable citizens and failed to properly account for fundamental principles of accuracy, accountability, and fairness, according to the former Australian human rights commissioner.
Ed Santow, who spent years warning of the dangers of poorly implemented technology as Australia’s human rights commissioner, said on Wednesday that Australia’s lack of strategic artificial intelligence (AI) skills was “incredibly dangerous” and could lead to more programs like the “robodebt disaster”.
The Online Compliance Intervention system, better known as robodebt, was launched by the Coalition government in 2016. It used an algorithm to average out a welfare recipient’s yearly income using data from the ATO and cross-matched this with income reported to Centrelink.
The scheme raised $1.7 billion in debts against 443,000 people and regularly incorrectly matched data, leading to inaccurate or non-existent debt notices being issued.
In awarding victims of the scheme a $112 million settlement in June, a Federal Court judge blasted the program as “unlawful” and a “shameful chapter” in Australia’s social security history.
Mr Santow was highly critical of robodebt as Australia’s Human Rights Commissioner during his five-year term, which ended in July.
Now leading a responsible technology program at the University of Technology Sydney as industry professor of responsible technology, Mr Santow said robodebt was an example of an AI or algorithm-based technology being deployed by governments without adequate ethics, testing or recourse for people impacted by it.
“[AI] is often pretty experimental technology,” Mr Santow said during a University of Technology Sydney (UTS) responsible AI webinar on Wednesday.
“Historically, not just in Australia, but a lot of countries have got a very bad record of beta testing new technology on literally the most vulnerable citizens in our community.
“And, frankly, that’s what seems to have happened with robodebt.”
Mr Santow said the automated technology should not have been trialled on vulnerable citizens without stringent safeguards, and potentially not at all.
Any deployment of AI or algorithm-based technology needs to satisfy fundamental principles of accuracy, fairness and accountability in order to minimise risks, Mr Santow explained.
But robotdebt had not adequately addressed any, he said.
Accuracy is “crucially important” when governments make decisions using technology, Mr Santow said, but the robotdebt scheme was found to have very high rates of error, including raising debts with individuals who owed the government nothing.
The scheme also lacked simple redress mechanisms for when errors were made, with victims forced to “untie the Gordian Knot” with lawyers to get any sort of accountability, the former human rights leader said.
Mr Santow also questioned the overarching fairness of a scheme that raised relatively small debts from many years prior.
“Sometimes when you’re cutting money back from someone that you may have overpaid $100 five, six, seven years ago, maybe it’s actually not really fair to claim that money back.
“So, you need to have an overarching kind of look at the system that you’re creating and making sure that it really works fairly for people.”
Mr Santow said Australia must learn the lessons from robodebt, but this will require more people with “strategic” AI skills to avoid similar problems.
The CSIRO has forecast that on current trends Australia will be 70,000 graduates short of what is required to meet demand for technical AI skills like data science by 2030. Mr Santow said the technical skills challenge is well understood and several moves are being made to address it.
But another “incredibly dangerous” skills gap of strategic AI expertise is widening, he said.
Mr Santow pointed to research showing organisations are under pressure to deploy AI technologies but most had almost no idea how to do it correctly. The pressure would lead to irresponsible and dangerous use of the technology, similarly to robodebt, he said.
The former human rights commissioner joined UTS this month to lead a responsible technology initiative. He will lead a push by the university to educate companies and government agencies from leadership down on the risks and opportunities of AI.
“We really want to be at the forefront of building Australia’s AI capability so that companies and government agencies can use artificial intelligence smartly, responsibly, and in accordance with our liberal democratic values. And that means respecting people’s basic human rights.”
Earlier this week, Perth lawyer Lorrainne Finlay was appointed as Australia’s next Human Rights Commissioner by the Coalition government. Ms Finlay has rarely broached technology issues in regards to rights and has been critical of the Australian Human Rights Commission in the past.
Crisis support is available from Lifeline on 13 11 14.
Do you know more? Contact James Riley via Email.
Whatever happened to the compensation payment that was awarded on top of the refund? Did the law firm keep it?
On 02/12/2015 a robodebt was raised against me. After applying it to a part 9 agreement I am still trying to get the full value of my robodebt returned for me. Centrelink is saying as they agreed with the 3rd party for a 72% return to recover the illegal debt that they are only required to pay back the 72% of their full robodebt to me. Dirty government can take pay rises for doing a conniving job and ripping off the lowest people in their society.
Does anyone really know what effects this rip-off truly had on part time workers that were properly disclosing any income and then to be treated like a criminal thanks so much for a couple of hundred dollars compensation for all the hours, weeks, months to plead my innocents while still trying to work!!! I would so love to have the wealth to sue the f***ers
The purpose of robodebt was to resolve the ridiculously high amount of debt owed to the government by parenting payment recipients.
What they have overlooked is the majority of debts owed are a result of child support not being paid, paying parents incorrectly reporting income resulting in the recieving parent/main care giver recieving a debt.
The gov has been quietly trying to back track and resolve this since implementing there algorithm last December (well timed over the Christmas period)
If I am able to identify and do the maths what strikes me as concerning is who is overlooking and approving these “changes”, who is providing information to the “approver” and why aren’t the implementation of algorithm going through user testing groups before they press the button.
It’s a reflection of the culture and issues with the APS, the staffing cap essentially let them promote themselves despite the fact they have no experience or capability, let them manage themselves with no HR present.. it’s a mess.
Implementing tech needs to go through proper testing. Let alone developed with someone that knows how the relevant legislation interacts.
I never got any money back and have not herd any more about it.its money i could certainly use right now
It actually had nothing do with that also concerning issue.
It was explicitly about earnings income. Also, it was logically and legally flawed from the outset.
I believe in December last year the government implemented algorithms to balance single parent fortnightly FTB with child support on a fortnightly basis, not the previous annual basis.
The algorithm did not take into account:
Child support only pays child support into recieving accounts on a monthly basis, 6th of each month.
Issue: If child support is calculated on a monthly basis and services Australia is applying it on a fortnightly basis the figures will be inaccurate.
Child support figures used in the algorithm were based on the amount assesed, not the actual amount paid/recieved.
Issue: FTB reduced, child support not recieved the main care giver is left below the poverty line with no pathway/access to effective and speedy recourse.
The previous minimum FTB rates were over ridden by the calc resulting in single parents with unpaid child support recieving 0 FTB, 0 Child Support and living of the 820 sole parenting payment.
The government acknowledges the parenting payment amount is only economically viable in combination with the Fortnightly FTB payments.
I never got any money back and have not heard any more about it. Please, where is our compensation money. I need it now.
Thank you