I fact-checked the ABC, Anglicare, and myself. We were all wrong. Then things got weird.
On Tuesday, 10 August, ABC reporter Michael Janda published a story titled “Universal basic income attracts 77pc support in Anglicare poll” (since changed to “Basic income attracts 77pc support in Anglicare poll”). The article went on to say that the survey “shows more than three-quarters of Australians back a basic income guarantee above the poverty line.” This too has been changed. The section, as of last night, read “more than three-quarters of Australians back a permanent basic income”. It had been changed again by this morning to say “some kind of basic income”. All versions of the article are wrong. That’s not what the polling shows.
The polling question from which the 77 figure is derived asks only “Should every Australian have an income above the poverty line?”, without specific reference – at least not in the question itself – to basic income, or any government policy. The question does not even ask, for example, whether it is the government’s responsibility to ensure everyone has an income above the poverty line, just whether respondents think all Australians “[s]hould… have” an income above a certain level. There is a description of a basic income in the poll, and one of the questions asks respondents how a “permanent basic income” would change their lives and behaviour, but there is no question which asks directly about the implementation of such a policy.
The poll numbers were touted in a report by Anglicare, which claims “[a] very strong majority of respondents supported the creation of a permanent basic income, with 77 percent in agreement overall.”
Anglicare have since explained the discrepancy as a “versioning error”, which lead to confusion regarding the wording of the question that was included in an Ipsos omnibus poll.
But Anglicare and the ABC were not the only ones who made a mistake. On the basis of what they had published, I also posted a story about it (since retracted) on our company blog, with those same numbers in the headline. However, in that blog I noted that neither the ABC’s article nor the report itself seemed to contain the exact phrasing of the question, and that I had emailed the author of the report asking for it.
Eventually, a spreadsheet of questions and reply data, as supplied to Anglicare by Ipsos, was provided to me. That is when I first noticed the actual wording of the question. I was anything but thrilled to uncover the error, as you can see from my reaction in the research portal immediately below.
Not captured in the system, but available here, is an initial exchange between myself and Mr Janda of the ABC, who I contacted about his error via an ABC webform on Friday 20 August. In it he defends the headline and the article, saying:
As outlined in the article, options presented for a basic income ranged from an expanded JobSeeker payment without mutual obligations (effectively making it universal) to a full aged pension scale payment to all adults. Clearly, the poll is not necessarily indicating 77 percent support for the latter, but it does show 77 percent support for policy changes to ensure that no-one lives below the poverty line, and that is certainly one version of a universal basic income.
This is incorrect. A Jobseeker payment without mutual obligations would not be universal, as it would presumably not be paid to those with incomes from jobs. That means while it might be free of a requirement to actively seek work, it would still be means tested. Also, the survey question does not explicitly show support for policy changes of any kind, only support for the statement all Australians “should… have” an income level above the poverty line, which is not a policy but an outcome. It does not ask “should the government provide [or guarantee] an income above the poverty line?”
Some percentage of people might believe the best way to achieve a zero-poverty outcome is a cultural change, such as millennials learning the value of a hard day’s work, or whatever. This subset, or a subset of this subset, might in fact be vehemently opposed to basic income as a policy, believing it will increase poverty in the long run by reducing the incentive to work and causing inflationary shortages.
If this first hurdle had been cleared (it was not, but if it had been) then a second hurdle would remain in terms of which policy the government should use to achieve this goal – be it a means tested income guarantee, a universal basic income, or some other policy, such as a job guarantee or wage subsidy.
People may have expected follow up questions where they would get to clarify that. If there had been a second question asking whether a basic income was the best way (or a way they would support) to achieve that outcome. If that question has been asked, I am certain there would have been some people who had responded positively to the question about outcomes who then responded negatively about a basic income as a policy.
Consider both the results of the YouGov poll commissioned by the green institute last year and the way it was reported by your colleague at the ABC.
The result found net support of 58% for a basic income, which was described *in the question* rather than at some other point in the call. Either public opinion on the issue has shifted by 19% in the intervening months, or the wording of the question affected the outcome. Since we cannot rule out the latter, we can have no confidence about the former.
Mr Janda also said that he had “previously had discussions on the issues you raise with professional pollsters and statisticians” but did not answer when asked, twice, whether those discussions related to this survey in particular, or had been more general. Nor did he respond to a question about why his reporting of the poll had not conformed to the standards set by the reporting of his colleague Gareth Hutchens when covering the Green Institute/YouGov poll from 2020, specifically regarding the inclusion of the polling company and exact question wording, or whether the ABC had specific rules or guidelines about reporting polls.
On 23 August, I published the first version of this article. That was the second article about the poll – correcting the first – but sharing with it a common research project. It was up for less than an hour.
I had written to both Maiy Azize and Michael Janda the previous Friday, telling them I had discovered what I thought was an important discrepancy, and would publish a correction to my now retracted blog (which had depended on them as sources) by 11am on Monday. At 11:30, just as I was sharing, I received an email from Azize with a Word document attached entitled “Final survey questions”, which Azize said was the correct phrasing of the survey questions as asked, with the spreadsheet containing inexact, outdated, or placeholder questions. I retracted the article to review this new material.
In the Word document the phrasing was: “Do you support the introduction of a permanent basic income in Australia?” Note that this is what is was reflected in Jandas first revision. But that was not correct, and the current wording is is still not correct.
Unsatisfied with an explination where the Ipsos data-set was casually mislabelled, I looped them into the conversation, asking them to verify that the wording in the Word document, not the spreadsheet, was what had been asked.
Before they responed, Ms Azize did, reversing her position following a conversation with Ipsos:
This reaction is quite different from that of Mr Janda, whose position was initially at least, that the difference between explicitly mentioning a basic income and the more general wording was inconsequential. I should note that it is my understanding that Mr Janda has been made aware that the spreadsheet (from which he originally worked) is correct .
I should also note that my correspondence with Mr Janda went beyond that included in the pdf linked to above. After that point, it moved across multiple email threads, sometimes involving other people and direct messages on Twitter. I do not feel his position has altered much from his initial stance. I have invited him to join me on a Zoom call to discuss this further if he feels I have unfairly represented his position. If I have done so, I hope he will agree that I have not done so out of malice.
I had no intention or expectation of catching anyone out. Nor do I claim to have stumbled on a vast conspiracy. But sometimes from the outside, incompetence and conspiracy are indistinguishable (to paraphrase Mike Duncan). The truth is that standards of accuracy, documentation, and general thoroughness are low across the industry. Mr Janda has by no means been shown to be anywhere near the worst offenders in the media. He is probably one of the better reporters, as far as I can tell.
I am sure that your motives are of the absolute best kind – compassion for the needy. And you are not responsible for the dysfunctional nature of the media ecosystem, where precision often gives way to a tactical skirmishes in rough ideological terrain.
But my day job, when I am not working to advocate for a basic income, is trying to improve that media ecosystem. In the long term, that will create an environment far more conducive to the kind of progressive change we both want.
That “day job” is running the team which built the research capture software this article demonstrates (it’s the video player above, and the software that recorded the screen-cap videos). That is how I can be so sure about exactly what happened when and how during this confusing and sometimes upsetting process. But capturing our research processes does not just change how we interact with them after the fact, it makes the research itself better, too.
In the 7th and 8th research highlights in the player above, you see me – fully convinced of the polls veracity – note verbally that I have not yet found the exact wording, and committing to contact the report’s author and request it. Had I not been using our software, I would not have verbalized these thoughts, and probably not emailed the author – I certainly would not have recorded myself doing it. But because of it I did, and so, one week later, I was compelled to do a follow-up I thought would be entirely perfunctory. It turned out to be anything but. By making my process transparent, I had removed the option of just losing interest when it got too hard. I was strapped in. But there was something liberating about that, too. I was never worried that I could come across as a bad faith actor, out to smear the good people of Anglicare and the ABC. If I was right, there was no malice in it. If I was wrong, I was wrong in good faith, and that would be clear to anyone who cared enough to look back over my process, which had been documented thoroughly.
Indeed I was somewhat dissappointed to have my suspicions bout the survey questions confirmed. I had grown quite attached to the headline of my correction’s correction, which I was going to call “I wasn’t wrong! I was right the first time.” It would have still been a great demonstration of the power of our tool – and the need for better information-hygene in the public sphere, which our tool is here to catalyse.
If, as I suggested to Mr Janda in our exhange, journalists at the ABC were to use a tool like this, we would see similar improvements in their performance, as they hold themselves to account before others have a chance to. I am sure our encounter has already affected Mr Janda’s future work. He won’t make the same mistake again. Stone is already changing the media, even with only one user.
The software is still in beta, but an open beta meaning anyone can use it. They might just experience some minor bugs. For example, in this story, the Zoom conversation with my colleagues which was uploaded as supporting media has minor audio issues and appears out of proper chronological order. You probably would not have noticed that if I hadn’t pointed it out. But I am convinced that in order to preserve and expand a functioning public sphere, at least some portion of those active in the public sphere must become, and you might want to sit down for this – it’s pretty cringe – accountability junkies. It’s getting people on board with that which is our real challenge, the technology is just a means to that epistemilogical ends.
This is the opposite approach taken by many in the stop-fake-news space, whose approaches, more or less, are all a variation on having a committee of experts (and/or an “AI”) tell people where and from whom they should get their news. Such approaches seek to build legacy journalism a grand fortress, in which it would only starve.
Where they seek to build walls, we instead build windows. Join the transparency revolution today.
Note: this article was updated again following its second publication, to reflect a second revision to Mr Janda’s article.