Initial criticism of the Media Bias chart was detailed in a 2018 article written by then-journalism student Claire Hamada and published on Medium. In addition to asserting that Ad Fontes Media had not released its data sets, Hamad wrote that the chart’s authors did not employ a coding reliability coefficient in their analysis.
Upon further examination of the Ad Fontes Media website in mid-August 2019, extensive data sets are not presented for scrutiny and there is no mention of the coding method employed to organise data, both of which are common practice. The lone presentation of raw data appears on this page where two stories are analysed, though the coding method is not explained in detail.
It is clear that Ad Fontes Media’s underlying data on not available for scrutiny. Ortero admits as much on the FAQ page: “The data I have compiled so far is in my own spreadsheets and notebooks.” She continues: “I do not have a study that documents and reports my data because… I didn’t start this project as a study… and… my current data set that I have developed since the beginning of this project is still incomplete and insufficient to satisfy standards of academic rigour.”
Ortero adds: “I am developing a study and an accompanying paper that will hopefully form the basis for ongoing data-driven content analysis and display.” This information was not available on the Ad Fontes Media website as of mid-August 2019, nor were data sets which Ortero claims should be available by “mid-2019.”
Other entries on the Ad Fontes Media site suggest that should the data be made available, it is likely that it would be subject to significant criticism or scrutiny. Consider the passage “The sources I initially chose include those I read most often and those I am exposed to most often through aggregators or other sources. The also include sources which I have reason to believe many others are exposed to most often.”
The subjective nature of Ortero’s analysis is made more apparent in the statement: “Note that I did not quantitatively determine how many sites are out there on each partisan side. Some people object to this and believe there are far more trash websites on one side or the other. I do not have the time or resources to conduct such a quantitative measure, so I did not conduct one”.
She goes on to suggest that the data sources are not listed as the additional word count would be “too much” for some readers. Moreover, on a page where Ortero asks for individuals to take part in their study, there is no detailed description of their expected tasks.
Taking these factors into consideration, Ad Fontes Media and items it produces – even when taking into account some experienced members of its Advisory Board – are likely best understood as informed but also amateur in nature. This assertion is supported by a number of different observations of Ad Fontes Media’s website, including a lack of concise language in the methodology, some grammatical errors (suggesting it has not been reviewed by many people), and the existence of a merch page.
But even if all these factors were to be set aside, it remains possible that engaging in such analysis is a largely fruitless endeavour, a point outlined in the aforelinked article by Tamar Wilner in the Columbia Journalism Review, in which the Ad Fontes Media Chart is harshly assessed. Regardless, without the underlying data being made public and subject to scrutiny, the question of whether this chart was produced using conclusions from a rigorous, robust process cannot be definitively answered.