How reliable is your AI content creation? The use of AI for content generation is increasing in popularity, but there are still questions about its accuracy and reliability. Concerns over potential inaccuracies range from distorted facts to incorrect interpretations of data to data bias and failure to include inclusive language.
The cost of wrong information may be trivial, but it will tell your readers that you might not be trustworthy and reliable. It is essential that users of AI content generation tools are aware of the risks and take steps to reduce the chances of inaccurate information.
It is not only fair but necessary to put AI through its paces on some of the issues discussed. Where appropriate, I have included examples of my interactions with OpenAI.
Concerns about AI Content Creation
The Telephone Game
As kids, we played The Telephone Game. The premise is simple: the first person whispers a sentence into the ear of the person sitting next to them. The second person in the circle does the same and this is repeated until the last person gets the message and shares the message with the group, often with hilarious results.
The problem that this game highlights is that small changes to the message accumulate over time, and the final has little resemblance to the original.
It is not necessarily true that anyone in the circle tried to change the message. The whole point is to try to convey the message as accurately as possible.
The sheer volume of information available today means this distortion of facts will happen.
Inaccurate Facts
We assume that “facts” are true and many dictionaries will agree with that. A better definition would be that a fact is what people believe or consensually accept as truth. The ancient Greeks believed the world to be round, but proponents of a flat earth held sway for centuries until someone challenged that “fact” and disproved it.
Inaccurate facts can be relatively harmless, such as when a business is open, or they may be more far-reaching.
Two Examples
I asked OpenAI to evaluate and correct the statement, “The population of Columbus, Ohio is 300,000 and consists mostly of people below the age of 52.” Its response was, “Yes, I can correct that statement. The correct statement would be ‘The population of Columbus, Ohio is 892,533, and the median age is 31.9 years.’”
Its source for this statement is https://worldpopulationreview.com/us-cities/columbus-oh-population. The data has since been updated and the variance is within reason and to be expected.
I asked if it was correct that 50% of the world’s population was female. OpenAI responded, “Yes, it is true that approximately 50% of the world’s population is female. According to the United Nations, the global female population is estimated to be 3.7 billion, while the global male population is estimated to be 3.6 billion.”
Its source for this information is https://www.un.org/en/sections/issues-depth/population/index.html which is no longer available. Because this is a site managed by the United Nations, the information at the time was likely reasonably correct.
Incorrect Interpretation of Data
Data interpretation is dependent on the quantity and quality of the underlying data sources.
One famous example of an incorrect interpretation of data is Amazon. Beginning in 2014, the company developed machine learning systems designed to screen candidates.
The idea was that the system could look through hundreds of applications and filter out the candidates who did not meet the basic requirements. They trained the system to vet applicants based on the resumes the company had received over the previous ten years. This introduced an incorrect bias that did not recommend women because most of the applicants during that time period the system trained on were men, reflecting the male dominance in the industry. (Reuters).
Not Using Verifiable Quality Sources
We see this problem even with Google and Bing searches today. There is a lot of content out there, and the quality of information varies dramatically. We want to think that what makes it to the top of the search rankings is the best and most authoritative answer to our questions.
There is no guarantee of that. A quick scan of the results will show that some information comes from reputable sources and some do not.
It would be nice if the search results could give me a consensus on the topic. However, all too often, the general consensus has been wrong. Even experts in any field will sharply disagree with each other, begging the question: who do we trust?
OpenAI provided me with good information, but I wanted to see if it could cite its sources. I asked for links to other interactions we had while researching this topic. It provided me with two links to articles, one by Forbes and the other by CIO.
Unfortunately, the links were no longer available. OpenAI finished training its models in early 2022 and some of the sites it trained on are no longer available. Systems with access to newer data should be able to provide current information.
Data Bias
Bias in datasets occurs when statistics based on those datasets do not accurately reflect the target population. Data bias can happen when:
- The sample size for the dataset is not large enough.
- Segments of the population are not included in the data collection.
- The data is influenced by historic and cultural beliefs.
Failure to Include Inclusive Language
“Inclusive language is more than just avoiding the use of a few antiquated or offensive terms and phrases. It is about embracing communication that acknowledges the power differentials and dynamics of our society and their deleterious effects. It is about showing appreciation for the diversity everyone brings to the table. And finally, it is about creating cultures where people can feel free to be their full authentic selves.” (Efua Andoh, American Psychological Association)
“Using gender-inclusive language means speaking and writing in a way that does not discriminate against a particular sex, social gender, or gender identity, and does not perpetuate gender stereotypes. Given the key role of language in shaping cultural and social attitudes, using gender-inclusive language is a powerful way to promote gender equality and eradicate gender bias.” (United Nations – Gender Inclusive Language)
AI systems can recognize exclusive language and even make recommendations on how to adapt text accordingly. It is ultimately up to the writer to decide to do so.
Tips for Reliable AI Content Creation
AI content creation holds promise but is not a replacement for well-researched and well-written human content. Readers look for, and Google wants to promote, content that demonstrates expertise, authority, and trustworthiness
- Research thoroughly and use quality sources to ensure the accuracy of facts.
- Take steps to verify sources and ensure that data is not biased.
- Double-check generated content for errors and ensure that it is up-to-date.
- Avoid making assumptions and look for alternative interpretations of data.
- Include inclusive language when generating content to ensure that it is not exclusionary.
These are best practices regardless of how you write the article.
AI systems are complex tools with a steep learning curve. It is entertaining to ask simple questions and get simple answers, thoughtful and more meaningful answers take time. The quality of the output is only as good as the quality of the input and that includes how you interact with them, the kinds of questions you ask, and how you phrase those questions. Recognize that these tools do make mistakes.
Perhaps the best advice is, “Use with caution; the contents may be hot.”