Why artificial intelligence must disclose that it's AI

File photo - A robot from the movie is on display for the premier of the motion picture Terminator 3 "Rise of the Machines" June 30, 2003 in west Los Angeles. (REUTERS/Mike Blake)

Google recently repitched Duplex to explicitly reveal to restaurant hosts and salon staff that they are speaking with the Google Assistant and are being recorded.

Google omitted this small but important detail when it first introduced Duplex at its I/O developer conference in May. A media backlash ensued, and critics renewed old fears about the implications of unleashing AI agents that can impersonate the behavior of humans in indistinguishable ways.

By tweaking Duplex, Google puts to rest some of that criticism. But why is it so important that companies be transparent about the identity of their AI agents?

Can AI Assistants Serve Evil Purposes?

"There's a growing expectation that when you're messaging a business, you could be interacting with an AI-powered chatbot. But when you actually hear a human speaking, you generally expect that to be a real person," says Joshua March, CEO of Coversocial.

March says that we're at the beginning of people having meaningful interactions with AI on a regular basis, and advances in the field have created the fear that hackers could exploit AI agents for malicious purposes.

"In a best-case scenario, AI bots with malicious intentions could offend people," says Marcio Avillez, SVP of Networks at Cujo AI. But Marcio adds that we may face more dire threats. For example, AI can learn specific linguistic patterns, making it easier to adapt the technology to manipulate people, impersonate victims, and stage vishing (voice phishing) attacks and similar activities.

Many experts agree the threat is real. In a column for CIO, Steven Brykman laid out the different ways a technology such as Duplex can be exploited: "At least with humans calling humans, there's still a limiting factor—a human can only make so many calls per hour, per day. Humans have to be paid, take breaks, and so forth. But an AI chatbot could literally make an unlimited number of calls to an unlimited number of people in an unlimited variety of ways!"

At this stage, most of what we hear is speculation; we still don't know the extent and seriousness of the threats that can arise with the advent of voice assistants. But many of the potential attacks involving voice-based assistants can be defused if the company providing the technology explicitly communicates to users when they're interacting with an AI agent.

Privacy Concerns

Another problem surrounding the use of technologies such as Duplex is the potential risk to privacy. AI-powered systems need user data to train and improve their algorithms, and Duplex is no exception. How it will store, secure, and use that data is very important.

New regulations are emerging that require companies to obtain the explicit consent of users when they want to collect their information, but they've been mostly designed to cover technologies where users intentionally initiate interactions. This makes sense for AI assistants like Siri and Alexa, which are user-activated. But it's not clear how the new rules would apply to AI assistants that reach out to users without being triggered.

In his article, Brykman stresses the need to establish regulatory safeguards, such as laws that require companies to declare the presence of an AI agent—or a law that when you ask a chatbot whether it's a chatbot, it's required to say, "Yes, I'm a chatbot." Such measures would give the human interlocutor the chance to disengage or at least decide whether they want to interact with an AI system that records their voice.

Even with such laws, privacy concerns won't go away. "The biggest risk I foresee in the technology's current incarnation is that it will give Google yet more data on our private lives it did not already have. Up until this point, they only knew of our online communications; they will now gain real insight into our real-world conversations," says Vian Chinner, founder and CEO of Xineoh.

Recent privacy scandals involving large tech companies, in which they've used user data in questionable ways for their own gains, have created a sense of mistrust about giving them more windows into our lives. "People in general feel that the large Silicon Valley companies are viewing them as inventory instead of customers and have a large degree of distrust towards almost anything they do, no matter how groundbreaking and life-changing it will end up being," Chinner says.

Functional Failures

Despite having a natural voice and tone and using human-like sounds such as "mmhm" and "ummm," Duplex is no different from other contemporary AI technologies and suffers from the same limitations.

Whether voice or text is used as the interface, AI agents are good at solving specific problems. That's why we call them "narrow AI" (as opposed to "general AI"—the type of artificial intelligence that can engage in general problem-solving, as the human mind does). While narrow AI can be exceptionally good at performing the tasks it's programmed for, it can fail spectacularly when it's given a scenario that deviates from its problem domain.

"If the consumer thinks they're speaking to a human, they will likely ask something that's outside the AI's normal script, and will then get a frustrating response when the bot doesn't understand," says Conversocial's March.

In contrast, when a person knows they are talking to an AI that has been trained to reserve tables at a restaurant, they will try to avoid using language that will confuse the AI and cause it to behave in unexpected ways, especially if it is bringing them a customer.

"Staff members who receive calls from the Duplex should also receive a straightforward introduction that this is not a real person. This would help the communication between the staff and AI to be more conservative and clear," says Avillez.

For this reason, until (and if) we develop AI that can perform on a par with human intelligence, it is in the interests of the companies themselves to be transparent about their use of AI.

At the end of the day, part of the fear of voice assistants like Duplex is caused by the fact that they're new, and we're still getting used to encountering them in new settings and use cases. "At least some people seem very uncomfortable with the idea of talking to a robot without knowing it, so for the time being it should probably be disclosed to the counterparties in the conversation," says Chinner.

But in the long run, we'll become accustomed to interacting with AI agents that are smarter and more capable at performing tasks that were previously thought to be the exclusive domain of human operators.

"The next generation won't care if they're speaking to an AI or a human when they contact a business. They will just want to get their answer quickly and easily, and they'll have grown up speaking to Alexa. Waiting on hold to speak to a human will be much more frustrating than just interacting with a bot," says Conversocial's March.

This article originally appeared on PCMag.com.