Opinion: China Isn't the AI-Powered Dystopia You Think It Is

In the 18th century, English philosopher Jeremy Bentham envisaged a theoretical “Panopticon” prison with one guard observing all the inmates, who were unable to tell whether they were being watched. Media reports have increasingly drawn parallels between Bentham’s panopticon and surveillance in China. These reports have noted various signs with apprehension: a pervasive use of facial recognition applications, surveillance systems, and plans for a universal social credit system.

China strives to be a world leader in new technologies, yet at the same time, it has created a cyber regime that may hamper Chinese innovation by making international collaborations problematic. Access to major overseas markets and research facilities will be hindered for some companies with a close association to the Chinese security apparatus.

An AI-powered Chinese dystopia faces many obstacles, including:

1. The immense practical difficulty of creating a unified social credit system;
2. The positive innovations of facial recognition companies;
3. Increased transparency from open source algorithms;
4. Greater demands for privacy protection from Chinese citizens.

Creating a unified social credit system is hard

The development of a social credit system in China may seem sinister, but three elements invite moderation: the system is not one unified apparatus, it began attempting to solve a huge problem–the large unbanked economy–and it has run into many hurdles.

In July last year, the Chinese government delayed the public launch of the program, announced in 2014, saying the siloed data was “unscientific”. This is because the government and eight tech companies each operate a separate pilot scheme, for which they use only their own proprietary data, refusing to share it with each other.

There’s now a widely shared view that China’s vast data reserves that power the pilot credit systems will also help Chinese AI development. But previously Chinese bureaucracies kept siloed and haphazard data sets, including for calculating personal credit.

Further, China is also notorious for its shadow banking industry. The future social credit system could help to control this murky shadow banking industry, currently without regulatory oversight.

Read more: China’s Social Credit System: AI-driven panopticon or fragmented foundation for a sincerity culture?

The private sector’s internet data sets can solve this problem.

These datasets are also relatively new. WeChat was only released in 2011 and WeChat Pay came along in 2014. Alipay was founded in 2004, super-charging Alibaba’s e-commerce empire with its Western legal version of trust: the escrow system. This is the same “trust” problem that these pilot credit systems attempt to resolve.

Some aspects of the pilot social credit systems are bizarre and highly disruptive. For example, when jaywalkers get fined, it lowers their credit score and the government will not allow people with low scores to travel via plane or train. For the companies running their pilots, like Alibaba and Tencent, this becomes a PR problem.

And there are also many global examples of companies making credit decisions based on “non-traditional but otherwise reliable data:” the Philippines’ Lenndo, German Kreditech, and Hong Kong’s WeLend, that also has a successful Chinese subsidiary (我来贷). These are private companies, and China’s social credit systems are initiated by the government, but creating a lending system through “non-traditional but otherwise reliable data” is viable with some checks and balances.

Social credit system watchers have known for a while, as academic Rogier Creemers’ put it, that: “[t]he primary role of the [social credit system] is to engineer trust in the marketplace and in social conduct, as well as providing means of internal oversight of government officials. In that sense, it’s actually quite boring”.

The jury is still out on China’s social credit experiment. In its final form, it may not be a government-run system of control but rather a badly needed banking product.

Tech companies offer multiple products

In China, you can start to board trains, flights, or even pay for your KFC meal using facial recognition. While much of this rapid innovation has been driven by consumer convenience, some of the products resemble an episode of the dystopian sci-fi TV series Black Mirror.

But tech companies offer multiple products. Companies that develop facial and voice recognition technologies, also build products in fields like medicine. iFlytek is investing resources into medical diagnostics, including creating a robot that passed China’s medical exam. This will help the scarcity of doctors in rural China.

Along with smart city partnerships with Microsoft, YITU Tech worked with Alibaba’s cloud computing arm to build a real-time cloud system for the Guizhou Traffic Police. The company is also looking at “tough medical problems we all face,” like using AI to diagnose lung cancer.

In addition to being deployed in hospitals around China, YITU Tech has signed strategic agreements with hospitals from Taiwan, South Korea, and Japan.

We cannot understate the tenacity and skill of Chinese entrepreneurs in building talented teams, creating new innovative products and better algorithms, and building global partnerships. They walk regulatory tightropes with great skill.

China’s “Skynet” is opaque

There are, of course, also various potentially more sinister aspects of facial recognition. Now the world’s second largest AI start-up firm, SenseTime said recently that as far as it knows, Chinese police have only used the company’s tech to catch criminals. For these companies, this is a hard PR problem for global expansion.

There are CCTV cameras everywhere globally, the issue is the opaqueness of the Chinese system.

Hikvision (海康), a real-time surveillance program for public security, is a key player in China’s “Skynet” surveillance system. It is also a major supplier in the surveillance industry globally. The Chinese government owns a 42% stake in Hikvision, which makes a fifth of all cameras currently sold worldwide.

In 2017, BBC reporter John Sudworth agreed to be tracked by the system. It took just seven minutes for it to locate and “apprehend” him.

At the same time, the company’s facial recognition system has reportedly decreased red light violations by over 90 percent at one intersection in Suqian, Jiangsu.

Xie Yinnan, VP of Megvii (China’s third largest AI company), told NPR that the government’s procurement of facial recognition technologies has helped rapid innovation. “The government is pushing the need for this technology from the top. So companies don’t have big obstacles in making it happen,” Xie said. “In America, people are too busy discussing how they should use it.”

Watch: Facial recognition is on the rise in China

“We just provide the government the technology, and they do their job with it. Cameras in China are set at 2.8 meters above the ground,” he added. “That means they won’t be able to capture human faces. That’s a rule. Chinese citizens know that, so they don’t think about it too much.”

But that statement needs to be validated, as privacy concerns increase in China and a new privacy regime emerges.

The Chinese demand for privacy

Western media often say that Chinese people don’t worry about privacy, but the evidence is mounting against this notion.

In late March, a Chinese TV investigation found that users of free WiFi apps did not actually understand that by using the apps, they were agreeing to upload a list of all the networks they have ever connected to—including their own homes and workplaces. Users were able to save on mobile data at the cost of giving their own personal data.

Furthermore, according to a recent survey carried out by State-run CCTV and Tencent Research, 76.3 percent of the 8,000 participants feel that AI is a threat to privacy. According to the survey, they mostly fear facial recognition technologies. There is a greater public consciousness of these new emerging technologies.

And China’s privacy regime is evolving: the government is legislating new protections like the country’s new Cyber Security Law that, amongst other aims, bans online service providers from collecting and selling users’ personal information. New Chinese privacy standards also took effect on May 1, 2018.

Of course, these legal rules may not pertain to the Chinese government but there are indications of growing citizen awareness of data privacy. This is a growing issue for China’s tech companies.

YITU Tech research scientist Dr. Wu Shuang said in March that better understanding of facial recognition is needed for reasonable public dialogue.

Dr. Shuang said the technology is still misunderstood: “I believe facial recognition has improved a lot in recent years,” he noted, “It is still not widely understood how much better we have gotten, even in the industry. The industry and general public have to be informed about this rapid progress, and then we can talk about how to apply such advanced technology in a more reasonable way. We need to establish industry standards, and also policy-makers need to set up policies and regulations for facial recognition to be used properly and widely, and to void the ambiguities and doubt about how it can be used.”

In other words, policy has not yet caught up with the technology.

A global push means an open source approach

Privacy is a universal issue and PR is increasingly important for companies seeking to go global. On March 28, Hikvision announced it would provide open source access to its AI technology. Hikvision manufactures front-end cameras and AI chip powered data analysis systems. The company will also be opening an AI development platform, and the company’s AI Cloud will be merging algorithms from other AI companies.

How this will work remains unclear but this Chinese report provides some explanation, saying Hikvision aims to build a global security industry ecosystem.

In early March, Hikvision also opened a new Source Code Transparency Center (SCTC) in California attempting to demonstrate its commitment to security and transparency. Similar data centers may also be possible in China under the country’s new Cyber Security Law.

These are either PR exercises or a genuine attempt to share source code. And there are caveats to these sorts of disclosures. But Hikvision’s global business partners can decide that.

Nevertheless, in late May the US government banned Hikvision in the US, citing security concerns.

An open source software approach does not prevent companies like Hikvision from closely assisting the Chinese government in creating an Orwellian surveillance system. However, it is a sign of the unresolved public-private tensions, as Chinese companies increasingly go global and exponentially need global R&D talent to develop AI products.

Public perceptions

This is a new era built on the polarity of decentralized open data blockchain and proprietary dataset-driven AI. Many ethical questions will need to be addressed by Google, Facebook, and Amazon and by Chinese AI companies: Baidu, iFlytek, Sensetime, and Hikvision.

So while fears of a Panopticon 2.0 persist, various domestic, international and commercial factors may prevent it from being realized. And as demands for privacy grow louder globally, China too is beginning that conversation. Bentham’s dystopia is not guaranteed.

Photo courtesy of TechNode