The surveillance economy has set its sights on smartphone sensors

The most successful companies of our time are those who’ve mastered user data – collecting it, analyzing it and profiting from it – typically at the expense of user privacy.

The Internet of things (IoT) has given tech companies, advertisers, data brokers and others in the surveillance economy the ability to track not just the actions we take on our screens but the actions we take in our homes, including what we say and what we do. Now these companies are going a step further, turning their attention to extracting valuable data from the sensors within the smartphones we take with us everywhere we go.

In this article, I’ll explain why smartphone sensors are the next big target for advertisers and look at some of the steps that users can take to defend against sensor-based mobile tracking.

The rise of sensor panic

A number of recent news stories have placed into focus the public’s fear of a world that’s always watching, listening and monitoring. In mid-February, an airplane passenger’s discovery of a camera located beneath the inflight entertainment screen was cause for alarm from many observers. That news was closely followed by revelations that Google, maker of the Nest Secure home security system, had originally failed to disclose that the device included a built-in microphone. This new phenomenon is so palpable that it’s even been given a name: sensor panic.

Fueling this phenomenon are two key trends that have been building over many years. The first trend is the explosion of inexpensive, Internet-connected sensors – including microphones and cameras – in everything from smart speakers to doorbells, under the banner of the IoT. The second trend is the erosion of trust in tech companies, the result of a steady stream of gross privacy violations like Google mischaracterizing its location tracking practices and Facebook providing certain companies with special access to user data.

Indeed, our collective techlash didn’t happen overnight; we’ve come a long way from brandishing pitchforks at Facebook’s introduction of its news feed in 2006 to simply shrugging our shoulders at yet another data privacy scandal in 2019. As Facebook, Google and Amazon evolved to dominate their industries and our lives off their ability to extract, make sense of and monetize user data – privacy be damned – we’ve gradually become numb to their violations. This is surveillance capitalism at work.

It’s not a coincidence that these three tech giants sell their own sets of sensors in the form of virtual assistant-based smart speakers (Amazon Echo, Google Home and Facebook Portal), giving each company a potentially valuable window into user behavior. To get an idea of how these companies may intend to leverage data from these devices, we can look at patents that have been filed. One of Amazon’s patent applications describes a system for deriving user preferences from ambient speech, while one of Google’s describes a system for recommending products based on identifying a user’s possessions.

Beyond fixed cameras and microphones

For most of us, smartphones are now the undisputed center of our digital universes. And the apparatuses of surveillance capitalism have followed suit, using the smartphone’s unique capacity for location tracking to track users in highly targeted ways. Given that tech giants have shown an understanding of how sensors can be used to observe and target users, it stands to reason that the next privacy battle will take place for the sensors built into the mobile devices that are always in our possession. In fact, we can already see this starting to happen in the wild. Importantly, while current permission models for both iOS and Android gate access to sensors like cameras and microphones, websites and apps are typically given full access to lesser-known sensors like the accelerometer and proximity sensor.

In recent years, researchers have started looking at some of the creepy possibilities of this brave new world of sensor-based mobile targeting and advertising. Here are some of the potential use cases:

• Device fingerprinting: Advertising networks rely heavily on browser fingerprinting, which involves capturing a user’s unique set of browser configurations (like version number and active settings) to track users across websites. But because mobile browsers have a relatively homogeneous set of configurations, this technique is much less effective for smartphones. Researchers have discovered an alternative, showing how motion sensors – specifically the accelerometer – can be used to fingerprint devices by measuring anomalies in signals; these anomalies are based on tiny variations in the electromechanical structure resulting from imperfections in the manufacturing and assembly processes.
• Inferring user activities: Researchers have also had success using the accelerometer and gyroscope to infer a user’s activities, whether that’s walking, sitting, standing, biking or jogging. With a good guess as to which activity a user is doing, ads can be micro-targeted to the user.
• Verifying ad impressions: Generally, ad impressions are counted every time an ad is delivered, meaning there’s no way to guarantee whether a real user actually received the message. But researchers have shown that the proximity sensor can be used to verify whether or not a real person is present when an ad is delivered.

And if an app or website is given access to a device’s cameras and microphones, it can potentially do much more, including:

• Inferring a user’s likes as detected through conversation keywords
• Serving ads based on identification of a user’s personal objects
• Tracking a user’s reactions to displayed content
• Serving or modifying ads based on detection of a user’s facial expressions.

To see how this sensor-based tracking may play out in the real world, let’s examine a scenario in the not-too-distant future. Let’s say that I’m training for a marathon and decide to take a quick action selfie using a social media app. I’ve already given the app access to my smartphone’s cameras and location. The app can:

• Infer that I’m running (via the motion sensors)
• Monitor my fatigue level (via the front camera and motion sensors)
• Determine where I am and where I’m headed (via location services)
• Infer that I love to drink smoothies, based on past web searches associated with my device’s sensor fingerprint.

The next ad I see is for a smoothie bar located in the direction I’m heading. The ad contains an eerily accurate headline: “You deserve a post-run pick-me-up.”

Defending against mobile tracking

While it can feel daunting to escape the tentacles of the surveillance economy, there are a number of steps you can take to keep your smartphone from being used for tracking purposes:

1. When installing/using an app or visiting a website, understand the privacy implications of every permission request.
2. Review your app settings periodically to ensure that only necessary permissions are granted for each app.
3. Utilize tools within your mobile operating system to limit sensor access; the release of Android Pie added the ability to restrict sensor usage by idle apps, for example, while a beta release of iOS 12.2 added the ability to restrict motion sensor access within the Safari browser.
4. Install a mobile ad-blocking app to prevent third-party tracking code from accessing your device.
5. Invest in an anti-surveillance smart case to keep cameras and microphones from being used when you don’t expressly permit them.

As in every battle for user data, there will be a tug-of-war between those on the side of privacy and those on the side of profit. Those of us standing on the side of user privacy will need to stay vigilant about how the sensors at the heart of our most important devices are used against us.