23

Siri must "listen" for commands, however, that being said if it does not receive the command prompt "Hey Siri," can users rely on the algorithm to not record/send conversations or other private audio?

If device specificity is required for the question. Assume an iPhone X with IOS 13.X

0
19

Your asked about the trust-worthiness of an algorithm. The answer to your question is a firm and absolute "No" - users can not rely on the algorithm to not record/send conversations or other private audio. Some algorithms are definitely better than others, but no algorithm is perfect - what is invented by the human mind can be defeated by the human mind.

Let's muse over another word in your question: rely. Reliance is not a binary state of mind; it's much more subtle than that. Consider that we may rely on the mail carrier to deliver our mail each day, but as the consequence of failure is not catastrophic, we do not rely upon it heavily. OTOH, if you are a diabetic living in a remote location, your reliance on the mail carrier's delivery of your insulin has far greater implications.

The "threat" is also a factor in our reliance. Security-wise, perhaps the thing we have going for us is that nobody really cares what we say or write or do. But if we happen to attract attention to ourselves - perhaps as a suspect, or the target of a criminal enterprise, or have become involved in "matters of national security" as this chap recently did, then our level of reliance on algorithms - and computer security in general - becomes a different matter. And so it is with your reliance on Apple's algorithm: does the mail carrier bring you junk, or does he/she deliver insulin?

Finally, let's consider the other variable in reliance: the quality of the algorithm. Reliance is not a binary state, and no algorithm is perfect. Nevertheless we must still consider how good it is in determining how heavily we should rely upon it.

Unfortunately it is a practical impossibility for us to know how secure Apple's algorithm is:

  • it's not open source - people outside Apple have to disassemble the code to inspect for flaws. They cannot read the source

  • even if it were open source, it wouldn't be perfect

  • even if it were perfect, Apple still makes mistakes

  • Apple could even abuse their privilege or not catch an employee that attempts this abuse

Instead, and as a substitute for a quantitative analysis of the algorithm itself, perhaps the best we can manage is an attempt at objective assessment of some facts. Consider the following:

  1. A complete and massive failure of the algorithm would be bad for Apple's business. Consequently, it seems likely that Apple would put a serious effort into the algorithm, and safeguards for the data.

  2. Apple has weighed up Siri's business value against the risks of failures of their algorithm, and have made the decision to retain Siri. Impose may be a better word than retain in this case as Apple has not provided an option to remove or un-install the Siri software.

These facts do not shed much light on the actual quality of Apple's algorithm, but they do speak to motives and motivation. We can safely conclude that Apple has reasonably strong financial incentives to design a reliable algorithm. However, it also seems safe to conclude that Apple considers our private data to be within their purview. Why else would they not provide an option for removal of Siri to accommodate their customers' preferences?

None of this is intended as "Apple-bashing". I am an Apple customer - I have spent more money than I'll ever admit on their products. But I feel it's also important to be realistic. Apple's interests do not coincide perfectly with ours. Should we rely on Apple's judgments and algorithms to protect and safeguard our security and privacy? I hope this answer has provided a little food for thought.


Epilogue:

I have just read @Bradley's answer to this question. I have to say that if the allegations leveled by Apple's contractor in Ireland over the "Siri Incident" are true, then Apple deserves far more than the "Apple-bashing" I tried to avoid in my answer. They deserve substantial fines - perhaps even criminal proceedings. It is simply unacceptable that anyone or any organization be allowed to use their wealth and influence to trample the rights of others. More to the point of answering this question, this episode (if true) speaks volumes on how we should consider relying on Apple's software.

0
12

The answer is it is always listening. The recent article in the Guardian, of the former apple contractor, demonstrates precisely why the best option for you is to disable Siri all together if you want to mitigate the risk of yourself being recorded. Equivalently, buy the cheapest non-smartphone.

This is not just an Apple-centric issue. Unfortunately, when you accept the terms and conditions, you are giving Apple, and other companies the right to use your data however they want. This should not be the case when you are not using an application, but many times over it is the case.

8
  • 2
    Excellent points! I wasn't aware that the contractors involved were in Ireland; this is a spectacular irony given all the tax breaks Ireland awarded Apple! And you are correct - these T&Cs that demand we surrender the rights to our data are simply an affront! I must take some issue w/ you on your point about disabling Siri - Disabled software is a far greater risk than non-existent (deleted) software. Finally, kudos to Le Bonniec - I suppose Apple regards him much as the NSA regards Edward Norton. – Seamus May 23 '20 at 19:54
  • 3
    @ankii This is not fear-mongering. If I had stated my answer without evidence, then I would say your criticism is fair, but I have provided justification for my claim. In addition to this, Siri is listening, as the contractor and article state, regardless of whether you consciously , or un-consciously give it an order. – Bradley May 24 '20 at 9:33
  • 3
    The article doesn’t mention if it was listening even after I turned off “hey siri”. And I don’t mind anyone listening if I do it after I hold my home button to activate siri. Further, the whistleblower demands punishment for the past actions, which I support. But also want to highlight that he doesn’t say if it is recording now too, after the said ios update. – anki May 24 '20 at 10:52
  • 1
    When you click the "disable !! Siri !!" button, the idea of "trusting" that some programmer disabled the audio from that point is ................ just astounding. I mean you might as well believe in garden fairies. Like, on what possible basis would you believe that the "Siri !! Off!!" button disables anything? Why? Like, there are no laws whatsoever compelling Apple (or similar organizations) to "disable! Siri!". For example - when you "unsubscribe!!!" (rofl) from some marketing lists, there are at least some vague laws in most jurisdictions making it mean something. – Fattie May 24 '20 at 21:47
  • 1
    @ankii As an example: Samsung "smart" web enabled TVs had a 'feature' where they would relay watching information back to Samsung. Users could 'disable' this feature. Disabling the feature disabled telling the user that it had not been disabled and was in reality still sending the same information to Samsung. . Really. || Some/many/most/all home market routers were find to have backdoors that allowed the manufacturers to interact with them - ostensibly for purposes of maintenance/upgrade. I do not know what use the NSA made of this feature. – Russell McMahon May 25 '20 at 7:08
11

We can only tell you what Apple publicly states. Can you rely on it, well that's entirely up to you. IMO, I trust Apple and Microsoft orders of magnitude more than I do Google, Facebook, Twitter, et al. That doesn't mean I trust them implicitly, either.

Specific to this question, it is addressed in this press release from Apple: Improving Siri’s privacy protections. Below is a quick summary of what's in the press release.

How Apple Protects your Privacy

  • They try to do as much of the Siri processing on the device as possible
  • Any data stored on Apple servers are not used to build a marketing profile nor is it sold to anyone
  • Siri tries to use the least amount of private data as possible. The example used was for general queries, they would use a general location, but for more specific and granular requests, more detail is used
  • Siri dictating messages is done on the device and not stored on the server

Changes they are making to improve your privacy

  • audio recordings to improve Siri are no longer kept, only computer transcriptions will be used to improve Siri
  • Siri's improvement program is opt in rather than opt out; meaning you have to affirmatively give Apple permission to use your recordings
  • Only Apple employees will be able to review customers recordings

So, based on this, is Siri listening?

Yes.

Am I worried about it?

Not worried, exactly. I trust it, for now, but remain skeptical. Siri is listening, but if most of what Siri does is on device, there's no need to record/process your conversation until it hears "Hey Siri;" and responding to that has no need to be sent to servers for initial processing.

1
  • Comments are not for extended discussion; this conversation has been moved to chat. – bmike May 25 '20 at 18:07
2

The German-language TV show Quer, broadcast by BR, the Bavarian public broadcaster, aired a segment back in March (prior to Covid-19-related restrictions in Bavaria) which can be found on Youtube (the auto-translated subtitles aren’t bad in my initial impression). About 1 minute in, they team up with the Chaos Computer Club to determine whether Siri sends data while listening without the trigger words Hey, Siri being added by putting themselves between the router and the internet to record every packet of data sent.

The positive control (a statement initiated with Hey Siri) immediately sends out data as supposed. Later in the segment (about 3:40) they perform a negative control and see absolutely no data usage. This negative control was designed to use a couple of trigger words (holiday, flight) that show the desire to purchase a non-everyday product making the speaker an incredibly interesting target for targeted advertising – one of the main criticisms that are being voiced.

Obviously, a 5-minute segment in a 45-minute show that covers a range of topics cannot cover all relevant aspects. Most notably, they only show checking for outgoing internet connections so any recordings kept locally on the phone would slip through the cracks. Furthermore, n = 1 for both positive control and experiment as of the broadcast segment which is obviously far below statistical significance. Nonetheless, this experiment provides an indication towards a No answer: at the very least it seems like Siri is not sending private communications to the outside world.

6
  • Actually, what we can conclude from this experiment is that Siri does not send out data immediately. Or, more precisely, that it does not do it all the time. – Federico Poloni May 25 '20 at 14:15
  • I wonder how Apple came to be in possession of all of this data? Do you feel that Thomas le Bonniec & his cohorts have distorted the facts? – Seamus May 25 '20 at 21:30
  • 1
    @FedericoPoloni That is a good point; but obviously leads to the reverse question of when it would be sending such data instead. The candidate that seems to suggest itself most is during nighttime backup (i.e. connected to wifi and charger while screen locked for extended periods of time); the nature of this makes it somewhat harder to determine what is being sent at this time. – Jan May 27 '20 at 3:11
  • 1
    @Seamus Imho, there’s a very simple way to reconcile both statements: The mic picked up something that it perceived to be an activation call but this was due to substandard voice recognition (He does mention that his job was to assist in speech-to-text). Another option is, as Federico suggested, a time-delayed data sending. At present, I don’t have enough evidence to discard either hypothesis; I did, however, feel that the aspect I posted was lacking in other answers so I felt the need to include it. – Jan May 27 '20 at 3:15
  • @Jan Consider also the option that there is a hidden switch to turn on 'snooping mode' remotely, on an individual basis for a single target user, for the benefit of various three-letter agencies. This may sound like conspiracy theory, but there is evidence that similar snooping techniques are used (Snowden's leaks, _NSAKEY, Staatstrojaner...), and frankly speaking if I were the head of NSA I would consider my staff to be highly incompetent if they could not pull this off somehow (by pulling strings at Apple, bribing or threatening a key sysadmin, etc.). – Federico Poloni May 27 '20 at 6:36

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .