Project

General

Profile

Feature #275

Audio playback service

Added by Soren Stoutner over 1 year ago. Updated 7 months ago.

Status:
New
Priority:
4.x
Start date:
05/04/2018
Due date:
% Done:

0%

Estimated time:

Description

I received the following in an email from a user:

Sorry to bother you, but maybe your knowledge might shine a
light on an old "playing an audio stream stops after screen went off"
issue. I have 6 phones here, Android 5.0.2, three 6.0.1 and two 7.1.2
versions, all showing the same behaviour with almost all browsers,
running both the Webkit or Gecko engine. To reproduce the issue you can
do the following (note: site needs 1st party cookies and JS):

1) go to http://radio.dos.nl/ and select one of the radio stations (you
can swipe L/R between pages)

2) turn off screen, the sound continues for a few minutes

3) after some time the sound stops, BUT: a network dump shows that
the stream continues (!)

4) switch on the screen, the sound restarts, but all you hear is what
remains in the buffer and when the buffer is empty after some seconds,
the sound stops. And again: the (inaudible) network stream still
continues.

5) The inaudible network stream stops when the "stop" button is
selected, so apparently PB still has some control over the stream.

6) the inaudible network stream will not continue eternally, after some
time the inaudible stream stops anyway.

Apparently PB (and many many other browsers) do not agree with the OS
on what to do with the stream. OTOH: there are a few browsers which
work as expected: e.g. Chromium, Chrome, FireFox and Opera, although
the latter lacks mp3 support due to license issues.

What these working browsers have in common is a loudspeaker icon in the
top bar and when the top bar is pulled down, a control panel is shown,
even when the screen is locked (see attachments)

Would that small icon hide the solution to this problem? Is there maybe
some vague library call that is used for playing audio streams which
causes the icon and control panel to be present?


Files

pulled-down.jpg (419 KB) pulled-down.jpg Soren Stoutner, 05/04/2018 11:03 AM
top.jpg (179 KB) top.jpg Soren Stoutner, 05/04/2018 11:03 AM
Android Profiler Screenshot.png (79.1 KB) Android Profiler Screenshot.png Soren Stoutner, 05/10/2018 03:54 PM

History

#1

Updated by Soren Stoutner over 1 year ago

This is the response I wrote to the email:

I have not spent the time necessary to completely analyze this exact scenario, but I have enough background with this topic that I think I can help shed some light on what is happening.

1) In Android, by default, when an app is switched away from the foreground, its processes are paused. Therefore, Android pauses JavaScript execution on browsers when the screen is turned off or a different app is moved to the foreground. In addition, Privacy Browser explicitly pauses JavaScript as part of the onPause command.

2) It is possible for an app to start a background service that runs all the time. This is separate from the main foreground app. It is that background service that creates the notification icon and allows music to play when an app is not in the foreground. Many of the major browsers include this type of background service for some types of audio playback. Privacy Browser does not currently include any background services. I am not categorically opposed to adding one, but before I would do so I would need to spend some time making sure there was no way a website could abuse such a service to compromise the privacy of a user. In addition, the development of such a service would likely be time consuming, so I would push it off until other more important features are implemented.

3) Regarding the continuation of the audio stream when the screen is turned off, audio streaming usually use the UDP protocol. UDP, unlike TCP, does not send a response packet indicating that the data arrived successfully. Devices connected to UDP streams typically send a periodic packet to the server to indicate they are still listening. If the server does not receive the packet at the expected interval, the stream is disabled. I worked with one streaming service once that would go continue streaming for 5 minutes between control packets. So even if the browser was closed and the device was turned off, the router would continue to receive packets until the 5 minute mark was reached. In the case of the Android device with the screen turned off, the OS is receiving the UDP packets and then discarding them because there is no program in a running state that is able to receive them.

4) When the screen is turned back on, the JavaScript state is restarted, including any data in the stack (which includes the audio buffer).

5) My guess is that when the browser is restarted it is not reconnected to the UDP audio stream, which, nevertheless, will continue until the timeout. I am not sure why this happens, but it likely has something to do with an interplay between the Android OS networking subsystem and the Android activity lifecycle. However, hitting the stop button sends the stop command to the server, which stops the stream.

https://developer.android.com/guide/components/activities/activity-lifecycle

#3

Updated by Soren Stoutner over 1 year ago

Ok, thnx for your time! Nice explanation! The only thing that differs
is that there are no UDP streams available on that site, and allmost
all radio streams are tcp streams, I haven't seen any UDP version yet,
and I wonder if the html5 <audio> tag is capable of playing UDP streams,
I have no idea.

I don't know the answer to that either.

In case of UDP you're right, but as stated before, this is all TCP, and
the phone is still tranmsmitting ACK replies to the server, otherwise
the server would stop to send packets immediately if it doesn't receive
ACK packets with the appropiate sequence numbers.

Sorry. As I mentioned, my explanation was based on my previous experience with audio steaming and not a thorough investigation of what is going on in this case. The end results are similar, in that the stream continues as long as the device is on, because when the OS networking stack receives the packet it sends an ACK. Later in the process, the OS networking stack discards the packet because there is no program ready to receive it.

But anyway, you say it's javascript, so I made a simple page without JS,
just the audio tag and a control setting (most browsers ignore the
"autoplay" option, that's why I use JS for that), I just made the page
and I see exactly the same behaviour:

http://radio.dos.nl/test.html

My understanding is that HTML 5 clients actually use the JavaScript engine (V8 in the case of WebView) to process audio and video tags, even if JavaScript is not explicitly mentioned in the HTML. That is why HTML 5 audio and video playback does not work when JavaScript is disabled.

After all, I conclude that this phenomenon is "normal" behaviour
if you do not explicitely exclude the radio stream from the onPause()
process, like Chrome and FireFox seem to do.

It isn't exactly that one needs to exclude the radio stream in onPause(). onPause() stops all of the app's foreground processes and there is not way to exclude any of them. Rather, it is that the app needs to spawn a separate background service and pass the audio off to it.

If you ever add the option to enable playing audio after onPause() I
think it would be a good idea to add it as a per site option like JS and
cookies.

Yes, once the background audio service is built, it shouldn't be that difficult to control it with domain settings. Note that if WebView does not provide controls for extracting the audio stream and passing it off to a background service, it may be necessary to wait until the Privacy WebView fork before this can be implemented. I haven't spent enough time looking at it yet to be sure, but if that is the case this feature will need to be moved to the 4.x series.

#4

Updated by Soren Stoutner over 1 year ago

I spent a bunch of time testing on various devices from Android 5.1.1 to 8.1.0. All of them were running Privacy Browser 2.8.1 with WebView 66.0.3359.126.

In my testing audio continued to stream after the screen was turned off or the app was placed into the background for both the JavaScript and non-JavaScript sites. Audio would typically stream for 3-5 minutes. It isn't clear to me if the audio was buffered before Privacy Browser was placed into the background and was simply being played out by the OS audio service, or if WebView is spanning a background service that continues to receive data and play audio until some timeout is reached (because the background audio service isn't communicating with the server correctly). My guess is the most likely explanation is that the OS is buffering the audio, which would explain the variation in the time the audio plays.

If the screen turns off and the user turns it back on before the audio stops playing Privacy Browser is able to reconnect to the stream. It is only a problem if the audio has stopped, in which case turning on the screen sometimes plays a brief few seconds of audio (my guess is audio that was still being processed by Privacy Browser and was paused with the other app resources) but then stops again and doesn't reengage until the play button is pressed again.

And you are correct. Both audio and video will play without JavaScript enabled if only the HTML5 tags are used.

#5

Updated by Richard Lucassen over 1 year ago

One of the things you did not mention in your research is that under water the stream continues to play even though you don't hear anything. And the few seconds of sound you hear after you turned on the screen is what's in the buffer, just before the audio was switched off.

After some time, 10 minutes or so, the inaudible network stream stops as well. I don't know if you're able to see real time what's happening on the network, I think it might be quite interesting for you to see what's happening. I'm working in the network business and I'm surrounded by lots of Linux boxes so inspecting the network is no problem for me.

There is another small thing that I'd like to mention: FireFox Klar 4.0.2 (in Fdroid store) works like a charm, even without the icon and a control panel like FireFox and Chrome, but an upgrade to 4.2 make FireFox Klar behave like all other browsers.

https://f-droid.org/packages/org.mozilla.klar/

Anyway, I think that your analysis that FireFox and Chrome are forking another program is most logic explanation. It seems to me that this might be the only correct solution to resolve this background problem.

R.

#6

Updated by Soren Stoutner over 1 year ago

I did see the network stream continue in my testing. I just didn't mention it because it had already been established.

Attached is a screenshot from Android Profiler. Just before 3:32 Privacy Browser is placed in the background. The network stream continues on the lowest chart.

What is interesting is the CPU activity. The hump at 3:37 is normal, in the sense that no matter what Privacy Browser is doing or not doing (playing audio, displaying a static page) this hump appears a few seconds after being placed in the background. I don't know exactly what is happening here, but it is likely some OS activity relating to the app losing foreground status.

Also, note that CPU activity continues with 1% blips every few seconds, but not as frequently as the 1-2% blips when Privacy Browser was in the foreground. Originally I thought this might be caused by WebView automatically spawning a service to handle the audio and that service's activity being displayed. But that doesn't match for several reasons. 1) A background service should have a different PID (instead of 20540) and so should not appear on this chart. 2) If a background service were being spawned and then stopping for some reason after a certain amount of time, I would expect that time to be consistent. But my testing showed an inconsistent timeout between 3-5 minutes.

This indicates an OS level audio buffer. I didn't think the OS would buffer sound for 3-5 minutes, but it appears to be possible and is the only explanation that matches all the behavior.

That still leaves unexplained what is causing the 1% blips in CPU usage for an app in a suspended state in the background. My guess is that it is actually a false reading, in the sense that the service running on the phone that reports the statistics to the Android Profiler is probably using resources to gather the data and they are being reported as belonging to Privacy Browser. I have seen this sort of behavior in the past, where the profiler consumed resources that were then reported as belonging to the app (which kind of defeats the point of having a profiler if the resource usage is different when it is running).

#7

Updated by Soren Stoutner 7 months ago

  • Priority changed from 3.x to 4.x

Because Android's WebView does not provide any method, that I am aware, to extract a current audio stream and pass it off to a background service, this feature request will likely have to wait until the 4.x series with Privacy WebView.

Also available in: Atom PDF