Why is no one address the Arlo Ultra live 4k lag...stop acting like it's my hardware

[fhang82]

Horrible lag using this arlo ultra security system. There is a 3 to 5 seconds lag when viewing 4k live and using two way audio. From the time I see my kids walk up the drive way and I say HI to them as they walked up to the house. They have already entered the house and have closed the door behind them. That is horrible live lag knowing that I would have been robbed by the time my live stream picked up the robber walking up my driveway. This is not a security camera if the lag is this bad.

[trinatron]

3 - 5 sec delay? Mine can be anywhere from 5 - 10sec with the 4K live view. I’m guessing it may have something to do with amount of data and compression required to transmit 4K over WiFi. I’ve heard the video doorbells have minimal delay but they’re not 4K

[StephenB]

---

@trinatron wrote:
I’m guessing it may have something to do with amount of data and compression required to transmit 4K over WiFi.

---

I don't think so (at least not for most people).  Ultras generally are sending ~ 3 megabits.  Most wifi networks can handle that fairly easily.

I don't know what the cause is,  just saying that I don't think it's that.

[trinatron]

---

@StephenB wrote:

---

@trinatron wrote:
I’m guessing it may have something to do with amount of data and compression required to transmit 4K over WiFi.

---

I don't think so (at least not for most people).  Ultras generally are sending ~ 3 megabits.  Most wifi networks can handle that fairly easily.

I don't know what the cause is,  just saying that I don't think it's that.

---

Just keep in mind that a typical 4K stream usually sends anywhere between 15Mbps - 25Mbps depending on the codec being used. Hence why I think there's some serious compression occuring at the Arlo camera if it's only sending around 3Mbps. The compression will take time to process.

Although I do believe Arlo should be able to get the delay down to a consistent 2 - 3 sec max as long as the cameras have strong WiFi connection with minimal interference.

[LandJS]

I don't even want to get into figuring out what goes on with the compression that would effect the delay.  What I can say from my own experience is that it was much faster 15 months ago when I first purchased mine.  It now takes at least twice as long as it did but that was quite a few firmware updates ago so many things have probably changed.

[trinatron]

We should also mention that Live View via the Apple HomeKit has no delay and often starts the Live View much quicker than Arlo’s own app. It’s not 4K though.

[StephenB]

---

@trinatron wrote:

---

---

Just keep in mind that a typical 4K stream usually sends anywhere between 15Mbps - 25Mbps depending on the codec being used. Hence why I think there's some serious compression occuring at the Arlo camera if it's only sending around 3Mbps. The compression will take time to process.

---

Sigh.  I've been working with video compression for about 30 years now.  

Compressing to a lower bit rate actually takes fewer resources in the camera than compressing to a higher bitrate. More blocks in the video are skipped, and fewer symbols are entropy coded. It is true that there are some advanced features in the compression that will use more resources if you use them.  But that's not really in play here. 

If the lag were a result of the camera compression, then we'd all be seeing this lag all the time.  But that's not the experience.  The lag can be very long some days, and  much shorter other days.  Some users rarely see it, others routinely see it.  

Personally my guess is that sometimes the lag is caused by the Arlo Cloud not being able to keep up.  In some cases, upload congestion could also be the cause - especially if your internet service has a slow uplink speed.

[trinatron]

---

@StephenB wrote:

---

---

Compressing to a lower bit rate actually takes fewer resources in the camera than compressing to a higher bitrate. 

---

I'm not sure which chips have been incorporated into the camera, but even with a dedicated VPU, it still won't have anywhere near the processing power of a modern PC.

---

@StephenB wrote:

Personally my guess is that sometimes the lag is caused by the Arlo Cloud not being able to keep up.  In some cases, upload congestion could also be the cause - especially if your internet service has a slow uplink speed.

---

But why should local live view need to even talk to the cloud or depend on uplink speed. It's connecting directly to the smarthub via WiFi? 

[LandJS]

trinatron best reply yet

[trinatron]

@LandJS 

Thankyou, it's much appreciated. Being a longterm member, I'm hoping you maybe able shed some light on something I've been noticing.

I'm new to the Arlo community, but it seems like that the only person from Arlo who tends to respond, is JamesC. Now i've got nothing against JamesC as I think he's doing an unbelievable job responding to all these queries, but we never really get responses or confirmation on the more technical queries. I think Arlo would benefit from having a couple of their engineers also monitoring this community or at least, engineers available for JamesC to talk too.

Then again maybe we're expecting too much from the community and such queries/issues should be lodged with technical support.

[LandJS]

yes JamesC has been on here as long as I can remember.  ShayneS is a moderator to but I see him more on the smart thread than the Ultra.  They do what they can but often can only send you to support. Support is the true weakness I believe.  I've gotten support people that pretty much read from a script,  had a couple that didn't even know some basic functions, gotten the e-mail of the conversation and wondered what the hell are you talking about, nothing like the conversation at all.    The length of time it takes to fix an issue is far more than any top line product should be.  This forum has been helpful but like most followers trying to help others.  There used to be many more on here but has dwindled to mostly new buyers which is often the case with forums.  I have found that if I finally make it to actually talking to someone higher up the chain the results have been pretty good.  Only JamesC or ShayneS could tell you who they have access to, I know with one issue they weren't on the same page as support but they both do what they can and I can't remember her name but there was a gal helping to.  When these were Netgear they sold and supported the product, now the business seems to be more about the cloud service and so features for those choosing not to use a subscription aren't exactly flying off the shelf.  No pics For the SD files, no ability to delete from the app, limited to the phone app only and only the primary subscriber, can only view at home unless you set up VPN or port fwd.  Some are happy with that and some feel why buy them and not buy the elite subscription.

[trinatron]

@LandJS 

Thanks for the reply. The information you provided is insightful but worrying. Going forward, I really hope Arlo don't just rely on good hardware but also offer a great service. 

[michaelkenward]

---

@trinatron wrote:

---

But why should local live view need to even talk to the cloud or depend on uplink speed. It's connecting directly to the smarthub via WiFi? 

---

Something tells me that it is a lot more complicated than that.

[StephenB]

To be clear - I agree with the main point of all this - the lag is simply too long.  It's ok for video streaming applications,  but not for anything interactive.  Talking with someone at the camera location requires a much shorter lag (a couple of hundred milliseconds is generally ok, but not seconds).  Certainly it can be lower than it is today - Zoom manages it after all.

Where I disagree is the speculation on the cause of the lag.  While there are certainly many things I don't know about the internals of the arlo system, I have been building and analyzing performance of internet conferencing systems for about 25 years.

---

@trinatron wrote:

---

---

I'm not sure which chips have been incorporated into the camera, but even with a dedicated VPU, it still won't have anywhere near the processing power of a modern PC.

---

Yes, but if you think it through it can't be the compression that's causing the lag. 

First of all, one way or another the compression has to keep up with real-time.  Livestreaming can continue for 20-30 minutes. There's a limit to how much buffering there can be.  

And if it were the compression itself, the frames waiting to be compressed have to be buffered in the camera memory - in uncompressed format.  Each 4K frame is 3840*2160.  It's YUV 4:2:0 format, so U and V are stored as 1920x1080.  Total memory to store each uncompressed frame is therefore 12,441,600 bytes.  The framerate is a bit variable, but is about 15 fps.  You're seeing 10 seconds of lag, so that's ~150 frames of buffering.  That requires about 2 GB of RAM.  No way the camera has that much video buffering.

---

@trinatron wrote:

---

---

But why should local live view need to even talk to the cloud or depend on uplink speed. It's connecting directly to the smarthub via WiFi? 

---

FWIW, I don't recall you saying if the 10 second lag was when your phone was connected to your local WiFi or if it was when your phone was using your mobile data.

As @michaelkenward says, it's more complicated than that.  The camera connects to the smarthub via the closed wifi, but then the smarthub has to reach your phone - which sometimes will require talking to the cloud (depending on where the phone is, and how it is connected).  It won't always use your internet uplink of course, but in many cases it will.  

One thing we don't know is the transport used to transport the live stream.  But I am thinking it is likely TCP - that would explain the large variations we are seeing in lag. 

TCP backs off its rate very aggressively whenever a packet is lost, and then slowly increases the rate again.  Since the data rate is fairly constant,  this can result if a fair amount of buffering whenever the data transport gets behind (because the packets are queued up for transmission somewhere in the system).  Smooth playback makes it hard to drain that buffering once it occurs (the only way to do it is to play the video and audio faster than real-time - either speeding up the video, or dropping frames on playback).

Note this buffering is of the compressed bitstream.  So at the full 3 megabit speed, 10 seconds of lag means there is 3-4 megabytes of buffering on the path (much less than 2 GB for the uncompressed frames).   And looking at the files I download the camera often is sending a lower bitrate than 3 megabits. 

This network buffering would be distributed across the camera, the smarthub, and your wifi router, and the phone if the connection is purely local.  Add in the cloud, and all the devices on the path from router -> cloud -> phone when you are away from home (or using your cellular data connection instead of your home wifi).

FWIW, interactive conferencing (video or voice) uses UDP, instead of TCP.  Network latency is much less, but packets are lost in transmission.  This is overcome by a combination of packet retransmission and a technique called "forward erasure correction".  The video stream can also be structured to allow some packets to be dropped without video corruption (though there's no evidence that Arlo uses that technique in the files I've looked at). 

UDP doesn't have built-in rate control, so packet loss doesn't cause a sharp drop in the transmission rate.  That greatly reduces the buffering in the network infrastructure - though you will end up with some packet buffering in the sender (camera) and the receiver (phone).

[LandJS]

trinatron they will have to if they plan their service as the primary business.  They will be up against both Google and Microsoft whose equipment will quickly offer the same features going fwd if they choose to.  Even Netgear cut them loose for a reason and Netgear stock to a good jump when it took place.

[LandJS]

StephenB all the tech speak is informative and even interesting though my experience was computers and not the internet, however for users it's the performance experienced that matters most.  There is no doubt that over the past 15 months the lag has increased and has done so in stages with firmware updates.  This last camera update has caused the longest lag I have experienced, that would seemed to suggest it's not a hardware issue.  This is also the case with many computer updates.  It's irritating but it's life as capabilities are added.  My security cameras however are a different matter, reaction time is one of it's most important purposes and that seems to be going in the wrong direction.

[StephenB]

---

@LandJS wrote:

 however for users it's the performance experienced that matters most.  

---

I agree with that.  But when people offer possible causes that clearly aren't true, I think it's important to address that.  Lots of people search this forum, and incorrect info can lead to basic confusion on how the system works, as well as ineffective troubleshooting.

I'm not sure if the lag has increased over the months, but the lag has always been too long for two-way communication.  

[LandJS]

perhaps you question  whether or not lag has increased but I guarantee you mine has and I'm sure others on here can.  As for peoples opinions or personal advice on a forum it has for years been subjective and so is who is correct since most consider themselves to be the expert.  You say you have years in the tech business so this is something you should be well aware of unless you only spend most of your time on profession based forums which many can read but not interact.  You show knowledge and are always here to help but that doesn't always mean correct for any of us.

[trinatron]

@StephenB 

We're all (inlcuding you) just trying to help people understand why they maybe experiencing unexpected results or offer possible solutions to the problems they maybe experiencing. With these types of technical issues, no one actually knows the cause except for the Arlo engineers themselves. You're also just speculating on buffering being the cause, so don't pretend like that is defintely the answer. I never said compression was definitely the problem I just said it could be the cause. Unless Arlo themselves respond, we're never going to know.

From my experience these are the only facts

• Local 4K Live View has the worst video/audio delay 5+ seconds
• Local 1080P Live View also has video/audio delay ~3 sec
• There is minimal video/audio delay using the Apple HomeKit instead
• Apple HomeKit only streams 1080P with my Ultra, not 4K
• The Arlo app uses an encrypted RTSP stream which is TCP. Most of the traditional security camera systems do this.
• Apple HomeKit uses HSL which is also TCP.

So the biggest delay is caused by steaming 4K.

---

@StephenB wrote:

 I have been building and analyzing performance of internet conferencing systems for about 25 years.

---

So what, that makes you 100% correct. FWIW I've been working as a network engineer for the past 15 years. I mainly work with Cisco gear and I know a lot about WiFi, comms and protocols etc. but I'll also be the first to say I definitely won't know everything about network engineering and I'll never pretend to.

---

@StephenB wrote:

As @michaelkenward says, it's more complicated than that. 

Actually it isn't if it's true Local 4K Live View. Camera ->ArloWiFi -> Smarthub ->Cable-> Router ->HomeWiFi-> Phone. I'm unable to monitor Arlo's WiFi but my home WiFi signal is strong with no interference/issues. Again having no issues using Apple HomeKit proves that there's no problems with the Home WiFi or Arlo's WiFi. The problem maybe just with the Arlo App and whatever else it's trying to do (e.g. talking to the Arlo servers) prior to launching and possibly during live view. 

"TCP backs off its rate very aggressively whenever a packet is lost, and then slowly increases the rate again"

You're referring to TCP Windowing and again I'm not seeing any comms issues when using the Apple HomeKit.  The only difference being a different app and not 4K.

 "forward erasure correction".  

You mean FEC (forward error correction)

[StephenB]

---

@trinatron wrote:

You mean FEC (forward error correction)

---

No I don't.  The terms are similar (and the underlying codes are often the same).  

Forward Error Correction will correct bit errors on the link.  Forward Erasure Correction will recover packets that are totally lost in transmission, but doesn't correct packets that have bit errors in them.  https://en.wikipedia.org/wiki/Erasure_code

Generally at layers 3 and 4 the assumption is that the link layers won't deliver errored packets, so erasure correction is what is used.  For example, simple parity can correct erasure or detect a bit error - but it can't correct a bit error unless you know which packet has the error.

Often people (and RFCs) will often use the wrong term.  It doesn't help that the acronym is the same.

---

@trinatron wrote:

I never said compression was definitely the problem I just said it could be the cause.

but I'll also be the first to say I definitely won't know everything about network engineering and I'll never pretend to.

---

I don't claim to know everything, and I didn't say with certainty that it was directly linked to TCP.  I just said that the variation in delay was consistent with the use of TCP (particularly if there was loss or congestion on the network path). It certainly does have to be buffering, as there is no other way to create this kind of delay in a real time system (other than sending the packets to the moon and back). 

FWIW, it sure sounded to me like you were saying that compression was the problem, and you hung on to that when I pointed out that it couldn't be.

But let's move on.

---

@trinatron wrote:

• Local 1080P Live View also has video/audio delay ~3 sec
• There is minimal video/audio delay using the Apple HomeKit instead

You're referring to TCP Windowing and again I'm not seeing any comms issues when using the Apple HomeKit.  The only difference being a different app and not 4K.

---

4K brings a higher data rate along with it of course.

I've seen the HomeKit comment from other posters, but it's not something I use myself.  It might be an important clue. How minimal is the delay you are seeing with HomeKit?  Not perceptable?

If the HomeKit delay is significantly less than 3 seconds, then the 3 second lag could be the arlo app playout buffer.  Though HomeKit would also use a different API in the base - so it could be an issue with the base station firmware.  I can't think of any way to tell if it's the sender or receiver.

But it wouldn't be anything in the cameras, since the camera->base link is identical for both paths.  And with local livestreaming, it's not in the Arlo Cloud either.

[trinatron]

---

@StephenB wrote:

I've seen the HomeKit comment from other posters, but it's not something I use myself.  It might be an important clue. How minimal is the delay you are seeing with HomeKit?  Not perceptable?

In my experience, less than 1 second. It feels like around the 600 - 700ms but I'll measure as soon as I get the chance. You can hold a 2-way conversation without any issues.

---

[paulseig]

I'm a New user within the last month, and was looking for some tips here on cutting lag time..  I have an Ultra system with 2 cameras, both running on AC power, and signal is good.. Hub is about 30 ft from front cam, and 10 ft from backyard cam..  The video is Excellent.. 4K and 1080.. but I also think the audio is poor..  I agree with prev post, trinatron ?  lag is longer on 4K, and a little better on 1080 ( about 3 secs.), I also get an echo on audio ?  My hub and Cameras did firmware updates as soon as I powered them on...  Someone said the audio used to be much better before firmware updates. Was it ALOT better ?      I hope they can improve the audio and Lag...  

[Ac34]

I just hooked mine up to the HomeKit and viewed my camera . It seems like the lag is just as someone said second or less.

[Ac34]

Just hooked my camera up to the HomeKit. New to HomeKit though. Any tips or advice with the cameras and HomeKit. Like settings or things you have done to improve the experience?

[LandJS]

I'm using a Samsung so homekit isn't something I'm familiar with but two things point to it not being hardware related.  First, that lag did not exist 16 months ago and has increased with more app updates with the latest being the worst lag ever.  Second would be that it doesn't seem to exist with homekit.  If it's not there with homekit, it's not hardware and Arlo app writers need to talk with Apple.