Arlo
Support

WB6NGC · ‎2019-11-07

Hub VMB3010r2 firmware 1.12.2.5_240_59657e5

Camera H7 firmware 1.3.319

This issue happened some while back and caused many complaints. It was fixed by arlo, apparently.

Now it's done it again. No alerts nor video saved from motion.

The motion test works. Manual recording is saved to library and viewable.

The hub and cameras have been power cycled and re-synced several times.

I've updated my case (Arlo case #41238594) several times with no further response.

Not a warm fuzzy feeling here.

Any other hints?

AudiTuner · ‎2019-11-08

Hopefully this works until Arlo fixes the issue:

I assume you're using Asuswrt-Merlin firmware? If so, in your ASUS router, go to LAN -> DNSFilter and configure it as shown. This should allow your ASUS router to still use DoT while allowing your Arlo base station to bypass the global DoT settings. In my case, I am using Cloudflare servers for my Arlo base station.

View solution in original post

WB6NGC · ‎2019-11-08

I have found out how to make it work from my end. I have DNS over TLS active on my router for several months and up until recently the arlo cam worked fine. Now I have to turn that feature off to get notifications and library video. I don't get it. It uses the same two DNS servers, just differs by whether the connection to them is encrypted or not. (1.1.1.1 and 9.9.9.9) So while I can have working cameras, it seems now it's at the expense of dns security. Go figure.
I would really like to have DNS over TLS (DoT) active. Any hints?

StephenB · ‎2019-11-08

@WB6NGC wrote:

I have found out how to make it work from my end. I have DNS over TLS active on my router for several months and up until recently the arlo cam worked fine. Now I have to turn that feature off to get notifications and library video. I don't get it. It uses the same two DNS servers, just differs by whether the connection to them is encrypted or not. (1.1.1.1 and 9.9.9.9)

While Cloudflare has unfiltered DNS, Quad9 does not - they filter out DNS names that they think are malicious. So maybe try re-enabling DNS over TLS, and removing Quad9 from your DNS as a test - then reboot the router and the base and see if anything changes.

If that works, then perhaps escalate with Quad9.

WB6NGC · ‎2019-11-08

Thanks for the suggestion. I tried as suggested but unfortunately it still only works for me if DoT is off.

Maybe if I swing a dead chicken overhead...😁

StephenB · ‎2019-11-08

@WB6NGC wrote:

Thanks for the suggestion. I tried as suggested but unfortunately it still only works for me if DoT is off.

Did you try it the other way around (using only Quad9)?

The dead chicken might help too, but only if you swing it counterclockwise. 🤔

WB6NGC · ‎2019-11-08

Yup. Several variations of providers in a list the option offers. Only turning off DoT does the trick so far. Bummer.

I have not seen any other site I visit exhibit any similar odd behavior. I wonder if it's some configuration setting in the arlo

servers but that's above my pay grade.

AudiTuner · ‎2019-11-08

Hopefully this works until Arlo fixes the issue:

I assume you're using Asuswrt-Merlin firmware? If so, in your ASUS router, go to LAN -> DNSFilter and configure it as shown. This should allow your ASUS router to still use DoT while allowing your Arlo base station to bypass the global DoT settings. In my case, I am using Cloudflare servers for my Arlo base station.

WB6NGC · ‎2019-11-08

I presume there was supposed be an image. It didn't make it.

However, I took your suggestion and enabled DNSfilter global router

then added my arlo as a client below with quad9 filter mode and

behold! it works! Thanks. So far no other odd behavior.

jsbeddow · ‎2019-11-08

Just to confirm, I can also add another up vote to this method: The real question here is why the Arlo firmware programmers decided to step in and interfere with DNS resolution that has not been an issue in the past. It is as if they seem to keep finding new ways to cripple a working system...anyway, here is the fix.

If using the enhanced Merlin firmware for your Asus router, and using secure DNS, aka DoT (DNS over TLS)

Step 1: On the LAN, DHCP Tab, Manually Assigned IP around the DHCP list, add the Arlo base station(s) as DHCP static lease devices (this ties the IP address and MAC address together each time DHCP renewal takes place).

Step 2: On the LAN, DNSFilter Tab, Set DNS based filtering to "On", Global Filter mode to "Router", add a Custom (user-defined) DNS1 of (your choice of DNS providers here: I found that Google DNS 8.8.8.8 worked well, Cloudflare DNS 1.1.1.1 also worked for me but seemed to slightly delay the notification time, YMMV). Next, in the Client List box below, add your Arlo base station (it should show up in the drop down list of available MAC addresses/device host names), set it's mode to Custom 1, Add button, then Apply button.

Step 3: Reboot or Power cycle your Arlo base station and test if working now. It should if all steps were followed. 😊

Cheers, I hope this has helped some other users searching for a solution: one that won't be found by spending an entire day of web chat with Arlo support (which is what I essentially did yesterday - what a waste of time that turned out to be). It eventually culminated in them agreeing to do an RMA on my base station, which I almost agreed to: luckily the forum user AudiTuner pointed me in the right direction today. Thanks.

StephenB · ‎2019-11-09

@AudiTuner wrote:

I assume you're using Asuswrt-Merlin firmware? If so, in your ASUS router, go to LAN -> DNSFilter and configure it as shown. This should allow your ASUS router to still use DoT while allowing your Arlo base station to bypass the global DoT settings. In my case, I am using Cloudflare servers for my Arlo base station.

Thx for posting the solution. It looks like your settings disable DNS filtering for the base, which suggests the the issue might not be in the Arlo firmware (rather that the domain name is being filtered for some reason).

In any event, your screenshot is visible now.

ChenXia · ‎2019-11-09

I don't run DoT, but rather a local DoH setup - and I can confirm having the same issues with very slow motion detection.

However, switching Arlo base away from the DoH setup and letting it use for example Cloudflare (DNS over 53) the normal way immediately solves the lag in motion detection and detection is as snappy as it always have been.

When it is using the DoH set up I see the base sending multiple DNS requests for the same hostname, this leads me to think that something in the Arlo base times out if it does not receive the DNS response fast enough (as sometimes both DoT and DoH takes a bit longer to initially establish the channels before query). Once queries are cached it reacts quicker. But since most of the Arlo servers points to AWS which have a 60 sec TTL, this behavior happens often due to the low TTL.

While I didn't react directly when this started to occur I do believe it began when DoH was enabled for me as well so it is something that Arlo reacts on. I cannot see any queries that are being filtered or blocked in the log.

For info, the way I let my Arlo base bypass my DoH setup is simply giving it its own set of DNS-servers in the DHCP configuration compared to other devices (if your router/firewall supports that sort of configuration).

Digging into things, I see that when I query some of the Arlo hostnames through DoH, if it's uncached I receive a DNS reply bigger in size than if it is cached by the server already. Perhaps these extra flags in the response the first time it's queried makes Arlo break, while response from the cached query it receives a "proper" DNS response and accepts it right off without having to re-query.

Example of the DoH and cached response, you ca see the difference in size at the last line:

; <<>> DiG 9.7.2-P2 <<>> vzweb31-prod.vz.netgear.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 2276
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;vzweb31-prod.vz.netgear.com. IN A

;; ANSWER SECTION:
vzweb31-prod.vz.netgear.com. 3600 IN A 52.50.174.128

;; Query time: 14 msec
;; SERVER: 172.17.43.10#53(172.17.43.10)
;; WHEN: Sat Nov 09 11:39:29 2019
;; MSG SIZE rcvd: 88

#########

; <<>> DiG 9.7.2-P2 <<>> vzweb31-prod.vz.netgear.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 43262
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 0

;; QUESTION SECTION:
;vzweb31-prod.vz.netgear.com. IN A

;; ANSWER SECTION:
vzweb31-prod.vz.netgear.com. 3590 IN A 52.50.174.128

;; Query time: 5 msec
;; SERVER: 172.17.43.10#53(172.17.43.10)
;; WHEN: Sat Nov 09 11:39:39 2019
;; MSG SIZE rcvd: 61

StephenB · ‎2019-11-09

@ChenXia wrote:

I don't run DoT, but rather a local DoH setup - and I can confirm having the same issues with very slow motion detection.

...

Thx for taking the time to post this information, it gives a clearer picture of what's going on.

As an aside, I am suprised that they are still using netgear DNS names.

gattaca · ‎2019-11-10

I am livid about this very issue. About a year ago in December Arlo's DevOpt team hozed up every camera in my setup with their "frequent updates". No motion, nadda nothing. To remedy this, I had to remove every camera (10) completely from my account and then readd them 1 by 1 to the base. That was up/down on a ladder about 25 times and at least 2+ hours of my time. -> https://community.arlo.com/t5/Arlo/Three-cameras-no-motion-detection/m-p/1658637#M61039

Now they have done it again. How could they not test and insure that a DoT setup is not going to be impacted by their code updates? I've been running DoT since January/February 2019 on my router and about 1 week ago, all my motions, notifications and vid library just stopped working. So I figure it's time to remove all the cameras again and start over. Then I find this thread saying that Cloudflare's DoT is now suddenly breaking my security cameras? I'm beyond livid. I'm 1/4" away from taking the Arlos down and eBaying the whole works b/c the Arlo developers are: a) clearly not testing their DevOpts software drops thoroughly enough before pushing it to the field and b) I'm not turing off DoT just to accommodate their lack of "getting it right the first time".

Fix your apps so they work with DoT/DoH and stop mucking with the DNS just because you want full control of the pipelines.

BWoodcock · ‎2019-11-11

That's not correct, actually. Quad9 has quite a few different combinations of features available. Each IP address has a different combination. If you want to test without malware blocking, just send your query to 9.9.9.10.

https://www.quad9.net/faq/#Is_there_a_service_that_Quad9_offers_that_does_not_have_the_blocklist_or_...

Also, there are quite a few different encryption methods available... DNS-over-TLS, DNScrypt, and DNS-over-HTTPs. Can't hurt to try others, and to test whether your TLS port is being blocked somewhere.

jsbeddow · ‎2019-11-11

Actually, the only user suggesting that DNS filtering (malware blocking) is the root cause is StephenB., I think the rest of us have accepted that Arlo is simply trying to control the entire path back to their servers. Those of us who were using Cloudflare (which does no content filtering) in combination with DoT had severe problems, essentially non-functional systems.

gattaca · ‎2019-11-11

^^^ Correct. I have been using Quad9's + DoT for many months with my 10 Arlo cameras via an ASUS/Merlin setup without a single issue. Then with this last mandatory Arlo update, all my external cameras just stopped recording anything or alerting to motion. Had it not been for the above posts, I would have never EVER suspected Arlo's new app/web infrastructure was suddenly not playing nice with Quad9 + DoT after all these months! For me, this is clearly not a Quad9 or Cloudflare DoT issue, it's an issue with Arlo's development teams DevOpting an app out the door and/or it's them trying to fully control the DNS lookups.

I have used both Quad9 and Cloudflare with DoT.

After spending at least 2-3 hours searching on this, I changed my ASUS router yesterday to "bypass" the DoT for the two base stations as outlined above and what'd you, alerting and recording is magically working again. I'm still using 9.9.9.9 and 1.1.1.1 as options, just no DoT but only with those two base station addresses.

THANK YOU GUYS so much for reporting and finding this. It's now Arlo's turn to make their stuff work properly with DoT - no matter the provider. I think we are all on the same page! THANKS!

StephenB · ‎2019-11-12

@jsbeddow wrote:
Actually, the only user suggesting that DNS filtering (malware blocking) is the root cause is StephenB.,

To be clear, I suggested that was a possible cause. The tests I suggested for that were run by the OP, and there was no change in the symptoms. So DNS filtering doesn't seem to be a factor.

A later post from @ChenXia suggested it might be linked to DNS caching (showing that with DoH running, the cached response wasn't the same as the uncached response). I haven't seen any followup to that idea.

AFAICT. the Arlo base has no direct knowledge of whether the router is using ordinary DNS, DoH or DoT to talk to it's DNS server. In all cases, the local network just sees ordinary DNS. The (undeniable) fact that it is affecting Arlo suggests that either that

the DNS server isn't always finding the requested hostname (depending on the transport used to reach it).
Arlo is sensitive to DNS latency
there is is something a bit different about the formatting of DNS messages coming from the Asuswrt-Merlin firmware (which appears to be used by some or all of the posters seeing this problem).

The first possibility was the DNS filtering theory, which I think we've ruled out. That does leave the other two.

@ChenXia's observation that the base is making a lot of DNS requests, combined with his discovery that going directly to CloudFlare's DNS solved his latency problem seems very relevant to me, and is making me wonder about latency. Arlo shouldn't have to ask for the same hostname in rapid succession, so it seems to me that they could easily prevent that.

Though the different size of the cached DNS message suggests that possibility (3) is potentially a factor too. Maybe looking at the message sizes when Arlo is set up to bypass DoH would rule that one out. Something that maybe would be worth the time to check.

@jsbeddow wrote:
I think the rest of us have accepted that Arlo is simply trying to control the entire path back to their servers.

I'm seeing that only in @gattaca 's post ( a reply to yours), and of course yours. Other posters seem to think it's just a bug in the Arlo base station that needs to be fixed.

FWIW, if I wanted to "control the entire path", I'd just have hard-wired my preferred DNS server IP address into the base station. Arlo clearly isn't doing that, they are using whatever DNS server they are given from DHCP. Plus, I don't understand how failing with DoH or DoT would give them any more "control". So that doesn't seem like a good explanation to me. It seems more likely that it's just a bug.

gattaca · ‎2019-11-12

This is all my laymen's understanding.

There is added latency to the DoT/DoH transmission streams over non-encrypted versions.
The ASUS routers + Merlin FW are "early adopters" of making DoT/DoH easier options to setup and enforce for everything behind the router. In fact, unless you setup the bypass like outlined above, all DNS requests are forced to use the DoT/DoH path.
There is added rejections in some of the tooling we run but the major change which started this working for me was the bypassing my DoT setup JUST for the 2 base station IPs/MACs. Since this has been reported several times we could consider it leading root-cause. Now is that b/c Arlos or sensitive to latency or don't like the DNS + DoT/DoH streams or both or something else?
Some of the Merlin tooling was was done to stop/thwart misbehaving applications which appear to have been hardcoded in their defaults use their OWN DNS servers vs. what the router says use. For instance, certain TVs, Microsoft OSes, Google, and other IoT-like devices want to collect your DNS data a/o claim to provide better services. Someone could watch the DNS stream to see if that's what Arlo is up to. IDK how or have the wireshark tooling or TCPDUMP skills to figure it out. The ASUS + Merlin router has the ability to force all DNS to use what we say, not what they want.
Whether that DNS flow is over DoT/DoH is a 2nd stage of the setup. The above workaround settings bypass that 2nd stage b/c I'm still using 9.9.9.9 and 1.1.1.1 as the bypass targets, just not with DoT.

I think we are making progress. I just don't know the root cause. If it is indeed Arlo hardcoding their own DNS, then very, very bad. If it latency related due to DoT/DoH, then that can be adjusted in the Base FW etc..

Interestingly, I'm reported that ONLY my BASE units were impacted by this... and NOT my standalone Q cams inside. They are happily using DoT and never exhibited this behavior. Different cameras, different FW... THANKS!!!

StephenB · ‎2019-11-13

@gattaca wrote:
If it is indeed Arlo hardcoding their own DNS, then very, very bad.

I don't see any privacy leakage in this instance (since the only DNS lookups the base should be doing are to connect to the Arlo Cloud servers and NTP). But I agree it would be a very bad design choice.

While it would ensure that the base could always resolve the cloud server hostnames, it would also assume that DNS traffic to their server was allowed through the network firewall. That's problematic for some networks, and an unnecessary limitation. Plus changing the DNS server IP would require pushing new firmware to all the bases in the field first, which is an operational nightmare.

In any event, I think @ChenXia's analysis shows clearly that they aren't doing it. If they were, he wouldn't see any cached DNS messages going to the base, and he'd have seen the DNS server IP address they were using in his network trace. You could also use WireShark or a similar tool to confirm this on your own network.

@gattaca wrote:

In fact, unless you setup the bypass like outlined above, all DNS requests are forced to use the DoT/DoH path.

That's certainly true if the device is using the DNS proxy built into the Asus. Hopefully it's not true if the device has a manually configured DNS server. Technically it's possible - Asus could simply block DNS requests outbound (other than the ones the router is proxying). But there are situations where people have good reasons to use a specific DNS server for a specific device.

So IMO that would also be a very bad design choice (in this case for Asus).

It is easy to test - just manually configure a PC to use 8.8.8.8 (google DNS), and see if name resolution still works.

jsbeddow · ‎2019-11-13

Frankly, I have moved on to more important topics, as the workaround functions for my needs. However, I am still furious that Arlo wasted at least five+ hours of my time last Thursday on three useless chat sessions: their resolution was an offer to exchange my base station (which I now know would not have resolved the problem) - of course, it would have been at my expense if I wanted advance shipping of the replacement unit. I just hope that this thread has made enough "noise" to have been noticed by the firmware programmers, and a fix is planned for the next release. We won't really know though (short of testing it ourselves), as the release notes for every firmware almost always say "Bug fixes", which I find incredibly lazy on their part.

Larzalere · ‎2019-11-22

Hi All -

Read through this discussion and although I have 20 years of using computers, the mentions of DoH and Cloudflare and so forth are striking fear into my heart. I recently had a suspected package stolen from my porch and went to check the Arlo library - no recordings, of any kind, on any of my cameras (I have 6). Now it sounds like Arlo support won't be able to help, but I'm not sure I'm capable of figuring out what you all have been discussing. Any tips for me to just get things working? I'm running a Mac with an Arris Surfboard SB6141 and Airport, 6 pro or pro 2 cameras and a pro base station. Firewall is off. Thanks for any and all guidance.

StephenB · ‎2019-11-23

@Larzalere wrote:
Read through this discussion and although I have 20 years of using computers, the mentions of DoH and Cloudflare and so forth are striking fear into my heart.

We did get into some fairly new/emerging stuff there, and it's almost certainly not related to your problem. You'd have had to play around with advanced router settings (that most routers don't even have at this point) for DoH/DoT to be a factor.

@Larzalere wrote:

I recently had a suspected package stolen from my porch and went to check the Arlo library - no recordings, of any kind, on any of my cameras (I have 6). Now it sounds like Arlo support won't be able to help, but I'm not sure I'm capable of figuring out what you all have been discussing. Any tips for me to just get things working? I'm running a Mac with an Arris Surfboard SB6141 and Airport, 6 pro or pro 2 cameras and a pro base station. Firewall is off. Thanks for any and all guidance.

Let's start at the beginning.

When you connect to https://my.arlo.com in your browser, are you seeing that the cameras and base station are online?
If they are on-line, can you make live stream any of the cameras?
Are you using Safari for the browser? Or are you using something else (for instance Chrome)?

gattaca · ‎2019-11-23

If you are not using DNS over TLS, then this thread likely will not solve your issues. What I can say is that the last time I had this sort of thing happen - all cams stop reporting, the Arlo DevOps team had delivered yet another major FW/App/Website update. It knocked my cams offline for a week+. The only way I was able to get them back was to delete every camera from the base units and then re-add each camera back to the base unit. After that, I had to setup the other cam parms such as recording sensitivity, quality, or motion triggers. Once that was done, the cameras behaved normally.

I had tried everything prior to that such as upgrading the firmware, rebooting the base stations. So like last time, I suspected that the Arlo forced updated had done me in again. It was not until I ran across this thread that I discovered the link to my DoT setup. Now, I believe the same thing happened. Arlo's team updated their firmware, apps, and web site and clearly did not test for DoT setups because this who setup has been using DoT for 11+ months without a hitch and then suddenly around the same time as their forced update, my cams go offline. - Sorry, NOT a coincidence.

Peace.

Arlo Support

Arlo App for Support

Arlo
Support

Arlo App
for Support