A couple of weeks ago, at my $dayjob, we implemented a recursive resolver with RPZ in an enterprise network.
After a few days, the customer got back to us with an issue – the DNS resolution of the domain esic.in failed with an NXDOMAIN response. After a cursory look at the problem, it became evident that esic.in resolved correctly but www.esic.in did not.
The customer also reported that if they switched the resolver to 8.8.8.8, the DNS resolution of www.esic.in was without any problems, and the website was accessible in the network.
So, what is causing the DNS issue with www.esic.in with the on-prem resolver?
Let’s find out. To start with the basics, here are the authoritative name servers of the domain esic.in,
$ whois esic.in | grep "Name Server:"
Name Server: ns-1089.awsdns-08.org
Name Server: ns-52.awsdns-06.com
Name Server: ns-1978.awsdns-55.co.uk
Name Server: ns-882.awsdns-46.net
If we traverse the DNS delegation from the root to esic.in, we get valuable insights,
. 518400 IN NS k.root-servers.net.
. 518400 IN NS l.root-servers.net.
. 518400 IN NS d.root-servers.net.
. 518400 IN NS e.root-servers.net.
. 518400 IN NS j.root-servers.net.
. 518400 IN NS b.root-servers.net.
. 518400 IN NS g.root-servers.net.
. 518400 IN NS a.root-servers.net.
. 518400 IN NS h.root-servers.net.
. 518400 IN NS m.root-servers.net.
. 518400 IN NS i.root-servers.net.
. 518400 IN NS c.root-servers.net.
. 518400 IN NS f.root-servers.net.
in. 172800 IN NS ns1.registry.in.
in. 172800 IN NS ns2.registry.in.
in. 172800 IN NS ns3.registry.in.
in. 172800 IN NS ns4.registry.in.
in. 172800 IN NS ns5.registry.in.
in. 172800 IN NS ns6.registry.in.
esic.in. 3600 IN NS ns-882.awsdns-46.net.
esic.in. 3600 IN NS ns-1978.awsdns-55.co.uk.
esic.in. 3600 IN NS ns-52.awsdns-06.com.
esic.in. 3600 IN NS ns-1089.awsdns-08.org.
esic.in. 300 IN A 115.113.201.36
esic.in. 300 IN A 218.248.15.136
esic.in. 172800 IN NS ns-1089.awsdns-08.org.
esic.in. 172800 IN NS ns-1978.awsdns-55.co.uk.
esic.in. 172800 IN NS ns-52.awsdns-06.com.
esic.in. 172800 IN NS ns-882.awsdns-46.net.
And, here is the delegation trace from the root to www.esic.in,
. 518400 IN NS a.root-servers.net.
. 518400 IN NS e.root-servers.net.
. 518400 IN NS c.root-servers.net.
. 518400 IN NS b.root-servers.net.
. 518400 IN NS m.root-servers.net.
. 518400 IN NS l.root-servers.net.
. 518400 IN NS h.root-servers.net.
. 518400 IN NS j.root-servers.net.
. 518400 IN NS d.root-servers.net.
. 518400 IN NS g.root-servers.net.
. 518400 IN NS i.root-servers.net.
. 518400 IN NS k.root-servers.net.
. 518400 IN NS f.root-servers.net.
in. 172800 IN NS ns1.registry.in.
in. 172800 IN NS ns4.registry.in.
in. 172800 IN NS ns5.registry.in.
in. 172800 IN NS ns6.registry.in.
in. 172800 IN NS ns3.registry.in.
in. 172800 IN NS ns2.registry.in.
esic.in. 3600 IN NS ns-882.awsdns-46.net.
esic.in. 3600 IN NS ns-1089.awsdns-08.org.
esic.in. 3600 IN NS ns-1978.awsdns-55.co.uk.
esic.in. 3600 IN NS ns-52.awsdns-06.com.
www.esic.in. 3600 IN NS lbr1.esic.in.
www.esic.in. 3600 IN NS lbr2.esic.in.
www.esic.in. 0 IN A 218.248.15.136
If you compare the two outputs and look closely, the authoritative nameservers have delegated www.esic.in to the name servers lbr1.esic.in and lbr2.esic.in
And at the time of the issue, the nameservers lbr1.esic.in and lbr2.esic.in did not respond to Do53(UDP) resulting in an NXDOMAIN!
DNSViz also reported the non-responsive nameservers as well as OpenDNS cachecheck,
At the time of writing this blog post, the name servers lbr1.esic.in. and lbr2.esic.in. were responding and www.esic.in was resolving correctly. But for more than 24+ hours, they were unresponsive resulting in some random people on the Internet in India being unable to access the website.
tl;dr If Jio VoWiFi isn’t working for you, set a different DNS resolver on the phone. While I am a big proponent of running your own resolver in the network, ( runyourownresolver.in ) , you could test by using open resolvers. The issue doesn’t seem to be impacting everyone and only a subset of users.
To begin with, there are multiple things broken in the authoritative name servers ns1.vowifi.jio.com. and ns2.vowifi.jio.com. of vowifi.jio.com which I’ll cover a bit later.
I came across reports ( See here & here ) of Jio VoWiFi not working for many and while the reports were sketchy, I decided to test this myself.
Below is a snippet from a log file of a dns query to vowifi.jio.com from my phone(192.168.1.137) to a recursive resolver(Unbound) which I run in my network,
May 28 15:54:35 root unbound: [1300:0] info: 192.168.1.137 vowifi.jio.com. A IN
Ideally, the domain is standardised & is made up of Mobile Network Code(MNC) and Mobile Country Code(MCC). For example – in the case of Airtel VoWiFi, the domain name that I see hitting my Unbound resolver is epdg.epc.mnc045.mcc404.pub.3gppnetwork.org. where MNC – 045 and MCC – 404 which signifies Airtel – Karnataka region.
However, oddly enough, Reliance Jio seems to be using vowifi.jio.com. Having said that, the standardised domain name works as well. For example – epdg.epc.mnc861.mcc405.pub.3gppnetwork.org. resolves to 49.44.59.36 and 49.44.59.38
Below is the dns resolution entire delegation chain. From my home network, I can see that the vowifi.jio.com resolves to 49.44.59.38 and 49.44.59.36
. 518400 IN NS a.root-servers.net.
. 518400 IN NS b.root-servers.net.
. 518400 IN NS c.root-servers.net.
. 518400 IN NS d.root-servers.net.
. 518400 IN NS e.root-servers.net.
. 518400 IN NS f.root-servers.net.
. 518400 IN NS g.root-servers.net.
. 518400 IN NS h.root-servers.net.
. 518400 IN NS i.root-servers.net.
. 518400 IN NS j.root-servers.net.
. 518400 IN NS k.root-servers.net.
. 518400 IN NS l.root-servers.net.
. 518400 IN NS m.root-servers.net.
com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS l.gtld-servers.net.
com. 172800 IN NS c.gtld-servers.net.
com. 172800 IN NS h.gtld-servers.net.
com. 172800 IN NS e.gtld-servers.net.
com. 172800 IN NS d.gtld-servers.net.
com. 172800 IN NS i.gtld-servers.net.
com. 172800 IN NS f.gtld-servers.net.
com. 172800 IN NS m.gtld-servers.net.
com. 172800 IN NS j.gtld-servers.net.
com. 172800 IN NS g.gtld-servers.net.
com. 172800 IN NS k.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.
jio.com. 172800 IN NS ns1.jio.com.
jio.com. 172800 IN NS ns2.jio.com.
jio.com. 172800 IN NS ns3.jio.com.
jio.com. 172800 IN NS ns4.jio.com.
vowifi.jio.com. 3600 IN NS ns1.vowifi.jio.com.
vowifi.jio.com. 3600 IN NS ns2.vowifi.jio.com.
vowifi.jio.com. 5 IN A 49.44.59.38
vowifi.jio.com. 5 IN A 49.44.59.36
At this point, I confirmed that VoWiFi on Jio works by putting the phone on Airplane mode while remain connected to WiFi. A ~22 minute call worked flawlessly.
To confirm that vowifi.jio.com was indeed the domain name that needs to resolve for VoWiFi to work on Jio, I configured an entry for vowifi.jio.com to return a NXDOMAIN answer in my DNS RPZ aka DNS Firewall in Unbound.
With that configured, any DNS query for vowifi.jio.com from any device in the network will be meted out with a NXDOMAIN answer. Below is a snippet from the Unbound log confirming the RPZ rule applied.
May 28 17:31:50 root unbound: [1191:0] info: 192.168.0.137 vowifi.jio.com. A IN
May 28 17:31:50 root unbound: [1191:0] info: RPZ applied [custom block to test vowifi] vowifi.jio.com. nxdomain 192.168.0.137@64521 vowifi.jio.com. A IN
;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 14747
;; flags: qr aa rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; vowifi.jio.com. IN A
;; ANSWER SECTION:
;; AUTHORITY SECTION:
;; ADDITIONAL SECTION:
;; Query time: 136 msec
;; SERVER: 192.168.0.250
;; WHEN: Thu May 28 18:03:42 2020
;; MSG SIZE rcvd: 32
In the context of VoWiFi, the other noticeable problems with DNS infrastructure of Jio –
A/AAAA records for ns1.vowifi.jio.com, ns2.vowifi.jio.com are missing
ns1.vowifi.jio.com(49.44.59.6), ns2.vowifi.jio.com(49.44.59.7) don’t respond to queries over TCP
The other interesting thing that is worth observing is that when you try resolving vowifi.jio.com from outside India or use a DNS resolver which is perhaps not geographically located within India, the authoritative name servers ns1.vowifi.jio.com(49.44.59.6), ns2.vowifi.jio.com(49.44.59.7) give out a different set of IP addresses – 49.45.63.1, 49.45.63.2
; <<>> DiG 9.16.3 <<>> @127.0.0.1 vowifi.jio.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13728
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;vowifi.jio.com. IN A
;; ANSWER SECTION:
vowifi.jio.com. 4 IN A 49.45.63.1
vowifi.jio.com. 4 IN A 49.45.63.2
;; Query time: 352 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat May 30 06:23:23 IST 2020
;; MSG SIZE rcvd: 75
To confirm this hypothesis, I decided to utilise the RIPE Atlas probes to run a measurement. If you’re unaware of the RIPE Atlas project, check an earlier post – Host a RIPE Atlas software probe in your network.
And the results of the measurement are interesting. Out of the 75 probes which participated in the measurement, there were many probes which received the response 49.45.63.1 & 49.45.63.2 to the DNS query to vowifi.jio.com
ASN
AS Name
DNS Response 1
DNS Response 2
Resolver IP address
4758
NICNET-VSNL-BOARDER-AP National Informatics Centre, IN’
49.45.63.2
49.45.63.1
164.100.3.1
4758
NICNET-VSNL-BOARDER-AP National Informatics Centre, IN’
49.45.63.1
49.45.63.2
164.100.3.1
24186
RAILTEL-AS-IN RailTel Corporation of India Ltd., Internet Service Provider, New Delhi, IN’
TATACOMM-AS TATA Communications formerly VSNL is Leading ISP, IN’
49.45.63.1
49.45.63.2
8.8.8.8
16509
AMAZON-02, US’
49.44.59.38
49.44.59.36
::1
15169
GOOGLE, US’
49.45.63.1
49.45.63.2
::1
139331
DCORP-AS-AP DevelentCorp., IN’
49.44.59.36
49.44.59.38
::1
24309
CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN’
49.45.63.2
49.45.63.1
192.168.1.10
9498
BBIL-AP BHARTI Airtel Ltd., IN’
49.45.63.1
49.45.63.2
192.168.139.245
internationalvowifi.jio.com also seems to indicate VoWiFi International calling, which resolves to 49.44.59.36 and 49.44.59.38 from my vantage point. The same resolves to 49.45.63.1 and 49.45.63.2 from every location that I’ve managed to check from outside India.
Looking at the results, most likely the issue is with how ns1.vowifi.jio.com & ns2.vowifi.jio.com are responding to client subnet (EDNS0) in DNS queries.
RIPE Atlas is a global network of devices, called probes and anchors, that actively measure Internet connectivity. RIPE Atlas users can also perform customised measurements to gain valuable data about their own networks.
At the time of writing, there were 11,009 probes connected, the total number of probes connected is maybe higher than this as probes go offline due to Internet disconnections, power issues especially in under developed/developing countries.
All this while, the probes have been hardware devices.
That changed sometime in February 2020, the RIPE NCC released a software version of the RIPE Atlas probe. This is super useful (apart from the fact that the hardware probe costs money to manufacture and ship it and most importantly Indian customs 😢 ) as you can run the software probe on RaspberryPi along with many other supported operating systems(CentOS7,CentOS8, Debian 9, Debian 10 and Docker).
For more information about installing the software probe and registration, please click the following link
If anyone needs any help in installing/registering the probe, feel free to ping 🙂
