CERT-In : Sensor for MSME networks for logs

If you are an MSME and are looking at complying to the CERT-In directives on logs, then, a sensor we’ve built for generating and storing logs of the entire network, might just be what you are looking for.

What do the CERT-In directives on logs state

All service providers, intermediaries, data centres, body corporate and
Government organisations shall mandatorily enable logs of all their ICT
systems and maintain them securely for a rolling period of 180 days and
the same shall be maintained within the Indian jurisdiction. These should
be provided to CERT-In along with reporting of any incident or when
ordered / directed by CERT-In.

Challenges faced in incident response environments (MSME) with no logs

The idea of building a sensor stemmed from our experiences of incident response in environments with zero security posture.

CERT-In sensor MSME logs

The same sensor can capture network packets and generate logs per the CERT-In directives.

At the btNOG-9 Conference on the 14th October 2022, I’ll be presenting Incident Response on a shoestring budget

In the presentation, I’ll share the challenges we faced in incident response environments with zero security posture, i.e. lacking logs, etc. The presentation will then focus on the solution – a sensor we built using open-source software such as Suricata and Zeek, logging DNS queries etc.

By deploying a sensor in the network, MSMEs can comply with the CERT-In directives and also facilitate incident responders to investigate security incidents.

Incident responders can leverage the rich logs by intercepting and ingesting packets into tools such as Zeek. If you are new to Zeek, check the blog post, Packets don’t lie – Threat Hunting with Zeek and the APNIC Academy page where a recording of the webinar will be available soon.

For a broader deep dive into why Network Security Monitoring is important in the context of incident response, check my presentation on Packets don’t lie – Network Security Monitoring (NSM) for the masses

Aside from the folks at BtCIRT, I am hoping there would be a bunch of other folks from a security background interested in incident response.

Little Snitch – Capturing traffic of a specific process

While investigating a bit of oddity with the Skype app on Mac OS X, I wanted to capture all traffic from only the Skype processes.

But first, a little background on the issue. All DNS traffic from my systems is routed through a WireGuard tunnel. The peer endpoint at the other end runs a recursive resolver with DNS Response Policy Zones (DNS RPZ).

The issue is – that as soon as the WireGuard tunnel is disabled, Skype will try connecting to Google DNS(8.8.8.8) and www.bing.com. Perhaps Skype bypasses the local recursive resolver set on the system and sends the DNS queries for its hosts to Google DNS directly? I cannot ascertain that yet, and it warrants a thorough investigation.

I was alerted to this by Little Snitch after I refreshed the rules. A ritual that I seem to follow every few months.

Little Snitch Network Monitor contains a hidden gem. The ability to capture traffic of a specific process.

For example, by right-clicking on any process, you should see the option to Capture Traffic. That should open a terminal window prompting for the sudo password.

After capturing traffic and interrupting should result in a PCAP on the Desktop. Neat!

Little Snitch documentation about this feature.

Shodan geoping and geodns -Quickly check ping and DNS resolution across multiple locations

Measuring ping and DNS from different vantage points using RIPE Atlas has been something that I have been using for some time now.

A few weeks ago, I came across Shodan’s geoping and geodns API, which provides ping and DNS lookup from a few locations and other details such as RTT. This is great because you can quickly check ping and DNS resolution on systems where you only have curl running.

You always have the RIPE Atlas project for more detailed and sophisticated use-cases. To get started with the RIPE Atlas project, check the webinar I delivered some time ago for APNIC.

curl https://geonet.shodan.io/api/geoping/139.59.19.245 | jq .
[
  {
    "ip": "139.59.19.245",
    "is_alive": true,
    "min_rtt": 41.439,
    "avg_rtt": 41.539,
    "max_rtt": 41.689,
    "rtts": [
      41.68868064880371,
      41.4891242980957,
      41.43881797790527
    ],
    "packets_sent": 3,
    "packets_received": 3,
    "packet_loss": 0,
    "from_loc": {
      "city": "Singapore",
      "country": "SG",
      "latlon": "1.3215,103.6957"
    }
  },
  {
    "ip": "139.59.19.245",
    "is_alive": true,
    "min_rtt": 229.823,
    "avg_rtt": 230.04,
    "max_rtt": 230.268,
    "rtts": [
      230.2682399749756,
      229.82311248779297,
      230.0271987915039
    ],
    "packets_sent": 3,
    "packets_received": 3,
    "packet_loss": 0,
    "from_loc": {
      "city": "Santa Clara",
      "country": "US",
      "latlon": "37.3924,-121.9623"
    }
  },
  {
    "ip": "139.59.19.245",
    "is_alive": true,
    "min_rtt": 183.42,
    "avg_rtt": 183.567,
    "max_rtt": 183.683,
    "rtts": [
      183.68268013000488,
      183.41970443725586,
      183.59804153442383
    ],
    "packets_sent": 3,
    "packets_received": 3,
    "packet_loss": 0,
    "from_loc": {
      "city": "Frankfurt am Main",
      "country": "DE",
      "latlon": "50.1025,8.6299"
    }
  },
  {
    "ip": "139.59.19.245",
    "is_alive": true,
    "min_rtt": 185.742,
    "avg_rtt": 185.865,
    "max_rtt": 185.993,
    "rtts": [
      185.99295616149902,
      185.86158752441406,
      185.74166297912598
    ],
    "packets_sent": 3,
    "packets_received": 3,
    "packet_loss": 0,
    "from_loc": {
      "city": "Amsterdam",
      "country": "NL",
      "latlon": "52.3740,4.8897"
    }
  },
  {
    "ip": "139.59.19.245",
    "is_alive": true,
    "min_rtt": 267.025,
    "avg_rtt": 267.047,
    "max_rtt": 267.061,
    "rtts": [
      267.0609951019287,
      267.05384254455566,
      267.0247554779053
    ],
    "packets_sent": 3,
    "packets_received": 3,
    "packet_loss": 0,
    "from_loc": {
      "city": "Clifton",
      "country": "US",
      "latlon": "40.8344,-74.1377"
    }
  },
  {
    "ip": "139.59.19.245",
    "is_alive": true,
    "min_rtt": 261.196,
    "avg_rtt": 261.239,
    "max_rtt": 261.279,
    "rtts": [
      261.1956596374512,
      261.24072074890137,
      261.2793445587158
    ],
    "packets_sent": 3,
    "packets_received": 3,
    "packet_loss": 0,
    "from_loc": {
      "city": "London",
      "country": "GB",
      "latlon": "51.5085,-0.1257"
    }
  }
]

The geodns API enables looking up DNS across multiple locations.

curl https://geonet.shodan.io/api/geodns/brainattic.in  | jq .
[
  {
    "answers": [
      {
        "type": "A",
        "value": "139.59.19.245"
      }
    ],
    "from_loc": {
      "city": "Clifton",
      "country": "US",
      "latlon": "40.8344,-74.1377"
    }
  },
  {
    "answers": [
      {
        "type": "A",
        "value": "139.59.19.245"
      }
    ],
    "from_loc": {
      "city": "Frankfurt am Main",
      "country": "DE",
      "latlon": "50.1025,8.6299"
    }
  },
  {
    "answers": [
      {
        "type": "A",
        "value": "139.59.19.245"
      }
    ],
    "from_loc": {
      "city": "London",
      "country": "GB",
      "latlon": "51.5085,-0.1257"
    }
  },
  {
    "answers": [
      {
        "type": "A",
        "value": "139.59.19.245"
      }
    ],
    "from_loc": {
      "city": "Amsterdam",
      "country": "NL",
      "latlon": "52.3740,4.8897"
    }
  },
  {
    "answers": [
      {
        "type": "A",
        "value": "139.59.19.245"
      }
    ],
    "from_loc": {
      "city": "Singapore",
      "country": "SG",
      "latlon": "1.3215,103.6957"
    }
  },
  {
    "answers": [
      {
        "type": "A",
        "value": "139.59.19.245"
      }
    ],
    "from_loc": {
      "city": "Santa Clara",
      "country": "US",
      "latlon": "37.3924,-121.9623"
    }
  }
]

The geodns command provides the output in shell format,

# geodns google.com
142.250.178.14                 London
142.250.186.46                 Frankfurt am Main
142.250.80.46                  Clifton
142.251.36.46                  Amsterdam
74.125.68.100                  Singapore
74.125.68.101                  Singapore
74.125.68.102                  Singapore
74.125.68.113                  Singapore
74.125.68.138                  Singapore
74.125.68.139                  Singapore

Similarly, the geoping command,

# geoping 8.8.8.8
Amsterdam (NL)                 0.863 ms       (min: 0.509 ms, max: 1.414 ms)
Clifton (US)                   1.985 ms       (min: 1.729 ms, max: 2.443 ms)
Frankfurt am Main (DE)         1.167 ms       (min: 0.754 ms, max: 1.979 ms)
London (GB)                    0.769 ms       (min: 0.527 ms, max: 1.229 ms)
Santa Clara (US)               2.273 ms       (min: 1.638 ms, max: 3.151 ms)
Singapore (SG)                  1.53 ms       (min:  1.13 ms, max: 2.204 ms)

The details about the geoping and geodns commands are available here

The curious case of esic.in DNS

A couple of weeks ago, at my $dayjob, we implemented a recursive resolver with RPZ in an enterprise network.

After a few days, the customer got back to us with an issue – the DNS resolution of the domain esic.in failed with an NXDOMAIN response. After a cursory look at the problem, it became evident that esic.in resolved correctly but www.esic.in did not.

The customer also reported that if they switched the resolver to 8.8.8.8, the DNS resolution of www.esic.in was without any problems, and the website was accessible in the network.

So, what is causing the DNS issue with www.esic.in with the on-prem resolver?

Let’s find out. To start with the basics, here are the authoritative name servers of the domain esic.in,

$ whois esic.in | grep "Name Server:"
Name Server: ns-1089.awsdns-08.org
Name Server: ns-52.awsdns-06.com
Name Server: ns-1978.awsdns-55.co.uk
Name Server: ns-882.awsdns-46.net

If we traverse the DNS delegation from the root to esic.in, we get valuable insights,

.	518400	IN	NS	k.root-servers.net.
.	518400	IN	NS	l.root-servers.net.
.	518400	IN	NS	d.root-servers.net.
.	518400	IN	NS	e.root-servers.net.
.	518400	IN	NS	j.root-servers.net.
.	518400	IN	NS	b.root-servers.net.
.	518400	IN	NS	g.root-servers.net.
.	518400	IN	NS	a.root-servers.net.
.	518400	IN	NS	h.root-servers.net.
.	518400	IN	NS	m.root-servers.net.
.	518400	IN	NS	i.root-servers.net.
.	518400	IN	NS	c.root-servers.net.
.	518400	IN	NS	f.root-servers.net.
in.	172800	IN	NS	ns1.registry.in.
in.	172800	IN	NS	ns2.registry.in.
in.	172800	IN	NS	ns3.registry.in.
in.	172800	IN	NS	ns4.registry.in.
in.	172800	IN	NS	ns5.registry.in.
in.	172800	IN	NS	ns6.registry.in.
esic.in.	3600	IN	NS	ns-882.awsdns-46.net.
esic.in.	3600	IN	NS	ns-1978.awsdns-55.co.uk.
esic.in.	3600	IN	NS	ns-52.awsdns-06.com.
esic.in.	3600	IN	NS	ns-1089.awsdns-08.org.
esic.in.	300	IN	A	115.113.201.36
esic.in.	300	IN	A	218.248.15.136
esic.in.	172800	IN	NS	ns-1089.awsdns-08.org.
esic.in.	172800	IN	NS	ns-1978.awsdns-55.co.uk.
esic.in.	172800	IN	NS	ns-52.awsdns-06.com.
esic.in.	172800	IN	NS	ns-882.awsdns-46.net.

And, here is the delegation trace from the root to www.esic.in,

.	518400	IN	NS	a.root-servers.net.
.	518400	IN	NS	e.root-servers.net.
.	518400	IN	NS	c.root-servers.net.
.	518400	IN	NS	b.root-servers.net.
.	518400	IN	NS	m.root-servers.net.
.	518400	IN	NS	l.root-servers.net.
.	518400	IN	NS	h.root-servers.net.
.	518400	IN	NS	j.root-servers.net.
.	518400	IN	NS	d.root-servers.net.
.	518400	IN	NS	g.root-servers.net.
.	518400	IN	NS	i.root-servers.net.
.	518400	IN	NS	k.root-servers.net.
.	518400	IN	NS	f.root-servers.net.
in.	172800	IN	NS	ns1.registry.in.
in.	172800	IN	NS	ns4.registry.in.
in.	172800	IN	NS	ns5.registry.in.
in.	172800	IN	NS	ns6.registry.in.
in.	172800	IN	NS	ns3.registry.in.
in.	172800	IN	NS	ns2.registry.in.
esic.in.	3600	IN	NS	ns-882.awsdns-46.net.
esic.in.	3600	IN	NS	ns-1089.awsdns-08.org.
esic.in.	3600	IN	NS	ns-1978.awsdns-55.co.uk.
esic.in.	3600	IN	NS	ns-52.awsdns-06.com.
www.esic.in.	3600	IN	NS	lbr1.esic.in.
www.esic.in.	3600	IN	NS	lbr2.esic.in.
www.esic.in.	0	IN	A	218.248.15.136

If you compare the two outputs and look closely, the authoritative nameservers have delegated www.esic.in to the name servers lbr1.esic.in and lbr2.esic.in

And at the time of the issue, the nameservers lbr1.esic.in and lbr2.esic.in did not respond to Do53(UDP) resulting in an NXDOMAIN!

DNSViz also reported the non-responsive nameservers as well as OpenDNS cachecheck,

At the time of writing this blog post, the name servers lbr1.esic.in. and lbr2.esic.in. were responding and www.esic.in was resolving correctly. But for more than 24+ hours, they were unresponsive resulting in some random people on the Internet in India being unable to access the website.

APNIC 52 – Threat Hunting using DNS

I presented on how we at my $dayjob do Threat Hunting using DNS at APNIC 52.

This is the same presentation I gave at SANOG 37, but luckily, I had the full quota of 20 minutes to complete the presentation without rushing into it.

Here is the video of the presentation,

Happy hunting!

sdns://2021 – Hyperlocal root and LocalRoot

Image Source: sdns2021.dnscrypt.info

I had the opportunity to present on Hyperlocal root and the LocalRoot project at sdns://2021 last week

I’ve written and presented about Hyperlocal root aka RFC 8806 in the past. In the context of privacy, Hyperlocal root does provide a possible solution to the problem,

Prevent snooping by third parties of requests sent to DNS root servers

RFC8806

Aside from that, faster negative responses to non-existent domains eliminates junk to the root

Speaking of junk to the root, I did mention in my presentation that Chromium 87 has stopped sending junk queries to the root based on the Chromium bug report and Verisign’s blog post Chromium’s reduction of root DNS traffic

I did a quick check with Google Chrome 92.0.4515.131 and oddly enough I am still seeing this behaviour,

Aug 16 09:03:24 unbound[1:0] info: 192.168.100.4 ckgydztukkdsta. A IN
Aug 16 09:03:24 unbound[1:0] info: 192.168.100.4 lubdcupibujjne. A IN
Aug 16 09:03:24 unbound[1:0] info: 192.168.100.4 ltvlataieb. A IN

This will need further researching and debugging which I will save for another post.

A big thank you to Frank for organising sdns://2021 and also to folks from Quad9 for their help.

For some reason as can be seen in the video, presentation is stuck at a specific slide, the PDF can be found here

All the presentations are available on Youtube.

SANOG 37 – Threat Hunting using DNS

PC: Mohan Thomas

At SANOG 37, I had the opportunity to share some of the ways in which we have been doing Threat Hunting using DNS at my $dayjob.

Here is the video of the presentation.

I also had a little demo but I decided to improvise and add slides instead, since the program was running a little behind schedule and I was the only one standing between everyone and their lunch. trouble was also lurking.

That aside, the same paper ‘Threat Hunting using DNS’ has been accepted at APNIC 52 and hopefully, I will be able to demo the juicy bits.

Jio VoWiFi issue – It’s always DNS!

tl;dr If Jio VoWiFi isn’t working for you, set a different DNS resolver on the phone. While I am a big proponent of running your own resolver in the network, ( runyourownresolver.in ) , you could test by using open resolvers. The issue doesn’t seem to be impacting everyone and only a subset of users.

To begin with, there are multiple things broken in the authoritative name servers ns1.vowifi.jio.com. and ns2.vowifi.jio.com. of vowifi.jio.com which I’ll cover a bit later.

I came across reports ( See here & here ) of Jio VoWiFi not working for many and while the reports were sketchy, I decided to test this myself.

Below is a snippet from a log file of a dns query to vowifi.jio.com from my phone(192.168.1.137) to a recursive resolver(Unbound) which I run in my network,

May 28 15:54:35 root unbound: [1300:0] info: 192.168.1.137 vowifi.jio.com. A IN

Ideally, the domain is standardised & is made up of Mobile Network Code(MNC) and Mobile Country Code(MCC). For example – in the case of Airtel VoWiFi, the domain name that I see hitting my Unbound resolver is epdg.epc.mnc045.mcc404.pub.3gppnetwork.org. where MNC – 045 and MCC – 404 which signifies Airtel – Karnataka region.

However, oddly enough, Reliance Jio seems to be using vowifi.jio.com. Having said that, the standardised domain name works as well. For example – epdg.epc.mnc861.mcc405.pub.3gppnetwork.org. resolves to 49.44.59.36 and 49.44.59.38

Below is the dns resolution entire delegation chain. From my home network, I can see that the vowifi.jio.com resolves to 49.44.59.38 and 49.44.59.36

.	518400	IN	NS	a.root-servers.net.
.	518400	IN	NS	b.root-servers.net.
.	518400	IN	NS	c.root-servers.net.
.	518400	IN	NS	d.root-servers.net.
.	518400	IN	NS	e.root-servers.net.
.	518400	IN	NS	f.root-servers.net.
.	518400	IN	NS	g.root-servers.net.
.	518400	IN	NS	h.root-servers.net.
.	518400	IN	NS	i.root-servers.net.
.	518400	IN	NS	j.root-servers.net.
.	518400	IN	NS	k.root-servers.net.
.	518400	IN	NS	l.root-servers.net.
.	518400	IN	NS	m.root-servers.net.
com.	172800	IN	NS	a.gtld-servers.net.
com.	172800	IN	NS	l.gtld-servers.net.
com.	172800	IN	NS	c.gtld-servers.net.
com.	172800	IN	NS	h.gtld-servers.net.
com.	172800	IN	NS	e.gtld-servers.net.
com.	172800	IN	NS	d.gtld-servers.net.
com.	172800	IN	NS	i.gtld-servers.net.
com.	172800	IN	NS	f.gtld-servers.net.
com.	172800	IN	NS	m.gtld-servers.net.
com.	172800	IN	NS	j.gtld-servers.net.
com.	172800	IN	NS	g.gtld-servers.net.
com.	172800	IN	NS	k.gtld-servers.net.
com.	172800	IN	NS	b.gtld-servers.net.
jio.com.	172800	IN	NS	ns1.jio.com.
jio.com.	172800	IN	NS	ns2.jio.com.
jio.com.	172800	IN	NS	ns3.jio.com.
jio.com.	172800	IN	NS	ns4.jio.com.
vowifi.jio.com.	3600	IN	NS	ns1.vowifi.jio.com.
vowifi.jio.com.	3600	IN	NS	ns2.vowifi.jio.com.
vowifi.jio.com.	5	IN	A	49.44.59.38
vowifi.jio.com.	5	IN	A	49.44.59.36

At this point, I confirmed that VoWiFi on Jio works by putting the phone on Airplane mode while remain connected to WiFi. A ~22 minute call worked flawlessly.

To confirm that vowifi.jio.com was indeed the domain name that needs to resolve for VoWiFi to work on Jio, I configured an entry for vowifi.jio.com to return a NXDOMAIN answer in my DNS RPZ aka DNS Firewall in Unbound.

With that configured, any DNS query for vowifi.jio.com from any device in the network will be meted out with a NXDOMAIN answer. Below is a snippet from the Unbound log confirming the RPZ rule applied.

May 28 17:31:50 root unbound: [1191:0] info: 192.168.0.137 vowifi.jio.com. A IN
May 28 17:31:50 root unbound: [1191:0] info: RPZ applied [custom block to test vowifi] vowifi.jio.com. nxdomain 192.168.0.137@64521 vowifi.jio.com. A IN
;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 14747
;; flags: qr aa rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; vowifi.jio.com.	IN	A

;; ANSWER SECTION:

;; AUTHORITY SECTION:

;; ADDITIONAL SECTION:

;; Query time: 136 msec
;; SERVER: 192.168.0.250
;; WHEN: Thu May 28 18:03:42 2020
;; MSG SIZE  rcvd: 32

And VoWiFi(Jio) stops working.

You can refer to the MNC and MCC codes list on Wikipedia – Mobile Network Codes in ITU region 4xx (Asia)

In the context of VoWiFi, the other noticeable problems with DNS infrastructure of Jio –

  1. A/AAAA records for ns1.vowifi.jio.com, ns2.vowifi.jio.com are missing
  2. ns1.vowifi.jio.com(49.44.59.6), ns2.vowifi.jio.com(49.44.59.7) don’t respond to queries over TCP

The other interesting thing that is worth observing is that when you try resolving vowifi.jio.com from outside India or use a DNS resolver which is perhaps not geographically located within India, the authoritative name servers ns1.vowifi.jio.com(49.44.59.6), ns2.vowifi.jio.com(49.44.59.7) give out a different set of IP addresses – 49.45.63.1, 49.45.63.2

; <<>> DiG 9.16.3 <<>> @127.0.0.1 vowifi.jio.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13728
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;vowifi.jio.com.			IN	A

;; ANSWER SECTION:
vowifi.jio.com.		4	IN	A	49.45.63.1
vowifi.jio.com.		4	IN	A	49.45.63.2

;; Query time: 352 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat May 30 06:23:23 IST 2020
;; MSG SIZE  rcvd: 75

@varkey at IBF pointed out the OpenDNS Cache Check website which also seems to confirm it.

To confirm this hypothesis, I decided to utilise the RIPE Atlas probes to run a measurement. If you’re unaware of the RIPE Atlas project, check an earlier post – Host a RIPE Atlas software probe in your network.

And the results of the measurement are interesting. Out of the 75 probes which participated in the measurement, there were many probes which received the response 49.45.63.1 & 49.45.63.2 to the DNS query to vowifi.jio.com

ASNAS NameDNS Response 1DNS Response 2Resolver IP address
4758NICNET-VSNL-BOARDER-AP National Informatics Centre, IN’49.45.63.249.45.63.1164.100.3.1
4758NICNET-VSNL-BOARDER-AP National Informatics Centre, IN’49.45.63.149.45.63.2164.100.3.1
24186RAILTEL-AS-IN RailTel Corporation of India Ltd., Internet Service Provider, New Delhi, IN’49.44.59.3649.44.59.38127.0.0.1
14061DIGITALOCEAN-ASN, US’49.44.59.3649.44.59.38127.0.0.1
18209BEAMTELE-AS-AP Atria Convergence Technologies pvt ltd, IN’49.44.59.3849.44.59.36202.53.8.8
18209BEAMTELE-AS-AP Atria Convergence Technologies pvt ltd, IN’49.44.59.3849.44.59.3649.207.46.6
135190UBERCORE-AS Ubercore Data Labs Private Limited, IN’49.45.63.149.45.63.2192.168.1.200
135817ESTOB-AS-AP Esto Broadband Private Limited, IN’49.45.63.149.45.63.28.8.8.8
18207YOU-INDIA-AP YOU Broadband & Cable India Ltd., IN’49.45.63.149.45.63.28.8.8.8
134316WORLD-AS World Star Communication, IN’49.44.59.3849.44.59.361.1.1.1
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’49.44.59.3849.44.59.36192.168.0.1
24309CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN’49.44.59.3649.44.59.3810.98.0.1
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’49.44.59.3849.44.59.36192.168.1.1
17625BLAZENET-IN-AP BlazeNet_s Network, IN’49.44.59.3849.44.59.36202.131.104.2
133661NETPLUS-AS Netplus Broadband Services Private Limited, IN’49.45.63.149.45.63.2192.168.1.2
133982EXCITEL-AS-IN Excitel Broadband Private Limited, IN’49.44.59.3649.44.59.38192.168.1.1
133318MAXTECHA-AS Maxtech, IN’None49.44.59.38192.168.1.1
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’49.44.59.3649.44.59.381.1.1.1
24309CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN’49.45.63.249.45.63.1fd00:1:2:3::1
131442DIGITALNETWORK-IN Digital Network Associates Pvt Ltd, IN’49.45.63.249.45.63.1192.168.10.1
135260FOURTY2COMM-AS 42 Communications Pvt. Ltd., IN’None49.45.63.1192.168.10.1
9430STPI-NOIDA Software Technology Parks of India,Block-IV, IN’49.44.59.3649.44.59.38192.168.1.1
132933CTPLAND-AS CharotarTelelink Pvt Ltd, IN’49.45.63.149.45.63.2192.168.1.1
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’49.44.59.3849.44.59.3645.90.28.112
17488HATHWAY-NET-AP Hathway IP Over Cable Internet, IN’49.44.59.3649.44.59.38202.88.152.8
55824NKN-CORE-NW NKN Core Network, IN’49.45.63.149.45.63.2200.200.200.7
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’None49.45.63.2200.200.200.7
18196SEVENSTAR-AS Seven Star Internet Service Provider, IN’49.45.63.149.45.63.28.8.8.8
9829BSNL-NIB National Internet Backbone, IN’49.45.63.149.45.63.22001:4860:4860::8888
134053EXPL-AS-IN ETHERNET XPRESS PVT. LTD., IN’49.44.59.3849.44.59.36fda9:ded9:2bc5::1
24309CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN’49.45.63.249.45.63.1192.168.1.1
9829BSNL-NIB National Internet Backbone, IN’49.44.59.3649.44.59.38192.168.1.1
24309CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN’49.45.63.149.45.63.210.0.0.1
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’49.44.59.3849.44.59.36202.56.215.55
55824NKN-CORE-NW NKN Core Network, IN’49.44.59.3849.44.59.36192.168.1.7
17747SITINETWORS-IN-AP SITI NETWORKS LIMITED, IN’49.45.63.149.45.63.2172.22.146.1
134053EXPL-AS-IN ETHERNET XPRESS PVT. LTD., IN’49.44.59.3649.44.59.3845.116.0.238
24186RAILTEL-AS-IN RailTel Corporation of India Ltd., Internet Service Provider, New Delhi, IN’49.44.59.3649.44.59.3810.0.7.253
23860ALLIANCE-GATEWAY-AS-AP Alliance Broadband Services Pvt. Ltd., IN’49.44.59.3649.44.59.38203.171.240.10
132215POWERGRID-IN Power Grid Corporation of India Limited, IN’49.45.63.149.45.63.210.0.0.1
9498BBIL-AP BHARTI Airtel Ltd., IN’49.45.63.249.45.63.1192.168.0.44
24309CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN’49.44.59.3849.44.59.36192.168.1.1
17813MTNL-AP Mahanagar Telephone Nigam Limited, IN’49.44.59.3649.44.59.3859.185.3.10
45528TIKONAIN-AS Tikona Infinet Ltd., IN’49.44.59.3849.44.59.36192.168.1.1
134325JETSPOTNETWORKSPVTLTD-AS JETSPOTNETWORKS PVT LTD, IN’49.44.59.3649.44.59.38192.168.1.1
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’49.44.59.3649.44.59.38192.168.1.1
24309CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN’49.44.59.3849.44.59.36202.83.21.43
17747SITINETWORS-IN-AP SITI NETWORKS LIMITED, IN’49.44.59.3649.44.59.381.1.1.1
9829BSNL-NIB National Internet Backbone, IN’49.44.59.3849.44.59.36192.168.1.1
134249MARGONW-AS Margo Networks Pvt Ltd, IN’49.44.59.3649.44.59.38172.28.242.252
12222AKAMAI, US’49.45.63.249.45.63.123.216.52.9
55836RELIANCEJIO-IN Reliance Jio Infocomm Limited, IN’49.44.59.3849.44.59.36192.168.29.1
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’49.44.59.3649.44.59.38192.168.1.1
136336TICFIBER-AS Thamizhaga Internet Communications Private Limited, IN’49.44.59.3849.44.59.36192.168.0.1
9829BSNL-NIB National Internet Backbone, IN’None49.44.59.36192.168.0.1
55824NKN-CORE-NW NKN Core Network, IN’None49.44.59.36192.168.0.1
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’49.44.59.3649.44.59.38125.22.47.125
55577BEAMTELE-AS-AP Atria Convergence Technologies pvt ltd, IN’49.44.59.3849.44.59.36192.168.1.254
134326AIRDESIGNBROADCAST-AS Airdesign Broadcast Media Pvt Ltd, IN’49.45.63.149.45.63.2208.67.222.222
138786CCBSPL-AS-IN Crystal Clear Broadband Services Pvt. Ltd., IN’49.45.63.149.45.63.28.8.8.8
56166IISERBNET-IN IISER Bhopal Campus, IN’49.44.59.3849.44.59.36172.30.1.2
24560AIRTELBROADBAND-AS-AP Bharti Airtel Ltd., Telemedia Services, IN’49.44.59.3849.44.59.36192.168.1.1
9829BSNL-NIB National Internet Backbone, IN’49.44.59.3649.44.59.38fdf6:a86d:4264::1
55836RELIANCEJIO-IN Reliance Jio Infocomm Limited, IN’49.44.59.3649.44.59.38192.168.31.1
139331DCORP-AS-AP DevelentCorp., IN’49.45.63.149.45.63.28.8.8.8
23860ALLIANCE-GATEWAY-AS-AP Alliance Broadband Services Pvt. Ltd., IN’49.44.59.3849.44.59.361.1.1.1
55836RELIANCEJIO-IN Reliance Jio Infocomm Limited, IN’49.45.63.149.45.63.28.8.4.4
17488HATHWAY-NET-AP Hathway IP Over Cable Internet, IN’49.44.59.3849.44.59.361.1.1.1
135718DISHAWAVESINFONET-AS DISHAWAVES INFONET PVT. LTD, IN’None49.44.59.361.1.1.1
4755TATACOMM-AS TATA Communications formerly VSNL is Leading ISP, IN’49.45.63.149.45.63.28.8.8.8
16509AMAZON-02, US’49.44.59.3849.44.59.36::1
15169GOOGLE, US’49.45.63.149.45.63.2::1
139331DCORP-AS-AP DevelentCorp., IN’49.44.59.3649.44.59.38::1
24309CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN’49.45.63.249.45.63.1192.168.1.10
9498BBIL-AP BHARTI Airtel Ltd., IN’49.45.63.149.45.63.2192.168.139.245

internationalvowifi.jio.com also seems to indicate VoWiFi International calling, which resolves to 49.44.59.36 and 49.44.59.38 from my vantage point. The same resolves to 49.45.63.1 and 49.45.63.2 from every location that I’ve managed to check from outside India.

Looking at the results, most likely the issue is with how ns1.vowifi.jio.com & ns2.vowifi.jio.com are responding to  client subnet (EDNS0) in DNS queries.

How to improve the root – Run it locally

Image shows the locations of the root server IP Anycast instances.
Source: https://root-servers.org/

Current State of DNS Root Servers

The DNS root server system uses IP Anycast.There are 13 root server operators with a total of 1084 instances all over the world. Let’s look at some of the problems in the context of the root server system,

Decrease the round trip time to the root servers

The round trip time to the root servers is dependent on multiple factors. Availability of a root server instance within the country and optimal routing. While the first can be addressed by installing an instance of the root server in a country, the second one is a bit hard to address. Routing determines whether the traffic to the root server from the last mile will reach the instance which is local or take the transit route to an instance outside the country.

If the traffic is transiting outside the country, the result is increased latency and poor performance in the context of DNS resolution.

Case in point, in the context of India, Netnod which is a root server operator managing the i-root-servers.net has an Ancast IPv4 node in Mumbai.

A traceroute from AS9498 to i.root-servers.net shows that traffic is not hitting the local instance but taking the transit route.

traceroute from AS9498 to i.root-servers.net
The above image has been taken from a RIPE Atlas measurement.

Similarly, RIPE NCC is the root server operator managing the k.root-servers.net. Again, in the context of India, there is an Anycast node IPv6 node in Mumbai and Noida.

A traceroute from AS9498 to k.root-servers.net shows that traffic is not hitting the local instance but taking the transit route.

traceroute from AS9498 to k.root-servers.net
The above image has been taken from a RIPE Atlas measurement.

If you aren’t aware of the RIPE Atlas project, check the earlier post

Prevent snooping of queries

In the case of traditional DNS or DNS over 53( Do53), the traffic is unencrypted. In response to the privacy concerns and to secure DNS traffic between the client and the recursive resolver, IETF standardised DNS-over-HTTPS (DoH) and DNS-over-TLS (DoT). While both of the protocols secure the communication between the client and the recursive resolver, traffic between the recursive resolver and the root servers is still in the open i.e unencrypted.

Faster negative responses to queries for non-existent domains

I would like to point you to the earlier posts on Chromium based browsers and DNS, and Junk to the root as that would set the context for this one.

The recent study by ICANN OCTO reveals that a vast majority of the queries to the root servers are for names which do not exist in the root zone. By providing faster negative responses to non existent domains to the stub resolver, eliminate sending the junk queries to the root servers entirely.

Increase the resiliency of the root server system

In the context of DNS, the primary intention of using IP Anycast is to have the topologically closest server provide the answer. This model fails if there is suboptimal routing as seen in the examples of traceroute to the root servers earlier.

The additional benefit of using IP Anycast is that considering optimal routing, in the event of a DDoS attack, the impact is limited in effect as it gets confined to certain areas. In the past, IP Anycast has helped to mitigate attacks on the root server system where the attack became limited in scope to certain Anycast instances of the root server and caused a saturation of network connections.

On the contrary, Mirai botnet attack on Dyn Infrastructure also tells us that large scale attack can cause congestion across the Anycast instances resulting in unavailability of services.

Finally, we get to a set of broader questions – How do we increase resiliency against a DDoS on the root server system ?  Since the root server system doesn’t penalise abuse (period), should we continue abusing it ? 

A probable solution as proposed in RFC 7706 is to run a local copy of the full root zone on the loopback. What this essentially suggests is that the full root zone on the loopback will serve as upstream to the recursive resolver and the recursive resolver should be able to validate the zone from the upstream using DNSSEC.

In order to implement this, one first needs a copy of the root zone. The following root servers currently allow transfer of the root zone using AXFR over TCP,

Sl. NoRoot Server Operator
1b.root-servers.net
2c.root-servers.net
3d.root-servers.net
4f.root-servers.net
5g.root-servers.net
6k.root-servers.net
7lax.xfr.dns.icann.org & iad.xfr.dns.icann.org (L-root server)
Root Server Operators which support transfer of the root zone

The process of manually pulling the root zone has an operational issue – one needs to periodically check if the root zone has changed in the root zone copy at the upstream and then update the copy of the root zone configured to run on the loopback.

Even though RFC 7706 is Informational, recursive resolver software such as ISC BIND, Unbound, Knot Resolver have built-in support.

Slaving of the root zone – ISC BIND 9.16.3(stable)

Image of an excerpt from named.conf showing the slaving of the root zone configuration

Part II of this post will contain operational instructions for running a local copy of the root zone and document some of the pitfalls of doing so.

DNS RPZ (Response Policy Zones) – Using DNS as a layer of defence – Part I

Update (06/08/2020)APNIC has published this post on their blog. Robbie Mitchell from APNIC was of great help in correcting a few things and polishing the article. You can read the Part 1 on the APNIC blog here

DNS(Domain Name System) is the crucial & ubiquitous fabric of the Internet.  While on the surface, users rely on accessing websites, apps, email etc underneath it’s the DNS database which provides the map for the Internet.

It’s fair to say that everything on the Internet begins with a DNS query. This means that the DNS is used for legitimate purposes and as well as abused by bad actors.

Adding a layer of security to a flat network

In the context of COVID-19, where most of us are working from home, security of the the devices & data being accessed from a hostile home network has become a major talking point over the last couple of months. The home network is atypical from an enterprise network from a security perspective and apart from its inherent flaws, it’s a flat network.

flat network is a computer network design approach that aims to reduce cost, maintenance and administration.[1] Flat networks are designed to reduce the number of routers and switches on a computer network by connecting the devices to a single switch instead of separate switches. Unlike a hierarchical network design, the network is not physically separated using different switches.

The topology of a flat network is not segmented or separated into different broadcast areas by using routers.

Wikipedia

Here is a representation of a flat network design,

The constraints of a flat network are,

  • No segmentation of traffic – Single broadcast domain
  • Easy & rapid propagation of malicious traffic within the network

One of the layers of security that can be brought into a flat network at an economical cost is by leveraging DNS. Before we look into how that can be implemented, here is a DNS primer for what happens when a domain name is accessed in a network,

Shift of the recursive resolvers

In the above diagrammatic representation, the part which is doing the most heavy lifting is the Recursive DNS Server or Recursive resolver. At the very beginning of the Internet, users themselves ran recursive resolvers on the machines or in the network. This model slowly shifted to the network operators (ISP’s) offering this as a bundled, free of cost offering along with the service. And the model has moved DNS resolution further away from the user with the advent of the Cloud/Quad DNS providers. To name a notable few, Google Public DNS (8.8.8.8, 8.8.4.4), Cloudflare (1.1.1.1, 1.0.0.1), Quad9(9.9.9.9) etc.

While each of these open resolvers services promote faster dns resolution, in reality they are still further away from the user from a round trip metric. Even though all of these open resolver services use IP Anycast, the proximity to the user cannot compete with a local resolver. In obvious terms, the recursive resolver which is in the users network or even the resolver provided by the Internet Service Provider will always be closest.

The one definitive advantage that the cloud/quad DNS open resolvers provide is the availability of a large cache.

If you aren’t convinced yet on running your own DNS resolver instead of outsourcing it to the cloud/quad DNS providers, I would urge you to read Why should I run my own DNS resolver?

And most importantly, if you want to leverage DNS Response Policy Zones (DNS Firewall) to add a layer of security in your network, you need to run a recursive resolver.

What is DNS Response Policy Zones(RPZ) ?

  • It’s currently an Internet-draft and not a standard yet. The latest draft is available here
  • It’s a vendor neutral – BIND, Unbound, PowerDNS Recursor support it
  • Allows policy to be applied to DNS queries. Set a differentiated route for the bad domains
  • Economical solution – a RaspberryPi can act as recursive resolver with DNS RPZ for the entire network – especially useful & low cost solution for home networks, SOHO etc

Just like the functioning of a firewall, RPZ is made up of TRIGGERS & ACTIONS.

This is all good but without threat intelligence data, a DNS Firewall doesn’t add any value.

Threat intelligence RPZ feeds

While there are many threat intelligence providers which provide a DNS RPZ feed, below are some of the free/community ones,

Part II of this post will contain instructions for configuring a RPZ feed in ISC BIND9.