If you are an MSME and are looking at complying to the CERT-Indirectives on logs, then, a sensor we’ve built for generating and storing logs of the entire network, might just be what you are looking for.
All service providers, intermediaries, data centres, body corporate and Government organisations shall mandatorily enable logs of all their ICT systems and maintain them securely for a rolling period of 180 days and the same shall be maintained within the Indian jurisdiction. These should be provided to CERT-In along with reporting of any incident or when ordered / directed by CERT-In.
Challenges faced in incident response environments (MSME) with no logs
The idea of building a sensor stemmed from our experiences of incident response in environments with zero security posture.
The same sensor can capture network packets and generate logs per the CERT-In directives.
At the btNOG-9 Conference on the 14th October 2022, I’ll be presenting Incident Response on a shoestring budget
In the presentation, I’ll share the challenges we faced in incident response environments with zero security posture, i.e. lacking logs, etc. The presentation will then focus on the solution – a sensor we built using open-source software such as Suricata and Zeek, logging DNS queries etc.
By deploying a sensor in the network, MSMEs can comply with the CERT-In directives and also facilitate incident responders to investigate security incidents.
Incident responders can leverage the rich logs by intercepting and ingesting packets into tools such as Zeek. If you are new to Zeek, check the blog post, Packets don’t lie – Threat Hunting with Zeek and the APNIC Academy page where a recording of the webinar will be available soon.
Thanks to the kind folks at APNIC for initiating the request and starting the email thread.
The gist of the presentation was about using Zeek to look for anomalies. Before jumping into Zeek, I introduced Network Security Monitoring. Spoke about conn.log and dns.log and used PCAPs from Stratosphere IPS Project to demonstrate threat hunting with Zeek.
Zeek logs are a great source in the context of threat hunting and Incident Response.
A total of 203 folks had registered for the webinar, and around 55-60 attended. That’s been my experience with online webinars and workshops – many folks will register, but a small fraction attend.
While one hour webinar is a brief period to talk about all-things-zeek, I hope the webinar gives a quick introduction to getting started.
But the most important thing was the interactive Q&A session at the end.
The webinar was recorded and should be available in a few days. I will update the blog post with a link to the recording and the slides.
Also, since I am on the topic of Zeek, ZeekWeek 2022 is an in-person event on October 12th – 14th in Austin, TX.
An excellent line-up of speakers, and the schedule is packed with goodness.
While investigating a bit of oddity with the Skype app on Mac OS X, I wanted to capture all traffic from only the Skype processes.
But first, a little background on the issue. All DNS traffic from my systems is routed through a WireGuard tunnel. The peer endpoint at the other end runs a recursive resolver with DNS Response Policy Zones (DNS RPZ).
I’ve been running a WireGuard tunnel and routing DNS from my phone through a recursive resolver(unbound + rpz). Here are the battery stats.
Prior to WireGuard, an OpenVPN tunnel on the phone would drain the battery out in no time! pic.twitter.com/0rJVcJk1iq
The issue is – that as soon as the WireGuard tunnel is disabled, Skype will try connecting to Google DNS(8.8.8.8) and www.bing.com. Perhaps Skype bypasses the local recursive resolver set on the system and sends the DNS queries for its hosts to Google DNS directly? I cannot ascertain that yet, and it warrants a thorough investigation.
I was alerted to this by Little Snitch after I refreshed the rules. A ritual that I seem to follow every few months.
Little Snitch Network Monitor contains a hidden gem. The ability to capture traffic of a specific process.
For example, by right-clicking on any process, you should see the option to Capture Traffic. That should open a terminal window prompting for the sudo password.
After capturing traffic and interrupting should result in a PCAP on the Desktop. Neat!
Measuring ping and DNS from different vantage points using RIPE Atlas has been something that I have been using for some time now. You can read about installing a RIPE Atlas software probe here
A few weeks ago, I came across Shodan’s geoping and geodns API, which provides ping and DNS lookup from a few locations and other details such as RTT. This is great because you can quickly check ping and DNS resolution on systems where you only have curl running.
You always have the RIPE Atlas project for more detailed and sophisticated use-cases. To get started with the RIPE Atlas project, check the webinar I delivered some time ago for APNIC.
A couple of weeks ago, at my $dayjob, we implemented a recursive resolver with RPZ in an enterprise network.
After a few days, the customer got back to us with an issue – the DNS resolution of the domain esic.in failed with an NXDOMAIN response. After a cursory look at the problem, it became evident that esic.in resolved correctly but www.esic.in did not.
The customer also reported that if they switched the resolver to 8.8.8.8, the DNS resolution of www.esic.in was without any problems, and the website was accessible in the network.
So, what is causing the DNS issue with www.esic.in with the on-prem resolver?
Let’s find out. To start with the basics, here are the authoritative name servers of the domain esic.in,
$ whois esic.in | grep "Name Server:"
Name Server: ns-1089.awsdns-08.org
Name Server: ns-52.awsdns-06.com
Name Server: ns-1978.awsdns-55.co.uk
Name Server: ns-882.awsdns-46.net
If we traverse the DNS delegation from the root to esic.in, we get valuable insights,
. 518400 IN NS k.root-servers.net.
. 518400 IN NS l.root-servers.net.
. 518400 IN NS d.root-servers.net.
. 518400 IN NS e.root-servers.net.
. 518400 IN NS j.root-servers.net.
. 518400 IN NS b.root-servers.net.
. 518400 IN NS g.root-servers.net.
. 518400 IN NS a.root-servers.net.
. 518400 IN NS h.root-servers.net.
. 518400 IN NS m.root-servers.net.
. 518400 IN NS i.root-servers.net.
. 518400 IN NS c.root-servers.net.
. 518400 IN NS f.root-servers.net.
in. 172800 IN NS ns1.registry.in.
in. 172800 IN NS ns2.registry.in.
in. 172800 IN NS ns3.registry.in.
in. 172800 IN NS ns4.registry.in.
in. 172800 IN NS ns5.registry.in.
in. 172800 IN NS ns6.registry.in.
esic.in. 3600 IN NS ns-882.awsdns-46.net.
esic.in. 3600 IN NS ns-1978.awsdns-55.co.uk.
esic.in. 3600 IN NS ns-52.awsdns-06.com.
esic.in. 3600 IN NS ns-1089.awsdns-08.org.
esic.in. 300 IN A 115.113.201.36
esic.in. 300 IN A 218.248.15.136
esic.in. 172800 IN NS ns-1089.awsdns-08.org.
esic.in. 172800 IN NS ns-1978.awsdns-55.co.uk.
esic.in. 172800 IN NS ns-52.awsdns-06.com.
esic.in. 172800 IN NS ns-882.awsdns-46.net.
And, here is the delegation trace from the root to www.esic.in,
. 518400 IN NS a.root-servers.net.
. 518400 IN NS e.root-servers.net.
. 518400 IN NS c.root-servers.net.
. 518400 IN NS b.root-servers.net.
. 518400 IN NS m.root-servers.net.
. 518400 IN NS l.root-servers.net.
. 518400 IN NS h.root-servers.net.
. 518400 IN NS j.root-servers.net.
. 518400 IN NS d.root-servers.net.
. 518400 IN NS g.root-servers.net.
. 518400 IN NS i.root-servers.net.
. 518400 IN NS k.root-servers.net.
. 518400 IN NS f.root-servers.net.
in. 172800 IN NS ns1.registry.in.
in. 172800 IN NS ns4.registry.in.
in. 172800 IN NS ns5.registry.in.
in. 172800 IN NS ns6.registry.in.
in. 172800 IN NS ns3.registry.in.
in. 172800 IN NS ns2.registry.in.
esic.in. 3600 IN NS ns-882.awsdns-46.net.
esic.in. 3600 IN NS ns-1089.awsdns-08.org.
esic.in. 3600 IN NS ns-1978.awsdns-55.co.uk.
esic.in. 3600 IN NS ns-52.awsdns-06.com.
www.esic.in. 3600 IN NS lbr1.esic.in.
www.esic.in. 3600 IN NS lbr2.esic.in.
www.esic.in. 0 IN A 218.248.15.136
If you compare the two outputs and look closely, the authoritative nameservers have delegated www.esic.in to the name servers lbr1.esic.in and lbr2.esic.in
And at the time of the issue, the nameservers lbr1.esic.in and lbr2.esic.in did not respond to Do53(UDP) resulting in an NXDOMAIN!
DNSViz also reported the non-responsive nameservers as well as OpenDNS cachecheck,
At the time of writing this blog post, the name servers lbr1.esic.in. and lbr2.esic.in. were responding and www.esic.in was resolving correctly. But for more than 24+ hours, they were unresponsive resulting in some random people on the Internet in India being unable to access the website.
I presented on how we at my $dayjob do Threat Hunting using DNS at APNIC 52.
This is the same presentation I gave at SANOG 37, but luckily, I had the full quota of 20 minutes to complete the presentation without rushing into it.
I had the opportunity to present on Hyperlocal root and the LocalRoot project at sdns://2021 last week
I’ve written and presented about Hyperlocal root aka RFC 8806 in the past. In the context of privacy, Hyperlocal root does provide a possible solution to the problem,
Prevent snooping by third parties of requests sent to DNS root servers
RFC8806
Aside from that, faster negative responses to non-existent domains eliminates junk to the root
I did a quick check with Google Chrome 92.0.4515.131 and oddly enough I am still seeing this behaviour,
Aug 16 09:03:24 unbound[1:0] info: 192.168.100.4 ckgydztukkdsta. A IN
Aug 16 09:03:24 unbound[1:0] info: 192.168.100.4 lubdcupibujjne. A IN
Aug 16 09:03:24 unbound[1:0] info: 192.168.100.4 ltvlataieb. A IN
This will need further researching and debugging which I will save for another post.
A big thank you to Frank for organising sdns://2021 and also to folks from Quad9 for their help.
For some reason as can be seen in the video, presentation is stuck at a specific slide, the PDF can be found here
At SANOG 37, I had the opportunity to share some of the ways in which we have been doing Threat Hunting using DNS at my $dayjob.
Here is the video of the presentation.
I also had a little demo but I decided to improvise and add slides instead, since the program was running a little behind schedule and I was the only one standing between everyone and their lunch. trouble was also lurking.
That aside, the same paper ‘Threat Hunting using DNS’ has been accepted at APNIC 52 and hopefully, I will be able to demo the juicy bits.
Over the course of the last year, the Flat Network at home has become an important extension of the enterprise network.
Figure 1 – A simple representation of a flat network
What is a flat network?
The network is not segmented i.e computers/devices can access any other computer/device in the network
A simple design with the goal to reduce cost, maintenance and administration
From a security perspective, the design poses a few challenges,
Systems/devices that are part of the network can communicate with one another – malware working the way laterally and propagating
Lack of visibility into the nature of communication between devices/systems in the network and the Internet – Getting answers to questions such as which IP addresses on the Internet does the smart bulb (or replace it with any Internet of Trash device) connect to or why is the Windows OS connecting to an IP address in China etc
Both of the above are traditionally addressed using Network Security Monitoring (NSM)
At my $dayjob, we implement NSM for enterprises and that requires investment on hardware, configuration and maintenance.
In the context of a home network, one wouldn’t be happy or interested in spending a large amount of money to setup a full fledged NSM at home (unless you are like me).
That’s exactly where the Corelight@Home project comes in.
Corelight is excited to announce the Corelight@Home program, bringing Corelight’s enterprise-class Network Detection and Response to home networks. While it is not a commercially available or officially supported product, it has all the same capabilities you’ll find in our Corelight Sensors. It combines all the goodness of open source Zeek and Suricata plus most of the value-added features of Corelight Sensors, FREE for home use. Put it all together on cheap, dependable hardware, and you can shine a light on suddenly vital home networks.
And the best part, it has been designed to be deployed on a RaspberryPi.
So, apart from the investment of a RaspberryPi, one will also need a network switch which can mirror packets or a network tap. Considering the goal of implementing NSM in the home network for cheap, a network switch is the only choice.
In India, the cheapest that I could find was the Netgear GS310TP for Rs. 8800 (inclusive GST) or around $120
Figure 2 – Netgear GS310TP
If you already have a managed/smart switch suggest that you check the manual and see if it supports port mirroring.
Figure 3 – Flat network with a NSM sensor (Raspberry Pi)
After receiving the login details, setup the RaspberryPi based on instructions in the Software Sensor Docs.
The next step is configuring the network switch to mirror packets (port mirroring). The simplest configuration is to mirror the uplink port (port on the network switch where the Internet router is connected) to the sensor port (port on the network switch which connects the RaspberryPi )
With the sensor configuration done, the last bit that remains is to ship the Zeek logs to Splunk or Humio for pretty dashboards.
Corelight Software Sensor supports a large number of exports,
Splunk (via the HTTP Event Collector) or Humio
Kafka
JSON over TCP
Syslog
Redis
The idea behind the implementation in the home network was to have visibility into the home network for cheap. Considering that, and the assumption that most won’t have decent compute to run an ELK stack etc, I went with Humio. They provide a Free SaaS tier with 2 GB inject per day and retention of one week.
Figure 4 – Humio dashboard with network insights
Note – One can also negate the Corelight Software Sensor and setup Zeek, Suricata and configure the RaspberryPi as a sensor. Aside from the simplicity of getting started with an NSM, the Corelight Software Sensor also provides more insight into encrypted traffic, built-in integration into Zeek and Suricata such that pivoting between them is easy.
tl;dr If Jio VoWiFi isn’t working for you, set a different DNS resolver on the phone. While I am a big proponent of running your own resolver in the network, you could test by using open resolvers. The issue doesn’t seem to be impacting everyone and only a subset of users.
To begin with, there are multiple things broken in the authoritative name servers ns1.vowifi.jio.com. and ns2.vowifi.jio.com. of vowifi.jio.com which I’ll cover a bit later.
I came across reports ( See here & here ) of Jio VoWiFi not working for many and while the reports were sketchy, I decided to test this myself.
Below is a snippet from a log file of a dns query to vowifi.jio.com from my phone(192.168.1.137) to a recursive resolver(Unbound) which I run in my network,
May 28 15:54:35 root unbound: [1300:0] info: 192.168.1.137 vowifi.jio.com. A IN
Ideally, the domain is standardised & is made up of Mobile Network Code(MNC) and Mobile Country Code(MCC). For example – in the case of Airtel VoWiFi, the domain name that I see hitting my Unbound resolver is epdg.epc.mnc045.mcc404.pub.3gppnetwork.org. where MNC – 045 and MCC – 404 which signifies Airtel – Karnataka region.
However, oddly enough, Reliance Jio seems to be using vowifi.jio.com. Having said that, the standardised domain name works as well. For example – epdg.epc.mnc861.mcc405.pub.3gppnetwork.org. resolves to 49.44.59.36 and 49.44.59.38
Below is the dns resolution entire delegation chain. From my home network, I can see that the vowifi.jio.com resolves to 49.44.59.38 and 49.44.59.36
. 518400 IN NS a.root-servers.net.
. 518400 IN NS b.root-servers.net.
. 518400 IN NS c.root-servers.net.
. 518400 IN NS d.root-servers.net.
. 518400 IN NS e.root-servers.net.
. 518400 IN NS f.root-servers.net.
. 518400 IN NS g.root-servers.net.
. 518400 IN NS h.root-servers.net.
. 518400 IN NS i.root-servers.net.
. 518400 IN NS j.root-servers.net.
. 518400 IN NS k.root-servers.net.
. 518400 IN NS l.root-servers.net.
. 518400 IN NS m.root-servers.net.
com. 172800 IN NS a.gtld-servers.net.
com. 172800 IN NS l.gtld-servers.net.
com. 172800 IN NS c.gtld-servers.net.
com. 172800 IN NS h.gtld-servers.net.
com. 172800 IN NS e.gtld-servers.net.
com. 172800 IN NS d.gtld-servers.net.
com. 172800 IN NS i.gtld-servers.net.
com. 172800 IN NS f.gtld-servers.net.
com. 172800 IN NS m.gtld-servers.net.
com. 172800 IN NS j.gtld-servers.net.
com. 172800 IN NS g.gtld-servers.net.
com. 172800 IN NS k.gtld-servers.net.
com. 172800 IN NS b.gtld-servers.net.
jio.com. 172800 IN NS ns1.jio.com.
jio.com. 172800 IN NS ns2.jio.com.
jio.com. 172800 IN NS ns3.jio.com.
jio.com. 172800 IN NS ns4.jio.com.
vowifi.jio.com. 3600 IN NS ns1.vowifi.jio.com.
vowifi.jio.com. 3600 IN NS ns2.vowifi.jio.com.
vowifi.jio.com. 5 IN A 49.44.59.38
vowifi.jio.com. 5 IN A 49.44.59.36
At this point, I confirmed that VoWiFi on Jio works by putting the phone on Airplane mode while remain connected to WiFi. A ~22 minute call worked flawlessly.
To confirm that vowifi.jio.com was indeed the domain name that needs to resolve for VoWiFi to work on Jio, I configured an entry for vowifi.jio.com to return a NXDOMAIN answer in my DNS RPZ aka DNS Firewall in Unbound.
With that configured, any DNS query for vowifi.jio.com from any device in the network will be meted out with a NXDOMAIN answer. Below is a snippet from the Unbound log confirming the RPZ rule applied.
May 28 17:31:50 root unbound: [1191:0] info: 192.168.0.137 vowifi.jio.com. A IN
May 28 17:31:50 root unbound: [1191:0] info: RPZ applied [custom block to test vowifi] vowifi.jio.com. nxdomain 192.168.0.137@64521 vowifi.jio.com. A IN
;; ->>HEADER<<- opcode: QUERY, rcode: NXDOMAIN, id: 14747
;; flags: qr aa rd ra ; QUERY: 1, ANSWER: 0, AUTHORITY: 0, ADDITIONAL: 0
;; QUESTION SECTION:
;; vowifi.jio.com. IN A
;; ANSWER SECTION:
;; AUTHORITY SECTION:
;; ADDITIONAL SECTION:
;; Query time: 136 msec
;; SERVER: 192.168.0.250
;; WHEN: Thu May 28 18:03:42 2020
;; MSG SIZE rcvd: 32
In the context of VoWiFi, the other noticeable problems with DNS infrastructure of Jio –
A/AAAA records for ns1.vowifi.jio.com, ns2.vowifi.jio.com are missing
ns1.vowifi.jio.com(49.44.59.6), ns2.vowifi.jio.com(49.44.59.7) don’t respond to queries over TCP
The other interesting thing that is worth observing is that when you try resolving vowifi.jio.com from outside India or use a DNS resolver which is perhaps not geographically located within India, the authoritative name servers ns1.vowifi.jio.com(49.44.59.6), ns2.vowifi.jio.com(49.44.59.7) give out a different set of IP addresses – 49.45.63.1, 49.45.63.2
; <<>> DiG 9.16.3 <<>> @127.0.0.1 vowifi.jio.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 13728
;; flags: qr rd ra; QUERY: 1, ANSWER: 2, AUTHORITY: 0, ADDITIONAL: 1
;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;vowifi.jio.com. IN A
;; ANSWER SECTION:
vowifi.jio.com. 4 IN A 49.45.63.1
vowifi.jio.com. 4 IN A 49.45.63.2
;; Query time: 352 msec
;; SERVER: 127.0.0.1#53(127.0.0.1)
;; WHEN: Sat May 30 06:23:23 IST 2020
;; MSG SIZE rcvd: 75
To confirm this hypothesis, I decided to utilise the RIPE Atlas probes to run a measurement. If you’re unaware of the RIPE Atlas project, check an earlier post – Host a RIPE Atlas software probe in your network.
And the results of the measurement are interesting. Out of the 75 probes which participated in the measurement, there were many probes which received the response 49.45.63.1 & 49.45.63.2 to the DNS query to vowifi.jio.com
ASN
AS Name
DNS Response 1
DNS Response 2
Resolver IP address
4758
NICNET-VSNL-BOARDER-AP National Informatics Centre, IN’
49.45.63.2
49.45.63.1
164.100.3.1
4758
NICNET-VSNL-BOARDER-AP National Informatics Centre, IN’
49.45.63.1
49.45.63.2
164.100.3.1
24186
RAILTEL-AS-IN RailTel Corporation of India Ltd., Internet Service Provider, New Delhi, IN’
TATACOMM-AS TATA Communications formerly VSNL is Leading ISP, IN’
49.45.63.1
49.45.63.2
8.8.8.8
16509
AMAZON-02, US’
49.44.59.38
49.44.59.36
::1
15169
GOOGLE, US’
49.45.63.1
49.45.63.2
::1
139331
DCORP-AS-AP DevelentCorp., IN’
49.44.59.36
49.44.59.38
::1
24309
CABLELITE-AS-AP Atria Convergence Technologies Pvt. Ltd. Broadband Internet Service Provider INDIA, IN’
49.45.63.2
49.45.63.1
192.168.1.10
9498
BBIL-AP BHARTI Airtel Ltd., IN’
49.45.63.1
49.45.63.2
192.168.139.245
internationalvowifi.jio.com also seems to indicate VoWiFi International calling, which resolves to 49.44.59.36 and 49.44.59.38 from my vantage point. The same resolves to 49.45.63.1 and 49.45.63.2 from every location that I’ve managed to check from outside India.
Looking at the results, most likely the issue is with how ns1.vowifi.jio.com & ns2.vowifi.jio.com are responding to client subnet (EDNS0) in DNS queries.
Recommended reading
If you enjoyed reading this blog post, you might find root hints vs RFC 8806 interesting.
