Malicious Bots Want This File

The internet is a scary place full of malware and malicious activity. Learn how to defend against malicious bots searching for sensitive information on your server. Specifically, we will learn how to hide hidden files/directories from malicious web crawlers and hackers, while whitelisting specific files/directories, using the Nginx location context.

Malicious Activity

If you have ever gone through your logs for a website you own, you might come across a lot of malicious activity that looks like this:

[01/Jun/2024:22:49:14 +0000]: 301 "GET /.env HTTP/1.1"
[01/Jun/2024:22:49:14 +0000]: 301 "GET /.env HTTP/1.1"
[01/Jun/2024:22:49:14 +0000]: 301 "GET /.env HTTP/1.1"

Here, a malicious bot/web crawler is hoping that my server is publicly exposing an environment variable file called .env. This is a common name for a file containing API keys, database credentials, etc. Files that start with a dot like this, such as .env, .htaccess, etc. are known as hidden files. They are hidden because they often contain sensitive information. Because of this, there are bots that crawl the internet looking for sites that have these files exposed.

Don't Serve Up Hidden Files

As these files are sensitive, they should not be served up on any request. In other words, whatever directory Nginx is serving static content from, these hidden files should not be there. For example, a common place to serve static content from with Nginx is the following directory.

/usr/share/nginx/html

No hidden files should be placed in this directory. A good way to prevent this is to have a dedicated folder for serving static content, such as a public folder. The code below is a Dockerfile for generating an Nginx image. Notice how the public folder is placed in the /usr/share/nginx/html directory. This is where the name public comes from, its contents will be available to the public.

FROM nginx:alpine
COPY ./public /usr/share/nginx/html
COPY ./conf/nginx.conf /etc/nginx/nginx.conf
COPY ./conf/upstream.default.conf /etc/nginx/templates/default.conf.template

WARNING: Notice how we also have a conf folder for holding configuration.

Denying Access to Hidden Files

Another form of prevention is to deny access to any hidden file. We can do this with the Nginx location context and regex.

location ~ /\. {
  deny all;
}

INFO: The ~ symbol is used by Nginx to compare a URI to a case-sensitive regular expression.

Here we are telling Nginx to deny any requests to URLs starting with a dot. Now if we send a cURL request to a hidden file Nginx will return a 403.

curl localhost/.env
<html>
<head><title>403 Forbidden</title></head>
<body>
<center><h1>403 Forbidden</h1></center>
<hr><center>nginx/1.25.5</center>
</body>
</html>

We can also hide some underlying implementation by returning a 404 as opposed to displaying a forbidden message.

location ~ /\. {
  deny all;
  return 404;
}

The return directive stops processing and returns the specified status code. Therefore, if we send a cURL we will now get a 404.

curl localhost/.env
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.25.5</center>
</body>
</html>

Allowing Access to Let's Encrypt

However, there are certain hidden directories/files that need to be accessible such as the .well-known directory Let's Encrypt uses for SSL. We can make this directory accessible by placing it before the catch all hidden location.

location ~ /\.well-known {
  return 200;
}

location ~ /\. {
  deny all;
  return 404;
}

INFO: Nginx applies locations with regular expressions in the order of their appearance in the configuration file.

Now if we make requests to /.well-known we will get a 200 while any other hidden file/directory will still return a 404.

curl localhost/.env           
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx/1.25.5</center>
</body>
</html>
curl localhost/.well-known 
curl localhost/.well-known/hi