I just started hosting a private web service on Vultr, a VPS provider, for my own convenience.

Though It has been 1~2 years since I started messing with stuff like tcp, http, server development, nginx… I’ve never really touch some of the key parts: how domain name works, how to configure dns, how to properly setup a web server machine.

It’s quite a learning experience that worth a blog post, so here we go.

VPS Provider: Vultr

I’ve been dealing with GCP in last 2 years, actually GCP is the only cloud provider I’ve ever tried, and the experience is not so good:

  • The console is very powerful, very complicated, and very fragmented. It’s packed with all sort of cloud stuff Google provides, and a lot of the products are left untouched most of the time, while still being there, costing your attention.
  • IMO, the UI is confusing, mainly becuase all the infomation is dumped on you, and most of the text elements are of same size.
  • Also, the UI is kinda slow in my experience, and we usually have to move between tabs a lot, which makes the slowness prominent.
  • The pricing is very complicated and dynamic. It’s a really good concept to devide all things up and charge separately according to usages, but it’s not quite friendly to users. I recently found that I have several disks of deleted vm that had been charging me for several months. Had I not noticed the money I paid was surpirsing, I may never notice.

Vultr is mainly about VPS, not much more, the UI is clean and easy to navigate.

As fo pricing, it’s cool. Vultr asks you deposit first, and it adopts fixed pricing model. I feel at ease knowing that my deposited $10 will run my machine for about 2 months.

The machine deployment is straightforward, complete the form and push the deploy button, it’ll start doing it job. In minutes, the machine’d be ready for you to ssh into it from your machine or the web console.

Domain Registrar: Namecheap

From time to time, I’d feel an urge to buy a domain for my idea, but had never really done once.

I decided to buy my domains on Namecheap, the buying part is easy, just filling forms and paying the money.

The hard part for me is configuring the DNS records. I have a mediocre understanding of how DNS works, but I knows n0thing about how to set it up… What is DNS record? And What are A, AAAA, CNAME, MX, SRV?

Googled them. Tried and errors. Hours gone. The key missing piece is that I don’t know that domain names point to IP, not IP:port, this one cost me quite some time to figure out.

It’s actually not complicated at all, at least for my purpose. The only 1 DNS Record needed is:

  • A RECORD: It decides what IP the domain points to. The Name of a A Record means subdomain, the Data is the IP address it points to. For example, if we setup a record whose Name is blog, Data is 1.2.3.4. Assuming the registered domain is cool.xyz, when we try to connect to blog.cool.xyz, the DNS tells our computer to go to 1.2.3.4. If it’s an IPV6 address, use AAAA Record instead.

The Namecheap experience is smooth, interfaces are easy to use and stuff are easy to understance. I’ve never tried other registrars, so nothing to complain.

Setting Up The Machine

The most familiar Linux OS for me is CentOS, so that’s what loaded on the mechine deployed on Vultr.

Some random notes:

  • ~/.ssh/authorized_keys is a file not a directory.

Setting up Nginx

To route http packets to the web service listening to another port, a web server listening to 80 or 443 is required. Nginx is my most familiar tool so it’s a no-brainer for me.

The installation steps:

  • yum install nginx
  • systemctl enable nginx
  • Wrote a .conf file to proxy http packets to my web service in /etc/nginx/conf.d so it gets included to http block in /etc/nginx/nginx.conf.

Things I learned:

  • By default, Nginx has a timeout of 60 seconds when proxying packets. configs like proxy_send_timeout 1h, proxy_read_timeout 1h are useful.
  • For proxy_pass, localhost and 127.0.0.1 seems different. The port is stripped if I use localhost.
  • Slashs matter. For easier understanding, take a look at the pseudo conf, notice the difference in location name: 
      server {
      server_name 123.com;
      location /a/ {
          proxy_pass 127.0.0.1:1234;
      }

      location /a/ {
          proxy_pass 127.0.0.1:1234/;
      }
  }

    When a request to 123.com/a/b arrived, the fist location block would proxy it to 127.0.0.1:1234/a/b, while the second one would proxy it to 127.0.0.1:1234/b.

Make It a Systemd Service

The web service I’m hosting is a single-file excutable, I just drop it into /usr/local/bin which I learned is a goto place for binaries not installed with package managers, it’ll also make things easier on SELinux enabled machine, we’ll cover this later.

I need the web service to auto start on boot and auto restart if any failure happens, it seems the simplest way is create a new systemd service config for it.

Just write a file like this into /etc/systemd/system, the naming convention is NAME.service,

Description=DUCT

Wants=network.target
After=syslog.target network-online.target

[Service]
User=root
ExecStart=/bin/bash -c "/usr/local/bin/duct serve"
RestartSec=6s
Restart=on-failure
KillMode=process

[Install]
WantedBy=multi-user.target

The pitfall here is make sure the ExecStart is formatted as /bin/bash -c "command" event if it’s already an executable, otherwise It’ll keep dying without telling you why.

After the file is in place, enabling the service with systemctl enable NAME.service, this makes it start on boot, but does not start it immediately.

To start it immediately, do systemctl start NAME.service.

The above NAME.service used in the commands can be abbrevated to NAME.

Good to Go?

With DNS, Nginx and my web service set-up, I thought I was ready to go… But no, I can’t connect to my web service.

Dealing with SELinux

Though "The most familiar Linux OS for me is CentOS" I’ve never confronting SELinux before! It’s… annoying. But I appreciate the strictness so I decided to stick to it, not disabling it.

Where Is It

In one of the previous sections, I mentioned that placing executables in /usr/local/bin is helpful.

SELinux maintains context of files. For example, if I place my duct in like /home, it won’t work, the execution would be denied by SELinux, because according to the configured contexts, execution happening in /home is not allowed.

A lot of the default paths are pre-configured, and I like the idea to stick to vanilla settings, so I chose to place my executable to /usr/local/bin. (Actually, this is the first time I really google about where should stuff go, and found a good post)

If you are interested with how to mess with file contexts, here is a link to the documentation.

But one more thing had to be done, when we move something into /usr/local/bin, SELinux won’t know that thing is now in a valid place to be executed.

I ran restorecon -rv /usr/local/bin/duct to update it, so it’s now allowed to be execute there.

Connection Denied

What was happening is even if I was doing curl from the machine itself, it’s not proxied to the web service, instead, Nginx would throw 502 Bad Gateway at me.

This is very weird. I wen to take a look at /var/log/nginx/error.log, it was something like this:

[crit] 3684#0: *56 connect() to [::1]:1234 failed (13: Permission denied) while connecting to upstream, client: 123.123.123.123, server: sub.1234.com, request: “GET /duct/1 HTTP/1.1”, upstream: “http://[::1]:1234/1”, host: “sub.1234.com”

Okay, (13: Permission denied).

There’s such a file /var/log/audit/audit.log that contains logs of what the audit system done. I took a look, found something looks like this:

type=AVC msg=audit(1633183449.281:813): avc:  denied  { name_connect } for  pid=3684 comm="nginx" dest=9002 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:http_port_t:s0 tclass=tcp_socket permissive=0
type=SYSCALL msg=audit(1633183449.281:813): arch=c000003e syscall=42 success=no exit=-13 a0=12 a1=560edfc92af0 a2=10 a3=7fff7e08e13c items=0 ppid=3683 pid=3684 auid=4294967295 uid=989 gid=985 euid=989 suid=989 fsuid=989 egid=985 sgid=985 fsgid=985 tty=(none) ses=4294967295 comm="nginx" exe="/usr/sbin/nginx" subj=system_u:system_r:httpd_t:s0 key=(null)ARCH=x86_64 SYSCALL=connect AUID="unset" UID="nginx" GID="nginx" EUID="nginx" SUID="nginx" FSUID="nginx" EGID="nginx" SGID="nginx" FSGID="nginx"

What a gibberish. With ausearch -m avc -i we can get a more readable version:

type=PROCTITLE msg=audit(10/02/2021 22:04:09.281:814) : proctitle=nginx: worker process
type=SYSCALL msg=audit(10/02/2021 22:04:09.281:814) : arch=x86_64 syscall=connect success=no exit=EACCES(Permission denied) a0=0x12 a1=0x560edfc92b18 a2=0x1c a3=0x7fff7e08e13c items=0 ppid=3683 pid=3684 auid=unset uid=nginx gid=nginx euid=nginx suid=nginx fsuid=nginx egid=nginx sgid=nginx fsgid=nginx tty=(none) ses=unset comm=nginx exe=/usr/sbin/nginx subj=system_u:system_r:httpd_t:s0 key=(null)
type=AVC msg=audit(10/02/2021 22:04:09.281:814) : avc:  denied  { name_connect } for  pid=3684 comm=nginx dest=1234 scontext=system_u:system_r:httpd_t:s0 tcontext=system_u:object_r:port_t:s0 tclass=tcp_socket permissive=0
  • Nginx was doing a connect syscall, but has no permission to do it.

    syscall=connect success=no exit=EACCES(Permission denied)

  • The actual denied action was name_connect.

    avc: denied { name_connect }

  • Nginx is a httpd.

    exe=/usr/sbin/nginx subj=system_u:system_r:httpd_t:s0

  • The target is a normal tcp port, which is what my web service listens to.

    tcontext=system_u:object_r:port_t:s0

  • name_connect by http_d to a tcp port is not allowed.

    permissive=0

Great, but how do we know what does it allow?

We can do sesearch --allow -s httpd_t | grep name_connect to further investigate, the sesearch statement would output all rules which apply to action comes from a httpd and is an allow rule, then the grep statement filters lines contains name_connect.

The output should looks like:

...
allow httpd_t http_port_t:tcp_socket name_connect; [ httpd_can_network_relay ]:True
...

The line is saying that if httpd_can_network_relay is true, name_connect from a httpd to any port of http_port_t is allowed. There was no line starts with allow httpd_t port_t:tcp_socket.

So it’s much clear about what happened. So, to solve this, we need to enable the option httpd_can_network_relay first with setsebool -P httpd_can_network_relay on (-P means it does not revert after reboot)

Then we have 2 options:

  • Allow httpd do name_connect to all tcp port.
  • Add the port we want to http_port_t.

Comparing the two, it’s very easy to comes to a conclusion that, the second one, which allows 1 more tcp port to be connected, is much safer than the first one, which allows all tcp ports to be connected.

If we do semanage port -l | grep http_port_t, we can see all the ports belonging to http_port_t.

To add a port to http_port_t, do semanage port -a -t http_port_t -p tcp [PORT NUMBER].

The permission issue was fixed.

Firewalld

After all these done, it still did not work. What could still be wrong… after all these obstacles I overcame?

The good old firewall.

What was happening is that curl says No route to host when I told it to connect to my domain. The message sounds very much like a DNS issue, or like the machine is not exposed to internet… Hey, that means there’s no route to get to my domain, is it?

Firewalld can be adjusted with firewall-cmd, I did these to open the door:

  • firewall-cmd --list-all, from the output it seems that 80 (http) is not allowed
  • firewall-cmd --permanent --add-service=http
  • firewall-cmd --reload
  • firewall-cmd --permanent --list-services, now the output shows that http is enabled

Bingo! My web service is online.