Octocurious

Christina O'Donnell

How to Set up a GitLab Instance on an Existing Nginx Server

March 21, 2024

I’ve been running a Git daemon instance on my VPS for a few months and it’s working fairly well at hosting my repos. If you didn’t know, one cute feature of Git is that it can be run as a somewhat rudimentary daemon. By simply passing --daemon to it. This is fantastic as a quick way to get your git repos hosted on your own server or VPS. But if you find yourself longing for more features, such as a web interface, then GitLab is a reasonable alternative 1.

This post will cover the installation procedure on a ‘self-hosted’ VPS if you already have an Nginx web-server running. If you don’t then the installation procedure should be more straight-forward, simply following the official GitLab installation procedure.

Warning: This guide involves chaning things on your server that may not be easy to reverse. As always, ensure you have recent and tested backups before getting started.

Installing GitLab on Debian/Ubuntu #

You’ll want to start by fetching GitLab Community Edition. There’s a trap on their site where the default installer they provide is called GitLab-ee. This stands for Enterprise Edition which contains non-free code components which may be distasteful to some.

You can build it from scratch, and I’m sure there’s lots of fun going along that route, but I personally would rather have it up and running sooner than later.

You can download the free installer for many distributions here. 2

On Debian, you have to add a few dependencies then do the Yikes curlbash:

sudo apt-get update
sudo apt-get install -y curl openssh-server ca-certificates perl

curl https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/script.deb.sh | sudo bash

You can inspect this script for yourself.

# ...

main ()
{
  detect_os
  curl_check
  gpg_check
  detect_version_id

It starts by calling some simple functions that I’ve ommitted that just exit with an error if a dependency isn’t present.

apt-get update &> /dev/null

# Install the debian-archive-keyring package on debian systems so that
# apt-transport-https can be installed next
install_debian_keyring

apt-get install -y apt-transport-https &> /dev/null

It then ensures the APT cache is up to date, stores itself in the APT keyring, so Debian can authenticate the packages being downloaded. apt-transport-https is a library that allows APT to download over HTTPS.

  gpg_key_url="https://packages.gitlab.com/gitlab/gitlab-ce/gpgkey"
  apt_config_url="https://packages.gitlab.com/install/repositories/gitlab/gitlab-ce/config_file.list?os=${os}&dist=${dist}&source=script"

  apt_source_path="/etc/apt/sources.list.d/gitlab_gitlab-ce.list"
  gpg_keyring_path="/usr/share/keyrings/gitlab_gitlab-ce-archive-keyring.gpg"

  # create an apt config file for this repository
  curl -sSf "${apt_config_url}" > $apt_source_path

  # import the gpg key
  curl -fsSL "${gpg_key_url}" | gpg --dearmor > ${gpg_keyring_path}

  # update apt on this system
  apt-get update &> /dev/null

  echo "The repository is setup! You can now install packages."
}

Here it’s simply adding two files: /etc/apt/sources.list.d/gitlab_gitlab-ce.list and /share/keyrings/gitlab_gitlab-ce-archive-keyring.gpg.

I’ve removed the error checking and a lot of other supporting code that isn’t needed to understand what it’s doing.

So code simply sets up APT to download GitLab. The final install is to run:

sudo apt install gitlab-ce

If you get a download without an error then you’ve successfully installed GitLab! If not then you should go through the curlbash script to check if everything succeeded.

Basic configuration #

The next step is to take down my current Git daemon to replace it with GitLab. We’ll start with HTTP only on a standalone system to start with and move on to making it work over HTTPS and then get it to play nice with my existing Nginx server using a ‘reverse proxy’.

The main file that we’ll be editing to get this to work is /etc/gitlab/gitlab.rb. If you open the file that GitLab has installed then you’ll see 3000 or so lines of commented out Ruby. This is the primary configuration file that you need to touch to get GitLab to work the way you want.

We only need to change one line to get GitLab’s web view working: external_url. Set this to your domain name with the protocol as HTTP (since we haven’t got certificates set up yet). Search for the line that says external_url under “GitLab URL” and set it to your domain name.

external_url 'http://git.mutix.org'

There’s a big comment above each variable with links to documentation and explanation of what will and won’t work as values.

make sure that your DNS is set up to point ‘A’ records of your domain to the server gitlab will be running on.

Now I need to take down my existing Git-daemon setup to proceed. If you don’t have an existing Git-daemon set up on your server, then you can safely skip the next section.

Taking down the existing Git-daemon (optional) #

My git --daemon setup had 3 components:

  1. Nginx daemon to recieve smart HTTP requests
  2. Git-daemon to serve up the Git files
  3. And, fastcgi to authenticate users

This is childs-play compared to the complexity of GitLab which has over a dozen components. To get everything working properly, we’ll have to go one step at a time, changing only the required components.

I need to start by (after backing up) disable my existing Git-daemon setup. This is done simply with systemctl disable.

sudo systemctl disable --now git-daemon
sudo systemctl disable --now fastcgi

For Nginx things need to be handled a little differently. since I have a live server on there, I simply stopped it for now. We’ll come back to getting both Nginx instances to work together. For now we need to temporarily disable while we set up GitLab.

sudo systemctl stop nginx

Reconfiguring and testing #

Now we can run sudo gitlab-ctl reconfigure and if that worked then you’ll be able to go to your domain name above to see if you can see gitlab’s login page.

If not then take a look at the log /var/log/gitlab/nginx/error.log and the system journal.

Generating a certificate with letsencrypt #

Scroll down to the “Let’s Encrypt” section, set letsencrypt[’enable’] to true, add your email address to the conact_emails list and reconfigure.

################################################################################
# Let's Encrypt integration
################################################################################
letsencrypt['enable'] = true
# This should be an array of email addresses to add as conacts
letsencrypt['contact_emails'] = ['admin@mutix.org']
letsencrypt['group'] = 'root'
letsencrypt['key_size'] = 2048
letsencrypt['owner'] = 'root'
letsencrypt['wwwroot'] = '/var/opt/gitlab/nginx/www'

If this succeeds then GitLab has registered its own certificate and we can go back and set the external_url as HTTPS:

external_url 'https://git.mutix.org'

Now it should seemlessly work with SSL!

The final step before we’re ready to use GitLab is to get our existing Nginx server (which is currently offline) to serve other webpages while GitLab is running. This can either be done through means of a ‘reverse proxy’, or by telling GitLab to serve its pages directly on our existing Nginx server.

The reverse-proxy route is somewhat fiddly, but should work better once GitLab is up and running, allowing GitLab updates to change the Nginx server settings. Alternatively, you can skip to the next section which covers how to set up a non-bundled web-server.

Setting up reverse proxy using Nginx #

A reverse proxy is a type of server or service that sits in front of one or more web servers, intercepting requests from clients (such as web browsers) before they reach the backend servers. It acts as an intermediary for requests from clients seeking resources from the servers. The reverse proxy forwards these requests to the appropriate backend server and then returns the server’s response to the client, making it appear as though the proxy itself originated the response. For more information, Nginx has a wiki page on the topic.

You’ll want to save a copy of your configuration because this could get hairy!

First you need a port number that’s not being used for anything else. Anything above 1000 but under 65000 is likely to work without clashing. I arbitrarily chose 6236.

server {
        server_name git.mutix.org;
        #listen 80;
        listen 443 ssl;
        location / {
                proxy_pass http://git.mutix.org:6236;
                proxy_set_header Host $host;
                proxy_set_header X-Real-IP $remote_addr;
                proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
                proxy_set_header X-Forwarded-Proto $scheme;
                # proxy_set_header X-Forwarded-Ssl on;
        }
        ssl_certificate /etc/gitlab/ssl/git.mutix.org.crt
        ssl_certificate_key /etc/gitlab/ssl/git.mutix.org.key
        # ssl_certificate /etc/letsencrypt/live/git.mutix.org/fullchain.pem; # managed by Certbot
        # ssl_certificate_key /etc/letsencrypt/live/git.mutix.org/privkey.pem; # managed by Certbot
        # include /etc/letsencrypt/options-ssl-nginx.conf; # managed by Certbot
        # ssl_dhparam /etc/letsencrypt/ssl-dhparams.pem; # managed by Certbot
}

################################################################################
## GitLab NGINX
##! Docs: https://docs.gitlab.com/omnibus/settings/nginx.html
################################################################################

nginx['enable'] = true

nginx['listen_port'] = 6236

nginx['listen_https'] = false

nginx['proxy_protocol'] = true

nginx['real_ip_trusted_addresses'] = [ "127.0.0.0/8", "YOUR_PUBLIC_IP/32"]

Now reconfigure GitLab again and you won’t be able to access it anymore. You’ll need to start up Nginx again.

https://docs.gitlab.com/omnibus/settings/nginx.html#configuring-proxy-protocol

Serving GitLab and your site from a single Nginx daemon #

Getting the reverse proxy working is not always the easiest option. Thankfully GitLab provides an easier way to configure Nginx: Serving GitLab pages from a single root Nginx instance. To get this working, first disable GitLab’s Nginx:

nginx['enable'] = true

Then you need to download the default configuration into /etc/nginx/sites-available/:

sudo wget https://gitlab.com/gitlab-org/gitlab/-/raw/master/lib/support/nginx/gitlab-ssl

Other server definitions are available on GitLab’s GitLab.

The first think you’ll want to do is change every instance of YOUR_SERVER_FQDN with your domain name.

After that, you’ll want to point the ssl_certificate and key at the nginx generated certificate:

## Strong SSL Security
## https://raymii.org/s/tutorials/Strong_SSL_Security_On_nginx.html & https://cipherli.st/
ssl_certificate /etc/gitlab/ssl/git.mutix.org.crt;
ssl_certificate_key /etc/gitlab/ssl/git.mutix.org.key;

Adjust the paths to point at the ‘Omnibus’ directory. (Omnibus is the curlbash installer that we used to install GitLab.)

upstream gitlab-workhorse {
  # GitLab socket file,
  # for Omnibus this would be: unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket
  server unix:/var/opt/gitlab/gitlab-workhorse/sockets/socket fail_timeout=0;
}

location ~ ^/(404|422|500|502|503)\.html$ {
  # Location to the GitLab's public directory,
  # for Omnibus this would be: /opt/gitlab/embedded/service/gitlab-rails/public
  root /opt/gitlab/embedded/service/gitlab-rails/public;
  internal;
}

That should be it. Don’t forget to reload Nginx, and reconfigure GitLab!

sudo systemctl reload nginx
sudo gitlab-ctl reconfigure

Logging in to the root account #

Now you’ve got a basic GitLab playing nice with another site on the same server, the next step (which is much more fun) is to make it yours. To log in to the root account, you need the root password, which is stored in /etc/gitlab/initial_root_password.

And then visit the login page, enter the username root, and the password provided. And you’re good to go.

Note that if you wait for longer than 24 hours then GitLab automatically deletes that path. In that case you’ll have to use the GitLab Rake interface:

sudo gitlab-rake "gitlab:password:reset"

It took a few seconds to give the prompt but once it does, enter the username root and a new strong password.

Conclusion #

If all of the above worked then you’ll have your own GitLab server running that you’re free to configure and start working with.

Troubleshooting #

If you hit any errors, the first place to look are the logs. Some logs go to the system log (normally accessible via journalctl), while others end up in /var/log. For our purposes, the main log files of interest are /var/log/nginx/error.log and /var/log/gitlab/nginx/error.log. These contain information for the root Nginx and GitLab Nginx instances respectively.

DNS #

If you can’t see any error page on your domain over HTTP with your main site offline, but GitLab appears to be running then you’ll want to check that your DNS records are pointing to your domain. You can test this out with the dig tool:

$ sudo apt install bind9-dnsutils
$ dig git.mutix.org A

; <<>> DiG 9.16.44-Debian <<>> git.mutix.org A
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 1207
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 0, ADDITIONAL: 1

;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 1232
;; QUESTION SECTION:
;git.mutix.org.			IN	A

;; ANSWER SECTION:
git.mutix.org.		300	IN	A	194.163.141.236

;; Query time: 12 msec
;; SERVER: 161.97.189.52#53(161.97.189.52)
;; WHEN: Sat Mar 23 09:52:34 CET 2024
;; MSG SIZE  rcvd: 58

If you don’t get anything in the answer section then double check your DNS records. Sometimes you just need to wait for it to propagate.

Let’s Encrypt #

If HTTP works but Let’s Encrypt doesn’t seem to be working then try disabling all but the essential Let’s Encrypt settings:

letsencrypt['enable'] = true
letsencrypt['contact_emails'] = ['admin@mutix.org'] # This should be an array of email addresses to add as contacts

This is all I needed to get it working. If that doesn’t work, then there’s a troubleshooting page in the GitLab documentation.

502 Bad Gateway with the reverse proxy method #

This tends to occur when the reverse proxy is having troube connecting to your existing Nginx configuration. The first thing to check are the following two log files:

2024/03/22 19:51:41 [error] 137377#137377: *30 upstream prematurely closed connection while reading response header from upstream, client: 81.147.82.218, server: git.mutix.org, request: "GET /favicon.ico HTTP/1.1", upstream: "http://127.0.0.1:5189/favicon.ico", host: "git.mutix.org", referrer: "https://git.mutix.org/users/sign_in"


2024/03/22 19:51:41 [error] 155413#0: *7 broken header: "GET /favicon.ico HTTP/1.0" while reading PROXY protocol, client: 127.0.0.1, server: 0.0.0.0:5189

Here GitLab’s Nginx is the ‘upstream’ of the reverse proxy. Here’s what’s happening in this example:

  1. The user is accessing https://git.mutix.org/.
  2. From the first log message, we can see the root Nginx is recieving this request and redirecting it to the upstream GitLab instance.
  3. Based on the second log message, GitLab’s Nginx is recieving the GET request, but saying that the header is broken.

Unfortunately, even with debug output, I wasn’t able to get anything more helpful than this. You’ll have to just fiddle with the headers for a bit until it works.

If you can’t get the reverse proxy to work then you can always try the other method of hosting GitLab from your root Nginx.

Sources #

https://tecadmin.net/secure-gitlab-with-lets-encrypt-ssl/ https://docs.gitlab.com/omnibus/settings/ssl/

  1. Source Hut and Gitea, both of them free and open source. ↩︎

  2. Notably no Guix package unfortunately. ↩︎