04
Dec 20

Django + Postgres + nginx + uWSGI = ❤️

To date, I set up a production web service thrice. (Yeah, I am a full stack dev!)

The first time I did many mistakes. That website still works, I maintain and update it, but now it would be difficult to fix things, I would have to stop the service and have some quite stressful time re-configuring it.

The second time I did it more or less right, but the deployment happened in the corporate network of the company that I no longer work for, so that knowledge is now lost to me.

When I tried to do it the third time, I had to recall and google many things again. Now it works, so it is a good moment to write a note to myself in future.

The plan

Five steps to quick and easy deployment by Alex who likes to code:

  1. Install nginx, uwsgi, python3, and postgres. (no-op)
  2. Set up PostgreSQL.
  3. Check out the Django project and set it up.
  4. Configure uWSGI.
  5. Configure nginx.

Setting up prerequisites is literally one call to apt. Too easy, I cannot write anything else about that. Other parts are longer.

PostgreSQL

As user postgres, run psql. This will take us into the DBMS console. There, run the following:

CREATE DATABASE db-name;
CREATE USER db-user WITH PASSWORD 'password';
ALTER ROLE db-user SET client_encoding TO 'utf8';
ALTER ROLE db-user SET default_transaction_isolation
    TO 'read committed';
ALTER ROLE db-user SET timezone TO 'UTC';
GRANT ALL PRIVILEGES ON DATABASE db-name TO db-user;

Most commands above are quite self-explanatory. The ALTER ROLE ones are not mandatory but may be useful.

Now exit the DBMS console and proceed to the next part.

Django

This part is the longest, it is in fact the actual deployment of the project, other parts are just… environment.

  1. Switch to the user that will normally do whatever is required regarding the source code, e.g., updating the checkout, editing the code and project configs, etc. Let it be user project.
  2. Check out the project into the common location for websites, like /var/www on Debian. Let it be /var/www/project (we will need it later a few more times).
  3. Create the virtual Python environment (venv); modern versions of Python can do that themselves via python -m venv {directory name}, I like to have it in the venv directory directly within the Django project, so the full path would be /var/www/project/venv.
  4. Activate the venv and install Python requirements. Do not deactivate it for the next few steps.
  5. Edit the project settings. See remarks below for some ideas about this.
  6. Test that the project is configured properly by running manage.py migrate.
  7. Test even more that the project is configured properly by running manage.py runserver. Terminate the server after it starts normally.
  8. Create the Django superuser by running manage.py createsuperuser. We may deactivate the venv now.

Remarks on preparing the project for deployment

The web service always has some site-specific settings, and putting them under the version control would be a pain. I like the idea of moving these settings into the separate file from the very beginning. Create site_settings.py near the standard settings.py, cut and paste whatever is going to differ between hosts, and then add from .site_settings import * into settings.py. Add site_settings.py into .gitignore.

We definitely want SECRET_KEY, DEBUG, and DATABASES to stay in the site config.

uWSGI

As root, create /etc/uwsgi/apps-available/project.ini (the .ini extension is important!) and put this inside:

[uwsgi]
# The root directory of the Django project.
chdir  = /var/www/project
# The Python module within the project that creates
# the application.  It is already there, built into
# every Django project.
module = project.wsgi
# The venv that should be used.
home   = /var/www/project/venv

# Used to connect to the web server.
socket       = /var/www/project/socket
# uWSGI has plugins for different platforms.
# This may require to install something like
# uwsgi-plugin-python3.
plugins      = python37

# Not important :-D but still needed.
master       = true
processes    = 5
vacuum       = true

Create a link to the newly created config to enable the application:

ln -s /etc/uwsgi/apps-available/project.ini \
        /etc/uwsgi/apps-enabled/project.ini

Restart uwsgi and ensure that it runs with no errors.

nginx

As user project, create /var/www/project/project/uwsgi_params and put there this:

uwsgi_param QUERY_STRING $query_string;
uwsgi_param REQUEST_METHOD $request_method;
uwsgi_param CONTENT_TYPE $content_type;
uwsgi_param CONTENT_LENGTH $content_length;
uwsgi_param REQUEST_URI $request_uri;
uwsgi_param PATH_INFO $document_uri;
uwsgi_param DOCUMENT_ROOT $document_root;
uwsgi_param SERVER_PROTOCOL $server_protocol;
uwsgi_param REMOTE_ADDR $remote_addr;
uwsgi_param REMOTE_PORT $remote_port;
uwsgi_param SERVER_ADDR $server_addr;
uwsgi_param SERVER_PORT $server_port;
uwsgi_param SERVER_NAME $server_name;

(This is in fact the standard file that is shipped with the nginx distribution.)

Now switch to root, create /etc/nginx/sites-available/project and put this inside:

server {
    listen      80;
    server_name project.org;
    charset     utf-8;

    location /static/ {
        alias /var/www/project/static/;
    }

    location / {
        \# Connection to uWSGI.
        uwsgi_pass unix:/var/www/project/socket;
        include    /var/www/project/project/uwsgi_params;
    }
}

Create a link to the newly created config to enable the website:

 ln -s /etc/nginx/sites-available/project \
    /etc/nginx/sites-enabled/project

Restart nginx and ensure that it runs without errors.

That is it!

References


13
Nov 20

HiDPI support in Chromium for Wayland

It all started with this bug. The description sounded humble and harmless: the browser ignored some command line flag on Wayland. A screenshot was attached where it was clearly seen that Chromium (version 72 at that time, 2019 spring) did not respect the screen density and looked blurry on a HiDPI screen.

HiDPI literally means small pixels. It is hard to tell now what was the first HiDPI screen, but I assume that their wide recognition came around 2010 with Apple’s Retina displays. Ultra HD had been standardised in 2012, defining the minimum resolution and aspect ratio for what today is known informally as 4K—and 4K screens for laptops have pixels that are small enough to call it HiDPI. This Chromium issue, dated 2012, says that the Linux port lacks support for HiDPI while the Mac version has it already. On the other hand, HiDPI on Windows was tricky even in 2014.

‘That should be easy. Apparently it’s upscaled from low resolution. Wayland allows setting scale for the back buffers, likely you’ll have to add a single call somewhere in the window initialisation’, a colleague said.

Like many stories that begin this way, this turned out to be wrong. It was not so easy. Setting the buffer scale did the right thing indeed, but it was absolutely not enough. It turned out that support for HiDPI screens was entirely missing in our implementation of the Wayland client. On my way to the solution, I have found that scaling support in Wayland is non-trivial and sometimes confusing. Since I finished this work, I have been asked a few times about what happens there, so I thought that writing it all down in a post would be useful.

Background

Modern desktop environments usually allow configuring the scale of the display at global system level. This allows all standard controls and window decorations to be sized proportionally. For applications that use those standard controls, this is a happy end: everything will be scaled automatically. Those which prefer doing everything themselves have to get the current scale from the environment and adjust rendering.  Chromium does exactly that: inside it has a so-called device scale factor. This factor is applied equally to all sizes, locations, and when rendering images and fonts. No code has to bother ever. It works within this scaled coordinate system, known as device independent pixels, or DIP. The device scale factor can take fractional values like 1.5, but, because it is applied at the stage of rendering, the result looks nice. The system scale is used as default device scale factor, and the user can override it using the command line flag named --force-device-scale-factor. However, this is the very flag which did not work in the bug mentioned in the beginning of this story.

Note that for X11 the ‘natural’ scale is still the physical pixels.  Despite having the system-wide scale, the system talks to the application in pixels, not in DIP.  It is the application that is responsible to handle the scale properly. If it does not, it will look perfectly sharp, but its details will be perhaps too small for the naked eye.

However, Wayland does it a bit differently. The system scale there is respected by the compositor when pasting buffers rendered by clients. So, if some application has no idea about the system scale and renders itself normally, the compositor will upscale it.  This is what originally happened to Chromium: it simply drew itself at 100%, and that image was then stretched by the system compositor. Remember that the Wayland way is giving a buffer to each application and then compositing the screen from those buffers, so this approach of upscaling buffers rendered by applications is natural. The picture below shows what that looks like. The screenshot is taken on a HiDPI display, so in order to see the difference better, you may want to see the full version (click the picture to open).

What Chromium looked like when it did not set its back buffer scale

Firefox (left) vs. Chromium (right)

How do Wayland clients support HiDPI then?

Level 1. Basic support

Each physical output device is represented at the Wayland level by an object named output. This object has a special integer property named buffer scale that tells literally how many physical pixels are used to represent the single logical pixel. The application’s back buffer has that property too. If scales do not match, Wayland will simply scale the raster image, thus emulating the ‘normal DPI’ device for the application that is not aware of any buffer scales.

The first thing the window is supposed to do is to check the buffer scale of the output that it currently resides at, and to set the same value to its back buffer scale. This will basically make the application using all available physical pixels: as scales of the buffer and the output are the same, Wayland will not re-scale the image.

Back buffer scale is set but rendering is not aware of that

Chromium now renders sharp image but all details are half their normal size

The next thing is fixing the rendering so it would scale things to the right size.  Using the output buffer scale as default is a good choice: the result will be ‘normal size’.  For Chromium, this means simply setting the device scale factor to the output buffer scale.

Now Chromium looks right

All set now

The final bit is slightly trickier.  Wayland sends UI events in DIP, but expects the client to send surface bounds in physical pixels. That means that if we implement something like interactive resize of the window, we will also have to do some math to convert the units properly.

This is enough for the basic support.  The application will work well on a modern laptop with 4K display.  But what if more than a single display is connected, and they have different pixel density?

Level 2. Multiple displays

If there are several output devices present in the system, each one may have its own scale. This makes things more complicated, so a few improvements are needed.

First, the window wants to know that it has been moved to another device.  When that happens, the window will ask for the new buffer scale and update itself.

Second, there may be implementation-specific issues. For example, some Wayland servers initially put the new sub-surface (which is used for menus) onto the default output, even if its parent surface has been moved to another output.  This may cause weird changes of their scale during their initialisation.  In Chromium, we just made it so the sub-surface always takes its scale from the parent.

Level 3? Fractional scaling?

Not really. Fractional scaling is basically ‘non-even’ scales like 125%. The entire feature had been somewhat controversial when it had been announced, because of how rendering in Wayland is performed. Here, non-even scale inevitably uses raster operations which make the image blurry. However, all that is transparent to the applications. Nothing new has been introduced at the level of Wayland protocols.

Conclusion

Although this task was not as simple as we thought, in the end it turned out to be not too hard. Check the output scale, set the back buffer scale, scale the rendering, translate pixels to DIP and vice versa in certain points. Pretty straightforward, and if you are trying to do something related, I hope this post helps you.

The issue is that there are many implementations of Wayland servers out there, not all of them are consistent, and some of them have bugs. It is worth testing the solution on a few distinct Linux distributions and looking for discrepancies in behaviour.

Anyway, Chromium with native Wayland support has recently reached beta—and it supports HiDPI! There may be bugs too, but the basic support should work well. Try it, and let us know if something is not right.

Note: the Wayland support is so far experimental. To try it, you would need to launch chrome via the command line with two flags:
--enable-features=UseOzonePlatform
--ozone-platform=wayland


01
Jul 20

Hello world!

I am Alex, working on Chromium since 2013, Igalian since 2019, and chromie for a while now.

Chromie? Yes, that is what Chromium engineers use when referring to themselves. However, I did not know that until I joined Igalia, started contributing to the project more often than I used to before, and joined some chats where chromies communicate. I wish I knew when this cute moniker was born.

So, I have been working on Chromium for quite a few years now. The project is huge, it evolves as the world does, and it tries to lead the web and to be ahead of what people want. We serve the web, but we also shape it and make it better. This mission is exciting and never gets boring.

Throughout my whole career I had been working for companies where blogging about work implied supervision from the management and was generally frowned upon. Software engineers were supposed to write code, not blog posts. Here at Igalia it is all different; we work with open source for the public, and speaking aloud about what we do is part of that work.

This seems to be a perfect opportunity to get back into blogging.