Pinning: Apt’s Best Kept Secret

I have recently discovered a feature of the apt(8) package manager that smooths over almost all of the annoyances that I've had with point-release distributions (Debian, Ubuntu, etc.). apt(8) provides a simple and effective way to install newer versions of packages than the ones included in the release version that you are running: pinning.[1]

First, a general observation about package versions: For most packages installed on a Linux system, the fact that they aren't the latest version is not very important. Ubuntu and Debian backport security and bug fixes to the supported versions of software for their major releases. This means you may miss out on some new features, but theoretically shouldn't be forced to live with any serious bugs or security vulnerabilities. It is only a small number of packages for which it is important to get the latest and greatest version. What those packages are is going to be pretty personal. For me, it's mpv(1), yt-dlp(1) and nvim(1). If you're into gaming, graphics drivers will probably make the list.

With this in mind, pinning in apt(8) is a feature that lets you control what repository apt(8) uses to install specific packages. There are two configuration files that we will be managing here: /etc/apt/sources.list and /etc/apt/preferences. The first of these files lists the repositories that apt(8) will pull packages from, and the second allows you to set priorities for these repositories, either on a package by package basis, or in general.

Let's consider getting an up-to-date version of yt-dlp(1) on Ubuntu 22.04 LTS as a specific example. Recently, YouTube pushed out an API update that broke this program, and the authors of yt-dlp(1) pushed an update to unbreak it. This updated package is not (or at least was not) available in the standard repositories for 22.04. [2] However, it is available in the repositories for the upcoming Ubuntu 23.04, Lunar Lobster. So let's configure apt(8) to install Lunar Lobster's version of yt-dlp(1) onto a machine running Jammy Jellyfish.

Adding new Repositories

First, we'll perform a full update of the packages installed on our system. We're going to attempt to do an update later to verify that we have configured our preferences correctly, so we want to make sure that we're fully up to date going in [3],

                sudo apt update
                sudo apt upgrade
            

Now, we need to update our /etc/apt/sources.list to add the repositories for Lunar Lobster. This is as simple as adding the following line to the file,

deb http://us.archive.ubuntu.com/ubuntu lunar main restricted universe multiverse

We now have added the Lunar repositories to apt(8). To verify this, run another update, and then use apt-cache(8) to verify that lunar is available. You should see a long list of the repositories that apt(8) is configured to interact with, which will include Lunar's.

                sudo apt update
                sudo apt-cache policy
...
500 http://us.archive.ubuntu.com/ubuntu lunar/multiverse amd64 Packages
    release v=23.04,o=Ubuntu,a=lunar,n=lunar,l=Ubuntu,c=multiverse,b=amd64
    origin us.archive.ubuntu.com
500 http://us.archive.ubuntu.com/ubuntu lunar/universe amd64 Packages
    release v=23.04,o=Ubuntu,a=lunar,n=lunar,l=Ubuntu,c=universe,b=amd64
    origin us.archive.ubuntu.com
500 http://us.archive.ubuntu.com/ubuntu lunar/restricted amd64 Packages
    release v=23.04,o=Ubuntu,a=lunar,n=lunar,l=Ubuntu,c=restricted,b=amd64
    origin us.archive.ubuntu.com
500 http://us.archive.ubuntu.com/ubuntu lunar/main amd64 Packages
    release v=23.04,o=Ubuntu,a=lunar,n=lunar,l=Ubuntu,c=main,b=amd64
    origin us.archive.ubuntu.com
...

The number in front of the repositories is the priority, which apt(8) uses to determine where to get packages when installing or updating. By default, you should see 500 used both here, and for Jammy's repositories. Generally speaking, higher numbers mean higher priority, though the details are a bit more complicated than that (we'll get there later).

The upshot of this default configuration is that, because lunar and jammy both have the same priority, apt(8) will prefer to install the most up-to-date package available from either of them. In practice, this means that lunar will be the default repository for your system. You can verify this by starting an apt upgrade; it will attempt to upgrade nearly every package on your system, even though we ensured it was fully up-to-date before getting here. Abort the upgrade--we aren't trying to update to a different version of Ubuntu right now.

Setting Repository Priorities

This behavior is great if it is your intent to upgrade to Lunar Lobster, but isn't so great if you only want to install a couple of packages from there. So, we need to adjust the priorities such that Jammy's repositories are preferred over Lunar's for general usage.

The way that you set the priority for repositories is by either creating an /etc/apt/preferences file, or adding files to /etc/apt/preferences.d, and specifying priorities in there. We'll go the single-file route. Create (if it doesn't already exist) /etc/apt/preferences and place the following inside it,

Package: *
Pin: release n=jammy
Pin-Priority: 700
 
Package: *
Pin: release n=lunar
Pin-Priority: -10

and then run apt update again to load the new preferences. If you attempt an apt upgrade after this, it shouldn't try to install any updates from the lunar repositories.

The first part of each section of this file specifies the packages that the configuration applies to. You can configure this on a package by package basis, or use *, as we do here, to apply the settings to all packages. If you did want to do this for a specific package, simply replace the * with the package name.

Then, we specify the repository using Pin: release n=x. If you look back at the output from apt-cache policy, you'll see that each repository has a couple of parameters listed, an n, a, o, etc. You can use any of these to specify which repository or repositories you are referring to with this setting. a is the most specific, so a=jammy will refer specifically to the jammy repository, but not jammy-updates or jammy-security, whereas n=jammy will collectively reference all three.

Finally, the Pin-Priority sets the priority of the specified repositories for the specific packages. The general rule is that the highest priority wins when trying to install or update, but it's actually a bit more complicated than that. The details are in the manual (apt_preferences(5)), but I'll repeat them here. Note that P refers to the priority number.

P >= 1000
   causes a version to be installed even if this constitutes a
   downgrade of the package


990 <= P > 1000
   causes a version to be installed even if it does not come from
   the target release, unless the installed version is more recent


500 <= P > 990
   causes a version to be installed unless there is a version
   available belonging to the target release or the installed
   version is more recent


100 <= P > 500
   causes a version to be installed unless there is a version
   available belonging to some other distribution or the
   installed version is more recent


0 <P > 100
   causes a version to be installed only if there is no
   installed version of the package


P < 0
   prevents the version from being installed


P = 0
   has undefined behavior, do not use it.

We'll discuss what a "target release" is next, but first some general comments. Priorities between 500 and 989 are the ones that behave as you probably expect: if all your repositories are in this range, the package will be installed and/or updated from the highest priority repository that has it. If there's a tie, the repository with the newest version wins. You will never downgrade a package--so if one is installed from a lower priority repository it won't get replaced by an older version in a higher priority repository.

The other priority ranges have special properties. If you set a priority over 1000, this will cause packages from that repository to be used, even if it results in a downgrade. It isn't often that this is what you want, so you want to generally stay away from this. Priorities below 0 will cause packages to never be installed, unless you manually set the repository as a target.

In this example, we've set the package priority for lunar to -10, which means that packages will never get installed from there. If we try to install a package that is in lunar, but not in jammy, the install will fail. If you were to instead set lunar to a priority of 600, then jammy will be the default source for packages, but if a package is not in jammy, lunar will be checked for it. I find this useful in Debian, where stable is sometimes missing things that you may want to install, but haven't needed it in Ubuntu as of yet.

Installing a Package from a Specific Repository

We have blocked installing packages from lunar by default. If we want to install something from there, we can force it by setting lunar as the target repository when we call apt(8). As an example, at the time of this writing the yt-dlp(1) package is on version 2022.04.08 in the jammy repositories. So if we install it, we'll see,

            sudo apt install yt-dlp
            yt-dlp--version
            2022.04.08
        

This version is actually broken now, because YouTube has updated its API and so trying to use it will result in an error,

            yt-dlp https://www.youtube.com/watch?v=NQ5uD5x8vzg
[youtube] NQ5uD5x8vzg: Downloading webpage
[youtube] NQ5uD5x8vzg: Downloading android player API JSON ERROR:
[youtube] NQ5uD5x8vzg: Unable to extract uploader id; please report
this issue on https://github.com/yt-dlp/yt-dlp/issues?q= ,
filling out the appropriate issue template. Confirm you are
on the latest version using yt-dlp -U 
        

However, we can forcibly install a newer version from lunar by using the -t flag in apt(8) to specify an install target repository,

            sudo apt install -t lunar yt-dlp
            yt-dlp --version
            2023.02.17
            yt-dlp https://www.youtube.com/watch?v=NQ5uD5x8vzg
            actually works now!
        

By specifying the target, we override the block on using packages from lunar. Further, this package will now be updated based on the repository it was installed from, so when a new version is pushed to lunar, apt(8) will download it without us needing to do anything special.

There is one caveat to this. If the package you install has any dependencies, these may be satisfied from lunar if necessary. This means that any other packages that share this dependency will need to be updated to their lunar versions too. If you later attempt to install a package that depends on an older version of one of these dependencies, apt(8) will fail. This is because the installed package will come from the jammy repositories, and the dependencies installed on your system will be the newer ones from lunar. Unless jammy is set with a priority greater than 1000, apt(8) won't downgrade these dependencies to match the package you're installing, and so it won't be able to install the package. Should this happen, you need to install the version of this package from lunar instead, using the -t lunar flag in the install command.

Rolling Everything Back

If you decide that you regret everything and want to roll back to the default jammy versions of all of your packages, apt(8) can do that, and it probably won't break your system. All you need to do is bump the jammy entry in your /etc/apt/preferences file to have a priority of over 1000, and apt(8) will force all the packages it can into compliance. This may result in some packages being uninstalled, if apt(8) can't make it work out.

More specifically, update /etc/apt/preferences so that it contains,

Package: *
Pin: release n=jammy
Pin-Priority: 1100
 
Package: *
Pin: release n=lunar
Pin-Priority: -10

and then run,

            sudo apt update
            sudo apt upgrade
        

This will cause apt(8) to downgrade packages as necessary to ensure that they all are coming from the jammy repositories.

Conclusion

Pinning packages with apt(8) is an incredibly useful feature, and has changed the way that I look at "stable" distributions like Ubuntu LTS and Debian. Using this feature, you can have a stable base system, without losing access to new versions of specific software that must be kept up to date, or without missing out on the latest features in some userland applications. In a sense, it gets you the best of both worlds: rolling release for selective userspace applications on top of a stable base system.

I used Arch Linux for over a decade before I learned about pinning, and now I'm happily running Debian as my main Linux distribution. Selectively installed packages from the unstable branch let me me have the latest toys, but I don't have to deal with the instability of running the core system components under a rolling release model.

apt(8) converted me away from Arch Linux--I just had to learn how to actually use it first.