Poudriere after a time

Poudriere is not perfect but it is brilliant and a powerful tool.  I think that it may have already become my primary ports build method as for the most part it does what I expect it to do.  With poudriere and pkg, used together and only my poudriere repo enabled, I can be fairly certain that what I install will be built how I wish it to be built.  I can go to any individual poudriere port directory in /usr/local/poudriere/ports/default to modify or manipulate the port build as I desire.  I have my minetest build adjustments copied to its poudriere path, the modified Makefile and hide or remove a patch in its files.  Whether something succeeds to build or not only affects what I will later install with pkg, and not necessarily cause other breakage or removals.

The testport function is very helpful and may one day assist me with porting something to FreeBSD.  This also helps to detect mistakes in the ports tree members we already have.  This capability can be enabled for bulk builds as well but sometimes requires skirting a problematic port.  One recent issue is a sort of build system regression which may only affect a subset of ports and as such it is not something I can solve.  The only way around it is to successfully build those ports which fail while ignoring the problem, then once built and in the poudriere cache the test mode can be reenabled.  Other ports which poudriere points out 'portlint' type errors are easily fixed, as most involve errors in the Makefile itself such as the preferred order of lines or missing dependencies, or typos.

I have been keeping an updated installed-pkgs list since this seems to be one of the few aspects of many of the other ports-mgmt tools that poudriere does not automate directly.  It is a very simple thing to do, to create this list, and even though poudriere chooses its own preferred build order, I still organize my list alphabetically for my own ease of use.  Below is how I periodically create the file.

pkg origin | sort -u -d > installed-pkgs

This works quite well.  One may believe that the pkg origin command would deliver a concise list but it seems to include a number of duplicates.  Even if it was my own mistake causing the duplicates, it is likely a good idea to guard against them anyway.

There is one flaw of poudriere which if addressed would cover one further step I need to take for that list to be used without problems.  My system contains a number of ports which have since vanished from the tree.  I would have expected that placing their port origins into the blacklist file would solve the issue but it does not.  The peculiar thing about the blacklist file is that poudriere uses it to ignore buildable ports listed but still fails on those ports if they do not exist in the tree.  If the blacklist also filtered those ports from the set of files in the bulk build list given to it, we could add them to the blacklist once and forget about them with successive lists created with the instruction above.  This small flaw was not caused by adding anything else to the poudriere bulkbuild command, such as the -t which would cause other testing of each port, because I did not use it.  The only other possible cause is that my poudriere config file enables portlint scrutiny which I had not enabled, however, I still believe the blacklist ought to first filter the bulkbuild list as just described.  Instead of the desired expected function of the blacklist, I have to find and remove those specific presently non-existent ports from the buildlist.  Maybe I had built and installed something which had a directory in the ports tree (perhaps a new unsubmitted port), so pkg gains the registration of the port, then an update (such as fetch extract) or other rewrite of the ports tree causes it to vanish.  It might be speedier to add a port origin to avoid in the blacklist file than to go to the parent Makefile to add or remove a directory.  The point being, that the blacklist ought to be a universal exclude.  Below, when those port origins are not removed from the installed-pkgs file is an example output and its failure to build due to the flaw:

root@ichigo:~ # poudriere bulk -f ~/installed-pkgs -j 12amd64
[00:00:00] Creating the reference jail... done
[00:00:01] Mounting system devices for 12amd64-default
[00:00:01] Mounting ports/packages/distfiles
[00:00:01] Using packages from previously failed build: /usr/local/poudriere/data/packages/12amd64-default/.building
[00:00:01] Mounting ccache from: /var/cache/ccache
[00:00:01] Mounting packages from: /usr/local/poudriere/data/packages/12amd64-default
[00:00:01] Copying /var/db/ports from: /var/db/ports
[00:00:01] Appending to make.conf: /etc/make.conf
[00:00:01] Warning: Blacklisting (from /usr/local/etc/poudriere.d/12amd64-blacklist): sysutils/pcbsd-libsh
[00:00:01] Warning: Blacklisting (from /usr/local/etc/poudriere.d/12amd64-blacklist): x11-fonts/font-bitstream-speedo
[00:00:01] Warning: Blacklisting (from /usr/local/etc/poudriere.d/12amd64-blacklist): x11-fonts/fontconfig-reference
[00:00:01] Warning: Blacklisting (from /usr/local/etc/poudriere.d/12amd64-blacklist): x11-wm/fvwm-crystal
/etc/resolv.conf -> /usr/local/poudriere/data/.m/12amd64-default/ref/etc/resolv.conf
[00:00:01] Starting jail 12amd64-default
[00:00:01] Copying host static ccache from /usr/local/bin/ccache
[00:00:03] Logs: /usr/local/poudriere/data/logs/bulk/12amd64-default/2020-10-19_00h09m04s
[00:00:03] Loading MOVED for /usr/local/poudriere/data/.m/12amd64-default/ref/usr/ports<
[00:00:05] Ports supports: FLAVORS SELECTED_OPTIONS
[00:00:05] Gathering ports metadata
[00:00:05] Error: MOVED: x11-fonts/font-bitstream-speedo 2019-10-15 Deprecatd upstream
[00:00:05] Error: MOVED: x11-fonts/fontconfig-reference 2020-09-19 Remove outdated reference documentation
[00:00:05] Error: MOVED: x11-wm/fvwm-crystal 2020-09-19 Has expired: Uses deprecated version of python
[00:00:05] Error: Fatal errors encountered gathering initial ports metadata
[00:00:05] Cleaning up
12amd64-default: removed
12amd64-default-n: removed
[00:00:05] Unmounting file systems
root@ichigo:~ #

I created a script to iterate through the config of each port in the buildlist outside of the poudriere jail.  As you may recall, I setup the port config directory and the /etc/make.conf as symbolic links for poudriere, so any changes I would make to port options or the make.conf will affect any other ports builds including poudriere.  The make.conf file has become quite extensive, and will be mentioned in another blog post.  Each port config option has a keyword which I may add to OPTIONS_UNSET+= or OPTIONS_SET+= in the make.conf.  Many of these keywords are unique but a lot are used across multiple ports, sometimes with the same purpose, sometimes not, and sometimes the same purpose is triggered by multiple keywords.

When I use poudriere to build my ports, I am primarily using the content of my make.conf file to configure them, because the last step is another script which removes the saved configuration options for each port in the buildlist.  This technique works rather well, except when an option causes a cascade of failed or skipped ports builds.  The other wrinkle with this, is when two ports share some options but one of them uses all tick boxes and the other uses radio buttons.  This is easy to fix but it takes a moment to discover the issue and then code the exception into the make.conf.

Except for the possible build system flaw, using either -t with bulkbuild, or testport for individual ports will help triage problems.  With nearly 2000 ports listed as installed, it takes a LONG time to manually check the options for conflicts, as well as deciding which keywords need to be added into the make.conf before sending it to poudriere.  The config script helps with simply iterating through the list, taking away the need to repeatedly type the command make -C /path/to/port/origin config but I still have to verify that no new keywords affect my overall choices.  The config removal script works without prompting through all ports in the list and takes a bit of time though I can do other things until it finishes.

I choose not to use pulseaudio if possible, and rather have alsa or oss or both.  Other port option choices take a bit more investigation about which one to prefer, such as if ports are identical other than license, or author's philosophy, or if one is lighter or more efficient or more capable, or if any is a dependency for a larger percentage of other ports.  The default port config options are nice but it is a one-size-fits-all approach and tends not to favor something other than oss, or may include nVidia enhancements which I cannot use, or is not adapted for my processor or gpu.  I am likely not alone as an admin of my own system or a FreeBSD user, in that both tend to prefer a greater level of control, or we'd be using something else.

The combination of poudriere, no saved port configs, and a rather large, detailed make.conf (once it and the buildlist are mostly finalized) mean that I can have semi-regular updates to the ports I use with few surprises.  I can set the job to run while I am asleep and much of the time have it nearly finished before I wake.  If something doesn't build, the older pkg remains in cache, and pkg upgrade will not bite me.  Maybe eventually I will investigate the poudriere daemon option or using a cron job to get things a bit more automated, but this is how I relate to poudriere at the moment.  It is a reliable tool I can keep using.