Remote poudriere build

What I had forgot to include in a previous blog post was how I managed to successfully setup poudriere on atlas for use with remote package serving to ichigo.  I initially followed a Vermaden blog post Simple FreeBSD poudriere harvester guide which helped me get the majority of the setup configured, adjusted.  I attempted to leave a comment on the blog post but for whatever reason it failed, so I am going to provide that here as well.  The process is not difficult but there is at least one detail left out which could be discernable to some but it took a little for me to discover.

Vermaden does a very good job for the community to provide insights and guides and is part why I chose to write my own blog as well, beginning years ago.  It became obvious to me that writing about my experiences on FreeBSD would help me to remember and help others to find their way more easily.  I would recommend to any FreeBSD user to do the same for at least the same or similar reasons, it may seem a no-brainer.

The essentials of Vermaden's blog, what I needed I highlighted:

The /boot/loader.conf file.

# cat /boot/loader.conf
security.bsd.allow_destructive_dtrace=0
kern.geom.label.disk_ident.enable=0
kern.geom.label.gptid.enable=0
cryptodev_load=YES
zfs_load=YES

The /etc/rc.conf file.

# cat /etc/rc.conf
  clear_tmp_enable=YES
  syslogd_flags="-ss"
  sendmail_enable=NONE
  hostname=fbsdpr
  ifconfig_em0="inet 10.0.10.123/24"
  defaultrouter="10.0.10.1"
  sshd_enable=YES
  dumpdev=AUTO
  zfs_enable=YES
  nginx_enable=YES
  memcached_enable=YES
  memcached_flags="-l localhost -m 8192"

The /etc/sysctl.conf file.

# cat /etc/sysctl.conf
vfs.zfs.min_auto_ashift=12

Those lines will permit a web browser such as firefox to view the progress and results of the current poudriere bulk build, and permit ssh including access to X windows applications via ssh -X and sftp for any LAN file transfer needs.  And then I add the packages or make sure these few are installed.

Now we can add needed packages.

# env ASSUME_ALWAYS_YES=yes \
    pkg install -y \
      poudriere \
      portmaster \
      screen \
      tmux \
      zsh \
      beadm \
      lsblk \
      nginx \
      git-lite \
      htop \
      tree \
      ccache-memcached \
      memcached \
      groff

Although when I read the blog post I believe I already had those installed and some other things he mentions.

The most important step that I needed was for the pkg signing which is a series of steps, each command as root:

1. mkdir -p /usr/local/etc/ssl/keys /usr/local/etc/ssl/certs /usr/ports/distfiles
2. chmod 0600 /usr/local/etc/ssl/keys
3. openssl genrsa -out /usr/local/etc/ssl/keys/poudriere.key 4096
4. openssl rsa -in /usr/local/etc/ssl/keys/poudriere.key -pubout -out /usr/local/etc/ssl/certs/poudriere.cert

I wasn't sure that mkdir would create multiple paths in one command, and I already had /usr/ports/distfiles, so I did them individually but testing a moment ago I now know I could have just copied that line entirely.  Those steps above are perfect for the poudriere build machine but a detail that is essential here which can be done now or anytime before trying to use the signed pkgs, is to copy the /usr/local/etc/ssl/certs/poudriere.cert to the client box(es).  This is one thing that is missing and tripped me up, surely it is assumed and obvious but not to me right then.

Since I already had poudriere running previously for use on the same machine, a poudriere.conf already existed.  I edited the file to be sure to add and revise as needed the lines he provided:

ZPOOL=zroot
BASEFS=/usr/local/poudriere
ZROOTFS=/dev/null
#FREEBSD_HOST=ftp://ftp.freebsd.org
POUDRIERE_DATA=/usr/local/poudriere/data
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
PKG_REPO_SIGNING_KEY=/usr/local/etc/ssl/keys/poudriere.key
URL_BASE=http://172.16.0.12
USE_TMPFS=all
TMPFS_LIMIT=5
MAX_MEMORY=4
MAX_FILES=2048
DISTFILES_CACHE=/usr/ports/distfiles
KEEP_OLD_PACKAGES=yes
KEEP_OLD_PACKAGES_COUNT=2
CHECK_CHANGED_OPTIONS=verbose
CHECK_CHANGED_DEPS=yes
CCACHE_DIR=/var/cache/ccache
RESTRICT_NETWORKING=yes

Since I use the same ports tree for the poudriere machine and also for poudriere, I did not create a seperate ports tree for it.  I also have the /etc/makeconf symbolic linked to /usr/local/etc/poudriere.d/make.conf because I want local things built to match what poudriere builds.  This way hopefully there is no difference but automation.

I setup nginx as described in Vermaden's blog post, possibly from a previous nginx poudriere progress view attempt, only needed to check it was accurate.  I have also had ccache setup for ages as well, so did not need to look at that much either.

One of the last steps in this whole process is to define the repos.  I had already setup poudriere.conf in /usr/local/etc/pkg/repos for the build machine to use the pkgs it makes, but I revised it to use the signature, the two lines were added.

poudriere: {
        url      : file:///usr/local/poudriere/data/packages/14amd64-default,
        signature_type: "pubkey",
        pubkey: "/usr/local/etc/ssl/certs/poudriere.cert",
        enabled  : yes,
        priority : 10
}

On the client machine, I needed a new pkg repo configuration file.

atlas: {
  url: "http://172.16.0.12/packages/14amd64-default",
  signature_type: "pubkey",
  pubkey: "/usr/local/etc/ssl/certs/poudriere.cert",
  enabled: yes,
  priority: 100
}

Since at the time that I originally followed the instructions I missed placing a cert file on my client machine, this setup failed inexplicably.  I used pkg -v -v update -f to try to figure it out.  The results did not clearly say that I had no cert or that that was the issue, because something else was interfering.  I had pf setup on the poudriere build machine, and this halted the client from requesting anything, it literally silently failed, no explanation.  I probably had it in the back of my mind that pf could be the reason, so I eventually tried service pf stop and tried again, this time it was able to update the meta file and others.  After that was solved, it took a bit for me to figure out that I needed to copy the cert file to the client.  Once everything was in place, all I had to remember was to periodically disable pf as needed.  I knew I could permanently cut a hole in pf for the pkg update or install requests but that was something for another time.