Poudriere bulk build lists

I may have briefely mentioned or possibly even described how I collect what is installed on my system for rebuilding via poudriere.  I have two methods that I use from time to time, one takes a list direct from those directories which exist in the ports tree, and the other uses pkg to collect what is installed already.  There are only two areas which I am most likely to install all or a majority of the ports and that is fonts and games.  I might use these lists periodically to ensure that new things were added.  I could use the ports tree built-in mechanisms to build every port in a category such as fonts or games but this means I would get items I do not desire.  Building via poudriere will permit me to limit to what I wish and more quickly handle the task as it would work upon a bulk list and has a blacklist capability.

The simple script to collect all the fonts that could be installed looks like this:

#!/bin/sh
[ -f /root/allfonts-list ] && rm /root/allfonts-list
for dir in `ls /usr/ports/x11-fonts`
 do
  echo x11-fonts/$dir >> /root/allfonts-list
done

The same for games is only changed a little:

#!/bin/sh
[ -f /root/allgames-list ] && rm /root/allgames-list
for dir in `ls /usr/ports/games`
 do
  echo games/$dir >> /root/allgames-list
done

My script to create the gross bulk list as well as lists divided into categories of what ports are installed is a bit larger and provides more feedback than most of my scripts.  It was quieter but I decided that it is nice to know what its doing rather than possibly failing, so more of my newer or re-written scripts include a bit more echoes.  You may notice in this script (below) that I specifically exclude the various llvm ports from the bulk list.  These I won't ever need built as non-dependent ports so I expect that poudriere or pkg would draw them in automatically.  This also eliminates the chance that I would have multiple versions of llvm in any bulk list which could become outdated and unneeded.  The blacklist can further eliminate old versions of llvm which could be dependencies but probably ought to be newer versions when upstream for those ports finally get to that task.

#!/bin/sh
echo "Clearing /var/tmp p-keglist files.."
rm /var/tmp/installed-pkgs-gross 2>/dev/null &&echo -n "."
rm /var/tmp/installed-pkgs-gross-added 2>/dev/null &&echo -n "."
rm /var/tmp/installed-pkgs-gross-sorted 2>/dev/null &&echo -n "."
rm /var/tmp/installed-pkgs-ungamed 2>/dev/null &&echo -n "."
rm /var/tmp/installed-pkgs-* 2>/dev/null &&echo -n "."
echo;echo "--"; echo


echo "Backing up installed-gross list to /var/tmp"
cp ~/installed-pkgs-gross /var/tmp/installed-pkgs-gross.backup
echo;echo "--"; echo


echo "Gathering list of installed pkgs.."
pkg origin >> /var/tmp/installed-pkgs-gross
echo;echo "--"; echo


echo "Organizing the list.."
cat /var/tmp/installed-pkgs-gross | sort -d | sort -u >> /var/tmp/installed-pkgs-gross-added &&echo -n "."
cat /var/tmp/installed-pkgs-gross-added | sed -E 's:devel\/llvm[0-9]*::'| sort -u -d > /var/tmp/installed-pkgs-gross-sorted &&echo -n "."
mv /var/tmp/installed-pkgs-gross-sorted /var/tmp/installed-pkgs-gross &&echo -n "."
cp /var/tmp/installed-pkgs-gross ~ &&echo -n "."
echo;echo "--"; echo


echo "Dividing the list.."
category=0

for portorigin in `cat /var/tmp/installed-pkgs-gross-added`
do
category=`echo $portorigin | cut -d / -f 1`
echo $portorigin >> /var/tmp/installed-pkgs-$category

echo $category $portorigin
if ! [ "$category" = "games" ] ; then
echo $portorigin >> /var/tmp/installed-pkgs-ungamed
fi
done
echo;echo "--"; echo


rm /var/tmp/installed-pkgs-gross-added 2>/dev/null

echo "Finalizing and checking list sizes, one looong moment.."
for listfile in `ls -1 /var/tmp/installed-pkgs-*`
do
wc -l $listfile |sed -e 's:\/var\/tmp\/::'
done

cp /var/tmp/installed-pkgs-* ~
echo;echo "--"; echo

I had noticed that my bulk list constructing script seemed to be including duplicates while my intent at that time had been to maintain a list of what had been installed which had newer items added.  I want to keep track of the things I consistently have installed on my system but which due to some upgrade might be left out due to becoming uninstalled (unintentionally).  During that moment if I were to create the bulk list, those items which I didn't truly want removed from my system would not be included in my bulk list.  My solution may have been faulty, to add-to my bulk list, which caused the duplicates in the bulk listings.  This is why I include a little bit of an attempt to remove possible duplicate lines.  There is also a bulk list which excludes games because those might not be what is most important at the time.

Now that the bulk lists exist, one could issue the following command for their poudriere build process poudriere bulk -j 14amd64 -f `p-keg-deblack installed-pkgs-games` with some small explanation that the "installed=pkgs-games" is the bulk list preceded with -f to indicate to poudriere to take the list from a file.  The "p-keg-deblack" is a script mentioned prevously which filters blacklisted items before sending the list to poudriere, it acts upon the bulk list and the content within the ` characters is seen by poudriere as "installed-pkgs-games-deblack" which is a new file it created.  The -j indicates the poudriere jail that it will use for building.  The blacklist can be crafted how ever you wish, my own and a revised p-keg-deblack script is shown in a previous blog post, Working on a blacklist which may be something you could use, however, Essential caveat: Use this script at your own risk, I make no promises.