My previous description of using sylve for porting software to FreeBSD did not include two features that I have decided to enable, but which also cause me some little bit of grief because the most simple method around it seems to no longer exist. I setup sylve which was easy, then I configured a jail, and I have used the jail for updating a number of ports since. I may use it to create new unofficial -dev ports. What I did not do, was begin from nothing for the start of any port. What I have been doing lately is just that, begin with an empty jail with the bare minimum required for any port to build from a Makefile, and then reset the jail after.
Sylve has the functionality already integral to the system. We need to create a ZFS snapshot after the minimal configured jail with minimal required ports are installed. After the snapshot exists, we can rollback any time at will and also configure sylve to automatically rollback when the jail is stopped. All of this is easy from within sylve. The other parts of the jail configuration include some allowed options for the jail, and the fstab entries to allow access to my home directory where my git directories are, and one for ports so I can avoid duplication as well as for consistency between jail builds and builds outside of it.
We need to go to the jail options tab to make some adjustments. Each item expands by clicking the property name, then going to the top to click the edit button.
The snapshot needs to be made but after it exists, clicking on the snapshot line copies the name for it. The whole path starts with the zfs pool name and the default path of sylve/jails and the id number for the jail followed by an @ and then the snapshot name you copied.
I have all of my -dev repo directories in my home directory organized using the same directory hierarchy as the ports tree (which is important). In my
Since we are starting from nothing but the most mandatory initial programs and configuration, any build from that first Makefile (such as for one of my unofficial -dev ports) will potentially cause a build and install of as many as 1400 other ports. Once upon a time, there was an environment variable that pkg honored which would tell it to install all dependencies as packages rather than build from source via ports. This functionality is no longer available when building directly from a Makefile in the ports tree. The good news, as slight as it may be, is that there is still a method to get close to the same result. I wrote a script for each step of the process.
The first script starts with make build-depends-list run-depends-list which we can pipe to sort to alphabetize and remove duplicates, then pipe to sed and awk to exclude the initial base ports and to remove /usr/ports/ from each line as well, and finally direct it to an output file.
#!/bin/sh # First Dependencies make build-depends-list run-depends-list | sort -u -d | sed -e 's:/usr/ports/::' -e 's:devel/bsddialog::' -e 's:devel/ccache::' -e 's:devel/desktop-file-utils::' -e 's:ports-mgmt/pkg::' -e 's:ports-mgmt/portconfig::' | sed '/^$/d' > first_deps_list
Another script takes a file name as input to send to pkg so the content of the file is installed.
#!/bin/sh # Prep yes|pkg install `cat $1`
The third script uses pkg origin output which is also similarly organized and cleaned up for an output file with a larger list.
#!/bin/sh # Most dependencies PORTNAME=`make -V PORTNAME` SELF=`pkg origin | grep $PORTNAME` pkg origin | sort -u -d | sed -e 's:devel/bsddialog::' -e 's:devel/ccache::' -e 's:devel/desktop-file-utils::' -e 's:ports-mgmt/pkg::' -e 's:ports-mgmt/portconfig::' | awk -vLine="$SELF" '!index($0,Line)' | sed '/^$/d' > most_deps_list
This whole process mostly works well, except that when there are conflicting pkgs installed, such as llvm-lite and llvm, or git-tiny and git, especially when an automatic removal of one of the lesser used dependencies causes something like sdl3 to also be removed. Not a problem but note when trying to build and install the updated or nascent unofficial -dev port, whether it still wishes to build dependencies. At that point we can leave and allow it to chug away while we are not present, or just cancel the process to install it from pkg, and then clean and resume the install until the next ones if there are any.
The other method if those scripts are not used, is to set the build process going, re-starting it when it unexpectedly fails to get a dependency of a dependency somewhere (which truly should never occur) and then once it is entirely complete, use pkg to get a list of what was installed. We can stop the build when we notice an extra long compile such as for llvm or any other thing, use pkg for that, clean up and restart the install. After the build is done, we use pkg origin piped to sort and direct into an organized file, then remove the five mandatory initial ports: devel/bsddialog, devel/ccache, devel/desktop-file-utils, ports-mgmt/pkg, and ports-mgmt/portconfig. Another two are useful to have for Makefile testing and cleanup: ports-mgmt/portfmt for portclippy, and ports-mgmt/porttools for portlint. Portclippy has a way of helping visualize how to organize your Makefile, portlint tests various things needed for a proper FreeBSD port.
Any of the secondary lists will likely be larger than a preferred group of dependencies because it is best (for testing and development) to accept the one-size-fits-most default configurations of each and not the most lean build of any or all ports. This is important because most users will use those defaults rather than a longer process of building customized ports locally. When the Makefile for the resulting port is used, we can choose other options that may reduce or change dependencies from those defaults. For testing, any options that affect what is installed should be ticked, any options that are unproven or may change upstream should also be tested, ticked. My own system denies wayland, so I already have a few things I must build for myself due to a wayland default in the official FreeBSD pkg repo.
Regardless of whether we use the speedier or slower method initially, I believe it is best to do another pkg origin directed into a file for the final actual dependency list, minus those five ports of course. Those mandatory ports are in the snapshot and so they will always remain after a rollback. We are making this all-origins file so that each time in the future when we update the -dev port from an effectively empty jail, hopefully we will not need to spend any extra time to build the dependencies. I could keep a jail for every port and leave those dependencies intact, but only up to the point that upstream changes them. By keeping these lists and re-populating the jail before a build, its a happy medium without taking too much time after its setup.
It seems that no matter how much care is taken when compiling the list of dependencies for speedy easy install by pkg, there is a chance that weird conflicts among all of the dependencies may occur. When this happens, automatically accepting by -y or piped yes, may also cause some dependencies to be removed just as quickly. Instead of wasting effort on that, just get the best list of dependencies as possible, use pkg to install them, and then continue with the fresh build of the port. I don't know how to manipulate pkg to get a better result or to ignore its sat solver. I also don't know whatever it would take to truly re-implement that lost capability to install all dependencies with pkg instead of building any that exist in ports.
The main reason we want to test ports, or attempt to port new software, or update ports from within a clean jail, is that dependencies are not silently satisfied and therefore missed in the Makefile. Another reason is that your main system also remains more clean and "uncorrupted" by all of the excess packages or ports that get installed to satisfy dependencies, or that are installed in hopes of satisfying dependencies. Inside the jail we can set it up to be much more vanilla than our personal preference outside of it. And one more good reason, is if we use what is built within the jail which means it is also installed outside of the jail, the update to a port or any testing done in the jail will not cause the executable such as a game, from being uninstalled or broken outside of the jail. A jail can also be setup to be less vanilla but also distinct from the system outside of it, and again, means that system outside is unaffected.
Use sylve to very easily manage a dozen different jails, or just one. Sylve makes the management accessible and visible, all of the options are present, it allows a fresh clean jail on demand, even if it is the same one you always use for a purpose. Working from within a web browser for the interaction may change some things a little bit but those are easily understood. I get around some of the differences by being able to work with git in the same directory from outside of the jail, but I think its the browser that is getting in the way rather than a quirk with sylve. The more sylve is used, the greater chance for improvement, but it is still a great tool in its relatively early days.
No comments:
Post a Comment