14-stable update, etcupdate, and a struggle

When I set out to update my system to use FreeBSD 14-stable in lieu of 13-stable, I knew there was a chance things could go awry.  Usually a switch between major versions will involve substantial changes to how things are done and thus indirectly invalidate a large swath of (or everything) that is installed.  I was ready for this task but first I had to get a running system to begin my other (userland) updates.

As usual, essentially the same way I've done many times, I updated my /usr/src to the most current 14-stable, and began the usual buildworld and kernel processes as described in another blog post,Kernel and world rebuild.  Since this was going to be a new ball of wax instead of some minor tweaks of 13-stable, I wisely chose to install a GENERIC kernel.

What I was not so aware of, was how issues with etcupdate could thwart a complete update.  After a couple attempts to rebuild and reinstall, I looked closer at etcupdate and what might possibly be the wrench in the works.  This discussion of my kernel and world challenge is being written from memory as I didn't remember to take any notes to be more precise.  What I recall is that I eventually clarified six or eight files so that they would match what was expected in the new system, etcupdate could then be satisfied (no more conflicts) and my usual process could be completed.

Aside from the etcupdate hiccup, the transformation from 13-stable to the new shiny 14-stable went rather smoothly and easily.  Since I now had a fresh install, I could work on getting my userland (ports) updated and back to a functioning desktop.  I now have a list of ports I need to re-install when I update my kernel and this along with a helpful script to iterate through each of them, would update anything needing the kernel and all the rest.

misc/pciids
graphics/libdrm
graphics/linux-c7-libdrm
graphics/libglvnd
graphics/libGLU
graphics/gpu-firmware-amd-kmod
x11/xorg
graphics/mesa-libs
graphics/mesa-dri
graphics/drm-kmod
devel/libudev-devd
x11-drivers/xf86-input-mouse
x11-drivers/xf86-input-keyboard

I had those ports rebuilt and reinstalled, and made sure that x11-wm/fvwm was installed as well.  When I tried to startx it told me there were no screens detected, which is an obscure way to say that there is no graphics driver, or that is just one cause of the message.  I tried to reinstall the few items I knew would affect this, and made sure to install graphics/gpu-firmware-amd-kmod because that is the hardware I have.  I was still unable to issue startx and get my fvwm desktop back up.  I figured that since I had trouble in the past and desktop-installer cured things before, I could use it now and get back a gui.

I recall there being a moment (but I am very uncertain where in the order of events) when I was unable to connect to the freebsd pkg repo, and I hadn't changed anything myself (directly, intentionally) so I looked online for some documentation about it.  At that time I may have noticed that ports were no longer updating.  Strangely, what had always worked was no longer functioning, I had to change my freebsd.conf to get that back.

This is my /usr/local/etc/pkg/repos/FreeBSD.conf with the former text all commented out with # characters, the working portion below.

# $FreeBSD$
#
# To disable this repository, instead of modifying or removing this file,
# create a /usr/local/etc/pkg/repos/FreeBSD.conf file:
#
#   mkdir -p /usr/local/etc/pkg/repos
#   echo "FreeBSD: { enabled: no }" > /usr/local/etc/pkg/repos/FreeBSD.conf
#
#
#FreeBSD: {
#  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
#  mirror_type: "srv",
#  signature_type: "fingerprints",
#  fingerprints: "/usr/share/keys/pkg",
#  enabled: yes
#  priority: 0
#}

FreeBSD: {
  url: "pkg+http://pkg.FreeBSD.org/${ABI}/latest",
}

Since desktop-installer succeeded to set me up with SDDM and fluxbox, I had a GUI to work on the rest of my updates.  I needed to figure out or disable SDDM and get fvwm going.  At some point I noticed that graphics/drm-kmod had a way to auto choose the right port to install based upon the version of FreeBSD.  The Makefile did not permit the newer version for 14-stable to be drm-515-kmod as it at that moment could only install drm-505-kmod.

I tried to get things updated using the file list mentioned earlier but kept running into problems until I happened to use graphics/gpu-firmware-kmod to install things instead of the more direct amdgpu in the list.  Once I did this, both drm-505-kmod or drm-515-kmod worked fine.  When I reinstalled that list, I kept thinking that maybe one thing in the list affected others, and maybe they do but repeated installs did not solve it.

So finally again I had a gui and everything seemed to be going the right direction.  All it would take to bring things back to a broken state yet again is to update those ports.  I liked the idea of sysutils/auto-admin and it mostly worked fine, aside from how it would insist to reinstall (from pkg) versions with one-size-fits-all default port configs.  The more ridicules was ftp/curl which by removal caused auto-admin scripts themselves to fail.

Eventually I realized that if auto-admin wanted to deinstall ftp/curl and use bad defaults for it (I must have static for some other port) I could build it the way I need then lock it.  This became the solution for a number of issues.  Pango also needed the libthai option because firefox insisted upon it, so that was built and locked.  Some time later, vlc was also affected by changed options, MAD being the only mp3 decoder available, so this too rebuilt and locked.  I know ffmpeg and firefox were both eventually locked as well.

It seems that within the week or so that I rebuilt and reinstalled FreeBSD from 13-stable to now 14-stable, more things mysteriously broke and had to be fixed than usual.  I do not know what to point to as the precise cause.  Most if not all of these difficulties are within the realm of FreeBSD ports and not kernel or world related at all.

Somehow after updating fvwm3 to its most recent commit, what will be the next release version, I exited out to start the new version and couldn't get back.  I was getting a very odd message related to some sort of X windows thing which at this moment I cannot recall.  It was something like how it wasn't sure which amdgpu driver or definition or something to use.  This was odd as when I looked at what I thought was pertinent, there was only one related to this.  I think it was related to drivers again, or I assumed it was, so back to that list and reinstalling.  I eventually added new things to the list after finally getting fvwm up again, mostly.

The trouble was, even though I had fvwm up, my mouse cursor was stuck at the center of the screen and things felt stalled, I think they *were* stalled.  I tried reinstalling the mouse driver and then went back to desktop-installer for the essentials and hopefully a fix.  Once more, desktop-installer cured the issue.  I looked at what drivers were now installed and added more to the list.

x11/libXxf86dga
x11/libXxf86vm
x11-drivers/xf86-input-libinput
x11/xf86dga
x11/xmodmap

I believe now after desktop-installer saved me yet one more time, everything should be working and I would have no troubles.  Except that I was wrong of course.  Now I am unsure how but something with my keyboard configuration now lacked anything for arrow keys.  After a lot of digging around I finally found how to fix them.  What I didn't discover until in the process of writing this is that my delete key does not function (fixed in below list).  I had to edit /usr/local/etc/Xmodmap to add some lines.  Later with xmodmap -pke > ~/.Xmodmap I told xmodmap to copy the file to my home directory.  Vermaden's blog post on FreeBSD desktop (part 9) was also useful to help me remember xev and its function, and some other things.

keycode 111 = 0xFF52
keycode 116 = 0xFF99
keycode 114 = 0xFF98
keycode 113 = 0xFF96
keycode 119 = BackSpace

I keep adding to my 'kernel rebuilt redo GUI' file list, at present this is the correct listing, missteps left out but vulkan stuff added just in case as it is an AMD card.

misc/pciids
graphics/libdrm
graphics/linux-c7-libdrm
graphics/libglvnd
graphics/libGLU
graphics/gstreamer1-plugins-vulkan
graphics/realesrgan-ncnn-vulkan
graphics/realsr-ncnn-vulkan
graphics/vapoursynth-waifu2x-ncnn-vulkan
graphics/vulkan-caps-viewer
graphics/vulkan-extension-layer
graphics/vulkan-headers
graphics/vulkan-loader
graphics/vulkan-tools
graphics/vulkan-validation-layers
graphics/gpu-firmware-kmod
x11/xorg
graphics/mesa-libs
graphics/mesa-dri
graphics/drm-515-kmod
devel/libudev-devd
x11-drivers/xf86-input-mouse
x11-drivers/xf86-input-keyboard
x11/libXxf86dga
x11/libXxf86vm
x11-drivers/xf86-input-libinput
x11-drivers/xf86-video-amdgpu
x11/xf86dga
x11/xmodmap

They always say what does not kill you makes you stronger, well, in the software/computer world it might be more like perseverence along with a little knowledge will always lead to a victory against those inanimate objects.  There is definitely an aspect of the former saying, in that, the greater the struggle, the more extreme the frustration, surpassing both will lead to greater knowlege and understanding, but in a nutshell: just don't give up.  Look for answers, look for help, embrace the puzzle.

Writing about our difficulties, successes, and failures, will help others who find the content.  You too could start a FreeBSD blog.

FreeBSD source via git

Some time ago I went through the process of switching to git from svn for updating /usr/src.  I believe the newly git-created directories sat unused after completing that step those 20+ days ago, until I finally set aside the time to work on rebuilding my kernel and world again.  I could have rebuilt and reinstalled my kernel and world right then but as it had been a while, I figured I should update my source.  This is where I needed to research a bit more, the conversion method is readily available but mention of how to update was not and the commands are different than svn so I couldn't just use update as before.

First the conversion process which I did, the first step simply as a precaution.  Copy your last /usr/src updated with svn to a backup directory:

cp -pRP /usr/src /usr/src-last_svn_update

Remove the directory and all contents then create a fresh directory.  As /usr/src contains hidden files or directories, this may be more appropriate and an easier sure step.  After, we still need the directory to use with git, so re-create it.

rm -rf /usr/src

mkdir /usr/src

Or just delete the /usr/src directory contents:

rm -rf /usr/src/*

Now that the way is prepared we can use git to get the source tree:

git clone -b master --single-branch --depth 1  git://github.com/freebsd/freebsd.git /usr/src

 Github changed security in September 2021 

git clone -b master --single-branch --depth 1  https://github.com/freebsd/freebsd.git /usr/src

Since I use the stable branch of version 12, below is how I obtain a specific branch, the manpage also indicates that --depth implies --single-branch, so we can omit that:

git clone -b stable/12 --depth 1 git://github.com/freebsd/freebsd.git /usr/src

git clone -b stable/12 --single-branch --depth 1  https://github.com/freebsd/freebsd.git /usr/src

The output is much less verbose than svn.  We can follow the initial clone command with a few more which provide added detail and verification.

Cloning into '/usr/src'...
remote: Enumerating objects: 84805, done.
remote: Counting objects: 100% (84805/84805), done.
remote: Compressing objects: 100% (71708/71708), done.
remote: Total 84805 (delta 17987), reused 37808 (delta 10001), pack-reused 0
Receiving objects: 100% (84805/84805), 276.31 MiB | 10.02 MiB/s, done.
Resolving deltas: 100% (17987/17987), done.
Updating files: 100% (81378/81378), done.

Looking further into the insanely extensive git manpages, I find git reflog which appears to display in reverse order, the most recent action at the top.

git reflog

94a5e942b (grafted, HEAD -> stable/12, origin/stable/12) HEAD@{0}: clone: from git://github.com/freebsd/freebsd.git

We can discover the config settings by typing the command below from /usr/src.  Many of those settings are defaults and I must have setup my email address previously for git.  I have used git sporadically for various things including my own repos of random projects so this is not a big surprise.  The manpage for git-config lists a LOT more options.

git config --list

user.email=username@mailsvc.com
user.name=username
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git://github.com/freebsd/freebsd.git
remote.origin.fetch=+refs/heads/stable/12:refs/remotes/origin/stable/12
branch.stable/12.remote=origin
branch.stable/12.merge=refs/heads/stable/12

The last thing I need to do is get the most recent updates, to refresh my /usr/src, which if you read the initial paragraph is really all I needed to do when I began this post, and all I will need to do from now on.  The first time updating /usr/src it will be a two step process, the first line is only needed once.

git config pull.rebase true

git pull

If you had not used that first line for your first update, git will give you a reminder as below.

hint: Pulling without specifying how to reconcile divergent branches is
hint: discouraged. You can squelch this message by running one of the following
hint: commands sometime before your next pull:
hint:
hint: git config pull.rebase false # merge (the default strategy)
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.

Lets set the config and show the resulting config list change.

git config pull.rebase true

git config --list

user.email=username@mailsvc.com
user.name=username
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git://github.com/freebsd/freebsd.git
remote.origin.fetch=+refs/heads/stable/12:refs/remotes/origin/stable/12
branch.stable/12.remote=origin
branch.stable/12.merge=refs/heads/stable/12
pull.rebase=true

Now that the git variant on obtaining FreeBSD source is reasonably described, the post about updating kernel and world can be updated to match as well as the specifically related distilled page.  Everything is setup, any further need to obtain source can be accomplished with one simple command:

git pull 

Kernel and world rebuild

I prefer to use the -STABLE branch which means there are many more updates compared to RELEASE.  I try to do an update once a week or at least a couple times a month.  This is not a particularly difficult task but I still refer to a printout of a tutorial page from an old BSD Now podcast.  I have a number of customizations which I have used in the past along with a custom kernel, but I have not used them lately, or at least not the customized KERNCONF.  This entire process will need to be done as root due to standard default permissions.

** Please note that FreeBSD switched to git and so svn is not the method since Dec 18th 2020 or so. **

If you have not done so already, you will need to obtain the source for the build you intend to use.

git clone -b master --single-branch --depth 1 git://github.com/freebsd/freebsd.git /usr/src

and then to prepare now for future updates do:

git config pull.rebase true

Since I use the stable branch of version 12, below is how I obtain a specific branch:

git clone -b stable/12 --single-branch --depth 1 git://github.com/freebsd/freebsd.git /usr/src

Assuming you hadn't just completed the above, or sometime next week you choose to rebuild, one of the first steps is to update /usr/src.  This can be as easy as changing to the directory /usr/src then type git pull.  Either before or directly after this step, it is very important not to have any leftover remnants from the previous builds, use the command make cleanworld, and then to be sure, the command make clean, prior to any building.

Now that the preparations have been made, we can begin.  You can take advantage of multiple cores in your system by setting one job per core or cpu.  If you are uncertain you can check by sysctl -n hw.ncpu and place the resulting number after -j for the number of jobs.  Wow, I just realized that I hadn't used all my six cores the last time I rebuilt my kernel and world.  I would have had I used the command below, because it will always provide the correct number, supposing I have 8 cores sometime in the future I wouldn't be using only six for the build.

make -j `sysctl -n hw.ncpu` buildworld

Now that that is complete, I can do the next step. This one I discovered can be combined with the step after it, as below.  This explicitly defines the kernel configuration file, which must reside in /usr/src/sys/amd64/conf as a real fle, or a symlink to it from your home directory which is a good way to keep it against accidental erasure. By using kernel rather than buildkernel the scripts involved will build and install the kernel in the default location, so this is particularly helpful if the process is unattended such as a slow task handled while the admin is sleeping or handling other matters.

make -j `sysctl -n hw.ncpu` kernel KERNCONF=GENERIC

Although I have had a custom kernel in the past, and I still have the modified kernconf file, I am using GENERIC because shockingly, I had some difficulties. My troubles with booting and with kernel builds have been primarily related to graphics. I think I have a decent handle on them now, but rather than use the monolithic configuration I made, I want to take a closer look at how it compares to GENERIC and also discover which items can be built as kernel objects. Once I know that, I would limit kernel object builds to those I need or might use, and reduce rather than increase the kernel by removing things I never use.

The next step is to go to single user.  This I do with a reboot which also verifies that the kernel I just installed functions as intended to that point.

reboot

Watch for the FreeBSD startup menu, press space as needed to pause it -- it may be set to be a very short countdown here, then choose 'singleuser' by hitting '2'.  It is NOT recommended to use another shell for handling the buildworld or buildkernel processes, so when it finishes and shows the prompt below, tap the return or enter key.

Enter full pathname of shell or RETURN for /bin/sh

As I mentioned in another blog post, now I can't boot the way to access /usr/src again requires some mounting actions, below mounts all items in /etc/fstab as read-write.

mount -u /

and then if your system is ZFS rather than UFS, this mounts all datasets:

zfs mount -a

Once in /usr/src there are a few more things to do and then we can reboot into the new system.  First, what the manpage calls a pre-buildworld mode mergemaster which compares the installed /etc/group and /etc/master.passwd with the new defaults so that you can choose what to adjust as needed, and then the install command.

 Mergemaster removed Nov 20, 2023 replaced by etcupdate. 

Function of etcupdate is essentially the same, pre-buildworld option remains -p for this step.

mergemaster -p

etcupdate -p

make installworld

Should you see the message below, which can appear,

At installworld Make: "/usr/src/Makefile" line 360: Check your date/time

it is a simple fix, type the following command to continue:

adjkerntz -i

Now that the error is corrected, finish the command that was interrupted:

make installworld

The next steps will help update configuration files, allow you to merge your customized lines with the new file as needed, while automatically handling those files which are essentially identical or have not been user-modified.

mergemaster -iUF

etcupdate -n

     -n             Enable “dry-run” mode.  Do not merge any changes to the
                    destination directory.  Instead, report what actions would
                    be taken during a merge

etcupdate

It is good practice to be certain old libraries and other assorted files are not left around to cause problems, especially true between updates of major versions, so the next two commands will help remove all of them automatically. The command below, yes repeats 'y' to fulfill prompts, thus agreeing to each deletion which automates the script.  Yes as described by its manpage can be used to repeat any text.

yes|make delete-old

yes|make delete-old-libs

Now that everything is finished, you can reboot one last time to load up and use your newly updated system.  If for any reason there is a failure with booting, refer to now I can't boot for some steps to help solve it.