Stuck with scfb

What an unusual change, that once I did my best to keep the scfb driver uninstalled and now I have to use it.  I recently upgraded my system from 14-stable to the newly available 15-stable.  The usual process of updating src via git had a small hiccup due to a missing update to the blog post I perused.  The minor detail I missed was that git switched to using https and that was what stopped the /usr/src update.  All my blog posts now mention that git change where I had described the old method.  The update was quick and building kernel and world went smoothly as expected.

When I was booted into the 15-stable kernel with world, I knew I had a lot of adjusting to do with regard to installed software or any other changes.  I had saw an easy way to handle the task somewhere online.  What I did was pkg upgrade -f right after I used gitup ports to update the ports tree.  What I didn't know was that pkg was missing or not in a working state, so I had to do pkg bootstrap -f and after I did the pkg upgrade -f, I expected most things to just work.

You can make an educated guess that of course things did not work as expected or hoped.  When I tried to start the X server so I could get back to using my pc as normal, it failed, some errors that seemed to be related to a mismatch with the kernel.  I figured I could get past this issue by being certain to build locally with my updated ports tree.  I rebuilt graphics/mesa-dri which pulled in graphics/libdrm (and I believe graphics/mesa-libs which makes sense), next I rebuilt graphics/drm-kmod which also provided gpu-firmware-kmod, after that was x11-drivers/xf86-video-amdgpu, next was x11-servers/xorg-server, and finally x11/xinit.  I believed I had everything remade compatible with 15-stable and so I tried to start X again.

I must have assumed that I needed to reboot to have everything load properly, but this simply made me reboot a second time into single user so I could remove all the graphics stuff.  Only after the graphics drivers were not trying to load could I boot into text only.

With all that I rebuilt, it gave an error that seemed to be a conflict between drivers?  In the /var/log/Xorg.0.log file it seemed to load a raven driver and then picasso, but it couldn't find the amdgpu_picasso_gpu_info_bin file.  No matter how many times I repeated the rebuild process to be sure I hadn't made any mistakes, I got the same result.  The only way I could succeed to boot was to make sure the graphics was uninstalled, and usually I had to do that via single user.

With too many attempts already made, I remembered that there was the vesa driver which could work, and also the scfb driver I could try.  I installed vesa first and tested it to see an unusable former 4k desktop squeezed into 640 x 480, everything huge on my 4k monitor.  I didn't have mouse, so I couldn't exit fvwm or use an xterm to do anything, but I believe an alternate virtual tty was accessible, and I killed it from there.  Next was the scfb driver which had always wanted to get in the way of the amdgpu driver in the past.  When I got into X again, it was still quite large but much more usable.  I also cured the mouse issue by building the mouse driver which was mysteriously missing.

I have during and since been searching for any recent reports or information about this issue.  Most of the results are rather old, like 2019, or (of course) very Linux specific which does me no good, since their solutions are purely for that distro and its install methods.  I did see mention somewhere about a change to driver naming or something, I am very unclear on that.  That was in Linuxland, so if there was a change in that realm, we need to wait for it to be propagated into FreeBSD before maybe I will see a fix and get back to my amdgpu driver with 4k.  I don't believe there is anything more I can do besides wait, but I will likely keep trying to get away from scfb.

UPDATE:
That was pleasantly fast.  I suffered with graphics being broken and forced into second class scfb for my driver for only two or three days.  Moments after posting the above, I went to /usr/local/etc/pkg/repos to edit the FreeBSD.conf file to disable FreeBSD-ports-kmods, and then upgraded pkg and updated, and after all the drivers were reinstalled, I have my 4k back.  While I watched the update listing, it showed those graphics drivers did indeed all get updated to a more recent version.  I cannot say whether there was a problem in the drivers or if any file was missing, or if strangely those kmod files were not pulled from FreeBSD-ports-kmods repo if that happened before.  All that I can say right now is that my experience with the situation and my suffering scfb for a short time was documented above.  How ever things got corrected, I am grateful for the work by those involved.

Zpool-upgrade loader.efi fail

I recently bought a Sony Walkman android device, which I hoped would permit me to use an app to control my Deco S4 wifi devices.  The wifi hardware was added in hopes of improving local wifi from a single Xfinity device and in a more secure way by connecting via my OPNsense box.  Once I solved the wifi device app issue, I still had the Sony Walkman for audio.

Perhaps foolishly, I wanted to use larger better quality flac files, and knowing I still had quite a large number of CDs to run through ripperX, I tried to change my vdev to better compression.  What seemed to me as being better compression algorithms were unavailable to me because my zfs needed an update.  I had done as many as two updates in the past, one which switched me to 'feature flags' and each time was warned about compatibility but never had an issue.

Without any further investigation, assuming that it would be just as easy a process as in the past, so I upgraded zfs and then changed the compression of the vdev for my music files.  Everything ran fine, no issues.  Later, I was playing minetest and after some time in the midst of what I was attempting on the minetest server, my display and mouse, and pc froze up.  There has been an issue with something somewhere which has caused me a panic and reboot, so all I had once more was the inconvenient interruption.  This time it didn't reboot itself, just seemed to remain frozen, so I rebooted.

This is when I discovered that something was wrong.  It started the boot but before it got very far I was greeted with errors including "ZFS: unsupported feature: com.klarasystems:vdev_zaps_v2" and mention that it couldn't find any bootable drives.  Luckily there was a specific term which I could search online for more information: klarasystems:vdev_zaps_v2.

One search result was FreeBSD forum post 14-0-release-zfs-features-gotcha.91085 but I also looked at the 14.0 release notes, and /usr/src/UPDATING:

20160708:
	The stable/11 branch has been created from head@r302406.

After branch N is created, entries older than the N-2 branch point are removed
from this file. After stable/14 is branched and current becomes FreeBSD 15,
entries older than stable/12 branch point will be removed from current's
UPDATING file.

COMMON ITEMS:

	General Notes
	-------------
	Sometimes, obscure build problems are the result of environment
	poisoning.  This can happen because the make utility reads its
	environment when searching for values for global variables.  To run
	your build attempts in an "environmental clean room", prefix all make
	commands with 'env -i '.  See the env(1) manual page for more details.
	Occasionally a build failure will occur with "make -j" due to a race
	condition.  If this happens try building again without -j, and please
	report a bug if it happens consistently.

	When upgrading from one major version to another it is generally best to
	upgrade to the latest code in the currently installed branch first, then
	do an upgrade to the new branch. This is the best-tested upgrade path,
	and has the highest probability of being successful.  Please try this
	approach if you encounter problems with a major version upgrade.  Since
	the stable 4.x branch point, one has generally been able to upgrade from
	anywhere in the most recent stable branch to head / current (or even the
	last couple of stable branches). See the top of this file when there's
	an exception.

	The update process will emit an error on an attempt to perform a build
	or install from a FreeBSD version below the earliest supported version.
	When updating from an older version the update should be performed one
	major release at a time, including running `make delete-old` at each
	step.

	When upgrading a live system, having a root shell around before
	installing anything can help undo problems. Not having a root shell
	around can lead to problems if pam has changed too much from your
	starting point to allow continued authentication after the upgrade.

	This file should be read as a log of events. When a later event changes
	information of a prior event, the prior event should not be deleted.
	Instead, a pointer to the entry with the new information should be
	placed in the old entry. Readers of this file should also sanity check
	older entries before relying on them blindly. Authors of new entries
	should write them with this in mind.

	ZFS notes
	---------
	When upgrading the boot ZFS pool to a new version (via zpool upgrade),
	always follow these three steps:

	1) recompile and reinstall the ZFS boot loader and boot block
	(this is part of "make buildworld" and "make installworld")

	2) update the ZFS boot block on your boot drive (only required when
	doing a zpool upgrade):

	When booting on x86 via BIOS, use the following to update the ZFS boot
	block on the freebsd-boot partition of a GPT partitioned drive ada0:
		gpart bootcode -p /boot/gptzfsboot -i $N ada0
	The value $N will typically be 1.  For EFI booting, see EFI notes.

	3) zpool upgrade the root pool. New bootblocks will work with old
	pools, but not vice versa, so they need to be updated before any
	zpool upgrade.

	Non-boot pools do not need these updates.

	EFI notes
	---------

	There are two locations the boot loader can be installed into. The
	current location (and the default) is \efi\freebsd\loader.efi and using
	efibootmgr(8) to configure it. The old location, that must be used on
	deficient systems that don't honor efibootmgr(8) protocols, is the
	fallback location of \EFI\BOOT\BOOTxxx.EFI. Generally, you will copy
	/boot/loader.efi to this location, but on systems installed a long time
	ago the ESP may be too small and /boot/boot1.efi may be needed unless
	the ESP has been expanded in the meantime.

	Recent systems will have the ESP mounted on /boot/efi, but older ones
	may not have it mounted at all, or mounted in a different
	location. Older arm SD images with MBR used /boot/msdos as the
	mountpoint. The ESP is a MSDOS filesystem.

	The EFI boot loader rarely needs to be updated. For ZFS booting,
	however, you must update loader.efi before you do 'zpool upgrade' the
	root zpool, otherwise the old loader.efi may reject the upgraded zpool
	since it does not automatically understand some new features.

	See loader.efi(8) and uefi(8) for more details.

and then to the manpage for loader.efi, specifically:

man 8 loader.efi

(excerpt)
EXAMPLES
   Updating loader.efi on the ESP
       The  following  examples	 shows	how to install a new loader.efi	on the
       ESP.

       First, find the partition of type "efi":

	     # gpart list | grep -Ew '(Name|efi)'
	     1.	Name: nvd0p1
		type: efi
	     2.	Name: nvd0p2
	     3.	Name: nvd0p3
	     4.	Name: nvd0p4
	     1.	Name: nvd0

       The name	of the ESP on this system is nvd0p1.

       Second, let's mount the ESP, copy loader.efi to	the  special  location
       reserved	for FreeBSD EFI	loaders, and unmount once finished:

	     # mount_msdosfs /dev/nvd0p1 /boot/efi
	     # cp /boot/loader.efi /boot/efi/efi/freebsd/loader.efi
	     # umount /boot/efi

SEE ALSO
       loader(8), uefi(8)

Since I had experienced similar issues with my box which meant I had to adjust what was on the hard drive, I knew that I would need bootable media so that I could reach the hard drive.  This is what became a more significant and time-consuming problem, at least partly due to my own stubborn foolishness.  I had a number of micro sd cards and a usb reader.  Out of three which had the potential to function as I needed, only one had an old NomadBSD installed upon it.  My mistake was that I insisted upon upgrading it from 12.x to 14.0 which ran into storage constraints and then breaking it from its normal startup process and eventually unable to boot at all.

Later I noticed an old Kingston Digital DataTraveler SE9 64GB USB 2.0 which I discovered has FreeNAS installed.  I could boot it and get to shell which allowed me to get the img file.  It was good enough for some attempts to create a bootable micro sd card but it eventually would fail too quickly during an ftp transfer or dd write, so that I had to find another option.  I was pretty sure that I had some cdroms or dvds which have some kind of FreeBSD, so once I found my cache, I decided to use a PC-BSD 8.2rc2 dvd.

This is when I was finally tired of the whole process and possibly more rested or something, so I actually, after a chunk of two days of attempts, finally made progress.  I booted the PC-BSD disc and after some tries for the GUI, I realized I only truly needed a shell.  From the shell, I was able to mount my target micro sd card which I used to store the downloaded (via ftp -a download.freebsd.org) img file.  I could also format an SSD which I bought more than 5 years ago in a group of four for adding zfs cache devices to my box (obviously still not accomplished).  I shifted the img file to the SSD so I could dd it to the micro sd card.  Once I had the FreeBSD 14.0 installer on the micro sd card, I booted it and began the process of the repair.

My initial plan was to first test the process by an update of the loader.efi on the micro sd card but got stymied.  I followed the steps above (in the loader.efi manpage example).  I discovered that the appropriate partition was ada2p1, which I mounted to /boot/efi and then found that there was no /boot/efi/efi/freebsd path.  What I had was /boot/efi/EFI/BOOT/BOOTxxx.EFI which I left alone.  Instead I created the needed freebsd directory in /boot/efi/EFI.  Once the path was setup, I could copy the loader.efi into the correct location.

This was my very first ever experience with this sort of repair or upgrade, so I was not sure of success until my reboot.  At some point during or shortly after this repair, I decided to properly install FreeBSD 14.0 onto the SSD as a failsafe against future similar problems, I kept the micro sd card as the FreeBSD 14.0 installer.  I used the SSD using the presently unavailable cables2go version of this:

Identified as a Generic-Adapter-Converter-Optical-External.   Aside from all of the above, I was able to connect to my local network with my Sony Walkman to look for answers and help.  If I can keep my Walkman able to be used for such emergencies, this may be an easier method than keeping an entire network and FreeBSD installed on a box functioning, all I will likely need is wifi and access.  I have no plan to ever modify the Walkman away from its functional install.

Definite relief after I was able to properly boot up my box and use it on my last of a series of three days off from work.  I hope that you update your loader.efi BEFORE you upgrade zfs so that you can avoid the excitement of doing the repair above.

Now I can't boot

There have been plenty of times by now that I have made some sort of adjustment on my system and then it doesn't boot.  We know the usual suspects are /etc/rc.conf and /boot/loader.conf but I'm sure there are others, possibly even a badly thought out recently built and installed custom kernel.  So now we are stuck, we have one box and it fails to boot but the way to solve the problem is to get online from it.  The situation with the broken kernel might be sidestepped easily, simply choose the option from the boot menu to use a different kernel.  If there were also mistakes with the buildworld, and lack of items means no booting, then there needs to be another way.

If you can get to single-user mode (another boot menu item), the changes will be easy to apply.  First, mount -u / and then if your filesystem is ZFS rather than UFS, zfs mount -a and now you can re-edit the typo out of your /boot/loader.conf or some other file, but what if your situation is a bit more complex?

You can use a usb stick to boot and from there mount the drives in your pc, make needed adjustments and get everything back to normal again.  This is where it can be fun, and when I say fun I mean not quite a nightmare though it is a real special pain.  You can probably use any bootable BSD which offers shell access to the machine, but since discovering NomadBSD, it has become my preference.  What NomadBSD has is a complete system which is self-contained within the usb media.  So if you are short of time and cannot fix your system, you can use it to get online to do something important, such as check your work schedule.  Of course, this immediate need situation means that you previously setup a web browser and installed and configured an addon (such as blur by Abine) which stores your passwords, and you had made any other needed adjustments to suit your needs.  So, getting online you have the solution to the problem and you've written it down and now you need to fix whatever is wrong on the HDD of your system.

What you need to do is mount the HDD of your system into the usb media that is loaded.  There should be a directory /media present already, if not, create it via mkdir /media because this will be the mount point to reach inside your system HDD.  I would assume that you are already in a shell window (xterm perhaps) or you used shell access from the menu when you booted the usb.  We have two pieces of the puzzle, the solution and the system running with a shell, what we need to do in order to make the changes is get to the drive.

With ZFS, there is a special command which will do what we need, change zroot to the name of your pool.
zpool import -f -R /media zroot

Many times the mistake was made or is corrected in either /etc or /boot, so now to reach those directories or any others on your HDD, you would prefix the desired directory with /media such as below.  Work slowly, re-read the command you've typed before committing to it by pressing return or enter.  While your HDD is mounted to your usb stick, /media is the root directory of your HDD, and / is the root directory of the usb stick itself.  Entering the specific directory in order to edit a file, such as rc.conf or loader.conf, may be better than remembering every time to prefix with /media, but always pay attention to your current working directory or path.
cd /media/etc
or
cd /media/boot

Only you can know what the problem and solution are.  Now that you have access  to your HDD you can make the corrections and reboot.  That zpool import command is only viable until you reboot and does not need to be turned off or disabled.  We have not made any permanent changes to how your drives are mounted in order to fix the problem, unless of course your problem and solution specifically involves a permanent adjustment to how your drives are mounted.  Right now I do not have any examples handy of the dumb things I have done which resulted in being unable to boot along with how they were fixed.

While tinkering with your system in ways that can only truly be done when it is open source, because you use FreeBSD for your need of control over all of it, you are setting yourself up with the potential for mistakes.  There is nothing wrong with unintentionally doing something incorrectly, it is the surest way to learn.  We may read somewhere how to do something but unknowingly miss a step or configure something wrong or assume their technique will work on our system.  The worst of these experiences involve Boot blockers and unfortunately they can at least temporarily halt all further progress.