FreeBSD source via git

Some time ago I went through the process of switching to git from svn for updating /usr/src.  I believe the newly git-created directories sat unused after completing that step those 20+ days ago, until I finally set aside the time to work on rebuilding my kernel and world again.  I could have rebuilt and reinstalled my kernel and world right then but as it had been a while, I figured I should update my source.  This is where I needed to research a bit more, the conversion method is readily available but mention of how to update was not and the commands are different than svn so I couldn't just use update as before.

First the conversion process which I did, the first step simply as a precaution.  Copy your last /usr/src updated with svn to a backup directory:

cp -pRP /usr/src /usr/src-last_svn_update

Remove the directory and all contents then create a fresh directory.  As /usr/src contains hidden files or directories, this may be more appropriate and an easier sure step.  After, we still need the directory to use with git, so re-create it.

rm -rf /usr/src

mkdir /usr/src

Or just delete the /usr/src directory contents:

rm -rf /usr/src/*

Now that the way is prepared we can use git to get the source tree:

git clone -b master --single-branch --depth 1  git://github.com/freebsd/freebsd.git /usr/src

 Github changed security in September 2021 

git clone -b master --single-branch --depth 1  https://github.com/freebsd/freebsd.git /usr/src

Since I use the stable branch of version 12, below is how I obtain a specific branch, the manpage also indicates that --depth implies --single-branch, so we can omit that:

git clone -b stable/12 --depth 1 git://github.com/freebsd/freebsd.git /usr/src

git clone -b stable/12 --single-branch --depth 1  https://github.com/freebsd/freebsd.git /usr/src

The output is much less verbose than svn.  We can follow the initial clone command with a few more which provide added detail and verification.

Cloning into '/usr/src'...
remote: Enumerating objects: 84805, done.
remote: Counting objects: 100% (84805/84805), done.
remote: Compressing objects: 100% (71708/71708), done.
remote: Total 84805 (delta 17987), reused 37808 (delta 10001), pack-reused 0
Receiving objects: 100% (84805/84805), 276.31 MiB | 10.02 MiB/s, done.
Resolving deltas: 100% (17987/17987), done.
Updating files: 100% (81378/81378), done.

Looking further into the insanely extensive git manpages, I find git reflog which appears to display in reverse order, the most recent action at the top.

git reflog

94a5e942b (grafted, HEAD -> stable/12, origin/stable/12) HEAD@{0}: clone: from git://github.com/freebsd/freebsd.git

We can discover the config settings by typing the command below from /usr/src.  Many of those settings are defaults and I must have setup my email address previously for git.  I have used git sporadically for various things including my own repos of random projects so this is not a big surprise.  The manpage for git-config lists a LOT more options.

git config --list

user.email=username@mailsvc.com
user.name=username
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git://github.com/freebsd/freebsd.git
remote.origin.fetch=+refs/heads/stable/12:refs/remotes/origin/stable/12
branch.stable/12.remote=origin
branch.stable/12.merge=refs/heads/stable/12

The last thing I need to do is get the most recent updates, to refresh my /usr/src, which if you read the initial paragraph is really all I needed to do when I began this post, and all I will need to do from now on.  The first time updating /usr/src it will be a two step process, the first line is only needed once.

git config pull.rebase true

git pull

If you had not used that first line for your first update, git will give you a reminder as below.

hint: Pulling without specifying how to reconcile divergent branches is
hint: discouraged. You can squelch this message by running one of the following
hint: commands sometime before your next pull:
hint:
hint: git config pull.rebase false # merge (the default strategy)
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.

Lets set the config and show the resulting config list change.

git config pull.rebase true

git config --list

user.email=username@mailsvc.com
user.name=username
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git://github.com/freebsd/freebsd.git
remote.origin.fetch=+refs/heads/stable/12:refs/remotes/origin/stable/12
branch.stable/12.remote=origin
branch.stable/12.merge=refs/heads/stable/12
pull.rebase=true

Now that the git variant on obtaining FreeBSD source is reasonably described, the post about updating kernel and world can be updated to match as well as the specifically related distilled page.  Everything is setup, any further need to obtain source can be accomplished with one simple command:

git pull 

Kernel and world rebuild

I prefer to use the -STABLE branch which means there are many more updates compared to RELEASE.  I try to do an update once a week or at least a couple times a month.  This is not a particularly difficult task but I still refer to a printout of a tutorial page from an old BSD Now podcast.  I have a number of customizations which I have used in the past along with a custom kernel, but I have not used them lately, or at least not the customized KERNCONF.  This entire process will need to be done as root due to standard default permissions.

** Please note that FreeBSD switched to git and so svn is not the method since Dec 18th 2020 or so. **

If you have not done so already, you will need to obtain the source for the build you intend to use.

git clone -b master --single-branch --depth 1 git://github.com/freebsd/freebsd.git /usr/src

and then to prepare now for future updates do:

git config pull.rebase true

Since I use the stable branch of version 12, below is how I obtain a specific branch:

git clone -b stable/12 --single-branch --depth 1 git://github.com/freebsd/freebsd.git /usr/src

Assuming you hadn't just completed the above, or sometime next week you choose to rebuild, one of the first steps is to update /usr/src.  This can be as easy as changing to the directory /usr/src then type git pull.  Either before or directly after this step, it is very important not to have any leftover remnants from the previous builds, use the command make cleanworld, and then to be sure, the command make clean, prior to any building.

Now that the preparations have been made, we can begin.  You can take advantage of multiple cores in your system by setting one job per core or cpu.  If you are uncertain you can check by sysctl -n hw.ncpu and place the resulting number after -j for the number of jobs.  Wow, I just realized that I hadn't used all my six cores the last time I rebuilt my kernel and world.  I would have had I used the command below, because it will always provide the correct number, supposing I have 8 cores sometime in the future I wouldn't be using only six for the build.

make -j `sysctl -n hw.ncpu` buildworld

Now that that is complete, I can do the next step. This one I discovered can be combined with the step after it, as below.  This explicitly defines the kernel configuration file, which must reside in /usr/src/sys/amd64/conf as a real fle, or a symlink to it from your home directory which is a good way to keep it against accidental erasure. By using kernel rather than buildkernel the scripts involved will build and install the kernel in the default location, so this is particularly helpful if the process is unattended such as a slow task handled while the admin is sleeping or handling other matters.

make -j `sysctl -n hw.ncpu` kernel KERNCONF=GENERIC

Although I have had a custom kernel in the past, and I still have the modified kernconf file, I am using GENERIC because shockingly, I had some difficulties. My troubles with booting and with kernel builds have been primarily related to graphics. I think I have a decent handle on them now, but rather than use the monolithic configuration I made, I want to take a closer look at how it compares to GENERIC and also discover which items can be built as kernel objects. Once I know that, I would limit kernel object builds to those I need or might use, and reduce rather than increase the kernel by removing things I never use.

The next step is to go to single user.  This I do with a reboot which also verifies that the kernel I just installed functions as intended to that point.

reboot

Watch for the FreeBSD startup menu, press space as needed to pause it -- it may be set to be a very short countdown here, then choose 'singleuser' by hitting '2'.  It is NOT recommended to use another shell for handling the buildworld or buildkernel processes, so when it finishes and shows the prompt below, tap the return or enter key.

Enter full pathname of shell or RETURN for /bin/sh

As I mentioned in another blog post, now I can't boot the way to access /usr/src again requires some mounting actions, below mounts all items in /etc/fstab as read-write.

mount -u /

and then if your system is ZFS rather than UFS, this mounts all datasets:

zfs mount -a

Once in /usr/src there are a few more things to do and then we can reboot into the new system.  First, what the manpage calls a pre-buildworld mode mergemaster which compares the installed /etc/group and /etc/master.passwd with the new defaults so that you can choose what to adjust as needed, and then the install command.

 Mergemaster removed Nov 20, 2023 replaced by etcupdate. 

Function of etcupdate is essentially the same, pre-buildworld option remains -p for this step.

mergemaster -p

etcupdate -p

make installworld

Should you see the message below, which can appear,

At installworld Make: "/usr/src/Makefile" line 360: Check your date/time

it is a simple fix, type the following command to continue:

adjkerntz -i

Now that the error is corrected, finish the command that was interrupted:

make installworld

The next steps will help update configuration files, allow you to merge your customized lines with the new file as needed, while automatically handling those files which are essentially identical or have not been user-modified.

mergemaster -iUF

etcupdate -n

     -n             Enable “dry-run” mode.  Do not merge any changes to the
                    destination directory.  Instead, report what actions would
                    be taken during a merge

etcupdate

It is good practice to be certain old libraries and other assorted files are not left around to cause problems, especially true between updates of major versions, so the next two commands will help remove all of them automatically. The command below, yes repeats 'y' to fulfill prompts, thus agreeing to each deletion which automates the script.  Yes as described by its manpage can be used to repeat any text.

yes|make delete-old

yes|make delete-old-libs

Now that everything is finished, you can reboot one last time to load up and use your newly updated system.  If for any reason there is a failure with booting, refer to now I can't boot for some steps to help solve it.

Goin High Def

I recently made a significant upgrade to the monitor my primary system uses.  I knew there would be a bit of adjustment to make, but when I unplugged the HDMI cable from the retired Samsung TV and attached it to the new Samsung 4k monitor, everything seemed to be about the same.  There was nothing to adjust, no issues with the display that I could see.  This all changed when I decided to reboot my pc.

There didn't seem to be anything I could do to make my GUI behave the same as it once had, to fill the whole of my new monitor without first having black bars on the left and right, then later it was all simply a horribly bad dpi on a formerly HD monitor.  One of the first things I tried, was in hopes that the solution was simple, that for whatever dumb reason my video card couldn't see the proper screen size and by extension wasn't reporting this to the OS.

My interim solution prior to this was to temporarily power up my old monitor, attach it by HDMI to my pc, then reboot and let the software set things up as they had been.  After X was up on the old monitor, I could plug the HDMI cable into the new display which would then retain the better dpi until a reboot.

I bought a new DisplayPort cable because it has more capability than even my supposedly HD capable HDMI cable which I bought because an older cable began to fail.  I plugged in the new cable, rebooted to get the graphics card to use the proper port, and nothing, things were no better.

The next change was in many ways a fluke, because it was an adjustment I made which had been working just fine so I had no idea that it could be the one detail to begin to get everything working right, finally.  Before this change, I tried variations on two lines for the kldload of the correct driver, first radeon and then amdgpu and back again.  When that was eliminated as the reason (and it was working though still bad resolution etc.) I removed the comments from what ultimately cured nearly everything.

#WITHOUT_DRM_MODULE="YES"
#WITHOUT_DRM2_MODULE="YES"

And now we both see why I am baffled.  I swear I had enabled those two lines which lead to the solution, but to write this blog post I just went to the /etc/rc.conf file to discover them commented out!  Did I not do as I believed I did?  Or maybe somehow something automatically recommented them out?  I have no idea, so I guess I will have to continue with the other issues which I solved and assume that I must have forgot, that the driver was the issue perhaps.

I regretfully do not have any screenshots to show, which would describe the remaining issues.

Nearly everything about firefox was extremely tiny.  Most webpages rendered with what appeared to be maybe an 8px font instead of a more legible 12px.  It took a bit of time to finally stumble upon the solution because we all know how easy it is to find technical answers with google.

What I discovered is that yet again firefox does things itself rather than abide by the host or its environment.  The fix is done by adjusting an about:config setting.  I discovered that mine had been set to .9 which may be default and this could be reason for some oddness in how firefox is different than other browsers.  The setting below seems pretty close to perfect.

layout.css.devPixelsPerPx 1.50

That was the first adjustment for firefox, as it still insisted upon using a second set of mouse cursors and at much larger size than my fvwm desktop UI, which was distracting and annoying.

It seems that at least for the *nix/*bsd world that gtk3 is used by firefox and this is what is overriding the mouse cursor settings.  So to fix this, or get rid of the gtk choice, I edit a file to remove one line.  Edit ~/.config/gtk-3.0/settings.ini to remove the line which defines it:

gtk-cursor-theme-name=whiteglass

So now that my firefox was looking almost normal again, I could work on the remaining GUI issues.  One is to adjust fvwm to have a larger mouse cursor as it is just barely usable size.  I figured out how to make whiteglass mouse cursor theme as an overall default on my desktop as I've always liked its style.  These changes were made for X and fvwm honors them.

Edit ~/.Xdefaults or ~/.Xresources to include lines:

Xcursor.theme: whiteglass
Xcursor.size: 22

My default font size for xterm was also modified by adding the line below to ~/.Xdefaults which may still be a little small but it is much better than what would be.

XTerm*Font:-misc-fixed-medium-r-normal--18-120-100-100-c-90-iso10646-1