Tool without fanfare

Due to our fairly recent VCS change and impending portsnap retirement, it is somewhat surprising that gitup was released to seemingly very little or no fanfare.  It might be used to solve the issue of svn being replaced by git for obtaining source, and both portsnap and svn for maintaining one's local ports tree.  It also is apparently even smaller or less dependency encumbered than git@tiny which should be a rather large positive for the average consumer as well as being quite simple to configure and use.

Firstly, the configuration file provided with the port when installed might need to be updated or modified to suit your needs.

# $FreeBSD$
#
# Default configuration options for gitup.conf.
# github.com/freebsd/freebsd-ports
#
# ${repo} can be replaced by doc, ports and src. (*)
#
# https://git.freebsd.org/${repo}.git
# ssh://anongit@git.freebsd.org/${repo}.git
# (*) In subversion, base is used for FreeBSD src repository (src).
{
        "defaults" : {
                "host"                  : "github.com",
                "port"                  : "443",
                "verbosity"             : "1",
                "work_directory"        : "/var/db/gitup",
        },

        "ports" : {
                "repository"            : "/freebsd/freebsd-ports.git",
                "branch"                : "main",
                "target"                : "/usr/ports",
                "ignores"               : [
                        "distfiles",
                        "packages",
                ],
        },

        "quarterly" : {
                "host"                  : "git.freebsd.org",
                "repository"            : "/freebsd/freebsd-ports.git",
                "branch"                : "quarterly",
                "target"                : "/usr/ports",
                "ignores"               : [
                        "distfiles",
                        "packages",
                ],
        },

        "release" : {
                "repository"            : "/src.git",
                "branch"                : "releng/12.2",
                "target"                : "/usr/src",
                "ignores"               : [
                        "sys/arm64/conf",
                        "sys/i386/conf",
                        "sys/pc98/conf",
                        "sys/powerpc/conf",
                        "sys/riscv/conf",
                        "sys/sparc64/conf",
                ],
        },

        "stable" : {
                "repository"            : "/src.git",
                "branch"                : "stable/13",
                "target"                : "/usr/src",
                "ignores"               : [
                        "sys/arm64/conf",
                        "sys/i386/conf",
                        "sys/pc98/conf",
                        "sys/powerpc/conf",
                        "sys/riscv/conf",
                        "sys/sparc64/conf",
                ],
        },
}

This is because, as we can see above, the defined aliases are no longer appropriate.  The present pending release is 13.0, and current is now 14, while stable is also 13.x, so this should be an easy adjustment.

There does not seem to be an option to simply show what the configured sections are but if one tries to initiate gitup with a bit of nonsense, the message will tell you and so now we can see how the above configuration works.

gitup blargh
gitup:
Cannot find a matching section in the command line arguments.  These are the configured sections:
         * ports
         * quarterly
         * release
         * stable
         * current
: Invalid argument

The short manpage gives further insight and tells of how it can be used to repair a repo and other things.

NAME
     gitup – A minimalist, dependency-free program to clone/pull Git
     repositories.

SYNOPSIS
     gitup section [-ckrV] [-C configuration file] [-h commit checksum]
           [-t tag] [-u pack file] [-v verbosity] [-w commit checksum]

DESCRIPTION
     gitup is a minimalist, dependency-free program used to clone or
     synchronize a local copy of a Git repository.  It is not intended to be a
     full-featured replacement for the devel/git port, but rather, is intended
     to provide a convenient way for users to update FreeBSD source files
     without having to install devel/git (and all of its related dependencies)
     and without the substantial disk overhead that normally accompany local
     Git repositories.

     gitup currently only supports anonymous, encrypted transfers via the
     "Smart HTTP" protocol.

OPTIONS
     Configuration options are stored in /usr/local/etc/gitup.conf and are
     grouped into commonly used sections (additional custom sections can be
     added to this file).  The following command line options can be used to
     override the default and/or section values:

     -C          The location of the configuration file to use.

     -c          Force gitup to clone the repository.

     -h          The "have" commit checksum of the repository to use.  Only
                 needed when importing a pack file generated by the official
                 Git client.

     -k          Save a copy of the pack data.

     -r          Repair the local repository, replacing any files that are
                 missing or have been modified.

     -t          Fetch the commit referenced by the specified tag.

     -u          Skip the download of the pack data and use the specified file
                 instead.

     -v          How verbose the output should be (0 = no output, 1 = show
                 only names of the updated files, 2 = also show commands sent
                 to the server and additional debugging information).

     -V          Display the version number and exit.

     -w          The "want" commit checksum of the repository to retrieve.

FILES
     /usr/local/etc/gitup.conf  Default location of the gitup configuration
                                file.

     /var/db/gitup              Default location where gitup stores its lists
                                of known files.  The files stored here are
                                used during subsequent runs to reconstruct the
                                local repository state and confirm that the
                                local tree is intact.

CAVEATS
     Because gitup neither uses nor updates the .git directory, it is not a
     drop-in replacement for the official Git client.  gitup should only be
     used on repositories originally cloned with the official Git client if
     there is no intention to ever use the official Git client on that local
     repository again.

     To keep its footprint as small as possible, gitup does not retain pack
     files downloaded from the repository (unless explicitly instructed to do
     so) and does not use a saved pack file (unless explicitly instructed to
     do so).  gitup relies on the known remote files lists stored in
     /var/db/gitup and the current state of the local repository to
     reconstruct data that would normally be stored in the pack file.
     Discrepancies in the local repository will be flagged as errors and gitup
     will attempt to any restore any missing or modified files from the
     server, returning the local repository to a clean state.

     Extra care should be given when specifying a commit to pull as gitup only
     performs shallow clones and does not have access to a repository's full
     commit history.  If a commit is passed in via the -w command line
     argument and that commit exists on a different branch, the Git server
     will neither object nor inform gitup of the branch change and will send a
     pack file that will convert the local repository.  If the change of
     branches is intentional, the relevant section in
     /usr/local/etc/gitup.conf will need to be manually updated.

EXAMPLES
     Using the configuration options for commonly used repositories/branches
     stored in /usr/local/etc/gitup.conf, to clone/pull a copy of the most
     recent revision of stable/12:

           gitup stable

     To retrieve a copy of freebsd-ports at commit
     0123456789abcdef0123456789abcdef01234567:

           gitup ports -w 0123456789abcdef0123456789abcdef01234567

SEE ALSO
     gitup.conf(5)

AUTHORS
     John Mehr  is the author of gitup and this man page.

What I like most about this tool is that by default it avoids repairs to the ports tree where I may have made modifications to Makefiles or hidden patch files or any other similar adjustments.  This way I can keep a slightly modified local ports tree which I can use to build individual ports (such as minetest) which I then explicitly blacklist for the poudriere repo pkg builds.  This requires that any pkg built and part of the poudriere repo is removed or a pkg upgrade will still see it.

The initial run, regardless of whether the repair option is used or a clean fresh ports tree is constructed by it, offers no feedback from this tool that it is doing anything and the processes or file transfers involved take quite a bit of time. I did enable maximum verbosity but that did not help, I would strongly recommend to the author that a spinning cursor or some other indication that the tool is doing something be added so that we know that our configuration is correct and we know everything is okay.  I am fine with things taking time to function (in this case about 14 minutes), to produce results, but silence is a killer as it makes me wonder and may cause me to end a process which is less than a second away from finally doing something visible.  The level of verbosity does not affect the 14 minutes of silence, only the output that follows it.  If this is used as part of an unattended script, I would recommend watching for results once outside of the script to be certain it acts as it should before hiding everything, or you may never know if anything is wrong such as an error in the configuration file.

FYI:

When the branch gets changed, which it has twice recently, first it shifted to main, and now its back to master once more, you may end up with an error should you have verbosity enabled for that repo. If you use poudriere, the message may appear similar to below, simply change it to what you might find is used at github/freebsd.

# Host: github.com
# Port: 443
# Repository: /freebsd/freebsd-ports.git
# Target: /usr/ports
# Have: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Want: 4010f7bbc03638d71781ce091bf40a0907fa12fe
# Branch: main
[00:00:00] Updating portstree "default" with git...fatal: couldn't find remote ref refs/heads/master

The solution for poudriere seems to be a bit overkill, but it is as follows below.

Delete the "default" ports tree:

poudriere ports -d

Re-create the "default" ports tree using branch=main by method=git. poudriere ports -c -B main -m git

FreeBSD source via git

Some time ago I went through the process of switching to git from svn for updating /usr/src.  I believe the newly git-created directories sat unused after completing that step those 20+ days ago, until I finally set aside the time to work on rebuilding my kernel and world again.  I could have rebuilt and reinstalled my kernel and world right then but as it had been a while, I figured I should update my source.  This is where I needed to research a bit more, the conversion method is readily available but mention of how to update was not and the commands are different than svn so I couldn't just use update as before.

First the conversion process which I did, the first step simply as a precaution.  Copy your last /usr/src updated with svn to a backup directory:

cp -pRP /usr/src /usr/src-last_svn_update

Remove the directory and all contents then create a fresh directory.  As /usr/src contains hidden files or directories, this may be more appropriate and an easier sure step.  After, we still need the directory to use with git, so re-create it.

rm -rf /usr/src

mkdir /usr/src

Or just delete the /usr/src directory contents:

rm -rf /usr/src/*

Now that the way is prepared we can use git to get the source tree:

git clone -b master --single-branch --depth 1  git://github.com/freebsd/freebsd.git /usr/src

 Github changed security in September 2021 

git clone -b master --single-branch --depth 1  https://github.com/freebsd/freebsd.git /usr/src

Since I use the stable branch of version 12, below is how I obtain a specific branch, the manpage also indicates that --depth implies --single-branch, so we can omit that:

git clone -b stable/12 --depth 1 git://github.com/freebsd/freebsd.git /usr/src

git clone -b stable/12 --single-branch --depth 1  https://github.com/freebsd/freebsd.git /usr/src

The output is much less verbose than svn.  We can follow the initial clone command with a few more which provide added detail and verification.

Cloning into '/usr/src'...
remote: Enumerating objects: 84805, done.
remote: Counting objects: 100% (84805/84805), done.
remote: Compressing objects: 100% (71708/71708), done.
remote: Total 84805 (delta 17987), reused 37808 (delta 10001), pack-reused 0
Receiving objects: 100% (84805/84805), 276.31 MiB | 10.02 MiB/s, done.
Resolving deltas: 100% (17987/17987), done.
Updating files: 100% (81378/81378), done.

Looking further into the insanely extensive git manpages, I find git reflog which appears to display in reverse order, the most recent action at the top.

git reflog

94a5e942b (grafted, HEAD -> stable/12, origin/stable/12) HEAD@{0}: clone: from git://github.com/freebsd/freebsd.git

We can discover the config settings by typing the command below from /usr/src.  Many of those settings are defaults and I must have setup my email address previously for git.  I have used git sporadically for various things including my own repos of random projects so this is not a big surprise.  The manpage for git-config lists a LOT more options.

git config --list

user.email=username@mailsvc.com
user.name=username
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git://github.com/freebsd/freebsd.git
remote.origin.fetch=+refs/heads/stable/12:refs/remotes/origin/stable/12
branch.stable/12.remote=origin
branch.stable/12.merge=refs/heads/stable/12

The last thing I need to do is get the most recent updates, to refresh my /usr/src, which if you read the initial paragraph is really all I needed to do when I began this post, and all I will need to do from now on.  The first time updating /usr/src it will be a two step process, the first line is only needed once.

git config pull.rebase true

git pull

If you had not used that first line for your first update, git will give you a reminder as below.

hint: Pulling without specifying how to reconcile divergent branches is
hint: discouraged. You can squelch this message by running one of the following
hint: commands sometime before your next pull:
hint:
hint: git config pull.rebase false # merge (the default strategy)
hint: git config pull.rebase true # rebase
hint: git config pull.ff only # fast-forward only
hint:
hint: You can replace "git config" with "git config --global" to set a default
hint: preference for all repositories. You can also pass --rebase, --no-rebase,
hint: or --ff-only on the command line to override the configured default per
hint: invocation.

Lets set the config and show the resulting config list change.

git config pull.rebase true

git config --list

user.email=username@mailsvc.com
user.name=username
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
filter.lfs.clean=git-lfs clean -- %f
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
remote.origin.url=git://github.com/freebsd/freebsd.git
remote.origin.fetch=+refs/heads/stable/12:refs/remotes/origin/stable/12
branch.stable/12.remote=origin
branch.stable/12.merge=refs/heads/stable/12
pull.rebase=true

Now that the git variant on obtaining FreeBSD source is reasonably described, the post about updating kernel and world can be updated to match as well as the specifically related distilled page.  Everything is setup, any further need to obtain source can be accomplished with one simple command:

git pull 

VCS changes again

Maybe some of you do better at keeping track of scheduled events, or you're a bit more in touch with the process.  Just a few moments ago I saw a tweet warning "heads up!"

Within the last few days I had noticed that it was 20-something days since my last reboot or update, so I did an svn update of /usr/src and rebuilt, reinstalled.  How serendipitous that was.  According the the wiki page mentioned in the tweet, this might not be possible later this month.  The image below might quickly become invalid but that was what I saw, note the 19th.

 

For some time FreeBSD has had svnlite in base which could be used to obtain an updated /usr/src or /usr/ports but with the switch to git I wonder if there will be git-lite or similar.  I would have to believe that poudriere would no longer be able to use svn+https to get /usr/src or /usr/ports.  So although the capability would not be removed since anyone could still use svn themselves for their private repo, it won't be for official FreeBSD.  Since I still have until December 18th for certain, I will make sure to do one last update to my /usr/src.  The small bit of trivia would be that once upon a time FreeBSD switched from CVS to SVN (2008) and they decided then that Git wasn't capable.  I cannot say whether Git changed or something else did but its surely good enough now, but for how long?  What will be the successor to Git and will it be BSD or MIT or similarly licensed?  If removing GPL from base would prevent git from being part of it, is there a non-GPL client much like svnlite which could be included?

They keep this around for entertainment:  https://wiki.freebsd.org/VCSWhy

I will have to investigate whether there is a way to do much of the same things as I had with svn, because after so many years I finally had everything figured out and the details put in one easy to access place.  I cannot imagine that all of the methods below have no equivalent.