Warning: there may be occasional oddness due to css and blog edits. **remodel of posts done. Overhaul of pages (distilled) in progress 3/9
Showing posts with label linux. Show all posts
Showing posts with label linux. Show all posts

Monday, August 31, 2026

Dogfood famine ecosystem

I keep going back to playing on luanti servers as my game for entertainment.  Not exactly because other options are missing but because of frustration, it works and other things I want won't build or are trapped by Windows dependency (and my struggles to get wine or its progeny to work for me), but my choice of server with luanti is also rather zen.  I do my best to keep my own -dev ports updated to the most recent commit.  I try to port things I think are cool, heavier emphasis on games since many other ports in the tree have plenty of representation and support by maintainers.  Along with a number of games that function in my -dev collection of unofficial ports, there is fvwm and feh and flameshot, and frustration with porting.  I use my -dev ports when they result in functional applications.

Porting software from Linux is our primary task, since software developed on and for FreeBSD is either rare or becomes a wider community project subsumed by Linux.  This may not necessarily be a bad thing, but it would also be nice sometimes for FreeBSD to be the originator and controlling interest in the result, for FreeBSD to be the upstream.

There should be no expectation of using portmaster or poudriere

When porting from Linux nothing is simple, not even when at first impression and initial investigation it could be.  There are not only multiple build management tools, such as CMake, Meson, Maven, Autotools, but even when those tools are used there may be non-standard ways of implementing the structure those facilitate.  I have long preferred to see CMake being used because I have become a bit more adept with it.  I can easily discover the options to expose in a Makefile for FreeBSD.  This is not always the case, sometimes CMake is not the method of setting options or not many options are set.  Sometimes dependencies come from another tool as with python eggs or rust crates or lua rocks.  And sometimes they use git submodules which are their own special pain to maintain.

I understand that the proper way for a port is a Makefile, distinfo file, pkg-plist and pkg-descr file, but most importantly the correctly designed Makefile.  On Linux or just outside of FreeBSD, the method for building and instructions given may be a shell script (frequently bash) or a series of commands which are not always directly transferrable into a Makefile.  Discovering the dependencies is usually easy but then FreeBSD ports has its own twist to navigate because we divide some things into a different logical set or use an alternate naming convention.  We may know the thing we're porting needs a certain qt or python module but it may take a bit of investigation to discover how to satisfy that dependency or build requirement.  We have a great porter's handbook which covers a very large swath of what we may need to know, but be careful of examples which may be out of date or could have typos.

A significant portion of all possible problems can be cured by testing.  The best testing is done by actually using it, and even better when the user is familiar with the tool or subject matter.  Who but someone dependent upon QT Creator, an IDE for development would know how it works, or which other files it needs but the port fails to install? It doesn't matter what port in the tree someone wants to build, it should build when given an accurate Makefile and preferably also an accurate distinfo file.  The options exposed should be enough to configure how it is built, the pkg-plist should be complete and up to date.  There should be no expectation of using portmaster or poudriere, when everything present at the origin for a port is meant to be sufficient to build it.  We do not require ccache be used while building ports but it can definitely speed up the process.  Distcc could be used as well, if infrastructure exists and distcc itself is without flaws or challenges.  Our pkg may be easy to use but even when using pkg, it is not immune to dependency issues, and the defaults it describes may be undesirable for those avoiding wayland or any other of a potentially growing list of unwanted or not necessary dependencies.  Poudriere is a very helpful and efficient tool, it can do the job very well.  However, it should not be the case that poudriere builds something significantly differently than directly from ports.  If it does, and it succeeds but building from ports fails, then maybe the exact same adjustments need to be made to the ports tree and its mechanisms.  Poudriere is an optional tool just as much as portmaster.  The result of building directly from ports must have the exact same result as when any additional tool is used.

The standard for being certain that a port has all of its dependencies, builds properly, and has its resulting ports tree files accurate should be to perform the process within a clean jail.  The make.conf used should include DEVELOPER="YES" and the tools portlint and portclippy should be used to help keep the Makefile consistent.  If portclippy or portlint expect an undesirable condition to be met, the tool can be adjusted.  Very often portlint and DEVELOPER="YES" cause problems to be found for correction, so those should be nearly mandatory to use.  With regard to the messages due to developer mode, the warnings about whether any certain library "may not be necssary" are confusing or sometimes inaccurate.  How it determines this seems to be imperfect when sometimes including a library may trigger that message but removing that same library triggers the need to add that dependency.  Sometimes the way to silence the "not necessary" message is to include the dependency another way, such as with a conditional instead of using the LIB_DEPENDS directly, but this doesn't feel right.  Once everything is finished building, the minimum testing that ought to be done is to see whether it will start up, will the executable work? Of course, better is to actually test its function, but at least if the application will start up and not complain about some flaw of dependencies or missing files, then its good to be updated in the ports tree.

We need to extend the /usr/ports/Mk structure for additional build systems, and improve things already present.  Why is it that there are only two primary git systems represented, and why do we do things differently for GitHub than GitLab? Why couldn't the necessary information be provided identically but used at each site in the way it prefers? We can keep the distinction between the two, but the account, project, tag, and install subdirectory (on the build side) could all be ordered identically with the same nomenclature.  How the git site uses or organizes that information should not necessarily mean we organize or provide it differently within the Makefile.  What about gitea or any other random as yet unnamed git repository? Why can't there be a generic git mechanism for the rest that would make it easier on those attempting to port with sources obtained at any existing git repo?

One thing that a potential port may have which becomes a significant bit of extra work, is a git submodule, or multiple of them.  The upstream developers have a way to obtain all of it in one shot but downstream as a FreeBSD port, the prescribed method is to add every one of those submodules as a unique git tuple and then check back each time upstream updates to see if any submodule, obtained by a seperate more extensive git pull, has changed too.  Why can't this be part of the initial git pull when obtaining the upstream sources? If we have any of those submodules as individual ports, and the upstream build can accept them as being provided like any other dependency, then we could do it that way.  However, its very excessive that we need to track down all of the submodules to create a tuple for each, and then repeat the process at each update.  A new Mk git option could be defined which might be Uses= git:sub and it would use the other method, the more extensive git pull that would include the submodules defined.  Those submodules would also automatically be in the expected position within the hierarchy and not be another custom post-extract or other operation in the Makefile.

Surely there may be more things that could be handled by the Mk files that would simplify the Makefile, or possibly improve its readability, clarity, or function.  Changes in the Mk files can certainly trigger informative messages about what is new or deprecated.  Improvements to the options could be made to add OPTNAME_CONTINGENT to be assigned other options that must be true for this option to also be set, although 'contingent' seems a bit long.  It may also be useful to add other kinds of checks, such as checking to see if a port is built with a specific option set, or some specific port is installed which may not be a dependency but could alter the install or configuration of the present port.  There will always be the flexibility of conditionals with .if and the pre-build or post-extract and other sections, but there are potentially simpler pre-defined constructions with less verbose language that could be added.

Makefile configuration option names should be unique when necessary and identical when they truly provide exactly the same function in all the ports that have such an option.  If a port requires wayland and has no way to build without it, then another port which depends upon it could potentially either have an option called wayland as well or mention in the option for that dependent port that it adds wayland.  There may also be times when upstream is not entirely clear about what a toggle in their build actually does, or are somewhat vague stating it adds documentation, which could be anything, and if we had but one Makefile config option across all ports of 'manpage' then it may not be accurate for a port that includes its README file.  Any config option should necessarily mean that the software is built differently, linking to different libraries, changes capabilities, or changes what is installed.  If the results are exactly the same, either the option was not set properly or something is not correct upstream.  There are ways to discover what upstream has for build options but each build management tool does it differently, oddly this is not mentioned in the porter's handbook for how these things can be discovered for each of the build systems, not even for the most common ones already mentioned.

A significant portion of all possible problems can be cured by testing.  The best testing is done by actually using it

Its nice that some things we can add to our Makefile defined in /usr/ports/Mk somewhere also include a number of defaults.  Defaults for CMake for example frequently cause "variable defined not used" messages, which is not actually a problem.  However, it might be helpful for additional CMake defaults or settings to be pre-defined, just to simplify invocations in the Makefile.  During the process of trying to understand a port, additional debugging information, particularly where dependencies are searched or other error messages with that regard, would be helpful.  Perhaps USES= cmake:debug or qt:debug or others, maybe it would be done another way.  I am no expert with Make or the design and function of the Mk files, but surely more can be done for some build systems, dependency mechanisms, and missing use cases that crop up within our growing ports tree.

The build system maven presently does not seem to have any helper in the Mk files which necessitates a potentially complex do-build and likely also a do-install subsection.  Surely this is only one instance where a do-build and possibly also do-install sections could be eliminated.  If the entire ports tree were checked for any of many potential additions to our Mk files, surely porting would be more likely to get easier for those who attempt to port or maintain.  Maybe we can automate upstream build option discovery, whether all those options get used (exposed) in the Makefile is up to the maintainer, but wouldn't it be cool if in the process of the build it could list all the possible options upstream provides?  Just like how there is a test with the DEVELOPER="YES" in the make.conf for linked libraries, maybe this could also trigger a list of those upstream options.  Would it depend upon whether the particular build management tool is in the USES line or simply auto-detect.  Maybe auto-detect because then it could also point at the build management tool if it wasn't already known.  Why not make our Mk infrastructure, mechanisms be ever more helpful and informative?

One tool that has potential to be improved or extended is portscout, since it can be used by those outside of port committers and maintainers.  Presently it can only check upstream for new release versions, but either portscout or a new tool with a similar capability to periodically, up to once daily (or configurable) to check upstream for new commits.  My own unofficial -dev ports which I maintain, were their number to grow significantly, would be helped with automation to check if a new commit, or the last of a series of commits had happened by midnight or the end of upstream's day.  Such a tool could be used not only to help maintain ports based upon last daily commit hash, but also with prompting testing that follows the build of that recent commit.

I want things to be better.  I want things to be easier.  I really want to be able to port anything under the Sun from Linux to FreeBSD, but there always seems to be an impasse where either the upstream coding somehow fails on FreeBSD, a dependency cannot be found while its location is provided, a dependency is entirely missing on our end, or something is weird with the build system or its resulting files.  I tried for ages to get dbeaver ported, I made progress but got stymied by time-date stamping of the resulting files to install.  I mostly got Mudlet functional on FreeBSD but to do so, may likely have broken translations.  Lutris had built last time I tried but seems only partially functional, and I am unsure how to fix it.  Beyond all reason eludes me for unknown reasons but I have made some progress on it.  There was something I was going to try to work on but it uses gitea and that git system is different enough that I am confused by it.

It is entirely possible that a lot of the stuff I am trying to get working for FreeBSD has coding issues upstream which cause me trouble.  As I said once before, we here in FreeBSD need to help upstream see us.  We need to help them help us, rather than we patch everything at downstream to work our way.  Upstream can add conditionals to their CMakeList.txt files (or other build system equivalent) to do things differently for us as necessary, and we can report back to them when its successful.  There is truly no reason why software written to work on Linux cannot be compiled to run under FreeBSD, with the few exceptions of systemd and kernel integration.  Patching all sorts of things on our end to get upstream's software to build how we need it, to install where we want it, and to fit everything else the way FreeBSD needs, is fine but this should be our last resort.  Our changes, patches, revisions should all be given to upstream so they can add them and so we can do as we always suggest to our downstream: give us your patches so we can maintain them instead of you.

We can raise the standard.  We can make porting easier.  Ideas first, testing and implementation next, then lets conquer the challenge of maintaining ports by additional automation and support structures, nothing is impossible but it may take time to build.

Wednesday, January 22, 2025

Gentoo does its way

Since I have been trying Gentoo, I have learned that it is definitely a bit more tedious and detailed to accomplish many things as compared to FreeBSD.  There isn't much automation in a default install and at the moment I am not sure what automation could be added.  Gentoo just does things differently, and you may need a crib sheet to know what to use or how to do something, until it becomes second nature.  It can be learned but there may be many bumps or hiccups along the way, correcting misunderstood methods and finally getting success.

For FreeBSD there is a bootable image which guides you through the entire process from preparing the storage medium (hard drives) all the way to installing and configuring the base system, and adding users.  A reboot into a fresh install drops you to a shell login prompt and then you need to setup your GUI.  Some flavors of FreeBSD such as long defunct PC-BSD and present Ghost BSD will finish by installing your desired GUI or a default GUI.  There is also sysutils/desktop-installer in ports which can be very helpful to automate a number of GUI installations.

If you have to setup a FreeBSD desktop, you can use a desktop installer and choose one of those it supports, or you can work from the basic level.  When you install a GUI for FreeBSD, you first need to get X windows with its assorted support mechanisms (including xterm), and then you choose a GUI like FVWM and maybe a login session manager such as SLiM.  After that is accomplished, you can continue to use pkg or the FreeBSD ports system to install further software.  The hierarchy of the ports system allows for easy discovery of software one may wish to add to their system.  There is plenty of associated documentation that is available online or installed, even as part of the base system, so that many questions can be answered easily.  One can fairly quickly and simply get a FreeBSD system up and running as desired with all the software of interest assuming it is already in ports or a pkg exists.

Gentoo does have documentation, it does have a handbook and wiki and forums.  What it does not seem to have, at least from a default basic install is automation or scripting all set-up to be used.  There is no poudriere, there is no ports tree with its hierarchy and predefined make files that easily facilitate the process.  Gentoo does have a build system, it does have packages.  Installing a fresh system is a long process, every step and detail the user must type and carry out.  There is great flexibility in this but some things may take using or trying to know if those are the best choices to make, the best configuration to use.  There are things that act like helpers, which complete tasks in a similar fashion as sysrc on FreeBSD can set variables for rc.conf, but aside from those it is a long journey with small step after small step process.  Gentoo has its niche, it has its advocates and fans, and it may aid me with porting things to FreeBSD, but if compared to Ubuntu, or as FreeBSD -release compared to -current, it is just a bit more difficult to handle and possibly not really for the novice.

This has though been the most time I have spent trying to use and understand any specific Linux distro and I believe the experience has been useful.  I am certainly not going to erase the long fought work of getting a usable Gentoo in Virtualbox, but instead move it to a faster machine to learn more.  My preference is to use FreeBSD, I have been using it sporadically since before 2000 as v2.2.6 and almost constantly since about 2008 beginning with PC-BSD 8.x which I eventually abandoned (due mostly to my tinkering) for FreeBSD 9.x and FreeBSD stable-9.  I eventually decided to work on porting things to FreeBSD in an unofficial capacity, which is where understanding Linux will be useful.  Gentoo in a virtual box might help me with that.

Wednesday, January 15, 2025

Involved with Gentoo

Since I have been stymied in my attempts to get gaming going on FreeBSD for myself, I decided to attempt another tactic.  After some effort I was able to get VirtualBox installed and setup to function properly, and it took a number of attempts to get the Gentoo iso to startup within it.  I chose Gentoo due to a remembered mention of it being somewhat like FreeBSD, maybe I was remembering something else but I went with Gentoo.  My plan was to see if I could get some stuff to work in the virtual environment which had failed me on FreeBSD.

The Gentoo install is involved.  We may hear complaints from those who try to install FreeBSD or any other BSD as it not being easy or is complicated, or that updating things later is difficult.  I would doubt most of those people have used Gentoo or have gone through all the details necessary to install it.  Luckily for me there is a guy in Kentucky who regularly (annually or so) records a step by step install video for Gentoo. He says that things change from time to time and he likes to keep an updated walkthrough for those who would need it.

I followed his video, pausing and rewatching in order to be sure I got the steps correct as I worked to setup Gentoo in virtualbox.  He tells of some settings that he might not use himself but that others may, and I chose to use some of those.  When it came to setting up the EFI partition and installing the needed files, it did not work.  It took me three iterations through the whole process to this point, restarting from a fresh install each time, to discover what I did wrong.  There was a path or file or directory name that I mistyped each of those previous times.  How I noticed that I made this mistake I am unsure, except that perhaps the emphasis on a third attempt made me look closer.  I made it past this and it felt like I would finally finish.

I was able to reach nearly the last steps when grub is setup for the last reboot.  This is the moment I had difficulty again.  What went wrong I may need to revisit, watching the video again to be sure.  I corrected whatever it was, and succeeded to get grub working, I thought.  Yes it would reboot and grub did function, but the trouble now was that when I went through the second video to install a GUI, I had made some mistakes with SLiM and FVWM3.  How I was trying to execute slim and direct it to start fvwm3 was inaccurate.  I solved this mistake but then discovered that due to the entire Gentoo install being rather bare bones, I was again stopped because although fvwm did load up, I had no way that I knew how to continue.

I had to install xterm, and I tried to solve a script issue that uses python.  Gentoo has a lot of things compile on your system with the emerge commands.  Getting most of what I needed built (and installed) took nearly a day to complete.  I still did it, using FreeBSD as the host, virtualbox provided with as much ram (22G) as I could and as many processor cores (5) as I was able.  The host was still mostly responsive but Gentoo inside virtualbox worked at a snails pace in comparison.

I hoped that if I could install Lutris, and get that to function, I would learn what I am missing with my unofficial port of it, the same for any other port like the ryzom client, or wine functionality, or steam.  Each install took forever which somewhat dampened my expectations, and then when I believed I had done everything for Lutris and the ryzom client to function, there were still unsatisfied dependencies or misconfigured things.  My hopeful panacea for Gentoo in virtualbox to help me solve problems became barely different than attempting directly from FreeBSD.

I still have a lot to learn about how to install a linux thing for Gentoo and how to find and discover dependencies, and where to look for documentation, and which documentation might be most accurate.  I have not yet abandoned it as both a challenge to learn more about and as a possible route to discovering how to make things function properly on FreeBSD.  The amount of non-automated detail that goes into doing things on Gentoo is mostly okay, I do tedious tasks on FreeBSD also, but a little more automation and speed would definitely not hurt, especially speed.  Waiting a day to verify success, to try something is a bit depressing.

I recently bought and assembled a new pc, about a month after I had my virtualbox experience above.  One massively beefier than my present Phenom IIx6, it is a Ryzen 3 and ram jumps from 32 gigabytes up to 128.  When I transfer the virtualbox machine to the new hardware (I have very little intent to go through THAT process again), perhaps the build and install process with emerge will not be so onerous.  Maybe at some future date I can write about how my understanding of Gentoo helped me succeed with something like dbeaver or lutris.

Update:

An updated friendlier to new user Gentoo installer video was made since this was originally posted. I also had forgotten to include the youtube video link in the second paragraph.

Frequently viewed this week