DragonFly On-Line Manual Pages
SYNTH(1) DragonFly General Commands Manual SYNTH(1)
NAME
synth - custom package repository builder for FreeBSD and DragonFly
SYNOPSIS
synth [help | configure | version | status | purge-distfiles]
synth [rebuild-repository | prepare-system | upgrade-system]
synth [everything | status-everything]
synth [status | build | just-build | install | force | test]
[file | port-origin ...]
DESCRIPTION
The synth program is an advanced concurrent ports building tool aimed at
system administrators that prefer or require the building of packages
from source rather than installing official binary packages. synth will
build packages in a clean environment can exactly mirror the system that
they are built on, it will create local package respositories and install
pkg(8) repository configuration file that causes locally built packages
to be used with the highest priority, all while allowing the system to
fully upgraded with a single command.
synth version
When synth is executed with the version command, an unknown command, or
no command at all, it will display a short block of text containing an
identification, summary, version number, copyright, usage, and a summary
of valid commands.
synth help
The help command displays lists each valid command with a short synopsis
of the command's function. It also elaborates the several commands expect
one or more port origins to be provided, or a path to a file with the
same information.
synth configure
The configure command launches an interactive menu for providing the user
with a clean method to configure synth. It works on the concept of
profiles, and the default profile "LiveSystem" is automatically installed
with values that are determined by scanning the system. Most of the
configurable items are directories that are used to construct the build
environment, but they aren't limited to that.
Ports directory If the PORTSDIR environment variable is set
when the initial profile is created, it
establishes the initial value. Otherwise,
synth will extract the value of PORTSDIR
from /usr/share/mk and use that. If
neither source points to a valid path,
/usr/dports and /usr/ports will be
successfully tried. If those don't exist
either, synth will exit with an error
message.
Packages directory This is the location of the local
repository metadata and all the packages
that are built by synth. It clearly must
have lots of available disk space. The
default value is /var/synth/live_packages
but it should be altered if the system has
a /var partition with limited space.
Distfiles directory This indicates where all the ports source
distribution files are stored, and the
initial value is determined by scanning the
existing configuration. The defaults are
defined by ports and are
/usr/ports/distfiles for (FreeBSD) and
/usr/distfiles for (DragonFly). Most users
will want to keep the initial value as that
directory should already contain distfiles.
Port options directory This is the directory where all the
selected options for ports are cached. The
initial value comes from a system scan, so
chances are it has the correct value.
However, if the user would like a separate
configuration area for port options, they
would create the new directory and set this
value accordingly (but the user would have
to ensure the new location is passed to the
port when configuring port options later in
that case).
Build logs directory A log is produced for every port built
(subsequent builds of the same port will
overwrite previous logs). This item
dictates where those logs are stored, which
is /var/log/synth by default. If the /var
partition is limited, the user will
definitely want to change this location as
the uncompressed logs can potentially
consume gigabyte of space.
Build base directory This is the mount point for all the
builders. When tmpfs is used, this location
requires very little space as it contains
empty directories that serve as mount
points. If tmpfs is not used, then this
location should have access to a lot
available diskspace as it will provide the
work area and virtual /usr/local for each
builder. The default location of
/usr/obj/synth-live should be fine for most
users (it generally has lots of space and
tmpfs is the default build mode).
System root directory Most users will use the base system as the
basis for the build environment, and thus
the default setting of / is correct.
Advanced users may use DESTDIR with
installworld to create an alternative build
base (e.g. i386 cross-build on amd64
architecture or a modified base that serves
as a fallout test for an upcoming feature).
Those would be the users that would modify
this value.
Compiler cache directory If ccache is properly installed on the
system, set this value to the location of
the cache. All the builders will leverage
it. It is disabled by default. It can be
disabled again by entering any invalid path
as a new value, e.g. "none".
Num. concurrent builders The represents the number of ports that can
be simultaneously built on the system. The
selected value is influenced by the number
of physical cores and hyperthreads the
system has, the amount of memory on the
system, the amount of available swap, and
if that swap is a solid-state drive or not.
Generally memory is the limiting resource
when tmpfs is used, so the default value
for the number of builders is generally 75%
of the number of CPUs of the system. The
user is free to balance jobs versus
builders as well.
Max. jobs per builder If memory is constrained, it's often a
better performance tradeoff to reduce the
number of builders and increase the number
of jobs per builder. The default value
varies depending on the system, but it will
never exceed 5.
Use tmpfs for work area If the initial scan shows there is at least
1.25 Gb per builder, this item will default
to "true". Building on tmpfs is a big
speed spot and it's recommended if the
memory resources allow it. This item is
directed at the work area for building the
port
Use tmpfs for /usr/local Before a port can start building, all the
build and library dependencies have to be
installed into the localbase. Extracting,
installing and later deinstalling is a lot
of work for a disk, so using tmpfs here is
another performance boost. It is
recommended that users with adequate memory
leave this enabled.
Display using ncurses During the concurrent building, the default
display is constructed using ncurses. It
shows a lot of good information and it
should be used. The display will fall back
to a text mode if the terminal doesn't
support it. This setting forces text mode
unconditionally, but few users will want or
need to use this option.
Fetch prebuilt packages When active, this option will scan the
external repository for suitable prebuilt
packages to satisfy build requirements. To
be considered suitable, the ABI, options,
and dependencies of the remote package must
all match requirements. If a package is
located, it will be fetched and placed in
the packages directory and later
incorporated into the local repository.
This option is not active by default,
meaning that synth will normally build
everything from source and ignore external
repositories.
To create alternative profiles, press the ">" key at the prompt. A new
menu will appear that provides options to switch to another existing
profile or create a new one that can be tailored and named. Any newly
created profile becomes the new default profile, but this can be easily
changed by repeating the profile selection process.
Pressing the Alphabetic key associated with each configuration item
brings up a prompt to change the value. Directory entries must be valid
paths except in the case of ccache. The boolean values are modified with
a single keypress ("T" or "F") and the numeric items require positive
integers. After changing the values, the old menu returns and altered
values are marked with an asterisk. Simply entering the "Enter" key
(carriage return) saves the changes while pressing the "Escape" key will
discard them.
If more than one profile exists, an option to delete profiles will appear
on the menu. By pressing the "<" key, a new menu will appear that lists
all the inactive profiles. Entering the number associated with one of
those profiles will immediately delete the profile. The deletion cannot
be undone.
synth upgrade-system
The upgrade-system command automates the full upgrade of the system upon
which synth is installed. It is expected that the ports tree has been
updated to the latest version before the command is executed. Using the
ports tree as a reference, synth will query pkg(8) to determine what is
currently installed and which software has newer versions available. It
will rebuild those packages and any package that depends libraries they
contain or that have a runtime dependency on the rebuild packages. In
addition, any port that has changed with respect to its options,
dependencies, or ABI will be rebuilt as well. After analysis, the
concurrent builder will begin building the set of packages that have been
identified as requiring a rebuild. When that phase is complete, the local
repository will be rebuilt without waiting for confirmation. Finally,
synth will command pkg(8) to upgrade the system using the local
repository, effectively bringing it completely up to date.
synth prepare-system
The prepare-system command has the same functionality of the
upgrade-system command with the exception that the packages will not be
installed. Once the local repository is fully rebuilt, the command exits.
synth rebuild-repository
The rebuild-repository command will perform a full sanity check on all
packages present in the packages directory and remove the ones which fail
the check. The local Synth repository will be created (or recreated)
using the remaining packages. This step is also done as part of several
other commands, so it mainly exists for scripting purposes.
synth status
The status command with no arguments performs a dry-run of the
upgrade-system command. It will not delete obsolete packages, nor rebuild
the local repository. It will list all the ports that will be rebuild
along with a total, and it also logs the same information to
/var/synth/synth_status_results.txt since the full list is often longer
than the terminal height.
synth just-build [file | port-origin ...]
The just-build command takes a list (one or more) of port origins (an
origin is the combination of a category and the port directory, e.g.
lang/python27) and it will builds the packages of those ports if
necessary. If analysis determines the packages are already up to date,
nothing is done.
This is the first of several commands that take arguments. The arguments
are either an unlimited number of port origins, e.g.
synth just-build editors/joe editors/nano editors/libreoffice
or they are limited to one, a path to a file, e.g.
synth just-build /tmp/build.list
A file that is equivalent to the first example would contain:
editors/joe
editors/nano
editors/libreoffice
When the building task is complete, synth exits.
synth build [file | port-origin ...]
The build command differs from the just-build command by asking the user
if they wish to rebuild the local repository when the building is
complete. The answer is usually "N" (for "no") because rebuilding the
repository can take a few minutes to complete, and it only makes sense to
do when all the building is complete. Should the user opt to rebuild the
repository, synth will then ask if the user wishes to install the
packages that come from the ports list on the system.
synth install [file | port-origin ...]
The install command is similar to the build command except that it will
not ask permission to rebuild the repository or install the resulting
packages. It will just do it all in sequence.
synth force [file | port-origin ...]
The force command is similar to the build command except that any
packages related to the ports list will be deleted first, regardless if
they are up to date or not. This results in that every port on the given
list will build. As with the build command, synth will request
permission to rebuild the repository and possibly install the packages
when the building phase is complete.
synth test [file | port-origin ...]
The test command will pre-delete any existing packages related to the
ports list before rebuilding them with extra DEVELOPER_MODE tests and
settings. When the building is complete, it will just exit without
offering to rebuild the repository.
synth status [file | port-origin ...]
This version of the status command provides a dry-run and reports on what
would happen if the just-build command was executed with the same
arguments. The output and logging is similar to the solo status command,
but the build scope is limited to the given ports list.
synth everything
The everything command is not meant for most users. It will attempt to
build every port in the ports tree, which can take several days depending
on the power and resources of the building machine. Even downloading the
tens of gigabytes worth of distfiles consumes significant time!
synth status-everything
The status-everything command is just a dry-run for the everything
command. Since synth works through increment building (that is, it will
not build a port if an up to date package already exists for it and all
the packages dependencies are also in place) , the resultant build list
can be much shorter than the full list of ports in the tree. Only those
rare users that wish to build the entire set of ports would use this
command.
synth purge-distfiles
This is a useful command that will first scan the entire ports tree to
gather a complete list of distfiles used by the ports tree, and then it
will scan the contents of the configured distfiles directory. All source
distribution files that cannot be matched to at least one port in the
tree will be removed. This can result in gigabytes of recovered disk
space if significant time has passed since the distfiles directory was
last cleaned.
ENVIRONMENT
PORTSDIR The PORTSDIR environment variable is only checked on the
very first signficant command of synth (meaning everything
except help and version) when the default profile is
created. It is also checked when adding new profiles. The
variable dictates the default location of the ports tree
with a higher priority than the standard locations.
SYNTHPROFILE When this environment variable is set to name of an
existing synth profile, that profile will be loaded when
synth is launch rather than the active profile. It is a
temporary override; removing the variable will allow the
default profile to load again.
WHYFAIL When this environment variable is defined to any value, the
sanity check routines will provide more detail on
dependency and option check failures on a separate output
line.
ENTERAFTER This environment variable only applies with the test
command, and then only when a single port is specified. If
both of those conditions are satisfied, and the variable
has the value of:
- extract
- patch
- configure
- build
- stage
- install
- deinstall
then an interactive build will be attempted. First, all the
port's prerequisites are built normally and synth will
return to text mode. If everything built successfully, the
specified port will build up to and including the point
specified by ENTERAFTER. At that point, control will be
given to the user by providing them with a tcsh shell with
the current directory set at the root of the builder. When
the user is done, they should terminate the shell with the
exit command to signal synth to clean up and terminate.
This is a developer tool to troubleshoot port build
problems and it is not intended for the average user.
LOCK This environment variable only applies with the test
command. When it is defined to any value, the builder's
localbase is remounted read-only prior to the configure
phase and restored to read/write permissions after building
is complete. This is a diagnostic tool aimed at
pinpointing file system violations as a port is not
supposed to write to localbase prior to staging. Note that
the localbase tmpfs setting will considered as False in
this mode, so building may be noticeably slower than
normal.
SELFTEST This environment variable only applies with the test
command. When it is defined, the "test" target is executed
as part of the build cycle; this target is not executed
under any other circumstances.
FILES
The following files may be used or created:
synth.ini This is the configuration file. It is automatically
generated and manipulated using the interactive
configuration menu. There is no need to touch this file
other than to remove unwanted profiles as currently
that is not possible from the program itself. It is
normally located at /usr/local/etc/synth/synth.ini
<profile>-make.conf
This is an optional, user-provided file. If it exists,
the builder's /etc/make.conf will be appended with the
contents of this file. For the default profile, the
file would normally be located at
/usr/local/etc/synth/LiveSystem-make.conf
<profile>-environment
This is an optional, user-provided file. It contains a
list of name-value pairs joins with an equals sign
(e.g. HTTP_PROXY=http://proxyserver:8888), one pair per
line. If the file exists, these variables will be
defined in the builder environment. For the default
profile, the file would normally be located at
/usr/local/etc/synth/LiveSystem-environment
<profile>-private.key
This is one of two files required to support the
signing of the repository by a local RSA key. It is
the output of the openssl genrsa command; see the
EXAMPLE section for how to generate it. The file
permissions are required to be set at 400 as well. For
the default profile, the file is normally located at
/usr/local/etc/synth/LiveSystem-private.key
<profile>-public.key
This is the second of two files required to support the
signing of the repository by a local RSA key. synth
must confirm the public counterpart of the signing key
is available because its path is required for the
generated pkg repository configuration file. For the
default profile, the file is normally located at
/usr/local/etc/synth/LiveSystem-public.key
<profile>-signing_command
This is one of two files required to support the
signing of the repository by an external signing
server. It consists of a single line containing the
command; see the EXAMPLE section. For the default
profile, the file is normally located at
/usr/local/etc/synth/LiveSystem-signing_command
<profile>-fingerprint
This is the second of two files required to support the
signing of the repository by an external signing
server. It consists of a single line containing the
SHA 256 hash of the public key that is the counterpart
of the key that signed the repository. For the default
profile, the file is normally located at
/usr/local/etc/synth/LiveSystem-fingerprint
<profile>-index The introduction of flavors within FreeBSD ports
destroyed the 1:1 relationship between port directories
and packages. Now any port can generated any number of
packages. This requires scanning every port in the
tree to determine the true origin range. This is a
time consuming task, so Synth caches the scan results
and reuses it as long as all ports files are older than
the generated index. When Synth detects the index is
out-of-date or if it doesn't exist, it is automatically
generated. One index exists for each profile, with the
default profile index normally located at
/var/cache/synth/LiveSystem-index
synth_status_results.txt
Whenever a status command is run, a list of ports that
would build with the actual command is produced at
/var/synth/synth_status_results.txt
synth_prefetch_list.txt
Whenever a status command is run, and the option to use
suitable prebuilt packages has been selected, a list of
packages that will be fetched as a result will be
written to /var/synth/synth_prefetch_list.txt
index.html For every build run, Synth installs a web summary at
the Report subdirectory of the build logs directory.
For the default profile, the file would be located at
/var/log/synth/Report/index.html. To remotely view the
web report with a browser, a web server must be present
and configured to expose the logs directory. The web
report is dynamic and updated 10 times per minute. The
complete history is available and searchable, with
quick-filters on "Built", "Failed", "Ignored" and
"Skipped" fields as well as on the "No." column which
is useful to examine cascaded skips. The search filter
can be quickly cleared by clicking on the "Total"
field.
port logs Every single attempt at building a port results in a
build log. The name of the log is always in the form of
[category]___[port].log so that subsequent builds will
overwrite previous attempts. They are created in the
directory specified by the configuration.
hook_run_start If this file exists, and it is executable (or it is a
symbolic link that leads to an executable file) then it
will be executed by synth at the start of the build.
The same requirement exists for all hooks. This hook,
and the next five hooks all define the following common
environment variables: PROFILE, DIR_PACKAGES,
DIR_REPOSITORY, DIR_PORTS, DIR_OPTIONS, DIR_DISTFILES,
DIR_LOGS, DIR_BUILDBASE. Uniquely, this hook also
defines PORTS_QUEUED which is the number of ports that
synth will attempt to build. This file must be colated
with synth.ini, so that would be
/usr/local/etc/synth/hook_run_start for most
installations.
hook_run_end This hook is fired when the build completes. In
addition to the common definitions, it defines
PORTS_BUILT, PORTS_FAILED, PORTS_IGNORED, and
PORTS_SKIPPED in the environment, which represents the
results of the build.
hook_pkg_success This hook is fired whenever a builder successfully
packages a port. In addition to the common definitions,
it defines RESULT=success, ORIGIN, and PKGNAME in the
environment.
hook_pkg_failure This hook is fired whenever a builder fails to package
a port. In addition to the common definitions, it
defines RESULT=failure, ORIGIN, and PKGNAME in the
environment.
hook_pkg_ignored This hook is fired prior to the start of a build when
synth prescans the queue for ports that are to be
ignored according to the ports tree. Once the build has
started, this hook is never fired again. In addition to
the common definitions, it defines RESULT=ignored,
ORIGIN, and PKGNAME in the environment.
hook_pkg_skipped This hook is fired whenever a package gets ignored or
fails which in turns cascades into a number of ports
getting removed from the build queue due to the
dependency's failure. This may fire before the build
starts due to ignored prescanning, or any time during
the build. In addition to the common definitions, it
defines RESULT=skipped, ORIGIN, and PKGNAME in the
environment.
SEE ALSO
pkg(8), pkg-repo(8)
NOTES
Graceful exit
During development, the interrupt signal (SIGINT) was captured and
handled. It turns out this handling extended to the builders, so any
event resulting in a SIGINT in the build would lock up the builder
permanently. There's no apparent workaround for this given the design, so
pressing Control-C is no longer handled. It will stop the build, but it
will leave everything in bad state. The next significant call to synth
will attempt to clean that up though. If a user wants to stop the build,
the best approach is to type the Control-Q combination. The program will
shut down as soon as it can. During the building phase no new builds will
start after the shutdown begins, but the active builders will continue
until completion.
Unprivileged users
At this time, synth can only be run by the root user.
Impulse indicator
A common question is "What does the Impulse indicator on the display
mean?" It is the package build rate (packages/hour) over the last 500
seconds. For the first roughly ten minutes of a bulk run, the "Impulse"
rate and the "Pkg/Hour" rate will be identical. After that, the "Impulse"
build rate will differ from the rate averaged over the entire elapsed
time, often by a great amount. It's just a way to gauge the package
production rate over the previous few minutes.
Supported architectures
Synth is written in Ada, and thus an Ada compiler or cross-compiler is
required to build it. For FreeBSD, that means synth is currently limited
to amd64 and i386. It is feasible to port GNAT to the ARMv7 architecture
as this has already been done (See lang/gnatdroid-armv7 port) but it's
not a trivial task to do it. Other architectures would be challenging,
although not impossible, to support.
EXAMPLE
To activate the signing of default profile's repository with a local RSA
key, generate the two key files:
% cd /usr/local/etc/synth
% openssl genrsa -out LiveSystem-private.key 2048
% chmod 0400 LiveSystem-private.key
% openssl rsa -in LiveSystem-private.key -out \
LiveSystem-public.key -pubout
To activate the signing of default profile's repository with an external
signing server, two files need to be created. It's assumed the signing
server has already been configured as described by the pkg-repo(8) man
page, and that the keys are located in the repo.pub and repo.key files.
On the signing server which is 192.168.0.44 in this example:
% sha256 -q repo.pub > LiveSystem-fingerprint
% echo "/usr/bin/ssh 192.168.0.44 /usr/local/bin/sign.sh" \
> LiveSystem-signing_command
Then move both files to /usr/local/etc/synth directory on the build
server. The build server root user needs to have a passphrase-free SSH
key access to the signing server in order for Synth to sign the
repository.
HISTORY
synth was conceived long before coding began in December of 2015. It was
designed to be system-agnostic from the beginning, and in particular
support for FreeBSD and DragonFly was added simulaneously. It was hoped
that it would attract users of the older PortMaster and PortUpgrade ports
management tools by providing them with a superior and well-maintained
alternative.
synth was written entirely by John Marino <marino@FreeBSD.org> and
released under the Internet Software Consortium license.
DragonFly 6.5-DEVELOPMENT January 27, 2021 DragonFly 6.5-DEVELOPMENT