HardenedBSD October 2021 Status Report

October was pretty quiet. This status may seem a bit more random.

I spent more time learning llvm's compiler toolchain, focusing specifically on llvm-readobj. For those following along, here's FreeBSD's exp-run of enabling WITH_LLVM_TOOLCHAIN in base: https://bugs.freebsd.org/bugzilla/show_bug.cgi?id=258872

I followed a few FreeBSD commits that affected HardenedBSD. Some of those commits deal with the signal trampoline code. Please let me know if you experience any oddities around signal delivery.

I've ordered a second ThunderX server for better arm64 support. The two SoftIron OverDrive 1000 systems the HardenedBSD Project has just don't deliver enough performance for doing operating systems development on.

Though it's not a part of the HardenedBSD project per se, I wrote a living article on my setup at home. Included are how I use HardenedBSD in several capacities, one being a PiHole alternative. I normally wouldn't include things of a personal nature in a HardenedBSD status update, but this article provides a decent peek into HardenedBSD's infrastructure. The link to the article: https://git.hardenedbsd.org/shawn.webb/articles/-/blob/master/personal/2...

HardenedBSD September 2021 Status Report

September saw a few changes. Most notably, in 14-CURRENT, FreeBSD introduced WITH_LLVM_BINUTILS, which we switched to. This makes it so that we use more llvm compiler toolchain tools, like llvm-objcopy, llvm-size, llvm-cxxfilt, etc. This causes a large number of ports to either fail or be skipped. Most notably, ports that include explicit DTrace support. I'm (slowly) learning how these extra llvm tools work to try to figure out how to fix these issues.

I hardened TCP and UDP blackholing (the `net.inet.tcp.blackhole` and `net.inet.udp.blackhole` sysctl nodes) such that connection attempts to unbound ports are ignored; that is, no TCP RST packet or ICMP unreach packets are sent on unbound ports.

I worked a bit on documentation, too. So the main project wiki is more up-to-date with regards to our features and changes. There's still more work to be done, but our documentation is indeed improving.

Loic troubleshooted some kernel panics and worked with FreeBSD to get those fixed. I still need to review a few patches he has submitted. I hope to get to that in the first week in October.

We received a new server donation. The server will be dedicated for Cross-DSO CFI development. We're grateful for any and all contributions. Your support keeps HardenedBSD alive. Thank you for giving the HardenedBSD team the opportunity to serve you.

Looking forward into October, I'm hoping to fix the llvm toolchain issues. I need to put my administrative hat on and take care of financials. I plan to reach out to recent donors, asking if they want their name added to the donor's page. I plan to perform a rather major overhaul of our perimeter firewall towards the end of the month.

HardenedBSD August 2021 Status Report

This month was spent in writing utilities and libraries for HardenedBSD's infrastructure. I wrote a little library, called liblattzfs, to help our infrastructure monitoring daemon (hbsdmon) monitor the various ZFS pools on our systems. Though liblattzfs is developed out-of-tree, I've already merged it into the source tree.

I also worked on another library, liblattutil, to make it so that our applications can have one centralized logging API. It also has a nifty SQLite3 wrapper. One could use this wrapper to convert a SQLite3 query's result to JSON. :-)

I created a new little application (rync) to convert our every-six-hour auto-sync scripts from csh to C.

I also disabled CFI for wpa_supplicant. It's my hope that one day, I/we finally get Cross-DSO CFI working in base so that we can re-add CFI to some of these applications. I'm hoping to get back to Cross-DSO CFI work in the coming week.

FreeBSD released some securty advisories, so I made sure that we had binary updates released in a timely fashion. The package builds are still running.

FreeBSD updated ports-mgmt/pkg from 1.16.3 to 1.17.1, introducing some major changes that caused issues with our package repo. It took me a few days to find and resolve the issues. If anyone else notices any issues with the package repo itself, please let me know.

Loic reported an issue with randompid calculation, so I fixed that. He also fixed a few ports and researched the failures of a few others.

Overall, August was a month spent on HardenedBSD's auxiliary applications with the goal of enhanced stability.

liblattzfs: https://git.hardenedbsd.org/shawn.webb/liblattzfs
liblattutil: https://git.hardenedbsd.org/shawn.webb/liblattutil
rync: https://git.hardenedbsd.org/hardenedbsd/rync

HardenedBSD 2021 Donation Run

Over the past couple years, we've focused on growing our infrastructure, replacing old, inherited servers with newer refurbished ones. The community's selfless help has enabled this growth.

The vast majority of our hosting is free, and we're incredibly grateful for those who provide free or low-cost hosting. We do have one leased server that costs $155/month USD. Additionally, we pay $10/month USD for the Pushover.net notification service.

Due to replacing the aging hardware, and replacing the occasional dead part (mostly hard drives), we're sitting at around six months worth of funding.

We need $120/month USD to keep our non-free services online. With much gratitude for existing donations and humility, we are asking the community to provide recurring monthly donations totaling a minimum of $120/month USD.

Please note that The HardenedBSD Foundation is a tax-exempt charitable organization in the USA. Donations by those in the USA are eligible for tax deduction.

We accept donations in the following ways:

  1. PayPal: shawn.webb@hardenedbsd.org
  2. GitHub sponsors: https://github.com/sponsors/lattera
  3. Bitcoin: 1FmbSRvZK4yC1b6ajeZWSvYXV2nmvwdWQq
  4. Ethereum: 0x9Ea8E44736AC8Ed806ef57f7F174a14D93689775
  5. Amazon Smile: https://smile.amazon.com/ch/83-1287321

HardenedBSD May 2021 Status Report

HardenedBSD saw a lot of work in May 2021. Here are the most notable changes:

  1. We hardened the kenv(2) syscall. Unprivileged or jailed users, even a jailed privileged account, should not be able to inspect the kernel environment. This is just one more hardening technique to mitigate "kernel infoleak as feature" type of issue.
  2. We fixed an issue with our ASLR implementation interfering with FreeBSD's ASR implementation. This was due to a mismerge a while ago when FreeBSD made changes to their ASR implementation and we had to adjust ours in the ELF image activator.
  3. We bumped bhyve's max vCPU count to 48 from 16. HardenedBSD's infrastructure now has systems with greater than 30 cores, so being arbitrarily limited to 16 meant we weren't utilizing the server to its greatest potential.
  4. We hardened dumping non-dumpable memory mappings. FreeBSD implemented an interface to tell the kernel "even though this mapping has been marked as non-dumpable, go ahead and dump it when dumping a process' core."
  5. FreeBSD's random stack gap implementation creates problems for certain applications, like ntpd. Rather than fix the issue, FreeBSD is explicitly disabling the stack gap for certain applications. We reverted those changes and are carrying on like normal.
  6. We performed network maintenance. The build infrastructure's firewall is now directly connected to the internet, rather than being behind the corporate firewall. This should drastically help increase the robustness of the network, especially our IPv6 setup.
  7. If you boot in UEFI mode, you'll notice a new bootloader screen. Thanks to Loic, HardenedBSD's logo should be nice and pretty on the UEFI bootloader screen.
  8. Loic and I fixed some ports entries, getting errant ports to compile again in our package builds.
  9. We're researching GitLab anti-spam measures. Ben La Monica from the HardenedBSD Foundation Board of Directors has been kindly maintaining our GitLab instance. We may experiment with different anti-spam measures and may experience minimal downtime as we do that research.
  10. And for the big one: We now build (nearly) all of 14-CURRENT/amd64 world with LTO, including libraries. We introduced a new build configuration, MK_LTOLIB, to tell the build framework whether to build static/shared libraries with LTO. As a precursor to supporting Cross-DSO CFI, we must build as much of the base OS with LTO as possible. This gets us closer to our goal of an OS with Cross-DSO CFI fully enabled for the userland. The compiler toolchain, libthr, and the various bootloaders are NOT compiled with LTO. A set of one or more follow-up commits will enable LTO for arm64 as well. We're doing our first package build with amd64 as we speak. It will still be a while before we support Cross-DSO CFI, but this is a great step in the right direction.

I'm excited for the progress we're making. I'm grateful to serve the community and I'm more grateful for the community's support.

HardenedBSD Switching IRC Servers

Last night, Freenode performed a hostile takeover of the officially-registered #hardenedbsd channel. A screenshot has been attached to this post for posterity. If you want to contact us via IRC, we are on LiberaChat (irc.libera.chat/#hardenedbsd).

If you have any questions, comments, or concerns, please reach out to me (shawn.webb [AT] hardenedbsd.org) directly.

HardenedBSD April 2021 Status Report

In April, HardenedBSD added a new developer named Loic! Loic is primarily focused on cleaning up old cruft in the HardenedBSD project and fixing bugs. It's my hope that, with time and increasing familiarity, he takes on more complex projects. Please join me in a warm welcome.

Just yesterday (30 Apr 2021), I set up the new 12-STABLE/amd64 nightly build server. After months of being offline, we are now building 12-STABLE every night again. I also updated all the servers/VMs/jails in our infrastructure.

I taught hbsd-update how to force IPv4 or IPv6. We now support the typical "-4" and "-6" options that most network applications support.

secadm is having issues right now. We're investigating the right fix. So, if you use secadm for toggling exploit mitigations, you'll probably want to use hbsd-control for now.

FreeBSD's ports migration to git and its impact on HardenedBSD

FreeBSD completed their ports migration from subversion to git. Prior to the official switch, we used the read-only mirror FreeBSD had at GitHub[1]. The new repo is at [2]. A cursory glance at the new repo will show that the commit hashes changed. This presents an issue with HardenedBSD's ports tree in our merge-based workflow.

I'm going to archive our old ports repo[3] and create a new repo[4]. Due to the nature of our changes and how far back our history goes, creating a new repo is necessary. Attempting to do a `git merge --allow-unrelated-histories` works, but brings gitlab to its knees, eventually failing.

For projects downstream of HardenedBSD using the same kind of merge-based workflow, you will need to effectively do the same thing we did. Here's the process I took:

  1. Match the last commit of the old repo (e7ad26d92beff76bbead7b6b675ad5c551e86fa9) with its corresponding commit in FreeBSD's old ports repo (4010f7bbc03638d71781ce091bf40a0907fa12fe).
  2. Clone FreeBSD's new repo to a temporary location
  3. Generate a diff between the old repo and the new: git diff 4010f7bbc03638d71781ce091bf40a0907fa12fe e7ad26d92beff76bbead7b6b675ad5c551e86fa9
  4. Create the hardenedbsd/main branch
  5. Apply the diff created in step 3
  6. Commit and push

Instead of committing directly to the master branch, we created a new branch "hardenedbsd/main". Upstream's branch in our repo is "freebsd/main".

I apologize for the breakage downstream to us. I made a "best effort" attempt at trying to prevent the breakage, but the ports repo is just too large to do it.

Going forward, please use the new repo at [4]. No merge requests and bug reports for the old repo at [3] will be accepted.

I plan to restore the every-six-hour autosync script this weekend. While there's still dust to settle, I plan to do the syncs myself. The dust should be settled this weekend.

[1]: https://github.com/freebsd/freebsd-ports/
[2]: https://cgit.freebsd.org/ports/
[3]: https://git.hardenedbsd.org/hardenedbsd/hardenedbsd-ports/
[4]: https://git.hardenedbsd.org/hardenedbsd/ports

HardenedBSD March 2021 Status Report

This month, I worked on finding and fixing the regression that caused kernel panics on our package builders. I think I found the issue: I made it so that the HARDENEDBSD amd64 kernel just included GENERIC so that we follow FreeBSD's toggling of features. Doing so added QUEUE_MACRO_DEBUG_TRASH to our kernel config. That option is the likely culprit. If the next package build (with the option removed) completes, I will commit the change that removes QUEUE_MACRO_DEBUG_TRASH from the HARDENEDBSD amd64 kernel.

I still have one new server to set up. I plan to use it for our 12-STABLE builds. I enabled the 14-CURRENT/arm64 nightly builds and we've now completed two production package builds.

I'm giving a virtual presentation on 07 Apr 2021 I'm giving titled "HardenedBSD 2021 State of the Hardened Union." It details the work we've been doing since the last HardenedBSD State of the Union.

As part of that presentation, I'd like to highlight areas in which HardenedBSD is used. If you or your employer uses HardenedBSD and would like me to add a slide about it, please reach out to me.

In April, I plan to focus on the ports tree. I'm going to audit all the ports that fail to build and determine if I can easily get them to build. A large number of ports ignore our setting -fPIC and -fPIE compiler flags and subsequently fail to build.

Jason Donenfeld of the Wireguard project is looking for a maintainer/developer for the Wireguard FreeBSD kernel module. If you are familiar with the networking kernel code and would like to help, please reach out to me. I'll get you in touch with Jason. I'm hoping that the HardenedBSD community can fill a gap where the FreeBSD community failed: developing a robust in-kernel Wireguard implementation properly blessed by the Wireguard project. I would be happy to dedicate some HardenedBSD infrastructure resources to help support this effort. Those resources include, but are not limited to: a repo on our self-hosted git server and a VM for nightly builds.

HardenedBSD February 2021 Status Report

This month was a busy one, especially for our infrastructure. We purchased and received three refurbished Dell servers and one Cavium ThunderX server. Two of the three Dell servers were deployed. One is being used for our GitLab migration[1][2]. The second one is being set up as the HardenedBSD 14-CURRENT nightly build and package build server.

With the ThunderX server, we now have a production-capable arm64 server. I'm hoping that we can promote arm64 as a teir 1 architecture by the end of March. Though the server is deployed, I still need to copy the build scripts over and perform is final configuration.

FreeBSD recently landed a W^X implementation (congrats!), but it interferes with our PaX-inspired implementation. In addition to infrastructure work, I've been looking into this. I'm only able to reproduce when building packages--when there's a heavy load on the system. I'm working on some candiate patches, but they're not ready for commit.

The new package build server can be reached at [3]. The Tor onion service address is [4].

The next non-hackery item on my list is to reach out to 2020 donors that donated $250 USD or more to HardenedBSD. I need to give them their tax statement in case they itemize their taxes. I also need to reach out to recent donors to see if they want their name listed on our donations page.

I appreciate the help and support from the community. It's because of you that this project is possible. Your donations enable me to serve you.

[1]: https://git.hardenedbsd.org/
[2]: https://groups.google.com/a/hardenedbsd.org/g/users/c/wZgihY3PR8E/m/gVbZ...
[3]: http://ci-08.md.hardenedbsd.org/
[4]: http://t3xwovqp3hijxp2wskcg2ot5jqpwjd63uaeodsp3ltduhvpxb47aegad.onion/

Pages

Subscribe to HardenedBSD RSS