$USER@bit-of-a-byte ~ $ cat $(find /var/log -name '*.log' -print | head -n 5)

More Blog Updates

There are yet more updates to the blog. The first of which is that we now have an actual domain name, which is zyradyl.moe. In keeping with my tradition of complete transparency, the domain was acquired through Gandi.net, after I found out that IWantMyName was unable to accept Discover cards. While I am still supportive of IWMN as a company, if they don’t accept my card it leaves me unable to use them.

Next, the DNS for this site is now handled through Cloudflare, which means that this site is now fully available via HTTPS with a valid SSL certificate. So, small victories.

While running through the process of updating the blog, I noticed several things were broken and went ahead and fixed those:

  • The “Site Version” link in the sidebar now properly links to the GitHub source repository.
  • A long standing issue with pagination has been corrected by updating to the jekyll-paginate-v2 gem, and rewriting the appropriate liquid blocks.
    • Github-Pages does not support the v2 gem. Therefore, the site has been downgraded back to the v1 gem, and the liquid blocks were cleaned up based on trial and error.
  • Related posts are now actually related! This is accomplished by iterating through tags at compile time and creating a list of related posts. While this may not always be accurate, it is far more accurate than the time based system jekyll uses by default.
  • A small issue has been corrected with the header file used across pages. There was a typo that was generating invalid HTML. It didn’t cause any visible issues, but it was a problem all the same.
  • The archive page now uses a new Liquid code block. This is to resolve the long standing </ul> problem, where the code would generate trailing closing tags.
  • HTTPS links have been enforced across the board. I cannot promise the site that you visit will have a valid SSL certificate, but we will certainly try to redirect the connection over SSL now.

HTML proofer is still throwing a few errors related to my consistent use of the Introduction and Conclusion headers, but these are not actual errors.

  • Even these errors have been fixed. HTMLProofer now returns a completely safe site.

I’m also in the process of going back through previous posts and cleaning up the YAML front matter. While this front-matter previously had very little impact on the site, it now can matter quite a lot with the way the related posts system works.

ThothBackup - Part 3

So, another week has gone, and it is time to update this blog with what I have learned. Unfortunately, experiments were not able to be run this week in the realm of data transfer. I decided to revisit the base system to focus on encrypting backup data while it is at rest on the system. This was one of the remaining security vulnerabilities with this process. While end-users still have to trust me, they can at least be assured the data is encrypted at rest.

Essentially, if the system was ever stolen, or our apartment door was broken down, we would just have to cut power and the data would be good. With that previous statement, please keep in mind that this week’s post only refers to the root drive. I didn’t make much progress because of things happening at work, but this is a nice, strong, foundation to build upon.

Many of the steps in this post were cobbled together from various sources across the internet. At the bottom of this post you can find a works cited that will show the posts that I used to gather the appropriate information.

End Goal

The end goal is to ensure that the operating system’s root drive is encrypted at rest. Full Disk Encryption is not an active security measure, it is a passive one. It is primarily there to ensure that should the system ever be stolen, it would not be readable. The root partition will not host any user data, so the encryption should be transparent and seamless.

In short, we will utilize a USB key to provide a Keyfile which will then be combined with LUKS encryption to unlock the LVM array to allow the initramfs to hand over control to the operating system.

Notes ## {: #thoth-3-notes }

Because we are using a solid state drive, and we will be filling the drive with data, it was important for me to over-provision the drive. The SSD we’re using comes with 240GB of space. We can assume that there is some form of manufacturer over-provisioning in play to get that number, if I had to guess I would assume there is actually 256GB of NAND memory on the drive, but only 240GB are made available to the user. This is a fairly reasonable level of over-provisioning.

However, with us planning to fill the drive with pseudorandom data in order to obfuscate the amount of data actually in use, this 16GB could potentially be used quite quickly. SSDs cannot actually rewrite sectors on the fly, they have to run a READ/ERASE/WRITE cycle. This is typically done by writing the new block to an over-provisioned area and then pointing the drive’s firmware at that block. In this way we avoid the ERASE penalty, which can be on the order of 0.5 seconds per block.

Essentially then, every single write to the drive will require a READ/ERASE/WRITE cycle, so padding the over-provisioning is a very good idea. It will help with wear leveling and prevent severe write amplification, while also making the drive “feel” faster.

Prior Work

Before we get into the new installation, we need to prepare the drive for its new role. Unless the flash cells are at their default state, the firmware will regard them as holding data and will not utilize them for wear leveling, thus rendering the over-provisioning useless.

To begin, boot the system via a Debian Live-CD and open up a root prompt using sudo.

If you, like me, prefer to work remotely, you will then need to run a sequence of commands to prep the system for SSH access. We need to add a password to the liveCD user, then install openSSH, and finally start the service. Once all of this is complete, you can log in from a more comfortable system.

# apt-get update
# apt-get install openssh-server
# passwd user
# systemctl start sshd

We will need to install one last software package, hdparm. Run apt-get install hdparm to grab it. Once you have done so, run hdparm -I /dev/sda. Under “Security” you are looking for the words “not frozen”. If it says frozen, and you are working remotely, you will need to access the physical console to suspend/resume the machine. This should unfreeze access to the ATA security system.

The first thing we need to do is to run an ATA Enhanced Erase. After this is done, I still like to run blkdiscard just to make sure every sector has been marked as empty. Finally, we will use hdparm to mark a host-protected-area, which the drive firmware will be able to use as an over-provisioning space. To calculate the HPA size, figure out what size you want to be available to you. Convert that into bytes, and divide by 512, which is the sector size. This will give you the number to pass to hdparm.

# hdparm --user-master u --security-set-pass Eins /dev/sda
# hdparm --user-master u --security-erase-enhanced Eins /dev/sda
# blkdiscard /dev/sda
# hdparm -Np390625000 --yes-i-know-what-i-am-doing /dev/sda
# reboot

Once this is done reboot immediately. There is a lot that can go wrong if you fail to reboot. At this point, I swapped out my disk for the Debian installer. If you are doing this on your own 2006-2008 MacMini, you may want to use the AMD64-mac ISO that the Debian project provides.

From here, we just have to confirm that the drive shows up how we want in the installer (200GB in size, in my case), and we can proceed with the installation.

Installation

Most of the Debian installation process is self explanatory. The only point where I will interject is partitioning. Because of the way the MacMini2,1 boots, it is important that we use an MBR based grub installation. You can do a 32bit EFI installation, but it is very fragile, and I’m not a fan of fragile things. That being said, I still wanted the ability to use GPT partitions. I like being able to label everything from the partition up to the individual filesystems.

Accomplishing this is actually fairly easy anymore. You just need to create a 1MB grub_bios partition as part of your scheme and you’re good to go. To get the level of control we need, we will select manual partitioning when prompted to set up our partitions in the installer.

Create a new partition table (This will default to GPT), and then lay out your initial partition layout. It will look something like this:

<PART #>  <SIZE>  <NAME>          <FILESYSTEM>  <FLAGS>
#1        1MB     BIOS_PARTITION  none          grub_bios
#2        1GB     BOOT_PARTITION  ext4          bootable
#3        199GB   ROOT_PARTITION  crypto        crypto

When you select “Physical Volume For Encryption” it will prompt you to configure some features. You can customize the partition there, but I actually wanted more options than the GUI provided, so I accepted the defaults and planned to re-encrypt later. Please make sure to allow the installer to write encrypted data to the partition. Since we have already set up a customized HPA, a potential attacker already knows the maximum amount of cipher text that can be present, and if the HPA is disabled they would likely be able to gain access to more. Therefore, it is important that we take every possible precaution.

Once this is done, you should scroll to the top where it will say “Configure Encryption” or something similar. Select this option, then select the physical volume we just set up, and it should drop you back to the partitioning menu. This time, however, you will be able to see the newly unlocked crypto partition as something that we can further customize.

Select that volume and partition it like so:

<PART #>  <SIZE>  <NAME>          <FILESYSTEM>  <FLAGS>
#1        199GB                   none          lvm

The LVM option will show up in the menu as “Physical Volume for LVM.” From here, we go back up to the top of our menu and select “Configure Logical Volume Manager.” You will then be taken to a new screen where it should show that you have one PV available for use. Create a new volume group that fills the entire PV and name it as you would like. For this project, I named it djehuti-root and completed setup.

Next we need to create a Logical Volume for each partition that you would like to have. For me, this looked like the following:

<Logical Volume>  <Size>  <Name>
#1                30GB    root-root
#2                25GB    root-home
#3                10GB    root-opt
#4                05GB    root-swap
#5                05GB    root-tmp
#6                10GB    root-usr-local
#7                10GB    root-var
#8                05GB    root-var-audit
#9                05GB    root-var-log
#10               05GB    root-var-tmp

Your layout may be similar. Once this is done, you can exit out and you will see that all of your logical volumes are now available for formatting. Since I wanted to stick with something stable, and most importantly resizable (more on why later), I picked ext4 for all of my partitioning. We will tweak mount options later. For now, the end product looked like the following:

<PARTITION>                       <FS>    <MOUNT POINT> <MOUNT OPTIONS>
/dev/sda2                         ext4    /boot         defaults
/dev/djehuti-root/root-root       ext4    /             defaults
/dev/djehuti-root/root-home       ext4    /home         defaults
/dev/djehuti-root/root-opt        ext4    /opt          defaults
/dev/djehuti-root/root-swap       swapfs  none          defaults
/dev/djehuti-root/root-tmp        ext4    /tmp          defaults
/dev/djehuti-root/root-usr-local  ext4    /usr/local    defaults
/dev/djehuti-root/root-var        ext4    /var          defaults
/dev/djehuti-root/root-var-audit  ext4    /var/audit    defaults
/dev/djehuti-root/root-var-log    ext4    /var/log      defaults
/dev/djehuti-root/root-var-tmp    ext4    /var/tmp      defaults

Once everything is setup appropriately, follow through the installation until you get to the task-sel portion. You really only want to install an ssh server and the standard system utilities pack. Once the installation completes, reboot into your server and make sure everything boots appropriately. We’re going to be doing some offline tweaking after this point, so ensuring that everything is functioning as is will save you a lot of headache.

Once you are satisfied the initial installation is functioning and booting correctly, it is time to move on to re-encrypting the partition with our own heavily customized parameters.

Re-Encryption

This process isn’t so much difficult as it is simply time consuming. Go ahead and reboot your system to the boot media selection screen. You will want to swap out your Debian Installation CD for the Debian LiveCD that we used earlier. Once the disks have been swapped, boot into the live environment and then bring up a shell. We will first need to install the tools that we will use, and then run the actual command. The command is actually fairly self explanatory, so I won’t explain that, but I will explain the reasoning behind the parameters below.

# apt-get update
# apt-get install cryptsetup
# cryptsetup-reencrypt /dev/sda3 --verbose --use-random --cipher serpent-xts-plain64 --key-size 512 --hash whirlpool --iter-time <higher number>

So, onto the parameters:

  • cipher - I picked Serpent because it is widely acknowledged to be a more “secure” cipher. Appropriate text from the above link is as follows: “The official NIST report on AES competition classified Serpent as having a high security margin along with MARS and Twofish, in contrast to the adequate security margin of RC6 and Rijndael (currently AES).” The speed trade-off was negligible for me, as the true bottleneck in the system will be network speed, not disk speed.
  • key-size - The XTS algorithm requires double the number of bits to achieve the same level of security. Therefore, 512 bits are required to achieve an AES-256 level of security.
  • hash - In general, I prefer hashes that have actually had extensive cryptanalysis performed to very high round counts. The best example of an attack on whirlpool, with a worst case situation where the attacker controls almost all aspects of the hash, the time complexity is still 2^128th on 9.5 of 10 rounds. This establishes a known time to break of over 100 years.
  • iter-time - The higher your iteration time, the longer it takes to unlock, but it also makes it harder to break the hash function. So if we combine what we know above with a large iteration time, we gain fairly strong security at the expense of a long unlock time when using a passphrase.

Once these specifications have been entered, you simply need to press enter and sit back and relax as the system handles the rest. Once this process is complete, you should once again reset and boot into the system to verify that everything is still working as intended. If it is, you are ready for the next step, which is automating the unlock process.

Auto-Decryption

There are a few ways to handle USB key based auto-decryption. The end goal is to actually use a hardware security module to do this, and I don’t anticipate the FBI busting down my door any time soon for hosting the data of my friends and family, so I opted for one that is easily extendable.

Essentially, the key will live on an ext4 filesystem. It will be a simple hidden file, so nothing extremely complex to find. This shouldn’t be considered secure at this point, but it is paving the way to a slightly more secure future.

The first thing that I did, though it isn’t strictly necessary, is write random data to the entire USB stick. In my case, the USB drive could be found at /dev/sdb.

# dd if=/dev/urandom of=/dev/sdb status=progress bs=1M

Once this is done, we’ve effectively destroyed the partition table. We will recreate a GPT table, and then create a partition that fills the usable space of the drive.

# apt update
# apt install parted
# parted /dev/sdb
(parted) mklabel gpt
(parted) mkpart KEYS ext4 0% 100%
(parted) quit

Now we just create the filesystem, a mount point for the filesystem, and make our new LUKS keyfile. Once the file has been created, we just add it to the existing LUKS header.

# mkfs.ext4 -L KEYS /dev/sdb1
# mkdir /mnt/KEYS
# mount LABEL=KEYS /mnt/KEYS
# dd if=/dev/random of=/mnt/KEYS/.root_key bs=1 count=4096 status=progress
# cryptsetup luksAddKey /dev/sda3 /mnt/KEYS/.root_key

After this point, the setup diverges a bit depending on what guide you follow. We will stick close to the guide posted to the Debian mailing list for now, as that guide got me a successful boot on the first try. The others are slightly more elegant looking, but at the expense of added complexity. As such, they may end up being the final configuration, but for this prototyping phase they are a bit excessive.

We have to modify the crypttab file to enable the keyfile to be loaded off of our freshly set up key drive.

sda3_crypt  UUID="..."  /dev/disk/by-label/KEYS:/.root_key:5  luks,initramfs,keyscript=/lib/cryptsetup/scripts/passdev,tries=2

At this point, we need to repackage our startup image, update grub, and reboot to test the whole package.

# update-initramfs -tuck all
# update-grub
# reboot

At this point the system should boot automatically, but you will notice a weird systemd based timeout that happens. This is mentioned in the guide posted to the Debian Stretch mailing list, and is fairly easy to solve. We just need to create an empty service file to prevent systemd from doing it’s own thing.

# touch "/etc/systemd/system/[email protected]_crypt.service"
# reboot

At this point, everything should boot correctly and quickly. You may notice a few thrown errors, but it shouldn’t be anything severe, more services loading out of order.

At this point, it used to be possible to allow for the creation of a fallback in the event that the key drive wasn’t present, but that seems to have been removed. I plan to look into it further when I have more time.

Conclusion ## {: #thoth-3-conclusion }

This concludes the first part of the Operating System setup process. The next step was originally planned to be thin-provisioning the partitions inside the djehuti-root volume group, but there seems to be some problems in getting the system to boot from a thin-provisioned root. I’m looking into a weird combined system, where the root is static but all the accessory partitions are thinly provisioned, but it will take time to tinker with this and report back.

Thin Provisioning isn’t strictly required, but it is a rather neat feature and I like the idea of being able to create more partitions than would technically fit. I’m not sure when this would be useful, but we will see.

Once all of this is finalized, we will move on to hardening the base system, and last but not least creating the Stage 1 Project page. Then it is back to experiments with data synchronization. This is a fairly large step back in progress, but I am hopeful it will result in a better end product, where security can be dynamically updated as needed.

Works Cited

The following sources were invaluable in cobbling this process together. I sincerely thank the authors both for figuring the process out and documenting the process online.

ThothBackup - Part 2

Hello! It’s that time of the week again, where I update everyone on my latest work. This episode is far less technical and focuses more on the concept of a “One and Done” backup solution, aka the holy grail of data maintenance.

It fucking sucks.

Introduction ### {: #thoth-2-introduction }

This entry is slightly unidirectional. The concept of a simple, easy to implement, catch everything you might ever need solution is quite literally the holy grail, yet it has never honestly been implemented. Sure, user data is generally scooped out, but in the day and age of game mods, and with some development projects taking place outside of the User directory, it seemed prudent to at least attempt the full backup. Well, I’ve been attempting it for seven days. Here’s what I’ve found.

Focus

We will not be focusing on the space impact of a complete backup. This is actually fairly negligible. With out-of-band deduplication, only one set of operating system files would ever be stored, so server side storage would reach a weird type of equilibrium fairly quickly. Instead, I’ll talk about three things:

  • Metadata Overhead
  • Metadata Processing
  • Initial Synchronization

There may be another post tonight talking about additional things, but this deserves it’s own little deal.

Metadata Overhead

A fully updated Windows 10 partition of your average gamer, aka my fiancé, is composed of 479,641 files and 70,005 directories which comprise a total data size of ~216 GiB. This is actually just the C drive and typical programs. If you factor in the actual game drive in use by our test case, that drive contains 354,315 files and 29,111 directories which comprise a total of ~385 GiB of space.

In summation, an initial synchronization of what is typically considered a “full system backup” comprises 833,956 files and 99116 directories comprising ~601GiB which results in an average filesize of ~755KiB and an average directory size of ~9 files.

SyncThing creates a block store that is comprised of, by default, 128KiB blocks. This means that for our system, assuming the data is contiguous, we need 4923392 Metadata Entries. Assuming the files are NOT contiguous, this is probably closer to about 5 Million metadata entries. As of right now, the server side metadata storage for the testing pool is at 1.7 GiB and initial syncronization is not yet complete. Extrapolating a bit, we can assume that 2.0 GiB would not be an unreasonable size for a final server side data store.

The client side store, at the time of writing, is approximately 1 GiB and may grow slightly larger. However, I will use 1 GiB. This means that there is a plausible total of 3GiB of metadata overhead representing an overhead percentage of ~0.5% across the pool. Scaling up, this means 10 clients with 1TB of data each would require 51.2GB of Metadata.

Should anything happen to the metadata store, it would need to be rebuilt by data reprocessing. This introduces a potentially massive liability, as scanning frequency would need to be reduced to not impact the rebuild operation.

Metadata Processing

The server is capable of a hash rate of 107MB/s. I am picking the server’s hash rate because it is both the slowest hash rate of the pool and would have the most metadata that would need to be rebuilt.

For a complete rebuild of the data of our current cluster, it would take the server ~96 Minutes during which no data synchronization could occur. This equates to a minimum of 1 Missed Hourly Update and could potentially result in up to 2 missed hourly updates if the timing was unfortunate enough.

For a complete rebuild of the data of our theoretical cluster, we will allow for a hash rate of 300MB/s. The total data needed to be rebuilt would be 10TB. This would result in a database rebuilt time of ~10 Hours which could result in up to 11 missed synchronization attempts.

Initial Synchronization

The initial syncronization is composed of three primary parts. First, the client and host must agree on what folders to syncronize. Second, the client must build a database of the content hosted locally. Next, utilizing a rolling hash algorithm, data is entered into the metadata cache and transmitted to the server.

Per the developer of SyncThing, millions of small files are the worst case scenario for the backup system. As of my independent, albeit anecdotal testing, After 7 days the synchronization process is still in effect. This represents a very poor user experience and would not be ideal for a widespread rollout.

Conclusion ### {: #thoth-2-conclusion }

The primary goal of a backup utility is to synchronize files and achieve cross system consistency as quickly as possible. While it is true that eventually consistent systems are utilized in large scale operations, this type of consistency is allowable only, in my opinion, at data sizes over 10TB. The current testing set is approximately 1TB at most, and thus this is unacceptable.

Either the backup paradigm must change, or the utility used to implement it must change. While I do not expect to find any faster utilities for performing the backup process, I do plan to continue to experiment. At this time, however, it seems that the most likely way to make the process as friendly as possible would be the implementation of a default backup subset, with additional data added upon user request, and after the high priority synchronization had been completed.

Blog Updates

Yes, it’s that time of year again. I have updated the blog! You can find more information below.

Analytics

Google Analytics are back. I understand that some people may not like being tracked, but at that point you should have an add or tracking blocker installed. I recommend looking at the Brave Browser, which is what I personally use, or installing uBlock Origin. The reason I have added this back to the blog is that I have noticed links to this blog appearing in various places over the web, and I would like to be able to detect how much traffic I am getting from these links.

I wholeheartedly understand if you disapprove of the use of Google Analytics. If anyone is able to suggest a better service, that collects less user data, please open an issue in the GitHub repository for this site. I will gladly change providers as long as the new one is also free.

Theme Updates

I forked the Lanyon repository and applied all the currently pending pull requests. This should keep everything up to date with the latest version of Jekyll. To make it easier for anyone else looking for that information, I created a new PR in the lanyon repository to my mergers. You can also find them under my GitHub site.

Fonts

Somehow the fonts on the site got nuked during the upgrade. They are back in place now.

Page Speed

The new updates have not yet been optimized. Previously I used Google’s page load speed thingy to optimize the site. Maintaining that by hand is one of the reasons the upgrade was so painful to implement. I’m looking for a way to automate the process the same way that I have currently automated the link tests that I run prior to a push. This will likely involve writing a new process in the Rakefile, so it will take some time. In the meantime, the only thing that is really being pulled is a few font files and the analytics script, so the load impact shouldn’t be too bad.

Organization

There is a weird issue between rendering the site on my local machine and the way GitHub pages renders it on their side. To solve this I created a new template and appended it to all the pages that should not appear in the sidebar. Hopefully this strikes a good balance between being able to use standardized templates, as well as ease of use. The new template simply imports the existing page template under a new name.

Images

While I am primarily focused on text on this blog, I have recently included a few images. These are also hosted on GitHub, so I cleaned up the way the image directory is laid out. This has impacted all of one image, but should make any expansion in the future much easier.

Liquid Changes

There was some issue with the way the Liquid on the Tags page was written. This has been corrected accordingly.

About Page

The about page has been correctly updated! My view on some of the listed issues has evolved in recent years. You will now find those things struckthrough with comments added underneath.

Future Improvements

There is still minifying, javascript inlining, and font work to be done to make the page run faster. Additionally, the tags page is simply a disaster. Between all of that and the project pages, there is still a lot left to be done. Hopefully, all will be accomplished in time.

Conclusion #### {: #blog-updates-conclusion }

Hopefully all of these changes make the site a bit more enjoyable to use. I do understand if the use of analytics bothers you. Please make an issue in the tracker, or if someone already has, comment accordingly. If there are enough people using this site that honestly care, I might consider removing the analytics while I do further research.

ThothBackup - Part 1

As many people may have guessed, this backup system very quickly got much larger than I initially expected. Because of the size of the backup project, the number of people interested, and how quickly things are changing along the way, I’ve decided to approach this project in a new way.

In the sidebar to the left you will notice there is a new link to a “Projects” directory. Here you will be able to find all my larger works. The project is now called ThothBackup, and what follows is a list of things I have learned along the way. All of this data will be consolidated and entered in a more coherent fashion into the project pages, so keep an eye out for those to update.

But for now, we have a lot of ground to cover, so let’s get to work.

Part 1 - Cross Platform is hard

One of the biggest parts of the backup project was its ability to be cross-platform. I want the system to be easy enough to use that anyone and everyone could grab a client, get it configured, and get going. To facilitate this, the initial idea was to use tools that were built into the operating system. On Linux and macOS this is easy enough, as rsync is installed on most distributions by default, and if it isn’t installed it is just a quick package manager installation away.

Then however, entered Windows. Initially I assumed that it would be easy to use with windows as well. After all, rsync.net has a nice little guide explaining how to set it up. However, their client can detect when you’re not using rsync.net servers (which is totally fair, there’s no hate from me on that) and limits using the program to 30 days. The other alternative is cwRsync, which was initially freeware, but has since changed to being a paid product. Obviously asking someone to play for a program to even be able to start to use the backup isn’t a great selling point.

The first idea that I had was to write something on my own. Maybe have shell scripts on all platforms check for required code and fetch anything that is needed. However, shell scripts are hard for many people to debug, and the sight of a command prompt can strike fear into the hearts of many windows users.

The second iteration of the idea was to write something in Python. However at that point the client is becoming a software project in its own right, and I didn’t start this to develop software, I started it because I wanted to set up a neat little backup service for my friends and family.

Thankfully, there are many other software suites that are both cross platform and useful for this task. We ended up going with SyncThing. SyncThing is an open-source (MPL2, which is a permissive form of copyleft) synchronization library that is cross-platform and written in Go. I’m a huge fan of Go even though I don’t actually write it myself, as it is a fantastic language for exactly this type of thing. Even better, SyncThing comes with easy to use and easy to understand GUIs, and is capable of NAT and Firewall punching via relays, and makes device configuration dependent on acceptance from both the server and the client. The protocol it uses is open source, and based on the usage reports at least one person is using it on 30 million files with 2,000 peers. Last, but most certainly not least, traffic is encrypted with 128 bit AES, and the protocol maintains perfect forward secrecy.

All of this (and a whole lot more, it really is an awesome bit of software) makes SyncThing perfect for our use case. This may not always remain the case, but it gives me somewhere to start. Even if we end up moving beyond SyncThing in the future, you really should give it a look. It is a phenomenal piece of software.

Part 2 - Changes in Sync Methods

As hinted above, the original plan to synchronize systems wasn’t going to work without more work than I was willing to put in to a single component of the system. Once we threw out the initial way the system was supposed to work, we had to retool the way things worked on the operating system too.

The original way the sync process was meant to work was that every user’s operating system would get its own Server Account, and rsync or some other synchronization system would be tunneled through SSH. I wasn’t sure if authentication would be handled by system accounts or LDAP, because I never got that far. But I did specifically pick the operating system (OpenSUSE) because of that distribution’s system configuration manager (YaST).

Now with the use of SyncThing, a daemon process would run under a single user, and all clients would then connect to that daemon process which would then write to disk using that daemon’s permission set. Thus, no need to worry about ACLs or anything of the like. It was interesting to work with ACLs though. You can see some of my old code if you browse through the commits history of the ThothBackup GitHub repository.

Part 3 - Filesystem Considerations

When I was testing the original synchronization strategy, I had everything being deposited onto BTRFS subvolumes that were mounted with the compress option. To be entirely honest, I wasn’t that impressed with the way the compression was working.

In the new system, BTRFS subvolumes are still being used (User, System Name, Operating System, Drive Name, Backup Client, Archive Client, etc) except now the subvolumes are mounted with compress-force option. Additionally, I have learned about out-of-band BTRFS deduplication and plan to play around with that at this stage in the project as well.

Part 4 - Operating Systems

I really, really like OpenSUSE. Like, a whole whole lot. It may very well be my favorite binary distribution, and I’ve used quite a few. I think the whole way it works is simply phenomenal, I like the company behind it, and it honestly boils down to just that: I like it.

But, after all the changes above I began to consider if I shouldn’t change distributions. Originally I thought of changing to BSD, but I was concerned about software availability. I know FreeBSD tends to have a very well maintained ports collection, but I was still.. concerned. Most tools in this arena seem to cater toward Linux, and if I was already changing multiple systems to avoid having to write new software, did I really want to run the risk of needing to write server side software?

After much deliberation, I ended up settling on Debian Stable with backports. The initial installation is extremely lean, and there is a truly massive amount of documentation available for Debian. It paid off well too. The initial install of Debian stable clocked in at 60MB of ram used, where as OpenSUSE was running around 200MB after reboot.

Conclusion #### {: #thoth-1-conclusion }

There is honestly still quite a bit more that needs to be discussed. One of the most amusing things that the past week or so has taught me is that Sydney’s computer is as good of a backup test as a normal single family household. His System has 4 drives, over one million files, a quarter of a million directories, and about a terabyte of used storage on it. Combining his single computer with my mac and a windows virtual machine, and we have as much testing as we could need.

Notes #### {: #thoth-1-notes }

I’m going to start including a little section at the bottom of each post to remind me what I need to work on. Hopefully having this publicly viewable will encourage me to actually follow through on writing more than one blog post every 18 days.

  • stage 1 project page
  • stage 2 project page
  • talk about security improvements that can be done
  • rewrite the server side new client script
  • talk about specific SyncThing configuration options used
  • write utility script to keep server config files up to date in git