Asustor Flashstor FS6812x/FS6806x — Experimental TrueNAS Support for AMD XGBE — Full Kernel Compile

After the success of getting Debian to work with the custom AMD patched kernel, I tried to see if I could get TrueNAS working as well — and yes, it works great using the same kernel and modules compiled for Debian.

Note: These instructions are not necessary for getting the AMD XGMAC 10 GbE NIC running on the Asustor Flashstor Gen 2 devices, it is enough to compile just the amd-xgbe module as described here. I'm only keeping them here for archival and reference purposes, or in case full kernel compilation is needed or desired.

I have to point out that I had never used TrueNAS before this, and had to fight with it to get even the smallest thing going (like editing a file)... I understand that "it's an appliance", but situations such as these that require a bit more tweaking of the system become impossible in that case. Anyway, it works, but I'm sure all of this breaks the "TrueNAS Warranty" of sorts, there won't be any support and everything will be reset on upgrade. Treat it all as experimental only, not to be used in production or with critical data.

Here are the steps to get this going:

1. Connect to TrueNAS, either via a USB NIC or directly on the console
2. Disable protections — WARNING: from this point forward, the configuration will become unsupported —

   • $ sudo su
     # /usr/local/libexec/disable-rootfs-protection

3. Make partitions writable:

   • # mount -o remount,rw 'boot-pool/ROOT/24.10.0.2/'
     

   • This needs to be redone after every reboot
4. Edit /etc/shadow and enable the root account, by copying the password hash from your main truenas_admin account
   • This needs to be redone after every reboot
5. Exit sudo and run su - instead with the truenas_admin password you just used
   • Note: we can't use sudo, as it has some strong hardening patches that kill processes it considers doing anything illegal, such as attempting to install a new kernel; I assume this is all done for a good reason, but it's very aggressive, with no logs or warnings to explain what's going on
6. You will need an actual Debian installation to complete this, since the ZFS modules are mandatory, and they don't come with the AMD kernel sources or the Debian kernel. This would be much easier if TrueNAS supported a form of DKMS, but I wasn't able to find anything like that. Thus, we need to install ZFS on Debian, compile them with DKMS for the AMD patched kernel, then copy over the ZFS modules to TrueNAS...
   • First follow the instructions right here to do this in a VM
   • Once the .debs are created and installed within the VM, add contrib and backports to sources.list, then run the following:

   • # apt update
     # apt install -t stable-backports zfsutils-linux zfs-dkms
     

   • This should take care of compiling the ZFS modules as well, including for the AMD patched kernel
7. Copy over the .deb packages compiled for Debian to the TrueNAS machine
8. Install the packages — This needs to be done under su, not sudo:

   • # apt install ./*.deb

9. Copy over /lib/modules/6.6.43asustor from the Debian VM to the TrueNAS machine (this includes the critical ZFS modules!
10. Update the initrd:

    • # update-initramfs -u -k 6.6.43asustor
      

11. Add an entry to GRUB:
    • Edit /boot/grub/grub.cfg and copy one of the existing menuentries, changing all instances of 6.6.44-production+truenas to 6.6.43asustor
    • This needs to be redone after every reboot
12. Reboot, choose the modified GRUB entry and check that everything is working
    • Note: If you've used a separate NIC to configure things so far, the AMD NIC will not be enabled properly by TrueNAS, because it only gets a link when using DHCP or having an IP assigned statically, and TrueNAS insists on only having DHCP on one NIC at a time. To solve this:
      • First, move the Ethernet cable to the AMD NIC
      • Second, go to the local TrueNAS console, open a shell and run dhclient enpXXX to start DHCP discovery on the AMD NIC
      • Once an IP is assigned, give it a few minutes for the TrueNAS scripts to restart the WebUI on the new IP
      • Go into the Network settings and move over the DHCP checkbox to the AMD NIC, test then save (this will cause a temporary disconnection, should be ok after that)
      • You can also assign static IPs to each interface to force them to come up without DHCP
13. Make it permanent — WARNING: highly hackish, not The TrueNAS Way, etc. —
    • In your home directory:

      • # cd /home/truenas_admin
        

    • Make a copy of the "fixed" grub.cfg file:

      • # cp /boot/grub/grub.cfg .
        

    • Edit it, copy the top menuentry into a new one, then edit the top one and modify the linux and initrd lines to point to the 6.6.43asustor versions, then modify the gnulinux-simple-xxx line to make it unique
    • Copy it over:

      • # cp grub.cfg /boot/grub/fixed-grub.cfg
        

    • Go to System -> Advanced Settings -> Add Init/Shutdown Script
    • Add Command to run as Post Init with the following content:
      • /home/truenas_admin/fix-grub.sh

    • fix-grub.sh:
      
      #!/bin/sh
      sleep 60s
      mount -o remount,rw 'boot-pool/ROOT/24.10.0.2/'
      cp /boot/grub/fixed-grub.cfg /boot/grub/grub.cfg
      mount -o remount,ro 'boot-pool/ROOT/24.10.0.2/'
      

    • The sleep should ensure that the file is copied back late enough in the boot process, after the TrueNAS scripts reset the original (otherwise it would get overwritten again)

I was able to confirm that this brings up the Flashstor with TrueNAS and the network working and is stable throughout a few reboots. However, it's clearly an unsupported configuration, and running a non-TrueNAS kernel may have other consequences later on that are not evident yet (certain apps/tools not working, unsupported modules, etc.). Ideally, the AMD patches for the XGBE NIC should be part of the upstream kernel, and then TrueNAS would "just work" as soon as it delivered an updated kernel. Alas, for now, the only course of action remains to ask the developers to add these patches as part of a bug/feature request.