What speed can you get over ssh to that board? The big issue I had with the Raspberry Pi 4 when using it as a low-powered server is when I run my backups to it over ssh, the network transfers top out at about 20 MB/s (just raw transfer speed going to /dev/null) vs. around 100MB/s I get on a regular x86 board.
I benchmarked it right now by copying a 6.1 GB file with scp from my laptop to the HC4 SSD. It eventually peaked between 77 and 80 MB/s (that's a B for bytes.) I have a 1 Gigabit interface and a 10 Gb/s cable so it reasonably saturated the interface. There were maybe another 10 or 20 MB/s to squeeze from it but I won't complain. It took 1m 13s.