Does anyone know the best hardware or software setup for TruNAS? I setup trunas a few months ago, and really like it. minus really poor performance. Which might be my hardware. The internet has just confused me more. Searching and reading speeds of Sata, SSD, and pcie is not straight. . . I am not sure where to even look to make changes for the better. Right now I have a AMD phenom II 6x with 16gig and 1 gig Ethernet. With seven Sata drives 2 of which are hot spares. My thoughts now. Are as follows. Pcie Sata card? Better nic? More ram? Bad settings? Any help would be wonderful.
Are you booting off a flash drive instance of TruNAS (recommended setup afaik)? If so, is it USB 3 on both ends? Other than that, just make sure the drives are 7200rpm and latest SATA spec should be fine. I don't use TruNAS itself, but use zfs pools on two physical hosts, which is what TruNAS is using "under the hood" So, I would check your boot media to begin with, then check which SATA implementation you have 1.5, 3, or 6 ... Mbps, and rpms of hard drive. Good luck. JMH On 8/10/22 3:39 PM, Wesley Robbins wrote:
Does anyone know the best hardware or software setup for TruNAS? I setup trunas a few months ago, and really like it. minus really poor performance. Which might be my hardware. The internet has just confused me more. Searching and reading speeds of Sata, SSD, and pcie is not straight. . . I am not sure where to even look to make changes for the better.
Right now I have a AMD phenom II 6x with 16gig and 1 gig Ethernet. With seven Sata drives 2 of which are hot spares.
My thoughts now. Are as follows. Pcie Sata card? Better nic? More ram? Bad settings?
Any help would be wonderful.
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org -- *Jonathan Haack* Haack's Networking phone: 505-310-6638 nextcloud: Haack's Networking <https://nextcloud.haacksnetworking.org> email: jonathan@haacksnetworking.org Haack's Networking <https://haacksnetworking.org/>
Gbps* On Sat, Aug 13, 2022 at 1:22 PM Jonathan Haack <jonathan@jonathanhaack.com> wrote:
Are you booting off a flash drive instance of TruNAS (recommended setup afaik)? If so, is it USB 3 on both ends?
Other than that, just make sure the drives are 7200rpm and latest SATA spec should be fine. I don't use TruNAS itself, but use zfs pools on two physical hosts, which is what TruNAS is using "under the hood"
So, I would check your boot media to begin with, then check which SATA implementation you have 1.5, 3, or 6 ... Mbps, and rpms of hard drive.
Good luck.
JMH On 8/10/22 3:39 PM, Wesley Robbins wrote:
Does anyone know the best hardware or software setup for TruNAS? I setup trunas a few months ago, and really like it. minus really poor performance. Which might be my hardware. The internet has just confused me more. Searching and reading speeds of Sata, SSD, and pcie is not straight. . . I am not sure where to even look to make changes for the better.
Right now I have a AMD phenom II 6x with 16gig and 1 gig Ethernet. With seven Sata drives 2 of which are hot spares.
My thoughts now. Are as follows. Pcie Sata card? Better nic? More ram? Bad settings?
Any help would be wonderful.
_______________________________________________ nmglug mailing listnmglug@lists.nmglug.orghttp://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org
-- *Jonathan Haack* Haack's Networking phone: 505-310-6638 nextcloud: Haack's Networking <https://nextcloud.haacksnetworking.org> email: jonathan@haacksnetworking.org [image: Haack's Networking] <https://haacksnetworking.org/>
Thanks. Ended up getting new hardware and did a total rebuild. But this time built the root disk on two usb disks. Now I have a 9+2 zfs raid. :). If you are interested, bought two asm1166 Sata cards that are 4xpcie 3.0. The 1x pcie Sata card seemed to be the biggest issue. It was really old. On Sat, Aug 13, 2022 at 1:43 PM Jonathan Haack <jonathan@jonathanhaack.com> wrote:
Gbps*
On Sat, Aug 13, 2022 at 1:22 PM Jonathan Haack <jonathan@jonathanhaack.com> wrote:
Are you booting off a flash drive instance of TruNAS (recommended setup afaik)? If so, is it USB 3 on both ends?
Other than that, just make sure the drives are 7200rpm and latest SATA spec should be fine. I don't use TruNAS itself, but use zfs pools on two physical hosts, which is what TruNAS is using "under the hood"
So, I would check your boot media to begin with, then check which SATA implementation you have 1.5, 3, or 6 ... Mbps, and rpms of hard drive.
Good luck.
JMH On 8/10/22 3:39 PM, Wesley Robbins wrote:
Does anyone know the best hardware or software setup for TruNAS? I setup trunas a few months ago, and really like it. minus really poor performance. Which might be my hardware. The internet has just confused me more. Searching and reading speeds of Sata, SSD, and pcie is not straight. . . I am not sure where to even look to make changes for the better.
Right now I have a AMD phenom II 6x with 16gig and 1 gig Ethernet. With seven Sata drives 2 of which are hot spares.
My thoughts now. Are as follows. Pcie Sata card? Better nic? More ram? Bad settings?
Any help would be wonderful.
_______________________________________________ nmglug mailing listnmglug@lists.nmglug.orghttp://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org
-- *Jonathan Haack* Haack's Networking phone: 505-310-6638 nextcloud: Haack's Networking <https://nextcloud.haacksnetworking.org> email: jonathan@haacksnetworking.org [image: Haack's Networking] <https://haacksnetworking.org/>
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org
I use a LSI 9211-8i (in IT mode) in my home media server. However, if I were to build another home media server today, this is what I would use. https://www.ebay.com/itm/144349058469 Any LSI HBA flashed to IT mode for direct disk passthru would work. Since you are wanting to use TrueNAS Scale, the options for HBA are a little more lax. But for FreeBSD based TrueNAS Core, you will want to stick with LSI. Just don't make sure you don't do hardware RAID, ZFS wants/needs to directly "talk" to each disk... I know people that use the onboard SATA ports on their motherboard as well, but for that many disks, you will typically need a PCIe card. As for performance, what model of drives are you using? In general, you will want to stick to using the same model of drive. Also, avoid SMR disks LIKE THE PLAGUE. You will basically need to stick to CMR disks (also known as PMR). Also, you will notice performance degradation when you use more than 80% capacity... https://www.wundertech.net/cmr-vs-smr-what-is-the-best-hard-drive/ Now, there are some issues with random 4K writes on TrueNAS Scale. https://www.youtube.com/watch?v=BoiHHnBDg0E&t=220s But they are working on it... https://www.truenas.com/community/threads/performance-testing-core-12-vs-13-... So basically, if you need ZFS but don't need to use docker. Then I recommend sticking with TrueNAS Core. If you need docker, then I would use TrueNAS Scale. For RAM, throw as much RAM as you can possible throw at it. ZFS caches it's data and metadata in RAM, though you can always tune this if you need some to save some RAM for containers or VMs. ECC is generally better, but it's not 100% necessary. What IS 100% necessary is to have multiple backups of your data! I would also recommend sticking with newer hardware, mostly for the power efficiency and higher performance with containers. Especially if TrueNAS is left powered on 24/7... https://www.youtube.com/watch?v=RWmfZHn2wLs As for the boot drive and USB support. The old recommendation is to use fast USB drives. However, that isn't true anymore. Especially for TrueNAS Scale since you most likely are going to use containers. https://www.truenas.com/docs/scale/gettingstarted/scalehardwareguide/#minimu... Hope this helps! ~ Jared On 8/14/22 21:11, Wesley Robbins wrote:
Thanks. Ended up getting new hardware and did a total rebuild. But this time built the root disk on two usb disks. Now I have a 9+2 zfs raid. :).
If you are interested, bought two asm1166 Sata cards that are 4xpcie 3.0. The 1x pcie Sata card seemed to be the biggest issue. It was really old.
On Sat, Aug 13, 2022 at 1:43 PM Jonathan Haack <jonathan@jonathanhaack.com <mailto:jonathan@jonathanhaack.com>> wrote:
Gbps*
On Sat, Aug 13, 2022 at 1:22 PM Jonathan Haack <jonathan@jonathanhaack.com <mailto:jonathan@jonathanhaack.com>> wrote:
Are you booting off a flash drive instance of TruNAS (recommended setup afaik)? If so, is it USB 3 on both ends?
Other than that, just make sure the drives are 7200rpm and latest SATA spec should be fine. I don't use TruNAS itself, but use zfs pools on two physical hosts, which is what TruNAS is using "under the hood"
So, I would check your boot media to begin with, then check which SATA implementation you have 1.5, 3, or 6 ... Mbps, and rpms of hard drive.
Good luck.
JMH
On 8/10/22 3:39 PM, Wesley Robbins wrote:
Does anyone know the best hardware or software setup for TruNAS? I setup trunas a few months ago, and really like it. minus really poor performance. Which might be my hardware. The internet has just confused me more. Searching and reading speeds of Sata, SSD, and pcie is not straight. . . I am not sure where to even look to make changes for the better.
Right now I have a AMD phenom II 6x with 16gig and 1 gig Ethernet. With seven Sata drives 2 of which are hot spares.
My thoughts now. Are as follows. Pcie Sata card? Better nic? More ram? Bad settings?
Any help would be wonderful.
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org <mailto:nmglug@lists.nmglug.org> http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org <http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org>
-- *Jonathan Haack* Haack's Networking phone: 505-310-6638 nextcloud: Haack's Networking <https://nextcloud.haacksnetworking.org> email: jonathan@haacksnetworking.org <mailto:jonathan@haacksnetworking.org> Haack's Networking <https://haacksnetworking.org/>
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org <mailto:nmglug@lists.nmglug.org> http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org <http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org>
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org
Thanks for all the info. Will be really looking into the CMR / SMR disks. For my disks I am running 9 small SSD drives. Might be worth buying 32g memory. As a side note I erased my first build and went with default install on new hardware with pcie 3.0. Seems to have made a big difference. On Fri, Aug 19, 2022 at 5:35 AM ABQLUG Events <community@abqlug.com> wrote:
I use a LSI 9211-8i (in IT mode) in my home media server. However, if I were to build another home media server today, this is what I would use.
https://www.ebay.com/itm/144349058469
Any LSI HBA flashed to IT mode for direct disk passthru would work. Since you are wanting to use TrueNAS Scale, the options for HBA are a little more lax. But for FreeBSD based TrueNAS Core, you will want to stick with LSI. Just don't make sure you don't do hardware RAID, ZFS wants/needs to directly "talk" to each disk...
I know people that use the onboard SATA ports on their motherboard as well, but for that many disks, you will typically need a PCIe card.
As for performance, what model of drives are you using? In general, you will want to stick to using the same model of drive. Also, avoid SMR disks LIKE THE PLAGUE. You will basically need to stick to CMR disks (also known as PMR). Also, you will notice performance degradation when you use more than 80% capacity...
https://www.wundertech.net/cmr-vs-smr-what-is-the-best-hard-drive/
Now, there are some issues with random 4K writes on TrueNAS Scale.
https://www.youtube.com/watch?v=BoiHHnBDg0E&t=220s
But they are working on it...
https://www.truenas.com/community/threads/performance-testing-core-12-vs-13-...
So basically, if you need ZFS but don't need to use docker. Then I recommend sticking with TrueNAS Core. If you need docker, then I would use TrueNAS Scale.
For RAM, throw as much RAM as you can possible throw at it. ZFS caches it's data and metadata in RAM, though you can always tune this if you need some to save some RAM for containers or VMs. ECC is generally better, but it's not 100% necessary. What IS 100% necessary is to have multiple backups of your data!
I would also recommend sticking with newer hardware, mostly for the power efficiency and higher performance with containers. Especially if TrueNAS is left powered on 24/7...
https://www.youtube.com/watch?v=RWmfZHn2wLs
As for the boot drive and USB support. The old recommendation is to use fast USB drives. However, that isn't true anymore. Especially for TrueNAS Scale since you most likely are going to use containers.
https://www.truenas.com/docs/scale/gettingstarted/scalehardwareguide/#minimu...
Hope this helps!
~ Jared
On 8/14/22 21:11, Wesley Robbins wrote:
Thanks. Ended up getting new hardware and did a total rebuild. But this time built the root disk on two usb disks. Now I have a 9+2 zfs raid. :).
If you are interested, bought two asm1166 Sata cards that are 4xpcie 3.0. The 1x pcie Sata card seemed to be the biggest issue. It was really old.
On Sat, Aug 13, 2022 at 1:43 PM Jonathan Haack <jonathan@jonathanhaack.com <mailto:jonathan@jonathanhaack.com>> wrote:
Gbps*
On Sat, Aug 13, 2022 at 1:22 PM Jonathan Haack <jonathan@jonathanhaack.com <mailto:jonathan@jonathanhaack.com>> wrote:
Are you booting off a flash drive instance of TruNAS (recommended setup afaik)? If so, is it USB 3 on both ends?
Other than that, just make sure the drives are 7200rpm and latest SATA spec should be fine. I don't use TruNAS itself, but use zfs pools on two physical hosts, which is what TruNAS is using "under the hood"
So, I would check your boot media to begin with, then check which SATA implementation you have 1.5, 3, or 6 ... Mbps, and rpms of hard drive.
Good luck.
JMH
On 8/10/22 3:39 PM, Wesley Robbins wrote:
Does anyone know the best hardware or software setup for TruNAS? I setup trunas a few months ago, and really like it. minus really poor performance. Which might be my hardware. The internet has just confused me more. Searching and reading speeds of Sata, SSD, and pcie is not straight. . . I am not sure where to even look to make changes for the better.
Right now I have a AMD phenom II 6x with 16gig and 1 gig Ethernet. With seven Sata drives 2 of which are hot spares.
My thoughts now. Are as follows. Pcie Sata card? Better nic? More ram? Bad settings?
Any help would be wonderful.
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org <mailto:nmglug@lists.nmglug.org> http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org <
http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org> -- *Jonathan Haack* Haack's Networking phone: 505-310-6638 nextcloud: Haack's Networking <https://nextcloud.haacksnetworking.org> email: jonathan@haacksnetworking.org <mailto:jonathan@haacksnetworking.org> Haack's Networking <https://haacksnetworking.org/>
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org <mailto:nmglug@lists.nmglug.org> http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org <http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org>
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org
nmglug mailing list nmglug@lists.nmglug.org http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org
Lots of great stuff in Jared's reply. I recently built a TrueNas Scale system that I have been pretty happy with. Using a Dell T110 II with 32GB of ram and a Xeon E3-1240. We used systems like these where I work and they seem power friendly for an old Xeon system. I put in it a9207-8i LSI card in ITmode. I am running 4, 8TB Hitachi/WB HGST Ultrastar drives. I have a 10GB Mellanox ConnectX-2 card in the machine as well. I have been very happy with the setup. It is far faster than the MediaVault NAS solution that I had been using, but the speed issues I had were in no way MediaVaults fault. MediaVault does a fine job. I landed on TrueNAS as I was attempting to build a system that would compare well to a Synology NAS that a friend had been looking at. I told him I could build something better for less. I succeeded, granted I got to make use of some less expensive used parts. Performance is performance. TrueNas Core is more polished. I built my machine and started testing in Core. Then I move to the beta of Scale. I decided to stick with Scale when they made the release version. Either would have worked for me. Yeah, SMR disks poison performance. Avoid those. Just saw your reply, sounds like you found the culprit. On Fri, Aug 19, 2022 at 5:35 AM ABQLUG Events <community@abqlug.com> wrote:
I use a LSI 9211-8i (in IT mode) in my home media server. However, if I were to build another home media server today, this is what I would use.
https://www.ebay.com/itm/144349058469
Any LSI HBA flashed to IT mode for direct disk passthru would work. Since you are wanting to use TrueNAS Scale, the options for HBA are a little more lax. But for FreeBSD based TrueNAS Core, you will want to stick with LSI. Just don't make sure you don't do hardware RAID, ZFS wants/needs to directly "talk" to each disk...
I know people that use the onboard SATA ports on their motherboard as well, but for that many disks, you will typically need a PCIe card.
As for performance, what model of drives are you using? In general, you will want to stick to using the same model of drive. Also, avoid SMR disks LIKE THE PLAGUE. You will basically need to stick to CMR disks (also known as PMR). Also, you will notice performance degradation when you use more than 80% capacity...
https://www.wundertech.net/cmr-vs-smr-what-is-the-best-hard-drive/
Now, there are some issues with random 4K writes on TrueNAS Scale.
https://www.youtube.com/watch?v=BoiHHnBDg0E&t=220s
But they are working on it...
https://www.truenas.com/community/threads/performance-testing-core-12-vs-13-...
So basically, if you need ZFS but don't need to use docker. Then I recommend sticking with TrueNAS Core. If you need docker, then I would use TrueNAS Scale.
For RAM, throw as much RAM as you can possible throw at it. ZFS caches it's data and metadata in RAM, though you can always tune this if you need some to save some RAM for containers or VMs. ECC is generally better, but it's not 100% necessary. What IS 100% necessary is to have multiple backups of your data!
I would also recommend sticking with newer hardware, mostly for the power efficiency and higher performance with containers. Especially if TrueNAS is left powered on 24/7...
https://www.youtube.com/watch?v=RWmfZHn2wLs
As for the boot drive and USB support. The old recommendation is to use fast USB drives. However, that isn't true anymore. Especially for TrueNAS Scale since you most likely are going to use containers.
https://www.truenas.com/docs/scale/gettingstarted/scalehardwareguide/#minimu...
Hope this helps!
~ Jared
On 8/14/22 21:11, Wesley Robbins wrote:
Thanks. Ended up getting new hardware and did a total rebuild. But this time built the root disk on two usb disks. Now I have a 9+2 zfs raid. :).
If you are interested, bought two asm1166 Sata cards that are 4xpcie 3.0. The 1x pcie Sata card seemed to be the biggest issue. It was really old.
On Sat, Aug 13, 2022 at 1:43 PM Jonathan Haack <jonathan@jonathanhaack.com <mailto:jonathan@jonathanhaack.com>> wrote:
Gbps*
On Sat, Aug 13, 2022 at 1:22 PM Jonathan Haack <jonathan@jonathanhaack.com <mailto:jonathan@jonathanhaack.com>> wrote:
Are you booting off a flash drive instance of TruNAS (recommended setup afaik)? If so, is it USB 3 on both ends?
Other than that, just make sure the drives are 7200rpm and latest SATA spec should be fine. I don't use TruNAS itself, but use zfs pools on two physical hosts, which is what TruNAS is using "under the hood"
So, I would check your boot media to begin with, then check which SATA implementation you have 1.5, 3, or 6 ... Mbps, and rpms of hard drive.
Good luck.
JMH
On 8/10/22 3:39 PM, Wesley Robbins wrote:
Does anyone know the best hardware or software setup for TruNAS? I setup trunas a few months ago, and really like it. minus really poor performance. Which might be my hardware. The internet has just confused me more. Searching and reading speeds of Sata, SSD, and pcie is not straight. . . I am not sure where to even look to make changes for the better.
Right now I have a AMD phenom II 6x with 16gig and 1 gig Ethernet. With seven Sata drives 2 of which are hot spares.
My thoughts now. Are as follows. Pcie Sata card? Better nic? More ram? Bad settings?
Any help would be wonderful.
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org <mailto:nmglug@lists.nmglug.org> http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org <
http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org> -- *Jonathan Haack* Haack's Networking phone: 505-310-6638 nextcloud: Haack's Networking <https://nextcloud.haacksnetworking.org> email: jonathan@haacksnetworking.org <mailto:jonathan@haacksnetworking.org> Haack's Networking <https://haacksnetworking.org/>
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org <mailto:nmglug@lists.nmglug.org> http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org <http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org>
_______________________________________________ nmglug mailing list nmglug@lists.nmglug.org http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org
nmglug mailing list nmglug@lists.nmglug.org http://lists.nmglug.org/listinfo.cgi/nmglug-nmglug.org
-- Jerred Weingarten "Really, I'm not out to destroy Microsoft. That will just be a completely unintentional side effect." Linus Torvalds, September 28, 2003
participants (4)
-
ABQLUG Events -
Jerred Weingarten -
Jonathan Haack -
Wesley Robbins