FreeNAS and alternatives?

[guddler]

Hi all,

Lot's of you play about with this stuff so here goes! Keeping up the disk theme, I bought 3 x 2TB drives for my HP Microserver yesterday. My current set up is thus...

(Server and services)
Host: ESXi 4.1 running from internal USB stick
Disk: 250GB SATA, bay 1. Has the datastore, ISOs, VMs etc.

CentosOne 1GB Mem
DNS
DHCP
Erm, etc.? Can't remember exacly but this is my general server

CentosTwo 1GB Mem (going to up to 2)
Work box - this has an instance of the work codebase running for testing. Well, they paid for the box after all!

vCMA 512K Mem
Lightweight VM for remote admin / access etc. with vSphere client

So. I just added another VM which is FreeNas 8.0.1-beta 3. I've given it 2GB ram (due to ZFS). I've created RAW VMDKs for the three new 2TB drives (bays 2 - 4) and added these to the FreeNas VM and created a RaidZ volume group with some datasets that are then shared using the FreeNas interface.

All fairly sweet.

So, I'm now starting to explore it. Before I go too mad loading stuff on I want to make sure it's all ok and the CPU is showing a bit odd in vSphere client:

Any thoughts on that one - pretty sure it's essentially idle.

A lot of people seem to talk of slow performance under FreeNas. I've not seen it yet, apart from copying 132GB of music onto it via USB took 12 hours (yes, really!!), but how do you measure this? People are quoting figures like 57Mb/s etc. - how do they know?

Finally, if the CPU or the performance proves to be a bit rubbish, any other suggestions for a setup on this box?

Couple of things to keep in mind:
1. Host Microserver only supports RAID0 and RAID1 (I want resilience really), and furthermore it's BIOS based so there's no ESXi drivers for it - I could switch to Hyper-V but I'd want RAID5 anyway really wouldn't I?
2. Actually, the other thing has completely slipped my mind! I'll edit if I think of it but I need to go look at some PCBs!!

Thoughts?

 

[guddler]

I've been chewing the fat on this one tonight. I may get rid of ESXi. While the flexibility of having the VMs and the ability to just throw a VM in at will, try it and junk it again is nice, I can do that on my desktop and ESXi is really causing me to get far more complicated than I need to.

Given that I've got 8GB of RAM and everything is linux based anyway, I think I might just go for one Centos VM, a software raid 5 (mdadm) with LVM on top. Only issue is how to go about migrating it - I really don't want to have to redo all the DDNS and stuff, that was a bitch!

And the 1 250G drive is a bit of a pain as well but I can't afford another 2TB one yet (and I don't need the space anyway). It just means I've got 1 drive that's not redundant.

Hmmm...

 

[kernow]

How strange, I have an HP microserver with freenas 7 on, set it up a couple of months ago. I love it. No virtualisation mind. *now reads thread*

Are those WD20EARS western digital 2TB drives you bought? I hope you used the wdidle3 tool to reduce the head parking value, apparently if its let default it can get dangerously close to its smart warning values in under a year - rendering the drive useless. I found a pre built ultimate boot cd ISO with the tool on and then just booted it from a pendrive using bootmyiso.exe, ran wdidle3 /s 300 and it just set both drives to 300 seconds instead of .. 8? Then rebooted and carried on with freenas setup as normal. I have 5GB ECC and it has never seemed to use over 38% mostly, just two simple zfs mirrors 2x160 and 2x2tb.

I keep logging into it to fiddle and realise its all set up and running, the way it should be really I guess.

Erm I have no idea about your problem though *spam*

kernow2011-07-14 10:19:00

 

[guddler]

That's interesting! As far as the drives go they're Hitachi's. A bit of a first for me I have to admit!

I'm thinking it may be one of two things. It could be the fact I'm using RaidZ across 3 drives and have only allocated 2G of ram. Though it's not showing high ram use. More likely, it could be the RAW drive mapping in ESXi. That's effectively introducing a software layer between the metal and zfs which probably isn't great?

I think I can use a VM to install a fresh copy of freenas to a new USB stick, remove my 250G drive for safety, boot and import the array. See how it runs without the VM. I am seeing pretty poor write performance at the moment. Kinda laggy and pausey.

I can't go that route permanently due to the other things I need the server to do but at least it will show me if that's the problem. If it is then I guess it's going to have to just become a stock Linux server with Samba or NFS shares etc.

 

[kernow]

Yeah, trying it without the vm layer and importing your zfs stuffs would probably be a good test. I love the little hp microserver though!

 

[guddler]

Oh, definately!! The very fact it can run ESXi with 4 or 5 VMs is amazing in itself for £99 + VAT!!

 

[gridrunner]

This does sound like an interesting bit of kit, will need to have a look. I use a Thecus N7700 with iSCSI targets and performance is respectable; but it's several hundred quid less disks, compared to £99!

As the previous poster suggested, dump the virtualisation layer for now as it may be contributing to your performance problems.

gridrunner2011-07-14 11:11:33

 

[guddler]

Created the install of FreeNas on a new USB stick and about to give it a whirl. Just doing some comparisons first as I'm on my laptop now which is technically a different network segment...

I guess I should clarify the price of the server

It was £99 + VAT after a £100 mail in rebate. But that money back came in a pretty timely fashion so it can be considered real. As stock it comes with a 1GB ram stick, I bought 2x4GB to replace that. It also comes with 1x250G HDD. I ran with just that for the first 4 months or so. Now that the price of 2TB's has dropped somewhat, I've jumped on 3x2TB @ £50.

So not mega, mega cheap, but all told for a dual core, 64bit processor (with VM tech.) based server, albeit 1.3Ghz, I think it's a damn fine bit of kit and I'd buy again in an instant if I was doing it again!

 

[trm]

Going back to your original problem, is that CPU graph for the VM host or for one of the individual VM guests (NAS VM)?

How many MHz per vertical graticule/division are there? If it's only a few tens of MHz then it's probably not relevant, but if that's a hundred MHz or more then something is definitely up. Are you seeing anything on the disk IO perf chart for host or guest?

The only times I've seen Vperf graphs like that are when something is broken

or the guest is doing something very specific (fixed load batch jobs via a minute crontab). No chance of a recurrence of the timer/tickless problem?

As for the RAW disk access, are you using RDM? (http://blog.davidwarburton.net/2010/10/25/rdm-mapping-of-local-sata-storage-for-esxi/) If you've got "disk-idle but unexplained activity visible" then I'd expect to see that activity recorded against the VM host and not the guest (the translation layer overhead is performed by the host not the guest), but that's based on ESX not ESXi although I doubt they're very different.

If you've gone for full disk 'fat' VMDKs then you should see even better throughput than a normal VMDK disk approach. But if you're looking at removing ESXi then there's not much point poking too far into this I guess.

Unfortunately can't recreate this on my ML110G6 as I went for Hyper-V as I need to do some certs for it.

 

[guddler]

CPU
Ummm, that graph is from the performance tab of the guest, not the host. Though the host obviously shows similar spikes. It's a graph of percentage CPU. The tallest points are about 21% and the bottom flat point is about 4%. The 4% is about right for the VM I would say. Slightly odd thing is that since last night, I'm now sometimes getting periods of calm, but not for long...:

Disk
Yes, I'm using that exact URL for RDM - at least the URL is 'visited' colour so I imagine I am. I did consider going the route of creating a single VMDK in the datastore that was the entire size of the disk (per disk) but didn't in the end as I thought performance would be better as RDM and I wasn't sure if restoration in the event of a disk failure might not be a bit of a nightmare? That was all based on pure speculation without any prior knowledge on my part though

How about the copying to / from performance? Might RaidZ have anything to do with that? I had the option of UFS too but I've no idea what that is so I went with ZFS / RaidZ

General
I'd way prefer to keep ESXi if I can. I just need to resolve these two performance items first. I love the flexibility that ESXi gives me. Though it's a right pain in the arse that they don't do a Mac version of the vSphere client. I'm generally using an XP VM on my Mac just to administer the server

What would your recommendation on the disks, and potentially the general NAS scenario be? I'm going to fire the thing up natively for a moment anyway just to guage performance and see if maybe FreeNas is just not doing so well under the VM, but they claim that most of their dev and testing is done on ESXi so I'd be surprised if it's specifically that?

I was wandering about just junking FreeNas and using my DDNS server (CentosOne) to do the sharing, having given it either the RAW disks (or VMDKs) that it can then software raid and use LVM on top of that?

 

[guddler]

Right, a little update...

A somewhat un-scientific experiment. Ie, a 5.1GB file (CD Image of OS X Lion so shouldn't be much blank space), a samba share and a stopwatch.

64bit FreeNas 8 in VM - ZFS RaidZ volume: 20MB/s

32bit FreeNas 8, physical - ZFS RaidZ volume: 32MB/s

32bit FreeNas 8, physical - UFS Raid3 volume: 35MB/s

Yep, I accidentally installed the 32bit version! So, still to do are UFS Raid3 on VM and 64bit physical UFS and ZFS. And I'll report back.

Can't really do much for now though as I've got to upload about 250MB of data to work and that's going to take 3hrs or so (I've got a terrible upload speed on ADSL Max). Can't really do measurements while that's going, wouldn't be a fair comparison.

 

[trm]

The bit I was keen to see are the markings in the far right of the snapshot you just included. Looking at that the darker lines are units of 750MHz and the lighter lines indicate 187.5 MHz.

Assuming the first screen cap has the same intervals it looks like last nights' VM performance was peaking around 500-ish MHz and averaging just above 220MHz (bit weird getting used to the VMware way of measuring load through completely arbitrary MHz units).

From that and the fact it seems to have slowed a bit today I'd probably think it was ZFS finalising the disk structuring. What does the disk perf chart for that guest show when compared to the CPU spiking? If I'm anywhere near the right lines then I'd expect to see a similar pattern on disk IOPs vs CPU util.

Going back to the future options, I understand that RDM is the fastest native disk IO path unless you can get a PCI(e) disk controller card and use VMDirectPath IO, but the last time I looked you had to dedicate the entire controller and all devices on it to VMDP IO so it's no good if your boot volumes are on the same controller as your data disks.

(I wrote this earlier when I saw your response above so I know some bits have been discussed on IRC but figured most of this was still relevant, so when I saw I'd not posted it I figured I would)

 

[guddler]

Ok, still plodding away with this!

Before the weekend I built a linux software raid 5 array using mdadm and then added LVM partitioning and volumes on top of that. Very nice in theory. This was all done in my Centos VM and the volume shared using Samba. Initial build of the array took 24 hours. Twice! I switched the server off when we went away for the weekend assuming it would just resume where it left off but when I started it up again Sunday evening it started from the beginning. So that's oddity number one. Can it be trusted??

So, now that it's all built I've run the same test as above, copying the same file, from the same machine on the same network segment. The result is frankly very difficult to believe. In a bad way. I get an incredibly bad 5.2MB/s sustained write speed with the 5.12GB file. Further more, when I delete that file the disks thrash like buggery for about 5 minutes.

Ok, so won't be going that route then

So, it looks like, unless anyone can suggest otherwise, the best I can hope for so far is a VM running FreeNas with a RaidZ RDM array. Hopefully I'll at least get my 20MB/s, and maybe it was doing it's initial build and things may speed up a bit?

 

[hellraiseruk]

Have you tried installing the VMware tools under your FreeNAS box? These are basically enhanced drivers which should boost network/disk performance under ESXi. Might also be worth using FreeNAS7 instead of 8, as 8 seems to have ridiculously high system requirements!

Ideally you should be using the vmxnet3 adapter for your VM, this is the one that does all the funky stuff like jumbo frames and TCP offloading - can make a big difference!

Ought to add, if you're doing RAIDZ, (or RAID 5 etc) then as it is software RAID, that poor little CPU is doing all the parity calculations when you write to it. This will impact quite negatively on performance, so might be an idea to create a single disk volume with no RAID and benchmark that...

Hellraiser...........>
hellraiseruk2011-07-27 11:45:15

 

[guddler]

Thanks for this info - I've not tried installing the tools, I was under the impression that it came with them as it seems to detect vmware on boot or something. I'll certainly investigate it though, anything I can do to improve the performance then all well and good!

 

[guddler]

Have you tried installing the VMware tools under your FreeNAS box?

Yep. Spent a few hours this evening setting up a new VM that I could install VSphere client on (how I wish they'd produce an OS X version of it). After all that, I fail at the first hurdle, which is './vmware-install.pl'.

No perl installed under FreeNAS.

Any ideas?