Comparison of HAAst to Commercial HA and Open Source HA

Features, capabilities, and information about HAAst
User avatar
johnley
Posts: 2
Joined: Mon Feb 06, 2017 8:14 pm

Comparison of HAAst to Commercial HA and Open Source HA

Postby johnley » Mon Feb 06, 2017 8:40 pm

We just completed (what was supposed to be) a 30 day comparison of HAAst to Open Source and FreePBX HA solutions, on behalf of a customer deploying Asterisk to retail outlets in major cities across California. After being pretty shocked by what we found I wanted to post the results on some forums for others to see.

First of all, the comparison didn't last beyond one week because of what we found. Here are some highlights of our tests:

  • Runaway process: We created a process which used almost 100% CPU. The FreePBX HA and Open Source solutions just sat there frozen or sluggish for an hour (until we manually rebooted them). Calls had choppy audio or didn't connect - just dropped. HAAst on the other hand immediately detected the problem and failed over to the other node.
  • I/O Overload: We created a process simulating heavy disk IO. The FreePBX HA and Open Source solutions acted up, froze intermittently (and dropped calls), etc! HAAst on the other hand immediately detected the problem and failed over to the other node.
  • Unplug the T1: FreePBX HA and Open Source didn't notice or care about the missing trunk; the first node just sat there not processing any calls! In contrast, HAAst detected the failure immediately and failed over. (By the way, HAAst could move the T1 trunk automatically, while the other 2 solutions required that we move a cable)
  • Fail SIP provider access: FreePBX HA and Open Source didn't notice or care that the SIP provider was no longer reachable; the first node just sat there not processing any calls! Again, HAAst in contrast immediately detected the failure and failed over.
  • Corrupt Asterisk: We corrupted the Asterisk process (overload the AMI socket), and FreePBX HA and Open Source didn't care, Asterisk just sat there brain dead, not bridging calls, but process alive. HAAst detected the Asterisk engine was misbehaving and failed over to the other node immediately.
  • Corrupt the disk: We simply wrote "1"'s to a file taking up all space. FreePBX HA and Open Source didn't notice or care until the disk was full (Asterisk just started misbehaving), and they just sat there unable to process calls; briding didn't work, calls hungup on arrival, etc. When we manually failed over the FreePBX HA and Open Source clusters the other node would not start! The disk space problem was mirrored to the other node! HAAst not only detected the Asterisk failure, it failed over automatically, and it did NOT corrupt the other node, and did not fill the other disk with "1"'s.
and more! The only test that all 3 systems handled equally well was unplugging the power on one node and the other node took over immediately. I'm not surprised by the Open Source results since it's up to me the admin to build whatever I want using my own code (write my own HA detection/recovery code, etc). However, after spending $3000 on the FreePBX HA software I was pretty shocked to find that it did nothing! There are no sensors! It turns out we spent $3000 just to get the open source products pre-installed into FreePBX, and that's it. (We discovered that FreePBX HA sells us the same open source packages we downloaded for free). Wow! And by the way, kudos to Telium for offering a free version so we could try it before buying.

I have posted the above on other sites where we compared products (one of them immediately took down my message by the way!!!). If you want details please email me [email removed] and I would be happy to share all of the test details and results.

One piece of advice Telium: get a marketing department! Why don't you advertise the difference between your products? The other guys have much better brochures, videos, and marketing stuff!

John Ley
Telecom Consultant
User avatar
Telium Support
Posts: 162
Joined: Sun Nov 27, 2016 3:27 pm

Re: Comparison of HAAst to Commercial HA and Open Source HA

Postby Telium Support » Mon Feb 06, 2017 9:02 pm

Thank you for the positive comments. A lot of our customers actually switch to HAAst after the built-in/open source solution fails in real life scenarios.

I should point out that all other competing HA products are based on the open source packages (and in the case of FreePBX HA you are paying $3000 for these free packages). These are great packages, meant for file level HA - not application level HA! There are many reasons that these packages don't constitute a proper Asterisk HA solution, not the least of which being that corruption on one peer is immediately copied to the other peer, they are not aware of VoIP issues, they don't have visibility into Asterisk/trunks/routes/etc.

You might be surprised that Elastix v4 offers (for free) the exact same 'HA' solution that FreePBX sells (both based on the same open source packages). They show you how to setup the open source modules so Elastix HA works identically to FreePBX HA (thumbs up to Elastix for being honest).

We don't actually compete with open source / FreePBX HA / Elastix HA modules - those tend to by hobbyists in home office / small office installations that aren't heavily impacted by a phone outage. HAAst is targeted squarely at commercial PBX's and critical call centers. There's a reason why HAAst is the only solution chosen by 911 / PSAP / healthcare / etc call centers. HAAst clusters do not share any logical or physical devices, have real sensors to detect a broad range of sensors, only synchronize data if a peer is healthy (so you never sync corrupted data), and more.

I will pass your feedback along to the Sales & Marketing group. Our philosophy to date has been 'our product speaks for itself'. But kind words from customers are always appreciated! I'll leave your post pinned for others so they can conduct similar tests.

By the way - I removed your email address from your message otherwise you will get lots of spam (our forums are indexed by all major search engines). Users are welcome to contact you using the forum Private Message feature.
iblurker
Posts: 5
Joined: Wed Feb 08, 2017 3:05 am

Re: Comparison of HAAst to Commercial HA and Open Source HA

Postby iblurker » Wed Feb 08, 2017 1:45 pm

Johnley: I would like to do my own test (of my current cluster and HAAst) to try this for myself. Can you share details of how you tested so that I can do my own tests?

I installed HAAst in a virtual machine (ESXi) and can test it easily, but I don't want to destroy my primary phone cluster (which came with built-in clustering with the FreePBX system). So only non-destructive tests only if possible.

Thank you.
User avatar
johnley
Posts: 2
Joined: Mon Feb 06, 2017 8:14 pm

Re: Comparison of HAAst to Commercial HA and Open Source HA

Postby johnley » Wed Feb 08, 2017 6:12 pm

Blurker: You can perform the first few tests easily, but I suggest that you do them after business hours because you will have lengthy phone outages if you try this using the FreePBX HA module. Here are some steps so you can test this for yourself:

  • Runaway process: We used a CPU load testing program (http://weather.ou.edu/~apw/projects/stress/) for our tests but you can probably accomplish the same with a three line bash script to create CPU load:

    Code: Select all

    #!/bin/bash
    while true; do
      ( while :; do :; done ) &
    done
  • I/O Overload: We used an I/O load testing program (http://gerardnico.com/wiki/io/iometer) for our tests but you can do a simple I/O load test with a one line command:

    Code: Select all

    dd if=/dev/zero of=/dev/null &
    Depending on the CPU power (and storage subsystem power) you may have to run a bunch of these commands in parallel to really load up the system. Or you can set of= to a real file, that will actually write to disk (but that may incapacitate both of your peers since FreePBX HA mirrors the disk).
  • Fail SIP provider: Just unplug the network cable (or unplug your firewall) and watch what happens.
  • Fill disk: Just create a bash loop that appends "1" to a file until it abort. Then watch the other peer and you will see DRBD has filled that disk too. (HAAst does NOT mirror the disk)
WARNING: All of these tests can mess up FreePBX, so run a FreePBX backup before the test. This is important because FreePBX can corrupt its MySQL database during a system freeze up and you will have no backup since the other node is an exact mirror of the first (also corrupted!) thanks to DRBD. That in itself is something to think about - real life disk corruption on one peer will destroy both cluster nodes in FreePBX HA. How can Schmooze sell clustering software where corruption on one node gets copied to the other node? Unbelievable - until you test it and see for yourself. (I realize know that DRBD is just a raid mirror across the network - so everything written to one disk writes to the other, including corruption).

Then try the same test with HAAst and watch the green lights move from one node to the other on the HAAst gui page, within a second of the test starting! Sweet. AND, the other node's database will NOT be corrupt because HAAst does not do disk mirroring. (Confirmed using fsck command)

Please post your results here! I would suggest you also post your results on the Sangoma/Shmooze website - but I suspect it will be removed and you will get a warning from the admin. (or get banned).
iblurker
Posts: 5
Joined: Wed Feb 08, 2017 3:05 am

Re: Comparison of HAAst to Commercial HA and Open Source HA

Postby iblurker » Fri Feb 10, 2017 3:17 pm

Ok I did the 3 tests on my FreePBX HA, and on my HAAst test VM's. I didn't realize that I could setup more sensors in HAAst so I spent an hour adding sensors for realistic things I could think of (very cool). According to Schmooze there is nothing I can or need to setup - it's all "automatic" in FreePBX HA. Which I learned means "no sensors" (seriously - it detects almost nothing, just a failed process).

My test results were like yours (which underscores the reason I will be removing FreePBX's HA module). I ran my tests Wednesday after hours and (like you) couldn't believe what I saw. So I contacted Schmooze on Thursday to make sure everything was setup right, which it was. So Thursday after hours I repeated the tests just to be sure.

The details you requested:
Runaway process: 3 out of 5 times the entire FreePBX HA cluster failed (the other node would not take over). HAAst worked perfectly each time.
IO overload: 5 out of 5 times the entire FreePBX HA cluster failed (the other node would not take over, or if it did it would not start right). HAAst worked perfectly each time.
Fail SIP provider: 5 out of 5 times the entire FreePBX HA cluster failed. FreePBX HA didn't seem to be aware of any environment (disk space, CPU, network connections, etc). HAAst worked perfectly as soon as the connection dropped. For fun I even tried unplugging the switch that node 1 was connected to, and node 2 took over with HAAst. I can see how this would be useful if I want to put a backup node in another data center.

Some additional bad news: three times my runaway IO or CPU process corrupted something in FreePBX HA (I had to restore the node from bare metal). I have to admit I'm a bit fed up with Schmooze and their attitude (maybe because they were bought by Sangoma, I don't know). I kind of blindly trusted them at first but their fax solution (which was supposed to be great) is a major disappointment, their routing module which we paid for is a joke, and their HA module (which they told me at ITExpo was the only 'certified' solution) is worthless. If the only real life scenario the FreePBX HA module can detect is unplugging the power then that's worthless. I've personally encountered a runaway process, network outage, and a bad disk over the past 2 years. And none of these would trigger a failover in FreePBX HA (or open source) - just a dead phone system, which is not acceptable (in case Schmooze is listening)!

I will be recommending to my manager that we immediately remove the FreePBX HA module and replace it with HAAst. I feel like I was scammed with the FreePBX HA module, but I suppose it's my own fault for not learning more before buying. (As I bend my leg around to kick myself in the ass for wasting $3000 with Schmooze).
User avatar
Telium Support
Posts: 162
Joined: Sun Nov 27, 2016 3:27 pm

Re: Comparison of HAAst to Commercial HA and Open Source HA

Postby Telium Support » Fri Feb 10, 2017 4:59 pm

Thanks for the feedback. I will leave this post pinned so others can try the tests for themselves.

We've very proud of the sophistication of HAAst sensors and sync technology. I have a feeling we need to do a better job in the marketing of what makes HAAst different from other solutions.

As well, I will let our Engineering group know that we need to improve the manual in regards to sensors. The range of things you can test with HAAst is really broad - all rolling into one health score. We've seen clients monitor system fans, environmental (room) systems, 3rd party software systems, storage arrays, T1 CSU's, etc. Telium is usually engaged to set these up for clients, which is why I suspect we don't have more examples in the manual.

I'm not sure if you realized it in your testing, but HAAst will not sync any data if it detects a failing node. So you will never have the situation you described with other products, where corruption on one node corrupts the other node. (Assuming you've configured HAAst accordingly).

You are also correct that you can safely put one node in one data center, and the other node in another data center. HAAst is the only product that doesn't make use of shared disk (DRBD), shared phone banks, etc. or any LAN-centric protocols. So HAAst is the only product that works properly across a WAN. For example, we have a customer with one PBX in Singapore and one in England and HAAst works perfectly creating a cluster across continents.
steveadmin
Posts: 1
Joined: Mon Sep 18, 2017 12:14 pm

Re: Comparison of HAAst to Commercial HA and Open Source HA

Postby steveadmin » Mon Sep 18, 2017 12:25 pm

We switched to HAAst after 1 year with FreePBX HA. We had high hopes for FreePBX HA after they pitched it at IT Expo (they also had lots of nasty things about 'competing products').

However, every time something went wrong with the FreePBX HA cluster Sangoma had excuses. From corrupt files, to not detecting the trunks being down, to not detecting a runaway process using all memory - all of which broke the cluster and prevented calls from being handled.

Most recently our PBX ran out of file descriptors, and the FreePBX HA solution just sat there not processing calls, and not failing over, but reporting the cluster was fine. Sangoma's view was that was "an operating system problem" and not the responsibility of the cluster. Seriously? So FreePBX can't detect anything other than unplugging the box?? Check out: [link to sangoma site removed by admin]

So we switched to HAAst 1/2 year ago - and guess what, it detects all of the above problems and fails over immediately. No more excuses. (I don't care if its the OS's fault, the disk's fault, the network's fault, FAIL OVER dammit! And HAAst actually does).

I have to agree with the OP above, we wasted $3000 on FreePBX HA. Adding a file system mirror (DRBD) and simple heartbeat does not make it a real HA PBX ! That type of solution is fine for an HA file server, or print server, but not for application level HA!!! There is a HUGE difference between application service availability and OS availability. Just because Linux is running doesn't mean Asterisk is working properly. That fact that you have to practically kill the box for FreePBX HA to detect a failure means this is pretty much useless. This is not real life HA; it's fine if you just want to tick off a checkbox saying you have HA, but it's not OK if you are on the hook for business loss due to an outage. FreePBX HA can cost a real telecom admin (like me) his/her job! I already had egg on my face for multiple outages after buying Sangoma's $3000 HA solution. Then I had to go back to my boss and explain that I wasted $3000 on a piece of #$%@ and now have to spend another $3000 to buy a REAL high availability solution.
EnterpriseRentalIT
Posts: 2
Joined: Sun Feb 25, 2018 2:34 am

Re: Comparison of HAAst to Commercial HA and Open Source HA

Postby EnterpriseRentalIT » Sun Feb 25, 2018 2:54 am

I wanted to comment on this topic because we come at this problem from a very different angle. Our company does not like to pay for any software, and we are 90%+ open source on the back end.

We run 12 PBX clusters and for 3 years we ran our own open source PBX clustering with LinuxHA/shares/Corosync/DRBD/and more. Our solution grew year after year - primarily because after every cluster failure we discovered another use case! After 3 years our code to manage the cluster was getting so big and unwieldy that we had a developer spending 1/4 his time maintaining the solution. And, we still had outages with every new use case we encountered. Believe me - you can't even think of every use case, and everything you should consider in determining the health of a node. (The people at HAAst however have figured out every use case we encountered in 3 years, and more)

When my manager calculated what we were spending on our "FREE" solution (1/4 a full time salary) plus the cost and embarrassment of each outage (much higher than a developer salary) my manager finally threw out our home grown "FREE" solution and bought HAAst. We actually tried some commercial alternatives but they proved too generic and really weren't PBX aware.

I'm a Linux admin of 20 years so I don't like closed/commercial software, but I have to admit this is one of those exceptions I'm happy to pay for. It's not the money that bothers me (our network management solution cost over $100k to put in place and maintain), HAAst is actually cheap, but rather the principle of sticking with open source.

Anyways, in the end paying for HAAst is a no brainer. Oh and I should point out that the other "commercial" solutions are some combination of the above free open source packages. As someone pointed out above, if you are stupid enough to pay a vendor to give you a free open source solution then you are an idiot, and deserve what you bought. I suspect it may even be illegal for them to call it their own module when they are re-branding a free open source product. Hint: If you pay for a F---PBX HA module that simply moves your F---PBX directories to a DRBD partition, then it's not a real HA solution. (But if you are stupid enough to pay over $3000 for the free DRBD package then you get what you deserve).

If you run a commercial PBX and need HA, buy HAAst. It's a bargain. If you are a home office / mom and pop shop, and don't understand Linux or Asterisk, give your money to a configuration generator vendor and bend over to receive the free product you just purchased. Configuration generator HA products are so generic that they won't help you recover from real life failures but at least you can tell your manager you have HA!

----

Oh and one more thing - we have called HAAst technical support at 4:45am Pacific time and got (excellent) support getting our cluster back online. With open source (or other commercial) who don't have any idea about Asterisk, or who don't answer their phones until business hours, you are shit out of luck. And of course with open source there is no one to call. Big thumbs up to the guys at HAAst on fantastic support!!!

I write this message on a Saturday night after one of our PBX's was shut down by HAAst. The disk was corrupting and HAAst demoted the node. I came into the office and confirmed the disk failure. (If we ran DRBD the other node would be corrupt as well). I will rebuild on Monday, but I called HAAst at 7pm tonight (Saturday) and they helped me confirm the diagnosis. They even offered to help me setup a temporary peer on AWS if I wanted to ensure we had a failover. Hats off to you guys! Have a great weekend.
ITTSadmin
Posts: 1
Joined: Thu Jul 28, 2016 9:22 pm

Re: Comparison of HAAst to Commercial HA and Open Source HA

Postby ITTSadmin » Wed Feb 28, 2018 9:22 pm

I've got a different background to share on this topic so I wanted to explain why we switched to HAAst.

We have 5 FreePBX clusters running at a hosting company. At first we used FreePBX's HA module but for exactly the reasons listed above we threw it out. (It's junk. What a waste of money.) We bought it because it was the only one that Sangoma said was FreePBX 'certified'. In other words, they sell it so they say everything else is bad. I didn't research before buying it, but I eventually discovered that I just bought some open source packages that I could have installed myself for free! And FreePBX HA recovered from nothing but the most simplistic of failures. Absolutely useless!

Our hosting company (which provides IaaS and PaaS) offered us a VMware + VMotion based solution, that promised to move a failing PBX, keep calls up, no interruption in service, etc. That too proved too good to be true. VMware is not aware of the internals of the PBX, trunks, routes, other services, ability to bridge calls, etc. Anything short of a box exploding would not trigger a Vmotion (so we again suffered outages).

We finally moved to HAAst because it just plain works. We've been with HAAst for 3 years (adding 2 new clusters over that time). We've had hard disks fail, CPU overload (from another customer sharing our hosted box), even a hosting company network failure!! (AT&T outage on the coast last year). Each time HAAst failed over perfectly. It also failed over to our backup site (just another hosting company on the opposite coast).

You find lots of people offering "HA in 5 minutes" solutions (check YouTube), Do-It-Yourself open source solutions (like FreePBX/Elastix). Been there, done that, it doesn't work. Everyone promises great results but no one delivers (aside from HAAst). If your PBX is critical there is nothing else on the market that works like HAAst. I got fed up with every tech company/admin telling me they have a great solution, how fantastic it is, that it can do everything HAAst does, etc...but they don't work in REAL LIFE scenarios. (But they look great until your first outage).

If you know what it's like to get an network management alert at 3am because a system crashed, you will appreciate that HAAst actually let's you go back to sleep and deal with it when you get back to the office the next day/ next week. As an admin that's priceless! I want to be able to take vacation and not have to worry about my PBX HA cluster. (Hint: if your HA solution includes DRBD / open source then you better be heading for the office at 3am)
j.martinez
Posts: 2
Joined: Fri Apr 06, 2018 2:39 am

Re: Comparison of HAAst to Commercial HA and Open Source HA

Postby j.martinez » Fri Apr 06, 2018 2:53 am

Our company switched to HAAst from "AsteriskHA" five years ago. The AsteriskHA vendor offered a very inexpensive HA solution for Asterisk ($500). It looked great on paper, and the sales person said it was identical to HAAst, just cheaper. They told me HAAst with a big rip off, theirs did the same, etc.

But, it was problem after problem and excuse after excuse with their software, and after 2 years of headache we just bought HAAst. It seems that every few years a new product comes out that says it's "just like HAAst" - and then after they collect peoples $ they close down and disappear, with a product that's half baked (at best). The vendor of the $500 package is gone (surprise). I think other companies underestimate the products; Telium doesn't advertise 90% of the stuff that HAAst does. For example, I was surprised that it was modifying AstDB keys for me in the background to support certain Dundi features. That's awesome! It's not in their brochures or web site anywhere. (Either because they don't want to give competitors a how to guide, or because they're really bad at advertising). Another example, our primary PBX node at a remote site became unstable (hardware was failing and we didn't realize it). After a several failovers HAAst kept the other node as the new primary because the first node was the cause of too many failures in a short period of time. Again, not in Telium's advertising anywhere, but brilliant design! (The lesson is don't look at the pretty "features" web page and assume that's a checklist of all functionality.)

I see "how to" guides on the internet/YouTube showing how to make an Asterisk cluster in 15 minutes. There's even a free e-book (published by the guy who runs Freepbx - Tony Lewis) that tries to tell you that the open source stuff is identical to HAAst. (Because FreePBX's $3000 solution is just a bunch of open source packages, and then they allow their code to run from a DRBD partition when you pay them $3000). Last year I saw a project on Freelancer where some guy spent $2000 to have someone build him a product identical to HAAst. After wasting his $2000, the project failed (all the developers who said they could easily rebuild a HAAst product couldn't deliver). He came back again spending even more money because the first system he built for $2000 did little more than open source products. He's been back 3 times to add missing functionality and people still can't make him a system anywhere close to HAAst. So that poor schmuck wasted 4 months and over $6000 and still doesn't have a solution that can do a fraction of what HAAst does. For $600 he could have bought the Flex version of HAAst and have a working product!

We wasted far too much time and money avoiding spending money. We have 8 PBX's clusters now, and all run HAAst. Everyone wants a rock solid HA product, but some people just don't want pay for anything. Seriously people, you get what you pay for. HAAst was recommended by our ITSP because that's what they use. The ITSP is cheap (they use all off the shelf PC boxes sitting on plastic Costco shelves, with 30 window air conditioners, running in 2 old building in neighboring cities) yet they use HAAst; and they are the biggest ITSP in the southern part of the state. If they run HAAst, then you can bet there is no cheaper option (that actually works)

I'm also sure that another new Asterisk HA product will pop up again, from some other vendor, and then disappear in 2 years, then another, etc. Reminds me of those 2am TV commercials for a "high quality stereo" from china, identical to a $3000 Denon stereo but only costs 3 easy payments of $19.99. You get what you pay for people!