10000 channels on asterisk milestone reached

Olle has posted a little more information on his claim for the steak prize a couple of days ago:

Hello Asterisk users around the world!

Recently, I have been working with pretty large Asterisk installations. 300 servers running Asterisk and Kamailio (OpenSER). Replacing large Nortel systems with just a few tiny boxes and other interesting solutions. Testing has been a large part of these projects. How much can we put into one Asterisk box? Calls per euro invested matters.

So far, we've been able to reach about 2000 channels of G.711 with quad core CPU's and Intel Pro/1000 network cards in IBM servers. At that point, we see that IRQ balancer gives up and goes to bed, and all the traffic is directed to one core and the system gives up. We've been running these tests on several systems, with different NICs and have been working hard to tweak capacity. New drivers, new cards, new stuff. But all indications told us that the problem was the CPU context switching between handling network traffic (RTP traffic) and Asterisk. This was also confirmed from a few different independent software development teams.

Imaging my surprise this Monday when I installed a plain old Asterisk 1.4 on a new HP server, a DL380 G6, and could run in circles around the old IBM servers. Three servers looping calls between them and we bypassed 10.000 channels without any issues. SIP to SIP calls, the p2p RTP bridge, basically running a media proxy. At that point, our cheap gigabit switch gave up, and of course the NICs. Pushing 850 Mbit was more than enough. The CPU's (we had 16 of them with hyperthreading) was not very stressed. Asterisk was occupying a few of them in a nice way, but we had a majority of them idling around looking for something to do.

So, please help me. I need answers to John Todds questions while he's treating me with really good expensive wine at Astricon. How did this happen? Was it the Broadcom NICs? Was it the Intel 5530 Xeon CPU's? Or a combination? Or maybe just the cheap Netgear switch...

I hope to get more access to these boxes, three of them, to run tests with the latest code. In that version we have the new hashtables, all the refcounters and fancy stuff that the Digium team has reworked on the inside of Asterisk. The trunk version will propably behave much, much better than 1.4 when it comes to heavy loads and high call setup rates.

We're on our way to build a new generation of Asterisk, far away from the 1.0 platform. At the same time, the hardware guys have obviously not been asleep. They're giving us inexpensive hardware that makes our software shine. Now we need to test other things and see how the rest of Asterisk scales, apart from the actual calls. Manager, events, musiconhold, agi/fastagi... New interesting challenges.

So take one of these standard rack servers from HP and run a telco for a small city on one box. While you're at it, buy a spare one, hardware can fail ( ;-) ).
But don't say that Asterisk does not scale well. Those times are gone.

/Olle

---
* Olle E Johansson
* Open Unified Communication - SIP & XMPP projects