Search
  • chalarm

Hyper-scale CGN (1200Gbps+)

About a year a go i worked as a Technical Consultant for a Service Provider with regional countrywide backbone and small international presence solely for peering purposes.


Like many ISPs out there these days we ran an CGN to prevent IPv4 shortage. The network was scheduled to be expanded into more areas and to provide more eyeball traffic as a result of this would go beyond point of available boxes (300Gbps is what's available from commercial vendors).


So how do you engineer that which cannot fit into a single box? You start thinking outside the box.


Like any country obviously you have regions (you can go and think all cardinal directions in this case), but how do you make this scalable and how do you provide fallback from perspective of what if a data center (Force majeure) becomes unavailable either from networking perspective or just a full blown blackout - can be as easy as some engineer playing with the wrong switch.


Taking into account that in this country there are strategically just two egress points towards the Internet. Let's say for the sake of argument this provider was in France, it wasn't but i like to use Force majeur.


So let's say France has egress traffic in Rennes and Geneva (again we all know this isn't true but hypothetical):

We now have subscribers from Rennes and Geneva we obviously want them to go out on the closest CGN due to latency but when it's unavailable we want it to still be able to work in the other geo location.


So design factors:

~ We have 2 egress (Internet facing) locations;

~ 800Gbps traffic which is still growing rapidly and the current solution doesn't provide fail-over in any way - if it breaks it breaks.


So, what did i end up designing:

~ 2 (sites) x 5 (boxes) x 400Gbps (200Gbps IN/200Gbps OUT) on both egress locations.

~ Design constraint: Bigger boxes unavailable at reliable price points (hard limit was 300Gbps at this time)

~ Anycast was used to work as a many-to-one service.


How would you solve this problem?

24 views0 comments

Recent Posts

See All

Review Commentary: Opensource for the Desktop

TL,DR: For now Linux Mint is the 'winner' but the question is if this is going to be forever, since i don't expect this to be the case. As many of you know I'm a proud enthusiastic user of a Opensourc

My two cents over: DNS over HTTP (DOH)

This post is an expression of my opinion over DNS over HTTP in short DOH. DOH is currently being pushed by Cloudflare and Mozilla. Target audience: - Managers read the Tl;dr + Management note and don'

New Years Resolution 2021

First of all, Welcome, This blog post is a summerization of my frustrations of past year. Due to the pandemic, which i'm not even going to call anymore i was forced by my government to close my busine