Hyper-scale CGN (1200Gbps+)
About a year a go i worked as a Technical Consultant for a Service Provider with regional countrywide backbone and small international presence solely for peering purposes.
Like many ISPs out there these days we ran an CGN to prevent IPv4 shortage. The network was scheduled to be expanded into more areas and to provide more eyeball traffic as a result of this would go beyond point of available boxes (300Gbps is what's available from commercial vendors).
So how do you engineer that which cannot fit into a single box? You start thinking outside the box.
Like any country obviously you have regions (you can go and think all cardinal directions in this case), but how do you make this scalable and how do you provide fallback from perspective of what if a data center (Force majeure) becomes unavailable either from networking perspective or just a full blown blackout - can be as easy as some engineer playing with the wrong switch.
Taking into account that in this country there are strategically just two egress points towards the Internet. Let's say for the sake of argument this provider was in France, it wasn't but i like to use Force majeur.
So let's say France has egress traffic in Rennes and Geneva (again we all know this isn't true but hypothetical):
We now have subscribers from Rennes and Geneva we obviously want them to go out on the closest CGN due to latency but when it's unavailable we want it to still be able to work in the other geo location.
So design factors:
~ We have 2 egress (Internet facing) locations;
~ 800Gbps traffic which is still growing rapidly and the current solution doesn't provide fail-over in any way - if it breaks it breaks.
So, what did i end up designing:
~ 2 (sites) x 5 (boxes) x 400Gbps (200Gbps IN/200Gbps OUT) on both egress locations.
~ Design constraint: Bigger boxes unavailable at reliable price points (hard limit was 300Gbps at this time)
~ Anycast was used to work as a many-to-one service.
How would you solve this problem?