Core Network Upgrades

Well into the planning stages of the core network upgrade, we were met with a new devastating BGP bug on our aging MX240 router, causing core dumps to occur every 2 minutes as a result of a BGP attribute path length error. See the reddit post attached; https://www.reddit.com/r/Juniper/comments/1530req/juniper_mx240_core_dump_due_to_bgp_attribute/

Juniper states this issue is resolved in later Juniper versions; however, one big issue. If you do not have Juniper vendor support, you cannot obtain certain pre-requisite software. So after multiple unsuccesful attempts to upgrade this router to 23.4R2.13 we had to make a decision.

Do we keep wasting precious time attempting to upgrade and thus put the core network at prolonged risk of a full outage? Or do we bite the bullet and perform open-heart surgery on a core network and replace the MX240 with a nice new and shiny Nokia SR 7750?

Smooth is fast & fast is slow

Whilst the pressure is on, it is easy to rush through these incidents. But the smoother and calmer you are the better.

Let me remind you that all of this is happening within the space of say 10 hours, from the first BGP flapping at 9pm, three of us jump onto a bridge call to begin investigating, and before you know it, it is 4 am and we are formulating plans for the open heart surgery.

Before proceeding, though, we had to stem the bleeding. Disabling all external BGP peers on this core router and turning it into an additional internal BGP hop had stopped the BGP flapping, which allowed us to get some rest at 5am, ready to reconvene at 10 am to plan our upgrade.

If you're enjoying this read so far you will have to wait for me to finish writing it up......