Whenever the backlight of my phone changes without ringing there is something happening. Usually that happens when the local PBX for debugging gets restarted. But today there was no debugging going on, and the light came on.
It is even worse when something like that happens just before you want to go home.
We realized that we were not connected to our hosted PBX any more. It was amazing how fast the others found out. PBX is a really critical service.
Then when trying to get to the server by SSH, there was no response. We tried from our home network, no response either. Tried to get to the console, no response. In panic we opened a ticket with the data center to check if the server is still alive.
Then while waiting for an answer, we tried other servers. The Paris server also did not respond. That did not fit into the picture because that runs on an entirely different infrastructure. Was it a DNS issue? Obviously not since we were able to resolve the addresses.
Then we found out that we could get to the web interface of the PBX from another server in South East Asia. Signs of life! If we can get to the server from there, then the server must be okay.
There must have been some massive internet problem in our area. I guess we will not read about it in the Wall Street Journal tomorrow, but someone must have pressed Enter on something stupid on an important router. Or what could be is that we are witnessing the end of IPv4 because the routing is now so screwed up that it becomes impossible to keep all those routes up.
We’ll see how long it takes. We are still offline. Cell phone failover is the killer feature of the day.