Boxtone (www.boxtone.com) provides enterprises with solid mobile user management and troubleshooting tools. Customers using their tool were kept up to date with updates throughout the outage which is a refreshing thing to hear.
Mitch from Boxtone was kind enough to send over their synopsis of the outage as compiled by Boxtone. Funny how they give us more information that RIM does…
- Between 3:00 and 4:00 PM – Problems with BBM and BIS internet browsing reported around the web (I personally experienced around 3:30 PM).
- Between 6:30 and 7:00 PM – The problem extended to BES email, preventing the delivery of BES emails to and from BlackBerry smartphones. At each of our customers, BoxTone detected a greater than normal quantity of users with messages pending, based on our learned baseline of what is normal for each server and carrier, and immediately generated a warning alert our customers before the flood of user calls (Sample email alert below). BoxTone also placed all affected BES and Carriers in a Critical state on our customers’ Operations Dashboards (depicted by the red dots next to each BES and carrier). The steady growth in Pending Messages beginning around 6:45 is annotated in the attached screenshot and continued until the issue was resolved early this morning. From our monitoring data, it appears that BES were able to communicate with the RIM NOC throughout the outage; however, the NOC was unable to deliver messages.
- At approximately 12:09 AM, BoxTone detected a brief disconnect in the SRP connection of each BES to the NOC; it appears RIM reset the NOC SRP connection to complete their fixes. Following this reset, delivery of BES mail resumed.
- By 2:45 AM or earlier, BoxTone detected that most of our customers had returned to their normal (baselined) service levels, and that the backlog of pending mail had been delivered. BoxTone generated notifications informing our users that their service levels had returned to normal and updated the status of the BES and carriers to Normal.
All times EST.
Customer-specific information has been obfuscated in the attached alert and screenshot.
