There have been several reports since Sunday (12/20/20) of eNBs becoming inaccessible and not passing any user traffic. These impacted eNBs also appear to not be passing any user traffic (despite the OMC showing the eNB as active and with UEs attached). Opening the eNB webGUI will result in an error message and SSH connections will frequently close. When connected via SSH, the admin user will have no permission to execute any commands.
It has been determined that the cause is due to the eNBs being hacked directly. In most cases, the impacted eNBs had their management ports exposed to the outside world. It also appears that the hacked eNBs are scanning the local network for other eNBs to infect. At this time, only the eNBs running RTS and RTD software (all version) appear to be at risk.
For eNBs that have been hacked, there is currently two recovery methods available:
Create software rollback task from the OMC. To perform this step (from OMC), go to eNB>Maintenance>Software Rollback page, then click the Add (+) button and select the appropriate eNBs and click OK to initiate the software rollback task. The task will eventually display a “failed” status, but the backend system rollback commands did take place. The reason the task lists as failed is due to the reboot command not executing. This means you will still need to power cycle the eNB to bring it back up.
Perform software rollback from eNB serial port. If the eNB was rebooted or is not connected to the OMC, the only remaining option is to perform a software rollback from the serial port, which can only be accessed by opening the eNB chassis. Access to the serial port will also require a RS232 to TTL adapter. Detailed instructions can be provided by reaching out to support at firstname.lastname@example.org.
Upon software rollback, priority should be placed on upgrading the eNB software and blocking public access to the eNB management ports, as well as locally infected eNBs, to avoid repeated hacks. Performing a software upgrade will overwrite the secondary software bank which was hacked. As to the recovery recommendation, the safest approach would be to rollback all eNBs at once and after power cycling them, to disable the Ethernet port which they are connected to and only bring the port up upon all hacked eNBs being recovered or powered off as well as the network being secured. You will want to change your passwords afterwards as well of course.
While we don’t yet know where the exact vulnerability is, the common belief is that the hack occurred over HTTP, especially with SSH listening on a non-standard port. As we learn more about the nature of the hack and where the exact vulnerabilities are, we will provide another update along with a software patch. In the meantime and especially if there is risk of management ports being exposed, it is suggested to disable HTTP/HTTPs access from the WAN interface. This can be disabled on the Network>WAN/LAN/VLAN by setting Allow management access over WAN to Disable. Once disabled, the webGUI can only be accessed via the LAN IP. The eNB would still be managed from the OMC.
For those looking to restrict outbound traffic to CloudCore, outbound access to the following is required for all managed devices. Note that the cloud server IP addresses are currently not static which is why FQDN names are being listed instead. This is something we are currently reviewing and will likely be switching to static IPs in the near future.
- Host FQDN: baiomc.cloudapp.net
- Ports: (TCP) 443, 8082, 48080
- Host FQDN: baicells-westepc-03.cloudapp.net, baicells-eastepc04.eastus.cloudapp.azure.com
- Ports: (UDP) 4500, 500; (ESP) IP protocol 50
- Additional Note: IPsec pass-through must be allowed
Last Update: 12/22/20 12:35PM CT