Some time ago, the Yarix Red Team was engaged on a red team assessment that included an onsite activity to test the physical security posture of the Customer. Although we would have used social engineering tactics to physically enter the Customer property, this would have given us a too short amount of time to stay connected to its internal network for a deep analysis of it. We therefore decided to build a small, remotely controlled drop box that would be hidden in the Customer’s premises and connected directly to its internal network, hoping that it would not be detected. However, we only had a couple of weeks before onsite activity began, so we had to build and test the drop box as quickly as possible while using as many components as we already had on hand. In this article, we will describe the components we used, the scripts and configurations we made, the challenges we faced and how we solved them.

The big small picture

As many of you may know, a drop box is a piece of equipment with different purposes which is intended to be left unattended on a targeted place. Will it be a customer office or data center, a public location or where else, if detected it can be lost or destroyed, so it must be cheap enough to justify its loss. In addition, it must also be small enough so that it can be hidden or made to go as unnoticed as possible. Nowadays, there are many possibilities to build such a device, since many different single-board computers exist and they are quite common due to their low cost.

However, because of the pandemic caused by the Covid-19 virus first and the current geopolitical environment later, we are in the midst of a microchip shortage that has led to the rising cost of these kind of computers. For example the Raspberry Pi Foundation, known for their cheap but powerful single-board computers, had to increase their prices for the first time ever. This made difficult to buy a new board which had an affordable price and that was available for delivery in a couple of days. Luckily, as said, they are common and we already had a Raspberry Pi 3 available in our lab. Thanks to its versatility, by simply swapping the micro SD from a previous project with a 32GB one, also already available to us, we were ready to start the installation of the operating system. Of course, if in the future we need to use the Raspberry Pi with the old project, we just need to swap back the micro SDs and we will be ready to go, obviously if it will not be somewhere in the field!

Before we start fiddling with the project, we had one more key point to unmark. The purpose of this drop box is to give us a direct but remote access to the Customer internal network and to achieve it we must be able to reach the device in some way. Many of the articles that can be found on the Internet explaining how to create a drop box, rely on leveraging the client’s network to connect to a remote control server. This is certainly the easiest scenario to implement since all that is needed is to setup DHCP on the drop box’s ethernet interface and obtain an IP address on the target network, but this does not mean that the connection to the Internet is a given. In fact, restrictions on browsing may be in place that can range from limiting available ports (e.g., ports 80 and 443 can be open to outgoing connections but not port 22), to the complete block of all egress ports and the use of a proxy server. Finally, of course, a DHCP server could not be available in the network, resulting in a failure from the very beginning.

For these reasons, we decided to add a module that could provide Internet access to the Raspberry Pi via the cellular network, thus making it autonomous. A few options are available, and the most noteworthy ones are cellular USB dongles and Hardware Attached on Top (HAT). While the former are typically less expensive and easier to use, their USB dongle form factor add chunkiness to the drop box, making it harder to hide. Moreover, those dongles may be bound to a specific cellular carrier. Raspberry Pi HATs, on the other hand, ship more features and allow for greater expandability, despite being more difficult to set up, especially at the beginning. Furthermore, although higher than a USB dongle, their price is still affordable.

Components list

Below is the complete list of the components we used in this project (mostly hardware) and a brief description of them.

• Raspberry Pi 3 Model B rev 1.2: The backbone of all the project, this single-board credit card sized computer can run an ARM version of Linux that has almost all the functions that you will find on a regular computer distro; moreover, its built-in ethernet interface is perfect to quickly connect it to the target network.
• Sandisk Ultra Micro SDHC Class 10 32 GB: A spare micro SD memory card that we had in our lab; its 32GB capacity is more than enough to hold the operating system and any additional packages we might need.
• 5V USB power supply: While testing the drop box, we initially used a generic purpose 5V 4A power supply but then, as we will explain later in this article, we switched to a 5.2V power supply to achieve more stability.
• Power bank: a generic 16750mAh (5V, 2.4A) power bank that we used on the field to power the drop box.
• Waveshare SIM7600E-H 4G HAT: The broadband cellular network HAT module we chose, it supports all cellular network standards up to 4G; it also has a lot of features, many of which are out of scope for this article, but that can be used for future additions (e.g., the GPS module or the breakout pins to connect an Arduino)
• SIM cards: The HAT supports many 3G and 4G bands and isn’t bound to a specific cellular carrier, so it is possible to use different SIM cards; in this case we used two SIMs from two distinct Italian operators.
• Amazon EC2 Linux instance: A simple Linux instance that we used as the control server the drop box connects to; as we will see, it did not need many configurations to be ready.

Raspberry Pi

The first step we needed to do was to choose the operating system to be installed on the Raspberry Pi (RPi). Of course it would be Linux based and, while Raspbian (one of the most used RPi Linux distribution based on Debian) could be a good choice, we chose the ARM version of Kali Linux, since it represents the de facto choice for penetration testing and red teaming. In fact, in case we needed any tool during the activity, it would already be present by default or would be easily obtainable and/or installable. The ARM image for the Raspberry Pi is available on the Kali Linux download page and, based on the advices found on their user instructions page, we chose the 32-bit version over the 64-bit one because the former is generally more tested and has more compatibility with 3rd party software. Following the instructions we copied the image onto the micro SD card using the dd command. Although ours was a class 10 micro SD, it took quite a while to complete the copy, but once it was finished and the SD card was inserted into the Rasperry Pi, the operating system was ready to use.

During the first boot, we set up some basic configurations (e.g., keyboard layout, time zone, changing default passwords, etc.) and then updated the operating system. Some more important changes we made are the disabling of any automatic power management, to prevent the drop box from shutting down and thus being unreachable due to inactivity, and changing the hostname: we set a name that mimics that of a potentially existing server so as not to reveal the nature of the device. If prior knowledge of the target environment is available, the hostname could be adapted to go further unnoticed. Finally, we set the ethernet interface configuration to manual and created a pair of SSH keys to authenticate to the control server; we will discuss these in more detail in the sections below.

No other changes were made, since from the beginning the idea was to keep the base OS installation as standard and general as possible, so as not to depend on specific packages, be less susceptible to updates and make any future hardware migrations easier. During the course of the project, only one package was installed that wasn’t included in the default installation of Kali but is available in its repositories (knockd, which we will discuss in the Port knocking section below).

Regarding the power supply for the Raspberry Pi, we initially used, during the installation and testing phases, a generic 5V and 4A power supply. However, we saw that while performing some operations, especially when related to executing AT commands on the 4G module, this power supply seemed to fail to deliver sufficiently stable tension, causing the video signal to be interrupted for few seconds. Therefore, we replaced it with a cellular power supply that delivered a slightly higher voltage (5.2V) and was able to withstand the load peaks that occurred, achieving greater stability. When the testing phase was over and the use of a display was no longer necessary, the drop box was powered with a generic 16750mAh power bank that delivered 5V at 2.4A without registering stability problems.

SIM7600 HAT

This is the only component we had to purchase but, if we do not consider the RPi itself, it is also the most important module in the whole project. Based on what was discussed above, we chose to use a RPi HAT modem instead of a USB dongle. There are several models of 4G HATs available from various manufacturers, and each has its strengths and drawbacks. After a comparison of the various HATs available, we chose to purchase the SIM7600E-H 4G HAT module from Waveshare (The “E” on the name identifies the EMEA version of the module, with regard to supported LTE bands). The analysis revealed some disadvantages regarding this module; for example, a few articles on the Internet reported it as less user-friendly as it is slightly more complex to configure than others. In addition, the antenna contained in the package is an external SMA type (the adapter to IPEX1 connector is included in the package, however) and therefore more bulky and visible for our purposes. However, the manufacturer’s site provides quite detailed documentation that shows step by step how to configure and use the module, while we can replace the antenna in the future with a PCB or FPC one (or two since the HAT also supports an auxiliary one) as it can be found for cheap on various online stores (e.g. on Ebay). The SIM7600E-H was also cheaper than similar modules from other vendors and was also available on Amazon, which allowed us to receive it the day after ordering it. Other plus points of this HAT were that the package included screws and standoffs for mounting on the RPi and the support for GPS and Arduino which, although not used at this time, may come in handy in the future.

We began fiddling with the HAT by inserting a Vodafone Italy SIM card into the slot, mounted it on the RPi’s GPIO pins and connected its micro USB port to a USB port on the RPi with the supplied cable. By powering up the RPi, the HAT also starts up as it takes power directly from the GPIO pins; this is useful in that even if the RPi were to be turned off, the HAT would continue to operate, at least until the power supply is completely disconnected from the mains. At first startup, the module was active and was already accepting AT commands but, as recommended by the manufacturer’s instructions, we compiled the appropriate drivers (simcom_wwan) and loaded them, thus disabling the default ones (blacklisting the kernel module qmi_wwan). Still following the instructions, we disabled ModemManager to prevent it from interfering in the setup operations of the HAT interface (wwan0) and configured it to use DHCP. Using some more AT commands, we got confirmation that the module had registered to the cellular network.

However, as seen in the image above, the module had registered on the GSM (2G) cellular network and not on the LTE (4G) one. After verifying that the network search was not locked on a specific technology and attempting to change 4G bands and other settings via AT commands, we tried switching SIM cards using that of another operator (Iliad Italy). In this case there was an improvement in that the latter registered on the 3G network and was therefore usable as a data connection. However, this was not optimal since the 3G network was not available for the operators Vodafone and TIM because they both had already completed the decommissioning of this technology. At this point, we contacted Waveshare support asking what the problem might be and how we could fix it. After sending them the outputs of some AT commands, they told us (thanks Huang!) to check the configured APNs, as the default ones were those used by a Chinese operator. Setting the correct ones (AT command: AT+CGDCONT=1, "IP", "mobile.vodafone.it") finally allowed it to register on the 4G network and correctly activate the NDIS connection (AT command: AT$QCRMCALL=1,1). The same solution was also valid for Iliad.

Now that everything was working properly, we started writing some scripts to automatically connect to the cellular network and get an IP address from the operator. Basically, the script was going to be in bash, but since it was also necessary to send some AT commands and since using the minicom terminal emulator was not very practical to be inserted directly into a script, we also created a python script that was run in turn by the bash script. Such a python script used the native serial library to communicate directly with the HAT. Finally, we added a cron job that run the bash script at boot.

AWS control server

Directly opening SSH access on the drop box is not a viable way to reach it, since, being it directly connected to the client’s environment, it would open a direct access to it and thus create a security hole. So, as mentioned earlier, the drop box would connect to a listening control server and from there we would connect back to it. In order to avoid directly exposing corporate assets on the Internet, we needed a machine that could be sacrificed in case something went wrong. The suitable solution was undoubtedly an Amazon EC2 Linux instance; in case the implemented security controls (that we will see later) were not sufficient to prevent a possible attack, it would be sufficient to delete the entire instance and create a new one.

To avoid public exposure of port 22, known to be the default port for SSH and the one we use from our lab to connect to the control server, we started a second instance of SSH listening on a different one. There are various tutorials available on the net explaining how to do this, but in a nutshell it was sufficient to create a copy of the original SSH configuration file (sshd_config) and change in this the listening port. We also created a copy of the SSH service startup file (sshd.service) and in this one we added the option to load the previous configuration file instead of the default one. Finally, we started the second instance of SSH and enabled its startup on boot. Now both instances were listening at the same time.

To create the SSH tunnel from the drop box we needed a specific user in the control server that, for security reasons, would have as few privileges as possible. So we created a new user and, since authentication to the control server will be allowed only by public key, in its home we created the .ssh folder. Then, inside it, we added the authorized_keys file and in it we inserted the public key of the drop box user along with some options restricting its use to tunnel creation only (no-agent-forwarding, no-X11-forwarding, no-pty). Finally, we set the nologin binary as the user’s shell, effectively denying the user login access.

Calling home

Once the connection to the Internet was established, calling home was a relatively simple point to implement. As anticipated earlier, the drop box will need to open a reverse tunnel to the control server’s exposed SSH port by forwarding its own SSH port (the default 22) to another control server port that will remain listening on the localhost interface. Once this tunnel is established, it will be possible to run SSH in the control server pointing to the listening port on localhost, finally gaining access to the drop box. We have implemented this by creating a bash script that runs every minute via another dedicated cron job. This script was based on the create_ssh_tunnel.sh script found in an online article. Initially the two scripts were pretty much the same (follows our version of createTunnel() function) and, as described in the article, we also created a cron job to have it run every minute. Over the course of the project, however, we made several changes and additions to the script; we will better describe these in the next two sections.

createTunnel() {
  /usr/bin/ssh -p $JUMPBOXSSHPORT -N -R $JUMPBOXLOCALPORT:localhost:22 -i $KEYFILE $JUMPBOXUSER@$JUMPBOXIP 2> $WARNSFILE &
  # Waiting some seconds to give time to SSH to write to the warns log file 
  sleep 5
  
  # Checking if backgrounded SSH is running 
  # Redirecting the output to /dev/null since we are only interested in the exit code
  ps -p $! > /dev/null
 
  if [[ $? -eq 0 ]]; then
    echo \[`/usr/bin/date +%Y%m%d_%H%M%S`\] Tunnel to jumpbox created successfully
  else
    echo \[`/usr/bin/date +%Y%m%d_%H%M%S`\] An error occurred creating a tunnel to jumpbox. Check the file $WARNSFILE \for more information.
  fi
}

One noteworthy change to the operating system that we made that affects this phase was to set the ethernet interface to manual mode and to disable NetworkManager completely. This was because, especially when the drop box was first started, it could happen that the ethernet interface would start before the HAT interface and then, if the ethernet cable was connected and DHCP was available, the wrong default gateway would be set, thus causing outgoing traffic to go to the target network instead of the mobile data one. With this configuration, the ethernet interface is handled completely manually and also has the advantage of greatly limiting the noise generated in the target network when the drop box is connected, lowering the possibility of immediate device identification.

Port knocking

As we have seen so far, we have implemented various security settings to protect access to the control server. However, leaving an instance of SSH publicly listening is still not a good idea. To overcome this problem, we have implemented an additional layer of protection, known as port knocking. This technique involves sequentially sending to the server a packet to every port of a predefined list. When this combination is identified by the listening daemon, it executes a predefined command that usually opens the desired port on the server’s firewall (in our case the SSH port) allowing the connection. We used the following iptables command, that opens the desired port accepting as source only the IP address that performed the knock.

/sbin/iptables -I INPUT -s %IP% -p tcp --dport 11111 -j ACCEPT

Once the SSH connection is established, a knock similar to the previous but on different ports is performed that closes the previously opened port. The closing command is similar to the previous one but instead of the -I argument which adds the rule on top of the existing ones, it uses the -D argument which deletes it. One note regarding port knocking is that while the knockd package on Kali was already in its repositories, for the centos/redhat-based OS of the control server it had to be downloaded and installed manually. Also, it is essential to keep in mind that iptables, when implementing the drop all rule, also blocks connections to localhost, not allowing SSH connect back using the port opened from the tunnel. Therefore, the following rule must also be added to iptables ones.

-A INPUT -i lo -j ACCEPT

Availability

Since the drop box would had to be attached to the target network, started up in the shortest possible time and would be unattended for the entire time of the activity, in case of problems (especially connection ones) we would not be able to go onsite to fix them. For this reason, it would have to be as reliable as possible. In order to achieve this, we modified the calling home script by adding various checks that would identify any problems and restart the connection and/or the entire drop box. Below is a quick list of the implemented checks.

Cellular data connection: we created a bash script that runs from a cron job every 5 minutes and checks whether the cellular data connection is up and working properly. If the check fails, the same script executes the DHCP release, runs another python script, which in turn disconnects the HAT from the cellular network using AT commands, and finally attempts to re-establish the connection from scratch using the init script we saw in The SIM7600 HAT section.

SSH process and tunnel: during testing, we identified several cases where SSH processes, both on drop box side and on control server side, became misaligned especially due to connection problems. This caused that, on one side or the other, the SSH process remained up but the tunnel was actually no longer active. We identified the following cases.

• Busy control server local port: following loss of connection, it happened that, in the control server, the local port remained in use by the lost SSH tunnel. When the drop box retried the connection, it detected that the port was busy but only printed a warning and still sent the SSH process into the background, even though the tunnel had not been restored. From the next minute on, the script would detect SSH as active and, even if the control server port was freed, the tunnel would never be restored. The check we added in the calling home script to resolve this case was to look for that specific warning and, if present, to kill the SSH process. As soon as the local port on the control server became available again, this would allow the tunnel to be re-established.
• Public IP change: we noticed, especially using the operator Iliad, that the assigned public IP changed frequently. This happened even when the SSH tunnel was established but it was not in use (i.e., we didn’t connect back from the control server to the drop box). This led to a mismatch similar to that seen in the previous point and thus the inability to re-establish the SSH tunnel. We therefore added a check that, each time the script was executed, kept track of the previous public IP address and, if it had changed, killed the SSH process. In order to avoid unplanned interruptions while we were connected to the drop box, the kill was executed only in case no one was currently connected via the call back.
• Generic network error: in some cases, it happened we lost the connection to the cellular data network, causing the usual SSH mismatch. Therefore, we implemented a check that verifies the status of the connection to the cellular network and, if it seemed down, killed the SSH process.

Forced refresh: in case any additional problems arise that were not expected from previous cases, we added a function that every half hour, regardless of the state of the connection and the SSH tunnel and if no one was connected to the drop box, killed SSH so that at the next minute the tunnel would be cleanly re-established.

Drop box restart: again in an effort to solve any unanticipated problems also if not related to the connection or SSH tunnel, we have added another function that every hour, if no one was connected to the drop box, completely restarted it. To avoid boot loops, the reboot was performed only when the uptime was more than one hour.

Thanks to these checks, if something went wrong we had to wait a minimum of one minute to a maximum of one hour to get back an SSH tunnel, obviously if we didn’t had an hardware problem (or someone pulled the cable!).

Future works

And that’s it, for now!

Despite the short time we had to build this drop box and to make it as reliable as we could, it did its job very well. During the project, we also came up with some ideas of additional features that, due to the limited time available, we could not add that could have extended the capabilities of the device. Some of them are: integration with a WHID Elite that we already have on hand, use of an encrypted partition where we can store any sensitive data, the addition of powerful USB wifi adapters for scanning and attacking wifi networks, the possibility to boot the drop box via calls and/or text messages sent directly to the HAT, the ability to bypass NAC (Network Access Control), generate various type of notifications that will alert us when the connection is established. Finally, we will definitely have to do some code review and optimization as well as find a way to better disguise it (for example, as a power supply) or at least create a special 3D printed case so that it does not look like an explosive ordnance!

Author

Mattia Merotto is a member of the Yarix’s Red Team and, between one penetration test and another, he likes to tinker with the hardware both to glean the secrets it hides and to repair it, even when the latter is derived from the former.