So as it turns out, if you manually attach a second ENI to an EC2 Ubuntu instance, it will not work right away. Neither will it work if you reboot the instance. In fact it just won’t work at all.
This is because Ubuntu is not able to recognize additional EC2 ENIs as ‘plug-in-play’ devices. You can confirm this by trying to ping from the first private IP to the second private IP, or by simply issuing an $ ifconfig command. You will see only eth0, and not eth1.
So, in order to be able to SSH into the secondary private IP (or public IP of the ENI), you first need to manually create a config file for the eth1 device at the same directory location of the eth0 config. This happens to reside at /etc/network/interfaces.d/
I created the file as eth1.cfg, and this is what I put inside it:
# secondary eth1 interface
iface eth1 inet static
At this point you can reboot your instance, and upon reboot you should now be able to ping from the first private IP to the second, and the eth1 should show up when you enter $ ifconfig. If not, you probably set up the config file wrong.
After testing to confirm the eth1 device is set up and configured correctly, you then need to add an internal route so that traffic can flow properly to and from the IP. For this, I used the following route add command:
sudo route add -net 10.0.0.0 netmask 255.255.255.0 dev eth1 gw 10.0.0.1
Now you should be good to go.