Ubuntu 20.04.1 LTS Libvirt + Cockpit + Docker + Portainer

Build minimal install box, add OpenSSH during installation
Watch your disk partitions! You will be using a lot (eventually)

Base install:

ssh to box:

# I'm bad, I do it all as root
sudo su - 
apt update
apt upgrade

libVirt install:

# libVirt install
apt install cpu-checker
apt install qemu-kvm libvirt-daemon-system libvirt-clients bridge-utils virtinst virt-manager
systemctl is-active libvirtd
# should output "active"
usermod -aG libvirt $USER
usermod -aG kvm $USER
exit
# Do this as your user also
sudo usermod -aG libvirt $USER
sudo usermod -aG kvm $USER
sudo brctl show

Cockpit:

sudo su - 
apt install cockpit -y
systemctl start cockpit
ss -tunlp | grep 9090
ufw allow 9090/tcp
apt install cockpit-machines cockpit-storaged cockpit-packagekit cockpit-networkmanager cockpit-dashboard cockpit-bridge
# Cockpit should now be available on https://ip:9090

Docker:

sudo apt-get update

sudo apt-get install apt-transport-https ca-certificates curl gnupg-agent 

sudo apt-get install software-properties-common

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository  "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt-get update

sudo apt-get install docker-ce docker-ce-cli containerd.io

sudo apt -y install mc iperf3 iptraf-ng

sudo docker run hello-world

Portainer

docker run -d --name portainer --restart unless-stopped -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer

NetData

apt install curl -y
bash <(curl -Ss https://my-netdata.io/kickstart.sh)

Links

# Cockpit is on:
https://SERVER_IP:9090/

# Portainer is on:
http://SERVER_IP:9000/

# Netdata is on
http://SERVER_IP:19999

This was needed in CentOS 8, not sure about Ubuntu 20.04 yet:

Set up network bridge:
Cockpit - Networking - Add Bridge
Select your ethernet card (em1) *(ymmv)
Name - bridge0
Reboot. Really. This may save some headaches later.


For desktop OS type guests, you should be good to go, for servers you have to use a bridge to get on the right network.

Assuming you added the bridge above (Cockpit – Network – Add Bridge)

Virtual machines – VM – Networking – Interface Type – Bridge to LAN, Source – bridge0, e1000e or virt, whatever.

Restart the VM, it should appear on your network. You may have to reboot the host if this is the first time through!

Beware that firewalld and iptables are all in this mess! You may have to enable ports or disable the firewall on the LAN side to get your server to work.

If you can’t build a new VM or can’t connect, reboot the box at least once before freaking out.

ISO’s to install from

ISOs should live in /var/lib/libvirt/images/ probably want to mount that as a read only nfs mount to your NAS:

Go to cockpit – storage and add the NFS share to that path, stupid easy.

Portainer templates

  • Open Portainer (http://SERVER_IP:9000/)
  • Go to Settings
  • Go to App Templates
  • Select “Use External Templates”
  • Paste in:
  • https://raw.githubusercontent.com/SelfhostedPro/selfhosted_templates/master/Template/template.json
  • Click “Save Settings”
  • Go to “App Templates” in the blue bar menu
  • Turn on “Show Container Templates”

Storage changes

If you need to change storage locations for libVirt, follow this: https://linuxconfig.org/configure-default-kvm-virtual-storage-on-redhat-linux#h6-1-create-new-virtual-storage-directory

Kali and Openvas GVM Setup

Cheat sheet:

# Start
sudo gvm-start
# Stop
sudo gvm-stop
# Update the feed
sudo gvm-feed-update

Hit the web UI at: https://your.ip.add.ress:9392/

Build-out:
Make a new Kali machine on libvirt VM, lxc, lxd, proxmox, whatever, just not docker (for Kali or Openvas, too many updates that get eaten/lost/etc. and OpenVas is HUGE). 

sudo su - 
apt update
apt upgrade
systemctl enable ssh.service


apt install openvas
apt install gvm
gvm-setup

(If it fails with ERROR: The default postgresql version is not 13 required by libgvmd) See fix below.

gvm-setup
(should work now and update the feed etc. takes a LONG time just let it go) GET THE PASSWORD AT THE END!

Make it listen on on more than 127.0.0.1:
gvm-stop
sed -ibak -e 's/127.0.0.1/0.0.0.0/g' /lib/systemd/system/greenbone-security-assistant.service
sed -ibak -e 's/127.0.0.1/0.0.0.0/g' /etc/default/greenbone-security-assistant
gvm-start
(Ignore the 127.0.0.1 there, it’s just in the script)
sudo gvm-feed-update
 to update the feed only
sudo gvm-start/stop 
to start or stop the service

Hit the web UI at: https://your.ip.add.ress:9392/
admin and the ong password it generated at the end of the setup

gvm-setup fix: edit:
vi /etc/postgresql/12/main/postgresql.conf
vi /etc/postgresql/13/main/postgresql.conf

Look for port = in both

Make v12 5433
Make v13 5432
systemctl restart postgresql

We aren’t using postgres for anything else here, so not being very careful with it
OR
FIX:
list your clusters
pg_lsclusters
and drop all clusters besides 13 cluster.
eg:
sudo pg_dropcluster 12 main –stop

Libvirt importing a virtual appliance

This directly from here: https://www.redhat.com/en/blog/importing-vms-kvm-virt-v2v

yum update

yum install virt-v2v

virsh pool-list
mkdir /var/lib/libvirt/Appliances
virsh pool-define-as Appliances –type dir –target /var/lib/libvirt/Appliances
virsh pool-start Appliances
virsh pool-autostart Appliances
virsh pool-list

Then import an OVA file:

virt-v2v -i ova /root/third_party_appliance.ova -o libvirt -of qcow2 -os Appliances -n default

Example: virt-v2v -i ova /home/brad/Nextcloud_VM_www.hanssonit.se.ova -o libvirt -of qcow2 -os Appliances -n default

If you get something like this:

[root@centos8 ~]# virt-v2v -i ova /home/brad/Nextcloud_VM_www.hanssonit.se.ova -o libvirt -of qcow2 -os Appliances -n default
[ 0.0] Opening the source -i ova /home/brad/Nextcloud_VM_www.hanssonit.se.ova
virt-v2v: warning: making OVA directory public readable to work around
libvirt bug https://bugzilla.redhat.com/1045069
[ 8.8] Creating an overlay to protect the source from being modified
[ 9.0] Initializing the target -o libvirt -os Appliances
[ 9.0] Opening the overlay
virt-v2v: error: libguestfs error: could not create appliance through
libvirt.

Try running qemu directly without libvirt using this environment variable:
export LIBGUESTFS_BACKEND=direct

Original error from libvirt: Cannot access backing file
‘/home/brad/Nextcloud_VM_www.hanssonit.se.ova’ of storage file
‘/var/tmp/v2vovl2af2bf.qcow2’ (as uid:107, gid:107): Permission denied
If reporting bugs, run virt-v2v with debugging enabled and include the
complete output:

virt-v2v -v -x […]

You work around this by running:
export LIBGUESTFS_BACKEND=direct
then do the virt-v2v again and it should work.

Now start your imported appliance:

[root@centos8 ~]# virsh list –all
Id Name State
1 kali running
Nextcloud_VM_www.hanssonit.se shut off

[root@centos8 ~]# virsh start Nextcloud_VM_www.hanssonit.se
error: Failed to start domain Nextcloud_VM_www.hanssonit.se
error: Cannot get interface MTU on ‘bridged’: No such device

You probably need to edit the network card to match your other VM’s

Centos 7 or 8 Libvirt + Cockpit virtual host

Build minimal install box of Centos 7 or 8. With 8 you can choose Virtual Host and Headless administration and it covers most of this right off the bat.

Watch your disk partitions! Changing storage after the fact isn’t terrible, but if you need to change storage, follow this: https://linuxconfig.org/configure-default-kvm-virtual-storage-on-redhat-linux#h6-1-create-new-virtual-storage-directory

SSH should be enabled and running:

sudo yum install openssh-server
sudo systemctl enable sshd
sudo systemctl start sshd
sudo systemctl status sshd
ssh to box:
yum -y update
yum -y install cockpit cockpit-machines
yum -y install qemu-kvm libvirt libvirt-python libguestfs-tools virt-install
yum -y install mc iperf3 iptraf-ng
yum -y install virt-install virt-viewer
systemctl start cockpit.socket
systemctl enable cockpit.socket
systemctl status cockpit.socket
firewall-cmd –add-service=cockpit –permanent
firewall-cmd –reload
# Cockpit is on: https://SERVER_IP:9090/

Centos 8 needs: 
sudo systemctl enable –now cockpit.socket

modprobe fuse
virt-host-validate
systemctl start libvirtd.service
systemctl enable libvirtd.service
systemctl status libvirtd.service

If needed, set up network bridge:
Cockpit – Networking – Add Bridge
Select your ethernet card (em1)

Should now be able to build virtual machines from within cockpit. You might need to reload the page or go out and back in for it to figure out libvirt is enabled.

If you can’t build a new VM, reboot the box at least once before freaking out.
ISO’s to install from:
ISOs should live in /var/lib/libvirt/images/ probably want to mount that as a read only nfs mount:
go to cockpit – storage and add the NFS or SMB share, stupid easy.

Optional
Netdata: Install netdata on CentOS 7 (and for docker host stats)

Docker: You *can* do docker too:
yum install -y cockpit-docker
sudo systemctl restart cockpit.socket

But compared to Portainer, there’s no comparison. Doesn’t mean you can’t do both on one box though, just that the cockpit plugin is garbage. Use Portainer.

sudo dnf config-manager –add-repo=https://download.docker.com/linux/centos/docker-ce.repo
sudo dnf repolist -v
dnf list docker-ce –showduplicates | sort -r
dnf install –nobest docker-ce –allowerasing (Removes buildah and cockpit-podman and podman, that’s ok, they suck.)

sudo systemctl enable –now docker
systemctl is-active docker
systemctl is-enabled docker

Docker compose:
curl -L “https://github.com/docker/compose/releases/download/1.23.2/docker-compose-$(uname -s)-$(uname -m)” -o docker-compose sudo mv docker-compose /usr/local/bin && sudo chmod +x /usr/local/bin/docker-compose

# Portainer
docker pull portainer/portainer:latest
docker volume create portainer_data
docker run -d –name portainer –restart unless-stopped -p 9000:9000 -v /var/run/docker.sock:/var/run/docker.sock -v portainer_data:/data portainer/portainer

Recover deleted files from a bitlocker’ed drive or partition

This uses dislocker, testdisk and photorec

You **WILL** need the recovery key for the bitlocker drive, this cannot get around that, but the drive will be unharmed in the process.

It’s a multistep process to be able to recover, this was accomplished with MX Linux 9.2  (a debian dervative), but anything new should be ok

I recommend making a Ventoy USB drive, copy the iso’s to it, boot up and select the OS/ISO to boot from.

Links: Ventoy MX Linux UltimateBootCD Windows 10 Kaspersky Live CD ESET Live CD (You can put whatever you want into the root of a Ventoy USB drive, it scans and lets you select them on bootup, very, very slick!)

Use a distro to clone the disk to your server first, so you can always restore it back to ground zero (such as it is) (Ultimate Boot CD recommended), then clone the drive FIRST I used Clonezilla to clone the disk to my server.

Pre-requisites:

You’ll need to mount a server drive or external disk to recover to!

Boot via USB Ventoy, Choose Ultimate Boot CD, Choose Clonezilla

Prep #
Open a terminal and run:
# like: sudo mount //HOST/SHARE mountpoint -o username=username,password=password
mkdir /mnt/server
sudo mount //192.168.0.253/backup /mnt/server -o username=brad,password=password
mkdir /mnt/decrypted-image
mkdir /mnt/decrypted-filesystem

# Use Clonezilla to back up the entire disk to your server

(Beyond the scope of this particular article, but there are many ways of doing it. Restoring can be difficult, so pay attention to how you backed it up.)

Recover deleted files from the free space on the disk
# Pre requisites (I switched to the MX Linux Distro for this part, again, using the Ventoy USB):
open a terminal
apt update
apt install dislocker testdisk photorec mc

# decrypt the partition and mount it as a loopback image (that’s the way it works):
dislocker /dev/sda2 -p111111-222222-333333-444444-555555-666666 -- /mnt/decrypted-image
# NOTE NO SPACE   after the -p and your Bitlocker key. You may have to play around with what sdX letter you use, but it is usually sda.
mount -o loop /mnt/decrypted-image/dislocker.file /mnt/decrypted-filesystem

you should now be able to see your regular Windows files in /mnt/decrypted-filesystem

testdisk has to be run with the disk as a parameter or it gets confused and you won’t be successful:

testdisk /mnt/decrypted-image/dislocker.file

Use testdisk to copy deleted files to /mnt/server/wherever:
Choose the Disk
Choose NONE as the partition type (I know, I know, just go with it)
Advanced
It should show NTFS
now Choose List
It should chug for a bit and then show your drive listing, if so, you’re good to proceed
Arrow right to Undelete
Press A to select all files
Press c to copy them
Select your destination (/mnt/server/wherever) and wait for it to copy

The recovered files are probably kinda underwhelming and disheartening, don’t freak out yet, go to the next phase of recovery below:

Recover LOTS MORE files! photorec is a lot better:

photorec /mnt/decrypted-image/dislocker.file

copy files to /mnt/server/wherever:
photorec /mnt/decrypted/dislocker.file
Choose the disk
Select the Partition (NTFS) Filesystem
choose Other FAT/NTFS/HFS+/ReiserFS
Choose free space or whole partition, your case may vary
Navigate to the directory you want to save files into
Press C when correct
WAIT. For a Very Long Time… 500GB spinning rust took about 36 hours over wifi , the wifi was not the bottleneck here, it takes a lot of overhead to reassemble the files.

Backup the regular files from disk:
copy the normal files out:
rsync -avx --progress /mnt/decrypted-filsystem /mnt/server/directory

You will want wired ethernet here, wifi chokes hard on sustained throughput (220MB/min vs. 1.5GB/min, do the math on how big your drive is and how long it will take)

Reset Windows password on Bitlocker drive

I was not able to reset the password using chgntpw or 0phtcrack on a Windows 8 box, but it can be done with a Windows boot CD (from Ventoy even!)

This does come with a bunch of caveats though…

  1. You will have to decrypt the drive. This takes a long time, SSD is your friend here.
  2. By decrypting the drive, you WILL (probably) lose any chance of recovering anything from the free space on the disk because the freespace is encrypted too, and will be decrypted, but the leftover data will very likely not be there. In theory, the bitlocker is whole disk, so when it decrypts every block, you should be able to recover, but if you can’t, it’s too late!
  • Boot up to the Windows 10 installer screen
  • DON’T GO THROUGH THE SETUP, LEAVE IT ON THE STARTUP SCREEN
  • press Shift + F10
  • At that command prompt run:

cd d:\windows\system32
move utilman.exe utilman.exe.bak
copy cmd.exe utilman.exe
wpeutil reboot

Note: once it starts back up, you have about 1 minute to do the next step or the utility will get replace and you have to do the commands above again (ask me how I know)

At the login screen, click the Utility manager in the corner, you should get another command prompt within your normal system now.

Run: net user [ENTER]

to see the usernames

To reset a password, assuming your username is brad, run:

net user brad my_new_password

To set up a new admin user, ignoring any local users:

net user bob /add
net localgroup administrators bob /add

exit

You should now be able to log in using the credentials/password you set above. You may have to reboot once for it to take effect. If you added a new admin user, log in as that user, and you can use the manage local users utilities to reset any user passwords on the box.

Continue reading “Recover deleted files from a bitlocker’ed drive or partition”

Flatten out a directory

find /dir1 -mindepth 2 -type f -exec mv -i ‘{}’ /dir1 ‘;’

take everything scattered throughout subdirectories and put them somewhere else in one big directory

sometimes you need this.

How to populate a ruby/erubis/erb template from a command line

Template test.erb:

# DESCRIPTORS

account      = “<%= account %>”

ownerorg     = “<%= org %>”

application  = “<%= application %>”

region       = “<%= region %>”

environment  = “<%= environment %>”

service      = “<%= service %>”

node_name_id = “<%= taxonomy %>”

version   = “<%= version %>”

# Populate with values:

erb key=value key=value template.erb >> output.file

erb account=12 org=23 application=34 region=45 environment=56 service=67 taxonomy=89 version=fred test.erb >> myfile.out

# DESCRIPTORS

account      = “12”

ownerorg     = “23”

application  = “34”

region       = “45”

environment  = “56”

service      = “67”

node_name_id = “89”

version   = “fred”

Cloud Engineer Resume

This may or may not sum up the resume’s we get nowadays:

Flobert Unpronouncable

“Computer engineer with a general-wide focus on cloud computing technologies; specifically, on virtualization, networking, storage, security, automation,

infrastructure as code, private and public cloud offerings. Directly and indirectly involved on the design, implementation, and support of virtual infrastructure

platforms for internal, external and virtual clients, aimed generally at different entities accounting for high-availability, disaster recovery, and scalability. Currently

working on a variety of internal/external/virtual learning and development initiatives by planning and supporting diverse training events for new and existing cloud

engineering practitioners on areas of interest like networking, security, cloud migration, virtualization, containerization, automation, and public cloud

platforms.”

  • buzzword

  • fragment

  • Place I worked at for a week in the 80’s

  • recipe for salsa

  • buzzword

  • incomplete senten

  • tuned carb on ’67 Impala

  • Atention to detial!

  • Punctuation( ,optional( @misused]}

  • Place I was fired from the day I started

  • Angular!

  • Buzzword buzzword

  • Something that happened within 30 feet of me and I’m claiming credit

  • 22 Years experience with AWS!

  • Windows 9 Experience

  • Buzzword buzzword buzzword buzzword

  • Lenox certiffied

Splitting big mp3’s into smaller pieces:

Splitting big mp3’s into smaller pieces:

mp3splt -a -t 15.1 -o Saint\ Death\ -\ Mark\ Dawson-@n -d Saint\ Death\ -\ Mark\ Dawson Saint\ Death\ -\ Mark\ Dawson.mp3

mp3splt -a -t 5.10 -o Lesson_01-@n -d Lesson_01 Lesson_01.mp3

The -a tells mp3splt to auto-adjust the split point with silence detection.

The -t 5.10 tells it to make the files 5 minutes and 10 seconds long since the file is a little over 30 minutes long. (This length may vary a bit due to the -a option).

The -o Lesson_01-@n tells it to name the files as Lesson_01 followed by a track number.

The -d Lesson_01 tells it to put the files in a directory called Lesson_01.

last is input file.mp3

So: mp3splt -a -t 5.10 -o outputfilename-@n -d OutputDirectory InputFilename.mp3

Plot lines no longer allowed:

  • Car won’t start
  • “You don’t understand!”
  • Just play along.
  • “Hide your wrongdoing, no one will ever find out, just keep spinning bigger and bigger lies!” Said every Disney live action kids show ever.
  • Stupid adults
  • Are we done here?/Can I go now?/Can I go home? (just admit that you’re guilty right now)
  • Cops/Military/Doctors that don’t listen to anything ever ever ever
  • Exploding cars/every car accident = fireball of doom
  • Showering at midnight at abandoned campground/mental facility/dorm, you know, like ya do.
  • Maniacal laughing. Constantly. For No. Reason. At. All. Somebody just shoot the fucker.
  • Girls that can’t wink should not try.
  • No one blows kisses, do not try.
  • No eating apples on screen, please. It’s not medieval, and it’s not fun to listen to.
  • Close-miking of anyone eating anything. Do not.
  • “No, no, really, it’s nothing…” from people that should damn well be reporting this shit asap.
  • Professional __________ that know shit about ___________
  • Outright dismissal of clear evidence, and just moving on like no one is going to notice.
  • Stupid dads that can’t change a diaper or wipe a nose or anything beyond grunt
  • Stupid dads that can’t fix anything ever
  • Water does not shoot straight up from a drain. Ever.
  • Don’t check if he’s dead, just assume everything goes as planned.
  • Super-Ultra eyeballs that can see a bulge in a pocket from across town, or spot someone in a crowd from 5 blocks away
  • Super-Ultra hearing so that one guy on the opposite end of a crowded stadium can speak normally, yet address thousands of people. In 1200 B.C.
  • “Look! There he is!” 300 yards away in a crowd. Seriously?
  • Snipers with handguns shooting things miles away
  • Snipers shooting with the barrel hanging out the window
  • Anyone with a handgun hitting anything beyond 50 feet while running
  • Security footage with obvious moves/pans/scans/zooms
  • Asking questions and getting no answers at all or utterly random responses like everyone is crazy
  • Zoom! Enhance!!
  • Ook Ook! Pretty nurse!! Grunt. Grunt. No answer question. Zog rather die. Ok, stepped on nail. With dick. Light bulb in ass cuz I fell on it.
  • For the love of God, WATCH THE ROAD!!! Especially through the intersection with the trucks!
  • Doctors being risky and edgy and almost losing their license every single day
  • Next of kin being in the room during surgery micro-managing the surgeon