git 2.19.1 upgrade

This should get handled as part of normal patching, but RHEL ships with a 1.8.x branch of Git. We aren’t cloning from public repos for most things, but I can’t be 100% about it, and workstations are probably vulnerable as well. We should all update our workstations, but it doesn’t appear to be that simple for Linux servers:

 

(2.19.1 is the version to upgrade to.)

 

Mac: brew upgrade git didn’t seem to update the cli git, still on 1.8.3.1, have to download and install from https://sourceforge.net/projects/git-osx-installer/files/git-2.19.0-intel-universal-mavericks.dmg/download?use_mirror=autoselect

 

Windows: https://github.com/git-for-windows/git/releases/download/v2.19.1.windows.1/Git-2.19.1-64-bit.exe

 

Linux: Looks like git does not have an rpm for a 2.x, it’s a clone and compile, which makes long term sustainability a pain.

CVE says version 2.18 is the oldest affected version, and the newest available from the yum repos is 1.8.3.1-14.

https://git-scm.com/download/linux says to download and compile.

 

This works:

#!/bin/bash

# update git to 2.19.1

git –version

which git

cp /usr/bin/git /usr/bin/git-1.8.3.1

yum -y install curl-devel expat-devel gettext-devel openssl-devel zlib-devel

yum -y install gcc perl-ExtUtils-MakeMaker

cd /usr/src

wget https://www.kernel.org/pub/software/scm/git/git-2.19.1.tar.gz

tar xzf git-2.19.1.tar.gz

cd git-2.19.1

make prefix=/usr/local/git all

make prefix=/usr/local/git install

rm -f /bin/git

ln -s /usr/local/git/bin/git /bin/git

git –version

 

chef-client fails with ERROR: The used Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

TL;DR:

Your path is probably wrong for the root user that is running chef-client. We had a long screwed up path that eventually included the right path, but had an old ChefDK preceeding it. Basically, you’re using a broken chef-client (too  new, too old, broken encryption, whatever)

Works:

PATH=/usr/bin:/bin:/etc:.:/usr/local/bin:/usr/sbin:/opt/OV/bin/OpC:/home/rundeck/:/opt/middleware/Tools

Does not: 

PATH=/usr/local/rvm/gems/ruby-2.3.1/bin:/usr/local/rvm/gems/ruby-2.3.1@global/bin:/usr/local/rvm/rubies/ruby-2.3.1/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/root:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rvm/bin:/root/bin

Fix:

export PATH=/usr/bin:/bin:/etc:.:/usr/local/bin:/usr/sbin:/opt/OV/bin/OpC:/home/rundeck/:/opt/middleware/Tools

Permanent fix:

vi /etc/bash.bashrc

Change to:

type rvm >/dev/null 2>/dev/null || echo ${PATH} | __rvm_grep “/usr/local/rvm/bin” > /dev/null || export PATH=”${PATH}:/usr/local/rvm/bin”

vi /root/.bashrc

export PATH=/usr/bin:$PATH:/root/bin

But this hoses up ruby, better to get the right version of chef-client:

mv /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client-12.21.1

mv /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client-12.21.1

mv /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client-12.21.1

/usr/bin/chef-client –version

cd /usr/local/rvm/gems/ruby-2.3.1/bin/

ln /usr/bin/chef-client chef-client

chef-client –version

If you are running a specific version of chef-client, but running bare chef-client gives the wrong version, check “which chef-client” and rename the ones buried in the ruby paths:

mv /usr/local/rvm/gems/ruby-2.5.3/bin/chef-client /usr/local/rvm/gems/ruby-2.5.3/bin/chef-client.12.19.36

mv /usr/local/rvm/gems/ruby-2.4.2/bin/chef-client /usr/local/rvm/gems/ruby-2.4.2/bin/chef-client.14.9.13

etc.

logout and back in

chef-client -v 

should give you the proper version.

Frequent failure modes:

Needs newer version 12.21.31+:

[2019-03-18T14:50:54+00:00] ERROR: Cookbook ‘wdpr_dnsmasq’ version ‘0.0.1’ depends on chef version [“>= 12.21.31”], but the running chef version is 12.19.36

[2019-03-18T14:50:54+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Needs older version until cook book is updated:

Indicative of running chef-client 14 against incompatible cookbooks:

Running handlers:

[2019-03-18T14:54:29+00:00] ERROR: Running exception handlers

Running handlers complete

[2019-03-18T14:54:29+00:00] ERROR: Exception handlers complete

Chef Client failed. 0 resources updated in 07 seconds

[2019-03-18T14:54:30+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out

[2019-03-18T14:54:30+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report

[2019-03-18T14:54:30+00:00] ERROR: can’t modify frozen Array

[2019-03-18T14:54:30+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

590  [2018-10-10 15:27:33] echo ${PATH}

  591  [2018-10-10 15:29:27] chef-client

  592  [2018-10-10 15:30:07] which chef-client

  593  [2018-10-10 15:30:28] /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client –version

  594  [2018-10-10 15:30:40] mv /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client-12.21.1

  595  [2018-10-10 15:30:43] which chef-client

  596  [2018-10-10 15:30:55] /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client –version

  597  [2018-10-10 15:31:08] mv /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client-12.21.1

  598  [2018-10-10 15:31:10] which chef-client

  599  [2018-10-10 15:31:20] /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client –version

  600  [2018-10-10 15:31:55] mv /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client-12.21.1

  601  [2018-10-10 15:31:57] which chef-client

  602  [2018-10-10 15:32:05] /usr/bin/chef-client –version

  603  [2018-10-10 15:32:15] chef-client

  604  [2018-10-10 15:32:24] which chef-client

  605  [2018-10-10 15:32:30] chef-client

  606  [2018-10-10 15:32:38] cd /usr/local/rvm/gems/ruby-2.3.1/bin/

  607  [2018-10-10 15:32:39] ll

  608  [2018-10-10 15:33:28] ln chef-client /usr/bin/chef-client

  609  [2018-10-10 15:33:38] ln /usr/bin/chef-client chef-client

  610  [2018-10-10 15:33:42] chef-client

  611  [2018-10-10 15:35:00] history

[rundeck][nl-fldi-02119][~]

$ chef-client –version

Chef: 12.19.36

Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

openssl enc -help 2>&1 | grep gcm

[bwilliam@nl-fldi-02119 ~]$ openssl enc -help 2>&1 | grep gcm

-aes-128-ctr               -aes-128-ecb               -aes-128-gcm

-aes-192-gcm               -aes-192-ofb               -aes-256-cbc

-aes-256-ecb               -aes-256-gcm               -aes-256-ofb

The used Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

knife data bag show −−secret-file=./rev_secret_key rev_secret revpass

knife data bag show −−secret-file=/etc/chef/encrypted_data_bag_secret users rundeck

[2018-10-10T09:38:11-04:00] ERROR: The used Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

Recipe Compile Error in /var/chef/cache/cookbooks/wdprt_rundeck_client/recipes/default.rb

  ================================================================================

  Chef::EncryptedDataBagItem::EncryptedDataBagRequirementsFailure

  —————————————————————

  The used Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

[root@nl-fldi-02119 chef]# chef-client –version

Chef: 12.21.1

knife data bag show users rundeck

  cipher:         aes-256-gcm

which knife

which chef-client

[rundeck][nl-fldi-02119][~]

$ chef-client –version

Chef: 12.19.36

knife data bag show users rundeck

Recovering AWS instance that is prompting for password (happens when using SSH Key only logins)

recover cloud init password aws reset password instance remount cloud-init

Part of this is directly cribbed from AWS Docs!

Symptoms:

Password change required but no TTY available.

WARNING: Your password has expired

Logging in with SSH but getting prompted to change your password for cloud-user, and those are random to begin with, so can’t change it cuz you don’t know it

Problem:

cloud-user account password has expired. account expiry has nothing to do with ssh key validity, as shown by getting connected and then getting the change password prompt (you’re already in via SSH, but PAM kicks in to force the password change)

Fix:

Stop the instance (sorry)(do not terminate!)

Copy out user data

Replace user-data with this:

Content-Type: multipart/mixed; boundary=”//”

MIME-Version: 1.0

<span style="font-size: 14px; color: rgb(51, 51, 51); font-family: Consolas, "Andale Mono WT", "Andale Mono", "Lucida Console", "Lucida Sans Typewriter", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Liberation Mono", "Nimbus Mono L", Monaco, "Courier New", Courier, monospace; font-variant-caps: normal; font-variant-ligatures: normal; line-height: 1.6em;"–<//

Content-Type: text/cloud-config; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”cloud-config.txt”

#cloud-config

cloud_final_modules:

– [scripts-user, always]

<span style="font-size: 14px; color: rgb(51, 51, 51); font-family: Consolas, "Andale Mono WT", "Andale Mono", "Lucida Console", "Lucida Sans Typewriter", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Liberation Mono", "Nimbus Mono L", Monaco, "Courier New", Courier, monospace; font-variant-caps: normal; font-variant-ligatures: normal; line-height: 1.6em;"–<//

Content-Type: text/x-shellscript; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”userdata.txt”

#!/bin/bash

/usr/bin/chage -d 65535 cloud-user

<span style="font-size: 14px; color: rgb(51, 51, 51); font-family: Consolas, "Andale Mono WT", "Andale Mono", "Lucida Console", "Lucida Sans Typewriter", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Liberation Mono", "Nimbus Mono L", Monaco, "Courier New", Courier, monospace; font-variant-caps: normal; font-variant-ligatures: normal; line-height: 1.6em;"–<//

start the instance again

WAIT AT LEAST ONE MINUTE!!

User data happens async from when the box starts up, so sit tight a minute or two.

SSH login should now work

Do you need to put your user data back?

You have to stop the instance to replace the user-data section, and it will fire again on startup. Decide if this is what you need, or leave it as is, or stop it and blank out user data.

If you replace your user-data to what it was before, add this section to prevent this from recurring:

# Fix the cloud-user password age issue

sed -i.bak -e ‘/Defaults.*requiretty/s/^/#/’ /etc/sudoers

chage -d 65535 cloud-user

Ok, so, scripted version:

You need:

my-user-data (below)

aws-replace-user-data.sh (below)

aws cli installed and working

aws-saml-auth installed and working

my-user-data:

#cloud-boothook

Content-Type: multipart/mixed; boundary=”//”

MIME-Version: 1.0

–//

Content-Type: text/cloud-config; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”cloud-config.txt”

#cloud-config

cloud_final_modules:

– [scripts-user, always]

–//

Content-Type: text/x-shellscript; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”userdata.txt”

#!/bin/bash -x

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

groupadd -g 25638 rundeck

#adduser -g rundeck rundeck

useradd -p $(openssl passwd -1 9837y45fhyiwurhef84yf93y4978yhfh) -g rundeck rundeck

echo “9837y45fhyiwurhef84yf93y4978yhfh” | passwd –stdin rundeck

usermod -c “Rundeck Service Account.” -u 35638 rundeck

usermod rundeck -G wheel

gpasswd -a rundeck wheel

getent passwd rundeck > /dev/null 2&>1

if [ $? -eq 0 ]; then

    chage -I -1 -m 0 -M 99999 -E -1 rundeck

else

    echo “Skipping, user does not exist”

fi

getent passwd cloudse > /dev/null 2&>1

if [ $? -eq 0 ]; then

    chage -I -1 -m 0 -M 99999 -E -1 cloudse

else

    echo “Skipping, user does not exist”

fi

getent passwd cloud-user > /dev/null 2&>1

if [ $? -eq 0 ]; then

    chage -I -1 -m 0 -M 99999 -E -1 cloud-user

else

    echo “Skipping, user does not exist”

fi

getent passwd ec2-user > /dev/null 2&>1

if [ $? -eq 0 ]; then

    chage -I -1 -m 0 -M 99999 -E -1 ec2-user

else

    echo “Skipping, user does not exist”

fi

# chage failures make the cloud-init fail, so use the logic blocks above

# chage -I -1 -m 0 -M 99999 -E -1 rundeck

# Fix rundeck sudo privs

rm -f /etc/sudoers.d/rundeck

cat > “/etc/sudoers.d/rundeck” << EOF

# This file is managed by Chef.

# Do NOT modify this file directly.

%rundeck ALL=(ALL) NOPASSWD:ALL

Defaults:%rundeck !requiretty

EOF

# Rundeck env setup

mkdir -p /home/rundeck/.ssh/

chown -R rundeck:rundeck /home/rundeck

rm -f /home/rundeck/.ssh/authorized_keys

# Add Rundeck key

cat > “/home/rundeck/.ssh/authorized_keys” << EOF

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzxyMM1ozCoxZNe0q7PeiJdtqUQc6VKhAY46dmDET4Q+lvcmkDdE3q8IVCkrI8MES2j9YBoCy00BV3kAWRDTilq0CArDSVpTp5lz/2Fgu/EaxTMQKk2XiGGW4M4QUkAQRgHDNT1k8lYIhRENFS8Csf9Bt2lFOgWH18sw7s4GuCULbBfDZdsMVHN6wctv0j2vbvcPdg+QX2gg7TY4HdWoDQ3OSOrWSyAeseXA81h3+OZSKhyFmTIFzk0+8uxuv18CdilfyhCiDJqGwVV5WpbOTXlimT92ea6R5V1H8KeuhZEMnpuiCUjbQzGKuUsbKHu+bVWQqQMh9VS9VlC9Q0kqxFw==

EOF

chmod -R 600 /home/rundeck/.ssh/authorized_keys

chown -R rundeck:rundeck /home/rundeck/.ssh/authorized_keys

chmod -R 600  /var/spool/mail/rundeck

chown -R rundeck:rundeck  /var/spool/mail/rundeck

chmod 440 /etc/sudoers.d/rundeck

# Temporary

# echo rundeck:8y5g9eyhrgy3875ty98374hfjhwef | chpasswd

touch /root/cloud-init-fix-was-here

–//

aws-replace-user-data.sh:

#!/bin/bash

EXPECTED_ARGS=6

E_BADARGS=65

if [ $# -ne $EXPECTED_ARGS ]

then

        echo “All parameters are required, in any order:”

        echo “-i instance_id ( i-0b1fd14e321099e6d )”

        echo “-r region ( us-east-1, us-west-2, etc )”

        echo “-f filename ( aws-recovery-user-data in current path )”

  exit $E_BADARGS

fi

while getopts “:i:r:f:” opt; do

  case $opt in

    i)

      echo “-i was triggered, Parameter: $OPTARG” >&2;

        INSTANCE=$OPTARG;

      ;;

    r)

      echo “-r was triggered, Parameter: $OPTARG” >&2;

        REGION=$OPTARG;

      ;;

    f)

      echo “-f was triggered, Parameter: $OPTARG” >&2;

        FILENAME=$OPTARG;

      ;;

    *)

      echo “Invalid option: -$OPTARG” >&2

        echo Example: ./aws-replace-user-data.sh -i i-0b1fd14e321099e6d -r us-west-2

      exit 1

        break

      ;;

    🙂

      echo “Option -$OPTARG requires an argument.” >&2

        echo Example: ./aws-replace-user-data.sh -i i-0b1fd14e321099e6d -r us-west-2 -f my-user-data

      exit 1

        break

      ;;

  esac

done

# You can preset or override some things if its your machine

echo Using instance ID: ${INSTANCE}

echo Using region: ${REGION}

echo Using filename: ${FILENAME}

# aws-saml-auth

# pip install –upgrade –user awscli

# Stop

aws ec2 –region ${REGION} stop-instances –instance-ids ${INSTANCE}

# Better!

while [ true ]

    do  aws ec2 describe-instances –filter “Name=instance-state-name,Values=stopped” –region ${REGION} –instance-id ${INSTANCE} |grep stopped && break

    echo Not stopped yet

    sleep 5

    echo retrying…

done

# Base64 encode your file:

echo “Encoding ” ${FILENAME}

rm -f ${FILENAME}.base64

base64 ${FILENAME} >> ${FILENAME}.base64

# Modify!

echo “Modifying instance: “

aws ec2 modify-instance-attribute –region ${REGION} –attribute userData –value file://${FILENAME}.base64 –instance-id ${INSTANCE}

# Start

echo “Starting instance: “

aws ec2 –region ${REGION} start-instances –instance-ids ${INSTANCE}

# Optional

while [ true ]

    do  aws ec2 describe-instances –filter “Name=instance-state-name,Values=running” –region ${REGION} –instance-id ${INSTANCE} |grep running && break

    echo Not started yet

    sleep 5

    echo retrying…

done

Usage:

./aws-replace-user-data.sh -i i-050bea589a55ac039 -r us-west-2 -f my-user-data

punch it in the face backdoor:

Content-Type: multipart/mixed; boundary=”//”

MIME-Version: 1.0

–//

Content-Type: text/cloud-config; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”cloud-config.txt”

#cloud-config

cloud_final_modules:

– [scripts-user, always]

–//

Content-Type: text/x-shellscript; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”userdata.txt”

#!/bin/bash

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

useradd -p $(openssl passwd -1 9837y45fhyiwurhef84yf93y4978yhfh) backdoor

echo “9837y45fhyiwurhef84yf93y4978yhfh” | passwd –stdin backdoor

usermod -c “Recovery account, please remove.”  backdoor

usermod backdoor -G wheel

gpasswd -a backdoor wheel

<span style="font-size: 14px;"–<//