recover cloud init password aws reset password instance remount cloud-init
SCRIPTED FIX IS FURTHER DOWN, KEEP SCROLLING
Part of this is directly cribbed from AWS Docs!
Symptoms:
Password change required but no TTY available.
WARNING: Your password has expired
Logging in with SSH but getting prompted to change your password for cloud-user, and those are random to begin with, so can’t change it cuz you don’t know it
Problem:
cloud-user account password has expired. account expiry has nothing to do with ssh key validity, as shown by getting connected and then getting the change password prompt (you’re already in via SSH, but PAM kicks in to force the password change)
Manual Fix:
Stop the instance (sorry)(DO NOT terminate!)
Copy out user data
Replace user-data with this:
Content-Type: multipart/mixed; boundary=”//”
MIME-Version: 1.0
<span style="font-size: 14px; color: rgb(51, 51, 51); font-family: Consolas, "Andale Mono WT", "Andale Mono", "Lucida Console", "Lucida Sans Typewriter", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Liberation Mono", "Nimbus Mono L", Monaco, "Courier New", Courier, monospace; font-variant-caps: normal; font-variant-ligatures: normal; line-height: 1.6em;"–<//
Content-Type: text/cloud-config; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”cloud-config.txt”
#cloud-config
cloud_final_modules:
– [scripts-user, always]
<span style="font-size: 14px; color: rgb(51, 51, 51); font-family: Consolas, "Andale Mono WT", "Andale Mono", "Lucida Console", "Lucida Sans Typewriter", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Liberation Mono", "Nimbus Mono L", Monaco, "Courier New", Courier, monospace; font-variant-caps: normal; font-variant-ligatures: normal; line-height: 1.6em;"–<//
Content-Type: text/x-shellscript; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”userdata.txt”
#!/bin/bash
/usr/bin/chage -d 65535 cloud-user
<span style="font-size: 14px; color: rgb(51, 51, 51); font-family: Consolas, "Andale Mono WT", "Andale Mono", "Lucida Console", "Lucida Sans Typewriter", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Liberation Mono", "Nimbus Mono L", Monaco, "Courier New", Courier, monospace; font-variant-caps: normal; font-variant-ligatures: normal; line-height: 1.6em;"–<//
start the instance again
WAIT AT LEAST ONE MINUTE!!
User data happens async from when the box starts up, so sit tight a minute or two.
SSH login should now work
Do you need to put your user data back?
You have to stop the instance to replace the user-data section, and it will fire again on startup. Decide if this is what you need, or leave it as is, or stop it and blank out user data.
If you replace your user-data to what it was before, add this section to prevent this from recurring:
# Fix the cloud-user password age issue
sed -i.bak -e ‘/Defaults.*requiretty/s/^/#/’ /etc/sudoers
chage -d 65535 cloud-user
Ok, so, scripted version:
You need:
my-user-data (below)
aws-replace-user-data.sh (below)
aws cli installed and working
aws-saml-auth installed and working
my-user-data:
#cloud-boothook
Content-Type: multipart/mixed; boundary=”//”
MIME-Version: 1.0
–//
Content-Type: text/cloud-config; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”cloud-config.txt”
#cloud-config
cloud_final_modules:
– [scripts-user, always]
–//
Content-Type: text/x-shellscript; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”userdata.txt”
#!/bin/bash -x
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
groupadd -g 25638 rundeck
#adduser -g rundeck rundeck
useradd -p $(openssl passwd -1 9837y45fhyiwurhef84yf93y4978yhfh) -g rundeck rundeck
echo “9837y45fhyiwurhef84yf93y4978yhfh” | passwd –stdin rundeck
usermod -c “Rundeck Service Account.” -u 35638 rundeck
usermod rundeck -G wheel
gpasswd -a rundeck wheel
getent passwd rundeck > /dev/null 2&>1
if [ $? -eq 0 ]; then
chage -I -1 -m 0 -M 99999 -E -1 rundeck
else
echo “Skipping, user does not exist”
fi
getent passwd cloudse > /dev/null 2&>1
if [ $? -eq 0 ]; then
chage -I -1 -m 0 -M 99999 -E -1 cloudse
else
echo “Skipping, user does not exist”
fi
getent passwd cloud-user > /dev/null 2&>1
if [ $? -eq 0 ]; then
chage -I -1 -m 0 -M 99999 -E -1 cloud-user
else
echo “Skipping, user does not exist”
fi
getent passwd ec2-user > /dev/null 2&>1
if [ $? -eq 0 ]; then
chage -I -1 -m 0 -M 99999 -E -1 ec2-user
else
echo “Skipping, user does not exist”
fi
# chage failures make the cloud-init fail, so use the logic blocks above
# chage -I -1 -m 0 -M 99999 -E -1 rundeck
# Fix rundeck sudo privs
rm -f /etc/sudoers.d/rundeck
cat > “/etc/sudoers.d/rundeck” << EOF
# This file is managed by Chef.
# Do NOT modify this file directly.
%rundeck ALL=(ALL) NOPASSWD:ALL
Defaults:%rundeck !requiretty
EOF
# Rundeck env setup
mkdir -p /home/rundeck/.ssh/
chown -R rundeck:rundeck /home/rundeck
rm -f /home/rundeck/.ssh/authorized_keys
# Add Rundeck key
cat > “/home/rundeck/.ssh/authorized_keys” << EOF
ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzxyMM1ozCoxZNe0q7PeiJdtqUQc6VKhAY46dmDET4Q+lvcmkDdE3q8IVCkrI8MES2j9YBoCy00BV3kAWRDTilq0CArDSVpTp5lz/2Fgu/EaxTMQKk2XiGGW4M4QUkAQRgHDNT1k8lYIhRENFS8Csf9Bt2lFOgWH18sw7s4GuCULbBfDZdsMVHN6wctv0j2vbvcPdg+QX2gg7TY4HdWoDQ3OSOrWSyAeseXA81h3+OZSKhyFmTIFzk0+8uxuv18CdilfyhCiDJqGwVV5WpbOTXlimT92ea6R5V1H8KeuhZEMnpuiCUjbQzGKuUsbKHu+bVWQqQMh9VS9VlC9Q0kqxFw==
EOF
chmod -R 600 /home/rundeck/.ssh/authorized_keys
chown -R rundeck:rundeck /home/rundeck/.ssh/authorized_keys
chmod -R 600 /var/spool/mail/rundeck
chown -R rundeck:rundeck /var/spool/mail/rundeck
chmod 440 /etc/sudoers.d/rundeck
# Temporary
# echo rundeck:8y5g9eyhrgy3875ty98374hfjhwef | chpasswd
touch /root/cloud-init-fix-was-here
–//
aws-replace-user-data.sh:
#!/bin/bash
EXPECTED_ARGS=6
E_BADARGS=65
if [ $# -ne $EXPECTED_ARGS ]
then
echo “All parameters are required, in any order:”
echo “-i instance_id ( i-0b1fd14e321099e6d )”
echo “-r region ( us-east-1, us-west-2, etc )”
echo “-f filename ( aws-recovery-user-data in current path )”
exit $E_BADARGS
fi
while getopts “:i:r:f:” opt; do
case $opt in
i)
echo “-i was triggered, Parameter: $OPTARG” >&2;
INSTANCE=$OPTARG;
;;
r)
echo “-r was triggered, Parameter: $OPTARG” >&2;
REGION=$OPTARG;
;;
f)
echo “-f was triggered, Parameter: $OPTARG” >&2;
FILENAME=$OPTARG;
;;
*)
echo “Invalid option: -$OPTARG” >&2
echo Example: ./aws-replace-user-data.sh -i i-0b1fd14e321099e6d -r us-west-2
exit 1
break
;;
🙂
echo “Option -$OPTARG requires an argument.” >&2
echo Example: ./aws-replace-user-data.sh -i i-0b1fd14e321099e6d -r us-west-2 -f my-user-data
exit 1
break
;;
esac
done
# You can preset or override some things if its your machine
echo Using instance ID: ${INSTANCE}
echo Using region: ${REGION}
echo Using filename: ${FILENAME}
# aws-saml-auth
# pip install –upgrade –user awscli
# Stop
aws ec2 –region ${REGION} stop-instances –instance-ids ${INSTANCE}
# Better!
while [ true ]
do aws ec2 describe-instances –filter “Name=instance-state-name,Values=stopped” –region ${REGION} –instance-id ${INSTANCE} |grep stopped && break
echo Not stopped yet
sleep 5
echo retrying…
done
# Base64 encode your file:
echo “Encoding ” ${FILENAME}
rm -f ${FILENAME}.base64
base64 ${FILENAME} >> ${FILENAME}.base64
# Modify!
echo “Modifying instance: “
aws ec2 modify-instance-attribute –region ${REGION} –attribute userData –value file://${FILENAME}.base64 –instance-id ${INSTANCE}
# Start
echo “Starting instance: “
aws ec2 –region ${REGION} start-instances –instance-ids ${INSTANCE}
# Optional
while [ true ]
do aws ec2 describe-instances –filter “Name=instance-state-name,Values=running” –region ${REGION} –instance-id ${INSTANCE} |grep running && break
echo Not started yet
sleep 5
echo retrying…
done
Usage:
./aws-replace-user-data.sh -i i-050bea589a55ac039 -r us-west-2 -f my-user-data
punch it in the face backdoor user data, this creates a “backdoor” account, use this when rundeck user is fubared and you can’t get in:
Content-Type: multipart/mixed; boundary=”//”
MIME-Version: 1.0
–//
Content-Type: text/cloud-config; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”cloud-config.txt”
#cloud-config
cloud_final_modules:
– [scripts-user, always]
–//
Content-Type: text/x-shellscript; charset=”us-ascii”
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: attachment; filename=”userdata.txt”
#!/bin/bash
PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin
useradd -p $(openssl passwd -1 9837y45fhyiwurhef84yf93y4978yhfh) backdoor
echo “9837y45fhyiwurhef84yf93y4978yhfh” | passwd –stdin backdoor
usermod -c “Recovery account, please remove.” backdoor
usermod backdoor -G wheel
gpasswd -a backdoor wheel
<span style="font-size: 14px;"–<//
