Making iTerm 2 highlight errors all the time

This helps when you have to chase down stupid stuff all day, every day:

Launch iTerm2

Go to iTerm2 – Preferences – Profiles – Advanced – Triggers – Edit

Click +

In the Regex, type:   (error|ERROR|Error)

Action: Highlight Text

Pick a color, Pick a background color that stand out for you

Check the Instant box

Close back out

I have:

(Fail|FAIL|fail)

(Differ|DIFFER|differ)

(Insufficient|INSUFFICIENT|insufficient)

(Access|ACCESS|access)

(Denied|DENIED|denied)

(Error|ERROR|error)

Sorting out (and finding!) connection resets on Linux:

Sorting out connection resets:

tcpdump -nn -v ‘tcp[tcpflags] & (tcp-rst) != 0’

or

tcpdump -ilo -nn -v ‘tcp[tcpflags] & (tcp-rst) != 0’

-n is to stop resolving DNS names

-nn is to stop resolving DNS names AND port numbers to names

This is really useful when an app is logging connection resets, but not telling you what it is trying to connect to in the first place.

Reading tags from CLI or bash in ec2 instances

# To read the Name tag:

TAG_NAME=”Name”

INSTANCE_ID=”`./ec2-metadata -i | cut -f 2 -d ” ” `”

REGION=”`././ec2-metadata -z | cut -f 2 -d ” ” `”

REGION=${REGION%?}

TAG_VALUE=”`aws ec2 describe-tags –filters “Name=resource-id,Values=$INSTANCE_ID” “Name=key,Values=$TAG_NAME” –region $REGION –output=text | cut -f5`”

Gives:

set |grep TAG_VALUE

TAG_VALUE=’brad test box’

# To see ALL tags to see WTF: (use region from above)

aws ec2 describe-tags –region us-east-1 –output=text

# To read the Created-by custom tag:

TAG_NAME=”created-by”

INSTANCE_ID=”`./ec2-metadata -i | cut -f 2 -d ” ” `”

REGION=”`././ec2-metadata -z | cut -f 2 -d ” ” `”

REGION=${REGION%?}

TAG_VALUE=”`aws ec2 describe-tags –filters “Name=resource-id,Values=$INSTANCE_ID” “Name=key,Values=$TAG_NAME” –region $REGION –output=text | cut -f5`”

set |grep TAG_VALUE

TAG_VALUE=test-kitchen

# To read the nifi-node-num custom tag:

TAG_NAME=”nifi-node-num”

echo “TAG_NAME is: ” ${TAG_NAME}

INSTANCE_ID=”`./ec2-metadata -i | cut -f 2 -d ” ” `”

echo “INSTANCE_ID is: ” ${INSTANCE_ID}

REGION=”`././ec2-metadata -z | cut -f 2 -d ” ” `”

REGION=${REGION%?}

echo “REGION is: ” ${REGION}

TAG_VALUE=”`aws ec2 describe-tags –filters “Name=resource-id,Values=$INSTANCE_ID” “Name=key,Values=$TAG_NAME” –region $REGION –output=text | cut -f5`”

echo “TAG_VALUE is: ” ${TAG_VALUE}

# You need the EC2 Metadata binary for the above to work

wget http://s3.amazonaws.com/ec2metadata/ec2-metadata

chmod u+x ec2-metadata

./ec2-metadata –help

You can also do this IF you have credentials:

aws ec2 describe-instances –region us-east-1 –instance-id i-09301dcede4431741|grep -A 200 Tag

                    “Tags”: [

                        {

                            “Value”: “False”,

                            “Key”: “data_sensitive”

                        },

                        {

                            “Value”: “10/31/2017”,

                            “Key”: “valid_thru”

                        },

                        {

                            “Value”: “nifi-al”,

                            “Key”: “Application”

                        },

                        {

                            “Value”: “Aplha”,

                            “Key”: “Cluster”

                        },

                        {

                            “Value”: “False”,

                            “Key”: “Docker”

                        },

                        {

                            “Value”: “TBD”,

                            “Key”: “BAPP_ID”

                        },

                        {

                            “Value”: “nifi-al-latest”,

                            “Key”: “Name”

                        },

                        {

                            “Value”: “WDPRTechnologyIAParksDataPlatform@disney.com“,

                            “Key”: “Owner”

                        },

                        {

                            “Value”: “1”,

                            “Key”: “nifi-node-num”

                        },

                        {

                            “Value”: “nifi-al-latest-asg”,

                            “Key”: “aws:autoscaling:groupName”

                        },

                        {

                            “Value”: “Non-Prod Sandbox”,

                            “Key”: “Environment”

                        },

                        {

                            “Value”: “yes”,

                            “Key”: “tag_compliance”

                        }

                    ],

                    “AmiLaunchIndex”: 2

                }

            ],

            “ReservationId”: “r-0a402050d68688b53”,

            “RequesterId”: “226008221399”,

            “Groups”: [],

            “OwnerId”: “876496569223”

        }

    ]

}

ALL ec2-metadata options:

Usage: ec2-metadata <option>

Options:

–all                     Show all metadata information for this host (also default).

-a/–ami-id               The AMI ID used to launch this instance

-l/–ami-launch-index     The index of this instance in the reservation (per AMI).

-m/–ami-manifest-path    The manifest path of the AMI with which the instance was launched.

-n/–ancestor-ami-ids     The AMI IDs of any instances that were rebundled to create this AMI.

-b/–block-device-mapping Defines native device names to use when exposing virtual devices.

-i/–instance-id          The ID of this instance

-t/–instance-type        The type of instance to launch. For more information, see Instance Types.

-h/–local-hostname       The local hostname of the instance.

-o/–local-ipv4           Public IP address if launched with direct addressing; private IP address if launched with public addressing.

-k/–kernel-id            The ID of the kernel launched with this instance, if applicable.

-z/–availability-zone    The availability zone in which the instance launched. Same as placement

-c/–product-codes        Product codes associated with this instance.

-p/–public-hostname      The public hostname of the instance.

-v/–public-ipv4          NATted public IP Address

-u/–public-keys          Public keys. Only available if supplied at instance launch time

-r/–ramdisk-id           The ID of the RAM disk launched with this instance, if applicable.

-e/–reservation-id       ID of the reservation.

-s/–security-groups      Names of the security groups the instance is launched in. Only available if supplied at instance launch time

-d/–user-data            User-supplied data.Only available if supplied at instance launch time.

If you don’t have credentials, some of this doesn’t work, but you can try this:

Websphere system core dump location is incorrect by default

Location of the system core dump is typically incorrect and just drops it in the middle of the host OS, you will need to edit the

/opt/apps/WebSphere/AppServer/bin/setupCmdLine.sh

# Add:

IBM_COREDIR=/net/cn-flor-nas01-prod.wdw.disney.com/data/TPR/WDW/Vol005/WDPRTHome/middleware/dumps/${HOSTNAME}

export COREDIR

# the COREDIR can also go at the end of the large export command at the end of the script.

JVM restart(s) are required to pick up the change. Even though the system dump is initiated, you have to pick the JVM that initiates it, so it is related to the JVM. A full restart of all the JVM’s is technically needed, as is changing this file on every node in the cluster.

“Incorrect” in that the dump files are freaking gigantic and run the box out of space pretty quickly. Redirect these to the NAS.

git 2.19.1 upgrade

This should get handled as part of normal patching, but RHEL ships with a 1.8.x branch of Git. We aren’t cloning from public repos for most things, but I can’t be 100% about it, and workstations are probably vulnerable as well. We should all update our workstations, but it doesn’t appear to be that simple for Linux servers:

 

(2.19.1 is the version to upgrade to.)

 

Mac: brew upgrade git didn’t seem to update the cli git, still on 1.8.3.1, have to download and install from https://sourceforge.net/projects/git-osx-installer/files/git-2.19.0-intel-universal-mavericks.dmg/download?use_mirror=autoselect

 

Windows: https://github.com/git-for-windows/git/releases/download/v2.19.1.windows.1/Git-2.19.1-64-bit.exe

 

Linux: Looks like git does not have an rpm for a 2.x, it’s a clone and compile, which makes long term sustainability a pain.

CVE says version 2.18 is the oldest affected version, and the newest available from the yum repos is 1.8.3.1-14.

https://git-scm.com/download/linux says to download and compile.

 

This works:

#!/bin/bash

# update git to 2.19.1

git –version

which git

cp /usr/bin/git /usr/bin/git-1.8.3.1

yum -y install curl-devel expat-devel gettext-devel openssl-devel zlib-devel

yum -y install gcc perl-ExtUtils-MakeMaker

cd /usr/src

wget https://www.kernel.org/pub/software/scm/git/git-2.19.1.tar.gz

tar xzf git-2.19.1.tar.gz

cd git-2.19.1

make prefix=/usr/local/git all

make prefix=/usr/local/git install

rm -f /bin/git

ln -s /usr/local/git/bin/git /bin/git

git –version

 

chef-client fails with ERROR: The used Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

TL;DR:

Your path is probably wrong for the root user that is running chef-client. We had a long screwed up path that eventually included the right path, but had an old ChefDK preceeding it. Basically, you’re using a broken chef-client (too  new, too old, broken encryption, whatever)

Works:

PATH=/usr/bin:/bin:/etc:.:/usr/local/bin:/usr/sbin:/opt/OV/bin/OpC:/home/rundeck/:/opt/middleware/Tools

Does not: 

PATH=/usr/local/rvm/gems/ruby-2.3.1/bin:/usr/local/rvm/gems/ruby-2.3.1@global/bin:/usr/local/rvm/rubies/ruby-2.3.1/bin:/usr/kerberos/sbin:/usr/kerberos/bin:/root:/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/usr/local/rvm/bin:/root/bin

Fix:

export PATH=/usr/bin:/bin:/etc:.:/usr/local/bin:/usr/sbin:/opt/OV/bin/OpC:/home/rundeck/:/opt/middleware/Tools

Permanent fix:

vi /etc/bash.bashrc

Change to:

type rvm >/dev/null 2>/dev/null || echo ${PATH} | __rvm_grep “/usr/local/rvm/bin” > /dev/null || export PATH=”${PATH}:/usr/local/rvm/bin”

vi /root/.bashrc

export PATH=/usr/bin:$PATH:/root/bin

But this hoses up ruby, better to get the right version of chef-client:

mv /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client-12.21.1

mv /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client-12.21.1

mv /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client-12.21.1

/usr/bin/chef-client –version

cd /usr/local/rvm/gems/ruby-2.3.1/bin/

ln /usr/bin/chef-client chef-client

chef-client –version

If you are running a specific version of chef-client, but running bare chef-client gives the wrong version, check “which chef-client” and rename the ones buried in the ruby paths:

mv /usr/local/rvm/gems/ruby-2.5.3/bin/chef-client /usr/local/rvm/gems/ruby-2.5.3/bin/chef-client.12.19.36

mv /usr/local/rvm/gems/ruby-2.4.2/bin/chef-client /usr/local/rvm/gems/ruby-2.4.2/bin/chef-client.14.9.13

etc.

logout and back in

chef-client -v 

should give you the proper version.

Frequent failure modes:

Needs newer version 12.21.31+:

[2019-03-18T14:50:54+00:00] ERROR: Cookbook ‘wdpr_dnsmasq’ version ‘0.0.1’ depends on chef version [“>= 12.21.31”], but the running chef version is 12.19.36

[2019-03-18T14:50:54+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

Needs older version until cook book is updated:

Indicative of running chef-client 14 against incompatible cookbooks:

Running handlers:

[2019-03-18T14:54:29+00:00] ERROR: Running exception handlers

Running handlers complete

[2019-03-18T14:54:29+00:00] ERROR: Exception handlers complete

Chef Client failed. 0 resources updated in 07 seconds

[2019-03-18T14:54:30+00:00] FATAL: Stacktrace dumped to /var/chef/cache/chef-stacktrace.out

[2019-03-18T14:54:30+00:00] FATAL: Please provide the contents of the stacktrace.out file if you file a bug report

[2019-03-18T14:54:30+00:00] ERROR: can’t modify frozen Array

[2019-03-18T14:54:30+00:00] FATAL: Chef::Exceptions::ChildConvergeError: Chef run process exited unsuccessfully (exit code 1)

590  [2018-10-10 15:27:33] echo ${PATH}

  591  [2018-10-10 15:29:27] chef-client

  592  [2018-10-10 15:30:07] which chef-client

  593  [2018-10-10 15:30:28] /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client –version

  594  [2018-10-10 15:30:40] mv /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client /usr/local/rvm/gems/ruby-2.3.1/bin/chef-client-12.21.1

  595  [2018-10-10 15:30:43] which chef-client

  596  [2018-10-10 15:30:55] /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client –version

  597  [2018-10-10 15:31:08] mv /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client /usr/local/rvm/gems/ruby-2.3.1@global/bin/chef-client-12.21.1

  598  [2018-10-10 15:31:10] which chef-client

  599  [2018-10-10 15:31:20] /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client –version

  600  [2018-10-10 15:31:55] mv /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client /usr/local/rvm/rubies/ruby-2.3.1/bin/chef-client-12.21.1

  601  [2018-10-10 15:31:57] which chef-client

  602  [2018-10-10 15:32:05] /usr/bin/chef-client –version

  603  [2018-10-10 15:32:15] chef-client

  604  [2018-10-10 15:32:24] which chef-client

  605  [2018-10-10 15:32:30] chef-client

  606  [2018-10-10 15:32:38] cd /usr/local/rvm/gems/ruby-2.3.1/bin/

  607  [2018-10-10 15:32:39] ll

  608  [2018-10-10 15:33:28] ln chef-client /usr/bin/chef-client

  609  [2018-10-10 15:33:38] ln /usr/bin/chef-client chef-client

  610  [2018-10-10 15:33:42] chef-client

  611  [2018-10-10 15:35:00] history

[rundeck][nl-fldi-02119][~]

$ chef-client –version

Chef: 12.19.36

Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

openssl enc -help 2>&1 | grep gcm

[bwilliam@nl-fldi-02119 ~]$ openssl enc -help 2>&1 | grep gcm

-aes-128-ctr               -aes-128-ecb               -aes-128-gcm

-aes-192-gcm               -aes-192-ofb               -aes-256-cbc

-aes-256-ecb               -aes-256-gcm               -aes-256-ofb

The used Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

knife data bag show −−secret-file=./rev_secret_key rev_secret revpass

knife data bag show −−secret-file=/etc/chef/encrypted_data_bag_secret users rundeck

[2018-10-10T09:38:11-04:00] ERROR: The used Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

Recipe Compile Error in /var/chef/cache/cookbooks/wdprt_rundeck_client/recipes/default.rb

  ================================================================================

  Chef::EncryptedDataBagItem::EncryptedDataBagRequirementsFailure

  —————————————————————

  The used Encrypted Data Bags version requires an OpenSSL version with “aes-256-gcm” algorithm support

[root@nl-fldi-02119 chef]# chef-client –version

Chef: 12.21.1

knife data bag show users rundeck

  cipher:         aes-256-gcm

which knife

which chef-client

[rundeck][nl-fldi-02119][~]

$ chef-client –version

Chef: 12.19.36

knife data bag show users rundeck

Recovering AWS instance that is prompting for password (happens when using SSH Key only logins)

recover cloud init password aws reset password instance remount cloud-init

Part of this is directly cribbed from AWS Docs!

Symptoms:

Password change required but no TTY available.

WARNING: Your password has expired

Logging in with SSH but getting prompted to change your password for cloud-user, and those are random to begin with, so can’t change it cuz you don’t know it

Problem:

cloud-user account password has expired. account expiry has nothing to do with ssh key validity, as shown by getting connected and then getting the change password prompt (you’re already in via SSH, but PAM kicks in to force the password change)

Fix:

Stop the instance (sorry)(do not terminate!)

Copy out user data

Replace user-data with this:

Content-Type: multipart/mixed; boundary=”//”

MIME-Version: 1.0

<span style="font-size: 14px; color: rgb(51, 51, 51); font-family: Consolas, "Andale Mono WT", "Andale Mono", "Lucida Console", "Lucida Sans Typewriter", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Liberation Mono", "Nimbus Mono L", Monaco, "Courier New", Courier, monospace; font-variant-caps: normal; font-variant-ligatures: normal; line-height: 1.6em;"–<//

Content-Type: text/cloud-config; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”cloud-config.txt”

#cloud-config

cloud_final_modules:

– [scripts-user, always]

<span style="font-size: 14px; color: rgb(51, 51, 51); font-family: Consolas, "Andale Mono WT", "Andale Mono", "Lucida Console", "Lucida Sans Typewriter", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Liberation Mono", "Nimbus Mono L", Monaco, "Courier New", Courier, monospace; font-variant-caps: normal; font-variant-ligatures: normal; line-height: 1.6em;"–<//

Content-Type: text/x-shellscript; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”userdata.txt”

#!/bin/bash

/usr/bin/chage -d 65535 cloud-user

<span style="font-size: 14px; color: rgb(51, 51, 51); font-family: Consolas, "Andale Mono WT", "Andale Mono", "Lucida Console", "Lucida Sans Typewriter", "DejaVu Sans Mono", "Bitstream Vera Sans Mono", "Liberation Mono", "Nimbus Mono L", Monaco, "Courier New", Courier, monospace; font-variant-caps: normal; font-variant-ligatures: normal; line-height: 1.6em;"–<//

start the instance again

WAIT AT LEAST ONE MINUTE!!

User data happens async from when the box starts up, so sit tight a minute or two.

SSH login should now work

Do you need to put your user data back?

You have to stop the instance to replace the user-data section, and it will fire again on startup. Decide if this is what you need, or leave it as is, or stop it and blank out user data.

If you replace your user-data to what it was before, add this section to prevent this from recurring:

# Fix the cloud-user password age issue

sed -i.bak -e ‘/Defaults.*requiretty/s/^/#/’ /etc/sudoers

chage -d 65535 cloud-user

Ok, so, scripted version:

You need:

my-user-data (below)

aws-replace-user-data.sh (below)

aws cli installed and working

aws-saml-auth installed and working

my-user-data:

#cloud-boothook

Content-Type: multipart/mixed; boundary=”//”

MIME-Version: 1.0

–//

Content-Type: text/cloud-config; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”cloud-config.txt”

#cloud-config

cloud_final_modules:

– [scripts-user, always]

–//

Content-Type: text/x-shellscript; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”userdata.txt”

#!/bin/bash -x

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

groupadd -g 25638 rundeck

#adduser -g rundeck rundeck

useradd -p $(openssl passwd -1 9837y45fhyiwurhef84yf93y4978yhfh) -g rundeck rundeck

echo “9837y45fhyiwurhef84yf93y4978yhfh” | passwd –stdin rundeck

usermod -c “Rundeck Service Account.” -u 35638 rundeck

usermod rundeck -G wheel

gpasswd -a rundeck wheel

getent passwd rundeck > /dev/null 2&>1

if [ $? -eq 0 ]; then

    chage -I -1 -m 0 -M 99999 -E -1 rundeck

else

    echo “Skipping, user does not exist”

fi

getent passwd cloudse > /dev/null 2&>1

if [ $? -eq 0 ]; then

    chage -I -1 -m 0 -M 99999 -E -1 cloudse

else

    echo “Skipping, user does not exist”

fi

getent passwd cloud-user > /dev/null 2&>1

if [ $? -eq 0 ]; then

    chage -I -1 -m 0 -M 99999 -E -1 cloud-user

else

    echo “Skipping, user does not exist”

fi

getent passwd ec2-user > /dev/null 2&>1

if [ $? -eq 0 ]; then

    chage -I -1 -m 0 -M 99999 -E -1 ec2-user

else

    echo “Skipping, user does not exist”

fi

# chage failures make the cloud-init fail, so use the logic blocks above

# chage -I -1 -m 0 -M 99999 -E -1 rundeck

# Fix rundeck sudo privs

rm -f /etc/sudoers.d/rundeck

cat > “/etc/sudoers.d/rundeck” << EOF

# This file is managed by Chef.

# Do NOT modify this file directly.

%rundeck ALL=(ALL) NOPASSWD:ALL

Defaults:%rundeck !requiretty

EOF

# Rundeck env setup

mkdir -p /home/rundeck/.ssh/

chown -R rundeck:rundeck /home/rundeck

rm -f /home/rundeck/.ssh/authorized_keys

# Add Rundeck key

cat > “/home/rundeck/.ssh/authorized_keys” << EOF

ssh-rsa AAAAB3NzaC1yc2EAAAABIwAAAQEAzxyMM1ozCoxZNe0q7PeiJdtqUQc6VKhAY46dmDET4Q+lvcmkDdE3q8IVCkrI8MES2j9YBoCy00BV3kAWRDTilq0CArDSVpTp5lz/2Fgu/EaxTMQKk2XiGGW4M4QUkAQRgHDNT1k8lYIhRENFS8Csf9Bt2lFOgWH18sw7s4GuCULbBfDZdsMVHN6wctv0j2vbvcPdg+QX2gg7TY4HdWoDQ3OSOrWSyAeseXA81h3+OZSKhyFmTIFzk0+8uxuv18CdilfyhCiDJqGwVV5WpbOTXlimT92ea6R5V1H8KeuhZEMnpuiCUjbQzGKuUsbKHu+bVWQqQMh9VS9VlC9Q0kqxFw==

EOF

chmod -R 600 /home/rundeck/.ssh/authorized_keys

chown -R rundeck:rundeck /home/rundeck/.ssh/authorized_keys

chmod -R 600  /var/spool/mail/rundeck

chown -R rundeck:rundeck  /var/spool/mail/rundeck

chmod 440 /etc/sudoers.d/rundeck

# Temporary

# echo rundeck:8y5g9eyhrgy3875ty98374hfjhwef | chpasswd

touch /root/cloud-init-fix-was-here

–//

aws-replace-user-data.sh:

#!/bin/bash

EXPECTED_ARGS=6

E_BADARGS=65

if [ $# -ne $EXPECTED_ARGS ]

then

        echo “All parameters are required, in any order:”

        echo “-i instance_id ( i-0b1fd14e321099e6d )”

        echo “-r region ( us-east-1, us-west-2, etc )”

        echo “-f filename ( aws-recovery-user-data in current path )”

  exit $E_BADARGS

fi

while getopts “:i:r:f:” opt; do

  case $opt in

    i)

      echo “-i was triggered, Parameter: $OPTARG” >&2;

        INSTANCE=$OPTARG;

      ;;

    r)

      echo “-r was triggered, Parameter: $OPTARG” >&2;

        REGION=$OPTARG;

      ;;

    f)

      echo “-f was triggered, Parameter: $OPTARG” >&2;

        FILENAME=$OPTARG;

      ;;

    *)

      echo “Invalid option: -$OPTARG” >&2

        echo Example: ./aws-replace-user-data.sh -i i-0b1fd14e321099e6d -r us-west-2

      exit 1

        break

      ;;

    🙂

      echo “Option -$OPTARG requires an argument.” >&2

        echo Example: ./aws-replace-user-data.sh -i i-0b1fd14e321099e6d -r us-west-2 -f my-user-data

      exit 1

        break

      ;;

  esac

done

# You can preset or override some things if its your machine

echo Using instance ID: ${INSTANCE}

echo Using region: ${REGION}

echo Using filename: ${FILENAME}

# aws-saml-auth

# pip install –upgrade –user awscli

# Stop

aws ec2 –region ${REGION} stop-instances –instance-ids ${INSTANCE}

# Better!

while [ true ]

    do  aws ec2 describe-instances –filter “Name=instance-state-name,Values=stopped” –region ${REGION} –instance-id ${INSTANCE} |grep stopped && break

    echo Not stopped yet

    sleep 5

    echo retrying…

done

# Base64 encode your file:

echo “Encoding ” ${FILENAME}

rm -f ${FILENAME}.base64

base64 ${FILENAME} >> ${FILENAME}.base64

# Modify!

echo “Modifying instance: “

aws ec2 modify-instance-attribute –region ${REGION} –attribute userData –value file://${FILENAME}.base64 –instance-id ${INSTANCE}

# Start

echo “Starting instance: “

aws ec2 –region ${REGION} start-instances –instance-ids ${INSTANCE}

# Optional

while [ true ]

    do  aws ec2 describe-instances –filter “Name=instance-state-name,Values=running” –region ${REGION} –instance-id ${INSTANCE} |grep running && break

    echo Not started yet

    sleep 5

    echo retrying…

done

Usage:

./aws-replace-user-data.sh -i i-050bea589a55ac039 -r us-west-2 -f my-user-data

punch it in the face backdoor:

Content-Type: multipart/mixed; boundary=”//”

MIME-Version: 1.0

–//

Content-Type: text/cloud-config; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”cloud-config.txt”

#cloud-config

cloud_final_modules:

– [scripts-user, always]

–//

Content-Type: text/x-shellscript; charset=”us-ascii”

MIME-Version: 1.0

Content-Transfer-Encoding: 7bit

Content-Disposition: attachment; filename=”userdata.txt”

#!/bin/bash

PATH=/usr/local/sbin:/usr/local/bin:/sbin:/bin:/usr/sbin:/usr/bin:/root/bin

useradd -p $(openssl passwd -1 9837y45fhyiwurhef84yf93y4978yhfh) backdoor

echo “9837y45fhyiwurhef84yf93y4978yhfh” | passwd –stdin backdoor

usermod -c “Recovery account, please remove.”  backdoor

usermod backdoor -G wheel

gpasswd -a backdoor wheel

<span style="font-size: 14px;"–<//

Bulk convert mobi ebooks to audiobooks (robotic)

It works, the quality, well, not so much. Works good on the mac, Linux is still spotty.

Install calibre:

apt-get update

apt install calibre

# Convert the mobi to txt files

for file in *.mobi; do ebook-convert “$file” “${file%.mobi}.txt” ; done

# Convert txt to audio

apt-get install espeak lame ruby

wget https://gist.githubusercontent.com/sentientwaffle/2186807/raw/69d7f4e974b9e553ee7fd7de1ac3a9bee3095cce/speak.rb

# Works if no spaces in names

for file in *.txt; do ruby speak.rb  “$file”  ; done 

for file in *.txt; do espeak -v en-us -f ${file} -s 130 -w “${file%.txt}.wav” –split=30  ; done

eSpeak will take a few minutes to convert your content, just sit tight, it will peg one CPU, it is not multithreaded.

Quality is still not great.

Ok, so wrap it up in one nice script, this works 100% but the voices in espeak are crap:

#!/bin/bash

# Save default separator definitions

oIFS=$IFS

# define new line as a separator, filenames can have spaces

IFS=$’\n’;

# For each file (f) listed as duplicated by fdupes, recursively

  for f in `ls *.mobi`

    do

      echo “Using filename: ” ${f}

      ebook-convert “$f” “${f%.mobi}.txt”

      espeak -v en-us -f “${f%.mobi}.txt” -s 130 -w “${f%.mobi}.wav” –split=30

done

  for f in `ls *.wav`

    do

      echo “Using filename: ” ${f}

       lame -V2 ${f} ${f%.wav}.mp3

       rm -f ${f}

done

# restore default separator definitions

IFS=$oIFS

Can do this on a mac using the built in Alex voice:

brew install caskroom/cask/calibre

brew install lame

brew install mp3splt

convert-ebook-to-audiobook-mp3-mac.sh

#!/bin/bash

# brew install caskroom/cask/calibre

# brew install lame

# brew install mp3splt

echo “Using filename: ” “${1%.mobi}”

# Convert ebook to text

ebook-convert “$1” “${1%.mobi}”.txt –enable-heuristics –html-unwrap-factor 0.1

# Edit out any weirdness

# Will probably need to have a list of standard/recurring screwups to replace too like dr. etc.

vi “${1%.mobi}”.txt

# Convert txt to speech

# Available Apple voices (US English): Alex Fred Samantha Victoria

# say -r 180 –voice Alex -f “${1%.mobi}”.txt -o “${1%.mobi}”.aiff

# say -r 180 –voice Fred -f “${1%.mobi}”.txt -o “${1%.mobi}”.aiff

say -r 180 –voice Samantha -f “${1%.mobi}”.txt -o “${1%.mobi}”.aiff

# say -r 180 –voice Victoria -f “${1%.mobi}”.txt -o “${1%.mobi}”.aiff

# Convert aiff to mp3

lame -V2 “${1%.mobi}”.aiff “${1%.mobi}”.mp3

# Clean up the aiff giant file

rm -f “${1%.mobi}”.aiff

# Split the giant mp3 into smaller pieces and dump into a directory

mp3splt -a -t 15.1 -o “${1%.mobi}”-@n -d “${1%.mobi}” “${1%.mobi}”.mp3

Splitting big mp3’s into smaller pieces:

mp3splt -a -t 15.1 -o Saint\ Death\ -\ Mark\ Dawson-@n -d Saint\ Death\ -\ Mark\ Dawson Saint\ Death\ -\ Mark\ Dawson.mp3

mp3splt -a -t 5.10 -o Lesson_01-@n -d Lesson_01 Lesson_01.mp3

The -a tells mp3splt to auto-adjust the split point with silence detection.

The -t 5.10 tells it to make the files 5 minutes and 10 seconds long since the file is a little over 30 minutes long. (This length may vary a bit due to the -a option).

The -o Lesson_01-@n tells it to name the files as Lesson_01 followed by a track number.

The -d Lesson_01 tells it to put the files in a directory called Lesson_01.

last is input file.mp3

So: mp3splt -a -t 5.10 -o outputfilename-@n -d OutputDirectory InputFilename.mp3

Sample example AWS cloudwatch time formats

The common datetime_format codes are listed below. You can also use

any datetime_format codes supported by Python, datetime.strptime(). The

timezone offset (%z) is also supported even though it’s not supported

until python 3.2, [+-]HHMM without colon(:). For more information, see

strftime() and strptime() Behavior.

%y: Year without century as a zero-padded decimal                                 number. 00, 01, …, 99                                                                                       

%Y: Year with century as a decimal number.1970,                                 1988, 2001, 2013                                                                                      

%b: Month as locale’s abbreviated name. Jan, Feb,                                 …, Dec (en_US);                                                                                       

%B: Month as locale’s full name. January,                                 February, …, December (en_US);                                                                                       

%m: Month as a zero-padded decimal number. 01,                                 02, …, 12                                                                                     

%d: Day of the month as a zero-padded decimal                                 number. 01, 02, …, 31                                                                                      

%H: Hour (24-hour clock) as a zero-padded decimal                                 number. 00, 01, …, 23                                                                                   

%I: Hour (12-hour clock) as a zero-padded decimal                                 number. 01, 02, …, 12                                                                                     

%p: Locale’s equivalent of either AM or                                 PM.                                                                                     

%M: Minute as a zero-padded decimal number. 00,                                 01, …, 59                                                                                       

%S: Second as a zero-padded decimal number. 00,                                 01, …, 59                                                                                        

%f: Microsecond as a decimal number, zero-padded                                 on the left. 000000, …, 999999                                                                                       

%z: UTC offset in the form +HHMM or -HHMM. +0000,                                 -0400, +1030                  

%s: Epoch time in seconds                                                                            1333234800            

                                                                                          

Example formats:                                                         

04 Oct 2017 23:47:09,795  – ‘%d %b %Y %H:%M:%S,%f’

10.180.39.110 – – [29/Sep/2017:00:00:03 +0000] – ‘%d/%b/%Y:%H:%M:%S %Z’

2017-09-25 20:51:21.595 – ‘%Y-%m-%d %H:%M:%S.%f’

29 Sep 2017 00:41:19 – ‘%d %b %Y %H:%M:%S’

Sep 24, 2017 00:41:19 AM – ‘%b %d, %Y %I:%M:%S %p’

ISO8601: ‘%Y-%m-%dT%H:%M:%S%z’, e.g.  2014-02-20T05:20:20+0000                                                                                                                        

Log4j: ‘%d %b %Y %H:%M:%S’, e.g. 24 Jan 2014  05:00:00                                                            

Syslog: ‘%b %d %H:%M:%S’, e.g. Jan 23 20:59:29                                                            

Typical Ambari/log4j: ‘%Y-%m-%d %H:%M:%S,%f’, e.g. 2017-09-24 05:48:25,218

18/04/17 18:58:21 INFO ShutdownHookManager: Shutdown hook called

%y/%m/%d %H:%M:%S 

Tagging versions of Chef recipes in git after-the-fact

Tagging versions of Chef recipes in git after-the-fact

Chef metadata.rb versions and Git Version tags are NOT CONNECTED AT ALL

#!/bin/bash

# Auto tag script

# Make sure you tag master and have the current tree

pwd

git checkout master

git pull

rm -f temp.txt

rm -f temp2.txt

# Get the versions and metadata.rb commits, dump to file

git log -p — metadata.rb  |grep -E ‘(commit|\+version)’ > temp.txt

# Need to whack sequential commit commit, and keep only the first one:

# see https://stackoverflow.com/questions/45478938/delete-duplicate-lines-through-pattern-in-bash

awk ‘$1==”commit” && p==”commit”{next} {print; p=$1}’ temp.txt > temp2.txt

# dump the file into a usable format:

# see: https://stackoverflow.com/questions/9999934/sed-joining-lines-depending-on-the-second-one

cat temp2.txt | sed ‘$!N;s/\n\s*+//;P;D’

cat temp2.txt | sed ‘$!N;s/\n\s*+//;P;D’ |sed -e ‘s/version//g’  |sed -e ‘s/commit//g’ |tr -d “” |tr “‘” “,” | awk -F’,’ ‘{print “git tag -a v” $2 ” ” $1 ” -m \”Automated tagging\”” }’ > autotag.sh

chmod +x autotag.sh

./autotag.sh

git push –tags origin master

rm -f temp.txt

rm -f temp2.txt

rm -f autotag.sh

Checking out a tag as a new branch (you can’t checkout a tag)

git checkout master

git pull

git checkout -b do_not_merge v2.4.0

How to do it by hand:

Pull down and sync a master copy of your repo locally, cd into that directory:

dev

cd wdprt_sece_zdd

git pull

git checkout master

git pull

git log -p — metadata.rb  |grep -E ‘(commit|\+version)’

CLI:

git log -p — metadata.rb  |grep -E ‘(commit|\+version)’

Gives:

commit d7eb053b80ff04c58eda52b89abeef53cf15736c

+version          ‘2.1.0’

commit 1deb63acb26b3e2028d2b44237560f960ee0baed

+version          ‘2.0.2’

commit b7011d2890e972c6d9ac1920e148f461d94e7425

+version          ‘2.0.1’

commit c710d94ac80667c1706c95bee3c001a29c7c7e8a

commit 0366b2d1be83c677df1ed8eb3cb2c4c4d700aac4

+version          ‘1.0.1’

commit a81a4222d60ec27ef559a1043d91bdfd6e1754c9

    initial commit

+version          ‘1.0.0’

Note the splits! Top commit goes with the first version listed, regardless of interim commits!

GUI:

on the Web UI, find the metadata.rb for your app and select it since this dictates the version to chef, we’ll match up the git tags to it

once in metadata.rb, click the history button for it

from the list of changes to metadata.rb, click the clipboard thing, then <>

use the SHA that it put on the clipboard and the version in the metadata.rb file that you’re looking at

ie:

69436410dae0f256214fa2f27f4bb96e7e83f28a  and  1.1.1

Both:

from your directory of that repo, construct commands like this (you can use the whole SHA or the first 6-ish chars)

If you have multiple metadata.rb versions, only tag the latest (highest in the list) one because someone committed a version without a version bump.

git tag -a v1.1.0 41fb370 -m “Tagging wdprt_sece_mq v1.1.0”

git tag -a v1.1.1 6943641 -m “Tagging wdprt_sece_mq v1.1.1”

git tag -a v1.0.4 98d39c1 -m “Tagging wdprt_sece_mq v1.0.4”

git tag -a v1.0.3 fe5fa88 -m “Tagging wdprt_sece_mq v1.0.3”

git tag -a v1.0.2 9d7f1f9 -m “Tagging wdprt_sece_mq v1.0.2”

git tag -a v1.0.1 0f071d9 -m “Tagging wdprt_sece_mq v1.0.1”

git tag -a v1.0.0 cf56449 -m “Tagging wdprt_sece_mq v1.0.0”

git tag -a v1.1.2 26f75a2 -m “Tagging wdprt_sece_mq v1.1.2”

git push –tags origin master

And now berks should behave since it can find a tag

How to delete tags if you goof:

WW-AM04041696:wdprt_tomcat bwilliam$ git push –delete origin v2.5.1

To https://github.disney.com/WDPRT-chef/wdprt_tomcat.git

– [deleted]         v2.5.1

WW-AM04041696:wdprt_tomcat bwilliam$ git tag -d v2.5.1

Deleted tag ‘v2.5.1’ (was dccffed)

WW-AM04041696:wdprt_tomcat bwilliam$ git tag -d delete

Deleted tag ‘delete’ (was dccffed)

# Then re-push the right tag

WW-AM04041696:wdprt_tomcat bwilliam$ git tag -a v2.5.1 2614f4baf955aeac142a94070a15bf48a8ba6e16 -m “Tagging v2.5.1”

WW-AM04041696:wdprt_tomcat bwilliam$ git push –tags origin master

Automatic tagging (WIP)

We get this:

commit 74fb42b61975ebc560dabbb0dd380f763a1a0d15

+version          ‘2.0.3’

commit a721d6f882e11f7889be0dcb4c9daa9d2b40b2ba

commit 04a05cc3bd5990aa7c6587ced88afd5f1f4d432d

+version          ‘2.0.2’

commit 1f3dc2742613b396291c054116239b67398f7c33

+version          ‘2.0.1’

commit 4a88517235d66ad0f623b67e5ac4a500aada2f27

+version          ‘2.0.0’

commit 26f75a269212d2743d98abd7314771056b1c6200

+version          ‘1.1.2’

commit 69436410dae0f256214fa2f27f4bb96e7e83f28a

+version          ‘1.1.1’

commit 41fb3704d9b78546acb5025b3d49f7af18f6eba0

+version          ‘1.1.0’

commit 98d39c11f80e499288cbb9bf3a641379ddd68257

+version          ‘1.0.4’

commit fe5fa88dc50b1459eda0cc7a28c13fda5afb26f0

+version          ‘1.0.3’

commit 9d7f1f91a820f3ec5c3102ea4acb837a5cc791ca

+version          ‘1.0.2’

commit 0f071d981b8b491e4f52b7b1d8d0a8b59daa18bf

+version          ‘1.0.1’

commit cf5644964a8830d178bbb9538049db141e64da88

commit d07c46b3b7fdb5bb50e38b08717247f9a8e979c4

commit f680e39ecc46ab643af3ab8edaa8e30ca6d22c24

commit d253d62fa3a72c8b40713c8da76d82b4293155c3

commit 36b4fd8e0ec33040d3236ce8ad8a010d6ad71be2

+version          ‘1.0.0’

so:

# Make sure you tag master and have the current tree

git checkout master

git pull

rm -f temp.txt

rm -f temp2.txt

# Get the versions and metadata.rb commits, dump to file

git log -p — metadata.rb  |grep -E ‘(commit|\+version)’ > temp.txt

#check the file, can be weird

# vi temp.txt

# Need to whack sequential commit commit, and keep only the first one:

# see https://stackoverflow.com/questions/45478938/delete-duplicate-lines-through-pattern-in-bash

awk ‘$1==”commit” && p==”commit”{next} {print; p=$1}’ temp.txt > temp2.txt

# dump the file into a usable format:

# see: https://stackoverflow.com/questions/9999934/sed-joining-lines-depending-on-the-second-one

cat temp2.txt | sed ‘$!N;s/\n\s*+//;P;D’

# gives: 

commit 74fb42b61975ebc560dabbb0dd380f763a1a0d15version          ‘2.0.3’

commit a721d6f882e11f7889be0dcb4c9daa9d2b40b2baversion          ‘2.0.2’

commit 1f3dc2742613b396291c054116239b67398f7c33version          ‘2.0.1’

#Beauty! clean it up:

# need it in this format:

# git tag -a v1.1.0 41fb370 -m “Tagging wdprt_sece_mq v1.1.0”

# cat temp2.txt | sed ‘$!N;s/\n\s*+//;P;D’ |sed -e ‘s/version//g’  |sed -e ‘s/commit//g’ |tr -d ” ” |tr “‘” “,”

# 74fb42b61975ebc560dabbb0dd380f763a1a0d15,2.0.3,

# a721d6f882e11f7889be0dcb4c9daa9d2b40b2ba,2.0.2,

# Make the script

# cat temp2.txt | sed ‘$!N;s/\n\s*+//;P;D’ |sed -e ‘s/version/ -a v/g’  |sed -e ‘s/commit/ git tag /g’  |sed -e ‘s/     / /g’ |tr -d “‘”|sed -e ‘s/v  /v/g’

# Makes it wrong, but interesting:

# git tag  74fb42b61975ebc560dabbb0dd380f763a1a0d15 -a v2.0.3

# git tag  a721d6f882e11f7889be0dcb4c9daa9d2b40b2ba -a v2.0.2

cat temp2.txt | sed ‘$!N;s/\n\s*+//;P;D’ |sed -e ‘s/version//g’  |sed -e ‘s/commit//g’ |tr -d ” ” |tr “‘” “,” | awk -F’,’ ‘{print “git tag -a ” $2 ” ” $1 ” -m \”Automated tagging\”” }’

# Makes:

git tag -a 2.0.2 a721d6f882e11f7889be0dcb4c9daa9d2b40b2ba -m “Automated tagging”

git tag -a 2.0.1 1f3dc2742613b396291c054116239b67398f7c33 -m “Automated tagging”

git tag -a 2.0.0 4a88517235d66ad0f623b67e5ac4a500aada2f27 -m “Automated tagging”

# GTG!!!

git push –tags origin master