Regression testing releases with Depicted (Dpxdt), Travis, & Saucelabs

Depicted is a release testing tool that compares before and after screenshots of your webpage, highlighting differences between the two.

Depicted supplements your release testing by allowing you to approve any visual changes a new release may cause.

I wrote a script during my time at Sprintly that would take a TravisCI build ID, pull related screenshots from our Saucelabs selenium tests, and upload them to a Depicted API server for comparison.

Before a new release would be deployed, we would manually run our Depicted release script and check and approve any changes.

This script was integrated as a Django management command for ease of use. Check out the full script below with comments.

Django, Redis & AWS ElastiCache primary/replica cluster

AWS’s ElastiCache service is a convenient way to launch a Redis cluster. If you’re using Django, both django-redis-cache and django-redis packages support an ElastiCache Redis instance. If you are launching ElastiCache Redis with any amount of replicas, some additional master-slave configuration is needed in your Django settings.

Here is an example of an ElastiCache Redis cluster with a primary instance and two replicas:

The following is an example of the correct settings for this cluster if you’re using django-redis-cache backend:

CACHES = {
    'default': {
        'BACKEND': 'redis_cache.RedisCache',
        'LOCATION': [
            "test-001.730tfw.0001.use1.cache.amazonaws.com:6379",
            "test-002.730tfw.0001.use1.cache.amazonaws.com:6379",
            "test-003.730tfw.0001.use1.cache.amazonaws.com:6379"
        ],
        'OPTIONS': {
            'DB': 0,
            'MASTER_CACHE': "test-001.730tfw.0001.use1.cache.amazonaws.com:6379"
        },
    }
}

https://django-redis-cache.readthedocs.io/en/latest/advanced_configuration.html#master-slave-setup

Apache Kafka plaintext authentication and kafka-python configuration reference

Apache Kafka config settings and kafka-python arguments for setting up plaintext authentication on Kafka.

You’ll need to follow these instructions for creating the authentication details file and Java options.

I exposed the auth endpoint to port 9095. All other ports were closed via AWS security groups.

Kafka config settings:

security.inter.broker.protocol=PLAINTEXT
sasl.mechanism.inter.broker.protocol=PLAIN
sasl.enabled.mechanisms=PLAIN
advertised.listeners=SASL_PLAINTEXT://example.com:9095,PLAINTEXT://example.com:9092
listeners = SASL_PLAINTEXT://0.0.0.0:9095,PLAINTEXT://0.0.0.0:9092

kafka-python client command for connecting:

from kafka import KafkaProducer
producer = KafkaProducer(bootstrap_servers='example.com:9095', security_protocol="SASL_PLAINTEXT", sasl_mechanism='PLAIN', sasl_plain_username='username', sasl_plain_password='password')

GitHub Desktop & GPG issues “gpg failed to sign the data”

I had some issues while trying to get GPG signing working while using GitHub Desktop. While their docs say the application doesn’t support GPG, a bunch of users seemed to have it working.

I ran into a few errors and got it working correctly. A “gpg failed to sign the data” error is what took a while to find a fix for.

Assuming you followed all the instructions in GitHub’s docs, also make sure your global git settings are pointing to the gpg command and signing is set to true:

user.signingkey=EEDDA4EE375C6D12
gpg.program=/usr/local/bin/gpg
commit.gpgsign=true

And what ultimately fixed my issue was disabling GPG terminal output via:

echo "no-tty" >> ~/.gnupg/gpg.conf

Firefox & python Selenium: Stopping auto update on browser test runs

Found a minor annoyance when running headless selenium browser tests on Ubuntu server 16. For some reason automated tests would start failing when opening Firefox. Apparently the configuration i’m running allows for Firefox to run auto update when opened.

To stop Firefox auto updates during your python Selenium test run, load a custom profile:

from selenium import webdriver

profile = webdriver.FirefoxProfile()
profile.set_preference('app.update.auto', False)
profile.set_preference('app.update.enabled', False)

driver = webdriver.Firefox(profile)

If this doesn’t seem to do the trick, verify that apt unattended-upgrades are not causing this behavior. In one case, I saw that the update was happening in the /var/log/unattended-upgrades/unattended-upgrades-dpkg.log log file.

I disabled auto updates via apt globally with the command:

dpkg-reconfigure -plow unattended-upgrades

npm install causing “No space left on device” Pivotal Cloud Foundry

Had an issue while deploying to a 2GB Pivotal Cloud Foundry instance during npm install buildpack step. My instance was limited to 2GB disk space, but somehow my 100mb application was filling up the drive.

Error was along the lines of:

cp: cannot create directory ‘/tmp/contents258369199/...’: No space left on device

Turns out npm’s package cache is on by default, and was saving all installed packages to a cache directory on disk. So if you’re limited on disk space during a build, turn this off with the environment variable:

NODE_MODULES_CACHE=false

Elastic Beanstalk npm install error EACCES: permission denied, mkdir ‘/home/webapp’

Ran into an error while deploying to an AWS Elastic Beanstalk instance set up for Rails + Node + Webpack.

Full error during npm install:

npm install
npm ERR! Linux 4.4.23-31.54.amzn1.x86_64
npm ERR! argv "/opt/elasticbeanstalk/support/node-install/node-v4.6.0-linux-x64/bin/node" "/usr/bin/npm" "install"
npm ERR! node v4.6.0
npm ERR! npm  v2.15.9
npm ERR! path /home/webapp
npm ERR! code EACCES
npm ERR! errno -13
npm ERR! syscall mkdir

npm ERR! Error: EACCES: permission denied, mkdir '/home/webapp'
npm ERR!     at Error (native)
npm ERR!  { [Error: EACCES: permission denied, mkdir '/home/webapp']
npm ERR!   errno: -13,
npm ERR!   code: 'EACCES',
npm ERR!   syscall: 'mkdir',
npm ERR!   path: '/home/webapp',
npm ERR!   parent: 'testing-app' }
npm ERR!
npm ERR! Please try running this command again as root/Administrator.

After some Googling, this Stack Overflow thread gave me some clues: Deploying a Rails / Ember app on AWS Elastic Beanstalk

The /home/webapp directory needed to be created during deploy. This was accomplished using a .ebextensions config with a commands stanza. Docs on commands here

The final commands I used are as follows:

commands:
  01_mkdir_webapp_dir:
    command: mkdir /home/webapp
    ignoreErrors: true
  02_chown_webapp_dir:
    command: chown webapp:webapp /home/webapp
    ignoreErrors: true
  03_chmod_webapp_dir:
    command: chmod 700 /home/webapp
    ignoreErrors: true

Ubuntu server 12.0.4 SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

Ran into some trouble with Ubuntu 12.0.4 and an “SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure” error while accessing the Zoho Docs API through Python. Apparently Zoho’s API endpoint enforces a certain SSL version. The full python error was:

SSLError: [Errno 1] _ssl.c:504: error:14077410:SSL routines:SSL23_GET_SERVER_HELLO:sslv3 alert handshake failure

After ruling out python and openssl, the fix was to simply update to Ubuntu 12.0.4.1 through apt:

apt-get update
apt-get upgrade

And for the hell of it did a distribution upgrade:

apt-get dist-upgrade

django-simple-redis-admin: Manage redis within the django admin

It’s been a while since I talked about anything i’ve been coding. A few months ago I released Django Simple Redis Admin, which allows you to manage redis cache keys within the Django admin. Something that annoys me with other django admin packages is the requirement to add database tables for their packages. django-simple-redis-admin gets around this by making a django model on the fly. No need for tables, just plug and play. Hopefully in the future i’ll be adding some filters and speeding things up. Check out django-simple-redis-admin on my GitHub page.

python-cloudcracker: a class to interface with the CloudCracker API

So I released a quick python class to interface with the CloudCracker API. CloudCracker is a new online password cracking service. Upload a pcap file, pay via BitCoin or Stripe, and they will crack the password for you. Pretty cool. Check out CloudCracker at cloudcracker.com and check out python-cloudcracker on GitHub.