Recently I found myself needing to setup Elasticsearch for a project and thought it’d be the perfect opportunity to utilize Docker. This article discusses a portion of how I setup a set of three VMs to quickly get Elasticsearch up and running with minimal fuss. Hopefully you find it useful and can adapt the information provided here into your own setup.

I chose to use Docker for this project for a few reasons. For one, it allowed me to avoid iterating over and over to create a puppet module/configuration when the work of installing Elasticsearch as already done for me and nothing more than a command away. It allowed me to concentrate on configuration Elasticsearch itself rather than how to express how it should be configured in a different environment. Another reason I went with Docker is that you end up naturally documenting how the service runs on that system because the configuration is actually part of the command line arguments. It’s visible in both the script I use to create the container as well as if you issue docker inspect. Everything you need to know is more centrally located rather than poking around at system files and possibly missing something on one system. In the end, I have a setup that easier to understand, easier to debug and ultimately more portable.

For my setup I started with three Ubuntu 16.04 VMs configured using puppet to have the latest version of Docker installed and ready to go. These machines were configured with 16GB of memory and were each given an IP address (192.168.0.51, 192.168.0.52 and 192.168.0.53). From there I created a simple script called start_elasticsearch.sh to create the container on each server:

#!/bin/bash

docker rm -f elasticsearch
docker run --restart always \
 --ulimit memlock=-1:-1 \
 --ulimit nofile=65536:65536 \
 --cap-add=IPC_LOCK \
 -d \
 -e ES_JAVA_OPTS="-Xms7g -Xmx7g" \
 -v /var/lib/elasticsearch/:/usr/share/elasticsearch/data \
 -v /etc/timezone:/etc/timezone -v /etc/localtime:/etc/localtime \
 -p 9300:9300 -p 9200:9200 \
 --name elasticsearch \
 elasticsearch:5.2 \
 -Ediscovery.zen.ping.unicast.hosts=192.168.0.51,192.168.0.52,192.168.0.53 \
 -Ebootstrap.memory_lock=true \
 -Enetwork.host=0.0.0.0 \
 -Enode.name=${HOSTNAME} \
 -Enetwork.publish_host=`ip addr | grep inet | grep ens160 | awk '{print $2}' | cut -d '/' -f 1` \
 -Ediscovery.zen.minimum_master_nodes=2

Be sure to change the IP addresses listed in the -Ediscovery.zen.ping.unicast.hosts line as well as the interface name in -Enetwork.publish_host line (grep ens160) to match the values of your systems. You can adjust other values as well to match your setup such as ES_JAVA_OPT to set heap size (use about 50% of total system memory). Save the script out and the mark it executable with chmod +x start_elasticsearch.sh. Tuck it away wherever you want, you only need to run it again if these parameters are updated.

Next, create a directory at /var/lib/elasticsearch. This is where the actual files will be stored. Repeat these steps on each server in your cluster. You can now run the script which will download the Elasticsearch version specified and start it with the parameters you provided. Once all instances are started they’ll discover each other and will just work.

From here you can access any of the servers at its IP address on port 9200. For production use, it’s wise to put some kind of load balancer in front of the cluster. I used nginx to get the job done.

In a future post, I’ll detail how to achieve this same setup using docker-compose and possibly docker swarm.

I’ve been putting a lot of time into this little project. Nobody uses it (yet?) and truth be told I barely use it in the house but it’s been such a great way to learn a number of different things including python, mDNS (bonjour), creating installer files for debian and OS X systems and even git that I can’t stop working on it.

I’m now releasing version 0.3.0. This version brings a few changes but most notably the Linux client is now ready. The next release will be coming shortly and will focus on making the client the more robust about how it deals with network disconnects.

You can read more about the 0.3.0 release at https://github.com/dustinrue/Dencoder/wiki

I just spent that last couple of hours trying to figure out why I couldn’t create a new software RAID set on my Ubuntu 10.04 system. Long story short, it turned out to be device mapper grabbing hold of the drives at boot. No amount of lsof would show that the devices were busy. The key was running dmsetup table and seeing that the drives in question were indeed “locked” by the device mapper.

This thread was the key I needed to get it all figured out – http://www.mail-archive.com/linux-raid@vger.kernel.org/msg10661.html

After issuing dmsetup remove followed by the device name shown in dmsetup table I was off and running.

Well not quite. Turns out people have been getting confused on the pricing grid Oracle has on their site for the various products they provide. The confusion comes from the Embedded version of MySQL not supporting InnoDB and that the community edition isn’t listed as part of the grid.

The community edition still has InnoDB built in as an available storage engine but you can’t buy support from Oracle.

http://www.mysql.com/products/
http://palominodb.com/blog/2010/11/04/oracle-not-removing-innodb

I’ve been trying for a while now to find a reasonably priced way to play video files that live on my computers on our TV or projector with acceptable quality while still being easy to use. Finding a solution has proven more difficult than I initially thought. I could never really find anything that gave me everything I wanted without some major compromise and didn’t cost a lot in terms of price or time spent configuring. I simply want to be able to take a file from the computer and play it on my TV or projector on a device that feels integrated with the rest of my setup.

Enter Western Digital’s WD TV Live. The WD TV Live does exactly what I want and does it in a tiny and silent, remote controlled package. It is able to stream virtually every common video or audio format in use today from a network share or from a locally attached USB drive. As an added bonus, it can also access YouTube, Pandora, Flickr as well as a couple of other online services. You can read more about the WD TV Live at http://www.wdtvlive.com/products/wdtv_live#highlights.

There are some isues with the device right now that are supposed to be cleared up in a soon to be released firmware update. It will play DVDs ripped to .iso format but doesn’t support menus, or even chapters, at all. This makes it nearly impossible to watch a TV series on DVD. The other issue the player currently has is with h264 files but oddly enough it is all in the name. A file with the extension .m4v will most likely have audio sync issues. Renaming the file to .mp4 will resolve the audio sync issue but will do nothing with the stuttering playback for all but the most basic of h264 encoded files. Incredibly, taking the same h264 file and putting it into mkv format will fix all of the issues and the file will play perfectly.

Despite these very easily fixed issues, this is the best device I’ve come across for playing videos at this price point. Yes, there are probably more capable devices available and there is certainly more capable software available (XBMC for one) but for the money, it’s hard to beat the WD TV Live. My next few blog posts will focus on how I’ve integrated the WDTV into my home network including how I’m encoding files for it using HandBrake, Linux and Automator in OS X.

My wife and I don’t watch a ton of TV but we do have a few TV series on DVD. Lately I’ve been taking the time to rip the DVDs to hard drive so we can travel with them more easily.

The problem of course is they take up a lot of space after awhile so I’ve started to compress some of them to the h264 format. I picked h264 because I like the quality and because newer Macs are able to decode h264 using hardware acceleration. One problem with the h264 format is that by itself it has no support chapter markers unlike say, the mkv container. To get around this, I wrote a couple of scripts. One of them determines how many chapters (episodes) are available on a disc and the other one does the encoding.

In order to put all of this to use you’ll need to have the HandBrakeCLI program installed on your computer. The HandBrake is site (http://handbrake.fr/) has information about how to get everything installed for your OS. This post assumes you are using Linux, Mac OS X or other UNIX like system.

Here is the script that retrieves the number of chapters on from a DVD that has been ripped to a hard drive. You’ll need to modify it for your environment. I called mine getchapters.sh but the name is arbitrary.

#!/bin/bash
HandBrakeCLI -i $1 -t 0 2>&1| grep "title 1 has" | grep chapters | awk '{print $
6}'

This script is called within another script that I called h264encode_episodic and looks like this

#!/bin/bash
chapters=`getchapters.sh $1`
if [ "$2" != "" ]; then
title=$2
else
title=$1
fi

for I in `seq 1 $chapters`
do
HandBrakeCLI -Z "High Profile" -i $1 -o ${title}E${I}.mp4 -c $I
done

Running this script in the same directory as the DVD iso or VIDEO_TS directory you will end up individual video files, one for each chapter or episode of your show. Here’s how it is called

h264encode_episodic Scrubs_S1D2.iso Scrubs_S1D2

Scrubs_S1D2.iso is the iso of the DVD I want to convert and Scrubs_S1D2 is part of how the resulting files will be named. The first episode will be named Scrubs_S1D2E1.mp4.