Scraping an Arris cable modem status page

Screenshot of the modem status page with arrows pointing to a screenshot of the data's final form in Grafana.
This project’s purpose is to start with a status page and end with Grafana graphs and alerts.

It felt good to complete this project that’s been on my list for quite some time. The main goal was to scrape the values from my modem’s status page and pipe them into InfluxDB, which feeds Grafana. Not only could I look at data trends, but I could receive alerts if certain values exceeded an acceptable threshold.

Overall this is a straightforward process:

  1. Pull in the HTML from the status page (which happens to not need any authentication, making it even easier)
  2. Parse the tables we care about (Downstream and Upstream) using XPaths
  3. Munge the data into something suitable for InfluxDB
  4. Insert the data into InfluxDB
  5. Query the InfluxDB data from Grafana

I knew I wanted to use Python for the project, so I first looked into Scrapy. After wrapping my mind around it (somewhat) I gave it a go and actually had a working solution… but it felt way over engineered and at times inflexible for what I wanted. I threw 90% of that solution away and went with a simpler script.

What I landed on was something that’s custom and lightweight, but extendable in case someone has a different status page or wants to use an alternative to InfluxDB.

Grafana screenshot showing that a fluctuation in downstream power around 10:00 a.m. caused the "Correcteds" values to spike.
I’ve had it running for a day and I’m already seeing interesting data!

See the repo on GitHub!

I bookmark. A lot.

I’ve been working on a Python script to help me parse (and make sense) of all the bookmarks I consume on a daily basis. I was interested in just how many I accumulate daily, monthly, and yearly. It turns out, a lot.Output

Published
Categorized as Scripts

Raspberry Pi + Garage door opener: Part 2

In Part 1 of my ‘Raspberry Pi + Garage door’ series, I showed a super simple way to control a garage door with a script that could potentially be ran from the Internet.

This part expands on that and tackles the issue: ‘How do I know the state of my garage door if I’m not at home?’

Because the code operates no differently than someone pressing a single button on the remote control, you would normally have to look with your own eyes to see if you were closing, stopping, or opening the garage door. This can be an issue when your eyes are nowhere close to it.

I wanted to come up with a solution that didn’t involve running new equipment such as a switch to detect the door’s orientation. I decided to utilize what I already had in the garage: A camera. Namely, this one: Foscam FI8910W

The idea is to use the camera to grab an image, pipe that image into OpenCV to detect known objects, and then declare the door open or closed based off of those results.

I whipped up a couple of shapes in Photoshop to stick on the inside of my door:

shapes_web

 

videostream
Shapes taped to the inside of the garage door

I then cropped out the shapes from the above picture to make templates for OpenCV to match.

Templates for OpenCV

The basic algorithm is this:

  1. Get the latest image from the camera
  2. Look for our templates with OpenCV
  3. If all objects (templates) were detected, the door is closed – otherwise it’s open

1423729418.65
Shapes successfully detected

 

To help make step 3 more accurate, I added a horizontal threshold value which is defined in the configuration file. Basically, we’re using this to make sure we didn’t get a false positive – if the objects we detect are horizontally aligned, we can be pretty certain we have the right ones.

I was happy to find that the shapes worked well in low-light conditions. This may be due to the fact that my garage isn’t very deep so the IR range is sufficient, as well as the high contrast of black shapes on white paper.

Currently I have some experimental code in the project for detecting state changes. This will not only provide more information (e.g. the door is opening because we detect the pentagon has gone up x pixels), but is good for events (e.g. when the alarm system is on, let me know if the door has any state change).

I’ve tested running this on the Raspberry Pi and it works fine, though it can be a good bit slower than a full-blown machine. I have a Raspberry Pi 2 on order and it’ll be interesting to see the difference. Since this code doesn’t need anything specific to the Raspberry Pi, someone may prefer to run it on a faster box to get more info in the short time span it takes for a door to open or close.

I’ve created a video to demo the script in action!

GitHub: https://github.com/twstokes/door_detect

Detecting Smith numbers

After learning about Smith numbers today I cooked up this script fairly quickly. There are many places where it could be optimized and cleaned up but that wasn’t really the point – I just needed something that was accurate, and I’m pretty sure this is.

It also helped me to learn about the ulimit command after running it with a HUGE number that had taken up about 7GB of swap before I had to kill the process remotely via SSH…


#!/usr/bin/python
# smith number detector - TWS 1-12-13

def detectSmith(num):
  if(sumFactors(num) == sumDigits(num)):
    return True
  else:
    return False

# calculate the sum of the factors
def sumFactors(num):
  factorList = factors(num)
  sum = 0

  for factor in factorList:
    if factor > 9:
      # if a number (base ten) has multiple digits it has to be broken down
      sum = sum + sumDigits(factor)
    else:
      sum = sum + factor

  return sum

# sum the digits in a number with two or more
def sumDigits(num):
  sum = 0  

  if num > 9:
    while num > 9:
      sum = sum + num % 10
      num = num / 10
    sum = sum + num
    return sum
  else:
    # should never reach this
    return num

def factors(num):
  primeFactorList = []

  factor = lowestPrimeFactor(num)
  currentNum = num

  while factor != -1:
    primeFactorList.append(factor)
    currentNum = currentNum / factor
    factor = lowestPrimeFactor(currentNum)

  if(currentNum != num):
    # preventing returning the number passed in
    primeFactorList.append(currentNum)

  return primeFactorList

def lowestPrimeFactor(num):
  for x in range(2, (num / 2) + 1):
    if(num % x == 0):
      return x
  return -1

for number in range(1, 1000):
  if(detectSmith(number)):
    print "The number", number, "is a Smith number."
Published
Categorized as Scripts

A simple Linux content search script

A simple script I use for work to search the contents of files. It just wraps up ‘find’ and ‘grep’ to recursively search and display the location of the file.

#!/bin/bash
# Tanner Stokes - 8-23-12
# Search contents of files

if [ -z "$1" ] || [ -z "$2" ]
then
echo "Usage: csearch \"filetype\" \"content\""
echo "ex: csearch \"*.php\" \"my_function()\""
exit 1
fi

find . -name "$1" -exec grep -Hn "$2" {} \;
Published
Categorized as Scripts

Changing an LCD string on a Dell server remotely with Python

Here’s a little Python script I cooked up for work. We often have to change the custom LCD string on boxes if the machine is given a new name. Before, we’d have to restart a machine to access the DRAC configuration to change the name. With this script, all you have to do is enable IPMI through the DRAC remotely (if it isn’t already enabled), and know the correct credentials.

If the LCD isn’t set to view the custom string, you should be able to change it from the front of the box. As you can see, the function that I have commented out at the bottom is supposed to do this for you, but I never got it to work.

You may have to change ‘/usr/sbin/ipmitool’ to your appropriate path.

#!/usr/bin/python
import os

# Tanner Stokes - tannr.com - 2-26-10
# This script changes the LCD user string on Dell machines that conform to IPMI 2.0

sp_hostname = raw_input ("\nEnter DNS or IP of SP: ");
user_string = raw_input("Enter LCD string: ")

hex_string = ""

for x in user_string:
 	hex_string += hex(ord(x))
	# add space between each hex output
	hex_string += " "

print '\nTrying to change LCD string on '+sp_hostname+'...'

return_val = os.system('/usr/sbin/ipmitool -H '+sp_hostname+' -I lan -U root raw 0x6 0x58 193 0 0 '+str(len(user_string))+' '+hex_string)

if (return_val == 0):
	print 'LCD string changed successfully.\n'
else:
	print '\nNon-zero return value, something went wrong.'
	print 'Make sure IPMI is enabled on the remote host and the DNS or IP is correct.\n'

# this function supposedly sets the user string to show on the LCD, but never got it to work
# this can be changed from the front of the box anyway
# os.system('/usr/sbin/ipmitool -H '+sp_dns+' -I lan -U root raw 0x6 0x58 194 0')
Published
Categorized as Scripts

Sending Growl messages over a network with Python

6-6-12 – This script is really old and may not work with current versions of Growl. There are some good libraries to choose from and I’d recommend looking at them instead.

Updated 12-4-10 – minor code updates
I have yet another script. This one is actually my first one in Python, which is kind of sad after all the good things I’ve heard about the language. This time I didn’t actually write the entire script – I simply added to one that had already been written to give it more functionality.

The original script was found here: http://the.taoofmac.com/space/Projects/netgrowl and almost all credit goes to it’s author, as I only added a little bit more to it.

What this version does is send a Growl message over UDP to a Mac (running Growl and configured to listen for incoming notifications) using Python. I thought this was really cool – I could script my remote Linux machines (or whatever OS) to send messages to my main Mac.

What I added was the ability to have command line arguments so that it would be easier to change options on the fly and also easier to script.

Running ./netgrowl.py -h will give you the following:

#!/usr/bin/env python

# Updated 12-4-2010
# Altered 1-17-2010 - Tanner Stokes - www.tannr.com
# Added support for command line arguments

# ORIGINAL CREDITS
# """Growl 0.6 Network Protocol Client for Python"""
# __version__ = "0.6.3"
# __author__ = "Rui Carmo (http://the.taoofmac.com)"
# __copyright__ = "(C) 2004 Rui Carmo. Code under BSD License."
# __contributors__ = "Ingmar J Stein (Growl Team), John Morrissey (hashlib patch)"

try:
  import hashlib
  md5_constructor = hashlib.md5
except ImportError:
  import md5
  md5_constructor = md5.new

import struct
# needed for command line arguments
import sys
import getopt

from socket import AF_INET, SOCK_DGRAM, socket

GROWL_UDP_PORT=9887
GROWL_PROTOCOL_VERSION=1
GROWL_TYPE_REGISTRATION=0
GROWL_TYPE_NOTIFICATION=1

def main(argv):

    # default to sending to localhost
    host = "localhost"
    # default title
    title = "Title"
    # default description
    description = "Description"
    # default priority
    priority = 0
    # default stickiness
    sticky = False

    # if no arguments are given, show usage
    if(len(argv) < 1):
        usage()

    try:
        opts, args = getopt.getopt(argv, "hH:t:d:p:s")
    except getopt.GetoptError:
        usage()
    for opt, arg in opts:
        if opt in ("-h"):
            usage()
        elif opt in ("-H"):
            host = arg
        elif opt in ("-t"):
            title = arg
        elif opt in ("-d"):
            description = arg
        elif opt in ("-p"):
            # acceptable values: -2 to 2
            priority = int(arg)
        elif opt in ("-s"):
            sticky = True

    # connect up to Growl server machine
    addr = (host, GROWL_UDP_PORT)

    s = socket(AF_INET,SOCK_DGRAM)
    # register application with remote Growl
    p = GrowlRegistrationPacket()
    p.addNotification()
    # send registration packet
    s.sendto(p.payload(), addr)

    # assemble notification packet
    p = GrowlNotificationPacket(title=title, description=description, priority=priority, sticky=sticky)

    # send notification packet
    s.sendto(p.payload(), addr)
    s.close()

def usage():
    print """Usage: ./netgrowl.py [-hs] [-H hostname] [-t title] [-d description] [-p priority]
Send Growl messages over UDP
  -h help
  -H specify host
  -t title
  -d description
  -p priority [-2 to 2]
  -s make sticky"""
    sys.exit(1)

class GrowlRegistrationPacket:
  """Builds a Growl Network Registration packet.
     Defaults to emulating the command-line growlnotify utility."""

  def __init__(self, application="NetGrowl", password = None ):
    self.notifications = []
    self.defaults = [] # array of indexes into notifications
    self.application = application.encode("utf-8")
    self.password = password
  # end def

  def addNotification(self, notification="Command-Line Growl Notification", enabled=True):
    """Adds a notification type and sets whether it is enabled on the GUI"""
    self.notifications.append(notification)
    if enabled:
      self.defaults.append(len(self.notifications)-1)
  # end def

  def payload(self):
    """Returns the packet payload."""
    self.data = struct.pack( "!BBH",
                             GROWL_PROTOCOL_VERSION,
                             GROWL_TYPE_REGISTRATION,
                             len(self.application) )
    self.data += struct.pack( "BB",
                              len(self.notifications),
                              len(self.defaults) )
    self.data += self.application
    for notification in self.notifications:
      encoded = notification.encode("utf-8")
      self.data += struct.pack("!H", len(encoded))
      self.data += encoded
    for default in self.defaults:
      self.data += struct.pack("B", default)
    self.checksum = md5_constructor()
    self.checksum.update(self.data)
    if self.password:
       self.checksum.update(self.password)
    self.data += self.checksum.digest()
    return self.data
  # end def
# end class

class GrowlNotificationPacket:
  """Builds a Growl Network Notification packet.
     Defaults to emulating the command-line growlnotify utility."""

  def __init__(self, application="NetGrowl",
               notification="Command-Line Growl Notification", title="Title",
               description="Description", priority = 0, sticky = False, password = None ):

    self.application  = application.encode("utf-8")
    self.notification = notification.encode("utf-8")
    self.title        = title.encode("utf-8")
    self.description  = description.encode("utf-8")
    flags = (priority & 0x07) * 2
    if priority < 0:
      flags |= 0x08
    if sticky:
      flags = flags | 0x0100
    self.data = struct.pack( "!BBHHHHH",
                             GROWL_PROTOCOL_VERSION,
                             GROWL_TYPE_NOTIFICATION,
                             flags,
                             len(self.notification),
                             len(self.title),
                             len(self.description),
                             len(self.application) )
    self.data += self.notification
    self.data += self.title
    self.data += self.description
    self.data += self.application
    self.checksum = md5_constructor()
    self.checksum.update(self.data)
    if password:
       self.checksum.update(password)
    self.data += self.checksum.digest()
  # end def

  def payload(self):
    """Returns the packet payload."""
    return self.data
  # end def
# end class

if __name__ == '__main__':

    # send command line arguments to main() function
    main(sys.argv[1:])
Published
Categorized as Scripts

Super simple BASH script to monitor a process

Here’s a little BASH script that I made to monitor a virtual machine on an OS X box. Basically when its CPU usage gets higher than 75%, I get a Growl notification and a spoken warning from Bruce – one of the many Mac voices.

One thing to note:

ps -eo %cpu 207 | tail -1

is where the magic is done. Replace ‘207’ with the PID of whatever you want to monitor. It would be wiser to make this script monitor the process by name, but in my case I have the VM running all the time.

#!/bin/sh

# CPU threshold
readonly THRESHOLD=75
alarmState=0

while true; do
    # make the output of our command a variable
    set `ps -eo %cpu 207 | tail -1`
    # turn float into int by truncating from decimal after
    toInt=${1/.*}
    # if our CPU usage is above our threshold
    if [ "$toInt" -gt "$THRESHOLD" ]; then	
        # if we haven't displayed a Growl notification for this alarm
        if [ "$alarmState" -eq "0" ]; then	
            alarmState=1
            # display a Growl notification
            growlnotify -sm "`date` Warning! CPU is at $toInt!"
        fi
        # shout out our warning
        say -v Bruce "Warning! CPU is at $toInt!"
        # consider putting 'sleep' here
    else
        # CPU usage is below our threshold
        alarmState=0
    fi
    # pause the script for a second
    sleep 1
done
Published
Categorized as Scripts