Moved Tech Blog

I know I haven't posted much recently, mainly due to being busy at work and thinking I needed my posts to be well thought out and interesting. I have come to the conclusion that they need not be either if they help me or might help someone else someday.

To that end, I have started a new tech blog with a friend over at http://twoguysarguing.wordpress.com/

If anyone still reads this blog or has it in there feed reader, please adjust your links accordingly.

Thank you,
The Management.

Cold Sholder

This is a VERY quick and dirty script I threw together to see if the conditions were right to go skating on a local pond (based on a friends best guess).

Simply, it checks an online weather service for the overnight temperatures of the past 10 days and if every night maintained temperatures below freezing, tells you to go skating. Otherwise, no skating for you.

require 'rubygems'
require 'simplehttp'
require 'time'

class TimedWeather
 attr_accessor :date_time, :temp
 
 def initialize(date_time_string, temp_string)
  @date_time = DateTime.strptime(date_time_string, '%Y-%m-%d %H:%M:%S')
  @temp = temp_string.to_f
 end
 
 def to_s
  "[(#{@date_time})-(#{@temp})"
 end
end

def build_daily_url(date)
 "http://www.wunderground.com/weatherstation/WXDailyHistory.asp?ID=KMOSTLOU11&day=#{date.day}&year=#{date.year}&month=#{date.month}&format=1"
end

def find_timed_weather_for_date(date = Date.today)
 response = SimpleHttp.get(build_daily_url(date))
 records = response.gsub(/\n<br>/, '').gsub(/<br>/, '').split(/\n/)[2...-1]
 records.map do |text_record|
  split = text_record.split(/,/)
  TimedWeather.new(split[0], split[1])
 end
end

def conducive_day?(daily_temps)
 not daily_temps.select { |time| conducive_time?(time) }.map { |temp| conducive_temp?(temp) }.include?(false)
end

def conducive_time?(time)
 time.date_time.hour > 17 or time.date_time.hour < 6
end

def conducive_temp?(temp)
 temp.temp < 32.0
end

conducive = true

(0..9).each do |offset|
 if not conducive_day?(find_timed_weather_for_date(Date.today - offset))
  conducive = false
  break
 end
end

if conducive
 puts "GO SKATE! ...... maybe"
else
 puts "NO SKATING TODAY!"
end

Maven and TweetPublisher

One of my recent clients used Maven for builds, dependency management, and packaging so I figured I would give it real try and build something with it. Over the years I have hear numerous good things about Maven (really nice dependency management, consistent builds that don't require a huge ant file to work, simplicity) along with many bad things (complex when you try to do anything that is not standard, annoyance with the dictated conventions, difficulties managing the things managing your dependencies, etc). After trying it out on a couple very small and simple projects, I very much like the good points and am not convinced that the bad ones won't happen on a larger project.

For small stuff though, it is very nice.

If you follow the conventions, you can easily get a small project structure built out along with the ability to easily compile, test, and package/install without having to tell the build too anything about HOW to do it, just where/when.

One issue I did run into was related to packaging my app. My first test app was a simple CruiseControl plugin called TweetPublisher. A coworker wanted his Cruise server to tweet the status of each build ... so I set to work. After googling a while about how to write a publisher plugin for CruiseControl and finding a suitable Twitter API (Twitter4J) I was pretty much done.

** The latest versions have a much nicer interface and a very simple configuration than what I remember. :-)

The most basic maven project type produces a simple project which builds to a single jar. Perfect. With Eclipse's maven support it was dead simple to search for the cruise control api and twitter4j libraries, add them as dependencies, hack together some code and I was done. A short "mvn package" later and I had a jar with my code.

Only 1 problem.

Maven knows about my dependencies. It keeps them up to date (depending on how you configure them), downloads them when needed, uses them at compile and/or test time, etc. However, it does nothing out of the box to give them to you when you package. More simply, after a package goal is executed I have a jar with my code, which was compiled against my dependencies, but I have no clue what version it was compiled against or know where they are. This does not leave me in the situation I wanted: package should leave you with everything packaged so you could go use it in production. Not so.

After much googling and asking of coworkers no one could show me how to work with maven and dependencies short of writing my own archetype to do what I wanted.

Finally, I stumbed upon the assemble plugin bywhich I could define an "assembly" which would move my code around and copy all of its dependencies flattened into my code's .jar file. "mvn assembly:assembly" Viola! Done.

The very simplistic and untested code for tweetpublisher can be found here on github.

** The maven integration with NetBeans is also very nice.

Computational Mind Teaser

Via GoogleReader and DZone ... I came across this post yesterday: http://blogs.digitss.com/programming/a-mind-teaser-good-one/

Of course I couldn't pass up the challenge and spent all of my free random moments yesterday afternoon trying things out. I was able to see how the subtract function was easily related to decrement and had several promising ideas on how decrement could work but nothing right. This morning a coworker and I sat down and finished all three in under 30 minutes. I post my solutions here b/c I feel the need to justify the time wasted ;-)

DEC(b) :=
  a = 0
  loop(b)
    b = a++

SUB(a,b) :=
  loop(b)
    DEC(a)

DIV(a,b) :=
  q = 0
  loop(a)
    SUB(a,b)
    c = a
    DEC(c)
    loop(a)
      q++
    SUB(q,c)
  a = q

* all functions leave the result in the first variable and assume the operator presents itself between the two parameters (e.g. SUB(a,b) == (a=a-b) ... prefix notation)

** I like my DIV better than those posted as of the writing of this article but still feel like it could be shortened / cleaner. I am happy to say that the hack of reducing the second loop to only a single increment while using the loop(0) as the "conditional" to stop working was my idea :-)

The coworker I figured this out with then tried to explain y-combinators in lambda calculus and how they might relate to functional technique he was trying in JavaScript ..... my head hurts ... in a good way.

List-It

In an effort to maintain a side-project, learn more Python (which is getting damn confusing with the Ruby I was learning a few months ago), and finally have a clean app to keep my To-Do list online .... I present List-It.

** Its still in super-alpha at the moment **

Myself along with a coworker are tinkering around with producing a simple to-do application utilizing Google AppEngine, Google Code (link to our project), and JQuery.

At the moment a VERY basic interface is available and some simple back-end plumbing has been done for the main model (ToDo objects).

I am fairly proud of my simple Actionizer code. Basically, since AppEngine doesn't use a DB but really uses Big Table ... I wanted to expose some simple actions for each of my models so that AJAX calls could easily update the models, delete, and add. Not wanting to bolt on a larger framework which would already have the functionality but not teach me nearly as much Python, I decided to roll my own.

Thus the Actionizer class. Create an instance with the actions you want to give your models and then actionize any/all of your model objects. The Actionizer will produce the appropriately named URL actions and piping along with functions to produce request handlers to handle each model's actions.

Not ground breaking stuff, but not bad for just a couple of hours tinkering around with Python.

If you have any suggestions/thoughts/criticism/would like to help
... please let me know.

** In working on this side project ... I realized it would be nice for our purposes to have AppEngine monitor our Google Code poject for commits and when one is sent, to redeploy the app directly from the repository. Allass I can't find a way to do this yet, its not formally supported and no one on StackOverflow knows how to either.

Selenium - Chemistry for your UATs

Off and on for the past few weeks I have been working on migrating my current project's suite of user acceptance tests (UATs) from WatiN to Selenium.

Why you ask? Several reasons.

1. Our current WatiN tests are slow and brittle. Some of this is WatiN's fault, some our own
2. WatiN doesn't have great support for AJAX testing
   
3. WatiN only works in IE on Windows. It does this well, but our app needs to work in 7 browsers at the moment (IE 6 & 7, FF 2 & 3 on Mac & Win, Safari on Mac) soon to be 10 (Safari on Win, Chrome, IE 8).
4. Our company is going to be creating a Selenium Grid with remote control nodes of various browsers on different platforms for all projects to be able to use.
5. Currently the UATs don't run as part of any automated build
   

Thus, we are slowly trying to see what kind of effort it will take to convert our tests, rewrite/fix/upgrade where we can and need to, and automate as much as we can.

The really nice thing is the Selenium Grid and the ability to have a large number of RC's standing by in VMs for any project that wants to run tests. If you don't know about Selenium Grid, read about it here.

In our initial efforts to convert the tests we have found a few things to be true:

• Selenium has some built in checks that help with testing/action on AJAX calls, but they aren't perfect and primarily rely on built in check-wait loops on elements being on the page or Javascript functions changing values.
• Firefox 3 is not yet fully supported. A simple hack allows it to be used, however not all of the integrations and XPath work correctly
• I had a fair bit of trouble getting Safari to work on the Mac with Selenium. Evidently there are some architectural issues the team is battling with related to proxies and libraries ... but I was able to get it to work eventually
• The Grid/Remote Control environment works well but still has some bugs. Closing the window for a running RC on windows doesn't unregister the RC from the grid.
• Selenium seems to be a small but noticeable amount faster than WatiN
• Converting tests is hard when trying to ensure that the same exact thing is being tested before and after

Other than these issues, things are progressing. Three larger problems we are seeing and havn't come up with solutions for yet are:

• How can we automate the UATs running on the continuous integration server to run all browsers in parallel (e.g. running each test against all browsers at the same time, then moving on to the next one) while still maintaining one build/failure point?
• Is it possible to integrate the running of UATs against multiple browsers to the developer's work flow? Possible tooling support?
• Is it possible to rework a test-runner to run all of our tests in parallel depending on the number of RC's of a particular type that are available (e.g. if we have a ton of FF 3 RCs, run several tests all at once instead of waiting for each other)

It looks like we are going to be in between contracts for about a week so we are hoping to come up with some solutions to these. I will post back if we do.

** If you have any suggestions on how to automate or parallelize the Selenium tests, check our my StackOverflow question.

Lunch n' Learn - Shoes

Two weeks ago I did a Lunch-n-Learn at work on the simple Ruby GUI toolkit, Shoes. Here is a quick overview of what I talked about ....

In working on a side project of mine I came to the realization that I needed a GUI frontend to my app in order to keep moving forward. I had been working in Ruby and found that there is not a standardized GUI toolkit. There are >10 toolkits around that boast a variety of features and pros/cons but none that was universally accepted. My criteria were that whatever I chose, it needed to be multi-platform and easy to work with for simple things. Enter Shoes.

Shoes is the creation of why the lucky stiff, or _why as he is known. If you have never seen/heard/read anything of _why's, I very much suggest you do. Just Google him. Between his talks, his blogs, his books (one on Shoes and another on learning Ruby), and anything else you will learn a deep appreciation for what is obviously a very passionate and sometimes strange being. :-)

Shoes is designed to be simple and small. The main website at Shoooes.net demonstrates a simple GUI with a single buttons as:

Shoes.app { button "PUSH!" }

Most of the constructs have origins in how the web works and/or how modern browsers display elements so nothing is too complicated. You can easily display images, draw 2d shapes on the canvas, add buttons/text/text boxes/progress bars/etc, link to the web or to other windows, and show any of several built in dialog boxes. Much of Shoes features come from its integration with native components ... there are seperate Windows/OS X/Linux GTK installs.

While the framework isn't yet production ready by my accounts, it is rapidly evolving through _why's efforts and a surpisingly dedicated mailing-list. Would I write a complicated IDE in it? No. Would I consider it if I needed a simple user interface or data visualization for a small program? Yes.

If you considering checking it out, look at the main website first, peruse the help area which has gotten drastically better in the last couple months, and check out some of the user submited demo apps at the ShoeBox. Finally, check out _why's online (and print) PDF book on Shoes - Nobody Knows Shoes. If you have never tried drugs, or would like to relive the experience, flip through its pages. Along with good information on using Shoes are some crazy graphics and entertaining side-bantor.

If your interested in my LnL presentation or my thoughts on Shoes drop me a comment. The main code portion of my presentation (done in Shoes of course) can be found here. Please no comments on code style or my Ruby abilities ... I am still learning and put it together to show people what Shoes could do, not to prove myself. ;-)

Helpful Scripts For Work

I can be pretty lazy at times ... but in a good way, I think. I am a firm believer in the philosophy heard numerous times in various forms: "If you have to do something more than twice, have a computer do it". While I don't strive to find tons of little shortcuts to daily life, I do enjoy them.

At work in the past week, two scripts have saved me countless seconds maybe even minutes, and I figured I would share them here:

DZone RSS Links Streamliner

I use Google Reader to peruse the zillions of RSS feeds I am subscribed to and find the many articles I want to read and the fewer that I actually find time to. DZone.com is a really nice article aggregator centered on tech and programming news from around the web (think Digg before it got way overcrowded and full of celebrity gossip posts and links to fark ... not that there isn't a need for them).

Often I will find several DZone links I want to read at once and will open them all into new tabs from Google Reader and then read them one by one ... only one small problem ... when the link opens its to the DZone page. This is fine except that I currently don't have a DZone account so I don't vote on storries and I rarely read the comments ... I just want the article (e.g. here). Obviously having to click twice is at least 1 too much ;-)

Thus, here is a GreaseMonkey script I added so that I don't have to click through that extra time:

// ==UserScript==
// @name DZone Forward
// @namespace yardspoon
// @description Forwards user past DZone rss link page
// @include http://www.dzone.com/links/rss/*
// ==/UserScript==

window.location = document.evaluate("//div[@class='ldTitle']/a/@href", document, null, XPathResult.STRING_TYPE, null).stringValue;

It uses FF's built in XPath evaluator to find a link to the real article and forward the browser on to the page. It might be overkill for 1 click, but I like it. I don't have anything agains't DZone, I just don't like clicking more than I need to.


User-Agent Check Cheater

Most of my career has been working on networked systems and applications ... most of which have had a web frontend. Despite the best laid plans, often I am left with a situation where I really need to see whats actually being sent back between the server and the browser/client/web service and that can be difficult to do depending on the information you want to see. FF plugins like FireBug, XPather, and LiveHTTPHeaders give you tons of good information but can't give you the whole picture of whats being sent over the wire.

Enter WebScarab. I found this Java based personal web proxy right out of college while at my first client and am still finding uses for it frequently. After starting up a listener in WebScarab and configuring your browser to use it as a web proxy, it can easily monitor and display information about every single request and response sent (HTTP headers, redirects, content over the wire, additional resource requests, timing, etc ... great of AJAX debugging). Along with viewing the content it has numerous other features that allow you to simulate network conditions, manually intercept requests/responses before they are sent, do security fuzzy logic tests, test web services and much more.

One of the less used features but very important this week was the ability to act on requests and responses directly in a scripting language (BeanShell). This past week I was trying to install an application that for whatever reason, during the install process, needed to download additional information from the company's website. Only one problem: the installer uses IE not the default browser to do this, and the company blocks IE for external use for security reasons (FF installed on every desktop). This meant the installer failed. I could put in a support request for a temporary hole in the network to allow IE to access this site but since it was such a small thing I figured I would try to solve the problem directly.

It turns out outgoing requests are simply filtered by their user-agent HTTP header value (effective for IE blocking and simple) but also simply circumvented.

Queue WebScarab, the following script overrites the user-agent header of all requests with the value for FF ... meaning IE works and I was able to install the software. I turned the script back off when I was done so as to not risk any security issues and not piss off the admins ... but it worked none the less.

import org.owasp.webscarab.model.Request;
import org.owasp.webscarab.model.Response;
import org.owasp.webscarab.httpclient.HTTPClient;
import java.io.IOException;

public Response fetchResponse(HTTPClient nextPlugin, Request request) throws IOException {
request.setHeader("User-Agent", "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.3) Gecko/2008092417 Firefox/3.0.3");
response = nextPlugin.fetchResponse(request);
return response;
}

One other great feature of WebScarab is that if you set IE to use the web proxy, your really setting the default proxy for connections by other Windows applications that use the defaults ... meaning you can suddenly see all of the traffic that random applications are sending without you knowing ... apps phoning home, checking for updates, reporting usage statistics, etc. Kinda scarry.

Also ... watch-out ... the request and responses can be seen in plain text and/or Base64 Encoded text can be converted .... so don't send your password through it unless your paying attention :-) and realize how much of your web interactions are sent unencrypted over the wire for anyone who wants to listen to find.

Probably Fun - Baseball Stats

The other day a coworker who is very interested/involved in baseball statistics posed an interesting question to me just before I left work:

What is the probability of a team playing in a 5 game series to win the series on a streak of 3 wins?
(e.g. team A sweeps team B and wins the first 3, A loses 1 and then wins 3, or A loses 2 and wins final 3)

I quickly had flash backs of college to my statistics, discrete mathematics, number theory and finite automata courses (all favorites other than statistics). We quickly worked through some of the simpler aspects of it and came up with an answer that seemed well reasoned but not quite right. Of course I couldn't stop thinking about why it was wrong and went home and did further calculations, searched for my old statistics book, and even consulted Big Red. I eventually found the right answer but couldn't give myself the proof of why it was right.

The next day, after consulting with several other coworkers (at one point at least 15 people were busy discussing the problem and getting zero work done) and getting a well timed email from Big Red with his thoughts we were settled on a right answer.

I still can't put together a very clever or clear proof of why this solution is true, and I feel that there is probably a much succinct formula I am confident that it is correct .... a Monte Carlo simulation produces the exact same probability. :-)

Our answer, abstracted a bit:

The probability of a particular team winning a standard playoff series (where once a team wins more than half the games, the series is over) where the probability of the team winning each game is P and the number of games needed to win are X is:

P^X * ( (1-p)⁰ + (1-p)¹ + (1-p)² + ... + (1-p)^(X-1) )

so for a 5 game series where the odds are even:

0.5³ * ( (1-0.5)⁰ + (1-0.5)¹ + (1-0.5)² )

0.125 * ( (0.5)⁰ + (0.5)¹ + (0.5)² )

0.125 * ( 1 + 0.5 + 0.25 )

0.125 * 1.75

0.21875

The probability of a particular team winning 3 games in a row in a 5 game series is 21.875% assuming each game is a 50/50 shot.

Without going into even more detail, the confusion was the result of trying to simplify the problem into "There are X outcomes possible and Y of them are the ones we want and since there is even probability of the games the total probability is Y/X" However, since various possible outcomes involve a different number of games, they are actually weighted differently and thus its not 30% but noticeably less at ~ 22%.

I have to thank Pip for the shout-out (see comment) on Fungoes.net (the STL SABR chapter blog).
Pip is a coworker who posed the original question.

First App Engine Application - Static Map Maker

Over the past couple weeks I have been "working" on a newer project idea related to geo/gis data and mapping. In working on and thinking about the project it became clear that it would be useful to be able to generate static map images for specific lat/long coordinates on Earth. Basically, I wanted Google Maps images without Google Maps so that I could use the map images as references in my app (to be outlined in future posts).

Thus ... I bring you Static Map Maker!
(the link goes to the live app which anyone can currently try after logging into their Google Account)

Being my first real venture into the land of Google App Engine applications and something that I was proud of and able to throw together in about an hour of work (plus a bit more research prior) ... I present to you this quick rundown of what I did and how I did it:

After a small bit of research I found that Google had opened up some of its mapping data in the form of a Static Map API which allows for the retrieval of specific static map images based on the given http url. Perfect. The only things needed were a Google Maps API Key (free with registration) and the specifics of the map you want.

I had downloaded the Mac GoogleAppEngineLauncher previously but needed to updated to the latest SDK version before I could upload any code. With the launcher's GUI, its quite simple to create a new app and get something very fast and dirty running in no time. After a few minutes of flipping through the Getting Started tutorial I had a index page which showed my desired form, a result page, and a couple controllers to do the heavy lifting.

After a bit of testing where the map key for localhost came in very handy, I had a very limited but working none-the-less application that would allow me to enter new parameters and see the resulting map. The current app has no data validation and lacks any sort of formal tests or great coding standards but its functional (I am not a Python man by trade).

I then registered the application on the app engine site (SMS verification of your intent to create a new application) and used the launcher to one-click upload my code. Viola! Finished!

** I will probably post the code either on code.google.com or github, but that can wait until tomorrow **

Project Euler

Some days I feel dumb

Well, no dumb ... just not as smart as I did in college ... learning new things and pushing my brain everyday.

Queue Project Euler.

The site has a couple hundred math/computational problems to solve. All (that I have read) are math based but don't require just mathematical proofs to solve. Most can be simplified through mathematical reasoning, but will require you to write an algorithm to solve (ideally under a minute). If you find an answer you can check it online and if its right (and maybe even if it isn't) see someone's real solution and a forum to discuss the problem.

I have done several of these over the past week or so and give me a nice problem for my brain to munch on in the background while I am at work or mowing the lawn.

**I have posted some of my solutions/thoughts about the problems in a Google Notebooke here if anyone is interested.

Synergy

Back in college I was a nerd. Honestly, I am still a nerd, but that's beside the point.

In college I had a Windows XP box as my main computer with two monitors. I also had a linux box with its own monitor. Not being one to want to have to switch back and forth between keyboards I quickly devised a complex KVM setup that might give me what I want (using multiple KVMS and crazy logic). After a roommate convinced me a it was a lost cause, another roomate or a random forum post somewhere (I honestly can't remember which) turned me on to Synergy. Voila! Problem solved.

This small opensource app allows two computers, each with their own monitors, to share a single keyboard and mouse. The setup is simple:

1. install the server on one machine
2. setup "screens" for each computer you want to connect
3. configure how the screens connect (e.g. the left side of my windows machine connects to my linux machine and the right side of the linux machine connects to windows)
4. press start
5. install the client on the other machine and tell it to connect to the first machine
6. enjoy

6 simple steps and I could sweet my mouse from one side of my dual monitor setup across to the other monitor and all the way onto the linux machine. Keyboard focus changing as I go.

The setup isn't without its drawbacks (it appears that the Mac client isn't totally complete including copy and paste and invariably you will attempt to drag a window off its host OS and wonder why the mouse stops moving) but for free its great.

Right now I am using it at work to make my testing of our app in Safari easier. Check it out.