Michael Jay Lissner

Project Idea: “Programming library for curse words”

2011-03-13T23:13:34-07:00

When programming, there are occasionally times when you need to detect or block curse words. At CourtListener, for example, we make URLs with ID numbers in them that are formed by converting an ID number to letters (so a → 1, b → 2, 27 → A, etc). Higher numbers create longer strings of letters, so over time, this creates curse words in the URL. Currently, the site is only has a few four letter strings, but I will rue the day when any of the seven dirty words is being shown to users on my site.

There are many lists of curse words on the web, but none that is maintained or curated. Having that alone would be a useful project. What would make it better would be libraries in popular programming languages that efficiently told you if a string contained a curse word.

The next feature would be to add additional languages, and then to add words like pen1s, which aren’t normally curse words, but are certainly words you’d want to eliminate.

It’d be a pretty simple project, so I may just go for it.

Only question is, what do I name it?

Swimlane Diagram Generator Written in XSLT

2010-09-06T23:26:18-07:00

For the past couple years, I’ve been wanting to make a swimlane diagram showing all of my roommates and which room they lived in. I considered drawing it out by hand with a charting program, but the idea of updating it whenever somebody moved in or out seemed daunting, and I decided that the best thing would be to make a program that could generate a chart from XML. My new job at Recommind requires that I learn and use XSLT, so I took the opportunity to write an XSLT script that converts the XML data to HTML, Javascript and SVG.

The final product looks something like this (anonymized with Presidential info for good measure):

(click through for complete demo)

For the technically inclined: The program is an XSLT script, which converts the XML into HTML and Javascript. The Javascript is then interpreted by the Raphael library, which finally generates the SVG you see. It’s overly complex, but it was a fun mis-mash of technologies to play with and the point was learning new things as much as anything.

The transformation should work to make all kinds of swimlane diagrams, so if you’re interested in the code, let me know.

A Python Function to Verify Twitter Credentials

2009-04-03T19:09:25-07:00

Thought I’d post this for the future generations, since I had a hard time finding a template anywhere on the web when I needed one. It’s nothing revolutionary, but a useful snippet nonetheless. This is for one of my projects this semester.

import pycurl


def verifyTwitterCredentials(username, password):
    c = pycurl.Curl()
    c.setopt(c.URL, 'http://twitter.com/account/verify_credentials.xml')
    c.setopt(c.USERPWD, username + ":" +  password)
    twitterfeed = c.perform()

    status = c.getinfo(c.HTTP_CODE)

    if str(status) == '200':
        verified = True
    else:
        verified = False

    c.close()

return verified

Working with matplotlib and pycairo

2009-01-19T16:25:32-08:00

I spent a good part of my winter break working on learning Python and using it for projects. One project was the Yelp scraper that I posted about previously, and another was a report for my old work.

The report is a statistical analysis of the development of about 2,000 children aged three and four. For those interested, I’ll try to post it here once the final version is ready to go. In the past when making the report, I had been frustrated because there was no easy way to script the creation of the 30 or so charts that need to be made. Excel had been our data analysis tool, and as such, we were stuck with either using VBA to create charts, or to do it by hand. Since nobody knew VBA, we always just buckled down and did the work by hand.

This time around, I discovered the matplotlib Python library, and used that to create the charts. It was an pretty rough experience all in all. While simple graphs can be created in about five lines of code, creating complicated ones took a good amount of work. For example, to change the tick markers on a graph requires that you create tick objects, and then manipulate them each individually in a for loop. Granted, I couldn’t customize them at all in Excel, but figuring out that kind of change was a pain indeed.

The report itself required about 1,000 lines of code, and each chart required about 100-200 lines. For custom charts, I didn’t find the library that useful, however towards the end of the report there are 30 charts, all of which are identical, except for the data. For these charts, I was able to make a for loop that created them all in about 20 minutes, whereas previously these took me a few hours to make by hand.

Another library I spent some time learning was pycairo, which allows pixel by pixel editing of pictures. I had planned to use it to do any editing to the charts that I was unable to accomplish with the matplotlib library, but in the end, it was unnecessary. I have another project coming up though that will use the pycairo library, so look for that soon.

Yelp Scraper to Get Business Info in a Geographic Area

2008-12-21T16:41:13-08:00

I spent the past couple days on one of my first Python projects - using the Yelp API to compile a list of restaurants in a defined geographic area.

It’s been a good project. Because of some limitations of the API, I had to do some interesting tricks to make it work. One problem with the API is that it only allows 20 hits per query, so if you want to do a big query, you have to divide it up into tiny queries that have fewer than 20 hits each.

To accomplish that, if a query gets 20 hits within those two points, it will divide the longer dimension of the rectangle created by the points in half, and perform a query on each of those two new rectangles. For each of those, if there are 20 hits, it will again divide it in two and perform two new queries, and so forth until less than 20 hits are found for the rectangle. Once less than 20 hits are found, the data is entered into a database. Once all the points have been added to the database, a comma separated file is created, and the program ends.

It was pretty incredible switching to Python for this project from my usual Java, and also using an official API for the first time. This project ended up being about 200 lines (half of which are comments). I can’t imagine how long it would be with Java, since I used some rather powerful Python modules to accomplish this (namely, csv, urllib & json).

If anybody is interested in seeing/using the code, let me know. It should be useful if you need a list of restaurants or other businesses in a certain area. Worthy causes only please!

PCT Data Project - DONE

2007-12-12T21:29:57-08:00

I’m happy to announce that the PCT data project is complete!

Over the past several weeks/months, I have been slaving away over my computer writing this program. When used, it will generate a dynamic graphing area that will load up temperature data for one to six PCT hikers.

All those that are interested in the most complicated programming assignment I have ever worked on are welcome to check it out.

I am officially a free man once again! Thanks to all who made this possible with their encouragement and patience!

The Great Temperature Data Project

2007-11-01T20:40:55-07:00

Back in ‘05 when I hiked from Mexico to Canada on the Pacific Crest Trail, I carried a little device called an iButton. This little device contains essentially three things: a clock, a bit of memory and a thermometer. It’s waterproof, accurate to 0.1 degree Celsius, and is about the size of five dimes stacked one upon another. There are a bunch of silly things you can do with these, but what I chose to do with mine was to have it record the temperature every hour on the hour for the entire time I was hiking, with the idea being to get some good data about the temperature out there on the PCT.

All in all, you can figure that the temperature was recorded 24 times a day for about 150 days, for an astounding 3600 data points, and about 150 oscillations from the daytime high to the nighttime low. I’ve spent some time working with the data, and it’s pretty much impossible to make much use of….unless you write a program to interpret it. You can see it for yourself if you’re interested.

Well, as fate should have it, I am currently enrolled in a Java programming class, and I have the option of doing a final project of my own choosing. Having not put this data to good use has been a burden on my soul for a couple of years now, and I’ve decided to make my final project an applet that will allow a user to plot this data on a graph for any date range and any time range that they choose (e.g. 5pm to 10pm for September 20th to 23rd).

Once this is done, I will attempt to post it here, but here’s the question to you dear reader, do you have any suggestions as to features that you would be interested in seeing in an applet of this sort? Thoughts?

I’m quite excited about getting this info out there. Finally.