Michael Jay Lissner

Some Thoughts on Celery

2014-11-01T00:00:00-07:00

We finally upgraded CourtListener last week and things went pretty well with the exception of two issues. First, we had some extended downtime as we waited for the database migration to complete. In retrospect, I should have realized that updating every item one row at a time would take a while. My bad.

Second, Celery broke again and that took me the better part of a day to detect and fix. As a central part of our infrastructure, this is really, truly frustrating. The remainder of this post goes into what happened, why it happened and how I finally managed to fix it.

Why?

First, why did this happen? Well…because I decided to log something. I created a task that processes our new audio files and I thought, “Hey, these should really log to the Juriscraper log rather than the usual celery log.” So, I added two lines to the file: One importing the log file and the second writing a log message. This is the little change that brought Celery to a grinding halt.

What the Hell?

If you’re wondering why logging would break an entire system, well, the answer is because Celery runs as a different user than everything else. In our case, as the celery user — a user that didn’t have permission to the log file I requested. Ugh.

Fine, that’s not so bad, but there were a number of other frustrating things that made this much worse:

The Celery init script that we use was reporting the following:

↪ sudo service celeryd start
celeryd-multi v3.0.13 (Chiastic Slide)
> Starting nodes...
    > w1.courtlistener.com: OK

But no, it was not starting “OK”. It was immediately crashing.

No log messages…anywhere. This appears to be because you have to detach stdin and stdout before daemonizing and according to asksol on IRC, this has been fixed in recent versions of Celery so even daemonizing errors can go to the Celery logs. Progress!
The collection of things that happens when celery starts is complicated:
1. I call sudo service celeryd start
2. service calls /etc/init.d/celeryd
3. celeryd does some stuff and calls celery.sh (another file altogether), where our settings are.
  
  Update: Apparently this is a CourtListener-specific customization, so this step probably won’t apply to you, but I have no idea where this wacky set up came from (it’s been in place for years).
4. Control is returned to celery, which starts celery itself with a command generated from celery.sh.
On top of this, there’s a celery binary and there’s a celery management command for Django. (Update the Django commands were removed in Celery 3.1. More progress!) celery --help prints out 68 lines of documentation. Not too bad, but many of those lines refer you to other areas of the documentation. For example, celery worker --help prints another 100 lines of help text. Jesus this thing is complicated.

Did I mention it has changing APIs?

I digress a bit, but the point here is that it fails silently, there are no log messages when it fails, and there’s no way to know which part of a complicated infrastructure is the problem. End rant.¹

Seeking Sanity

It took me a long time to figure out what was going wrong, but I did eventually figure it out. The process, in case you run into something similar, is to modify celeryd so it prints out the command that it eventually runs. At that point you’ll have the correct command. With that, you can run it as the celery user and with some luck you’ll see what the problem is. There’s a modified init script for this purpose, if you like.

Other tips:

If you have a new enough version of Celery, there are some troubleshooting tips that should help. They did nothing for me, because I haven’t upgraded yet for fear of the changing APIs.
There seem to be a handful of different command line flags that Celery can use to be sent to the background. You’ll need to disable these when you’re testing or else you won’t see error messages or anything (apparently?).

Moving Forward

So, I feel bad: I’ve ranted a good deal about Celery, but I haven’t proposed any solutions. It looks like a lot of things have been improved in recent versions of Celery, so part of the solution is likely for us to upgrade.

But this isn’t the first time I’ve spent a long time trying to make Celery work, so what other ideas it take to make Celery a less complicated, more reliable tool?

The ideas I’ve come up with so far are:

More documentation for installation and set up troubleshooting with the possibility of a wiki.
- But already I rant about how much documentation it has.
A simpler interface that eliminates a number of edge uses.
- But I have no idea what, if anything, can be eliminated.
Support for fewer task brokers.
- But I use RabbitMQ and am considering switching to Redis.
A more verbose, more thorough debug mode.
- But apparently this is already in place in the latest versions?
Let Celery run as the www-data user as a general practice?
- But apparently that’s a bad idea.
  
  Update this is a bad idea in general, but it’s not particularly bad if you don’t expose Celery on the network. If you’re only running it locally, you can probablly get by with Celery as a www-data user or similar.

As you can tell, I don’t feel strongly that any of these are the right solution. I am convinced though that Celery has a bad smell and that it’s ripe for a leaner solution to fill some of its simpler use cases. I’m currently considering switching to a simpler task queue, but I don’t know that I’ll do it since Celery is the de-facto one for Django projects.

We deserve a good, simple, reliable task queue though, and I wonder if there are good ideas for what could be changed in Celery to make that possible. I, for one, would love to never spend another minute trying to make RabbitMQ, Celery and my code play nicely together.

In truth Celery is a classic love/hate relationship. On the one hand, it evokes posts like this one, but on the other, it allows me to send tasks to a background queue and distribute loads among many servers. Hell, it’s good enough for Instagram. On the other hand, god damn it, when it fails I go nuts. ↩

New tool for testing lxml XPath queries

2012-05-20T15:48:06-07:00

I got a bit frustrated today, and decided that I should build a tool to fix my frustration. The problem was that we’re using a lot of XPath queries to scrape various court websites, but there was no tool that could be used to test xpath expressions efficiently.

There are a couple tools that are quite similar to what I just built: There’s one called Xacobeo, Eclipse has one built in, and even Firebug has a tool that does similar. Unfortunately though, these each operate on a different DOM interpretation than the one that lxml builds.

So the problem I was running into was that while these tools helped, I consistently had the problem that when the HTML got nasty, they’d start falling over.

No more! Today I built a quick Django app that can be run locally or on a server. It’s quite simple. You input some HTML and an XPath expression, and it will tell you the matches for that expression. It has syntax highlighting, and a few other tricks up its sleeve, but it’s pretty basic on the whole.

I’d love to get any feedback I can about this. It’s probably still got some bugs, but it’s small enough that they should be quite easy to stamp out.

Update: I got in touch with the developer of Xacobeo. There’s an --html flag that you can pass to it at startup, if that’s your intention. If you use that, it indeed uses the same DOM parser that my tool does. Sigh. Affordances are important, especially in a GUI-based tool.

The Winning Font in Court Opinions

2012-01-27T22:15:58-08:00

At CourtListener, we’re developing a new system to convert scanned court documents to text. As part of our development we’ve analyzed more than 1,000 court opinions to determine what fonts courts are using.

Now that we have this information, our next step is to create training data for our OCR system so that it specializes in these fonts, but for now we’ve attached a spreadsheet with our findings, and a script that can be used by others to extract font metadata from PDFs.

Unsurprisingly, the top font — drumroll please — is Times New Roman.

Font	Regular	Bold	Italic	Bold Italic	Total
Times	1454	953	867	47	3321
Courier	369	333	209	131	1042
Arial	364	39	11	41	455
Symbol	212	0	0	0	212
Helvetica	24	161	2	2	189
Century Schoolbook	58	54	52	9	173
Garamond	44	42	41	0	127
Palatino Linotype	36	24	24	1	85
Old English	42	0	0	0	42
Lincoln	27	0	0	0	27

Using Pylint in Geany

2010-08-11T12:07:23-07:00

Pylint is a tool that tells you when your Python code is broken or when it has coding problems. As a newish Python coder, using it has taught me a lot about conventions, and has helped to make my code significantly cleaner. Enabling it in my IDE, Geany, makes it so that using it is just another part of my development workflow.

Enabling Pylint in Geany is easy. Simply open Geany, and create a new build command that uses pylint -r no "%f" as the command, and (W|E|F):([0-9]+): (.*) as the error regular expression. After you’ve done this, using this build command instead of saving your work will run Pylint on your current file, showing you warnings, errors and fatal errors in red.

Using Revision Control on a Django Project Without Revealing Your Passwords

2010-02-24T17:15:54-08:00

Just a quick post today, since this took me way too long to figure out. If you have a django project that you want to share without sharing the private bits of settings.py, there is an easy way to do this.

I tried for a while to to set up mercurial hooks that would strip out my passwords before each commit, and then place them back after each commit, thus avoiding uploading them publicly. This does not work however because all of the mercurial hooks happen after snapshots of the modified files have been made. So you can edit the files using a hook, but your edits will only go into effect upon the next check in. Clearly, this will not do.

Another solution that I tried was the mercurial keyword extension. This could work, but ultimately it does not because you have to remember to run it before and after each commit — something I know I’d forget sooner or later.

The solution that does work is to split up your settings.py file into multiple pieces such that there is a private file and a public file. I followed the instructions here, with the resulting code looking being checked in here and here. There is also a file called “20-private.py” which is not uploaded publicly, and which contains all the private bits of code that would normally be found in settings.py. Thus, all of my settings can be found my django, but I do not have to share my private ones.

A Python Function to Verify Twitter Credentials

2009-04-03T19:09:25-07:00

Thought I’d post this for the future generations, since I had a hard time finding a template anywhere on the web when I needed one. It’s nothing revolutionary, but a useful snippet nonetheless. This is for one of my projects this semester.

import pycurl


def verifyTwitterCredentials(username, password):
    c = pycurl.Curl()
    c.setopt(c.URL, 'http://twitter.com/account/verify_credentials.xml')
    c.setopt(c.USERPWD, username + ":" +  password)
    twitterfeed = c.perform()

    status = c.getinfo(c.HTTP_CODE)

    if str(status) == '200':
        verified = True
    else:
        verified = False

    c.close()

return verified

Working with matplotlib and pycairo

2009-01-19T16:25:32-08:00

I spent a good part of my winter break working on learning Python and using it for projects. One project was the Yelp scraper that I posted about previously, and another was a report for my old work.

The report is a statistical analysis of the development of about 2,000 children aged three and four. For those interested, I’ll try to post it here once the final version is ready to go. In the past when making the report, I had been frustrated because there was no easy way to script the creation of the 30 or so charts that need to be made. Excel had been our data analysis tool, and as such, we were stuck with either using VBA to create charts, or to do it by hand. Since nobody knew VBA, we always just buckled down and did the work by hand.

This time around, I discovered the matplotlib Python library, and used that to create the charts. It was an pretty rough experience all in all. While simple graphs can be created in about five lines of code, creating complicated ones took a good amount of work. For example, to change the tick markers on a graph requires that you create tick objects, and then manipulate them each individually in a for loop. Granted, I couldn’t customize them at all in Excel, but figuring out that kind of change was a pain indeed.

The report itself required about 1,000 lines of code, and each chart required about 100-200 lines. For custom charts, I didn’t find the library that useful, however towards the end of the report there are 30 charts, all of which are identical, except for the data. For these charts, I was able to make a for loop that created them all in about 20 minutes, whereas previously these took me a few hours to make by hand.

Another library I spent some time learning was pycairo, which allows pixel by pixel editing of pictures. I had planned to use it to do any editing to the charts that I was unable to accomplish with the matplotlib library, but in the end, it was unnecessary. I have another project coming up though that will use the pycairo library, so look for that soon.

Yelp Scraper to Get Business Info in a Geographic Area

2008-12-21T16:41:13-08:00

I spent the past couple days on one of my first Python projects - using the Yelp API to compile a list of restaurants in a defined geographic area.

It’s been a good project. Because of some limitations of the API, I had to do some interesting tricks to make it work. One problem with the API is that it only allows 20 hits per query, so if you want to do a big query, you have to divide it up into tiny queries that have fewer than 20 hits each.

To accomplish that, if a query gets 20 hits within those two points, it will divide the longer dimension of the rectangle created by the points in half, and perform a query on each of those two new rectangles. For each of those, if there are 20 hits, it will again divide it in two and perform two new queries, and so forth until less than 20 hits are found for the rectangle. Once less than 20 hits are found, the data is entered into a database. Once all the points have been added to the database, a comma separated file is created, and the program ends.

It was pretty incredible switching to Python for this project from my usual Java, and also using an official API for the first time. This project ended up being about 200 lines (half of which are comments). I can’t imagine how long it would be with Java, since I used some rather powerful Python modules to accomplish this (namely, csv, urllib & json).

If anybody is interested in seeing/using the code, let me know. It should be useful if you need a list of restaurants or other businesses in a certain area. Worthy causes only please!

Dear God, This is a Terrible Interface

2008-12-13T19:17:04-08:00

The UI for a KDE Python IDE is about the worst I have ever seen:

That’s about 90 buttons.