Michael Jay Lissner

Some Thoughts on Celery

2014-11-01T00:00:00-07:00

We finally upgraded CourtListener last week and things went pretty well with the exception of two issues. First, we had some extended downtime as we waited for the database migration to complete. In retrospect, I should have realized that updating every item one row at a time would take a while. My bad.

Second, Celery broke again and that took me the better part of a day to detect and fix. As a central part of our infrastructure, this is really, truly frustrating. The remainder of this post goes into what happened, why it happened and how I finally managed to fix it.

Why?

First, why did this happen? Well…because I decided to log something. I created a task that processes our new audio files and I thought, “Hey, these should really log to the Juriscraper log rather than the usual celery log.” So, I added two lines to the file: One importing the log file and the second writing a log message. This is the little change that brought Celery to a grinding halt.

What the Hell?

If you’re wondering why logging would break an entire system, well, the answer is because Celery runs as a different user than everything else. In our case, as the celery user — a user that didn’t have permission to the log file I requested. Ugh.

Fine, that’s not so bad, but there were a number of other frustrating things that made this much worse:

The Celery init script that we use was reporting the following:

↪ sudo service celeryd start
celeryd-multi v3.0.13 (Chiastic Slide)
> Starting nodes...
    > w1.courtlistener.com: OK

But no, it was not starting “OK”. It was immediately crashing.

No log messages…anywhere. This appears to be because you have to detach stdin and stdout before daemonizing and according to asksol on IRC, this has been fixed in recent versions of Celery so even daemonizing errors can go to the Celery logs. Progress!
The collection of things that happens when celery starts is complicated:
1. I call sudo service celeryd start
2. service calls /etc/init.d/celeryd
3. celeryd does some stuff and calls celery.sh (another file altogether), where our settings are.
  
  Update: Apparently this is a CourtListener-specific customization, so this step probably won’t apply to you, but I have no idea where this wacky set up came from (it’s been in place for years).
4. Control is returned to celery, which starts celery itself with a command generated from celery.sh.
On top of this, there’s a celery binary and there’s a celery management command for Django. (Update the Django commands were removed in Celery 3.1. More progress!) celery --help prints out 68 lines of documentation. Not too bad, but many of those lines refer you to other areas of the documentation. For example, celery worker --help prints another 100 lines of help text. Jesus this thing is complicated.

Did I mention it has changing APIs?

I digress a bit, but the point here is that it fails silently, there are no log messages when it fails, and there’s no way to know which part of a complicated infrastructure is the problem. End rant.¹

Seeking Sanity

It took me a long time to figure out what was going wrong, but I did eventually figure it out. The process, in case you run into something similar, is to modify celeryd so it prints out the command that it eventually runs. At that point you’ll have the correct command. With that, you can run it as the celery user and with some luck you’ll see what the problem is. There’s a modified init script for this purpose, if you like.

Other tips:

If you have a new enough version of Celery, there are some troubleshooting tips that should help. They did nothing for me, because I haven’t upgraded yet for fear of the changing APIs.
There seem to be a handful of different command line flags that Celery can use to be sent to the background. You’ll need to disable these when you’re testing or else you won’t see error messages or anything (apparently?).

Moving Forward

So, I feel bad: I’ve ranted a good deal about Celery, but I haven’t proposed any solutions. It looks like a lot of things have been improved in recent versions of Celery, so part of the solution is likely for us to upgrade.

But this isn’t the first time I’ve spent a long time trying to make Celery work, so what other ideas it take to make Celery a less complicated, more reliable tool?

The ideas I’ve come up with so far are:

More documentation for installation and set up troubleshooting with the possibility of a wiki.
- But already I rant about how much documentation it has.
A simpler interface that eliminates a number of edge uses.
- But I have no idea what, if anything, can be eliminated.
Support for fewer task brokers.
- But I use RabbitMQ and am considering switching to Redis.
A more verbose, more thorough debug mode.
- But apparently this is already in place in the latest versions?
Let Celery run as the www-data user as a general practice?
- But apparently that’s a bad idea.
  
  Update this is a bad idea in general, but it’s not particularly bad if you don’t expose Celery on the network. If you’re only running it locally, you can probablly get by with Celery as a www-data user or similar.

As you can tell, I don’t feel strongly that any of these are the right solution. I am convinced though that Celery has a bad smell and that it’s ripe for a leaner solution to fill some of its simpler use cases. I’m currently considering switching to a simpler task queue, but I don’t know that I’ll do it since Celery is the de-facto one for Django projects.

We deserve a good, simple, reliable task queue though, and I wonder if there are good ideas for what could be changed in Celery to make that possible. I, for one, would love to never spend another minute trying to make RabbitMQ, Celery and my code play nicely together.

In truth Celery is a classic love/hate relationship. On the one hand, it evokes posts like this one, but on the other, it allows me to send tasks to a background queue and distribute loads among many servers. Hell, it’s good enough for Instagram. On the other hand, god damn it, when it fails I go nuts. ↩

Every Now and Again, I Rant About iPods

2009-05-24T23:34:44-07:00

Like the title says, every now and again, I have to rant about how annoying iPods (and by extension, Apple) are. This is one of those times.

The picture below represents the folder hierarchy of my 20GB iPod, as displayed in the Ubuntu Disk Analyzer:

Right now, this iPod is nearly full. I filled it with a very normal hodge-podge of songs. Naturally, I like some artists better than others, and so, I have more of their songs. The Beatles and Zeppelin, for example, I have a lot of. A-Ha, and Four Non Blonds, not so much. When this is the case, I’ll have a folder filled with lots of songs - a big folder. If you look at the image above though, you’ll see that my iPod is organized into 50 equal directories, named r1 to r50.

Now, why would Apple do this? Why would they take perfectly organized songs in neat and tidy folders and then split them into 50 folders with meaningless names? Because that makes it more of a pain in the butt to figure out what music is on the iPod, and in theory makes it harder to share the songs. For the copyright violators, this should slow them down.

For the rest of the world, it’s a pain, because rather than browsing our iPod like we would a normal music player, instead we must use iTunes, or, if we use Linux, reverse engineer the thing, which invariably doesn’t work that well.

Apple could have made a device that worked normally on all computers, like everybody else did, but instead they have locked us into iTunes, a weird directory structure, and a system designed to be as obscure as possible. Not cool.

Cheney is so shady, it pisses me off.

2008-12-19T21:35:28-08:00

Cheney is at work to keep his records from becoming public as part of the National Archives when he goes out of office.

The vice president alone may determine what constitutes vice presidential records or personal records, how his records will be created, maintained, managed and disposed, and are all actions that are committed to his discretion by law.

I can think of no reason why this would be necessary unless he’s the shadiest person ever in office. Oh, right.

As an ex-government worker, I can testify to how ridiculous this is: It’s really fucking ridiculous.

I Am Me, Dammit!

2008-09-15T15:00:30-07:00

Well, it happened. The bastards stole my identity. They somehow got my credit card number, address, phone number, etc, took them, and put them to use. Grand total losses: $309. Not too bad, considering.

I’ve called my credit card company, and they promise to reimburse me for my losses. This is when I thank the federal government for making them do this. Thank you feds!

I’m thinking about how this happened, and I have a theory. I am about 95% sure this happened as a result of shopping for car parts online. One of the retailers I went to (lost to me now) decided to share my contact information with a number of “partners” so that a part I was looking for could be found more easily. I ended up getting the part, but I think they might have shared more info than they should have. Arg.

After all of the services that credit cards have tried to sell me over the years to protect me from this kind of thing, I’m happy to say that I’m fine without them. It’s really not the end of the world…it’s annoying though, yes.

Amazon Has mp3’s! I Wish I Could Download Them…

2008-01-22T19:04:40-08:00

It made some headlines, and is about to make more during superbowl. Amazon has started selling mp3’s from all four of the major record labels. It’s a pretty cool revelation, so I went ahead and checked it out.

Guess what? Linux users, you are screwed. In order to download albums from Amazon, you must “Get the Amazon mp3 downloader”…which is not available for Linux. Excuse me? Say what? I have downloaded literally hundreds upon hundreds of files from the Internet, and never have I needed a special tool other than Firefox. Why now?

Amazon, if you’re going to do something great like this, do it whole hog. Don’t make me install junkware. After all, I’m choosing to give you money, right?

An Important Distinction

2007-11-26T17:02:25-08:00

For the record people:

Quite: I haven’t quite finished my soup. This is quite a nice restaurant.
Quiet: This is a quiet restaurant. I enjoy peace and quiet.

We recently looked for roommates, and we subtracted points from anybody that wrote in their response email the word “quite” where they meant to say “quiet.” It was extraordinary how many applicant roommates think they are “a quite person”.