Michael Jay Lissner

Editing a File on Github

2014-10-06T00:00:00-07:00

When writing programs, developers have a choice of whether they want their work to be public or private. Programs that are made public are called “open source” and ones that are not are called “closed source”. In both cases the developer can share a program with the world as a website or iPhone app, or whatever, but in the case where the code is shared publicly it’s also possible for anybody anywhere in the world to change the program to make it better. (For more detail on this and other jargon, see the definitions at the end.)

This is very cool!

But I hear you asking, “How do I, a non-developer, make use of this system to make the world a better place?” I’m glad you asked — this article is for you.

And then there was Git

Git is an extremely popular system that developers use to keep track of the code they write. The main thing it does is make it so that two developers can work on the same file, track their individual changes and then combine their work, as you might do in Microsoft Word. Since all programs are just collections of lots of files that are together known as a “repository”, this lets a number of developers work together without tramping on each others changes.

There are a million ways to use Git but lately a lot of people use Git through a website called Github. Github makes it super-easy to use Git, but you still need to understand a few steps that are necessary to make changes. The basic steps we’ll take are:

You: Find the file
You: Change the file and save your changes
You: Create a pull request
The manager (me or somebody else): Merges the pull request, making your changes live

For the purpose of this article, I’ve created a new repository as a playground where you can try this out.

The playground is here: https://github.com/mlissner/git-tutorial/

Go check out the playground and create a Github account, then come back here and continue to the next step, changing a file.

Make your change

Like the rest of this, the process of making a change is actually pretty easy. All you have to do is find the file, make your change, and then save it. So:

Find the file

When you look at the playground, you’ll see a bunch of files like this:

Click the file you want to edit. In this case, it’s we’ll actually be changing file called “your-name.txt”. Click it.

Once you do that, you’ll see the contents of the file — a list of names, mine at the top — and you’ll see a pencil that lets you edit the file.

Click the pencil!

Change the file

At this point you’ll see a message saying something like:

You are editing a file in a project you do not have write access to. We are forking this project for you (if one does not yet exist) to write your proposed changes to. Submitting a change to this file will write it to a new branch in your fork so you can send a pull request.

Groovy. If you ignore both the jargon and the bad grammar, you can go ahead and add your name to the bottom of the file, and then you’ll see two fields at the bottom that you can use to explain your change:

This is like an email. The first field is the subject of your change, something brief and to the point. The second field lets you flesh out in more detail what you did, why it’s useful, etc. In many cases — like simply adding your name to this file — your changes are obvious and you can just hit the big green “Propose file change” button.

Let’s press the big green button, shall we?

Send a “pull request”

At this point you’ll see another form with another somewhat cryptic message:

The change you just made was written to a new branch in your fork of this project named patch-1. If you’d like the author of the original project to merge these changes, submit a pull request.

I think the important part of that message is the second sentence:

If you’d like the author of the original project to merge these changes, submit a pull request.

Ok, so how do you do that? Well, it turns out that the page we’re looking at is very similar to the one we were just on. It has two fields, one for a subject and one for a comment. You can fill these out, but if it’s a simple change you don’t need to, and anyway, if you put stuff on the last page it’ll just be copied here already.

So: Press the big green button that says “Create pull request”.

You’re now done, but what did you do, exactly?

Let’s parse what’s happened so far

At this point, you’ve found a file, changed it, and submitted a pull request. Along the way, the system told you that it was “forking this project for you” and that your changes were, “written to a new branch in your fork of this project”.

Um, what?

The most amazing thing that Git does is allow many developers to work on the same file at the same time. It does this by creating what it calls forks and branches. For our purposes these are basically the same thing. The idea behind both is that every so often people working on a file save a copy of the entire repository into what’s called a commit. A commit is a copy of the code that is saved forever so anybody can travel back in time and see the code from two weeks ago or a month ago or whatever. 95% of any Git repository is just a bunch of these copies, and you actually created one when you saved your changes to the file.

This is super useful on its own, but when somebody forks or branches the repository, what they do is say, “I want a perfect copy of all the old stuff, but from here on, I’m going my own way whenever I save things.” Over time, everybody working in the repository does this, creating their own work in their own branches, and amazingly, one person’s work doesn’t interfere with another’s.

Later, once somebody thinks that their work is good enough to share with everybody, they create what’s called a “Pull Request”, just like you did a moment ago, and the owner of the repository — in this case, me — gets an email asking him or her to “pull” the code into the main repository and “merge” the changes into the files that are there. Once this is done, everybody gets those changes from then on.

It’s a brilliant system.

My turn: Merging the pull request

When you created that pull request a moment ago, you actually sent me an email and now you have to wait for me to do something. Eventually, I’ll get your email, and when I do I’ll go to Github and see a screen like this:

I’ll probably make a comment saying thank you, and then I’ll press the Big Green Button that says, “Merge pull request”.

This will merge your changes into mine and we’ll both go about our merry way. Mission accomplished!

Why this works so well

This system is pretty amazing and it works very well for tiny little projects and massive ones alike (for example, some projects have thousands of active forks). What’s great about this system is that it allows anybody to do whatever they want in their fork without requiring any permission from the owner of the code. Anybody can do whatever they want in their fork and I’m happy to see them experimenting. That work will never affect me until they issue a pull request and I merge it in, accepting their proposed changes.

This process mirrors a lot of real world processes between writers and editors, but solidifies and equalizes it so that there’s a right way to do things and so that nobody can cause any trouble. The process itself can be a little overwhelming at first, with lots of jargon and steps, but once you get it down, it’s smooth and quick and works very well.

As you might expect, there are tons of resources about this on the Web. Some really good ones are at Github and there are even entire online books going into these topics. Like all things, you can go as deep as you want, but the above should give you some good basics to get you started.

Some Definitions

Open vs Closed Source: This is a topic entire theses and books have been written about, but in general open source is way of creating a program where a developer shares all of their code so anybody can see it. In general when a program is open source, people are welcome to edit the code, help file and fix bugs, etc. On the other hand, closed source development is a way of creating a program so that only the developers can see the code, and the public at large is generally not welcome to contribute, except to sometimes email the developer with comments.

In a way, the product of open source development is a combination of the code itself plus the program it creates, while in closed source projects the product is the program alone. There are thousands of examples of each of these ways of developing software. For example, Android and the Linux Kernel are open source, while Microsoft Word and iPhones are not. (See how I couldn’t link to the latter two?)
Repository: A collection of files, images, and other stuff that are kept together for a common purpose. Generally it’s a bunch of files that create a website or program, but some people use repositories for all kinds of things, like dealing with identity theft (shameless plug), holding the contents of this very webpage (shameless plug), or even writing online books teaching lawyers to code (not a shameless plug!).
Pull request: A polite way to say, “This code is ready to get included in the main repository. Please pull it in.”
Merging: The process of taking a branch or fork and merging the changes in it into another branch or fork. This combines two people’s work into a single place.

How to help end Boy Scouts of America’s ban on gays

2013-04-19T10:18:05-07:00

On May 3rd, the Boy Scouts are considering lifting their ban on gays, and are putting a vote to the local and national councils. This means that it’s easy to influence the vote by calling in and expressing your opinion. It’s simple to do so, and more voices could change the direction of the Boy Scouts of America, allowing all boys to be included and accepted.

Here’s what to do:

Find your local council
Give them a call or send them an email telling them your opinion.
Contact the national council via email: feedback@scouting.org.

Most important is to contact your local council. We need their phones to be ringing off the hook with people expressing their opinions. It truly takes no more than two minutes. Here’s the San Diego council: (619) 298-6121, and the East Bay council: (925) 674-6100.

If you prefer a form letter, you can just do this one through the Human Rights Campaign (34,000 people already have).

Here’s a simple transcript for you to follow:

I’m [calling/wrriting] to express my desire that the Boy Scouts immediately lift their ban on gays. This ban is discriminatory, outdated and pointless. Scouting teaches many great lessons to thousands of adolescents across the U.S. One lesson that Scouting should not teach is that homosexuality is somehow wrong or means for discrimination.

On May 3rd, I hope that you will help the BSA finally makes the right decision so it can continue to lead boys into being mature and accepting men.

Send this to the email above, and call your local council. This could make a difference.

Adding New Fonts to Tesseract 3 OCR Engine

2014-10-06T00:00:00-07:00

Status

I’m attempting to keep this up to date as Tesseract changes. If you have corrections, please send them directly using the contact page.

I’ve turned off commenting on this article because it was just a bunch of people asking for help and never getting any. If you need help with these instructions, go to Stack Overflow and ask there. If you send me a link to a question on such a site, I’m much more likely to respond positively.

Introduction

Tesseract is a great and powerful OCR engine, but their instructions for adding a new font are incredibly long and complicated. At CourtListener we have to handle several unusual blackletter fonts, so we had to go through this process a few times. Below I’ve explained the process so others may more easily add fonts to their system.

The process has a few major steps:

Create training documents
Teach Tesseract about the documents

Create training documents

To create training documents, open up MS Word or LibreOffice, paste in the contents of the attached file named ‘standard-training-text.txt’. This file contains the training text that is used by Tesseract for the included fonts.

Set your line spacing to at least 1.5, and space out the letters by about 1pt. using character spacing. I’ve attached a sample doc too, if that helps. Set the text to the font you want to use, and save it as font-name.doc.

Save the document as a PDF (call it [lang].font-name.exp0.pdf, with lang being an ISO-639 three letter abbreviation for your language), and then use the following command to convert it to a 300dpi tiff (requires imagemagick):

convert -density 300 -depth 4 lang.font-name.exp0.pdf lang.font-name.exp0.tif

You’ll now have a good training image called lang.font-name.exp0.tif. If you’re adding multiple fonts, or bold, italic or underline, repeat this process multiple times, creating one doc → pdf → tiff per font variation.

Train Tesseract

The next step is to run tesseract over the image(s) we just created, and to see how well it can do with the new font. After it’s taken its best shot, we then give it corrections. It’ll provide us with a box file, which is just a file containing x,y coordinates of each letter it found along with what letter it thinks it is. So let’s see what it can do:

tesseract lang.font-name.exp0.tiff lang.font-name.exp0 batch.nochop makebox

You’ll now have a file called font-name.exp0.box, and you’ll need to open it in a box-file editor. There are a bunch of these on the Tesseract wiki. The one that works for me (on Ubuntu) is moshpytt, though it doesn’t support multi-page tiffs. If you need to use a multi-page tiff, see the issue on the topic for tips. Once you’ve opened it, go through every letter, and make sure it was detected correctly. If a letter was skipped, add it as a row to the box file. Similarly, if two letters were detected as one, break them up into two lines.

When that’s done, you feed the box file back into tesseract:

tesseract eng.font-name.exp0.tif eng.font-name.box nobatch box.train .stderr

Next, you need to detect the Character set used in all your box files:

unicharset_extractor *.box

When that’s complete, you need to create a font_properties file. It should list every font you’re training, one per line, and identify whether it has the following characteristics: <fontname> <italic> <bold> <fixed> <serif> <fraktur>

So, for example, if you use the standard training data, you might end up with a file like this:

eng.arial.box 0 0 0 0 0
eng.arialbd.box 0 1 0 0 0
eng.arialbi.box 1 1 0 0 0
eng.ariali.box 1 0 0 0 0
eng.b018012l.box 0 0 0 1 0
eng.b018015l.box 0 1 0 1 0
eng.b018032l.box 1 0 0 1 0
eng.b018035l.box 1 1 0 1 0
eng.c059013l.box 0 0 0 1 0
eng.c059016l.box 0 1 0 1 0
eng.c059033l.box 1 0 0 1 0
eng.c059036l.box 1 1 0 1 0
eng.cour.box 0 0 1 1 0
eng.courbd.box 0 1 1 1 0
eng.courbi.box 1 1 1 1 0
eng.couri.box 1 0 1 1 0
eng.georgia.box 0 0 0 1 0
eng.georgiab.box 0 1 0 1 0
eng.georgiai.box 1 0 0 1 0
eng.georgiaz.box 1 1 0 1 0
eng.lincoln.box 0 0 0 0 1
eng.old-english.box 0 0 0 0 1
eng.times.box 0 0 0 1 0
eng.timesbd.box 0 1 0 1 0
eng.timesbi.box 1 1 0 1 0
eng.timesi.box 1 0 0 1 0
eng.trebuc.box 0 0 0 1 0
eng.trebucbd.box 0 1 0 1 0
eng.trebucbi.box 1 1 0 1 0
eng.trebucit.box 1 0 0 1 0
eng.verdana.box 0 0 0 0 0
eng.verdanab.box 0 1 0 0 0
eng.verdanai.box 1 0 0 0 0
eng.verdanaz.box 1 1 0 0 0

Note that this is the standard font_properties file that should be supplied with Tesseract and I’ve added the two bold rows for the blackletter fonts I’m training. You can also see which fonts are included out of the box.

We’re getting near the end. Next, create the clustering data:

mftraining -F font_properties -U unicharset -O lang.unicharset *.tr 
cntraining *.tr

If you want, you can create a wordlist or a unicharambigs file. If you don’t plan on doing that, the last step is to combine the various files we’ve created.

To do that, rename each of the language files (normproto, Microfeat, inttemp, pffmtable) to have your lang prefix, and run (mind the dot at the end):

combine_tessdata lang.

This will create all the data files you need, and you just need to move them to the correct place on your OS. On Ubuntu, I was able to move them to;

sudo mv eng.traineddata /usr/local/share/tessdata/

And that, good friend, is it. Worst process for a human, ever.

Enclosures

Installing Tracker from Source

2009-02-23T00:13:40-08:00

I’ve been working over the past several weeks on getting Tracker to work better on my system. There are a couple reasons that I’m doing this. The first is that by default on Ubuntu, Tracker doesn’t support a number of meta formats (such as the tags in JPEGs, ID3 info in MP3s, and the like). The second was that the RDF parsing code in the default Ubuntu version is a bit buggy, and the new version is better. It’s been a bit of a pain figuring out the install process, so I figured I’d post here so others might have an easier time.

The online instructions say to simply download the code, and to install it. No big deal, right? Well…in reality, it’s a bit harder than that. The process I went through was to download the source from here per the instructions, unpack the source files, and to run the configure command.

After the configure command is run each time, it will give you a summary of which components will be installed, and which will not. If you have all the dependencies necessary, and include a couple of arguments to the configure command, everything will get installed. If not, certain pieces will be missing.

The list of dependencies is a bit long, so before you run ./configure, you might as well install them. To do so, run:

sudo aptitude install libgmime-2.0-2a libgmime-2.0-2-dev dbus-glib-1-dev libdbus-glib-1-dev libhal-dev libhal-storage-dev sqlite3-dev libsqlite3-dev libexif-dev libdeskbar-tracker libgsf-1-dev libjpeg62-dev libtiff4-dev libxine-dev libpoppler-dev libgstreamer0.10-dev libpoppler-glib-dev libtotem-plparser-dev libunac1-dev libexempi-dev libraptor1-dev libtracker-gtk-dev libgnome-desktop-dev libgnome-desktop-dev libnotify-dev

Once those are installed, run:

./configure --enable-deskbar-applet --enable-tracker-applet --prefix=/usr --sysconfdir=/etc

After this, it should say that pretty much everything will be installed. If so, you can proceed to the commands below, and once those are complete, the latest version should be installed with full functionality.

make
sudo make install

If after running your configure script (./configure) it doesn’t indicate that everything will be installed, put in a comment below, and we’ll see what we can do.