Script to Clean Up F-Spot Database

One of the more popular photo management applications for Linux is f-spot, but unfortunately it has a rather glaring bug. It uses a sqlite3 database internally to track which pictures you've imported into it, what tags they have, etc. However, if you delete, move or rename any of the files that f-spot is tracking, the next time you browse to that photo within f-spot, it will crash the program. It's annoying, and there's no particularly easy way to deal with it...until now.

The script below (and attached) simply iterates through all of the photos that are in f-spot's database, and checks to see if those photos exist on your hard drive. If you run it in demo-mode, it will show you which files look problematic, and if you run it in normal mode, it will delete those database entries so that your database is cleaned up (but not before backing up your database, just in case).

In my very limited testing, it works very well, any additional feedback or bug reporting is more than welcome.

#!/bin/bash
 
# A script to find missing files in the f-spot database, and then delete them.
# At present these files crash f-spot. It's frustrating as all hell.
 
echo "Welcome to the f-spot database cleaner. All the usual disclaimers apply, as you might imagine."
echo "What would you like to do: " 
echo "  1) Run in demo-mode " 
echo "  2) Clean up your f-spot database" 
echo "  3) Quit" 
read -p "Your choice: " choice
 
case $choice in 
    1) demomode="true";;
    2) demomode="false";;
    3) exit 0;;
esac
 
# With that beginning stuff out of the way, let us do some functions
# First, a function to gather the database contents and to print out the ones that are orphans
 
function findAndFixOrphans {
    # find our db, and set a var. Checking for XDG path first, since it's the more recent location of the db
    if [ -f $XDG_CONFIG_DIR/f-spot/photos.db ] #checks if the $XDG_CONFIG_DIR variable is in use
    then
        DBPATH=$XDG_CONFIG_DIR/f-spot/photos.db
    elif [ -f $HOME/.config/f-spot/photos.db ] #uses the default $XDG location, if that's being used.
    then
        DBPATH=$HOME/.config/f-spot/photos.db
    elif [ -f $HOME/.gnome2/f-spot/photos.db ] #uses the old location of the DB, if the former aren't in use.
    then
        DBPATH=$HOME/.gnome2/f-spot/photos.db
    else
        echo "Error: Could not find database. Damn." 
        exit 1
    fi
 
    # Select the filenames, and put them in a variable.
    filenames=$(sqlite3 $DBPATH "SELECT URI FROM PHOTOS")
    filenames_versions=$(sqlite3 $DBPATH "SELECT URI FROM PHOTO_VERSIONS")
 
    # Chomp off the first instance of file://, and replace the rest with newlines.
    filenames=$(echo $filenames | sed 's/file:\/\///' | sed 's/file:\/\//\n/g'  )
    filenames_versions=$(echo $filenames_versions | sed 's/file:\/\///' | sed 's/file:\/\//\n/g' )
 
    if [ $demomode == "true" ]
    then            
        while read -r line
        do
            # Decode the filename
            decodedLine=$(echo -e "${line//\%/\\x}")
            if [ ! -f  "$decodedLine" ] 
            then
                # If the file doesn't exist, we output the filename, if in demomode, or we fix it if we are not in demomode.
                echo  "Errant record found in the photos table: $decodedLine"
            fi
        done <<< "$filenames"
 
        # We do the same for the photo_versions table
        while read -r line
        do
            # Decode filename
            decodedLine=$(echo -e "${line//\%/\\x}")
            if [ ! -f "$decodedLine" ]
            then
                # If the file doesn't exist, we output the filename
                echo "Errant record found in the photo_versions table: $decodedLine"
            fi
        done <<< "$filenames_versions"
 
    else
        # We backup the database, and make the correction
        cp $DBPATH $DBPATH.`date -I`.bak
        if [ $? -eq 0 ]
        then
            #The backup worked, tell the user.
            echo "Your database has been backed up to $DBPATH.`date -I`.bak"
        else
            echo "Error backing up your database."
            exit 3
        fi
 
        # First we do the photos table
        while read -r line
        do
            # Decode the filename
            decodedLine=$(echo -e "${line//\%/\\x}")
 
            if [ ! -f "$decodedLine" ]
            then
                # Do some sql here.
                foo="file://${line}"
                echo -n "Deleting URI $line from the database table photos..."
                sqlite3 $DBPATH "DELETE FROM PHOTOS WHERE uri = '$foo'"
                echo "done."
            fi
        done <<< "$filenames"
 
        # Then we do the photo_versions table
        while read -r line
        do
            # Decode the filename
            decodedLine=$(echo -e "${line//\%/\\x}")
 
            if [ ! -f "$decodedLine" ]
            then
                #Do some sql stuff
                foo="file://${line}"
                echo -n "Deleting URI $line from the database table photo_versions..."
                sqlite3 $DBPATH "DELETE FROM PHOTO_VERSIONS WHERE uri = '$foo'"
                echo "done."
            fi
        done <<< "$filenames_versions"
 
    fi
 
}
 
if [ "$demomode" == "true" ]
then
    echo "Great. Proceeding in demomode."
 
    findAndFixOrphans
 
    echo "Demomode successfully finished. Exiting."
    exit 0;
elif [ "$demomode" == "false" ]
then
    echo "Great. Cleaning up your database."
 
    findAndFixOrphans    
 
    echo "Database cleaned successfully."
    exit 0;
else
    echo "Something strange happened. See the script for details. Exiting."
    exit 2;
fi

AttachmentSize
fspot-cleaner.txt4.55 KB

For the new f-spot database you need to change some lines: You have to change all references to "uri" to "base_uri || filename" that has to be done for the SQL Select statements and for the Delete statements.

Yeah, I noticed this just yesterday, but haven't made the fix in the script yet. To handle it properly, I'll need to start checking which version of the DB is being used, and script accordingly. For now though, it seems like your solution should work.

On the upside, missing/moved photos aren't crashing the thing anymore.

I tried to use the script (with the proposed modification) but got following error:

Welcome to the f-spot database cleaner. All the usual disclaimers apply, as you might imagine.
What would you like to do:
1) Run in demo-mode
2) Clean up your f-spot database
3) Quit
Your choice: 1
fspot-cleaner.txt: 22: function: not found
fspot-cleaner.txt: 57: Syntax error: redirection unexpected

I'm a newbie in linux ... do you have an idea where the problem might be ?
(I'm using Ubuntu karmic)

Thanks,
Boris

I found out ... using sh command I was calling dash (on Ubuntu) and not bash ... so using /bin /bash works!

thank you
boris

Awesome. Good to hear it.

Hi, very interesting work! Thanks for sharing.
Bad news is it isn't working on my Lucid with F-Spot 0.6.1.5. Maybe they changed database structure.
Here is the output:

What would you like to do:
1) Run in demo-mode
2) Clean up your f-spot database
3) Quit
Your choice: 2
Great. Cleaning up your database.
Error: no such column: URI
Error: no such column: URI
Your database has been backed up to /home/stefano/.config/f-spot/photos.db.2010-05-05.bak
Deleting URI from the database table photos...Error: no such column: uri
done.
Deleting URI from the database table photo_versions...Error: no such column: uri
done.

Yeah, they changed the DB model. If you want to update the script, let me know via the contact form, and I'll share anything with you that you need to know. I don't have time to keep it up to date though, sadly.

Hi,

thanks a lot for your piece of software, unfortunately as you said it seems that they changed the database model. I think we all understand that you do not have time for fixing, however do you have any general suggestions? I peeked at the new DB in lucid 10.04, admit I do not know very well sqlite3, however I guess the model is similar, maybe few substitutions on your shell will be enough...? If you can only post few lines with guidelines, some of us can fix it and post it here for you.

Thanks anyway,

Greg

The relevant lines are these:

filenames=$(sqlite3 $DBPATH "SELECT URI FROM PHOTOS")
filenames_versions=$(sqlite3 $DBPATH "SELECT URI FROM PHOTO_VERSIONS")

Basically, they need to be updated so that the select statements point to the correct places in the new DBs, I think.

If you get it to work, please post notes! If you need more help, use the contact form, so we can sort it out offline.

# Select the filenames, and put them in a variable.
filenames=$(sqlite3 $DBPATH "SELECT BASE_URI||FILENAME FROM PHOTOS")
filenames_versions=$(sqlite3 $DBPATH "SELECT BASE_URI||FILENAME FROM PHOTO_VERSIONS")

If there is interest, this should be spun into its own repository, and ultimately recreated as an add-on for f-spot proper. I'm game to put it in a repo, and manage patches, but I don't have time to develop it or keep it up to date.

Please express interest here if you can also contribute in some way.

Here's the output of diff showing the changes I made for this to work for me (on Lucid):
uu@pc ~> diff f-spot_photo_cleanup_original.sh f-spot_photo_cleanup.sh
41,42c41,44
< filenames=$(sqlite3 $DBPATH "SELECT URI FROM PHOTOS")
< filenames_versions=$(sqlite3 $DBPATH "SELECT URI FROM PHOTO_VERSIONS")
---
> ## filenames=$(sqlite3 $DBPATH "SELECT URI FROM PHOTOS")
> ## filenames_versions=$(sqlite3 $DBPATH "SELECT URI FROM PHOTO_VERSIONS")
> filenames=$(sqlite3 $DBPATH "SELECT BASE_URI||FILENAME FROM PHOTOS")
> filenames_versions=$(sqlite3 $DBPATH "SELECT BASE_URI||FILENAME FROM PHOTO_VERSIONS")
96c98,99
< sqlite3 $DBPATH "DELETE FROM PHOTOS WHERE uri = '$foo'"
---
> ##sqlite3 $DBPATH "DELETE FROM PHOTOS WHERE uri = '$foo'"
> sqlite3 $DBPATH "DELETE FROM PHOTOS WHERE base_uri || filename = '$foo'"
112c115,116
< sqlite3 $DBPATH "DELETE FROM PHOTO_VERSIONS WHERE uri = '$foo'"
---
> ##sqlite3 $DBPATH "DELETE FROM PHOTO_VERSIONS WHERE uri = '$foo'"
> sqlite3 $DBPATH "DELETE FROM PHOTO_VERSIONS WHERE base_uri || filename = '$foo'"

Steve, this is very cool, but can you update it to work on various versions of the DB? It would be awesome if it inspected what version you had, and then did the appropriate deletions. Failing that, this does look like a great solution.

Hiya,

Here's a version that tests the db. It's a lazy version ie I haven't looked at the db structure.
Quick 'n Dirty as they call it.

Enjoy!

#!/bin/bash

# A script to find missing files in the f-spot database, and then delete them.
# At present these files crash f-spot. It's frustrating as all hell.

echo "Welcome to the f-spot database cleaner. All the usual disclaimers apply, as you might imagine."
echo "What would you like to do: "
echo " 1) Run in demo-mode "
echo " 2) Clean up your f-spot database"
echo " 3) Quit"
read -p "Your choice: " choice

case $choice in
1) demomode="true";;
2) demomode="false";;
3) exit 0;;
esac

# With that beginning stuff out of the way, let us do some functions
# First, a function to gather the database contents and to print out the ones that are orphans

function findAndFixOrphans {
# find our db, and set a var. Checking for XDG path first, since it's the more recent location of the db
if [ -f $XDG_CONFIG_DIR/f-spot/photos.db ] #checks if the $XDG_CONFIG_DIR variable is in use
then
DBPATH=$XDG_CONFIG_DIR/f-spot/photos.db
elif [ -f $HOME/.config/f-spot/photos.db ] #uses the default $XDG location, if that's being used.
then
DBPATH=$HOME/.config/f-spot/photos.db
elif [ -f $HOME/.gnome2/f-spot/photos.db ] #uses the old location of the DB, if the former aren't in use.
then
DBPATH=$HOME/.gnome2/f-spot/photos.db
else
echo "Error: Could not find database. Damn."
exit 1
fi

filenames=$(sqlite3 $DBPATH "SELECT URI FROM PHOTOS" 1>/dev/null 2>/dev/null)
if [ -z "$filesnames" ]; then version=v3; else version=v2; fi
echo $version

# Select the filenames, and put them in a variable.
if [ "$version" == "v3" ]
then
filenames=$(sqlite3 $DBPATH "SELECT BASE_URI||FILENAME FROM PHOTOS")
filenames_versions=$(sqlite3 $DBPATH "SELECT BASE_URI||FILENAME FROM PHOTO_VERSIONS")
else
filenames=$(sqlite3 $DBPATH "SELECT URI FROM PHOTOS")
filenames_versions=$(sqlite3 $DBPATH "SELECT URI FROM PHOTO_VERSIONS")
fi

# Chomp off the first instance of file://, and replace the rest with newlines.
filenames=$(echo $filenames | sed 's/file:\/\///' | sed 's/file:\/\//\n/g' )
filenames_versions=$(echo $filenames_versions | sed 's/file:\/\///' | sed 's/file:\/\//\n/g' )

if [ $demomode == "true" ]
then
while read -r line
do
# Decode the filename
decodedLine=$(echo -e "${line//\%/\\x}")
if [ ! -f "$decodedLine" ]
then
# If the file doesn't exist, we output the filename, if in demomode, or we fix it if we are not in demomode.
echo "Errant record found in the photos table: $decodedLine"
fi
done <<< "$filenames"

# We do the same for the photo_versions table
while read -r line
do
# Decode filename
decodedLine=$(echo -e "${line//\%/\\x}")
if [ ! -f "$decodedLine" ]
then
# If the file doesn't exist, we output the filename
echo "Errant record found in the photo_versions table: $decodedLine"
fi
done <<< "$filenames_versions"

else
# We backup the database, and make the correction
cp $DBPATH $DBPATH.`date -I`.bak
if [ $? -eq 0 ]
then
#The backup worked, tell the user.
echo "Your database has been backed up to $DBPATH.`date -I`.bak"
else
echo "Error backing up your database."
exit 3
fi

# First we do the photos table
while read -r line
do
# Decode the filename
decodedLine=$(echo -e "${line//\%/\\x}")

if [ ! -f "$decodedLine" ]
then
# Do some sql here.
foo="file://${line}"
echo -n "Deleting URI $line from the database table photos..."
if [ "$version" == "v3" ]
then
sqlite3 $DBPATH "DELETE FROM PHOTOS WHERE base_uri||filename = '$foo'"
else
sqlite3 $DBPATH "DELETE FROM PHOTOS WHERE uri = '$foo'"
fi
echo "done."
fi
done <<< "$filenames"

# Then we do the photo_versions table
while read -r line
do
# Decode the filename
decodedLine=$(echo -e "${line//\%/\\x}")

if [ ! -f "$decodedLine" ]
then
#Do some sql stuff
foo="file://${line}"
echo -n "Deleting URI $line from the database table photo_versions..."
if [ "$version" == "v3" ]
then
sqlite3 $DBPATH "DELETE FROM PHOTO_VERSIONS WHERE base_uri||filename = '$foo'"
else
sqlite3 $DBPATH "DELETE FROM PHOTO_VERSIONS WHERE uri = '$foo'"
fi
echo "done."
fi
done <<< "$filenames_versions"

fi

}

if [ "$demomode" == "true" ]
then
echo "Great. Proceeding in demomode."

findAndFixOrphans

echo "Demomode successfully finished. Exiting."
exit 0;
elif [ "$demomode" == "false" ]
then
echo "Great. Cleaning up your database."

findAndFixOrphans

echo "Database cleaned successfully."
exit 0;
else
echo "Something strange happened. See the script for details. Exiting."
exit 2;
fi

Since some (new?) base_uri entries in the database have no ending "/" this script may corrupt your database.

FIX:
Replace all
"base_uri||filename" with "base_uri||'/'||filename"
(It's not problem that some paths now have double slashs in it. Script works fine.)

Updated script by BBB:


#!/bin/bash

# A script to find missing files in the f-spot database, and then delete them.
# At present these files crash f-spot. It's frustrating as all hell.

echo "Welcome to the f-spot database cleaner. All the usual disclaimers apply, as you might imagine."
echo "What would you like to do: "
echo " 1) Run in demo-mode "
echo " 2) Clean up your f-spot database"
echo " 3) Quit"
read -p "Your choice: " choice

case $choice in
1) demomode="true";;
2) demomode="false";;
3) exit 0;;
esac

# With that beginning stuff out of the way, let us do some functions
# First, a function to gather the database contents and to print out the ones that are orphans

function findAndFixOrphans {
# find our db, and set a var. Checking for XDG path first, since it's the more recent location of the db
if [ -f $XDG_CONFIG_DIR/f-spot/photos.db ] #checks if the $XDG_CONFIG_DIR variable is in use
then
DBPATH=$XDG_CONFIG_DIR/f-spot/photos.db
elif [ -f $HOME/.config/f-spot/photos.db ] #uses the default $XDG location, if that's being used.
then
DBPATH=$HOME/.config/f-spot/photos.db
elif [ -f $HOME/.gnome2/f-spot/photos.db ] #uses the old location of the DB, if the former aren't in use.
then
DBPATH=$HOME/.gnome2/f-spot/photos.db
else
echo "Error: Could not find database. Damn."
exit 1
fi

filenames=$(sqlite3 $DBPATH "SELECT URI FROM PHOTOS" 1>/dev/null 2>/dev/null)
if [ -z "$filesnames" ]; then version=v3; else version=v2; fi
echo $version

# Select the filenames, and put them in a variable.
if [ "$version" == "v3" ]
then
filenames=$(sqlite3 $DBPATH "SELECT base_uri||'/'||filename FROM PHOTOS")
filenames_versions=$(sqlite3 $DBPATH "SELECT base_uri||'/'||filename FROM PHOTO_VERSIONS")
else
filenames=$(sqlite3 $DBPATH "SELECT URI FROM PHOTOS")
filenames_versions=$(sqlite3 $DBPATH "SELECT URI FROM PHOTO_VERSIONS")
fi

# Chomp off the first instance of file://, and replace the rest with newlines.
filenames=$(echo $filenames | sed 's/file:\/\///' | sed 's/file:\/\//\n/g' )
filenames_versions=$(echo $filenames_versions | sed 's/file:\/\///' | sed 's/file:\/\//\n/g' )

if [ $demomode == "true" ]
then
while read -r line
do
# Decode the filename
decodedLine=$(echo -e "${line//\%/\\x}")
if [ ! -f "$decodedLine" ]
then
# If the file doesn't exist, we output the filename, if in demomode, or we fix it if we are not in demomode.
echo "Errant record found in the photos table: $decodedLine"
fi
done <<< "$filenames"

# We do the same for the photo_versions table
while read -r line
do
# Decode filename
decodedLine=$(echo -e "${line//\%/\\x}")
if [ ! -f "$decodedLine" ]
then
# If the file doesn't exist, we output the filename
echo "Errant record found in the photo_versions table: $decodedLine"
fi
done <<< "$filenames_versions"

else
# We backup the database, and make the correction
cp $DBPATH $DBPATH.`date -I`.bak
if [ $? -eq 0 ]
then
#The backup worked, tell the user.
echo "Your database has been backed up to $DBPATH.`date -I`.bak"
else
echo "Error backing up your database."
exit 3
fi

# First we do the photos table
while read -r line
do
# Decode the filename
decodedLine=$(echo -e "${line//\%/\\x}")

if [ ! -f "$decodedLine" ]
then
# Do some sql here.
foo="file://${line}"
echo -n "Deleting URI $line from the database table photos..."
if [ "$version" == "v3" ]
then
sqlite3 $DBPATH "DELETE FROM PHOTOS WHERE base_uri||'/'||filename = '$foo'"
else
sqlite3 $DBPATH "DELETE FROM PHOTOS WHERE uri = '$foo'"
fi
echo "done."
fi
done <<< "$filenames"

# Then we do the photo_versions table
while read -r line
do
# Decode the filename
decodedLine=$(echo -e "${line//\%/\\x}")

if [ ! -f "$decodedLine" ]
then
#Do some sql stuff
foo="file://${line}"
echo -n "Deleting URI $line from the database table photo_versions..."
if [ "$version" == "v3" ]
then
sqlite3 $DBPATH "DELETE FROM PHOTO_VERSIONS WHERE base_uri||'/'||filename = '$foo'"
else
sqlite3 $DBPATH "DELETE FROM PHOTO_VERSIONS WHERE uri = '$foo'"
fi
echo "done."
fi
done <<< "$filenames_versions"

fi

}

if [ "$demomode" == "true" ]
then
echo "Great. Proceeding in demomode."

findAndFixOrphans

echo "Demomode successfully finished. Exiting."
exit 0;
elif [ "$demomode" == "false" ]
then
echo "Great. Cleaning up your database."

findAndFixOrphans

echo "Database cleaned successfully."
exit 0;
else
echo "Something strange happened. See the script for details. Exiting."
exit 2;
fi