Posted on

Personal Digital Archiving – Documents

I’m taking a Digital Preservation class and one of my assignments involves looking at my own personal digital archive. I usually pride myself on being organized and having a pretty minimal and clean digital presence…but formally looking at my stuff made me realize how much digital junk I really had!

So now I’m slowly working through my personal digital archive and am about to tackle my school documents. I thought I would post my thoughts and process on this blog for future reference.

Step 1 – Delete Stuff

I had a lot of junk lurking around from my school days. Old drafts, purely administrative documents, and other worthless things. Delete all of that stuff.

Step 2 – Copy Stuff

This is just an experiment so I’m planning on copying all of my school work (at least for the time being). Therefore I’ll be preserving the original file (in the original file format) as well as an archival master (as a PDF).

Step 3 – Rename Stuff

Right now, I use folders to sort and organize my files. This approach really works for me – but I wanted to explore an alternative way of describing and organizing files. So I’m going to embed more metadata in the filename itself. To do this bulk rename, I’m going use my Mac’s built in File Rename tool. Here is a video:

I’m going to re-work my folder structure into file names following these conventions:

SchoolSchoolYear-Semester#_CourseAbbreviation_FileName

So, for example,

  • NPU1-1_BTS1850_FinalPaper = North Park University, freshman year, first semester, BTS 1850, Final Paper.
  • NPTS2-2_BIBL5111_FinalPaper = North Park Theological Seminary, second year, second semester, BIBL 5111, Final Paper.

I’m planning on adding a readme to this folder – but the designated community for this project is just me and I think I can remember.

Step 4 – Migrate to PDF

I’m not dealing with ancient files here…but I am dealing with Word documents from 2002. So far, all of these files are still readable (and the Word format seems relatively safe for the foreseeable future…) but I wanted to get some experience bulk converting to PDF.

I’m going to use the Mac’s Automator feature again and run a custom AppleScript that I found online here: http://igikorn.com/batch-convert-word-documents-pdf/.

Step 5 – Back Up

At this stage, I have a folder – currently called “Personal Digital Archive” that has all of my school documents with uniform file names in the PDF format. Looks great! I’m planning on backing up these files using my standard back-up method (external hard drive) and I’m also exploring a cloud storage solution that is not DropBox or Google Drive.

 

Posted on

Python Cheat Sheet

I’m taking a class on Python and thought it would be helpful to compile my own “cheat sheet” of commands, functions, and common constructions that might be useful in the future. Following the mantra of “Do Good Work in Public” I’m posting it here.

Basic Stuff

Functions

Functions are a type of a Python object that can take parameters and can return something (a number, value, etc.) Basic design:

def functionname(parameter1, parameter2...):
    statements

Repetition

Use a for loop to do something many times. Example

for i in range(10):
    do something

This does something ten times. This uses the range function which can be formatted in three ways

  1. range(stop)
  2. range(start, stop)
  3. range(start, stop, step)

Relational Operators

Table from: http://pymbook.readthedocs.org/en/latest/operatorsexpressions.html

Operator Meaning
< Is less than
<= Is less than or equal to
> Is greater than
>= Is greater than or equal to
== Is equal to
!= Is not equal to

Selection Statements

if <condition>:
    <statements>
else:
    <statements>
  • The “else” statement is not necessary; an if statement can stand alone
  • Nesting selection statements is also fine – just indent appropriately.
  • A “tail nesting” is when all the nesting occurs within the last else. Python provides the shortcut elif to address this situation.

String/List Functions

Indexing - [] - Access an element in a sequence
Concatenation - + - Combine sequences together
Repetition - * - Repeated concatenation
Membership - in / not in - Asks whether an item is in a sequence / asks whether an item is not in a sequence.
Length - len - Asks the number of items in a sequence
Slicing- [:] - Extracts a part of a sequence.

There are also lots of ways to format strings – see table 3.2.

List Methods

append - listname.append(item) - adds a new item to the end of the list
insert - listname.insert(i, item) - inserts an item at the i spot in the list
pop - listname.pop() - removes and returns the last item from the list
pop - listname.pop() - removes and returns the (i) item from the list.
sort - listname.sort() - modifies a list to be sorted.
reverse - listname.reverse() - modifies a list in reverse order.
index -

Accumulator Function/Structure

Example. See page 95

Shortcut!

These do the same thing!

num = num +1
num += 1

Data Storage Options aka Collections

Strings

Strings are collections of stuff enclosed in quotation marks. For example, “Andy Meyer” is a good Python string. You can also retrieve things strings using the index. Strings cannot be changed; you can only point somewhere else.

nameString = "Andy Meyer"
nameString[3] would return "y" because the character "y" is in the third index position.

Lists

List are like strings…but also different in several ways:

  • List are enclosed in square brackets [].
  • Can be heterogeneous (can store numbers, string, characters, etc.)
  • List are mutable – they can be changed and updated. Items can be added and deleted.
nameList = ["Andy","Meyer", 30]
nameList[1] will return "Meyer" because that is the item in index position 1.

There are a lot of list methods. See table 4.2 on page 125.

Dictionaries

Dictionaries are unordered, associative collections that associate keys with values. Dictionaries use curly brackets {} and follow this formula {key, value}

agedictionary = {"Andy":30}

You can do cool stuff with dictionaries!

agesdictionary["Jim"]=31 # This adds a new entry (key, value) to the dictionary.
agesdictionary["Andy"] # this would return the value (30) associated with that key.

Working with Files

Three concepts:

  1. File name = the name of a file as a string
  2. File handler = the
  3. Contents of the file =

Examples!

filename = 'filename.txt'
filehandle = open(filename, "r") # this opens the file.
filehandle = open("anotherfilename.txt","w") # this opens the 
contentsofthefile = filehandle.read()

Read the File

filename.read() = reads and returns the whole file as a single text string
filename.read(n) = reads and returns the first n characters as a single text string
filename.readline() = reads and returns the file one line at a time. # You can iterate this and read an entire file, line by line.
filename.readlines() = reads and returns a list - each line is a seperate item in the list.

###
filename.seek(n) = seeks out the nth place in a document.
Posted on

MacroExpress and Libraries – Part 2

Continuation of an earlier post: http://www.meyerwebsite.com/macroexpress-and-libraries/.

In addition to the “simple” commands like text expansion and automating simple commands, Macro Express can also be used to perform very complicated tasks across different platforms and sources. Rather than outlining all possible options – a task I’m not qualified to do! – I’ll focus on a basic workflow that I use frequently. I think of it as “working through a list.” This is a way to automate some of the most repetitive tasks for managing bibliographic records.

Background

This process works best when you have two things:

  1. A clearly defined list of records that need to be changed/updated/moved/deleted.
  2. A clearly defined process to perform.

By “clearly defined” I mean defined with mathematical precision; Macro Express takes things very literally and you have to be precise. It’s also helpful to imagine all possible cases to anticipate problems along the way. Let’s look at each step in turn.

A Clearly Defined List

To start, you will need a clearly defined list. Thing like system supplied ID number for bib records, MFHDs, items, or patrons are ideal; ISBNs, ISSN, and exact name might seem good enough but they usually cause problems.

I generate almost all of these lists through the reporting side of Voyager and have started always including the system supplied ID numbers. Depending on the nature of your query, you may need to de-duplicate this list.

A Clearly Defined Process

I’ll share my general approach and then some tips and tricks. My process is usually to set my mouse aside and try accomplish what I need to do using only keyboard shortcuts. MacroExpress can control the mouse…but this always seemed very problematic; keyboard shortcuts are the way to go. Start the process with the ID number of a record to change on your clipboard. As you go, write down you commands on a pad of paper in sequential order. This will take some time.

A Recipe

Text File Begin Process: "filename.txt"
Activate Window: "Name of ILS system"
Wait For Window: "Name of ILS system"

This set of commands starts reading a text file (one that contains all of the system ID numbers) and activates the window you want to use.

Next, bring up your search window and search by that system number. Do this by using the “Text Type” command in MacroExpress and following your paper notes. You may have to incorporate some “Delays” to give your machine time to keep up. This should bring up the record you want to look at.

Do something! Ideally, you’ll write more MacroExpress code to automate the process totally.

However, a very simple option and one that is useful in complicated situations is to simply add the following command.

Wait for Key Press: <something weird>

Replace <something weird> with a key that you are unlikely to use in that process. Now you have a macro that brings up a record and waits… waits until you make and save a change, waits until you inspect it, just waits. You can set a time-out requirement if that is helpful.

Lastly, write some text type functions that will close and save the record you are working on. I also like to use the following command to keep a record of my work

Variable Modify String: Append %T1% to Text File

This copies the system ID number from the original text file and appends it to a new text file that will list all the records changed in the process.

 

Posted on

Macro Express and Libraries

I’m a big fan of library automation technologies and I thought I would write up a short piece on one of my favorite tools – Macro Express – to share with the library world.

What is Macro Express

Macro Express describes itself as:

the premier Windows macro utility. With Macro Express, you can record, edit and play back mouse and keyboard macros. Its powerful tools and robust features will make you more productive.

Although it’s not glamorous, I think this description is pretty accurate. Macro Express works across the Windows OS to automate tedious tasks. My general outlooks is this: if it’s easy to do once but really hard to do 300 times, you should try creating a Macro to automate this work.

Setting up Macro Express

  1. Download from http://www.macros.com/. They provide a 30 day trial and the license fees are quite cheap. Totally recommend purchasing this for your library.
  2. It comes pre-loaded with a few macros but you’ll certainly want to create your own. One important distinction, a “macro” is stored in a “macro file.” So if you are creating a new macro, select “New Macro” and not “New Macro File.”
    macrosnip1
  3. Macro Express should then prompt you “Choose Default Activation.” Macros can be triggered in a number of different ways. The ones I use most often are:
    1. HotKey – This creates a “normal” action that mimics other keyboard shortcuts. Just like CTRL C and CTRL V trigger certain actions, you can program keyboard combinations to execute certain commands.
    2. ShortKey – This activation method uses a special identifier and allows you to execute a macro using a “short key.” I believe the system default is ##. For example, when I type “##ip” Macroexpress replaces that text with the EZ Proxy URL prefix. It’s awesome.
    3. No Activation – this is pretty obvious; this means that macros can only be run from the MacroExpress editor
  4. After selecting an activation option (I will survey all three approaches) click on “Scripting Editor” and you are ready to create a new macro!

Basic Macros

Let’s make a macro that adds your signature to things that we can activate with a ShortKey.

  1. Create a new macro.
  2. Set default activation as a shortkey. I’ll use my initials “ajm” as my shortkey.
  3. Click on scripting editor. This should bring you to the following screen:
    macrosnip2
  4.  This is the “Scripting Editor” window. All of the available commands are on the left side (under “Commands”) and the actual macro you are writing is on the right/main side of the screen (under “Macro Script”)
  5. To get started, you’ll need to add a command. Either find a command by expanding the categories or by searching for a command (bottom left corner). We are looking for the “text type” command which is under “Text.”
  6. Double click the command to add it to the macro or click the small arrows to add it to the macro.
  7. You will get a box that allows you to enter keystrokes. Type out your signature (or copy/paste it in the box.
  8. Click okay and look over your macro.
  9. To give your macro a name (which I recommend!) click on the properties tab from the top band and add a nickname. Then click “Save” in the upper left corner.
  10. Congratulations! You just created a new macro!

I’ll try and write other posts that detail the other activation methods and how they might be useful in a library context. More later!

Posted on

Automation on an Apple – Batch renaming files on a Mac

The subtitle of this page could be “how to carefully rename 1,000 files.”

In my work with the Center for Railroad Photography and Art I handled a range of digitization and file management project. Each has required it’s own approach and I’ve learned a lot about Apple automation technologies working in a recent aspect of the project. It’s using the Apple tool “Automator” in two different ways. While I have some experience using other automation programs like MacroExpress and other File Renaming tools on a PC this was my first attempt on a Mac. And I think it went pretty well! Here was the problem and the solution I developed to solve this problem.

The Problem

Springer file names

The digital files were named using the actual photographic metadata. While this might seem great, it created two fundamental problems: (1) the digital files were out of order (relative to the original order of the collection) and (2) it did not match the file naming conventions I had established for this collection. So my task was to remedy this problem and (1) put the digital files in the same order as the physical collection and (2) extend the current file naming conventions to this part of the collection.

Organizing the Files

The first step was to organize the files. To do this, I decided to rename each file in some sort of alpha-numeric system. This seemed both easy and tedious. To make it slightly easier, I using the Automator service on the Mac to create a service. On a macro level, I could click a button and make a copy of the file that would then be organized in box order in a separate folder.

Here is this workflow in detail:

Screenshot of Apple Automator

  1. Copy Finder Item to a new folder – This created a second copy of the file that allowed me to explore and experiment without trouble.
  2. Rename the folder by adding the date/time function “seconds from midnight” to the front of the file name- The “seconds from midnight” function was totally arbitrary but provided a nice, easy way to sort the files.
  3. Save this as a “service” – Saving this as a service gave me easy access to trigger this automated workflow from anywhere in the finder.

After I established this workflow, I started going though the box. Once I found the digital file that corresponded to my first negative I would trigger this service. Thus, as I progressed through the box of negatives I would create a new folder of digital images in the correct order. Huzzah!

Renaming the Files

The end result of the first automated process was a folder of files in the correct order. However, the file names were terrible. For example:

Screenshot of organized files.

The leading numbers allow this list to be sorted into the correct order but this file naming convention was a huge mess. However, another automated workflow quickly solved this problem. The steps of this process were:

  1. Find Finder Items – This step ingested the files I created through the rename service
  2. Sort Finder Items – Although this step might seem redundant it provided to be critical to get the sorting correct. I simply sorted by name in ascending order to make this workflow work.
  3. Copy Finder Items – Again, because this was my first time using the Automator I wanted to create back-up copies of my work. I copied the items to yet another folder so that I could sort them.
  4. Make Sequential – This is the important step! I made them sequential by adding a new name (in this case I replaced the existing name with “Springer_01”) and then by placing a number after the name. I separated this number with an underscore and asked the program to make all numbers three digits long.
Apple's Automator - my new best friend.
Apple’s Automator – my new best friend.

The End Result

The result of these two processes created a perfectly organized folder of items that was named according to our established conventions. Doing this renaming work manually (without the Automated workflows) would have been possible but extremely labor intensive and prone to many different errors. This workflow saved hours of work and streamlined the process to prevent errors.

Additionally, this workflow is very general and could easily be applied to other file renaming projects.

Relative to other automation tools, Apple’s Automator is a strong contender. Because it is part of the Mac OS it integrates perfectly and natively with many Apple services and systems and offers a wide range of options. I would definitely seek to use the Automator on all future digital projects in a Mac environment.

Posted on

E-Resource Workflows in CORAL

A while back, I read the book “The Checklist Manifesto.” It was a very worthwhile read that was very relevant to libraries and librarians. That has been in the back of my mind as I craft workflows for our electronic resources. The books main premise, as summarized by Steven Levitt is simply this:

No matter how expert you may be, well-designed check lists can improve outcomes.

Two small details from that book have stuck with me.

First, useable workflows are short. Long checklists seem doomed to fail – it is better to break those up into smaller, more manageable lists. Many of the books examples are drawn from the airline and health care industries; it would be crazy to craft comprehensive workflows in those situations. So they have checklists for particular situations that, if need be, direct you to other checklists. I’ve adopted and adapted that idea here.

Second, checklists can either be READ-DO lists or DO-CONFIRM lists. READ-DO checklists are like recipes; you read something and then do something. DO-CONFIRM checklists reverse the order; do the things that need to be done and then confirm they are done. I think too many library checklists are READ-DO checklists when I believe that DO-CONFIRM lists are better suited to this part of library resources. Why? Because the diversity of library resources sometimes demand different things and because, often, the order of steps isn’t vitally important. I could be swayed on this – but right now I’ve been happy with my DO-CONFIRM checklist.

In CORAL, workflows are set for Acquisition Type, Format, and (optionally) Type. All of these fields can be defined in the admin setting of the resources tab. I’ve decided not to really discriminate based on resource type or resource format. So, essentially, we have four basic workflows for electronic resources. They are as follows:

Active Trial/Electronic/[Any]

This is the starting place for all new electronic resources and is the basic workflow when setting up a trial.

  1. Resource information is complete in CORAL.
  2. Organization information is complete in CORAL.
  3. Collection decision has been made.
  4. Appropriate Access Points have been determined and recorded.
  5. Pay invoice; gather license; change Acquisition Type in CORAL.

Paid/Electronic/[Any]

If after the trial period we decide to purchase  resource, I will change the acquisition type to “Paid” – this will trigger a number of different CORAL workflows based on the resource type. Here is the basic and general outline.

  1. Resource and Organization information is complete in CORAL.
  2. A copy of the invoice is saved in CORAL.
  3. A copy of the license is saved in CORAL and has been parsed for ILL rights.
  4. Appropriate access points have been created.
  5. Access has been checked from both on-campus and off-campus.
  6. SUSHI (or alternative statistic measuring tool) is in place.

Free/Electronic/[Any]

I’ve decided to add all of the free resources that North Park provides access to in CORAL for a number of reasons. First, I’d like CORAL to be a centralized and complete inventory of all our electronic resources. Second, if we are providing access we need to commit to supporting that access. Third, because so many valuable information resources are moving forward open access models.

  1. Title information is relatively complete in CORAL.
  2. Applicable access points (Catalog, Database pages, Libguides, SFX) have been created.
  3. Check access points.

Additionally, a few of these resources are actually cancelled print+online titles that guarentee perpetual access. Because the workflows are identical, I’d added these to this category with notes detailing the acquisition history. They are identifiable by having a resource type of “ejournal (single title)”. NOTE: If we start adding many open access titles to our list, I might have to specify what sort of “free” we are talking about; free because of a past subscription or free because it’s open access.

Cancelled/Electronic/[Any]

Right now, our “Historical” Acquisition type is for cancelled titles are trials that we decided not to purchase. In addition to changing the status as “archives” we have a one step workflow for these resources.

  1. All Access Points have been removed, disabled, or deleted.

Additionally, I note why the product was cancelled and/or why we opted not to purchase the product.

So that’s it. I have other workflows for the more detailed processes (adding a title to SFX, setting up SUSHI, etc.) but those exist best as separate workflows.

Posted on

Changing the LDR using MarcEdit

Just a quick update with a new workflow to add to this informal guide to my use of MarcEdit. The way our consortium (CARLI) displays format facets in VuFind recently changed and some of the MARC records for ebooks needed to be updated. One thing that needed to change was the “Bibliographic Level” in the leader needed to be changed from “d” to “a.”

I was unsure how to do this, but a simple google search resulted in Terry Reese’s helpful blog that explained the process and the function. I snapped a quick screenshot for future reference.

change LDR in MARC records

Unlike other subfields, the Leader is character specific and things cannot simply be deleted; they much be carefully replaced to preserve the order and spacing. The “Bibliographic Level” is recorded in the 7th position, so I ran this global change command and quickly made the switch to all 3,500 records.

Posted on

Bulk Importing Special Characters

I’ve been working on the North Park Archon project for the past few weeks and noticed a troubling trend. When I tried to upload the box/folder lists for our finding aids through the CSV import process, the system would reject all the special characters. Not outright reject them – just truncate the field before the character and move on to the next metadata sub-field.

Obviously, this was annoying and unfortunate – especially because I didn’t catch it until I had imported all the data.

It turns out that the problem was in the encoding language of the CSV document. For some reason, my default was set as ANSI; switching it to UTF made the difference and Archon’s scripts were able to import the special characters in the collection content field.

Posted on

Adding URL proxy information using MarcEdit

Another in the short series of posts dedicated to using MarcEdit to manipulate batches of MARC records. Right now, we use MarcEdit to batch catalog our electronic resources. This page is dedicated to a very simple and very common task: adding our institutional proxy to the vendor provided URLs.

Adding our proxy

In MarcEdit, go to Tools>Edit Subfield.

Why is this screen red? Read more below…

You want to edit the $u subfield of the 856 field, so select that. Entering ^b as the “field data” – and checking the “use regular expressions” box! – will use regular expressions to insert the proxy information at the very beginning of that subfield. That’s all there is to it!

Well…not really. To save myself even more time and further automate these tasks, I’ve saved this command as part of a task in MarcEdit. I called this task “Proxy all the URLs” and that is why the screen above is red. I’ve also added a few other very standard commands to this task. In order, they are:

Deleting all the existing 856$z notes. Get rid of them.

Deleting all the 856$z notes.

Adding our own, new an improved public notes and link text.

Changing the link text (856$y).

It’s worth mentioning that MarcEdit does these commands in order. If – for example – the delete 856$z was my final command, MarcEdit would very happily delete the nice new 856$z that was just created. So order matters!

That’s a quick overview of the “Proxy the URLs” task I wrote in MarcEdit. I like it because of it’s simplicity and consistency across multiple records and removes the possibility of error that typing or even cutting and pasting would introduce.

As per the usual, Terry Reese himself has a very helpful video on MarcEdit’s task automation tool.

 

Posted on

Adding and formatting the 300 field using MarcEdit

We use MarcEdit primarily to deal with ebook records, which sometime have thorny 300 fields. We want to standardize these fields as much as possible while preserving valuable information. Adding a 300 field to every single record is quite easy – see this page for information.

However, some batches will be mixed; some records will have 300s and some won’t. In this situation, we’ve opted to add a generic 300 field to records that lack one completely. This involve just the smallest amount of conditional logic.marcedit8This adds a 300 field only to records that don’t already have one.

Reformatting physical descriptions to be appropriate for ebooks is more difficult – but possible using MarcEdit. More details forthcoming.