Working with Chicago Data (part 1)

We are still talking a lot of data at North Park – in particular Chicago data. So I’m going to start getting my hands dirty working with this data to build capacity for future partnerships with faculty and students. So here is the first in what I hope to be many installments of the “Working with Chicago Data” series.

Mapping Chicago’s Grocery Stores

First step: Download data from the Chicago Data Portal (https://data.cityofchicago.org/). I’m using the Grocery Store 2013 dataset for this example.

Grocery Stores – 2013

Powered by Socrata

The data itself seems pretty clean and well formatted. I’m going to use Tableau for this example because that’s the tool I’m learning right now. I opened Tableau and imported the spreadsheet from the Chicago Data Portal. I ended up creating 4 different visualizations based on this data.

The first is a map of grocery store locations. It uses the latitude and longitude from the dataset to create points. Pretty standard and vanilla.

These next map is much more interesting. It takes into account the size of the store (measured in square footage) and codes that as size and color. Larger stores have larger, darker circles.

The last two maps were just variations on the second map. One version filtered out “small stores” that were less than 10,000 square feet. The other filtered out stores with the work “liquor” in the title. On a technical levels, these filters were easy to apply. However, I’m completely aware of the cultural assumptions I’m bringing to bear here. When I (white, affluent, middle class) think about a grocery store I think about a large store that doesn’t have the word “liquor” in the title.

That’s that! It was pretty easy to get this data and put it to use in the form of a map. I used Tableau here but I could also use Excel (with the power map add) or a more specialized tool like ArcGIS.

In terms of next steps or extensions:

  1. It would be interesting to compare results using a different tool. Might be good to showcase the basic steps for using each tool.
  2. It would be very interesting to add neighborhood boundaries and/or other information such as demographic information and/or economic status. I’ll have to look at ways to incorporate this data.
  3. It would also be very interesting to combine this data with user feedback like Yelp reviews.

Roadmap to Library Publishing

First, I’ve determined that there is no one “roadmap” that will lead my library into digital publishing. So, instead of creating a map, I’m going to do the best I can to sketch out the terrain ahead and think about questions that can guide our path.

Broader Context

This section tries to address two main questions: What is happening in the world of scholarly publishing that is relevant to North Park? What is happening within the North Park setting that is relevant to a library published endeavor? Quick thoughts:

  • Continued movement toward Open Access. There is still work to be done in our local context but that is the clear movement. The Covenant Quarterly and Journal of Hip Hop Studies indicate this trend is taking root on campus.
  • Institutional branding. There is a renewed focus in institutional branding and online presence. There could be powerful connections to make here.
  • Publishing and the North Park mission. My sense is that North Park values diverse contributions to the academic community more than creating a specialized repository
  • Chicago. There might be some opportunities to promote North Park within the regional context through research and student projects.

Scope

We need to define the scope of this project. There are many different efforts that fall under the broad category of “digital publishing”, including:

  • Institutional Repositories
  • Digital Humanities
  • Data Repository
  • Open Educational Resources
  • Campus multimedia (lectures, performances, etc.)

Of these options, I think the most appropriate level and scope would be an institutional repository that contains simple/static documents such as PDFs. A next step would be to curate multimedia from across campus.

Even within this scope, the library will need to make editorial and collection development decisions to make sure that (1) we have a critical mass of content and (2) that there is some editorial scope. I think we should prioritize the following content areas and focus on building relationships with relevant parties.

Student Research

  • Honor’s Projects and Papers
  • Student Research
  • Master’s Thesis
  • NPPress Student Research
  • Covenant History Papers
  • Partnerships with different courses/programs.

Faculty/Staff Scholarship

  • Journal Articles
  • Faculty/Staff Presentations and other “gray” literature
  • Papers from campus symposiums
  • Offer hosting/support for existing campus projects

Political Realities/Soft Skills

We would need some strong support from across campus to take on this project and lead the campus here. Given the proposed scope of this project, here are the people I think it would be important to connect with:

  • The President
  • Provost
  • Campus Deans
  • The University Marketing and Communication Office
  • Honors Program
  • Seminary Faculty
  • Faculty/Tenure Committee
  • NPPRESS Leadership
  • Student Research Committee

Some of these needed connections blend into the next set of questions that seeks to define the scope of this project and effort. I think if we have 5 strong allies (willing to contribute the content they are responsible for) that would make a strong starting point.

Management

Do we have the technical and social workflows to produce, distribute and preserve this content? There are many overlapping questions here, but here is an attempt to list the important ones:

  • Do we have the rights/permissions to publish these materials? Who will work with each group to determine these permissions and who will maintain the paperwork?
  • Do we have the staff expertise, staff time, and faculty/staff connections to successfully manage this projects?
  • What is the ongoing cost of this project in terms of hosting costs, incentives and open access fees, etc.?
  • Where does this rank compared to other library/institutional priorities?
  • What are peer institutions doing? What can we learn from them?

Northfield Historical Society (behind the scenes) updates!

I just finished some “behind the scenes” updated to the Northfield Historical Society site and wanted to document that process here. It was definitely a bit messy at times and quite labor intensive, but I think it was the best way to deal with the situation I faced.

Background

Starting last year, I got about 260 images from the old Northfield Historical Society site (archived here: http://www.oldsite.northfieldhistoricalsociety.org/). These images varied greatly in quality; there were a few large, high quality TIFF images but most files were small JPEGs the size of thumbnails. In order to get intellectual control over these files, I renamed them and manually formatted metadata (using Dublin Core) to create an Omeka site. This was part of my “Web Design for Organizations” class I took through GSLIS.

However, the low quality images didn’t look great online. In fact, they looked pretty bad. So I inquired if higher quality versions of the files existed somewhere else. After some searching, they were able to get higher quality images from another source. Success!

This new batch of files was a treasure trove…but also had a few problems:

  1. Very different file naming conventions.
  2. Included many additional photos not found in the initial ingest.
  3. Did not include all the files from the initial ingest.
  4. Included both TIFF and JPEG images.

So I needed to match the new files with the older set of images (keeping the highest quality image in each case) and then incorporate the new files into the file naming convention I established earlier.I used two tools that were particularly helpful in this process. One was a Batch Rename tool and the other was a Image Duplication tool. Both were extremely helpful.

Results

Here are two screenshots of the same “item” with two very different image files.

old-image

 

 

 

 

 

The improvement is hard to miss!new-image

 

Python Cheat Sheet

I’m taking a class on Python and thought it would be helpful to compile my own “cheat sheet” of commands, functions, and common constructions that might be useful in the future. Following the mantra of “Do Good Work in Public” I’m posting it here.

Basic Stuff

Functions

Functions are a type of a Python object that can take parameters and can return something (a number, value, etc.) Basic design:

def functionname(parameter1, parameter2...):
    statements

Repetition

Use a for loop to do something many times. Example

for i in range(10):
    do something

This does something ten times. This uses the range function which can be formatted in three ways

  1. range(stop)
  2. range(start, stop)
  3. range(start, stop, step)

Relational Operators

Table from: http://pymbook.readthedocs.org/en/latest/operatorsexpressions.html

Operator Meaning
< Is less than
<= Is less than or equal to
> Is greater than
>= Is greater than or equal to
== Is equal to
!= Is not equal to

Selection Statements

if <condition>:
    <statements>
else:
    <statements>
  • The “else” statement is not necessary; an if statement can stand alone
  • Nesting selection statements is also fine – just indent appropriately.
  • A “tail nesting” is when all the nesting occurs within the last else. Python provides the shortcut elif to address this situation.

String/List Functions

Indexing - [] - Access an element in a sequence
Concatenation - + - Combine sequences together
Repetition - * - Repeated concatenation
Membership - in / not in - Asks whether an item is in a sequence / asks whether an item is not in a sequence.
Length - len - Asks the number of items in a sequence
Slicing- [:] - Extracts a part of a sequence.

There are also lots of ways to format strings – see table 3.2.

List Methods

append - listname.append(item) - adds a new item to the end of the list
insert - listname.insert(i, item) - inserts an item at the i spot in the list
pop - listname.pop() - removes and returns the last item from the list
pop - listname.pop() - removes and returns the (i) item from the list.
sort - listname.sort() - modifies a list to be sorted.
reverse - listname.reverse() - modifies a list in reverse order.
index -

Accumulator Function/Structure

Example. See page 95

Shortcut!

These do the same thing!

num = num +1
num += 1

Data Storage Options aka Collections

Strings

Strings are collections of stuff enclosed in quotation marks. For example, “Andy Meyer” is a good Python string. You can also retrieve things strings using the index. Strings cannot be changed; you can only point somewhere else.

nameString = "Andy Meyer"
nameString[3] would return "y" because the character "y" is in the third index position.

Lists

List are like strings…but also different in several ways:

  • List are enclosed in square brackets [].
  • Can be heterogeneous (can store numbers, string, characters, etc.)
  • List are mutable – they can be changed and updated. Items can be added and deleted.
nameList = ["Andy","Meyer", 30]
nameList[1] will return "Meyer" because that is the item in index position 1.

There are a lot of list methods. See table 4.2 on page 125.

Dictionaries

Dictionaries are unordered, associative collections that associate keys with values. Dictionaries use curly brackets {} and follow this formula {key, value}

agedictionary = {"Andy":30}

You can do cool stuff with dictionaries!

agesdictionary["Jim"]=31 # This adds a new entry (key, value) to the dictionary.
agesdictionary["Andy"] # this would return the value (30) associated with that key.

Working with Files

Three concepts:

  1. File name = the name of a file as a string
  2. File handler = the
  3. Contents of the file =

Examples!

filename = 'filename.txt'
filehandle = open(filename, "r") # this opens the file.
filehandle = open("anotherfilename.txt","w") # this opens the 
contentsofthefile = filehandle.read()

Read the File

filename.read() = reads and returns the whole file as a single text string
filename.read(n) = reads and returns the first n characters as a single text string
filename.readline() = reads and returns the file one line at a time. # You can iterate this and read an entire file, line by line.
filename.readlines() = reads and returns a list - each line is a seperate item in the list.

###
filename.seek(n) = seeks out the nth place in a document.

MacroExpress and Libraries – Part 2

Continuation of an earlier post: http://www.meyerwebsite.com/macroexpress-and-libraries/.

In addition to the “simple” commands like text expansion and automating simple commands, Macro Express can also be used to perform very complicated tasks across different platforms and sources. Rather than outlining all possible options – a task I’m not qualified to do! – I’ll focus on a basic workflow that I use frequently. I think of it as “working through a list.” This is a way to automate some of the most repetitive tasks for managing bibliographic records.

Background

This process works best when you have two things:

  1. A clearly defined list of records that need to be changed/updated/moved/deleted.
  2. A clearly defined process to perform.

By “clearly defined” I mean defined with mathematical precision; Macro Express takes things very literally and you have to be precise. It’s also helpful to imagine all possible cases to anticipate problems along the way. Let’s look at each step in turn.

A Clearly Defined List

To start, you will need a clearly defined list. Thing like system supplied ID number for bib records, MFHDs, items, or patrons are ideal; ISBNs, ISSN, and exact name might seem good enough but they usually cause problems.

I generate almost all of these lists through the reporting side of Voyager and have started always including the system supplied ID numbers. Depending on the nature of your query, you may need to de-duplicate this list.

A Clearly Defined Process

I’ll share my general approach and then some tips and tricks. My process is usually to set my mouse aside and try accomplish what I need to do using only keyboard shortcuts. MacroExpress can control the mouse…but this always seemed very problematic; keyboard shortcuts are the way to go. Start the process with the ID number of a record to change on your clipboard. As you go, write down you commands on a pad of paper in sequential order. This will take some time.

A Recipe

Text File Begin Process: "filename.txt"
Activate Window: "Name of ILS system"
Wait For Window: "Name of ILS system"

This set of commands starts reading a text file (one that contains all of the system ID numbers) and activates the window you want to use.

Next, bring up your search window and search by that system number. Do this by using the “Text Type” command in MacroExpress and following your paper notes. You may have to incorporate some “Delays” to give your machine time to keep up. This should bring up the record you want to look at.

Do something! Ideally, you’ll write more MacroExpress code to automate the process totally.

However, a very simple option and one that is useful in complicated situations is to simply add the following command.

Wait for Key Press: <something weird>

Replace <something weird> with a key that you are unlikely to use in that process. Now you have a macro that brings up a record and waits… waits until you make and save a change, waits until you inspect it, just waits. You can set a time-out requirement if that is helpful.

Lastly, write some text type functions that will close and save the record you are working on. I also like to use the following command to keep a record of my work

Variable Modify String: Append %T1% to Text File

This copies the system ID number from the original text file and appends it to a new text file that will list all the records changed in the process.

 

Macro Express and Libraries

I’m a big fan of library automation technologies and I thought I would write up a short piece on one of my favorite tools – Macro Express – to share with the library world.

What is Macro Express

Macro Express describes itself as:

the premier Windows macro utility. With Macro Express, you can record, edit and play back mouse and keyboard macros. Its powerful tools and robust features will make you more productive.

Although it’s not glamorous, I think this description is pretty accurate. Macro Express works across the Windows OS to automate tedious tasks. My general outlooks is this: if it’s easy to do once but really hard to do 300 times, you should try creating a Macro to automate this work.

Setting up Macro Express

  1. Download from http://www.macros.com/. They provide a 30 day trial and the license fees are quite cheap. Totally recommend purchasing this for your library.
  2. It comes pre-loaded with a few macros but you’ll certainly want to create your own. One important distinction, a “macro” is stored in a “macro file.” So if you are creating a new macro, select “New Macro” and not “New Macro File.”
    macrosnip1
  3. Macro Express should then prompt you “Choose Default Activation.” Macros can be triggered in a number of different ways. The ones I use most often are:
    1. HotKey – This creates a “normal” action that mimics other keyboard shortcuts. Just like CTRL C and CTRL V trigger certain actions, you can program keyboard combinations to execute certain commands.
    2. ShortKey – This activation method uses a special identifier and allows you to execute a macro using a “short key.” I believe the system default is ##. For example, when I type “##ip” Macroexpress replaces that text with the EZ Proxy URL prefix. It’s awesome.
    3. No Activation – this is pretty obvious; this means that macros can only be run from the MacroExpress editor
  4. After selecting an activation option (I will survey all three approaches) click on “Scripting Editor” and you are ready to create a new macro!

Basic Macros

Let’s make a macro that adds your signature to things that we can activate with a ShortKey.

  1. Create a new macro.
  2. Set default activation as a shortkey. I’ll use my initials “ajm” as my shortkey.
  3. Click on scripting editor. This should bring you to the following screen:
    macrosnip2
  4.  This is the “Scripting Editor” window. All of the available commands are on the left side (under “Commands”) and the actual macro you are writing is on the right/main side of the screen (under “Macro Script”)
  5. To get started, you’ll need to add a command. Either find a command by expanding the categories or by searching for a command (bottom left corner). We are looking for the “text type” command which is under “Text.”
  6. Double click the command to add it to the macro or click the small arrows to add it to the macro.
  7. You will get a box that allows you to enter keystrokes. Type out your signature (or copy/paste it in the box.
  8. Click okay and look over your macro.
  9. To give your macro a name (which I recommend!) click on the properties tab from the top band and add a nickname. Then click “Save” in the upper left corner.
  10. Congratulations! You just created a new macro!

I’ll try and write other posts that detail the other activation methods and how they might be useful in a library context. More later!

INDEX and MATCH functions in Excel

I do a lot of work in Excel and I’ve used the INDEX and MATCH functions together instead of a VLOOKUP or HLOOKUP but always stumble through that process. I wanted to write a simple guide to help me in the future.

ID Item Color
1 Shoes Black
2 Socks Brown
3 Pants Blue
4 Shirts Red

Pretending for a minute that is our spreadsheet, let’s look at the functions in turn.

INDEX

Here is the function syntax: INDEX(array, row_num, [column_num]). Here is how I think of it and use it: INDEX(some rectangular shaped data set in excel, row number, column number)

Using the example, =INDEX(A1:C5,2,3) would return “Black” because that’s the data in the second row and third column of the rectangular shaped data set A1:c5.

MATCH

Here is the function syntax: MATCH(lookup_value, lookup_array, [match_type]) Here is how I think of it and use it: MATCH(what are you looking for, what rectangle of data should we search?, what kind of match are you looking for?)

Using the example, =MATCH(“Socks”,B1:B5,FALSE) would return “3” because “Socks” is in the 3rd column of that array.

CC image - https://flic.kr/p/nK7kRW
CC image – https://flic.kr/p/nK7kRW

INDEX and MATCH together!

Those functions are pretty useless on their own…but together they are awesome. Let’s say I want to know the color of the item with an ID of “4” – using these functions I can retrieve that data!

=INDEX(the column that has the information I want, MATCH(thing I’m looking for, column that has the data I’m starting with, FALSE))

Using the example, =INDEX(C1:C5,MATCH(“Shoes”,B1:B5)) would return “Black” because that is the data point at the intersection of column C and the row that contains the data point shoes.

Question to help the process:

  • What column has the information you want?
  • What data are you starting with?
  • What column has the data you are starting with?

Review – Publisher of the Community: We’re All Doomed

I just watched R. David Lankes’s recent keynote and wanted to post a short review of the material he covered.

Information

“Publisher of the Community: We’re All Doomed” Closing keynote for the NISO Workshop on “Using the Web as an E-Content Distribution Platform: Challenges and Opportunities.” http://quartz.syr.edu/blog/?p=6265

Abstract: We need to build platforms for scholarship and knowledge development, not information and content delivery. These platforms are not about APIs and eContent, but about people and content. We need to strive not for discovery, but epiphanies.

[vimeo id=”109836897″]

Reflection/Response

I really enjoyed this presentation and will take a few new ideas from it. Beyond new ideas, it was refreshing and inspiring and perhaps a bit challenging to hear his thoughts about collections and technology.

Lankes began by talking about the “death of documents” in that the definitions of documents are changing in rapid and new ways. Digital stuff is different at a fundamental level and the editorial process is radically different because digital information resources are living, changing things. There is continuity in this change but, overall, the change has been a massive paradigm shift. The words “ebooks” and “ejournals” are really just metaphors – we are trying to fit new information into old information formats.

This section flooded my mind with thoughts.

  • In my work moving the Covenant Quarterly online, how can I respect both the “old format” while embracing the new opportunities of a digital environment?
  • I asked a very similar question about my work with the Northfield Historical society – how do I work with that community and not just their collection? Great questions that are constantly reframing my approach.

Lankes also argues for a “sea change” in the history of library and library services – we need to stop thinking about content. There have been massive shifts in the last 40 years:

  • Collection focus. Our primary role was collecting and preserving documents. This worked…but only worked with a functional infrastructure of knowledge producers (academics and publishers) and libraries. Technical revolutions changed that – things like cheap paper and ink – and libraries had to change and evolve. The modern conception of a “book palace” – a building full of books – is a product of this period of time. This model is breaking – and totally broken by the digital revolution and the internet. Creating MARC records for webpages is crazy.
  • Service focus. Cataloging is a service – that has both internal and external audiences. Question – are we really applying our services to the external audience? From library orientation to information literacy – that’s a sign of the shift from collections to service focused libraries. This is a big change…but it’s not enough. It’s still operating in the “consumption” model of information. This model creates an artificial line between the knowledge consumption process and the knowledge creation process.
  • Conversation focus. It’s not about putting content online – it’s about making libraries as centers for conversations as mediators and participants. Conversations create revolutions. Move beyond collecting and organizing metadata toward capturing scholarship itself.

A Netflix for education doesn’t work because education isn’t based on consumption – it’s based on learning and learning is based on conversation and participation.

However, I do take issue with some of these claims. First, this presentation created vocational confusion. Lankes argues – persuasively, I might add – that libraries must move beyond a focus on collections to a focus on service and even beyond that to community conversations. However, my library and my position in the library are organized around the first two areas – collections and services – and my job is focused squarely on maintaining access to our collections and a few of our services. So while I’m inspired by the claims that we need to focus beyond those things I wondered what those new services and platforms for community conversations could actually look like and – as importantly – how do we move in that direction without the required skills? Lankes said that “People create restrictions that technology has removed” but that gives me pause for several reasons. First – the technical challenges to what Lankes proposed are not trivial. The technical infrastructure takes time and energy and expertise that many libraries and community groups simply lack. Second, how do we manage this change?

These – of course – are criticisms as much as they are struggling to think through the ramifications of his argument in my local context and in the context of my future.

CC image from Flickr - https://flic.kr/p/hKQMcN
CC image from Flickr – https://flic.kr/p/hKQMcN

Looking forward, I’d like to think more about the following things and perhaps explore them in library school:

  • The history of the book and the history of libraries. It seems like many librarians take this background and context for granted but I think it deserves intentional study.
  • Focus on building conversations. I think this means a focus on reference and instruction but – beyond that – how can I work with the communities I’m a member of?
  • How can I embody that belief – that the real goal of learning isn’t the consumption of knowledge or the production of knowledge but mediating and participating in that conversation? I try and do that here on my blog – but my attempts have been mixed. I will renew that effort!

Okay. I think that’s enough in the way of reflections. That’s plenty to think about and plenty to wrestle with for now.