Data revision history

Longitudinal Public Library Data File (PLDF2)

Abstract

The purpose of these pages is to document a longitudinal set of public library data constructed from the annual Public Library Survey of the National Center for Education Statistics (NCES) under the Federal State Cooperative System (FSCS) program. From this survey, NCES publishes the Public Library Data File, State Summary/State Characteristics Data File, the Public Library Outlet Date File, and the E.D Tabs reports entitled Public Libraries in the United States for the various years. This compilation is of the Public Library Data File (PLDF2). The dataset contains 128,287 observations (an observation being data from one library for one year) and a maximum of 72 variables per observation. The data were originally compiled and published for the years 1987-2001. As a result of the compilation of this dataset, it became clear that these data could not be used to analyze trends and, as a result, this set of data formed the basis of another dataset, PLDF3.

For the time being, the PLDF2 data are not available. However for almost every use of these data, PLDF3 will be better. Other sets of data than those described here will be made available by request.

Background

NCES continues a rich tradition of compiling and publishing library data by the Department of Education and its predecessors that goes back to the seminal Public Libraries in the United States of America: Their History, Condition, and Management, (Washington, GPO: 1876) (commonly called the "1876 Report"). The report was published by the then Bureau of Education. There are actually older library data published by the US government but the subject of this history will have to wait for another time. A good place to start is Robert V. Williams, "The Making of Statistics of National Scope on American Libraries, 1836-1986: Purposes, Problems, and Issues." Libraries and Culture 26(2): 464-485 (Spring, 1991).

The name of this dataset, "PLDF2," reflects the fact that it is in a continuum. In the discussion below under variables, the reader will see that over the years, a number of variable names were used to describe essentially the same variables. PLDF2 records Zipcodes in a variable called ZIP which was the name used in the 1998-2001 datasets. However ZIP1 was used from 1991 to 1997, LIBZIP from 1988-1990 and FLDE in 1987. (Note that in the dataset, NCES variable names are in uppercase while variables I created are lowercase and italicized here for clarity.) PLDF1 was the first merge of the various datasets and in it, all four of these variables representing the zipcode--indeed all the various variables used in each year--appear in the merged data. The result was a dataset that had something close to 150 variables and was about 230 megabytes in the master dataset in SAS format. Using it would require the analyst to sort through the documentation and variable names and to write code to collapse the same variables with its various names into one variable name. This is what PLDF2 does. It has, as mentioned, a maximum of 72 variables and is 79 megabytes as a SAS dataset. Derivative versions of the dataset have different sizes and are discussed elsewhere.

Collapsing variables in this fashion makes using the dataset easier but it has the risk of introducing error. We know that to err is human and we have here a dataset of 128,287 observations with 72 variables or 9,236,664 things to get wrong. It is virtually certain that as this dataset is used and as new datasets are derived from it, that errors will be discovered and that PLDF2 will be changed to reflect what is learned. Thus, this dataset will likely change. A link to a revision history appears near the top of this page. Also, note that this documentation reflects a description of the concept behind it but the actual programs written on the raw data are the method by which the concept was executed. This secondary nature of documentation introduces another chance for error. Although the programs that generated this dataset are not documented here, they are available to anyone who wants them. I have the SAS code, log files, and orginal data and will be happy to discuss them and to correct errors.

A Methodological Caveat

The most likely source of error occurs in the correction of the imputations that appear in many of the sets of data. Conceptually, imputation includes a set of Statistical techniques used to create numbers where none exist. If a library does not report how much money it spent on its collection but Congress and the library community in considering legislation want a national estimate of how much was spent, what are we to do about the fact that some libraries did not report? Normal practice is to "impute" the missing data, that is, to make up data. When data are imputed, a guess is made by considering related factors. For instance, libraries of similar size, in the same region, with this many books and employees will spend $X and that number is entered into the NCES data when none is reported. One of NCES's missions is to supply such national estimates. However, people in the research community do not want imputed data in the data they analyze. One can well impute data from raw data or develop other methods for imputation but one should have clean data for analysis. After all, imputed data is based on a theory about relationships between variables. Should we use our theories to derive data or derive our theories from data? The research community's answer to that question is clear. NCES distributes imputed data but documents which data are imputed so it is conceptually possible to remove those specific observations. The program coding, alas, is rather more tricky. The intent was to change all imputed values to a period (.) used conventionally in SAS and other data formats as a place holder with no value and are normally referred to as "missing." As a result, such values do not interfere with calculations.

In addition, there is the custom of encoding variables with "-1" (and similar codes) used conventionally to mean that the value is missing or to distinguish between "Not Applicable" or "Not Available." It should be relatively easy to take out these values and replace them with a period indicating it is missing. It should, should it not? But, in examining the raw data one finds all sorts of things. For numerical variables, such as expenditures, it is important to get out the -1s so they do not interfere with calculations and the method I chose was to look for anything less than 0 and replace it with the period. I wonder about that method but it has so far seemed to yield the correct results. Character variables, however are a different problem. In examining the raw data, -1 has appeared as "-1   " (-1 and three spaces in ZIP), "   -1" (three spaces and -1), "-1-1-1-1-1-1-1-1" (in another variable), "---------", and all manner of things. (For example, see these cases where ADDRESS="M" may mean missing.) Given they are character and not numeric variables, each must be separately discovered and changed exactly because -1 has no numeric meaning--it is just some characters. On the other hand, the harm here is less because one does not make calculations on character fields--that is, there is no average street name.

A Peek Into Future Datasets

PLDF2 is, in effect, the base from which other longitudinal datasets derived from the Public Library Data File can be constructed. Although the variables have been collapsed and that should aid analysis, we will still be left with a set of data with different numbers of libraries each year. Towards that end, we need to know, for instance, which libraries have been in any set of data for the whole time. Libraries change over time: there are marriages, divorces, and new library systems are formed. In analyzing a set of libraries over time, a key is to have the same set of libraries so that observed changes are not a result of adding or subtracting new libraries from the set of data. We would want, of course, to analyze all in some circumstances but we also will want to analyze the same libraries over time in other cases.

How to do that? Well, we need a unique number that is a key to each institution and will use that to write code to sort out libraries in each year or in key years. The variable FSCSKEY is almost that variable but it is not. I initially looked at all libraries in four states in order by FSCSKEY and had to reorder them by other means to discover that the FSCSKEY was not consistent since 1987. One library had five but it appears that in the last five or so years, these keys are consistent. A further discussion is available on the main page of PLDF3's documentation.

Then we can begin work on PLDF4 which will be a smaller dataset but will include data from public libraries in all years of the data going back as far as we can go. We have sample data from LIBGIS from the 1970s, and large collections of universe data going back to the 19th century. If we have 100 such libraries, it would be remarkable and valuable.

Future additions to these datasets will also include mapping the variables in PLDF2 to the NISO Z39.7 Library Statistics standard and other related standards, and including XML descriptors. The XML descriptors will aid making the data available over the Web and in a manner that requires less technical knowledge than PLDF2 does. A paper A New Direction for Library Data? discusses this notion in detail. Briefly, it envisions "the development a comprehensive and systematic plan to encompass all variables, from all open source surveys, for all years, from all types of libraries and other information agencies, in one system based, ultimately, on XML or whatever is developed from it." Standards are important to ensure interoperability and comparability with other data collections. Among the criticisms of our data is that they are Balkanized and the work of standards bodies is an effort to overcome this problem.

Basics of PLDF2

Variables

The table lists the variables that were dealt with in constructing PLDF2. The intent was to group all like variables in the various years in one variable name. Rarely was there any ambiguity in reading the documentation from year to year. In one case, HRS_OPEN, in 1987-88, the variable was defined as a count from one week and from 1989, the number is annual. For 1987-88, as is noted below, I multiplied the number by 52 in order to convert the numbers for those first two years to something (roughly) comparable to the data from later years.

The documentation proved helpful and only one obvious error was discovered. According to the documentation, in 1989 the variables PSUNDUP and SVCINUSE were not reported but they are in the dataset. It is not clear whether they were reported for all institutions or only those who reported in spite of the data's not being collected or whether the documentation is in error and the data are collected appropriately.

Two variables have been added to each year's data in order to ease processing: state and year. Year was hard coded in one of the programs and is the nominal year of collection and the same as the year that appears on publications. State is the two-digit postal code. This variable was made from various variables in the dataset as noted below.

Five variables were discarded completely but could be readily included in a separate dataset if someone were to want any of them. They are HRSCD, POPCD, PUBSEQ, SEQ, and STPOP. They are discussed below. Of course, when variables such as ZIP replaced FLDE and LIBZIP, those replaced variables were discarded also but the data they reported are in PLDF2 in ZIP. The data from these discarded five are not in PLDF2.

An Excel spreadsheet presents these variables in another format.

The following table has the variable in the topical order used in 2001. The convention followed is to use the 2001 variable name except where the data are no longer collected and the short definitions and notes come, largely, from the documentation for 2001. The wording differs on occasion for variables documented in the earlier years but the definitions overall are pretty consistent.

PLDF2 Data

Complete data are available in a number of formats.

Thanks to--

Elaine Kroe for the SAS datasets of the 1991-2000 data and to Libby Law for the data from 1990.

Bob Molyneux

April 19, 2004