Parsing Federal Procurement Data System

How can we analyze the entire procurement database of the United States Federal Government? Sometimes, even when data is provided, using that data can require some technical intervention.

Not All Data is Easy to Use!

The Federal Procurement Data System is record of all contracts awarded by the Federal Government to contractors, and is available online to peruse by the public.  The system is accessible at http://www.fpds.gov, and provides a rich source for analysis of how government contracts are awarded, how these contracts have changed over time, and what the spending patterns are within a fiscal year.  While much economic data is presented in a format that facilitates analysis, the FPDS only provides limited access to the underlying data via its convenient web portal.

How Can I Analyze This Data?

When data is presented in an inconvenient format, it can be a challenge to import it into your favorite analysis package.  That's where we come in!  The research technology consultants at IQSS have the skills to write scripts that can download large amounts of data and reformat it in a way that makes sense, and is convenient for researchers.  Furthermore, we can walk you through the entire process and give you the tools that are necessary to perform this processing yourself.  From the FPDS, data can be exported several records at a time, and then converted to files that can be loaded into R, Stata and even Excel.  By leveraging technology to download and convert data, a script can download not just a small subset of the records, but every possible record!

What if it takes too long for my computer to download the files?

Consultants at IQSS can not only help write programs to download and convert data, but we can also utilize the technical infrastructure of the Research Computing Environment (RCE) to separate your important research problems from your personal laptop or desktop.  That way, even if your computer crashes, your process will keep running remotely, even if you disconnect from the Internet.  Programs on the RCE will not only run faster, but you can access them from multiple computers, 24 hours a day, from anywhere on the globe.  By harnessing technology, your research horizons can be greatly expanded, and our consultants can help work with you on your specific problems.

Get in touch!

E-mail us any time to set up a free consulting session: support@help.hmdc.harvard.edu

Data Processing

A rich database of contracts is a valuable resource, but only if it can be analyzed in a statistical software package that researchers are familiar with.  How can data be transferred from an online database into a spreadsheet or other format for convenient analysis?  Research technology consultants were able to use Python to download specific records and translate them into a common format.