General Information

Data Carpentry is an organization with the goal of teaching basic concepts, skills and tools for working more effectively with data. The rapid generation of large amounts of data is fundamentally changing how research is done. This deluge of data presents great opportunities, but also many challenges in managing, analyzing and sharing data. Data Carpentry teaches the skills that will enable researchers to be more effective and productive, to learners with little to no prior computational experience. In particular, as large corpora of text are becoming digitized and broadly available, it is now possible to take a quantitative approach to analyzing text to address questions in public policy, develop guidelines for implementation or assess the effectiveness of policy. As many of the types of data collected for this type of assessment are text-based, in the form of free text responses in surveys, written reports, newspaper articles or a variety of other sources, the ability to conduct text mining and analysis effectively and reproducibly will have an impact on the ability to understand and contribute knowledge to policy making.

Data Carpentry workshops are for any researcher who have data they want to analyze, and no prior computational experience is required. This hands-on workshop teaches basic concepts, skills and tools for working more effectively with data. We will cover an introduction to R, how to use the tm package to convert text into numbers, and how to analyze and visualize the data in R. By the end of the workshop learners should be able to more effectively manage and analyze data and be able to apply the tools and approaches directly to their ongoing research.

Participants should bring their laptops and plan to participate actively. Before the workshop you will need to install R and RStudio. Please see the installation instructions for more details.

Data Carpentry's aim is to teach researchers basic concepts, skills, and tools for working with data so that they can get more done in less time, and with less pain.

Schedule

Before the workshop

Updates will be posted to this website as they become available.

The data file for the workshop are available at the following link (right-click and 'save as' to download to your computer)

The file of additional code for cluster analysis, topic modelling, reading in PDFs, etc. can be downloaded here (right-click and 'save as' to download to your computer)

The etherpad for this workshop can be found here

Instructors: Ben Marwick, Tracy Teal

Who: The course is aimed at faculty, research staff, postdocs, graduate students, advanced undergraduates, and other researchers in any field. No prior computational experience is required.

Requirements: Data Carpentry's teaching is hands-on, so participants are encouraged to bring in and use their own laptops to insure the proper setup of tools for an efficient workflow once you leave the workshop. (We will provide instructions on setting up the required software several days in advance, and the classroom will have computers with the software installed). There are no pre-requisites, and we will assume no prior knowledge about the tools. Participants are required to abide by Software Carpentry's Code of Conduct.

Contact: Please email tkteal@datacarpentry.org for questions and information not covered here.

Twitter: #datacarpentry @datacarpentry

Acknowledgements & Support

Data Carpentry is supported by the Gordon and Betty Moore Foundation and a partnership of several NSF-funded BIO Centers (NESCent, iPlant, iDigBio, BEACON and SESYNC) and Software Carpentry, and is sponsored by the Data Observation Network for Earth (DataONE). The structure and objectives of the curriculum as well as the teaching style are informed by Software Carpentry.

Additional Resources

R

• http://www.statmethods.net/ - good for data organization, basics stats and graphs
• http://www.gardenersown.co.uk/Education/Lectures/R/anova.htm - basic parametric and non-parametric stats
• http://www.cyclismo.org/tutorial/R/index.html - R tutorial
• R in Action - good book as an R reference
• http://www.twotorials.com/
• http://www.r-bloggers.com/
• http://tryr.codeschool.com/
• Advanced R Programming by Hadley Wickham
• Beginner's Guide to R from Computer World
• R for Journalists
• R Bloggers
• inside-R
• rOpenSci

Working with text in R

• http://cran.r-project.org/web/packages/tm/vignettes/tm.pdf - Introduction to the tm Package Text Mining in R
• http://www.jstatsoft.org/v25/i05/ - Text Mining Infrastructure in R (Journal of Statistical Software)
• http://cran.r-project.org/web/views/NaturalLanguageProcessing.html - CRAN Task View: Natural Language Processing
• http://handsondatascience.com/TextMiningO.pdf - Hands-On Data Science with R: Text Mining
• https://en.wikibooks.org/wiki/R_Programming/Text_Processing - R Programming/Text Processing
• http://www.matthewjockers.net/text-analysis-with-r-for-students-of-literature/ - Matthew L. Jockers' Text Analysis with R for Students of Literature
• https://heuristically.wordpress.com/2011/04/08/text-data-mining-twitter-r/ - Text Data Mining with Twitter and R

Plotting in R

• http://www.harding.edu/fmccown/r/ - Very simple graphs
• A variety of R gallery recommendations
• ggplot gallery - extensive and comprehensive; a great resource
• R Graphics Cookbook - highly recommended
• Some of R Graphics Cookbook plots - a set of some of the plots from the R Graphics Cookbook by Winston Chang
• R Graphical Manual - plots from apparently every R CRAN package

Getting help when working with R

• ? function, eg. ?mean to get built-in help on the mean function
• sos::findFn("rose diagram") searches all installed packages for the search term, using the sos package
• Most major packages come with vignettes that narrate typical uses of the package's core functions. Vignettes can be accessed with the command vignette(packagename).
• Google searches in the form: r help [your search terms or error message]
• A custom search engine of R resources: http://www.rseek.org/
• Graphical output from the examples in the documentation for all CRAN packages: http://rgm3.lab.nig.ac.jp/RGM
• All R package documentation (including CRAN, GitHub and Bioconductor packages) is online in an easy-to-ready format at http://www.rdocumentation.org/
• Stackoverflow is an online Q&A point-scoring website where questions and answers can be voted on to indicate their quality: http://stackoverflow.com/questions/tagged/r. Many highly skilled R programmers are active participants.
• You could send a message to the official r-help email list, but do be sure to read, follow and cite the posting guide. The list is also searchable.