Using the Rdatasets API

This tutorial explains the usage of the Rdatasets API in Data Retriever. It includes both the CLI (Command Line Interface) commands as well as the Python interface for the same.

Command Line Interface

Listing the Rdatasets

The retriever ls rdataset command displays the Rdatasets.

$ retriever ls rdataset -h (gives listing options)

usage: retriever ls rdataset [-h] [-p P [P ...]] all

positional arguments:
    all           display all the packages present in rdatasets

optional arguments:
    -h, --help    show this help message and exit
    -p P [P ...]  display a list of all rdatasets present in the package(s)

Examples

This example will display all the Rdatasets present with their package name, dataset name and script name

$ retriever ls rdataset

List of all available Rdatasets

Package: aer              Dataset: affairs                   Script Name: rdataset-aer-affairs
Package: aer              Dataset: argentinacpi              Script Name: rdataset-aer-argentinacpi
Package: aer              Dataset: bankwages                 Script Name: rdataset-aer-bankwages
...
Package: vcd              Dataset: vonbort                   Script Name: rdataset-vcd-vonbort
Package: vcd              Dataset: weldondice                Script Name: rdataset-vcd-weldondice
Package: vcd              Dataset: womenqueue                Script Name: rdataset-vcd-womenqueue

This example will display all the Rdatasets present in the packages vcd and aer

$ retriever ls rdataset -p vcd aer

List of all available Rdatasets in packages: ['vcd', 'aer']
Package: vcd              Dataset: arthritis                 Script Name: rdataset-vcd-arthritis
Package: vcd              Dataset: baseball                  Script Name: rdataset-vcd-baseball
Package: vcd              Dataset: brokenmarriage            Script Name: rdataset-vcd-brokenmarriage
...
Package: aer              Dataset: affairs                   Script Name: rdataset-aer-affairs
Package: aer              Dataset: argentinacpi              Script Name: rdataset-aer-argentinacpi
Package: aer              Dataset: bankwages                 Script Name: rdataset-aer-bankwages
...

This example will display all the Rdatasets present in the package vcd

$ retriever ls rdataset -p vcd

List of all available Rdatasets in packages: ['vcd', 'aer']
Package: vcd              Dataset: arthritis                 Script Name: rdataset-vcd-arthritis
Package: vcd              Dataset: baseball                  Script Name: rdataset-vcd-baseball
Package: vcd              Dataset: brokenmarriage            Script Name: rdataset-vcd-brokenmarriage
...

This example will display all the packages present in rdatasets

$ retriever ls rdataset all

List of all the packages present in Rdatasets

aer         cluster   dragracer  fpp2           gt        islr     mass        multgee         plyr      robustbase  stevedata
asaur       count     drc        gap            histdata  kmsurv   mediation   nycflights13    pscl      rpart       survival
boot        daag      ecdat      geepack        hlmdiag   lattice  mi          openintro       psych     sandwich    texmex
cardata     datasets  evir       ggplot2        hsaur     lme4     mosaicdata  palmerpenguins  quantreg  sem         tidyr
causaldata  dplyr     forecast   ggplot2movies  hwde      lmec     mstate      plm             reshape2  stat2data   vcd

Downloading the Rdatasets

The retriever download rdataset-<package>-<dataset> command downloads the Rdataset dataset which exists in the package package. You can also copy the script name from the output of retriever ls rdataset.

Example

This example downloads the rdataset-vcd-bundesliga dataset.

$ retriever download rdataset-vcd-bundesliga

=> Installing rdataset-vcd-bundesliga
Downloading Bundesliga.csv: 60.0B [00:00, 117B/s]
Done!

The downloaded raw data files are stored in the raw_data directory in the ~/.retriever directory.

Installing the Rdatasets

The retriever install <engine> rdataset-<package>-<dataset> command downloads the raw data, creates the script for it and then installs the Rdataset dataset present in the package package into the provided engine.

Example

This example install the rdataset-aer-usmoney dataset into the postgres engine.

$ retriever install postgres rdataset-aer-usmoney

=> Installing rdataset-aer-usmoney
Downloading USMoney.csv: 1.00B [00:00, 2.52B/s]
Processing... USMoney.csv
Successfully wrote scripts to /home/user/.retriever/rdataset-scripts/usmoney.csv.json
Updating script name to rdataset-aer-usmoney.json
Updating the contents of script rdataset-aer-usmoney
Successfully updated rdataset_aer_usmoney.json
Updated the script rdataset-aer-usmoney
Creating database rdataset_aer_usmoney...

Installing rdataset_aer_usmoney.usmoney
Progress: 100%|█████████████████████████████████████████████████████████████████████████████████████████████| 136/136 [00:00<00:00, 2225.09rows/s]
Done!

The script created for the Rdataset is stored in the rdataset-scripts directory in the ~/.retriever directory.

Python Interface in Data Retriever

Updating Rdatasets Catalog

The function update_rdataset_catalog creates/updates the datasets_url.json in the ~/.retriever/rdataset-scripts directory, which contains the information about all the Rdatasets.

>>> import retriever as rt
>>> rt.update_rdataset_catalog()

Note

The update_rdataset_catalog function has a default argument test which is set to False. If test is set to True, then the contents of the datasets_url.json file would be returned as a dict.

Listing Rdatasets

The function display_all_rdataset_names prints the package, dataset name and the script name for the Rdatasets present in the package(s) requested. If no package is specified, it prints all the rdatasets, and if all is passed as the function argument then all the package names are displayed.

Note

The function argument package_name takes a list as an input when you want to display rdatasets based on the packages. If you want to display all packages names, set package_name argument to all (refer to the example below).

>>> import retriever as rt
>>>
>>> # Display all Rdatasets
>>> rt.display_all_rdataset_names()
List of all available Rdatasets

Package: aer              Dataset: affairs                   Script Name: rdataset-aer-affairs
Package: aer              Dataset: argentinacpi              Script Name: rdataset-aer-argentinacpi
Package: aer              Dataset: bankwages                 Script Name: rdataset-aer-bankwages
...
Package: vcd              Dataset: vonbort                   Script Name: rdataset-vcd-vonbort
Package: vcd              Dataset: weldondice                Script Name: rdataset-vcd-weldondice
Package: vcd              Dataset: womenqueue                Script Name: rdataset-vcd-womenqueue
>>>
>>> # Display all the Rdatasets present in packages 'aer' and 'drc'
>>> rt.display_all_rdataset_names(['aer', 'drc'])
List of all available Rdatasets in packages: ['aer', 'drc']
Package: aer              Dataset: affairs                   Script Name: rdataset-aer-affairs
Package: aer              Dataset: argentinacpi              Script Name: rdataset-aer-argentinacpi
Package: aer              Dataset: bankwages                 Script Name: rdataset-aer-bankwages
...
Package: drc              Dataset: spinach                   Script Name: rdataset-drc-spinach
Package: drc              Dataset: terbuthylazin             Script Name: rdataset-drc-terbuthylazin
Package: drc              Dataset: vinclozolin               Script Name: rdataset-drc-vinclozolin
>>>
>>> # Display all the packages in Rdatasets
>>> rt.display_all_rdataset_names('all')
List of all the packages present in Rdatasets

aer         cluster   dragracer  fpp2           gt        islr     mass        multgee         plyr      robustbase  stevedata
asaur       count     drc        gap            histdata  kmsurv   mediation   nycflights13    pscl      rpart       survival
boot        daag      ecdat      geepack        hlmdiag   lattice  mi          openintro       psych     sandwich    texmex
cardata     datasets  evir       ggplot2        hsaur     lme4     mosaicdata  palmerpenguins  quantreg  sem         tidyr
causaldata  dplyr     forecast   ggplot2movies  hwde      lmec     mstate      plm             reshape2  stat2data   vcd

Downloading a Rdataset

>>> import retriever as rt
>>> rt.download('rdataset-drc-earthworms')

Installing a Rdataset

>>> import retriever as rt
>>> rt.install_postgres('rdataset-mass-galaxies')

Note

For downloading or installing the Rdatasets, the script name should follow the syntax given below. The script name should be rdataset-<package name>-<dataset name>. The package name and dataset name should be valid.

Example:

• Correct: rdataset-drc-earthworms

• Incorrect: rdataset-drcearthworms, rdatasetdrcearthworms