R/get_eurostat_toc.R
get_eurostat_toc.RdDownload Table of Contents (TOC) of Eurostat datasets if it is not cached previously.
get_eurostat_toc(
mode = "xml",
cache = TRUE,
update_cache = FALSE,
cache_dir = NULL,
compress_file = TRUE,
lang = "en",
verbose = FALSE,
...
)a character string either xml or txt defining the download mode. Depending on the mode the 'xml' version or the 'text' version of the TOC is downloaded.
The deafult value is xml as it provides more information (e.g. number of values, short description and download links in different formats (SDMX, TSV))
a boolean whether to load/save the TOC from/in the cache or not. The default value is TRUE, so that the TOC is checked first in the cache and if does not exist then downloaded from Eurostat and cached.
a boolean to update cache or not. The default value is FALSE, so the cache is not updated. Can be set also with options(restatapi_update=TRUE)
a path to a cache directory. The default is NULL, in this case the TOC is cached in the memory (in the '.restatapi_env'). Otherwise if the cache_dir directory does not exist it creates the 'restatapi' directory in the temporary directory from tempdir() to save the RDS- file. Directory can also be set with option(restatapi_cache_dir=...).
a logical whether to compress the RDS-file in caching. Default is TRUE.
a character string either en, de or fr to define the language version for the table of contents. The default is en - English.
A boolean with default FALSE, so detailed messages (for debugging) will not printed.
Can be set also with options(restatapi_verbose=TRUE)
parameter to pass on the load_cfg function
A data table with the following columns:
title | The name of dataset/table in the language provided by the lang parameter |
code | The codename of dataset/table which can be used as id in other functions |
type | The type of information: 'dataset' or 'table' |
lastUpdate | The date when the data was last time updated for tables and datasets |
lastModified | The date when the structure of the dataset/table was last time modified |
dataStart | The start date of the data in the dataset/table |
dataEnd | The end date of the data in the dataset/table |
values | The number of values in the dataset/table, and it is filled only if the download
mode is "xml" |
unit | The unit name for tables in the language provided by the lang parameter, for
dataset it is empty and this column exists only if the download mode is "xml" |
shortDescription | The short description of the values for tables in the language provided by the
lang parameter, for dataset it is empty and this column exists only if the download mode is "xml" |
metadata.html | The link to the metadata in html format, and this column exists only if the
download mode is "xml" |
metadata.sdmx | The link to the metadata in SDMX format, and this column exists only if the
download mode is "xml" |
downloadLink.tsv | The link to the whole dataset/table in tab separated values format in the bulk
download facility and this column exists only if the download mode is "xml" |
The TOC is downloaded from Eurostat websites through the REST API for the xml (default) version or from the bulk download facilities for txt version.
From the downloaded TOC the values in the 'code' column can be used as id in the get_eurostat_dsd, get_eurostat_raw, get_eurostat_bulk, and get_eurostat_data functions.
For more technical information see the detailed documentation of the API.
# \dontshow{
if (parallel::detectCores()<=2){
options(restatapi_cores=1)
}else{
options(restatapi_cores=2)
}
# }
# \donttest{
if (!(grepl("amzn|-aws|-azure ",Sys.info()['release']))) options(timeout=2)
toc_xml<-get_eurostat_toc(cache=FALSE,verbose=TRUE)
#>
#> get_eurostat_toc - API version:2 - number of cores:2
#> get_eurostat_toc - Downloading https://ec.europa.eu/eurostat/api/dissemination/catalogue/toc/xml
#> get_eurostat_toc - class(xml_leafs): xml_nodeset
#> get_eurostat_toc - number of nodes: 9148
#> get_eurostat_toc - number of cores: 2
head(toc_xml)
#> title code type
#> <char> <char> <char>
#> 1: Current account - quarterly data ei_bpm6ca_q table
#> 2: Financial account - quarterly data ei_bpm6fa_q table
#> 3: Current account - monthly data ei_bpm6ca_m table
#> 4: Financial account - monthly data ei_bpm6fa_m table
#> 5: International investment position - quarterly data ei_bpm6iip_q table
#> 6: Consumers - monthly data ei_bsco_m dataset
#> lastUpdate lastModified dataStart dataEnd values unit shortDescription
#> <char> <char> <char> <char> <num> <char> <char>
#> 1: 2024.02.20 2024.02.20 1992-Q1 2023-Q4 271230
#> 2: 2024.02.20 2024.02.20 1992-Q1 2023-Q4 48728
#> 3: 2024.03.21 2024.03.21 1991-01 2024-01 220443
#> 4: 2024.03.21 2024.03.21 1991-01 2024-01 71324
#> 5: 2024.01.12 2024.01.12 1992-Q4 2023-Q3 59859
#> 6: 2024.02.28 2024.02.28 1980-01 2024-02 308925
#> metadata.html
#> <char>
#> 1: https://ec.europa.eu/eurostat/cache/metadata/en/ei_bp_esms.htm
#> 2: https://ec.europa.eu/eurostat/cache/metadata/en/ei_bp_esms.htm
#> 3: https://ec.europa.eu/eurostat/cache/metadata/en/ei_bp_esms.htm
#> 4: https://ec.europa.eu/eurostat/cache/metadata/en/ei_bp_esms.htm
#> 5: https://ec.europa.eu/eurostat/cache/metadata/en/ei_bp_esms.htm
#> 6: https://ec.europa.eu/eurostat/cache/metadata/en/ei_bcs_esms.htm
#> metadata.sdmx
#> <char>
#> 1: https://ec.europa.eu/eurostat/api/dissemination/files?file=metadata/ei_bp_esms.sdmx.zip
#> 2: https://ec.europa.eu/eurostat/api/dissemination/files?file=metadata/ei_bp_esms.sdmx.zip
#> 3: https://ec.europa.eu/eurostat/api/dissemination/files?file=metadata/ei_bp_esms.sdmx.zip
#> 4: https://ec.europa.eu/eurostat/api/dissemination/files?file=metadata/ei_bp_esms.sdmx.zip
#> 5: https://ec.europa.eu/eurostat/api/dissemination/files?file=metadata/ei_bp_esms.sdmx.zip
#> 6: https://ec.europa.eu/eurostat/api/dissemination/files?file=metadata/ei_bcs_esms.sdmx.zip
#> downloadLink.tsv
#> <char>
#> 1: https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/EI_BPM6CA_Q?format=tsv&compressed=true
#> 2: https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/EI_BPM6FA_Q?format=tsv&compressed=true
#> 3: https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/EI_BPM6CA_M?format=tsv&compressed=true
#> 4: https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/EI_BPM6FA_M?format=tsv&compressed=true
#> 5: https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/EI_BPM6IIP_Q?format=tsv&compressed=true
#> 6: https://ec.europa.eu/eurostat/api/dissemination/sdmx/2.1/data/EI_BSCO_M?format=tsv&compressed=true
toc_txt<-get_eurostat_toc(mode="txt", lang="de")
#> There is a warning by the download of the txt TOC file. Run the same command with verbose=TRUE option to get more info on the issue.
#> There is a warning by the reading of the downloaded txt TOC file. Run the same command with verbose=TRUE option to get more info on the issue.
head(toc_txt)
#> NULL
options(timeout=60)
# }