▼Datasets Core
►2 Reference
On this page:
2.1 Registry and loading
dataset-names
dataset-packages
dataset-info
load-dataset-table
2.1.1 Registries
2.2 Downloads and the cache
dataset-cache-directory
dataset-cached?
fetch-dataset
clear-dataset-cache
2.2.1 Converted sets
2.2.2 Disabling downloads
2.3 Tables
dataset-table
dataset-table?
dataset-table-names
dataset-table-columns
dataset-table-column
dataset-table-row-count
dataset-table-info
2.4 Missing values
dataset-missing
dataset-missing?
2.5 Metadata fields
9.3.900
Racket
  top  contents    up   

2 Reference🔗ℹ

 (require datasets/core) package: datasets-core

This section specifies the core API. For a step-by-step introduction, see the Guide. Contracts below describe accepted arguments and results using the notation of The Racket Reference.

2.1 Registry and loading🔗ℹ

Procedures in this module that discover registries or take a dataset name (dataset-names, dataset-packages, dataset-info, load-dataset-table, fetch-dataset, dataset-cached? and clear-dataset-cache) are not thread-safe: concurrent calls can silently omit the datasets of other packages. Callers using several threads must serialize these calls.

procedure

(dataset-names [#:package package]) → (listof symbol?)

  package : (or/c #f string?) = #f
Returns the identifiers of every dataset declared by an installed registry, sorted alphabetically, including fetch-tier datasets. If package is a string, returns only that package’s datasets; a package that declares no installed dataset raises exn:fail:contract listing the available packages. Conflicting or malformed registries raise exn:fail; see Registries.

Example:
> (dataset-names #:package "datasets-core")

'(air-passengers

  airquality

  anscombe

  breast-cancer

  diabetes

  faithful

  iris

  mtcars

  plant-growth

  titanic

  tooth-growth

  us-arrests

  wine)

procedure

(dataset-packages) → (listof string?)

Returns the names of the packages that declare at least one installed dataset, sorted by string<?. Every identifier in (dataset-names) belongs to exactly one of them, recorded in its metadata as package.

Examples:
> (dataset-packages)

'("datasets-core")

> (hash-ref (dataset-info 'iris) "package")

"datasets-core"

procedure

(dataset-info name) → (and/c hash? immutable?)

  name : symbol?
Returns deeply immutable metadata for name. An unknown identifier raises exn:fail:contract. Metadata describes the complete source dataset; it is independent of any column selection. See Metadata fields for its fields.

procedure

(load-dataset-table name    
  [#:columns columns    
  #:download? download?]) → dataset-table?
  name : symbol?
  columns : (or/c #f (non-empty-listof string?)) = #f
  download? : boolean? = #t
Reads the observations for name and returns a table. Bundled data files are resolved relative to the installed module and read when this procedure is called; the current directory does not affect loading.

If columns is #f, all columns are returned in source order. Otherwise, it must be a nonempty list of distinct known column names. Its order determines the table’s column order. Row order and row count are preserved. Unknown datasets, unknown columns, empty selections, duplicate columns, and invalid argument types raise exn:fail:contract.

The result contains immutable column vectors. Strings inside those vectors are also immutable. Missing observations are dataset-missing. Repeated loads are equal; callers should not rely on object identity.

Bundled data is stored gzip-compressed and decompressed on each call; a fetch-tier dataset is read from (dataset-cache-directory) in the same form, after downloading it as by fetch-dataset if it is not cached. Column arguments are checked before any download. If the dataset is not cached and download? is #f, raises exn:fail naming the dataset and how to fetch it (see Disabling downloads). Every load checks the data’s SHA-256 against the sha256 metadata, for bundled and cached data alike, and returns a table only when it matches; a mismatch or an unreadable file raises exn:fail naming the file, in preference to any parse error. For a cached file that cannot be read or is not the pinned download, the error gives clear-dataset-cache as the remedy. For a pinned, readable one, and for any data that matches its SHA-256 but does not parse, it names the dataset and package instead: the registry entry is wrong.

Examples:
> (dataset-table-names
   (load-dataset-table 'mtcars #:columns '("mpg" "model")))

'("mpg" "model")

> (load-dataset-table 'iris #:columns '("unknown"))

load-dataset-table: unknown column

  column: "unknown"

  available: '("sepal-length" "sepal-width" "petal-length"

"petal-width" "species")

2.1.1 Registries🔗ℹ

A registry is a file mapping dataset names to metadata hashes. Any installed package can declare registries with the datasets-registries key in a collection-level info.rkt: a list of paths relative to that file’s directory. Each dataset’s data is data/name.rktd.gz beside its registry: gzip-compressed UTF-8 text of an opening parenthesis and newline, each row on its own line as two spaces followed by its write form, then a closing parenthesis and newline. Loading expects this form, with exact integers of at most 1000 digits; other text may be rejected. The registries of all installed packages are merged when first used. Each later call checks the registry files and declaring info.rkt files of the packages already found, and merges again when one changes. The installed package index is read again when the collection search path changes, as with current-library-collection-paths, when raco setup runs in the same process, or after a call whose discovery failed or was interrupted, so a package installed after the first call is usually not seen until the process restarts. Package datasets-core is reserved for the registries that datasets/core-data/info.rkt lists. Another installation’s datasets/core-data collection is skipped unread; any other registry declaring datasets-core is an error naming its file. A name declared twice is an error naming both packages. A malformed registry is an error naming its file. Either error makes every registry operation raise until the package at fault is fixed or removed. A fetch-tier entry has no data file; see Downloads and the cache.

When reading the installed package index fails with a permission error, or a security guard denies it as in a default racket/sandbox evaluator, only the datasets-core registries are used: a warning is logged to the datasets topic, an unknown dataset or package error gives the reason, and the next call tries discovery again. Any other failure to read the index raises.

2.2 Downloads and the cache🔗ℹ

Fetch-tier datasets are not stored in any package. Their data is fetched once into the cache directory and read from there, and their license gives the upstream terms of use rather than of redistribution, or NOASSERTION when the source states none, with the basis in notes. A dataset whose fetch format is rktd.gz is downloaded ready to read: its fetch hash has urls (mirrors, tried in order) and the sha256 of the download. A converted set is built locally from upstream files (see Converted sets). Other fetch keys are ignored. A cached file is named by a SHA-256 its registry entry pins, of the download for rktd.gz and of the converted content for a converted set, so a registry update that changes the data fetches it again rather than reusing the old file.

procedure

(dataset-cache-directory) → complete-path?

Returns the directory holding downloaded datasets: the value of DATASETS_HOME when it is set and nonempty, otherwise (build-path (find-system-path 'cache-dir) "racket-datasets"). The variable is read on every call. A relative value makes this procedure, fetch-dataset, dataset-cached?, clear-dataset-cache, and loading a fetch-tier dataset raise exn:fail; loading a bundled dataset is unaffected. The directory is created by the first download, not by this procedure.

Example:
> (parameterize ([current-environment-variables
                  (make-environment-variables #"DATASETS_HOME" #"/srv/racket-datasets")])
    (dataset-cache-directory))

#<path:/srv/racket-datasets>

procedure

(dataset-cached? name) → boolean?

  name : symbol?
Returns #t if loading name needs no download: always for a bundled dataset, and for a fetch-tier dataset when its file is in (dataset-cache-directory). Never contacts a server and does not read the file; load-dataset-table checks its SHA-256. An unknown dataset raises exn:fail:contract. For a converted set only the converted file counts; upstream files are never kept.

Example:
> (dataset-cached? 'iris)

#t

procedure

(fetch-dataset name    
  [#:idle-timeout idle-timeout]) → complete-path?
  name : symbol?
  idle-timeout : (and/c real? positive?) = 60
Makes name’s data available locally and returns the path of the gzip-compressed .rktd file load-dataset-table reads: the file in its package for a bundled dataset, the cached file for a fetch-tier one. A fetch-tier dataset that is not cached is downloaded from the urls in its fetch metadata, in order, until one succeeds. Each download goes to a temporary file beside its destination and is renamed into place only after its SHA-256 matches the metadata, so an interrupted or failed download never leaves a partial file under the cached name. A URL whose body delivers no data for idle-timeout seconds fails. The idle timeout bounds waits for body data only; connecting, the TLS handshake and waiting for response headers, including redirects, have no timeout. If every URL fails, raises exn:fail listing each URL and its failure, with the expected and actual SHA-256 for a mismatch. Raises exn:fail without contacting a server when the format is not supported; an unknown dataset raises exn:fail:contract.

For a converted set the converter is loaded first, then each of its files is downloaded as above to a temporary file, and the converter’s rows become the cached file. The temporary files are deleted whether or not this succeeds. A file whose every URL fails, a converter that cannot be loaded or raises, a malformed row, or content whose SHA-256 differs from the sha256 metadata raises exn:fail naming the dataset and the file or converter, and nothing is cached. A contract violation from the converter propagates unchanged (see Converted sets).

Example:
> (file-exists? (fetch-dataset 'iris))

#t

procedure

(clear-dataset-cache [name]) → void?

  name : (or/c #f symbol?) = #f
Deletes downloaded files from (dataset-cache-directory): those of name, or, when name is #f, those of every dataset, including datasets whose package is no longer installed. Only files the cache created (verified downloads, converted files and leftover temporary files) are removed; other files and the directory itself are left alone. A bundled dataset has nothing to delete. An unknown name raises exn:fail:contract. Clearing a dataset while another program downloads it makes that download fail.

2.2.1 Converted sets🔗ℹ

A fetch-tier dataset whose fetch format is convert is built on first use from upstream files. Its fetch hash has files, a nonempty list of hashes each with a name, urls (mirrors, tried in order) and that file’s sha256, and converter, a hash with a module (a collection-based module path, as a string) and a binding it provides. Its source-sha256 is the SHA-256 of the files’ sha256 values joined by newlines, in files order, and its source-url may name the upstream page:

(hash "format" "convert"
      "files" (list (hash "name" "covtype.data.gz"
                          "urls" (list "https://archive.ics.uci.edu/ml/machine-learning-databases/covtype/covtype.data.gz"
                                       "https://ndownloader.figshare.com/files/5976039")
                          "sha256" "614360d0257557dd1792834a85a1cdebfadc3c4f30b011d56afee7ffb5b15771"))
      "converter" (hash "module" "datasets/fetch-private/convert" "binding" "covtype-rows"))

The converted rows are written in the form bundled data is stored in (see Registries). The dataset’s sha256 pins the SHA-256 of that text, which is checked before the file is cached and again on every load. A converter change that alters the data therefore takes effect only with a registry update that pins the new content.

converter

(converter files info) → sequence?

  files : (hash/c string? complete-path? #:immutable #t)
  info : (and/c hash? immutable?)
The protocol a converter binding follows. files maps each file’s name to its verified download and info is the dataset’s dataset-info. The result is a sequence, possibly lazy, of rows in source order, each a list of column-count exact integers, flonums, strings or 'missing, within the limits of the data format above. A contract violation raised while loading the converter or producing rows, such as a binding that is not a procedure of two arguments or a result that is not a sequence, propagates unchanged; any other exn:fail is reported as described for fetch-dataset.

2.2.2 Disabling downloads🔗ℹ

A download happens only when a fetch-tier dataset is not cached. #:download? #f disables it for one call to load-dataset-table; a refused download raises exn:fail naming the dataset, the expected cache file and how to fetch it. Bundled and cached datasets load as usual. A converted set is neither downloaded nor converted while downloads are disabled. Every load checks the data’s SHA-256, for both tiers; a corrupt cached file raises exn:fail naming clear-dataset-cache as the remedy.

Downloads are logged at level 'info with topic 'datasets; set PLTSTDERR to info@datasets to see them.

2.3 Tables🔗ℹ

procedure

(dataset-table names columns [#:info info]) → dataset-table?

  names : (non-empty-listof string?)
  columns : (non-empty-listof (or/c list? vector?))
  info : hash? = (hash)
Returns an immutable table with one column per name, in order. Each column is a list or vector of cells: exact integers from (- (expt 2 63)) to (sub1 (expt 2 63)), flonums, strings or dataset-missing, its other cells all numbers or all strings. Names are distinct and columns equally long; that length is the row count, which may be zero.

The metadata (dataset-table-info) is info over defaults, each key of info replacing its default whole, unchecked against the data. The defaults give the row and column counts, #f or empty for the source fields of Metadata fields, and one columns description per name whose type is "integer" when every nonmissing cell is an exact integer, "string" when every one is a string and "real" otherwise. info is an equal?-based hash table with string keys whose values are strings, numbers, booleans, symbols, lists and such hash tables, without cycles; its columns, if present, is a list of hash tables each with a string name.

The table is a copy: later changes to the arguments do not reach it. A copy taken through an impersonator is checked again, and one that no longer passes raises exn:fail:contract. Rebuilding a loaded or generated table from its names, columns and metadata gives an equal? table.

Examples:
> (define t (dataset-table '("x" "label") (list (list 1.5 dataset-missing 3) '("a" "b" "a"))))
> (dataset-table-row-count t)

3

> (map (lambda (c) (hash-ref c "type")) (hash-ref (dataset-table-info t) "columns"))

'("real" "string")

> (define iris (load-dataset-table 'iris))
> (equal? iris (dataset-table (dataset-table-names iris)
                              (vector->list (dataset-table-columns iris))
                              #:info (dataset-table-info iris)))

#t

> (dataset-table '("a" "a") '((1) (2)))

dataset-table: contract violation;

 duplicate column

      names: '("a" "a")

  in: (->i

       ((names (non-empty-listof string?))

        (columns

         (names)

         (non-empty-listof dataset-column/c)))

       (#:info (info dataset-info/c))

       #:pre/name

       (names)

       "duplicate column"

       (not (check-duplicates names))

       #:pre/name

       (names columns)

       "one column per name"

       (= (length names) (length columns))

       #:pre/name

       (columns)

       "every column the same length"

       (equal-lengths? columns)

       (result dataset-table?))

  contract from:

      <pkgs>/datasets-core/datasets/core.rkt

  blaming: top-level

   (assuming the contract is correct)

  at: <pkgs>/datasets-core/datasets/core.rkt:36:11

procedure

(dataset-table? value) → boolean?

  value : any/c
Returns #t if value is a dataset table, #f otherwise. Tables come from load-dataset-table, the datasets/synthetic generators and dataset-table; there is no mutator.

procedure

(dataset-table-names table) → (listof string?)

  table : dataset-table?
Returns the selected column names in order. Each string is immutable.

procedure

(dataset-table-columns table) → (and/c vector? immutable?)

  table : dataset-table?
Returns an immutable vector of immutable column vectors, in the same order as dataset-table-names. Each column has (dataset-table-row-count table) elements.

procedure

(dataset-table-column table name) → (and/c vector? immutable?)

  table : dataset-table?
  name : string?
Returns the column named name from table. A name that is not among the selected columns raises exn:fail:contract.

procedure

(dataset-table-row-count table) → exact-nonnegative-integer?

  table : dataset-table?
Returns the number of source observations, including rows with missing values.

procedure

(dataset-table-info table) → (and/c hash? immutable?)

  table : dataset-table?
Returns the full source metadata. Its "column-count" may exceed the number of selected columns in table. For a table from dataset-table, see there.

2.4 Missing values🔗ℹ

value

dataset-missing : any/c

procedure

(dataset-missing? value) → boolean?

  value : any/c
dataset-missing is the missing-value singleton. The predicate recognizes that value by identity. It returns #f for #f, zero, NaN, strings, and symbols, including 'missing. The singleton is not a number and must be handled explicitly before arithmetic.

2.5 Metadata fields🔗ℹ

All metadata hashes, lists, and strings are immutable. Dataset hashes have these string keys:

Key

  

Meaning

name

  

Dataset identifier as a string.

rows

  

Number of source rows.

column-count

  

Number of source columns, including identifiers.

columns

  

Ordered list of column-description hashes.

identifiers

  

Column names identifying observations; an empty list if none.

suggested-targets

  

Optional response or label column names; an empty list if unspecified.

source-url

  

URL of the pinned upstream archive or page.

source-version

  

Upstream version or hash-pinned snapshot description.

source-sha256

  

Hexadecimal SHA-256 of the upstream archive.

sha256

  

Hexadecimal SHA-256 of the normalized .rktd content's UTF-8 bytes, before compression.

license

  

Accepted upstream terms.

citation

  

Source attribution and bibliographic citation.

notes

  

Transformations, representation choices, and historical quirks.

time-series

  

Time-series description hash, or #f.

tier

  

bundled (shipped with its package) or fetch (downloaded on first use).

package

  

Name of the package whose registry declares the dataset.

fetch

  

#f for bundled data, a hash for fetch-tier data; see Downloads and the cache.

A package registry may add further keys, which core passes through unchecked to dataset-info; datasets-islr, for example, adds islr2-chapters.

Each column-description hash has name, original-name, type, levels, unit, description, and missing-count keys. Semantic types are strings: integer, real, categorical, identifier, or string. Category levels are ordered strings; a noncategory has an empty level list. A unit of #f means unspecified. The missing count refers to that column in the complete source dataset.

A time-series hash has frequency (observations per unit of the first index column, such as 12 for monthly data indexed by year and month), index (the names of the index columns, outermost first), and start and end (the index values of the first and last rows, one per index column).

  top  contents    up