data-frame🔗ℹ

A data frame is a data structure used to hold data in tables with rows and columns. It is meant for conveninent access and manipulation of relatively large data sets that fit in the process memory. The package also provides functions for loading and saving data from data frames from various formats, as well as several utilities and helper functions for statistical calculations, plotting and curve fitting.

The package was initially written to support data the data processing needs for ActivityLog2, and since that application mostly deals with time series based data, the package supports data which is naturally ordered (whether by a timestamp or some other data column). However, the package does support more general data which is not necessarily ordered and also supports efficient lookups using secondary indexes, allowing multiple traversal orders to be defined for the data.

With data science a popular topic, a package named data-frame inevitably brings the question of how does it compare with other implementations with similar names and descriptions. The package author does not know the answer to that question, except to say that any resemblance to other implementatins is purely coincidental.

1 Creating data frames🔗ℹ

The functions below allow constructing new data frames. They are mainly intended for writing functions that load data into data frames from different sources, rather for direct use in other programs. To load data into data frames, see df-read/csv or df-read/sql and for manually creating data frames, for/data-frame.

If series is null, the data frame will not contain any series, and its size will be determined by the first series added, see also df-add-series!.

cmpfn specifies an ordering function to use. If present, values can be looked up in this series using df-index-of and df-lookup. The data must be ordered according to this function

na specifies the "not available" value for this series, by default it is #f.

contractfn is a contract function. If present, all values in the data series, except NA values, must satisfy this contract, that is, the function must return #t for all values in the series.

If the data frame already contains a series with the same name, that series will be replaced.

See also df-row-count and make-series.

This operation will materialize all lazy series (see df-add-lazy!), making it a possibly costly operation if you have lazy series. Indices using old-name will also be updated to use new-name, but this operation is fast, and no re-indexing will be done.

If a series named name already exists in the data frame, it will be replaced.

See df-add-derived! for the parameter names.

If the series with name name is delayed, see df-add-lazy!, the series will be materialized first.

If the data in the series is not already sorted, and you want to lookup values using df-index-of or df-lookup, consider adding a secondary index using df-add-index!.

2 Loading and Saving Data🔗ℹ

The functions construct data frames by loading data from CSV files or by running an SQL query.

na represents the value in the CSV file that represents the "not available" value in the data frame. Strings equal? to this value will be replaced by #f. Alternatively, this can be a function which tests a string and returns #t if the string represents a NA value

When quoted-numbers? is #t, all quoted values in the CSV file will be converted to numbers, if possible. E.g. a value like "123" will be converted to the number 123 if quoted-numbers? is #t, but will remain the string "123" if the parameter is #f.

start and stop denote the beginning and end rows to be written out, by default all rows are written out.

3 Inspecting data series🔗ℹ

4 Working With Properties🔗ℹ

A data frame can have arbitrary data attached to it the form of key-value pairs, where the keys are symbols. This is usefull for attaching additional meta-data to data frames. The functions allow working with properties.

5 NA Values🔗ℹ

Data series support the concept that a value "not available". This is done using a special value, usually #t, but separate for each data series. The functions below allow working with “NA” values. The NA value is specified when the series is created using make-series.

6 Get and Set Individual Values🔗ℹ

7 Indexing And Row Lookup🔗ℹ

df-add-index! will create an index on a single series and use the lt function for comparing elements, this function must provide a strict less-than ordering, and suitable values would be < for numbers and string<? for strings, although any function can be defined.

The na-in-front? determines where the “NA” values in the series are placed. If it is #t, they are placed before all other values, otherwise they are placed at the end.

The df-add-index*! will define a multi-column index on all the series specified as a list, the lt parameter fot this function is a list of comparison functions, one for each columns. Such an index will sort by the first column, than, for all equal values in the first column, it will sort on the second column, and so on.

A multi-column index is mosty used for defining a multi-column iteration order, however, such an index can still be used for fast lookup for elements in the first indexed series.

The series must either be sorted, see df-set-sorted!, or an index must be defined for it, see df-add-index! and df-add-index*!, otherwise the calls will raise an error.

exact-match? defines what to do when the value(s) are not found in the data series. If it is #t and the value is not found, the functions return #f.

If exact-match? is #f, the value need not be present in the series, in that case, the returned index is the position of the first element which comes after the value, according to the sort function. This is the position where value could be inserted and still keep the series sorted. A value of 0 is returned if value is less or equal than the first value of the series and a value of (df-row-count df) is returned if the value is greater than all the values in series.

The series must be sorted (see df-set-sorted!), or else this will error.

The given value need not be present in the series. If this is the case, the lower bound and upper bound are the same and represent the position of the first element which comes before value, according to the sort function. This is the position in which the given value could be inserted and keep the series sorted.

The series must be either sorted, see df-set-sorted!, or have an index defined for it, see df-add-index! and df-add-index*!, otherwise an error is reported.

df-lookup* allows looking up multiple values and will return a list of the corresponding values.

These functions combine df-index-of and df-ref into a single function and has the same restrictions as df-index-of: the series must either be sorted or an index defined for it.

exact-match? has the same meaning as for df-index-of

An interpolation function can be specified, if the default one is not sufficient. This function is called once for each value resulting series (i.e. it interpolates values one by one).

8 Extracting Data🔗ℹ

start and stop indicate the first and one-before-last row to be selected. filter, when present, will filter values selected: only values for which the function returns #t will be added to the resulting vector.

If there is no filter specified, the resulting vector will have (- stop start) elements. If there is a filter, the number of elements depends on how many are filtered out by this function.

HINT: to select data from all rows where a colunm has a specified value, define an index for the column and select using the same value for both from and to.

A multi-column index can also be iterated using a single value, in which case only the first column of the index is used. For example, if you have a data frame with Country, City and CityPopulation series, you can define an index on Country and CityPopulation, than select City and CityPopulation on that index with a specific country as the from and to arguments. This will return all cities in that country ordered by their population.

The by-index* versions of these functins allow specifying a multi-value key for a multivalue index.

The function returns #t if all elements of item (which can be a vector or a list) are not #f.

9 Iterating Over Rows🔗ℹ

fn is a function of ether one or two arguments. If fn is a function with one argument, it is called with the values from all series as a single vector. If fn is a function of two arguments, it is called with the current and previous set of values, as vectors (this allows calculating "delta" values). I.e. fn is invoked as (fn prev current). If fn accepts two arguments, it will be invoked as (fn #f current) for the first element of the iteration.

df-map will iterate over rows in the data frame between start and stop positions, while df-map/by-index and df-map/by-index* will iterate in the order defined by the index index-name between from and to values in the indexed series.

See df-select/by-index* for a discution on the differneces between by-index and by-index* versions.

fn is a function of ether two or three arguments. If fn is a function with two arguments, it is called with the fold value plus the values from all series is passed in as a single vector. If fn is a function of three arguments, it is called with the fold value plus the current and previous set of values, as vectors (this allows calculating "delta" values). I.e. fn is invoked as (fn val prev current). If fn accepts two arguments, it will be invoked as (fn init-val #f current) for the first element of the iteration.

df-fold will iterate over rows in the data frame between start and stop positions, while df-fold/by-index and df-fold/by-index* will iterate in the order defined by the index index-name between from and to values in the indexed series.

See df-select/by-index* for a discution on the differneces between by-index and by-index* versions.