16 Digs
(require denxi/dig) | package: denxi |
The digsite metaphor says that one uses a shovel to dig for artifacts. Despite putting the same amount of effort into each scoop, what the shovel encounters may change. This metaphor helps explain non-deterministic attempts to find a resource.
In Racket, a shovel is a shovel/c procedure and a dig is underway when such a procedure has program control.
The digsite metaphor implies some level of repeated effort (digging) towards uncertain results. For example, the built-in shovels for denxi/dig/filesystem resolve symbolic links as a way to “scoop” towards a possibly non-existent file. In a similar scenario, a server hosting data may respond differently to the same request, or issue redirects a shovel may follow.
struct
(struct $dig:no-artifact $dig (shovel-name hint))
shovel-name : (or/c string? symbol?) hint : any/c
value
= (-> any/c (subprogram/c artifact?))
procedure
(install-found-artifact hint link-path [ shovel]) → (subprogram/c (cons/c path-record? path-record?)) hint : any/c link-path : path-string? shovel : shovel/c = (current-shovel)
setting
DENXI_INSTALL_ARTIFACTS : (listof (list/c string? string?)) = ()
The path of a symbolic link to create with respect to (current-directory).
A string S, such that (current-shovel S) returns an artifact.
procedure
(find-artifact hint [shovel]) → (subprogram/c artifact?)
hint : any/c shovel : shovel/c = (current-shovel)
If hint is already an artifact, then it will be returned as-is. Otherwise, the result depends on the implementation of the shovel.
value
The default value is broken-shovel.
value
procedure
(dig-failure shovel-name hint) → subprogram?
shovel-name : (or/c string? symbol?) hint : any/c
procedure
(shovel-cons first second) → shovel/c
first : shovel/c second : shovel/c
procedure
(shovel-list shovel ...) → shovel/c
shovel : shovel/c
16.1 Memory Shovels
(require denxi/dig/memory) | package: denxi |
procedure
(make-memory-shovel contents) → shovel/c
contents : (hash/c any/c artifact?)
(define dig (make-memory-shovel (hash 'a (artifact (text-source "hello"))))) (dig 'a) ; OK (dig 'b) ; Fails
procedure
(make-memory-shovel/pkgdef contents defaults) → shovel/c contents : hash? defaults : package-query-defaults-implementation/c
contents is a nested hash table keyed first by provider names, then package names, then edition names, then both revision names and numbers as shown.
(define providers contents) (define packages (hash-ref providers "example.com")) (define editions (hash-ref packages "rpg")) (define revisions (hash-ref editions "directors-cut")) (artifact? (hash-ref revisions 0)) (artifact? (hash-ref revisions 28)) (artifact? (hash-ref revisions (hash-ref revisions "re-release")))
As indicated by the last line, each hash table may map keys to other keys in the same table.
e.g.
(hash-ref providers (hash-ref providers "alias.example.com"))
The shovel will recursively resolve keys in this way until it encounters a value of a type not used as a key, in observance of the digsite metaphor. If the search yields cyclic keys, this function will not terminate.
Package queries are autocompleted using defaults, and converted to canonical form with respect to a canon based on the hash table’s shape.
Example:
(define dig (make-memory-shovel/pkgdef (hash "default" "jon" "jon" (hash "calculator" (hash "scientific" (hash 0 (artifact #"...") 8 (artifact #"...") "initial" 0 "beta" 8)))))) (dig ":calculator:scientific:initial:beta")
16.2 Filesystem Shovels
(require denxi/dig/filesystem) | package: denxi |
You can bind shovels to filesystem directories. In observance of the digsite metaphor, the shovels defined herein always follow symbolic links.
procedure
(make-digest-file-path path chf) → path?
path : path-string? chf : symbol?
Used as a conventional location for unencoded digests for whatever is located at path.
procedure
(make-signature-file-path path) → path?
path : path-string?
Use as a conventional location for unencoded signatures.
procedure
(make-filesystem-shovel directory-path chf public-key-source) → shovel/c directory-path : complete-path? chf : symbol? public-key-source : source-variant?
relative-path complies with (and/c path-string? (not/c complete-path?))
(build-path directory-path relative-path) must exist as a readable file, or as a link to a readable file.
For example, assume /tmp/example exists as a file in a Unix-like system for a Denxi process to read. From here you can bind a dig procedure to /tmp like so.
(define S (make-filesystem-shovel "/tmp" 'md5 public-key-source))
From here, (S "example") will produce an artifact based on the existing file.
(artifact (file-source (string->path "/tmp/example")) #f #f)
If a file does not exist, then S behaves like broken-shovel. Note that S is non-deterministic, and may begin returning artifacts depending on the state of the directory at the time.
We specified 'md5 as an example CHF. To add integrity information for /tmp/example, create a /tmp/example.md5. It must hold the unencoded bytes of the digest produced from the contents of /tmp/example. If you do this, then (dig "example") will produce this enhanced artifact.
(artifact (file-source (string->path "/tmp/example")) (integrity 'md5 (file-source (string->path "/tmp/example.md5"))) #f)
Formally, integrity information appears in the artifact when (build-path directory-path (~a relative-path "." chf)) exists as a readable file.
Singature information may appear in another adjacent file. The complete directory listing should now appear as follows:
/tmp/example |
/tmp/example.md5 |
/tmp/example.md5.sig |
/tmp/example.md5.sig must hold the unencoded bytes for a signature produced from the digest’s contents. That is, for /tmp/example.md5.
In this scenario, (dig "example") will produce a complete artifact.
(artifact (file-source (string->path "/tmp/example")) (integrity 'md5 (file-source (string->path "/tmp/example.md5"))) (signature public-key-source (file-source (string->path "/tmp/example.md5.sig"))))
Formally, signature information appears in the artifact when (build-path directory-path (~a relative-path "." chf ".sig")) exists as a readable file.
procedure
(make-filesystem-shovel/pkgdef directory-path chf [ defaults]) → shovel/c directory-path : complete-path? chf : symbol?
defaults : package-query-defaults-implementation/c = default-package-query-defaults
For a given provider P, package name K, edition E, revision name A, and revision number N, the following invariants must hold for the files in directory-path.
If (build-path directory-path P "public-key") exists, it must be a readable file containing the public key for P. Corollary: K must not be equal? to "public-key".
file-name-string? returns #t for P, K, E, A, and N.
If (build-path directory-path P K E) exists, it is a readable directory suitable for use with make-filesystem-shovel.
If (build-path directory-path P K E N) exists, it is a readable file containing a package definition.
If (build-path directory-path P K E A) exists, then it must be a readable link pointing to exactly N. The link’s name must be a revision name from the corresponding package definition file (Digest and signature links are not required).
In terms of the above, here is a valid directory structure for a catalog.
~/denxi-catalog |
├── alice |
│ ├── default -> renderer |
│ ├── public-key |
│ ├── raw-input |
│ │ └── default |
│ │ ├── 0 |
│ │ ├── 0.sha3-384 |
│ │ ├── 0.sha3-384.sig |
│ │ ├── 1 |
│ │ ├── 1.sha3-384 |
│ │ ├── 1.sha3-384.sig |
│ │ ├── 5 |
│ │ ├── 5.sha3-384 |
│ │ ├── 5.sha3-384.sig |
│ │ ├── open-beta -> 5 |
│ │ └── xbox-controller-support -> 1 |
│ └── renderer |
│ ├── default -> vulkan |
│ ├── directx |
│ │ ├── 0 |
│ │ ├── 0.sha3-384 |
│ │ └── 0.sha3-384.sig |
│ └── vulkan |
│ ├── 0 |
│ ├── 0.sha3-384 |
│ ├── 0.sha3-384.sig |
│ └── default -> 0 |
├── default -> alice |
└── john |
├── calculator |
│ ├── default |
│ │ ├── 0 |
│ │ ├── 0.sha3-384 |
│ │ ├── 0.sha3-384.sig |
│ │ ├── 1 |
│ │ ├── 1.sha3-384 |
│ │ ├── 1.sha3-384.sig |
│ │ ├── 2 |
│ │ ├── 2.sha3-384 |
│ │ ├── 2.sha3-384.sig |
│ │ ├── initial -> 0 |
│ │ └── post-feedback -> 2 |
│ └── scientific |
│ ├── 0 |
│ ├── 0.sha3-384 |
│ ├── 0.sha3-384.sig |
│ ├── 1 |
│ ├── 1.sha3-384 |
│ ├── 1.sha3-384.sig |
│ └── zero-day-patch -> 1 |
├── calendar |
│ ├── chinese |
│ │ ├── 0 |
│ │ ├── 0.sha3-384 |
│ │ └── 0.sha3-384.sig |
│ └── gregorian |
│ ├── 0 |
│ ├── 0.sha3-384 |
│ └── 0.sha3-384.sig |
└── public-key |
In this example, the providers are Alice, a video game developer, and John, an office application developer.
There is a gap in revision numbers in Alice’s raw-input package. This is fine, because it reflects the possibility of missing information. The catalog will return the latest available match for queries like alice:raw-input:::open-beta.
Finally, each directory may contain a symlink named after a string one could use in a package query. You can leverage this to support getting the latest available version using queries like alice:raw-input.
16.3 HTTP Client Shovels
(require denxi/dig/http) | package: denxi |
procedure
(make-http-shovel base-url [ optional-chf optional-pubkey]) → shovel/c base-url : url-variant? optional-chf : (or/c #f symbol?) = #f optional-pubkey : (or/c #f source-variant?) = #f
The shovel accepts a url-variant? value as an argument. If a different value type is provided, the shovel behaves like broken-shovel. Otherwise, the shovel will create an artifact with http-source components w.r.t. base-url. Path and query string elements from the argument are appended to base-url’s respective fields. This implies that duplicate query string keys are allowed.
Invariant: All http-sources in created artifacts must yield the raw, unencoded bytes of the requested resource. This implies that HTTP response headers must include Content-Type: application/octet-stream.
If optional-chf is #f, then an output artifact will always lack both integrity information and signature information. Otherwise, the artifact will contain integrity information using the given CHF, along with a corresponding (~a "." optional-chf) extension to the last path element of the URL used to fetch the digest.
If optional-pubkey is #f, then an output artifact will always lack signature information. Otherwise, the artifact will contain signature information with optional-pubkey as the public key’s source, along with a corresponding (~a "." optional-chf ".sig") extension to the last path element of the URL used to fetch the signature.
Examples follow. Arguments to http-source are shown as URL strings for brevity. The actual output will include instances of url.
Given a shovel with no additional expectations...
(define dig (make-http-shovel "https://example.com"))
...both (dig "/some-file") and (dig (string->url "/some-file")) will produce
(artifact (http-source "https://example.com/some-file") #f #f)
Given a shovel with all expectations...
(define dig (make-http-shovel "https://example.com" 'sha3-384 (http-source "https://example.com/public.pem")))
...both (dig "/some-file") and (dig (string->url "/some-file")) will produce
(artifact (http-source "https://example.com/some-file") (integrity 'sha3-384 (http-source "https://example.com/some-file.sha3-384")) (signature (http-source "https://example.com/public.pem") (http-source "https://example.com/some-file.sha3-384.sig")))
16.3.1 Caveats for HTTP Shovels
Due to the volume of possible network conditions and API definitions for servers, denxi/dig/http provides no mechanisms for checking artifact availability, and no package query canon for package queries. Therefore, make-http-shovel will return artifacts that may produce no content when they are actually used. make-http-shovel only offers reasonable, minimal defaults for downloading compatible artifacts that follow the same naming conventions as make-filesystem-shovel.
If you want a client to check resource availability in advance of artifacts, you will need to implement your own HTTP shovel. If you want an HTTP server to work with make-http-shovel, you can simply serve the files from a directory suitable for make-filesystem-shovel. This will add an invariant where the shovel and the server must agree in advance about what files are available, which you can represent as arguments to make-http-shovel.
Those implementing services will also need to consider the relationship of package queries to artifacts representing package definitions, and implement a package query canon to allow that transition. All of the related machinery for distributing artifacts over the network can be delivered to clients in a launcher, which warrants additional scrutiny since launchers have all privileges of the process’ user.