http-easy: a high-level HTTP client
(require net/http-easy) | package: http-easy-lib |
This library wraps net/http-client to provide a simple interface for day-to-day use. It automatically handles:
connection pooling
connection timeouts
SSL verification
automatic compression and decompression
streaming responses
authentication
redirect following
cookie storage
multipart file uploads
1 Guide
1.1 Making Requests
Getting started is as easy as requiring the net/http-easy module:
> (require net/http-easy)
And using one of the built-in requesters to perform a request:
> (define res (get "https://example.com"))
The result is a response? value that you can inspect:
> (response-status-code res) 200
> (response-status-message res) #"OK"
> (response-headers-ref res 'date) #"Sat, 24 Apr 2021 06:58:51 GMT"
> (subbytes (response-body res) 0 30) #"<!doctype html>\n<html>\n<head>\n"
Connections to remote servers are automatically pooled so closing the response returns its underlying connection to the pool:
> (response-close! res)
If you forget to manually close a response, its underlying connection will get returned to the pool when the response gets garbage-collected. Unless you explicitly use Streaming Responses, you don’t have to worry about this much.
1.2 Streaming Responses
Response bodies can be streamed by passing #t as the #:stream? argument to any of the requesters:
> (define res (get "https://example.com" #:stream? #t))
The input port containing the response body can be accessed using response-output:
> (input-port? (response-output res)) #t
> (read-string 5 (response-output res)) "<!doc"
> (read-string 5 (response-output res)) "type "
Using response-body immediately drains the remaining data and closes the input port:
> (subbytes (response-body res) 0 10) #"html>\n<htm"
> (subbytes (response-body res) 0 20) #"html>\n<html>\n<head>\n"
> (port-closed? (response-output res)) #t
1.3 Authenticating Requests
The library provides an auth procedure for HTTP basic auth:
> (response-status-line (get "https://httpbin.org/basic-auth/Aladdin/OpenSesame")) #"HTTP/1.1 401 UNAUTHORIZED"
> (response-json (get "https://httpbin.org/basic-auth/Aladdin/OpenSesame" #:auth (basic-auth "Aladdin" "OpenSesame"))) '#hasheq((authenticated . #t) (user . "Aladdin"))
And for bearer auth:
> (response-json (get "https://httpbin.org/bearer" #:auth (bearer-auth "secret-api-key"))) '#hasheq((authenticated . #t) (token . "secret-api-key"))
The above example is equivalent to:
> (response-json (get "https://httpbin.org/bearer" #:auth (lambda (uri headers params) (values (hash-set headers 'authorization "Bearer secret-api-key") params)))) '#hasheq((authenticated . #t) (token . "secret-api-key"))
1.4 Sending Data
You can supply a list of pairs to be sent as an application/x-www-form-urlencoded payload:
> (define res (response-json (post "https://httpbin.org/post" #:form '((a . "hello") (b . "there"))))) > (hash-ref res 'form) '#hasheq((a . "hello") (b . "there"))
Alternatively, you can supply the #:json keyword argument to send an application/json payload:
> (define res (response-json (post "https://httpbin.org/anything" #:json (hasheq 'a "hello" 'b "there")))) > (hash-ref res 'json) '#hasheq((a . "hello") (b . "there"))
To send data using arbitrary formats, you can use the #:data keyword argument:
> (define res (response-json (post "https://httpbin.org/anything" #:data #"hello"))) > (hash-ref res 'data) "hello"
To gzip the payload, use the gzip-payload combinator:
> (define res (response-json (post "https://httpbin.org/anything" #:data (gzip-payload (pure-payload #"hello"))))) > (hash-ref res 'data) "data:application/octet-stream;base64,H4sIALDBg2AAA8tIzcnJBwCGphA2BQAAAA=="
> (define res (response-json (post "https://httpbin.org/anything" #:data (gzip-payload (json-payload (hasheq 'hello "world")))))) > (hash-ref res 'data) "data:application/octet-stream;base64,H4sIALHBg2AAA6tWykjNyclXslIqzy/KSVGqBQDRQQnYEQAAAA=="
1.5 Cookie Storage
To store cookies between requests pass a cookie-jar<%> into your session?:
> (require net/cookies net/url racket/class) > > (define jar (new list-cookie-jar%))
> (define session-with-cookies (make-session #:cookie-jar jar)) >
> (parameterize ([current-session session-with-cookies]) (get "https://httpbin.org/cookies/set/hello/world") (response-json (get "https://httpbin.org/cookies"))) '#hasheq((cookies . #hasheq((hello . "world"))))
>
> (for ([c (in-list (send jar cookies-matching (string->url "https://httpbin.org")))]) (printf "~a: ~a" (ua-cookie-name c) (ua-cookie-value c))) hello: world
1.6 UNIX Sockets
To make a request to a UNIX domain socket, pass http+unix as the scheme and url-encode the path to the socket as the host.
> (response-status-code (get "http+unix://%2Fvar%2Frun%2Fdocker.sock/info")) 200
2 Reference
procedure
(get uri [ #:close? close? #:stream? stream? #:headers headers #:params params #:auth auth #:data data #:form form #:json json #:timeouts timeouts #:max-attempts max-attempts #:max-redirects max-redirects #:user-agent user-agent]) → response? uri : (or/c bytes? string? url?) close? : boolean? = #f stream? : boolean? = #f headers : headers/c = (hasheq) params : query-params/c = null auth : (or/c false/c auth-procedure/c) = #f
data : (or/c false/c bytes? string? input-port? payload-procedure/c) = #f form : query-params/c = unsupplied json : jsexpr? = unsupplied timeouts : timeout-config? = (make-timeout-config) max-attempts : exact-positive-integer? = 3 max-redirects : exact-nonnegative-integer? = 16 user-agent : (or/c bytes? string?) = (current-user-agent)
procedure
(post uri [ #:close? close? #:stream? stream? #:headers headers #:params params #:auth auth #:data data #:form form #:json json #:timeouts timeouts #:max-attempts max-attempts #:max-redirects max-redirects #:user-agent user-agent]) → response? uri : (or/c bytes? string? url?) close? : boolean? = #f stream? : boolean? = #f headers : headers/c = (hasheq) params : query-params/c = null auth : (or/c false/c auth-procedure/c) = #f
data : (or/c false/c bytes? string? input-port? payload-procedure/c) = #f form : query-params/c = unsupplied json : jsexpr? = unsupplied timeouts : timeout-config? = (make-timeout-config) max-attempts : exact-positive-integer? = 3 max-redirects : exact-nonnegative-integer? = 16 user-agent : (or/c bytes? string?) = (current-user-agent)
procedure
(delete uri [ #:close? close? #:stream? stream? #:headers headers #:params params #:auth auth #:data data #:form form #:json json #:timeouts timeouts #:max-attempts max-attempts #:max-redirects max-redirects #:user-agent user-agent]) → response? uri : (or/c bytes? string? url?) close? : boolean? = #f stream? : boolean? = #f headers : headers/c = (hasheq) params : query-params/c = null auth : (or/c false/c auth-procedure/c) = #f
data : (or/c false/c bytes? string? input-port? payload-procedure/c) = #f form : query-params/c = unsupplied json : jsexpr? = unsupplied timeouts : timeout-config? = (make-timeout-config) max-attempts : exact-positive-integer? = 3 max-redirects : exact-nonnegative-integer? = 16 user-agent : (or/c bytes? string?) = (current-user-agent)
procedure
(head uri [ #:close? close? #:stream? stream? #:headers headers #:params params #:auth auth #:data data #:form form #:json json #:timeouts timeouts #:max-attempts max-attempts #:max-redirects max-redirects #:user-agent user-agent]) → response? uri : (or/c bytes? string? url?) close? : boolean? = #f stream? : boolean? = #f headers : headers/c = (hasheq) params : query-params/c = null auth : (or/c false/c auth-procedure/c) = #f
data : (or/c false/c bytes? string? input-port? payload-procedure/c) = #f form : query-params/c = unsupplied json : jsexpr? = unsupplied timeouts : timeout-config? = (make-timeout-config) max-attempts : exact-positive-integer? = 3 max-redirects : exact-nonnegative-integer? = 16 user-agent : (or/c bytes? string?) = (current-user-agent)
procedure
(options uri [ #:close? close? #:stream? stream? #:headers headers #:params params #:auth auth #:data data #:form form #:json json #:timeouts timeouts #:max-attempts max-attempts #:max-redirects max-redirects #:user-agent user-agent]) → response? uri : (or/c bytes? string? url?) close? : boolean? = #f stream? : boolean? = #f headers : headers/c = (hasheq) params : query-params/c = null auth : (or/c false/c auth-procedure/c) = #f
data : (or/c false/c bytes? string? input-port? payload-procedure/c) = #f form : query-params/c = unsupplied json : jsexpr? = unsupplied timeouts : timeout-config? = (make-timeout-config) max-attempts : exact-positive-integer? = 3 max-redirects : exact-nonnegative-integer? = 16 user-agent : (or/c bytes? string?) = (current-user-agent)
procedure
(patch uri [ #:close? close? #:stream? stream? #:headers headers #:params params #:auth auth #:data data #:form form #:json json #:timeouts timeouts #:max-attempts max-attempts #:max-redirects max-redirects #:user-agent user-agent]) → response? uri : (or/c bytes? string? url?) close? : boolean? = #f stream? : boolean? = #f headers : headers/c = (hasheq) params : query-params/c = null auth : (or/c false/c auth-procedure/c) = #f
data : (or/c false/c bytes? string? input-port? payload-procedure/c) = #f form : query-params/c = unsupplied json : jsexpr? = unsupplied timeouts : timeout-config? = (make-timeout-config) max-attempts : exact-positive-integer? = 3 max-redirects : exact-nonnegative-integer? = 16 user-agent : (or/c bytes? string?) = (current-user-agent)
procedure
(put uri [ #:close? close? #:stream? stream? #:headers headers #:params params #:auth auth #:data data #:form form #:json json #:timeouts timeouts #:max-attempts max-attempts #:max-redirects max-redirects #:user-agent user-agent]) → response? uri : (or/c bytes? string? url?) close? : boolean? = #f stream? : boolean? = #f headers : headers/c = (hasheq) params : query-params/c = null auth : (or/c false/c auth-procedure/c) = #f
data : (or/c false/c bytes? string? input-port? payload-procedure/c) = #f form : query-params/c = unsupplied json : jsexpr? = unsupplied timeouts : timeout-config? = (make-timeout-config) max-attempts : exact-positive-integer? = 3 max-redirects : exact-nonnegative-integer? = 16 user-agent : (or/c bytes? string?) = (current-user-agent)
2.1 Sessions
value
: (or/c 'delete 'head 'get 'options 'patch 'post 'put symbol?)
value
value
form-data/c : (listof (cons/c symbol? (or/c false/c string?)))
value
: (listof (cons/c symbol? (or/c false/c string?)))
parameter
(current-session session) → void? session : session?
= (make-session)
procedure
(make-session [ #:pool-config pool-config #:ssl-context ssl-context #:cookie-jar cookie-jar #:proxies proxies]) → session? pool-config : pool-config? = (make-pool-config)
ssl-context : (or/c #f ssl-client-context? (promise/c ssl-client-context?)) = (delay (ssl-secure-client-context)) cookie-jar : (or/c false/c (is-a?/c cookie-jar<%>)) = #f proxies : (listof proxy?) = null
The #:ssl-context argument controls how HTTPS connections are handled. The default implementation verifies TLS certificates, verifies hostnames and avoids using weak ciphers. To use a custom certificate chain or private key, you can use ssl-make-client-context.
The #:cookie-jar argument specifies the cookie jar to use to store cookies between requests made against a session. The default is to discard all cookies. See list-cookie-jar%.
The #:proxies argument specifies an optional list of proxies to use when making requests.
Changed in version 0.3 of package http-easy-lib: Added the #:proxies argument.
Changed in version 0.6: The #:ssl-context argument accepts promises.
procedure
(session-close! s) → void?
s : session?
procedure
(session-request s uri [ #:close? close? #:stream? stream? #:method method #:headers headers #:params params #:auth auth #:data data #:form form #:json json #:timeouts timeouts #:max-attempts max-attempts #:max-redirects max-redirects #:user-agent user-agent]) → response? s : session? uri : (or/c bytes? string? url?) close? : boolean? = #f stream? : boolean? = #f method : method/c = 'get headers : headers/c = (hasheq) params : query-params/c = null auth : (or/c false/c auth-procedure/c) = #f
data : (or/c false/c bytes? string? input-port? payload-procedure/c) = #f form : form-data/c = unsupplied json : jsexpr? = unsupplied timeouts : timeout-config? = (make-timeout-config) max-attempts : exact-positive-integer? = 3 max-redirects : exact-nonnegative-integer? = 16 user-agent : (or/c bytes? string?) = (current-user-agent)
Response values returned by this function must be closed before their underlying connection is returned to the pool. If the close? argument is #t, this is done automatically. Ditto if the responses are garbage-collected.
If the close? argument is #t, then the response’s output port is drained and the connection is closed.
If the stream? argument is #f (the default), then the response’s output port is drained and the resulting byte string is stored on the response value. The drained data is accessible using the response-body function. If the argument is #t, then the response body is streamed and the data is accessible via the response-output function. This argument has no effect when close? is #t.
The method argument specifies the HTTP request method to use.
Query parameters may be specified directly on the uri argument or via the params argument. If query parameters are specified via both arguments, then the list of params is appended to those already in the uri.
The auth argument allows authentication headers and query params to be added to the request. When following redirects, the auth procedure is applied to subsequent requests only if the target URL has the same origin as the original request. Two URLs are considered to have the same origin if their scheme, hostname and port are the same.
The data argument can be used to send arbitrary request data to the remote end. A number of payload procedures are available for producing data in standard formats:
> (define res (post #:data (json-payload (hasheq 'hello "world")) "https://httpbin.org/post")) > (hash-ref (response-json res) 'data) "{\"hello\":\"world\"}"
The form argument is a shorthand for passing a form-payload as the data argument.
The json argument is a shorthand for passing a json-payload as the data argument.
The data, form and json arguments are mutually-exclusive. Supplying more than one at a time causes a contract error to be raised.
The timeouts argument controls how long various aspects of the request cycle will be waited on. When a timeout is exceeded, an exn:fail:http-easy:timeout? error is raised. When redirects are followed, the timeouts are per request.
The max-attempts argument controls how many times connection errors are retried. This meant to handle connection resets and the like and isn’t a general retry mechanism.
The max-redirects argument controls how many redirects are followed by the request. Redirect cycles are not detected. To disable redirect following, set this argument to 0. The Authorization header is stripped from redirect requests if the target URL does not have the same origin as the original request.
Changed in version 0.3 of package http-easy-lib: Added support for the http+unix scheme to allow requests to UNIX domain sockets.
procedure
(url/literal? v) → boolean?
v : any/c
procedure
s : string?
procedure
(url/literal->string u) → string?
u : url/literal?
Literal URLs are used automatically when handling redirects to avoid issues that may pop up when decoding an re-encoding URLs from standards-non-compliant servers.
Added in version 0.7 of package http-easy-lib.
2.2 Responses
value
status-code/c : (integer-in 100 999)
syntax
(response clause ...)
clause = #:status-line e | #:status-code e | #:status-message e | #:http-version e | #:history e | #:headers heads maybe-rest | #:body e | #:json e heads = ([header-id e] ...) maybe-rest =
| e
> (require racket/match) >
> (match (get "https://example.com") [(response #:status-code 200 #:headers ([content-type (and (regexp #"text/html") the-content-type)])) the-content-type]) #"text/html; charset=UTF-8"
procedure
(response-status-line r) → bytes?
r : response?
procedure
(response-http-version r) → bytes?
r : response?
procedure
r : response?
procedure
r : response?
procedure
(response-headers r) → (listof bytes?)
r : response?
procedure
(response-headers-ref* r h) → (listof bytes?)
r : response? h : symbol?
procedure
(response-history r) → (listof response?)
r : response?
procedure
(response-body r) → bytes?
r : response?
procedure
(response-output r) → input-port?
r : response?
procedure
(response-json r) → (or/c eof-object? jsexpr?)
r : response?
procedure
(response-xexpr r) → xexpr?
r : response?
procedure
(response-xml r) → document?
r : response?
procedure
(read-response r) → any/c
r : response?
procedure
(read-response-json r) → (or/c eof-object? jsexpr?)
r : response?
procedure
(read-response-xexpr r) → xexpr?
r : response?
procedure
(read-response-xml r) → document?
r : response?
procedure
(response-drain! r) → void?
r : response?
procedure
(response-close! r) → void?
r : response?
2.3 Connection Pooling
value
limit/c : (or/c +inf.0 exact-positive-integer?)
procedure
(pool-config? v) → boolean?
v : any/c
procedure
(make-pool-config [ #:max-size max-size #:idle-timeout idle-timeout]) → pool-config? max-size : limit/c = 128 idle-timeout : timeout/c = 600
The max-size argument controls the maximum number of connections in a pool. Once a pool reaches this size, leasing a connection blocks until one is available or until the lease timeout is reached.
The idle-timeout argument controls the amount of time idle connections are kept open for.
2.4 Proxies
Proxies tunnel requests to one host through another. See the socks5 documentation for a SOCKS5 proxy implementation that is compatible with this library.
procedure
(make-proxy matches? connect!) → proxy?
matches? : (-> url? boolean?) connect! : (-> http-conn? url? (or/c #f ssl-client-context?) void?)
Added in version 0.3 of package http-easy-lib.
procedure
(make-http-proxy proxy-url [matches?]) → proxy?
proxy-url : (or/c bytes? string? url?)
matches? : (-> url? boolean?) = (λ (u) (equal? (url-scheme u) "http"))
Added in version 0.3 of package http-easy-lib.
procedure
(make-https-proxy proxy-url [matches?]) → proxy?
proxy-url : (or/c bytes? string? url?)
matches? : (-> url? boolean?) = (λ (u) (equal? (url-scheme u) "https"))
Added in version 0.3 of package http-easy-lib.
2.5 Authentication
procedure
(basic-auth username password) → auth-procedure/c
username : (or/c bytes? string?) password : (or/c bytes? string?)
procedure
(bearer-auth token) → auth-procedure/c
token : (or/c bytes? string?)
2.6 Payload Procedures
Payload procedures produce data and associated headers to be sent to a remote server.
procedure
p : payload-procedure/c
Added in version 0.8 of package http-easy-lib.
procedure
v : form-data/c
procedure
v : jsexpr?
procedure
p : payload-procedure/c
procedure
v : (or/c bytes? string? input-port?)
procedure
(field-part name value [content-type]) → part?
name : (or/c bytes? string?) value : (or/c bytes? string?) content-type : (or/c bytes? string?) = #"text/plain"
procedure
name : (or/c bytes? string?) inp : input-port? filename : (or/c bytes? string?) = (~a (object-name inp))
content-type : (or/c bytes? string?) = #"application/octet-stream"
procedure
(multipart-payload f ... [ #:boundary boundary]) → payload-procedure/c f : part? boundary : (or/c bytes? string?) = unsupplied
> (define resp (post #:data (multipart-payload (field-part "a" "hello") (file-part "f" (open-input-string "hello world!"))) "https://httpbin.org/anything")) > (hash-ref (response-json resp) 'form) '#hasheq((a . "hello"))
> (hash-ref (response-json resp) 'files) '#hasheq((f . "hello world!"))
2.7 Timeouts
procedure
(timeout-config? v) → boolean?
v : any/c
procedure
(make-timeout-config [ #:lease lease #:connect connect #:request request]) → timeout-config? lease : timeout/c = 5 connect : timeout/c = 5 request : timeout/c = 30
The lease argument controls the maximum amount of time leasing a connection from the connection pool can take.
The connect argument controls how long each connection can take to connect to the remote end.
The request argument controls how long to wait on a request before its response headers are returned.
2.8 Errors
procedure
(exn:fail:http-easy? v) → boolean?
v : any/c
procedure
v : any/c
procedure
→ (or/c 'lease 'connect 'request) e : exn:fail:http-easy:timeout?
2.9 User Agents
parameter
(current-user-agent) → (or/c bytes? string?)
(current-user-agent user-agent) → void? user-agent : (or/c bytes? string?)