4 Search (SXPath)🔗ℹ

The W3C "XPath" standard describes a standardized way to perform searches in XML documents. For instance, the XPath string "/A/B/C" (thank you, Wikipedia) describes a search for a C element whose parent is a B element whose parent is an A element that is the root of the document.

The sxpath function performs a similar search over SXML data, using either the standard XPath strings or a list of Racket values.

> (require sxml/sxpath)

> ((sxpath "/A/B/C")    '(*TOP* (A (B (C)))))

'((C))

> ((sxpath '(A B C))    '(*TOP* (A (B (C)))))

'((C))

> ((sxpath "//p[contains(@class, 'blue')]/text()")    '(*TOP*      (body       (p (@ (class "blue")) "P1")       (p (@ (class "blue green")) "P2")       (p (@ (class "red")) "P3"))))

'("P1" "P2")

(This documentation desperately needs more examples.)

If we use Neil Van Dyke’s html parser, we might parse this into the following sxml document:

> (define example-doc     '(*TOP*       (aaa "\n" "  "         (bbb "\n" "     "           (ccc) "\n" "     "           (www " www content " (xxx) (www "\n" "     " (zzz) "\n" "  ")))         "\n" "  "         (xxx "\n" "     " (ddd " content in ccc \n" "     ") "\n" "  ")         "\n")))

To extract the xxx’s inside the aaa from the example document:

> ((sxpath '(aaa xxx)) example-doc)

'((xxx "\n" "     " (ddd " content in ccc \n" "     ") "\n" "  "))

To extract all cells from an HTML table:

> (define table     `(*TOP*       (table        (tr (td "a") (td "b"))        (tr (td "c") (td "d")))))

> ((sxpath '(table tr td)) table)

'((td "a") (td "b") (td "c") (td "d"))

To extract all cells anywhere in a document:

> (define table     `(*TOP*       (div        (p (table            (tr (td "a") (td "b"))            (tr (td "c") (td "d"))))        (table         (tr (td "e"))))))

> ((sxpath '(// td)) table)

'((td "a") (td "b") (td "c") (td "d") (td "e"))

One result may be nested in another one:

> (define doc     `(*TOP*       (div        (p (div "3")           (div (div "4"))))))

> ((sxpath '(// div)) doc)

'((div (p (div "3") (div (div "4")))) (div "3") (div (div "4")) (div "4"))

There’s also a string-based syntax, txpath. As shown in the grammar above, sxpath assumes that any strings in the path are expressed using the txpath syntax.

So, for instance, the prior example could be rewritten using a string:

> (define doc     `(*TOP*       (div        (p (div "3")           (div (div "4"))))))

> ((sxpath "//div") doc)

'((div (p (div "3") (div (div "4")))) (div "3") (div (div "4")) (div "4"))

More generally, lists in the s-expression syntax correspond to string concatenation in the txpath syntax.

So, to find all italics that appear at top level within a paragraph:

> (define doc     `(*TOP*       (div        (p (i "3")           (froogy (i "4"))))))

> ((sxpath "//p/i") doc)

'((i "3"))

Handling of namespaces in sxpath is a bit surprising. In particular, it appears to me that sxpath’s model is that namespaces must appear fully expanded in the matched source. For instance:

> ((sxpath "//ns:p" `((ns . "http://example.com")))    '(*TOP* (html (http://example.com:body                   (http://example.com:p "first para")                   (http://example.com:p                    "second para containing"                    (http://example.com:p "third para") "inside it")))))

'((http://example.com:p "first para")   (http://example.com:p    "second para containing"    (http://example.com:p "third para")    "inside it")   (http://example.com:p "third para"))

But the corresponding example where the source document contains a namespace shortcut does not match in the same way. That is:

> ((sxpath "//ns:p" `((ns . "http://example.com")))    '(*TOP* (@ (*NAMESPACES* (ns "http://example.com")))            (html (ns:body (ns:p "first para")                           (ns:p "second para containing"                                 (ns:p "third para") "inside it")))))

'()

It produces the empty list. Instead, you must pretend that the shortcut is actually the namespace. Thus:

> ((sxpath "//ns:p" `((ns . "ns")))    '(*TOP* (@ (*NAMESPACES* (ns "http://example.com")))            (html (ns:body (ns:p "first para")                           (ns:p "second para containing"                                 (ns:p "third para") "inside it")))))

'((ns:p "first para")   (ns:p "second para containing" (ns:p "third para") "inside it")   (ns:p "third para"))

Ah well.

4.1 Filtering🔗ℹ

It may sometimes be the case that you’re looking for abc nodes that contain def nodes. You could try

(sxpath '(// abc def))

... but that would give you the inner def nodes, not the abc parent nodes that contain the def nodes. If you pore over the expansion above, you will discover that you can do this, using ... well, essentially, using a pair of nested lists:

((sxpath '(// (abc (def))))

'(x (x (abc (x)

(def "1")))

(abc (x) "2")

(abc (def) "3")))

Note that this is also the right way to go if for instance you’re looking for a div with a particular id:

((sxpath '(// (div (@ id (equal? "wanted")))))

'(body (foo (div (@ (id "a"))

(div (@ (id "b")) "abc")

"def")

(div (@ (id "wanted"))

(div (@ (id "c")) "qq")

"ghi"))))

But what if you want to check not that the string is equal to a fixed value, but rather that it contains a given value? This is common in the case of the "class" attribute, which often has a space-separated list of tokens. It turns out that SXML’s combinator library can handle this just fine, but in order to use it, you’ll need to unzip the sxpath to allow you to use the combinators. To see this, let’s amend the earlier example to find a class containing the token "wanted".

Our first step is to "unzip" the sxpath syntax. The following query produces exactly the same result as the previous one:

((node-join (sxpath '(//))

(node-reduce

(sxpath '(div))

(sxml:filter (node-join (sxpath '(@ id))

(select-kids

(node-equal? "wanted"))))))

'(body (foo (div (@ (id "a"))

(div (@ (id "b")) "abc")

"def")

(div (@ (id "wanted"))

(div (@ (id "c")) "qq")

"ghi"))))

At this point, we can replace node-equal? with any predicate on strings. In the following example we generalize it to look for occurrences of the token, and then we add a bunch of other junk to the specified id, to show that it still works:

;; does the given string occur as a "word" in the text?

(define (str-includes? str)

(λ (text) (member str (string-split text))))

((node-join (sxpath '(//))

(node-reduce

(sxpath '(div))

(sxml:filter (node-join (sxpath '(@ id))

(select-kids

(str-includes? "wanted"))))))

'(body (foo (div (@ (id "a"))

(div (@ (id "b")) "abc")

"def")

(div (@ (id "wanted bar baz"))

(div (@ (id "c")) "qq")

"ghi"))))

Deprecated; use sxpath instead.

procedure (sxml:filter pred) → sxml-converter

pred : sxml-converter-as-predicate

procedure (sxml:complement pred) → sxml-converter-as-predicate

pred : sxml-converter-as-predicate

procedure (node-reduce converter ...) → sxml-converter

converter : sxml-converter

procedure (node-or converter ...) → sxml-converter

converter : sxml-converter

procedure (node-closure converter) → sxml-converter

converter : sxml-converter

The following procedures depend explicitly on the root node.

Here’s an example:

> (((sxml:preceding (ntype?? 'www)) example-doc) ((sxpath `(aaa xxx)) example-doc))

'((www "\n" "     " (zzz) "\n" "  ")   (www " www content " (xxx) (www "\n" "     " (zzz) "\n" "  ")))

> (define doc '(*TOP* (div (p "foo") (p "bar")                            (img "baz") (p "quux"))))

> (((sxml:preceding-sibling (ntype?? 'p)) doc) ((sxpath '(// img)) doc))

'((p "bar") (p "foo"))