Grabing OpenGraph data with Common Lisp

This is going to be a short article, mainly to warm up myself for other posts that I have shelved too long. Lately I've trying to complete one project that I work on for quite a long time, although I haven't put more than 8 hours of work in it. This stale's project name is discordia-chan and it will be a service, that you can self-host and add an imageboard inspired discussion thread in your static blog, podcast page or other website.

The front-end of the service is developed with VueJS and already published, although admittedly at a very early stage, and the back-end is developed with Common Lisp and will be published soon as well.

One of the things that I had to decide, was how to create the initial post, ie how to fill the starting post of the thread. My idea is to use the ubiquitous Open Graph protocol¹, so that we just need to fetch the data from the URL and have titles, images, descriptions and any other data we need to initiate the conversation. This also allows a lot of customization for the user of the service, since she can choose the tags to use for the thread and with what data to provide it.

The starting point for this was the ability to fetch such data. This way when the server gets asked for a thread (identified by its URL), if such thread doesn't exist, it can grab the Open Graph data and create a new one.

To do this with Common Lisp turned out to be fairly easy. Using the drakma² http client, and lquery³ for parsing the html and selecting the appropriate data, it turned out to be just a few lines of code:

(defvar *opengraph-css-selector* "meta[property^=og:]")
(defvar *opengraph-char-removal* "og:")

(defun get-opengraph-data (url)
  "Grabs a url and returns an alist of its opengraph data."
  (let* ((html (http-request url))
         (parsed-html ($ (initialize html)))
         (properties (map 'list (lambda (x)
                               (string-trim *opengraph-char-removal* x))
                      ($ parsed-html *opengraph-css-selector* (:attr "property"))))
         (contents ($ parsed-html *opengraph-css-selector* (:attr "content"))))
    (map 'list #'cons properties contents)))

Now calling:

(get-opengraph-data "https://www.imdb.com/title/tt0092675/")

gets you:

(("url" . "https://www.imdb.com/title/tt0092675/") ("site_name" . "IMDb")
 ("title" . "Μέχρι τελικής πτώσεως (1988) - IMDb")
 ("description"
  . "Μέχρι τελικής πτώσεως: Directed by Newt Arnold. With Jean-Claude Van Damme, Donald Gibb, Leah Ayres, Norman Burton. \"Bloodsport\" follows Frank Dux, an American martial artist serving in the military, who decides to leave the army to compete in a martial arts tournament in Hong Kong where fights to the death can occur.")
 ("type" . "video.movie")
 ("image"
  . "https://m.media-amazon.com/images/M/MV5BNTJmMGEzMTQtMzQzMi00YjE1LWI4MTctNjY0NWZiYzE2MDVhXkEyXkFqcGdeQXVyMTQxNzMzNDI@._V1_FMjpg_UX1000_.jpg")
 ("image:height" . "1561.218836565097") ("image:width" . "1000"))

There is a misconception sometimes that Common Lisp doesn't have enough libraries and while this can be true sometimes, for most cases I find myself covered. Both of the used libraries are excellent and really well developed.

Anyways, that's all for now. –,–@