Incorporating Lua with Livtet

One of the goals I have with Livtet is to incorporate a means of extensibility. I didn't want to deeply write support for things like Calibre, Goodreads or Bookwyrm because these services can change their outward facing implementations on a whim, in response to a security change or the like. Having an extension system felt like the best way to work around this. Also I try to imagine this tool replacing my use and need for Calibre, that's still some time off. But I do look to how it allows plugins to manipulate and extend its behavior^[1]. In the case of Livtet, at the time of writing, plugins aren't declarable by the user in the interface, you'd have to modify the manifest file^[2] with the following:

search_method = "sqlite"

[libraries.01J5F20ZD9822Z0V64B1EGB01S]
signature = "file"
payload = "/home/ovid/Documents/My Stuff"

[libraries.01J5FBE22M4NRWM4YTSZSSX9Q8]
signature = "lua"
payload = "bookwyrm"

[libraries.01J5F21QB5JZ75BSCJ6GHEA1BQ]
signature = "lua"
payload = "open-library"

This is a list of three library lookup definitions and in the case of loading Lua extensions, Livtet knows to do that by looking at the payload value and wrapping that in a contextual extension for that plugin. I chose to share the Lua context across plug-ins to allow for plugins to share logic as opposed to spinning up a new one for each plugin^[3]. The way a Lua extensions talks to Livtet is straight-forward: it declares an (currently undocumented) interface to the host and Livtet invokes some code to extract that value from it to complete an operation. For example, if I wanted to get the list of books across my shelves from my Bookwyrm profile, some of the Lua code looks like this:

local function items()
	local user_profile = livtet.options("user_profile_url")
	local book_items = {}
	for shelf, url in pairs(USER_URLS) do
		local full_request_url = user_profile .. url
		local feed_index = livtet.fetch_json(full_request_url)
		local first_url = feed_index["first"]
		if first_url ~= nil then
			book_items[shelf] = fetch_all_from_feed(first_url)
		end
	end

	local books = {}

	for shelf, book_list in pairs(book_items) do
		for _, book_info in pairs(book_list) do
			local book = expand_book(book_info)
			table.insert(books, book)
		end
	end

	return books
end

M.library = {
	items = items,
	-- other API methods would go here.
}

Livtet provides a per-invocation global method, options, that allows some information that the plugin requires for operation^[4]; in this case, the Bookwyrm extension requires the profile URL that it'll work with. The act of calling this from Livtet looks like the following:

async fn items(
    &self,
    paging: livtet::library::ItemPaging,
) -> Result<Vec<livtet::Book>, livtet::Error> {
    let shared_lua = self.ext.lua().map_err(crate::Error::from)?;
    let lua = shared_lua.lua().map_err(crate::Error::from)?;
    let string = format!(
        "return require('{module_name}.init').library",
        module_name = self.ext.module_name
    );
    let metadata_value: mlua::Value = lua.load(string).eval().map_err(crate::Error::from)?;
    let items_function: Function = match metadata_value {
        mlua::Value::Table(library_table) => {
            library_table.get("items").map_err(crate::Error::from)?
        }
        mlua::Value::Function(library_func) => {
            let library_table: mlua::Table = library_func.call(()).map_err(crate::Error::from)?;
            library_table.get("items").map_err(crate::Error::from)?
        }
        _ => return Err(crate::Error::Lua(Error::InvalidValue).into()),
    };
    let page = lua.to_value(&paging).map_err(crate::Error::from)?;
    let books: Vec<Book> = items_function
        .call(page)
        .and_then(|v| lua.from_value(v))
        .map_err(crate::Error::from)?;

    Ok(books
        .into_iter()
        .map(|b| b.0.into())
        .collect::<Vec<livtet::Book>>())
}

Quite a bit is going on here. So a run-down:

• We grab a safe reference to the Lua context,
• Compose the Lua script we'll evaluate for our needs,
• We determine what kind of value M.library returns and pluck items from it; this mirrors the trait definition.
• We call it and convert each of the value returned into ones Livtet can work with.

This approach is something that's been simple to replicate. Writing out tests for this currently require hitting actual remote services^[5] which slow down the tests just a tad and run the risk, if I ran this on every commit — for example, of being blocked. I don't think it'll happen but I'm known for thinking of "what if". That said, these changes allow me to present these collections locally with little to no effort!

This is shifting my thinking about how I want libraries to show in Livtet. But that's for another post!