Jacky with gold-brimmed glasses looking to the left. Jacky Alciné
Status
Available · contract
Email
yo@jacky.wtf
Calendar
cal.com/jackyalcine
Résumé
/files/resume.pdf
Feeds
essays, links, notes

Data unions are the way forward for the Atmosphere

A scene captured by Compagons from Unsplash.
A scene captured by Compagons from Unsplash.

Tynan wrote a post on Bluesky that quoted their blog post about the advent of a new lexicon held up by https://lexicon.community, one for books. As someone who's helped drive this forward and is actively working on a client for books, I can't say how excited (positively!) for this step to be taken. There was an initial call some time ago to talk about some of the things we wanted to see in the lexicon and going forward and a point that came up that repeated Tynan's point in their Bluesky post: the need for a data union.

A Data Union?

A data union is a organization that collects information on behalf of its members. These members then get to dictate how the information is used — a bit like a cooperative. In the case of metadata around books, this information gets tricky. Things like copyright and general licensing can make for confusion on how and where information can be shared. Hardcover mentions this challenge themselves around redistribution.

A bit similar to Hardcover, for Livtet, I use vendors like Google's book API and ISBNDB to help source information like descriptions, identifiers and covers that you end up seeing on a page for books. Google's API is free though (generously!) rate-limited. ISBNDb's limits on their basic plan is good enough (for me) and easy enough to plan around such that the casual user of Livtet can build up their library. If you connect your account with Hardcover on Livtet, you also get the added benefit of being able to source information from their high-quality library — down to the covers used.

What a data union for the Atmosphere would provide would be a reduction in duplicate efforts of compiling information about books, their identifiers and covers and instead allow folks to focus on providing a pristine interface around this information. This is a bit similar to how Bluesky PBLLC provides https://public.api.bsky.app/ as a way to do Bluesky things (microblogging) across the entire space (that's accessible to https://web.plc.directory, that is). It's how things like https://mu.social, https://blacksky.community, https://eurosky.social and so many others can end up making their own interface over the data that the protocol makes public.

Such a shift would mean, with more common lexicons for behaviors around books, that when you leave a review for a book using https://popfeed.social/, it could show up in https://bookhive.buzz/ or https://livtet.olamaelcu.net/ by merit of the applications speaking the same underlying language not just for the books being interacted with but the interactions themselves.

Enter OpenLibrary

In the case of a data union, and to avoid reinventing the wheel, the easiest route of action would be if OpenLibrary produced a PDS of all of the books in its library and allow for more librarians to contribute updates to those records. This would provide immediate access as a source of data for those working in the Atmosphere and could incentivize other companies and organizations to contribute data to increase the quality of the information stored there.

OpenLibrary provides data dumps and an API that we can use to seed this data union and we can work on a limited replication of the data to make this work. It'd also be helpful if we contributed information back to the repository to help the general community as well[1].

The most important thing is that we try to move to have atmospheric applications see it as a net negative to attempt to collect information about books in a way that requires them to rebuild the entire world of books.

The lower effort solution would be OpenLibrary providing an AppView that responded to the two procedures defined by the book lexicon that munged their information in such a way that it was reproducible for other ATProto clients. I'm more in favor of this because it's one less service for OpenLibrary to serve and in my opinion, would move along their support of OAuth 2.1 for user profiles (by allowing folks to use access tokens for higher rate limits).

The 'We can just do things' Energy

We don't necessarily have to wait for OpenLibrary. The work of Nick behind Bookhive has already accrued information about 150,000 books and my own work with Bibliograph, merging in Nick's freely available data brings that number up to 315,000 (as of last writing with backfills from historically popular books and crawling other book lists). I'd love to work on some sort of sister project (or retrofit my AppView's existing name) such that it:

  • operates as a PDS to hold all of the book information
  • issues labels to accounts that are to be given librarian access, recognized as an author, etc
  • an AppView that aggregates things like reviews, reactions (relying on https://lexicon.community extensions) and other backlinks to provide network-wide rich information on books
  • a means of applying for librarian status managed by the members of the data union (which can be granted to anyone with a OpenLibrary account bounded to their ATProto identity, perhaps)
  • a Web client that provides librarians with tooling to manage the repository of book metadata

These features would allow for clients[2] like Resenha — a one optimized solely for Bookhive — to move from only working with Bookhive data to being its own client that works on open standards on the open protocol. This would encourage (if not enable) a proliferation of more clients since the data union's representation would be open to the public; even for local testing.

Disclaimer: some of this work is stuff I'm already working on with Bibliograph — the administrative view permits some of these operations. What's missing is the labeler that'd mint folks librarian access to records to update them - for starters.

I'm also working on a change to OpenLibrary to have them provide XRPC endpoints that can be used to resolve from a PDS they can run to serve information about books.

Things that can be enabled through this work by way of unification:

  • fast edition, contributor and publisher typeahead
  • uniform cover information
  • format-specific unit detailing (seconds for audiobooks, pages for lexical reads)
  • chapter-level delineation for material

The possibilities grow when we work together.

Your Thoughts

I wrote this to open up some discussion on what this opportunity could provide the Atmosphere, especially as we begin to look at opportunities for more non-Bluesky-centric services and applications. Those who are inclined to track what they read are very open to trying applications: it's how Goodreads is able to remain in the center of the space that it's in and other contenders like Fable, Margins and Readwise can come into the game as well and be successful. With this, we can finally create an experience where folks aren't exporting/importing and rebuilding community on each application: they can walk through the door with it.

The data's there; it's just up to us to democratize it.


  1. We could automate the writing of these contributions back to OpenLibrary once they land OAuth2 support to allow for folks to sign in and associate their accounts with OpenLibrary. ↩︎

  2. I put client in quotes here because it's truly a client towards one's repository from my understanding — not directly to Bookhive. Bookhive doesn't provide an AppView but a repository of information that works on lexicons for one's repo. ↩︎

Thanks for reading!

I'm open to feedback on the things I write. You can e-mail your thoughts to yo@jacky.wtf at anytime.

I have an infrequent newsletter that you can subscribe to (check the archives). I send between one to three e-mails a month, usually expanding on posts like this one and giving more updates on life and things I've engaged with. There's 117 people subscribed — come join us!