TiddlyWiki is quite a famous note-taking system. Describing itself as “a non-linear personal web notebook”, it’s actually quite a bit more than that. Thanks to a plugin system, it’s basically an app platform, and plenty of cool stuff has been made for it.
And it exists as a unique thing: a self-contained web application—a single HTML file with both the app and the content—that you can take around on a USB flash drive. Oh wait… who even uses those anymore for anything other than OS installers and bringing files to a print shop?
So of course, a bunch of storage solutions have sprung up.
There’s an official node.js based server mode where it becomes more like a traditional web app,
there are browser extensions and even modules inside of TW itself for saving the updated single file to a server,
there are mobile and desktop apps that try to combine that saving with some kind of offline support,
there are people using a file sync solution like Syncthing – I’ve even heard of some people using Syncthing with the server version,
.tid files between devices and running the server in Termux on their phones to access them on mobile.
Oof. While I’m nerdy enough to be able to do that, I’m also enough of a spoiled basic consumer to know that that’s not the user experience I want.
What I want is something that works like an app. Fully working offline, syncing efficiently (not by POSTing a multi-megabyte HTML file), quickly and reliably when online. And with client-side encryption, because there’s just no reason to let the server side read the contents here.
There has actually been one good attempt at bringing this kind of sync to TiddlyWiki: NoteSelf, which integrated PouchDB for storage. I liked the sound of it, but in practice it wasn’t up to my standards. It’s a heavyweight modification of TiddlyWiki that doesn’t keep up with latest core updates, PouchDB/CouchDB feel a bit heavy themselves, there’s no encryption, and the offline support is just “run it from your hard drive”.
So… this is where I come in. Last year, looking to try TiddlyWiki once again, looking through the options and getting frustrated with the aforementioned everything, one thing inspired me. I stumbled upon a basic IndexedDB plugin which made me realize that there is an API for storage backends inside TiddlyWiki. I realized that there’s no need for core modifications, that I could just start with this and—knowing what I know about the web platform—take it to the next level. Make a plugin that combines IndexedDB, encryption, a sync engine (with a custom super-lightweight server counterpart for it), and adds a Service Worker and a Web Manifest to turn TW into a Progressive Web App that works offline and installs on mobile with “add to home screen”. That was a whole Vision. And I knew it had to be done. It just had to exist. So I got to work, putting aside my typical FreeBSD system-level work to get back to good old web dev.
Now, a whole year after that inspiring moment, having gone through huge personal life changes and a reverse putting-aside in favor of my other work again… it’s finally ready for the public to see.
It’s still rough around the edges, but I’ve put a lot of effort into the setup experience, hiding all the complexity of the flexibility available with how the system is structured and presenting a very simple “happy path” where if you open the app from a sync server it’s already preconfigured to sync with that server and so on.
I’ve also tried to document it, though I have to say that detailing the security model was a lot easier than trying to explain the whole synchronization/server/content-versus-app thing. Hopefully at least for those familiar with TiddlyWiki it’s not too hard to understand that there’s the “app wiki” with the core/plugins/themes and the separate encrypted content, and that the sync server can work with both (differently).
Now, to justify the whole existence of this blog post, let me also list some random fun facts and whatnot – everything that didn’t fit in the documentation itself :)
Because of how browsers work, I’ve had to take extra care to make the storage work across multiple running tabs without corrupting anything. Moreover, I made it all explicitly work together well and it’s kind of a hidden feature now. Thanks to a combination of BroadcastChannels and Web Locks, you can enjoy a multi-tab experience. Browse the same wiki simultaneously in two tabs, the navigation will be independent but any changes will be visible everywhere.
This whole argon2ian thing you might’ve seen was indeed created for TiddlyPWA! I’ve started out using PBKDF2 because it’s available in the Web Crypto API, but eventually decided to upgrade to a modern password hash rather than cranking out iteration counts. I wasn’t satisfied with the state of existing Argon2 WASM+JS wrappers, so I set out to make my own, code-golfed to the tiniest size possible. The yak shaving stack got even deeper during that subproject. Also, I have used the very new Compression Streams API to be able to bundle the Wasm binary as compressed while not having to bunlde a decompressor. And this has led to the creation of a very funny bug…
…so since that API was already used there I started using it to compress long-enough tiddler contents as well. Stream APIs are kind of annoying when you don’t want to stream, so I went with the whole “a stream is an async iterator” thing. When I first gave TiddlyPWA to an external tester I got a report of not being able to save a particular tiddler. Turns out, that iterator thing is only available in Firefox as of right now. I accidentally made it so that tiddlers longer than 240 bytes would only work in Firefox. :D
Another really funny bug is something I bugzilled here for Firefox Android.
When using FlorisBoard, pressing “enter” would result in the password mask (bullet ••• characters) becoming the actual value of the input.
This is something that I have discovered while typing passwords into my TiddlyPWA password prompt!
I also ran into an already reported
prompt() bug in Firefox on iOS.
The server was initially written with SQLite as it is now, but when Deno KV was announced I got excited and rewrote it to use KV, hoping to leverage Deno Deploy for an easy server hosting experience. I quickly ran into a 64KiB limit per entry when trying to do the whole “save the HTML page” thing, but I found kv-toolbox, a module that would do chunking for me. Then, after everything was ready, already with the aforementioned other person testing it, I discovered (and ranted about) the undocumented 10 mutations in a transaction limitation. I wasn’t about to give up on atomicity, so I just rewrote everything back to SQLite in a burst of anger and now I hold a bit of a grudge against Deno KV >_< Even though I recognize that, obviously, it’s just that my use case is absolutely not the intended one.
There was a very last minute backwards-incompatible change I suddenly managed to do right before release.
I probably could’ve tried to make it as a commercial product with a centrally hosted server, but that’s not what I made.
TiddlyPWA is fully free (both “as in beer” and “as in freedom”) because I believe in software that empowers everyone to own their data and the software itself, that
is never gonna give you up doesn’t die for business reasons.
Right now TiddlyPWA is still early in its development, still missing a lot of things, but hopefully is solid enough to make you check it out. If you like what you see, please consider helping me continue working on it:
- go ahead and try TiddlyPWA first of course!
- report issues, read the code and suggest code changes on Codeberg;
- support me on Patreon :3 I’m trying to do this open source thing full time, which is only possible with your support!
- follow me on the fediverse;
- share TiddlyPWA with anyone who might be interested.