tl;dr: go play with the new thing there, report issues there and support me there :3 but please do read the full story below!
TiddlyWiki is quite a famous note-taking system. Describing itself as āa non-linear personal web notebookā, itās actually quite a bit more than that. Thanks to a plugin system, itās basically an app platform, and plenty of cool stuff has been made for it.
And it exists as a unique thing: a self-contained web applicationāa single HTML file with both the app and the contentāthat you can take around on a USB flash drive. Oh waitā¦ who even uses those anymore for anything other than OS installers and bringing files to a print shop?
So of course, a bunch of storage solutions have sprung up.
Thereās an official node.js based server mode where it becomes more like a traditional web app,
there are browser extensions and even modules inside of TW itself for saving the updated single file to a server,
there are mobile and desktop apps that try to combine that saving with some kind of offline support,
there are people using a file sync solution like Syncthing ā Iāve even heard of some people using Syncthing with the server version,
syncing the .tid files between devices and running the server in Termux on their phones to access them on mobile.
Oof. While Iām nerdy enough to be able to do that, Iām also enough of a spoiled basic consumer to know that thatās not the user experience I want.
What I want is something that works like an app. Fully working offline, syncing efficiently (not by POSTing a multi-megabyte HTML file), quickly and reliably when online. And with client-side encryption, because thereās just no reason to let the server side read the contents here.
There has actually been one good attempt at bringing this kind of sync to TiddlyWiki: NoteSelf, which integrated PouchDB for storage. I liked the sound of it, but in practice it wasnāt up to my standards. Itās a heavyweight modification of TiddlyWiki that doesnāt keep up with latest core updates, PouchDB/CouchDB feel a bit heavy themselves, thereās no encryption, and the offline support is just ārun it from your hard driveā.
Soā¦ this is where I come in. Last year, looking to try TiddlyWiki once again, looking through the options and getting frustrated with the aforementioned everything, one thing inspired me. I stumbled upon a basic IndexedDB plugin which made me realize that there is an API for storage backends inside TiddlyWiki. I realized that thereās no need for core modifications, that I could just start with this andāknowing what I know about the web platformātake it to the next level. Make a plugin that combines IndexedDB, encryption, a sync engine (with a custom super-lightweight server counterpart for it), and adds a Service Worker and a Web Manifest to turn TW into a Progressive Web App that works offline and installs on mobile with āadd to home screenā. That was a whole Vision. And I knew it had to be done. It just had to exist. So I got to work, putting aside my typical FreeBSD system-level work to get back to good old web dev.
Now, a whole year after that inspiring moment, having gone through huge personal life changes and a reverse putting-aside in favor of my other work againā¦ itās finally ready for the public to see.
So.
Itās still rough around the edges, but Iāve put a lot of effort into the setup experience, hiding all the complexity of the flexibility available with how the system is structured and presenting a very simple āhappy pathā where if you open the app from a sync server itās already preconfigured to sync with that server and so on.
Iāve also tried to document it, though I have to say that detailing the security model was a lot easier than trying to explain the whole synchronization/server/content-versus-app thing. Hopefully at least for those familiar with TiddlyWiki itās not too hard to understand that thereās the āapp wikiā with the core/plugins/themes and the separate encrypted content, and that the sync server can work with both (differently).
Now, to justify the whole existence of this blog post, let me also list some random fun facts and whatnot ā everything that didnāt fit in the documentation itself :)
Random Development Tidbits (ha)
Because of how browsers work, Iāve had to take extra care to make the storage work across multiple running tabs without corrupting anything. Moreover, I made it all explicitly work together well and itās kind of a hidden feature now. Thanks to a combination of BroadcastChannels and Web Locks, you can enjoy a multi-tab experience. Browse the same wiki simultaneously in two tabs, the navigation will be independent but any changes will be visible everywhere.
This whole argon2ian thing you mightāve seen was indeed created for TiddlyPWA! Iāve started out using PBKDF2 because itās available in the Web Crypto API, but eventually decided to upgrade to a modern password hash rather than cranking out iteration counts. I wasnāt satisfied with the state of existing Argon2 WASM+JS wrappers, so I set out to make my own, code-golfed to the tiniest size possible. The yak shaving stack got even deeper during that subproject. Also, I have used the very new Compression Streams API to be able to bundle the Wasm binary as compressed while not having to bunlde a decompressor. And this has led to the creation of a very funny bugā¦
ā¦so since that API was already used there I started using it to compress long-enough tiddler contents as well. Stream APIs are kind of annoying when you donāt want to stream, so I went with the whole āa stream is an async iteratorā thing. When I first gave TiddlyPWA to an external tester I got a report of not being able to save a particular tiddler. Turns out, that iterator thing is only available in Firefox as of right now. I accidentally made it so that tiddlers longer than 240 bytes would only work in Firefox. :D
Another really funny bug is something I bugzilled here for Firefox Android.
When using FlorisBoard, pressing āenterā would result in the password mask (bullet ā¢ā¢ā¢ characters) becoming the actual value of the input.
This is something that I have discovered while typing passwords into my TiddlyPWA password prompt!
I also ran into an already reported prompt() bug in Firefox on iOS.
The server was initially written with SQLite as it is now, but when Deno KV was announced I got excited and rewrote it to use KV, hoping to leverage Deno Deploy for an easy server hosting experience. I quickly ran into a 64KiB limit per entry when trying to do the whole āsave the HTML pageā thing, but I found kv-toolbox, a module that would do chunking for me. Then, after everything was ready, already with the aforementioned other person testing it, I discovered (and ranted about) the undocumented 10 mutations in a transaction limitation. I wasnāt about to give up on atomicity, so I just rewrote everything back to SQLite in a burst of anger and now I hold a bit of a grudge against Deno KV >_< Even though I recognize that, obviously, itās just that my use case is absolutely not the intended one.
This TypeScript thing that I referenced on fedi was actually for the SQLite part of the TiddlyPWA server.
There was a very last minute backwards-incompatible change I suddenly managed to do right before release.
So, TiddlyPWA Needs YOU!
I probably couldāve tried to make it as a commercial product with a centrally hosted server, but thatās not what I made.
TiddlyPWA is fully free (both āas in beerā and āas in freedomā) because I believe in software that empowers everyone to own their data and the software itself, that is never gonna give you up doesnāt die for business reasons.
Right now TiddlyPWA is still early in its development, still missing a lot of things, but hopefully is solid enough to make you check it out. If you like what you see, please consider helping me continue working on it:
- go ahead and try TiddlyPWA first of course!
- report issues, read the code and suggest code changes on Codeberg;
- support me on Patreon :3 Iām trying to do this open source thing full time, which is only possible with your support!
- follow me on the fediverse;
- share TiddlyPWA with anyone who might be interested.
Thanks!!