Re-writing Firn in rust - The End

Tagged: rust

Ahead - a fairly long monologue on the wrap-up of the re-write of Firn. This is mostly written for myself so it might be a bit boring (yak-shave-y) to read. You've been warned.

Why did I do it?

When I originally wrote Firnº I was coming back into the world of programming after a short break. I was still very interested in Clojure (I still am) and so I made pretty typical decisions driven by my fascination and enjoyment of a particular tool; in this case, I wrote a CLI tool using Clojure. At the time I didn't realize that I was setting myself up to go against what would become certain software platitudes that I would, somewhat abashedly and yet grudgingly, integrate into my ethos in the future:

  • Pick the right tool for the job (perhaps the most cliché).

  • Why use two when you could use one?

  • Keep it simple when you can.

  • Favour less dependencies and more hand-rolled code.

  • Don't work on performance unless it's actually too slow to use.

  • Avoid hulking virtual machines?

My goals for the project would emerge over time:

  • keep it light (use fewer dependencies).

  • Make it portable and easily installable (single binaries).

  • be able to keep the domain in your head for as long as possible.

  • only work on performance when it gets too slow for you.

The wrong tool for the job

I picked clojure because that's what I liked/what I knew/what was the right tool to fit the energy I had at the time. When I first built Firn I couldn't have mustered the energy to learn Rust (it looked so intimidating on top of that!). So instead, I looked at how I could use Clojure to build the tool I wanted. It was a strange collision of technologies that made building Firn with Clojure and a dash of Rust possible; the emergence of the GraalVM and its Native Image utility enticed me to be able to build a CLI tool that could be dropped into place and used immediately (in retrospect, I am perhaps overly concerned with this facet) - and using some of the existing work in the community I was able to tie together my application code in Clojure with the parser library and a small bit of Rust (just enough to be dangerous, as they say) all to have it compile into a binary for mac and linux. Very cool ( I still think what GraalVM offers is neat.)

So why was it the wrong tool for the job?

Two languages

I didn't need to have two languages. The answer was sitting there from the beginning - the parser that I would choose to use was in Rust and was available as a library - I should have written the code in Rust from the beginning. All I would be doing is writing application code to read files, template some html, and spit out some files in the end (I'm being reductive, it took longer than that for what I wanted).

Too-New-Tech

GraalVM looks very cool, but it's too new. I had to really dig for answers and I was out of my league trying to figure things out. First off, I was one or two steps removed from getting answered because my Java knowledge is not quite good - I've been able to put off lots of Java domain knowledge while still writing Clojure - however Clojure isn't the the first-language-in-line to use GraalVM. Finding people who were doing Clojure-y things with GraalVM-y things was hard. I was often just pilfering code from borkdude.

Other challenges with Graal - compiling the whole thing on my computer was a drag. It took forever, sent the fans blazing, and I couldn't really even test the end artifact (everything was done in a REPL) - this made testing my end-use-case scenarios to be more of a pain - I was waiting on CI builds to spit out a 50mb artifact for testing. Not ideal.

I had never connected two languages before, and while it worked, I realized that when working between two languages (especially two fairly new languages, and niche ones at that) that it can be hard to figure out how to do things. I don't know much about NIF's or the JNI so I won't say much more - but it mostly felt like a gray area of non-confidence - which I didn't want to have floating around in the code base.

Difficult for contributors

I never assume that people will contribute to my projects but by using this slew of tools I was not exactly making it easy - it's challenging enough to find folks who use emacs+org-mode, let alone also know clojure ... and a bit of rust.

All in all, the whole thing felt like it was built on a pile of mismatched bricks and stones. It felt like a tower of blocks that would come falling down, if one small brick was pulled out. I'm running out of good analogies here. Let's move on.

Feature Parity

I reached a place that I considered near feature-parity for the two versions. Normally, I wouldn't have cared about this, but because a few users seem to use Firn I decided to get everything I needed into the new release, and then try and integrate most of the features that already existed (a few got scrapped). Since this software is still pre 1.0.0, I'm fine with breaking changes.

Experience with Rust

I should write a bit about Rust. So far I feel like I've learned maybe 50% of the language. The first 20 hours of the project felt like I was swimming upstream as I worked to understand what the compiler was telling me, and navigating the syntax of the language. Lots of things that felt weird feel totally normal now (modules, imports) while other things still seem hazy (when to use String/&str, or Path/PathBuf) and other things are still completely just me hacking without bothering to understand what's going on behind the scenes (I don't know what derive does or what traits really are. I don't quite get lifetimes). But this is just the way I work. I'm a little embarrassed writing this now, but I also know I can't force myself to understand everything about a new topic / language / domain and then write a program in it. It's better for me to just write and learn when I get stuck.

Some things that are still behooving me a bit with the project:

  • I think I've got some code that could be DRY'd out in which I'm parsing the strings for org content more than once - I can't seem to work out how to pass the Orgize struct (which has a lifetime) into the templating engine's (Tera) context - instead I have to re-parse the string for a file more than once if a user is rendering say, two separate files and a TOC (that's 3 different instances of parsing the original org string).

  • The file organization is a confusing mess. Orgize provides an html writer (which I don't quite understand) that interacts with the parsed org tree - it iterates over things and then writes "start" and "end" strings for each type of element it encounters. But before using that writer, I have to iterate over the parsed content myself and transform the values based on user input (does a user want to change a headline level, or render just a single headline?). Ugh.

  • I made a flamegraph thing and I'm pretty sure the result is not good. Not sure how to debug that yet.

With Rust, I have a (perhaps self-imposed) desire to be writing efficient and fast code. That might be because everywhere I look people are referring to Rust code as BLAZING FAST (which is pretty obnoxious to me, tbh) - so if I'm doing something inefficient I think my ego gets a little in the way and I have the urge to do optimization work. This brings me to one of the aforementioned programming platitudes - don't work on perf stuff unless you actually have to do it. It's easy to obsess over that stuff and feel like you are getting stuff done. I'll worry about perf when this wiki takes longer than 10 seconds to reload when running firn serve or if builds take longer than 30 seconds.

Firn the Testless

Today I released v0.0.16 of Firn, and after that (and a bowl of ramen), I laughed to myself at the thought that I have not written any tests for it. I have not yet even learned how tests operate in rust (although from reading src code I see that they are inlined in the same file as the source). If I'm building a project for myself with no intended audience but me, I will almost never write tests unless it speeds up my workflow quite a bit.

For the original version of Firn, written in Clojureº, I had to write tests. I had a fair few tests, actually. I think I should still migrate some of them, or write a new suite entirely, but there will certainly be less. Using a language with types (especially one with a type system such as Rust's) really eliminates a whole class of errors. I just hadn't really realized this until actually writing rust. And of course, the types make the process of writing code quite pleasant as I can see definitions in emacs as I go. The compiler catches quite a bit and while I am sometimes waiting for crates to compile, I'm not that put off by the time to test a new line or block of code then when I was using the repl with Clojure.

Closing issues

Just by migrating to Rust and out of Clojure I was able to fix a few bugs that had been reported by users. Frankly, this could be in part due to the naive application code in the clojure code base. In particular, the clojure version was rendering html from a deep tree of JSON representing an org file when instead the rust versions is a flat iter that writes html based on start and end of a tag. Even just switching to this fixed a handlful of issues.

Going forward

Moving forward I'm going to implement some niceties for time-tracking stuff (one of the main reasons I run this wiki and use org-mode). I should be able to spit out some fun charts that graph my motivation and spells of energy based on my logs.

I will probably also be able to build a link-graph similar to what org-roam offers out of the box with canvas, which could be a fun project. I'm not sure that I will include that in the binary so much as just write something with canvas using the data that Firn can attach to the js window object.

Read more?

This post is title "The End" as I think it qualifies as the last post summing up the work of the re-write. Anything in the future about Firn will be about new features that make their way into future releases.

If you made it this far and want even more backstory and devlog-y stuff, you can read some of the other posts in this saga.