Technical History of Snowdrift.coop

This post aims to provide some perspective on the challenges we've had with building Snowdrift.coop using Haskell and on my personal experience getting into this as someone with virtually zero tech background.

My background

Although a competent computer user, I made it to my early 30's successfully avoiding learning any programming. After years as an Apple user, I was deeply worried about the future. I saw the iThings business model taking over, and that means: walled-gardens, censorship including the effective banning of community-built free software,[1] and advertising and tracking infused into every last crevice of our technology. In early 2012, I managed to struggle through installing GNU/Linux on an extra laptop, and that began my crazy journey deep into the free software world.


For more about how I got into free/libre/open issues and GNU/Linux, check out my story at my user profile. I also tell another framing of my story in my talk from Open Source Bridge 2015 (video is on YouTube): Bringing Non-Technical People to the Free/Libre/Open World and Why It Matters.

Getting into programming

The first several months into starting Snowdrift.coop, I knew nothing about the technical side of the project. My programming experience amounted to about 10 total hours of BASIC in elementary school and knowledge that C, C++, PHP, Java, JavaScript, Python, and Ruby were names of popular programming languages today. When David (my friend and co-founder who convinced me to do this at all) chose Haskell, I knew we'd have downsides due to using a less popular language that didn't already have all the robust libraries of something like Drupal, but I also had no perspective and trusted David's judgment.

I mostly worked on non-code areas: research, writing, and design concepts. But David told me to learn Vim. He then showed me how to use the Hamlet files in Yesod's Shakespearean Templates to write basic HTML but with whitespace and no need for closing tags.

Initial workflow

My early workflow for Snowdrift.coop looked like this:

  • ssh into the snowdrift.coop server where David had set up a user for me
  • open .hamlet files with vim
  • go to insert mode (which is mostly all I used at first)
  • write basic prose about the project, the politics, the plans, using basic HTML tags
  • save
  • tell David I had updated a file

More tools to learn

Eventually, we moved to our wiki system, and I had to learn Markdown. I slowly also learned more HTML and basic CSS (and Bootstrap specifically). Then, David encouraged me to start building the code locally on my own computer. For that, I had to learn about Git. I felt somewhat hesitant and resistant at each of these steps because they all seemed like further and further delay keeping us from getting launched. I really wanted to get more experienced programmers involved, not to become a programmer myself. I wanted Snowdrift.coop to exist, and I didn't lack for non-programming tasks to work on.

Cabal hell

When I first tried to build the code locally, I ran into some dependency conflicts and had to ask David for help. I remember feeling totally baffled, and I struggled for days to get going. Later, after a long process of David updating the site to Yesod 1.2, I had dependency conflicts yet again.

I also started recruiting others in conversations around the GNU/Linux music community and general GNU / Free Software community — and those who volunteered also ran into dependency conflicts. A few, notably Jason (JazzyEagle), persisted, but it took literally days — even weeks sometimes — to get things straight. Even after successful builds, something would break later and all the troubleshooting and trial-and-error would have to happen yet again.

At that time, the only way to determine versions needed for all the libraries was in the Snowdrift.cabal file which specified bounds. If we pinned a specific version and then had to change it later, we'd be back to troubleshooting again. All the libraries were system-wide, so anyone with multiple Haskell projects had it especially rough.

Cabal version 1.18 introduced sandboxes. That meant we could build separate dependencies per project. We also got cabal freeze in version 1.20. That made it easier to generate separate cabal.config files that pinned specific versions of all dependencies separately from the Snowdrift.cabal general bounds. With all that in place, building was more reliable.

Database setup

The other challenge in our building was the database setup. I initially didn't understand the concept of having a separate database. We use PostgreSQL, and setting that up varies somewhat among different systems. We had to run a whole series of commands to get the database working. And I had yet another tool and programming concept to make sense of.

Eventually, Mitchell wrote a script to automate the database setup, and later Nikita updated it to be a more robust Haskell executable that accepted various arguments.[2]

Working for reliable builds

Even with sandboxes and cabal.config version specifications, getting volunteers set up on their various systems remained quite a burden. None of these Cabal library tools helped with the GHC compiler itself. When different dependencies or code changes were needed for different GHC versions, we ran into further conflicts and challenges. Some people had outdated versions, and we had to have them go through a tedious manual process to update. Others (esp. on Arch Linux) got new updates early on, and then they ran into conflicts each time. Then, we had to struggle to either support multiple GHC versions or make everyone on all systems update to the new version.

Trying VM, Nix, Halcyon, Vagrant…

We tried many options for managing all the different issues. Early in 2015, Joe spent time preparing a full virtual machine image based on Gentoo so that others could potentially download and run the entire system and at least have it working. Unfortunately, that required additional manual steps for volunteers to then access the code or the running site outside of the VM. That solution wasn't completed smoothly enough and never caught on.

Nikita later worked to get everything building with Nix, but when I tested it, I ran into some GHC bug. Nix also required a whole separate structure and system to learn.

Mietek, the developer of Halcyon, actively helped us get going with his tool. Halcyon allows systems to install multiple versions of GHC side-by-side along with handling many other issues of sandboxing and even functions for downloading prebuilt sandboxes for various systems. Although it was a potential solution, somehow we didn't get it all figured out (maybe just misunderstanding or lack of clarity about who was going to finalize things). I don't quite understand the issues, but Halcyon does some odd things with installing in a new /app directory in the root system, and there were questions about how Yesod works where Snowdrift is both a library and a separate executable.

Seeing all this hassle, Paul (singpolyma) set us up with a Vagrant build. That solution finally worked, although it required a huge download and some funniness. Essentially, Vagrant is a virtual machine running our code which allows more smooth interaction with a user's regular system. While workable, Vagrant was still a bit cumbersome, and not everyone wanted to run a whole virtual machine just to hack on Snowdrift.

Stack to the rescue

This summer 2015, about 2.5 years after starting Snowdrift.coop, we finally have a pretty satisfying build process. Stack is a new tool similar to Halcyon but with an easier overall process. Bryan got set up Stack for us alongside substantial other organizing and technical tool updates.

Stack works in conjunction with something called Stackage which is a listing of specific sets of common libraries known to build well together. Stack lets us specify a version of Stackage along with any extras we need otherwise and build the site with a simple command: stack build.

Stack doesn't require separate manual setting up of sandboxes. It will automatically install the right GHC if not found already. It doesn't cause conflicts with any existing installation. It works well on a wide variety of systems. It feels like we finally have dependency hell behind us. Our build process is finally short, simple, and reliable.

Documentation

Each step of the way, whenever we got over some obstacle or I learned a new procedure, I always added it to our documentation. "Oh hlint? What's that? Huh… I guess I'll add it to the docs so others know about it…" Over this time, the documentation became a complete listing covering issues related to text-editors, Haskell, PostgreSQL, Yesod, Bootstrap, JavaScript, Git, and more.

Of course, we can't know exactly the background of each person looking at the docs. I wrote with no assumptions, considering someone like me who knew nothing. So, others took the docs and split them up so it would be more accessible for different audiences. We put the core items in the README, made a new basic BEGINNERS file (thanks to Peter (pharpend) for substantial help with that), and put more in-depth technical details and system-specific notes in a GUIDE file.

Learning Haskell

Throughout this process, I got pushed more and more into studying Haskell to understand the actual code. I read several books and websites. My favorite was the Haskell Wikibook for several reasons: The Wikibook is fully FLO, it's a wiki (so I improved it as I went!), and the quality is actually excellent. I'm not sure I could today write a substantial Haskell program, and I'm not yet familiar with the common Haskell libraries, but I can read our code, somewhat figure out what's going on, and I can tweak and fix things that others have written. I also now understand Git to reasonably advanced levels, use more advanced operations in Vim, and prefer Markdown over WYSIWYG text editing.

Looking forward

Over the nearly three years we've been working on Snowdrift.coop, I've become familiar with so many technologies I didn't previously know existed. It's been a fascinating experience. I would never have guessed that a simple website would be so complex. We haven't even touched or discussed some areas like analytics.

I think if we had used a language with more mature tools, we would be much further along today. Of course, it matters that the people doing the work choose tools they know well, and I see lots of advantages to Haskell. I'm not really qualified to compare languages — I still don't know any others besides Haskell, but I've never met a single person who knows Haskell and doesn't love it, and I can't say that for any other language. Experienced Haskellers tell me how much other languages would disappoint me now that I know this supposedly ultimate wonderful language. Accepting that Haskell has all these advantages technically, I hope it pays off in the long-term, even though it has delayed our initial launch.

I don't need to go into details here about why Haskell is so great (that's written about elsewhere by people who know more than I). What I know is: we lost a lot of time dealing with dependency conflicts, but now things are better. If I were starting a project today, I'd be much more enthusiastic about Haskell thanks to Stack and other newer developments. The Haskell community is wonderful, friendly, enthusiastic, and growing. We have people learning Haskell just because they want to help Snowdrift.coop, and we have people coming to help Snowdrift.coop specifically because they want to work in Haskell.

I wish it hadn't been a struggle to get where we are, but I'm ready to move forward. I still don't identify myself as a "programmer", but I now believe I have the capacity for it, and I know a lot about the overall issues and tools. However, I still think a lot about the experience of non-technical people and how to make sure our project is welcoming and accessible to them. I remain quite conscious of how many levels of complexity I went through to get to the understanding I have now (which I still think of as a beginning level).

Come help!

With our better (and still improving) tools and documentation, we're that much more welcoming. Yet with all the lost time, we're also behind on our development and that much more in need of volunteers to help us build this thing and realize our mission.

For anyone even interested in helping with any of the tech stuff: I made contributions back when I knew virtually nothing, not even much HTML, and you can too! The first step is just to follow the README and build the code. Let us know if things are as smooth now as we hope. We'll take it step-by-step from there. It's a great time to get involved. Together, we can build a better world and learn a whole lot along the way!


  1. I used several programs such as Audacity which contributors around the world build and share under the GNU GPL. Apple's App Store terms ban any user from even installing such software! See Apple's Crystal Prison. ↩︎

  2. And we have someone else working today to improve this still to make a better setup experience avoiding the need for the script to use sudo.] ↩︎