700,000 lines of code, 20 years, and one developer: How Dwarf Fortress is built

[Ed. note: While we take some time to rest up over the holidays and prepare for next year, we are re-publishing our top ten posts for the year. This is our number one post of 2021! Thanks for reading and we’ll see you in the new year. ]

Dwarf Fortress is a kind of oddball ardour tasks that’s damaged into Web consciousness. It’s a free recreation the place you play both an adventurer or a fortress stuffed with dwarves in a randomly generated fantasy world. The simulation runs deep, with new video games creating a number of civilizations with histories, mythologies, and artifacts. 

It has change into infamous, and rightly so. Particular person dwarves have emotional states, favourite gems, and grudges. And all of it takes place in an ASCII interface that appears imposing to newbies, however feels just like the textual content crawl in The Matrix: craftsdwarf, river, legendary megabeast. 

All the recreation is product of 1 developer, Tarn Adams, aka Toady One, who has been engaged on Dwarf Fortress since 2002. For the primary 4 years it was an element time challenge, however since 2006 it’s been full time. He writes all of the code himself, though his brother helps out with design and creates tales based mostly on the sport. Up till now, he’s relied on donations to maintain him going, however he’s at the moment engaged on a model with pixel graphics and a revamped UI that can be out there for buy on Steam. 

I reached out to Tarn Adams to see how he’s managed a single, rising codebase over 15+ years, the perils of pathing, and debugging useless cats. Our dialog beneath has been edited for readability. If you would like extra, we additionally spoke with Tarn on the podcast.


Q: What programming languages and different applied sciences do you employ? Principally, what’s your stack? Has that modified over the 15-20 years you’ve been doing this?

A: DF is a few mixture of C and C++, not in some sort of customary obeying manner, however type of a large number that’s accreted over time. I’ve been utilizing Microsoft Visible Studio since MSVC 6, although now I’m on some model of Visible Studio Neighborhood. 

I take advantage of OpenGL and SDL to deal with the engine issues. We went with these as a result of it was simpler to port them to OSX and Linux, although I nonetheless wasn’t in a position to do this myself after all. I’m undecided if I’d use one thing like Unity or Unreal now if I had the selection since I don’t know the best way to use both of them. However dealing with your individual engine can also be an actual ache, particularly now that I’m doing one thing past textual content graphics. I take advantage of FMOD for sound. 

All of this has been fixed over the course of the challenge, besides that SDL obtained launched a couple of years in so we might do the ports. On the mechanical facet of the sport, I don’t use a whole lot of exterior libraries, however I’ve occasional picked up some random quantity gen stuff—I put in a Mersenne Twister a protracted whereas in the past, and most not too long ago I adopted SplitMix64, which was featured in a chat on the final Roguelike Celebration.

Q: What are the challenges in creating a single challenge for thus lengthy? Do you suppose that is simpler to do by your self? That’s, since you wrote each line, is it simpler to take care of and alter?

A: It’s straightforward to neglect stuff! Looking for ‘;’, which is a free methodology however shut sufficient, we’re as much as 711,000 strains, so it’s simply not attainable to maintain all of it in my head now. I attempt to title my variables and objects persistently and memorably, and I depart sufficient comments round to remind myself of what’s happening after I arrive at a spot of code. Generally it takes a number of searches to seek out the precise thread I’m making an attempt to tug on after I go and revisit some piece of the sport I haven’t touched for a decade, which occurs fairly a bit. I’d say most modifications are targeted solely on sure components of the sport, so there’s sort of an energetic molten core that I’ve a a lot better working information of. There are a couple of actually crusty bits that I haven’t checked out since earlier than the primary launch in 2006.

Concerning the relative ease of doing issues on my own, definitely for me, who has no expertise engaged on a big multi-person challenge, that is the best way to go! Folks clearly get good at doing it the opposite manner, for instance over within the AAA video games context, and clearly a number of engineers are wanted over there to get issues performed on time. I’d be hesitant to say I can go in and alter stuff sooner than they will, essentially, since I haven’t labored in that context earlier than, however it’s true that I don’t have any team-oriented or bureaucratic hurdles to leap by after I need to make an alteration. I can simply go do it. However I additionally should do it alone.

Q: What’s the most important refactor/change that you just needed to make?

A: There have been some refactors which have lasted for months, redoing sure information constructions and so forth, although I’m undecided something is ever a refactor strictly right here since there’s at all times alternatives to push the mechanics ahead concurrently and it is sensible to take action when the code information is contemporary. 

Including the Z coordinate to make the sport mechanically 3D (whereas nonetheless being textual content) was one other one, and actually probably the most mind-numbing factor I’ve in all probability ever performed. Simply weeks and weeks and weeks of taking logic and performance calls that relied on X and Y and seeing how a Z suits in there.

Making the merchandise system polymorphic was in the end a mistake, however that was an enormous one. 

Q: Why was this was a mistake?

A: While you declare a category that’s a sort of merchandise, it locks you into that construction far more tightly than if you happen to simply have member parts. It’s good to have the ability to use digital features and that sort of factor, however the tradeoffs are simply an excessive amount of. I began utilizing a “instrument” merchandise within the hierarchy, which began to get numerous performance, and might now assist something from a stepladder to a beehive to a mortar (and pestle, individually, ha ha), and it simply feels extra versatile, and I want each crafted merchandise within the recreation had been below that umbrella. 

We do a whole lot of procedural technology, and if we wished to, say, generate an merchandise that acts partially like one factor and partially like one other, it’s simply manner more durable to do this if you find yourself locked down in a category hierarchy. Including issues like diamond dependencies and all that simply find yourself tying you in knots when there are cleaner methods to do it. If totally different elements can simply be turned on and off, it’s simpler, and permits you to do extra. 

I feel some recreation builders consult with this as an entity element system, although it’s my understanding that harder-core optimizer folks consider that as one thing else the place you’re truly breaking issues down by particular person fields. Utilizing a single object with totally different allotted subobjects is sort of definitely worse for cache misses, which is a complete different factor, however the advantages in group, flexibility, and extensibility simply can’t be ignored, and the totally different subfields of the instrument merchandise aren’t used so typically that it turns into an optimization challenge.

Q: Did you run into any points transferring from 32 bit to 64 bit? That appears like a kind of issues that was large on the time however has change into fairly accepted.

A: Under no circumstances! I’m struggling to think about a single challenge. Thankfully for us, we already had our byte sizes below management fairly effectively, because it comes up saving and loading the worlds; the format wanted to be nailed down again after we set that up, particularly as a result of we’ve needed to take care of endian stuff between OSes and all that. And we don’t do any gnarly pointer operations or different stuff that may have gotten us in hassle. It simply ended up being actually good code for 64 bit conversion resulting from our different practices, fully accidentally. The primary challenge was simply getting the time collectively to make the change, after which it didn’t find yourself taking almost so long as I believed it could.

Q: I’ve seen different video games just like DF die on their pathfinding algorithms.What do you employ and the way do you retain it environment friendly?

A: Yeah, the bottom algorithm is just a part of it. We use A*, which is quick after all, however it’s not ok by itself. We are able to’t make the most of a few of the improvements on that (e.g. jump point) since our map modifications a lot. Typically, folks have used approaches that add numerous bigger constructions on high of the map to chop corners, and due to the altering map, these simply take too lengthy to take care of, or are in any other case a trouble. So our strategy has been to simply hold monitor of related elements reachable by strolling. These are fairly straightforward to replace even when the map modifications shortly, although it does contain some flood-filling. As an illustration, if water cuts the fortress in half, it must flood out from one facet and replace a complete half of the fortress to a brand new index, however as soon as that’s performed, it’s good, usually. Then that permits us to chop nearly all failed A* calls from the sport—our brokers simply want to question element numbers, and if the element numbers are the identical, they know the decision will succeed.

It’s quick to take care of, however the draw back is that the element indices are maintained for strolling solely. Which means flying creatures, as an example, don’t have international pathfinding intelligence that’s any totally different from a walker. In fight and some different conditions, we use short-range flood fills with their precise logic to present them some benefits although. But it surely’s not very best for them.

I’m undecided we’ll try different constructions right here to make it work any higher. For our map sizes, they’ve all failed, together with some exterior makes an attempt. In fact, it is perhaps attainable with a extremely concerted effort, and I’ve seen different video games which have managed, as an example, some rectangular overlays and so forth that appear promising, however I’m undecided how risky or massive their maps had been. 

The most straightforward thought would simply be one thing like including a brand new index for fliers, however that’s a big reminiscence and pace hit, since we’d want to take care of two indices without delay, and one is dangerous sufficient. Extra particular overlays can monitor their pathing properties (and then you definately path by the overlays as a substitute of the tiles), however they’re exhausting and gradual to take care of because the map modifications. There are numerous different concepts floating round, like monitoring stairs, or doing a little restricted path caching, and there are in all probability some beneficial properties to be made there. We’re definitely on the fringe of what we will at the moment assist by way of brokers and map complexity, so one thing’ll have to present if we need to get extra out of it.

Q: On that word, you’re simulating a whole lot of issues suddenly—how do you handle so many so many actors asynchronously (or do you)?

A: If we’re speaking about asynchronous as in multithreading, then no, we don’t do any of that, apart from the graphical show itself. There’s a whole lot of promise right here, even with microthreading, which the group has helped me out with, however I haven’t had time to dive into. I don’t have any expertise and it’s a bug-prone factor.

Q: Have you ever tried different tasks/applied sciences alongside DF?

A: Positive! The facet challenge folder that’s migrated between computer systems for the final ten years or so has about 90 tasks in it. A few of them lasted for days, some for a number of years. They’re principally different video games, nearly at all times in different genres, however there are additionally a couple of DF helper tasks, like the parable generator prototype. Nothing near seeing the sunshine of day, however it’s enjoyable to mess around.

Q: Along with your ~90 facet tasks, have you ever explored some other programming languages? In that case, any favorites?

A: Ha ha, nope! I’m extra of a noodler over on the design facet, fairly than with the tech. I’m certain some issues would actually pace up the belief of my designs although, so I ought to in all probability no less than study some scripting and mess around with threading extra. Folks have even been type sufficient to provide some libraries and issues to assist on the market, however it’s simply troublesome to dam facet challenge day trip for tech studying when my facet challenge time is for stress-free.

Q: You may have probably the most fascinating launch notes. What’s your favourite bug and what brought on it?

A: It’s in all probability boring for me to say, however I simply can’t beat the drunken cat bug. There’ve been a couple of movies made about it by this level. That was the one the place the cats had been exhibiting up useless all around the tavern flooring, and it turned out they had been ingesting spilled alcohol once they cleaned their paws. One quantity was off within the ingest-while-cleaning code, and it despatched them by all of the signs of alcohol poisoning (which we added after we spruced up venomous creatures.)

If you wish to attempt Dwarf Fortress for your self, you may download it from their website.

Tags: dwarf fortress, solo developer, video games

More Posts