Image 01

Transneptune

beyond the Kuiper Belt, over the sea

Let’s talk about Sphinx

September 11th, 2017 by Kit

I’ve fielded a few questions lately from folks about Sphinx, reStructuredText, and documentation in the Python world and more broadly. So I thought I’d write up a bit of an intro to how I use and understand these tools, and how better documentation makes me a better developer.

Restructured understanding of reStructuredText

There’s a lot of cargo-cultish behavior and confusion around writing Python docs, and I think that a lot of this comes from seeing some Sphinx documentation, finding something curious in the source, and then not being able to learn what it is or where it comes from. The first source of confusion, I think, is between Sphinx and reStructuredText. Here’s the distinction: reStructuredText is a markup language (with a processor implemented in the docutils package), and Sphinx is a documentation builder or compiler, that takes a bunch of reStructuredText files and makes them into a whole suite of documentation.

Since reStructuredText is in some ways the basis, let’s start by talking about it. It’s a fairly straightforward markup language. If you’ve used Markdown before, it may be just similar enough to cause confusion. Headings are indicated with a series of =, -, or ~ on the following line, paragraphs are broken with two newlines (a single newline in a paragraph is ignored), italic is indicated with surrounding *...*, bold with surrounding **...**, and fixed-width with surrounding `...`. Lists have leading - or *. All broadly familiar.

(There are fancier things, like links and tables, but you can look those up yourself when you need them.)

Now, there are two things that reStructuredText supports that make it particularly useful as a basis for Sphinx: directives and roles. These are custom bits of markup that you can use to define special structures in the abstract representation of the document, and emit into the final compiled document in whatever way needed.

Roles are inline, and generally look like this: :role:`text`. The role is whatever the name of the custom role is (ref, class, index, or weirder things), and the text is whatever text gets wrapped in that role.

Directives are block-level, and look like this:

.. directive_name::
   :directive_arg1:
   :directive_arg2: value

   Some text that gets the directive applied to it.

They can have some keyword arguments (in the immediately following lines, with : surrounding) and a whole block of arbitrary paragraphs following.

There’s a small set of roles and directives included in reStructuredText itself, and a larger set in Sphinx, and vast wide world of third-party ones. Some builtin ones include directives for sidebars, warnings, topics, and roles for things like links.

The Sphinx’s riddle

So now we can see what’s in a pure reStructuredText file. What’s Sphinx give us? It gives us a whole passel of things, but most importantly it gives us a range of output formats, support for cross-references, and support for pulling documentation information from our Python source.

Sphinx starts with a file called conf.py in the root folder, which sets a number of values. Most crucially, it configures the location of the root reStructuredText file, and sets the theme and options used for the HTML, PDF, ePub, and other output. It also lets you specify extensions and third-party packages that enable more roles and directives.

You obviously don’t want your documentation to exist as one giant .rst file, though, right? Just like you wouldn’t want any webpage to be a single giant page. So you break your docs into many distinct .rst files, and cross-link them. The most crucial form of cross-reference is a .. toctree:: directive in the root .rst file, linking together and ordering all the other .rst documents. But you can also use the ref role to cross-link documents in the middle of a paragraph. For instance, make a reference target using this directive:

.. _some-ref:

# Wotta Heading

This is a section I want to refer to later.

And then in another document, link to it:

So, if you consider :ref:`some-ref`.

And that’ll render like this:

So, if you consider <a href="./some_file.html#wotta-heading">Wotta Heading</a>.

(There’s some magic in there, about how headings are turned into URL fragments, and how cross-linking finds the right file, but I hope the general idea is clear.)

Where do I go from here?

Start a documentation project! Or add docs to an existing project. Here’s a quick recipe:

pip install sphinx
sphinx-quickstart

Now you’ve got a conf.py and an index.rst. Write some docs, run make html, and open _build/html/index.html in your browser.

That’s all! You’ve got docs.

One more thing

Now, personally, I like using the dirhtml builder, instead of the html builder, because I like my URLs to end with a /, not a .html. But that makes it harder to view changes locally. So I made a little tool to serve the docs locally, and while it’s at it, to rebuild them as you change them. It’s called phix, and you can get it from PyPI. Assuming you’re running Python 3, just run pip install phix, and then run phix in the root of your documentation project. You can then see your docs at http://localhost:8000/ as you work on them. Enjoy!

Next steps

This is probably the first in a series. I’ve got a lot of other things I’d like to write about on this subject. I’ll link them here as I write them:

  • getting your docs on Read the Docs
  • building docs for different targets
  • writing custom Sphinx themes
  • writing Sphinx extensions
  • why you should try documentation-driven development

Defense against the Dark Arts

May 18th, 2017 by Kit

I’ve just been reading Carole Cadwalladr’s excellent journalism on the subject of Robert Mercer, Cambridge Analytica, and the Brexit and Trump votes. You can see the articles here and here. They’re chilling, and upsetting, and important.

At some level, what Cambridge Analytica is doing is something familiar: propaganda. It’s different in two respects, though: one, it’s targeted at the people most vulnerable to it, and two, it’s crafted especially to trigger their vulnerabilities. So instead of trying to find a message that you can spread all over the airwaves to convince everyone of your message, you find the voters at the margin, who are undecided, find the segment of them that you can sway, and then you find just what emotional buttons to push to sway them. Because you can automate so much of this, it becomes much more cost effective. It also has the added advantage of never showing your message to people who might get polarized away from your goal by it.

Another important factor here is that people can be targeted because of the amount of data we’re just casually throwing off in our day-to-day lives now. Every time you use the internet (and remember, “using the internet” now includes things like buying something with a card, or with a loyalty program), go by a security camera, or more, you leave a footprint that contributes to a shockingly holistic image of your behavior. We have techniques these days to turn that image into a predictive model, and guess with good accuracy whether you are someone who can be swayed.

But this is something we can combat. Education is always the first and best bulwark against propaganda. Critical thinking is like a memetic immune system, allow you to expose yourself to ideas without uncritically absorbing them, and being colonized by them.

That’s all well and good, but what’s scary about Cambridge Analytica, and its ilk, is that they can target the weakest. It’s a basic principle of security that you are only as secure as your weakest point. You can lock the doors as much as you want, but at a certain point, an intruder will come in through the windows.

So what do we do?

I don’t really know, but I know that whatever we do has to have two dimensions: in the social space, there’s educating the public, to bring up the most vulnerable; and in the technical space, there’s starving the propagandists of data. Helping more people, and especially the most vulnerable, be digital ghosts is a vital task.

(And at least CV Dazzle will give us all a great cyberpunk aesthetic for this brave new world we’re entering into. John Waters can rest easy.)

Great American Institutions

January 5th, 2017 by Kit

This country likes to talk at length about how much it values democracy, as a core concept. For a long time, I’ve said that this strikes me as strange, when two out of three of our major cultural institutions are (mostly) based around (substantial) autocracy.

That is, while we pursue democratic government, we maintain autocratic military (in service to a democratic government at least, which, I think, makes sense) and autocratic businesses (which we know we have alternatives to, but rarely pursue).

This is why I have always been opposed to the rhetoric of “running the country more like a business”. To me, that sounds like “run the country autocratically”.

The End of the Third Age

December 18th, 2016 by Kit

I’m going to talk about the Lord of the Rings. Or maybe America, at this juncture in time. Tolkien, famously, hated allegory and rejected allegorical interpretations of his work. And yet, time and time again, people have found an allegorical reading of his Lord of the Rings inevitable and tempting. There are plenty of reads mapping the various stocks (hobbits, elves, dwarves, orcs, men, and more) to various nationalities or ethnicities. But I want to talk about an angle Max Gladstone mentioned recently.

Gladstone suggests that the One Ring can be read as representing the ideological and practical foundations of a militarized fascist state. The meeting point of Walter Benjamin’s aestheticization of politics with the military-industrial complex. I think this is an important read, but not just for what it says about the One Ring. If the One Ring is fascist ideology, what, then, does that mean for the rest of the pieces of the Lord of the Rings?

At the time of the Lord of the Rings, Middle-Earth is at an inflection point. The age of the elves is drawing to a close, and the lands of the elves are proving increasingly untenable. Three holdouts of peace, beauty, and wisdom remain: Rivendell, under the rule of Elrond, Lothlórien, under the rule of Galadriel, and Lindon, under the rule of Círdan. Each of these rulers had one of the Three, three great rings of power that enabled them to hold their realms together despite the changing of the world. But these rings, as all of the other rings of power, were subordinate to the One Ring. They derived their power, ultimately, from the One Ring, and could not survive its destruction.

Where does that leave us? The realms of elves, these havens of beauty and justice, were built on a foundation that was intimately entangled with the roots of what we will read as fascism. With the one, came the other. And, in pursuit of true justice and peace, the elves knew that their realms must cease to be.

I live in Boulder, Colorado. It is growing quickly and particularly attracting those with advanced degrees and technical knowledge. Its history as a hippie enclave and the general political preferences of the well-educated mean that this is a very left-leaning town: including the more conservative suburbs, registered voters are 41% Democratic and 20% Republican. In the aftermath of the election, many people here have been looking around and wondering what the rest of the country is like. We knew we lived in a bubble, but we didn’t think it was that bad. We are, perhaps, in our own Lothlórien. To dismantle the power of the One Ring, we must sacrifice our Lothlórien. In this case, that will require actively pursuing political, economic, and racial diversity that we lack here. In other cases, it will require other sorts of changes.

Ta-Nehisi Coates talks, in his book Between the World and Me, about what he calls “the dream”. I think that I refer to, in some degree, the same thing he does. The Dream is about something more than comfort, it’s about what we call making it, not just living a life that by the standards of most of human history is unimaginably safe, comfortable, and healthy, but doing better than those around you. No one would frame it that way, either explicitly or to themselves, but that is part of the power of the One Ring that we build our havens on: the off-loading of risk, pain, and bodily harm onto other people, in other places.1 Put another way is Gibson’s famous saying: “The future is already here—it’s just not very evenly distributed.”

In the Lord of the Rings, the connection between destroying the One Ring and the loss of the elves’ power to sustain their realms is mystical, powered by magic and allowing action at a distance. In our material world, I think it is quite the opposite: I think that the work of destroying fascism intrinsically requires us to set aside our Lothlóriens, to recognize that however peaceful and plentiful they may seem, that peace is built on the bodies of others and on the same golden-age lies that breed fascism, white supremacy, and death. We must understand that a simple passive desire for harmony perpetuates segregation, and that we must therefore work proactively, struggle proactively, to make our communities more accessible and just and integrated. (What about those (white) liberals living in the most diverse cities in the USA? Yes, even your cities have this work to do. Segregation writ small is everywhere, and economic opportunities are very unevenly spread.) And this will not leave our world as it has been, and while there will be costs in comfort and familiarity, this is good, because this age has been, as all ages are, built on injustice.


  1. More generally, this is something capitalism excels at: accounting fraud. I don’t mean that in a legal sense, but rather in a holistic one. Everything we do in this world has costs, and the more we can put those costs on someone else’s books, the more we can lower our personal tally. There is no way out of this but pursuit of an ethos of holistic accounting, where nothing is dismissed as an “externality”, but rather accounted for. 

South Pacific

December 8th, 2016 by Kit

I grew up listening to a lot of Rodgers and Hammerstein; it was some of my mother’s favorite music, and so there was a lot of it in the house. More than any other musical of theirs, I listened to South Pacific.

Now, there are problems, of course, with that show. It’s a product of its time, and the original cast recording captures in amber some choices we might not make today. But for all that, the writing of the show, the message of it, is pointedly anti-racist. The two main plotlines each deal with different facets of specifically white, American racism, and different answers to it.

I’m listening to it again now, for a whole confluence of reasons, and really appreciating the message of it. Talking with Allie about it, she asked an interesting question: how were the negative responses to the anti-racism of it framed at the time? Where now people might likely say things like “PC gone too far!”, what did people say at the time to try to frame themselves as the “good guys” while opposing a message of anti-racism?

I ended up doing a little reading and research on this. It seems that the two pillars American white supremacy relied on at that time for its public defense were “decency” and “anti-communism”. When South Pacific was on tour, the Georgia legislature apparently introduced a bill outlawing entertainment containing “an underlying philosophy inspired by Moscow.”

How did people at the time undermine that? I don’t know. But I do know that Rodgers and Hammerstein stuck to their guns. They would not change one bit of the show to play better in the South. They turned out to be right not just morally, but artistically and economically: it ended up being one of their longest-running shows and eventually one of the canon of classic musicals, and “You’ve Got to be Carefully Taught” (the most explicitly anti-racist number) is one of the most remembered songs from the show.

There are many fronts we are fighting on today. We’re not just fighting racism and a renewed Jim Crow, but the rise of an authoritarian kleptocracy. But on the cultural battlefront, we can and must make our art unapologetic, as good as we can, and, well, convincing.

Craft and writing prose

March 28th, 2016 by Kit

Prose is important; even if you’re writing for no one else, you’re writing for yourself in the future. The best developers I know write a lot of prose, both documentation and commit messages and comments. The value is not immediately visible, it won’t make your tests pass, but it actually is part of keeping technical debt low; some tech debt comes from not knowing in detail what a piece of code does. Good docs and comments and commit messages let you know at least what it’s supposed to do.

So, prose is a part of the craft of software development, but not just any prose: particularly technical prose, explanatory, lucid prose. And it’s not something we, as developers, are usually taught.

We’re, generally speaking, a group that’s pretty good at self-education, though. We pick up new languages, frameworks, protocols, and principles. So not only must we do this, but we can do this. It’s hard, because it’s big and fuzzy (there are no tests that can tell you if your prose is correct), but still, we can do it.

This post is just an exhortation, not a guide, though. The subject is too large to cover with a simple trick or a general principle. So, how do you begin to write better? First, write more and read more. Second, get your stuff edited.

Thanks to Ryan and Owen.

Programmers aren’t special

March 21st, 2016 by Kit

Listen to craftspeople of all sorts. Learn from outside the bubble. We are not magical. What we do is not magical. It has some cool properties, so do other things. Learn.

(Yes, this means learn about how writers write. Yes, this means learn about how carpenters carpent. How psychologists psychologize. How baristas bar. How sailors sail. All of it.)

Embellish later

March 14th, 2016 by Kit

So, @wholemilk said something I liked:

Yeah. But sometimes that’s hard. Why?

I am a big believer in the value of considering the emotional landscape of any labor, but in my day to day, that means mostly programming.

When you have something that’s working but still incomplete, it can be a big emotional effort to break it so you can start to move it towards the next stable point, the next point at which it’s working(-but-incomplete).

It’s like editing a document: partway through applying edits, you have broken sentences lying around, half-formed ideas not fully explicated, and the thing’s a mess. Sometimes that even shows through, if you fail to clean up after a change to the first half of a sentence leaves the last half incoherent.

It is similar with code.

Fundamentally, this is why source control matters. It is a tool to help with the emotional labor of breaking your work. If you know you can always go back to a stable point, swinging out into the void for a bit becomes exploratory, not all-or-nothing.

Teamwork and Crosswords

March 7th, 2016 by Kit

Allie and I have been doing a lot of crosswords lately. While half the fun is yelling at Will Shortz when a clue is weak, the other half is in working together to solve a problem. And in the process, I’ve noticed something I find interesting. Teamwork can be about unlocking each others’ potential.

In this particular case, we had just done a Thursday crossword in record time, and I congratulated Allie on a job well done together. She countered that I had done three-quarters of the puzzle, and that she didn’t feel like she’d contributed her fair share. I however, felt that she had been crucial, and I realized it was because if she hadn’t been there doing this crossword with me, I would have been stuck at one-fifth and never completed it.

It was a very clear illustration to me of how teamwork, particularly in a complex creative task, is not about everyone doing the same amount of a generic and commoditized work, but rather about doing the right thing at the right time to help the team as a whole achieve more than the individuals would.

I’ve also been reading some pieces lately about Google’s Project Aristotle. It seems to come to a similar lesson, but in more concrete terms: a group’s effectiveness and intelligence is determined by the degree to which the members of the group can all contribute.

I think it’s worth remembering that effort and labor don’t just fill up a meter, and ding when it’s done. The complex web of human relations can make it hard to attribute success to any one piece of the system.

Code Folding

December 7th, 2015 by Kit

Nick made a comment about code folding, and I promised him an essay. Here it is. It’s not very long.

I like code folding. When I know a mature project well, and the modules are kinda big, I want to open a file and not be distracted by seeing things I’m not working on.

I dislike code folding. When I don’t know a project well, I want to be able to see everything without having to drill down into it. When I do know it well, I want the modules to be small enough that there’s just one topic in each file, and so opening a file does the thing that opening a code fold is supposed to do.

So, the second case is ideal, but also idealistic—and we don’t live in an ideal world. Sometimes there are other pressures against breaking a project into a billionty well-named modules, too, and you have to live with the fact that a given module might have some variety of content.

Code folding, like window focus, should ideally follow my unknown innermost desires—the computer should always know just what I want without my having to tell it. Alas.