This talk isn’t about any of those. It’s about mostly-text sites that, for unfathomable reasons, are growing bigger with every passing year.
This is so true that it makes me laugh and cry at the same time. I weep. I weep for the Internet. The Internet we know today was made possible by advertising, because too many of us don’t understand how reality works. That’s a good thing—that the Internet happened and grew to be as pervasive as it is—but the current trajectory does not lead to the best possibilities.
My fear—or maybe it’s better written, as “my lament”?—is that for every made-it-big tech person who represents the worst of avarice and greed, there is a sea of regular tech people who are being ground up by the works. Countless pasty faces staring at screens, drinking diet soda, trying to live in the bites of life they can grab after hours, (taking their phone so they can be summoned, of course!) stressed-out, burnt-out…
So when I hear people talk about “tech people” as if we’ve collectively done something wrong and messed up the world, I look around and all I see are people who’ve been broken and smashed. The grass is no greener on the inside-tech side of the fence. To everyone outside-tech, what gets done inside tech is magic—it’s not, it’s factory work, round two.
I don’t mean this as a repost to what people say when they lament what has happened to the world, but as a commiserating plea: “Yes! Yes! The problem is everywhere.”
I was exceptionally lucky to be born into this moment. I got to see what happened, to live as a child of acceleration. The mysteries of software caught my eye when I was a boy, and I still see it with the same wonder, even though I’m now an adult. Proudshamed, yes, but I still love it, the mess of it, the code and toolkits, down to the pixels and the processors, and up to the buses and bridges. I love the whole made world. But I can’t deny that the miracle is over, and that there is an unbelievable amount of work left for us to do.
This hit me right in the feels. I think I’ve had a larger share of the upsides and a smaller share of the downsides than Ford. But this feels like a good overview of my formative years in tech.
Somewhere I read, “the messiness cannot go into the computer.” That summarizes what I believe is the cause of my neurosis; I’ve spent so many years now taking real-world problems, and real-world interactions with people, and factoring them into computers—and I’m left with the messy parts of the problem stuck in my mind. I’m not sure one can even understand what I’m talking about until you’ve spent 30 years, daily, working on refactoring the fuzzy of the real world into the binary of the computer world. Maybe I can reword it this way:
Computers and brains are very different. I’ve spent decades using my brain to understand computers, work with computers, and program computers.
What if that has fundamentally changed my brain?
How can I possibly pretend that, “what if,” is not utter bullshit…
This article got out of hand, and is 3,000 words. I encourage you to skip around; There’s more discussion of specific tools and how-to material in the last third.
In the beginning
In the beginning, podcasts were simply audio files which people shared directly either by emailing them to each other, or by providing download links on their web sites.
RSS technology has been around for a while and provided a way for a web site to publish a “here’s what’s new” feed. It was soon realized that RSS could be used to provide a feed specifically of podcasts. Software able to retrieve and understand a podcast feed, could then automatically download the podcast files, and alert you of new episodes. Sharing a new show with friends then meant simply giving them the “feed URL” which one would “subscribe to” by adding it to your feed reader.
It was only a matter of time before someone realized that a central directory that stored all the feed URLs would be extremely useful. That directory would enable me to search for new shows, new episodes (across any show), and so on.
All of this depends on meta-data; information about the shows and about each episode. Applications for mobile platforms, desktops, and web sites sprang up each of which communicated only with some directory. At first, it was just Apple’s iTunes directory of podcasts, but now there are many different directories maintained by various entities.
Today, “submitting” your show means getting it added to a directory.
About the feeds
A feed contains two things: Information about the podcast (name, author, description, etc.) and a list of episodes and their information (title, description, download URL for the audio file, etc.). Specifically, it’s a file marked-up with XML which is meant to be read by programs. Here’s a screenshot of a podcast feed shown as raw text:
Here’s the same feed shown using a tool which understand the XML and can present it in a way slightly easier for humans to read:
Ignore those yellow highlights as it’s just commentary from the parser. Note that the display goes for more than 1,600 lines.
No one creates these feeds by hand. They need to change frequently, (each time an episode is published,) and it’s tedious. So show feeds are generated automatically. This is why you “publish” your show in one place, and then you have to “submit” your show to the directories, but then you can publish new episodes without having to notify anyone/anything.
You can install software on your web site which will let you create your episodes and it will generate your feed from there. That software would tell you the URL where it created your feed. You can join a service which lets you upload your episodes, and type in some of the meta-data. The service will then handle playing the file to people who use their web site or mobile app, and it will, (perhaps,) generate a feed for you too. In both these cases you can take your feed URL and submit it to other directories. (I’ll talk more about how this leads an ecosystem, below.)
The key take-away here is: Some software or service is creating your podcast feed (the “publisher”) and some service(s) are consuming your podcast feed (the “directories.”)
The publishers—whether this is your own WordPress web site with Podcast publishing plugins, or service like Simple Cast—accept your episodes when you upload them, take the meta-data you enter (title, description) and create your feed. The directories accept your show’s initial registration and then continuously consume all of the feeds from all show publishers, building a database of all known shows and episodes.
Potential problem: Detecting updated feeds
Directories have a big challenge: If there are +500,000 shows, by definition that means there are that many feed URLs. People want to see episodes appear immediately. The directory has to monitor those half-million URLs for changes. How often does it “poll” each URL?
After over a decade, things are quite advanced. Directories can ask, “is it updated?” instead of “give me the whole thing” and checking against their last fetched data. They can also learn how often each show generally publishes, and then adjust their checking to better align with that. But this is still a HUGE number of requests they need to make.
On the other side, where your feed is published, do you really want each directory (there are dozens) asking your publishing service, “is it updated?” “is it updated?” every five minutes?
This is simply an effect of the nature of RSS feeds; They are “pulled” by the consumer. The solution, created long ago, is to create a push-pull system, (I prefer to call it “bump-pull.”) The publishing service, (WordPress, Simple Cast, etc.,) sends a small alert to a hub whenever the podcast feed is updated. The directories subscribe to podcast feeds by telling the hub, “notify me if this feed URL notifies you.”
Now the directories can vastly cut down their workload. They immediately fetch any RSS feed when the hub bumps them to inform that the publisher bumped them. They can also look for stale feeds which haven’t been updated recently and proactively make a direct request to the feed publisher, “hello? is this thing still on?” and see what they get back.
This started as a proprietary project and has become an open standard, WebSub.
Having covered the publishers, and the directories, the last part is the players (the software which plays the audio files.) Your favorite podcast player has three parts:
A searching and subscribing front end that enables you to find things in somedirectory,
a download-and-store or streaming (or both) manager for the podcast files,
and an audio player. (…hey look, finally! …this is the entire point of podcasting!)
This creates a beautiful ecosystem where the publishing services do their job creating standardized feeds, and serving out the podcast audio files when they are downloaded or streamed-live. The directories create large databases of what shows and episodes exist, their titles, their content ratings, where’s the episode cover art, etc. The players (mobile apps, desktop apps, and web sites) handle all the end-user interface features (playback speed, silence-skipping, rewind, subscribe, sharing, episode play ordering, auto-download for offline listening, and on and on.)
Potential problem: Who pays for this?
I don’t believe our species can survive unless we fix this. We cannot have a society, in which, if two people wish to communicate, the only way that can happen is if it’s financed by a third person who wishes to manipulate them.
Publishing: My personal soapbox is that the podcast creator must choose to pay for the publishing part of the ecosystem. There are a few ways to do this, from choosing a partner which charges to create your feed, to rolling your own from scratch with a hosting company. It’s a good sign that there are a huge number of ways to “publish” your podcast; This part of the ecosystem is doing very well.
Listening: For your podcast player, you should always choose one which is notfree; It’s best if it’s one which charges a recurring fee, (annual is nice,) to the software provider to support their work. Never use free-with-ads players, unless you can pay to remove the ads (and you should do so to support that project.) This part of the ecosystem is doing mediocre.
Directories: Unfortunately, the directories you can’t actually pay. Some of the publishers are their own directory, which they use to enable only their own players. So in some cases you are effectively supporting a directory through your choice of publisher. That may or may not be a good thing.
Fracturing of the ecosystem
Suppose I want to listen to a show. (My friend said I should listen to, “Movers Mindset.”) I open my favorite player and search, but I can’t find it. What do I do? Well, as the end-listener, I can do nothing. As a show creator you are going to find this endlessly frustrating.
What’s going on is that some entities want to control the listeners (the people using the players,) so they entice podcast creators to submit their show feed to their closed directory. There are many of these closed directories. Even worse, they always come with odious click-wrap contracts; To submit your show, you have to agree that they can alter it by adding ads, or that you must defend them in court cases, and so on.
Apple’s iTunes directory is currently the only open directory; Meaning anyone can create player-software that uses their directory. Creating a player is not easy, and there are rules from Apple about directory use, but it is open to anyone’s use. There are other open directories, but they don’t have nearly as many shows as Apple’s, yet.
Currently (spring 2019) the BBC is embroiled in a PR mess for having removed all of their shows from Google’s podcasts directory. (Google’s podcast directory is not open, but Googles player apps use it, and that’s a lot of listeners.) The BBC wants people to use the BBC-provided players. But the rest of the podcast universe is not in the BBC’s directory, and so not available in their player. That means I have to have two players: one for the BBC and one for all the other plays-well-with-others podcast shows. In reality, I feel it’s best to simply shun any shows which refuse to be good podcast citizens and feed into the public directories.
On the other hand, it is not a good idea to have one single entity stewarding the only open directory that is large enough to be useful. We’re all currently relying on Apple to remain the adult in the room as the caretaker of the directory. I don’t like this idea either. Switching our lone egg to another basket controlled by someone else is no better.
What we, the podcast creators need is a self-assembled directory that we all co-create and take part in supporting. It would be built by the creators, and run by funding from the creators. This would become the One True Directory. Each show would pay a small annual fee to register in the global directory. This may sound familiar. Solving the problem of who controls the domain names on the Internet is the same problem of cooperation. The U.S. government started the domain name system, and some wise people worked really hard to create and shift everything to a global entity which now controls it democratically.
Potential problem: How many episodes can a show have?
The answer would seem to be “as many as the creator wants” but there’s a potential problem with the podcast feed structure. How many episodes are actually included in the feed? Directories do not store episodes’ information if they disappear from a feed; You wouldn’t be able to delete an episode if they did that. (Well, not easily, and not without some changes to how RSS works.)
The directories just keep the information which is currently in each feed. If the feed has 12 items, (some real-world defaults are this low,) when you publish episode 13, episode 1 “falls off” the bottom of your feed. This makes it disappear from the directories and from the players. (Cue your friends saying, “I can’t find that episode in my player.”)
If you want every episode of your show to remain in the directory, and therefore findable and playable, you need to ensure the feed will include that many episodes.
Here’s where it starts to be clear why you might want to control your own publishing deployment. What if your chosen publisher only includes 25 episodes? (More realistically, the free version of their publishing service limits you to a very small number to encourage you to step up.) If you have control of your podcast feed publication you can always change anything. The downside of course is the need for the technical knowledge to operate and publish the feed.
How to examine your feed
There are many web sites that will validate XML. (Remember, your show’s feed is marked up with XML.) A great example, which has been around as long as I can remember, is from the World Wide Web Consortium (W3C). Their XML validator is https://validator.w3.org/feed/ and here’s what is has to say about the Movers Mindset feed:
That’s rather gnarly looking. But it is also very accurate. If it mattered, I would change the feed to fix every one of those problems it reports. As it turns out though, the directories are not so strict when it comes to reading these feeds.
What we really need is an XML validator that understand we are trying to verify our podcast feed. Turns out this is also a thing. Meet your new best friend, the Cast Feed Validator. Here’s what it has to say about the Movers Mindset feed:
(Go find your feed URL and put it into the Cast Feed Validator.)
This is much easier to understand. The Cast Feed Validator is doing two jobs: Checking the structure of our feed (the XML validation) and checking for “podcast sanity.” It does a beautiful job of visually presenting the information in a way that we can see what’s going on, without having to directly read the XML ourselves.
We can take this a step further. Podnews.net runs a free lookup service which checks a lot of things about your feed—not the feed XML validity per se. I call this “meta-commentary” as they will tell you which directories you are visible in, and other nuances such as wether you are doing your titles in a way that makes Apple happy. The Movers Mindset page from Podnews’ includes a lot (not shown here) but this section is priceless:
One day, something is wrong… is it with the publisher of your show’s feed, the directory, or the player?
First off, I’d like to point out that since you are now able to understand how those three interact, you are way ahead on problem solving. You’ll know to ask questions to start teasing out, (from whoever reported the problem,) in which of the three parts of the ecosystem does the problem lie. “What player are you using?” is something you will grow tired of asking.
The vast majority of problems I have encountered, (or heard of,) fall into two categories:
the directory doesn’t include the feed at all, or it hasn’t fetched the feed recently
the information in the feed is wrong, mangled, missing, etc.
The first problem is mostly beyond the control of the podcast creator. We can submit our show’s feed URL, and we can reach out to support if it’s not yet included. But generally, directories poll for changes as they see fit and you’re either in, or you’re out. However, “I can’t find it in my player,” is almost alwaysbecause your show isn’t in the corresponding directory. This isn’t really a problem. It’s really just expected behavior.
The second case is where all the problems fall.
But first, I want to show you one more tool that will take you to Wizard-level.
Saving a copy of your feed to a text file
If you can save your podcast feed to a file, then you can do “before and after” comparisons. Imagine you find a problem and you think you’ve fixed it. (Perhaps you’ve made a change in your publishing service’s web interface; maybe you want to change all your episode titles to conform to Apple’s latest rules.)
Unfortunately, I cannot provide specific instructions because there are so many environments. But if you can figure out how to save your feed from your web browser—point your browser at your podcast feed, what happens? …can you save-as… to capture it? Or, if you know how to use a command-line on your computer, do you have the “curl” program installed? And so on…
I know I’m way out in the weeds here, but here’s a screenshot from a tool for showing the differences between two files. In this case, I saved a copy of the feed before and after making some text change. (I edited the HTML tags in the sponsorship section of an older podcast.) I can quickly verify that the onlydifferences are what I wanted:
If I didn’t know how to do this with my feed, I’d have to wait for some directory to load the feed and then wait for my favorite player app to refresh… and then realize I have typo… arrrr! another day-delay… Instead, save a copy, make a change, verify the change, and I’m done.
Namespaces in XML
Wow, you’re still reading?
You may have noticed in the screenshots of XML, that some tags are, <title>and some have a colon in them like, <itunes:title>. The itunes: part is an XML namespace. This enables everyone to invent their own tags without having to all agree on the tag names. So Apple’s directory will use the <title>information, unless you specify an <itunes:title>—then they’ll prefer their tag. You’ll see this come up when you read things like, “if you use Apple’s tags, you can…”
There are lots of namespaces and they’re declared at the top of the XML files with xmlns:URL in the (in this case) main <rss ...> tag that begins the XML. Don’t bother learning them—even I’m not that crazy. Instead, just remember that namespaces are a thing that groups tags.
In early 2019 there was a lot of noise about Apple changing the rules about podcast tiles. Your first thought should be, “why does Apple make the rules?” …they don’t, they make rules about what their directory will accept. Technically, they always had the rule that caused the fuss. What happened was they emailed all the creators and said they were going to start enforcing it. The rules is that numbers—among many other things—are not allowed in titles. So, “23. Interview with John Doe” is not allowed as an episode title by Apple.
Everyone freaked out.
A few people noted the rule is actually: Numbers are not allowed in Apple’s title tag. Which means, not allowed in the title tag in Apple’s namespace, our friend the <itunes:title> tag. So I went in and fixed all my <itunes:title> tags. In the end this was an improvement to our feed overall. Now I have my general <title> tags with what I want, (we happen to use numbers as prefixes,) and I conform to Apple’s guidelines for their Apple-namespace titles.
Remember that feeds are meant to be read by the computers. You shouldn’t have to ever mess with any of this, but now that you know how…
I didn’t bother talking about the various tags in the feed file. Now that you have some tools, you’ll discover the tag names have obvious meanings.
At the dawn of the internet, posting a commercial message was the indicator used by everyone to point and say, “that is spam.”
This was a huge mistake. Because it led to a deep rabbit-hole of requiring us to answer the question: Is this message commercial?
I think it’s commercial? …do you? Wait what is “commercial” is it any time we exchange any amount of value? That’d be two people talking! “Commercial” isn’t inherently bad… Ok, but we need to agree so we can make a decision! Is “we” a few of us in this space, or does the poster’s opinion matter? Does their “street credit” in the space affect how much we value their opinion? Maybe we can rate-limit how many border-line-commercial messages each person can… Oh, wait, I know! Let’s appoint someone to be the arbiter of this space and… deep. deep. rabbit. hole.
And we went to great length to try to place (move, cajoul, beg, etc) the commercial stuff into designated areas.
It’s not commercial that is the problem. SURPRISE is the problem. If something is unexpected, it better be perceived as desired. It’s not the content of the message (post, email, phone, whatever) that matters, it’s the recipient’s REACTION that matters.
That phone call at dinner from the caller ID you do not recognize—unexpected and undesired—spam!
The garage that fixed my car that later robo-calls me to beg me to 5-star rate them—unexpected and undesired—spam!
The web site pop-up dialog talking about…—spam!
So the first challenge is to get control of the channels. I’ve moved away from anything where random people can easily interrupt me. (Where “moved away” means everything from literally eliminate said thing, to change or reconfigure how it works, etc. My “inner circle” of people can easily surprise me, of course!) This drastically reduces surprises, and so drastically reduces spam.
Then the second challenge is to locate the channels that contain the information—including commercial information—which I want to receive. My favorite clothing retailer has learned that I like to be surprised with email from them. Commercial? …absolutely. Spam? …yes, please.
They just assumed I must be just like all the other people they represent- hungry and desperate and willing to sign anything.
~ Jason Korman
People sometimes ask, when the Movers Mindset podcast isn’t available in their favorite podcast-player app, why not?
tl;dr: odious clauses in click-wrap contracts.
You should see some of the things! Obviously, there are “hold harmless” clauses obsolving them of any possible responsibility—sure, ok, that’s fine, I am deriving benefit from having our podcast distrbuted through your thing. But some of them want the right to insert ads—not just run ads before or after. Sure, ok, again, you need to pay for your thing; I get that. But insert ads in the middle? Or how about clauses that bind me to defend them in any lawsuit. Not even just related to the content we created—but any lawsuit. Or how about my not being allowed to mention in our advertising that we’re being carried by their thing . . .
sorry i digress
These odious cases have arisen because it’s a lovers’ triangle: The thing/app convinces the users that everything is rosy. The users lean on me because they can’t hear the podcast, and then the thing/app extorts me. Which is all very closely related to Jaron Lanier’s comment about “our society cannot survive, if…” (And that’s a link to the web site where you can always listen to the podcast, for free, because we control our web site.)
Please—you reading right now—please start paying for things. Choose a podcast player app which is not free. That makes you the customer, and enables them to build a great app. Then they don’t have to strong-arm me. Choose a messaging system, choose a source of information, choose everything(!) by being part of a fair trade with another party.
If you find yourself in a position, where you’re thinking, “this is great and free!” please look around and try to figure out who is actually being taken advantage of… it’s clearly not you, but I assure you, it’s someone else.
The programmer, who needs clarity, who must talk all day to a machine that demands declarations, hunkers down into a low-grade annoyance. It is here that the stereotype of the programmer, sitting in a dim room, growling from behind Coke cans, has its origins. The disorder of the desk, the floor; the yellow Post-it notes everywhere; the whiteboards covered with scrawl: all this is the outward manifestation of the messiness of human thought. The messiness cannot go into the program; it piles up around the programmer.
~ Ellen Ullman
“The messiness cannot go into the program.”
I’ve never thought of it quite that way before. Every once in a great while, you feel the ground move beneath your feet. That sentence moved the ground for me.
I spent an enormous amount of time being a thorn in people’s sides as I clamored to get them to resolve the messiness so I could then manipulate the machines. I tried explaining the machines. I tried explaining the messiness and what I thought might be ways to resolve it. None of that turned out well for the machines, the people or me. Along the way, I realized that dealing with that every day has fundamentally changed how I think. Up until that sentence at the top, I didn’t have a good way to explain my predicament. I only had this fuzzy idea that reality is one thing, computers work this other way, and here I am stuck in the middle.
The messiness cannot go into the computer.
Maaaaybe, I can use that to remind myself that some particular bits of messiness are ok to ignore?
If chatbots are approaching the stage where they can answer diagnostic questions as well or better than human doctors, then it’s possible they might eventually reach or surpass our levels of political sophistication. And it is naïve to suppose that in the future bots will share the limitations of those we see today.
~ Jamie Susskind
This is an interesting read surveying a variety of ways that chatbots might crowd humans out of the very spaces we created.
It struck me that while, yes, chatbots are primitive (compared to “real” AI), they are still having a real affect on our social spaces. Not simply, “it’s noisy in here with all these chatbots,” but rather that our social spaces are in danger of being lost to chatbots.
The no-phones policy illuminated something about smartphone use that’s hard to see when it’s so ubiquitous: our phones drain the life out of a room. They give everyone a push-button way to completely disengage their mind from their surroundings, while their body remains in the room, only minimally aware of itself. Essentially, we all have a risk-free ripcord we can pull at the first pang of boredom or desire for novelty, and of course those pangs occur constantly.
~ David Cain
It has always seemed obvious to me that being focused on a screen, at the expense of the other person, was obviously bad. This used to bother me.
Now, when it happens I check my premises: Am I, right this instant, actually more interesting than the entire world in their hands?