On this day

October 31st

Selecting quotations using Chat-GPT

October 31

With great power, comes great responsibility. Large language models (LLM) like Chat-GPT are powerful tools. How do I use it responsibly?

I want to find and present great quotations from guests on my podcast episodes. What happens when I try to get Chat-GPT to do it? Following is a really deep dive into exactly what happens, along with my best efforts to work with this power tool in a way which accurately represents what my guests say, while showing them in the best possible lighting.

The transcripts the LLM works from

For this to make any sense, you need to know that I start from a machine-generated transcript. I get them from the recording service, or from another service where I upload audio (for older episodes before machine-generated-from-the-service was available.)

Below is a screenshot.

  • In the left-margin are line numbers. Line 107 and 123 are too long to fit, so my text editor has visually, “soft” wrapped them for ease of reading.
  • The files have time codes in them (the format does vary somewhat too). They have silly amounts of precision: 16:51.57 is 16 minutes and 51.57 seconds. Hours appear in the front as another number with another colon.
  • The LLM understands which person is the guest, because it understands who introduces the show, and introduces the other person.

I break the very long transcript text file into chunks, because there’s a maximum amount of text you can paste into the LLM interface. The screenshot above is from the 2nd or three chunks from my conversation with Martin. The whole chunk is 144 lines and about 8,000 characters.

Imagine having to read through the whole transcript to find the best part to quote. That’s very hard for me to do (nevermind I don’t have the time to do it), but the LLM can do it in a blink. LLMs are tireless and patient.

The prompt

There’s much discussion about “prompt engineering.” It’s an art. The best clues I can give you are: Explain it to a 5-year-old. And, the 5-year-old does not get confused by ordered lists, even if you nest them.

I give the LLM all the transcript chunks. Then I give it this prompt:

Select 5 direct quotations (not from Craig) from the conversation. I prefer longer quotations which include more context. For each quotation you select, do three things: First, show me your selected quotation. Do not rewrite the quotations. You may remove verbal tics such as “ah”, “yeah” and “um”. You must leave the speaker’s false-starts and restarts in place, ending those with an em-dash and a space. Second, show the nearest time from before the selected quotation. Show that time exactly as it appears in the original transcript. Third, show the exact original transcript from which you selected the quotation. For context, show several lines of the original, unedited transcript before and where you selected the quotation.

Oh! Reviewing this post, I even found a problem in the prompt above. Can you see it? Below, you’ll see a complaint about the LLM response. Now I think it’s not an LLM error, but an error in my prompt above. :slight_smile:

(This entire post about quotations is actually just part 2 of a much larger prompt which starts with, “Perform the following 5 tasks. Include a numbered heading before your response for each of these tasks:”)

It spits the result out in one long stream of text. I’ll break it apart…

Quote 1

There are many things that I have to consider as I look at that:

Is it really a good quote, based on what I remember of that conversation? Meh, it’s okay. But that’s why I as it for 5 selections.

I don’t love that it refuses to give me context after the part where it selected the quote. I’ve tried, but after hours of work, I’m done prompt engineering and want to start this post. (As I mentioned above, I think this is because there’s an error, above, in my prompt.)

In this particular conversation, Martin talks a lot about “parkour”—thus “park order” (and many other variations I see a lot)—is just an error in the raw transcript. Ignore that for today.

Most importantly: Is it hallucinating?? Well, it’s easy to use that timestamp. 1 minute 22 seconds is definitely in the 1st chunk… a moment of scrolling…

Here’s the actual, original chunk I uploaded, and the LLM’s output side by side…

Okay, that’s sane. If I was going to pick this quotation, I’d have to work some form of my question into that quote, so his quote has some context…

Moving on, I’m just going to give you the screenshots for each.

Quote 2

Quote 3

Quote 4

Quote 5

Closing thoughts

I use LLMs to write my show notes. Getting a quote or two is just one part of that.

“Write show notes” is not “the work only I can do” (as Seth would say.) And, I simply do not have the time to do show notes from scratch.

Yes, I’ve spent hours today on prompt engineering, but I have 319 more podcast episodes from 2022 and earlier (!) that I want to have show notes for. Those episodes would be better with show notes. A few hours spent here, enable me to copy-and-paste… wait a few minutes (the LLM is not instantaneous) and I have a really good starting point for show notes.

ɕ

Watching the light go on

October 31, 2022

This was Seth Godin’s second appearance on Brian Koppelman’s show The Moment and captures a slightly less polished version of some of his usual messaging. It’s easier to see his path to really polished books like This is Marketing and The Practice.

I think we talked last time about watching the light go on for people. That is my mission. That is what I’ve been doing since I was 18 years old, that when I’m doing my best work, what I’m doing is engaging with someone and helping them see the world differently and let them do work that they care about. And sometimes you can do that with a book. And the magic of books used to be that millions of people would go to a store waiting for a light to be turned on. So it’s scaled. And it was a combination of solitary endeavor, but a community one as well. I do it in person with people I care about. But that doesn’t scale. So the question is, is there a way in this post book world to be able to create environments where people change.

~ Seth Godin ~3:17 in the July 7, 2015 episode of The Moment with Brian Koppelman

slip:2se2.

Depending on where you normally listen, this episode might be hard to find. It’s more than 300 episodes back (7 years) in Koppelman’s The Moment podcast. I couldn’t find it on the main web site for the show. (I originally found it because I have a way of manually, human-reading RSS feeds, beginning from the first entry in a “drip” system.)

The link below is to a service from Overcast (the podcast player app) which will let you play it in your favorite web browser.

https://overcast.fm/+HS-ViyTws

ɕ

Life is ample

October 31, 2020

But life is ample, of course, for men who keep themselves detached from involvement. None of their time is transferred to others, none is frittered away in this diretion and that, none is committed to Fortune, none perishes of neglect, none is squandered in lavishness, none is idle: All of it, so to speak, produces income. A very little is therefore amply sufficient, and hence, when his last day comes, the philosopher goes to meet his death with a steady step.

~ Seneca

slip:4a412.

Territorial, not hierarchical

October 31, 2019

It has to be territorial, not hierarchical. Meaning real success comes from the inside out, not the outside in. Real success is the process, not the product. It’s what we would do if there were nobody else in the world, yet it depends in the end on everyone else in the world. The essential expression of our art is that of a gift. We draw from that which is most ourselves–and then offer that essence to our fellow travelers on this planet, to help them, entertain them, show them they’re not alone … asking nothing in return (well, maybe enough to pay the rent, we hope.)

~ Steven Pressfield

slip:4a489.

This is a classic that has nothing at all to do specifically with writing. If you are involved in creating anything, you will find this is a great article with a long list of elements of success. (“Elements of Success” is his title.) After you read this, you should run—not walk—and get a copy of his book War of Art; you can thank me later.

Anyway.

The paragraph above really spoke to me. The idea that “success is the process” is something I keep losing hold of. Like a swimmer who keeps forgetting that kicking effectively and continuously is a necessary part of staying afloat and getting there, I keep forgetting that the process is success and I begin to struggle.

ɕ

The secret to connecting with people

October 31, 2018

Let the other person have the privilege of being the first one to be understood. The biggest distraction to understanding someone else is self-importance. Needing to say something means you have to be thinking about it, and thinking about it means you have very little mental capacity left for empathy. Free up yours, and it will free up theirs.

~ David Cain from, The Secret to Connecting With People

slip:4urate8.

Empathy is the most useful ability I have ever developed. Sure, I first had to develop my abilities of self-awareness and self-assessment. But at that point, the need for empathy and compassion became plain as day.

ɕ

Lawsplainer: The Ninth Circuit and Compelled Speech

October 31, 2016

When a court applies scrutiny, it’s holding the government’s justification for a challenged law to a standard. How tough the standard is depends on the nature of the law and how the plaintiff says it’s defective. In some situations, courts apply strict scrutiny — for instance, laws that punish speech based on its content generally trigger strict scrutiny. If a court applies strict scrutiny, the government must show that the law in question serves a compelling government interest and is narrowly tailored to achieve that interest. Practically speaking, applying strict scrutiny almost always means that the court will strike down the law.

~ Ken White from, «https://popehat.com/2016/10/17/lawsplainer-the-ninth-circuit-and-compelled-speech-about-abortion/»

Don’t knee-jerk-react to the topic of the case (it’s about abortion). Instead, read this short piece as an explanation of one facet of the interation between our federal courts and our state legislatures. One of the beautiful aspects of our Republic is the way our founders envisioned a system to combine the states under a federal umbrella. I note particularly the concept of the federal courts NOT being able to just randomly intervene in the states’ affairs.

ɕ

Territorial

October 31, 2013

It has to be territorial, not hierarchical. Meaning real success comes from the inside out, not the outside in. Real success is the process, not the product. It’s what we would do if there were nobody else in the world, yet it depends in the end on everyone else in the world. The essential expression of our art is that of a gift. We draw from that which is most ourselves–and then offer that essence to our fellow travelers on this planet, to help them, entertain them, show them they’re not alone … asking nothing in return (well, maybe enough to pay the rent, we hope.)

~ Steven Pressfield

slip:4a489.

Mercy

October 31, 2013

Mercy is refraining from ridiculing or abusing the young men who lost 91-0. Mercy is complimenting them for their effort. Mercy is treating them like athletes and competitors after such a loss. Mercy is applauding when they return to the line time after time against hopeless odds, and when they return to practice again the next day. Mercy might be sitting down with them and telling them about times you lost badly and how you felt and how you got up again. Mercy is pointing out to other kids the character it takes to keep trying under such circumstances, and challenging them to have character like that. Mercy is teaching kids how to deal with the adversity they will certainly face in their life.

It is not merciful to teach them they have a right not to lose badly.

~ Ken White from, «http://www.popehat.com/2013/10/23/what-does-mercy-look-like/»

ɕ