A new Paradigm for thread organization
Let's talk about what people do. When I'm actually taking part in a conversation, I agree with you. Discussions are natural things that go where they will, and to restrain discussion from going where it will significantly restricts the breadth of discussion that can be had. But when I'm looking for a place to jump in, or when I make a thread about a Talon rework, I'd like discussion there to stay about Talon, or at least if moves to talk about Leblanc, it's in the context of Leblanc as a comparison point for Talon. I didn't make a Talon thread so I could hear more LB qq.
So, what I'd like to see is a system that both allows the discussion to go where it will, and one that keeps a given thread on a single topic.
At first these seems like opposing goals. However, that's under the assumption that thread == discussion.
Take twitter. Twitter does a great job of allowing discussion to happen naturally. There can be back-and-forth of people replying to each other's tweets, or others can jump in, but there's not a huge amount of inherent hierarchy. It's more like a web, where you can jump from topic to topic. The main point here is that each comment in a discussion has the same weight as "an OP" (a concept that doesn't really exist), and that this facilitates natural conversation flow.
Now take Boards. Boards does a good job of focused discussions. They still drift somewhat, but because that's natural for discussions and avoiding it is really hard. In part this is achieved by structuring a discussion around the OP and in part because discussion view (which is the only realistic option at this point) gets messy at really high indent levels, so conversations have to stay short. Though I find on a given topic I've mostly said everything there is to say by the time the nesting limit is reached.
Finally, take reddit. reddit is a middle ground (though way closer to Boards) because it has an (IMO, underused) feature to allow for discussion to wander: permalinks. When you permalink a comment, it shows ONLY that comment and its sub-comments, like so:
https://i.imgur.com/25DFXgP.png
It's almost like that comment forms its own OP -- using permalinks allows you to isolate a discussion by starting from the original off-topic comment, if you wish.
I propose we take the conclusions from the Boards and Twitter systems, and take use a modified version of the reddit system: instead of allowing a permalink to simulate a new OP, instead allow an OP to be embedded into the comments of another post, by posting a bare link in a comment. The comment would be pre-populated with the linked post's title, preview (like current mouseovers), and a Show More link. Clicking Show More would load the full OP, and its comments, into the thread you're currently viewing. (You could even create an OP from within a comment, with a checkbox or something).
Since embedded discussions and their replies are hidden by default except for a brief summary, they don't clog up a thread with off-topic conversation, but it's easy to either load in those comments to continue a discussion naturally, or simply move to the other thread to continue the discussion.
As an aside, it would also be a small step towards grouping of related content, an issue that I'll write more about later and elsewhere.
Here's a reply that would show what this thread would look like if it were embedded in another, except that the "Show More" button would load THIS thread's comments, rather than linking here.