This grew out of an answer I posted on a writing forum, but I got carried away and wrote an essay, so I’m posting it here.
The question: How might space-faring nations inside the same star system define their borders?
Borders between different star systems are easy, since stars don’t appreciably move relative to one another over the kind of time scales that nations tend to exist for. Those are the sorts of borders one most often finds talked about in space opera, where empires claim multiple star systems each. You can easily define borders in terms of distance: anything within X distance of the star belongs to that empire.
Borders between planets in the same star system, though, are where things get tricky. Suppose e.g. that Earth and Mars need to establish their borders in the aftermath of the Martian War of Independence. The two planets are sometimes on the same side of the sun and fairly close to one another; at other times they’re on opposite sides of the sun and a long way apart. You can’t take a map of the solar system and draw lines on it because the Earth and Mars are moving.
The easiest option would be to say that the nations’ territories are the space immediately around each planet, and everything outside of that is a no-man’s-land. (That’s what I do in Belt Three, which involves asteroid-based city states: the cities move relative to one another but you’re not in a city’s territory until you’re quite close.) That might not satisfy our Terran and Martian imperial rulers who want to divide the whole solar system between them, though.
If the nations are planets in more-or-less circular orbits, one option would be to define borders in terms of distance from the sun. Everything between distance X and distance Y from the sun belongs to Earth; everything between distance Y and distance Z belongs to Mars. Each planet’s territory is a big ring, within which the planet orbits. There would probably have to be special rules to allow ships from one nation to pass through the other’s territory on the way to somewhere else.
(Actually, the only work I can think of that defines borders in that way is the intro video to the game Pax Imperia: Eminent Domain. The solar system is divided between the ‘Inner Worlds’ and ‘Outer Worlds’ nations, so an Inner Worlds ship travelling through the asteroid belt to Pluto has to dodge Outer Worlds patrols. This system of borders is only used in the intro video: the game itself has each planet owned by a nation but the space between the planets is no-man’s-land.)
A cleverer option (and this is where it gets interesting) would be to think of borders not as fixed areas on a map, but as rules for determining whether an object is in a particular territory. Those rules could take into account velocity as well as position. In space, everything is moving, so how it’s moving is as important as where it is. A ship on the other side of the solar system whose orbit will intercept that of Mars is of great interest to the Martian government; a ship that will pass fairly close to Mars but whose orbit ends up intercepting Earth’s is of less interest. The Martian border “rule” could therefore be something like: a ship is considered to be in Mars’s territory if, based on its current position and velocity, it will come within X distance of Mars within Y days.
You could make the rule more advanced by taking into account a ship’s potential to change its course: a ship is considered to be in Mars’s territory if it could change its course such that it would come within X distance of Mars within Y days. That relies on the nations knowing at least the approximate capabilities of one another’s ships, and also might lead to awkwardness when the planets are positioned such that the best trade route between to of Earth’s outposts happens to be close to the course that would lead to Mars.
How the borders end up being defined would be up to who, in-universe, sets the borders, not necessarily what the most elegant solution would actually be. Perhaps the Space Council decides that everything within these distances from the sun belongs to Earth, and everything within those distances belongs to Mars–simple rules devised by politicians, to the chagrin of space navigators who know the complexities of orbital mechanics! Or perhaps the newly independent Mars issues a system-wide warning: every craft whose course will take it close to Mars is in our territory; all such ships must change course or be destroyed. If they’ve got the firepower to back that up, that’s what the border is.
At the end of the day a border only means as much as the nation’s ability to police it. Perhaps the de facto rule, therefore, would be: a body is in a nation’s territory if that nation’s Border Defence Force ships could reach it more quickly than those of any other nation.