So I dug down a little bit more into the probabilities of FS3 and found something that I think exemplifies what happens when you design a challenge resolution system using a semi-arbitrary 'this sort of feels good' mechanism.
Before I go on I would like the say that I still feel that the probabilities in FS3 work out pretty well and my comment about how it is designed isn't meant to come off as dismissive. It is just that at the moment I've been grinding through some math and my language centers are a little out of whack so I can't think of a better term to use that would be a clear and concise summary for how the probability system was derived without making it so that some people might perceive me as being dismissive.
The flaw I've spotted is essentially one of 'countering modifiers'. To explain the term in most games if you take an action to get a +1 to attack and then someone takes an action that gives them a +1 to defense then you end up back where you started. They take some form of action that counters your modifier (although usually in doing so they have to take some other form of penalty themselves).
In FS3 this doesn't happen. If you've got 6 dice to attack someone and you attack a person with 6 dice to defend then your chance of hitting is 61.69%. If both people get a 3 die boost (from spending Luck) the odds drop to 59.60%. So even though relative positions remain completely the same (both people are equally skillful, both people catch lucky breaks) the attacker loses 2% on their possibility to hit. While this isn't a whole lot it is still awkward and this is compounded by the fact that it isn't a consistent change. If both people only have 2 dice to begin with the odds shift from 69.60% to 62.76%, nearly a 7% change. Even more inconsistent if someone with 2 dice attacks someone with 6 dice they go from a 23.56% chance to hit to a 29.25% chance to hit, so they in fact gained nearly 6%.
Now again, I'm really not trying to say that FS3 is a horrible system or anything. Honestly, if I didn't think you were looking for data like this and you weren't able to use it constructively (and in this case 'constructive' even includes saying 'you know what? I -like- that this happens') I wouldn't bother. Kind of how we were talking earlier about aspects of realism and different people having different thresholds people will interpret these 'hiccups' differently. I'm just making sure you know they are there.
Now in a tabletop game I would be a lot more accepting of these small aberrations in the probability curves. It's more important to keep things moving along quickly and so you want to avoid complex calculations. That means there's lots of mechanisms that get used to generate probability curves and character progression schemes that are 'good enough'.
We're not playing on a tabletop, however and a lot of the things that would give us a headache the computer can handle very, very easily. As an example I would never try to play a game in which I'm told that the odds of success are 1/(1+2.718281828^((level difference)*-0.183102))*100. I mean what the Hell is that? On the other hand having the computer look at the two characters and say 'Oh! There's a 6 level difference. That means the odds of success is 75%' is something a computer is -really- good at doing. Computers are so good at it that they don't even need to round the result to 75% but can actually have the 'real' probability of 74.99999458% because they don't need to roll 2 10-sided dice to generate a probability.
Even better than that, because you are using mathematics to calculate probabilities you can avoid another pitfall that is left over from tabletop games; the Hell a character goes through to increase a skill they are really good at. You're a Great pilot and you want to increase your skill another point? That's going to take you 3 months during which absolutely nothing changes because you can't roll 2/3 of an 8-sided die. Using math, however, there's not anything wrong with being partway between Great and Expert. The computer has no more difficulty calculating the chance of success between characters of equal level and characters who are 2.1 levels apart.
Now in a lot of settings players might have some difficulty wrapping their heads around some of these aspects and I can completely understand that. One of the nice things about FS3, however, is that a lot of these mechanics are already hidden under the hood and you could change the system with the players barely noticing. Just allow the players to put XP straight into a stat like Pilot. If they put 4 or 5 XP in the skill they get feedback on their sheet that shows that their skill is 'Fair'. As soon as they add the 6th point of XP to Pilot the stat now reads 'Good'. However, a pilot who has put in 5 XP has a small advantage over a pilot who puts in 4 XP. When they shoot the computer looks at their stats, compares them to the stat of the defender, and tells them whether they hit or not. Practically nothing has changed for them (and that's assuming you let them buy partial levels. While I'm all in favor of that it is hardly a requirement. It's an advantage of the system buy hardly mandated).
The biggest problem you would have with such a system is dialing in the initial values. I actually had created a spreadsheet a few years back that did the math for me so I could concentrate on shaping the curve however I wanted.
I've gone ahead and cleaned it up and added some stuff that allow you to sort of 'mock up' tests using attributes that are roughly the same scale as what FS3 is doing. I did make one cheat, however, in the mockup and rather than using the XP chart that FS3 uses I change the calculations for the levels of skills and attributes to a pair of formulas.
I just find it funny to hear a comment like: "Oh, if only someone with more math background worked on it..."
