There are two different rosters that are significant for MLB teams. The first is the 40-man (major league) roster, in force year-round, which consists of players who are under reserve to the team. The second is the 25-man (active) roster, which exists only during the season and consists of players that may be used in games.
The 15 players on the major league but not the active roster play in the minor leagues and are said to be serving an optional assignment. If a player on the 40-man cannot be sent on optional assignment, and the team does not have room on the 25-man (or doesn't want him on it), then the player can be designated for assignment (DFA). The team then has 10 days to trade the player, release him, or expose him to outright waivers where other teams can claim and add him to their 25- and 40-man rosters. If the player clears waivers, under certain circumstances (see below), he can be outrighted off the 40-man roster with the team still retaining his rights.
Players who can be sent on optional assignment, or can be outrighted, have value in terms of providing flexibility to their teams, especially in the event of injuries or player struggles. Free agents who sign a minor league contract with a team are not on the 40-man roster.
Optional Assignments
In general, teams are granted three option years on a specific player. While it is common to refer to players as having finite number of "options", that get used up whenever the player is sent on optional assignment, this is a misnomer as optional assignments are for entire seasons. Players can be optioned and recalled multiple times within a season while only using one option year. For example, Mike McCoy was optioned to Las Vegas and recalled six times in the 2011 season, but that only counted as one option. If a player spends less than 20 days on optional assignment, then the player is not deemed to have had an option year used (but the player gets service credit and MLB salary for the time spent in the minors).
A player is considered out of options in the season following the use of his last option year, and he must stay on the 25-man roster or be removed from the 40-man roster. In addition, players with more than five years major league service time cannot be optioned without the consent of the player, which is known as Veteran's Consent. In practice, this is unlikely to be granted, and it is safe to assume that a team cannot option a player with five years of MLB service.
Fourth Option Year
In certain uncommon circumstances, a player may qualify for a fourth option year. This occurs when a player's three option years have been exhausted, but he does not have five professional seasons. The crux of the matter is in how "professional season" is defined for option purposes. A season only counts as a professional season if the player spends 90 or more days on an active roster (or 60 days or more days on the active roster and 30 or more on the DL). Rookie and short season leagues do not last for 90 days, so players assigned to these levels cannot qualify (unless first assigned to a full season league).
Most players who qualify for a fourth option year are international free agents signed at age 16 or 17. If they play a couple years in rookie ball, by the time they are added to the 40-man roster four years after signing to be protected from the Rule 5 draft, they may only have one year in a full season league and only one qualifying season. Assuming optional assignments the next three years, the player will only have four qualifying seasons. In general, if a player has two or more full seasons when added to the 40-man, it is very unlikely he will qualify for a fourth option; if he has less than two, there is a reasonable chance he will qualify though it is not guaranteed.
Optional Waivers
A final complicating factor, with respect to optional assignments, are optional waivers. When a team seeks to option a player who is more than three years removed from the first date he reported to a major league team, it must first secure optional waivers on the player. Theoretically, this means that any team could claim the player being put through optional waivers. The waivers are revocable, therefore the player can be pulled back if a claim were made (though he would be blocked from being optioned). In practice, these optional waivers are ignored by other teams and do not represent an obstacle to sending players on optional assignment. Minor Leaguer wrote a more detailed account of this rule and its implications.
For the purposes of calculating optional waivers above, the player's debut date and not his reporting date was used. In the vast majority of cases, there will be either no difference or an immaterial one. The exception is for players who are called up for the first time and report, but do not debut for a significant period of time. An example is Brian "Moonlight" Jeroloman, who has yet to debut in a MLB game, despite reporting to the Blue Jays in late August 2011 and spending all of September on the active roster.
Outright Assignments
As noted above, when a player is designated for assignment and clears waivers, he is removed from the 40-man. At that point a few things can happen, assuming the team does not release him. If a player has less than three years of service time and has not been previously outrighted, then his team can assign him outright to a minor league team while retaining his rights. If a player has more than three years service, or has been previously outrighted, then he can reject the outright assignment and chose to immediately become a free agent. The player can also chose to accept the outright assignment, but retains the right to elect free agency at the end of the season unless returned to the 40-man roster. It is common for players to do this, if outrighted in-season, but the player is in control and the team cannot assign him unilaterally. An example of this is Scott Richmond, who was sent outright in 2011, and then again in 2012. He accepted the 2012 outright assignment but then elected free agency at the end of the season, which he wouldn't have otherwise been entitled to do.