Both propellants fill very similar mission roles. Hydrazine has been used for 50+ years while LMP-103S was debuted in 2010 and is a serious contender in trades for future missions, especially those with significant size constraints
Chemical Performance as Propellants
Specific impulse is predominantly a function of combustion temperature and molecular weight. To maximize Isp, propellants with high combustion temperatures and low molecular weights need to be used (pressure ratio effects are much less important here). This statement is purely in pursuit of maximum Isp and doesn't not considering other design limitations like material properties and manufacturability. From CEA, the combustion temperature of LMP-103S is about 1570 C and the products' molecular weight is about 19.7. The combustion temperature for hydrazine depends on the degree of ammonia dissociation allowed by the engine design (it is a function of the catalyst type, size, and geometry, the chamber pressure, and the dwell time) but for 1N hydrazine engines is typically in the range of 850-900C. The molecular weight is then about 10.7.
In chemical rocket propulsion, the concept of characteristic velocity (c*) is used to compare different propellant and chamber design combinations. From the same CEA runs, the c* value for LMP-103S is about 1300 m/s and for hydrazine is about 1190 m/s (at the same chamber pressure).
LMP-103S is about 1.23 times more dense than hydrazine at room temperature. This directly increases the amount of propellant that can be carried in a given volume propellant tank, thereby increasing the total system delta-V and more than counters the steeper drop in performance at lower inlet pressures seen in the 1N engines above. The density comparison over a typical temperature range is shown below.
Here it is important to note that hydrazine freezes at 2 degrees C, whereas LMP-103S does not have a distinct freezing point. Since LMP-103S is a solution, there is a temperature (about -10 C) when the solution starts to become saturated and solid ADN crystals will begin to form in the liquid solvents. This is a reversible phase change and the solids return to solution when the temperature is raised. This allows for some interesting options with thermal design like letting the tanks stay cold and only heating the propellant lines and inlet valves.
To illustrate the difference between the two propellants at a system level we can look at the cumulative delta-V delivered to a 99 kg dry mass spacecraft with 11.3 liters of propellant tank volume as an example of a LEO spacecraft. Here the initial tank pressure is kept constant at 18.5 bar(a) as well as the end-of-life tank pressure of about 4.5 bar(a) (ie same blowdown ratio). Because of the large difference in density (see above), the hydrazine system can only load about 8.5 kg of propellant, while the LMP-103S system can fit 10.5 kg. Combined with the performance differences throughout the blowdown curve, this results significantly higher system delta-V from LMP-103S for a given volume.