Military simulations requiring realistic damage effects such as building structural damage from a direct air strike have typically resorted to look-up tables to determine the damage caused by munitions. More complex responses such as a physics-based calculation of structural collapse are beyond the capabilities of currently fielded simulations. Although look-up tables are a quick way to approximate damage, the main drawbacks are the lack of realism and limited flexibility in handling a wide range of damage effects and munitions. A physics-based approach offers much better realism and also adapts to diverse inputs, yet physics-based effects have historically been relegated to analytical simulations which do not require real-time or near real-time response. In this paper we will discuss lessons learned in developing a physics-based damage server for a real-time urban simulation using HLA. The damage server produces physically accurate penetration holes, craters and rubble for complex 3D building models using a wide variety of munition types. The server can output damage descriptions in a variety of polygonal formats for visualization. In addition, the extreme environments resulting from the weapon detonation are stored and available to compute personnel casualty and equipment damage. The damage server uses DoD accredited physics models.