Converting other animation files to Anim8or is complicated not because of the math, but because the other animation formats have different skeleton standards and frames of reference. The first hurdle in converting formats is finding a way to correlate the foreign skeleton to an Anim8or skeleton. I found this to be the most challenging part because of two things: (1) most foreign formats use joints and Anim8or uses bones, and (2) the reference frames of foreign formats differs from Anim8or's. Once you solve the correspondence of foreign joints/bones to Anim8or's bones, you must deduce the the difference in world reference frames and rotate the skeleton into Anim8or's world reference frame.
The next challenge is deducing the reference frame of the foreign bone's orientation. In Anim8or, the orientation of each bone is relative to the parent (BVH and SMD are similar in that respect), but the reference frame in Anim8or is y-axis along the bone, left-handed axis system. What is the foreign format reference frame axis system?
The foreign format's often give Euler angles for orientation, but does it use XYZ, YXZ, ZYX or other order? This must be deduced. Internally, Anim8or uses quaternions, so Euler to quaternion conversions must be done. The math for this conversion is different depending on what the world and local reference frame axis systems are.
These problems, coupled with the brain-twisting non-intuitiveness of quaternions will make your brain explode! Each part of the conversion of a foreign format to Anim8or's is dependent on the previous part. You can get all but one step correct and the result will look as wrong as getting them all wrong.
The math (and code for the math) itself is fairly straightforward. Pretty much everything you need to know math-wise can be found on Martin Baker's site
http://www.euclideanspace.com/maths/geometry/