cooldude234,
Whenever I have tried to determine whether a line segment (like an edge) intersects a polygon, I have done it in two steps:
1) determine if and where the line segment intersects the plane of the polygon
2) determine if that intersection is actually inside the polygon
It is unclear from your code segment what Ray1 and Ray2 actually are, but if they are points that define a line segment, then this is how I would solve this problem:
facenormal=normalize(crossproduct(vert3-vert2,vert2-vert1)) //get normal of polygon plane
rayvector= ray2 - ray1 //get vector of ray line segment
ray_plane_angle=dot(facenormal,rayvector) //actually the arccos of the angle between
if ray_plane_angle != 0 //if zero, the two are parallel
ray_plane_intersection_point = ray2 + ((dot(vert1 - ray2,facenormal) / ray_plane_angle) * rayvector
//above, calculate the intersection point of vector of the line segment and plane
//next add angles of intersection point to two of the vertices at a time.
//if the angles add to 360 deg (or 2*pi rad) then inside, otherwise the intersection point is outside
total_angles= acos(dot(normalize(vert1-ray_plane_intersection_point),normalize(vert2-ray_plane_intersection_point)))
+acos(dot(normalize(vert2-ray_plane_intersection_point),normalize(vert3-ray_plane_intersection_point)))
+acos(dot(normalize(vert3-ray_plane_intersection_point),normalize(vert1-ray_plane_intersection_point)))
if (abs(total_angles - 2*pi) < 0.001) contact_point=ray_plane_intersection_point
If you are actually dealing with a line segment, rather than an endless line, then you need to check whether the intersection point is actually between ray2 and ray1:
if length(ray2-ray_plane_intersection_point)+length(ray1-ray_plane_intersection_point) > length(ray2-ray1)
//intersection point is not between ray2 and ray1
I wrote the code in pseudo code rather than a particular language.
I hope that helps.
NickE