My first rack used Yakima mounts and bars- was not ideal. Yakima makes great stuff but their rack rails are not really intended for this application, they are for carrying moderate loads and mainly on cars and SUVs- not heavy racks and loads on a 4 wheel drive trail. While mine did not break, they would get some serious side to side sway going on any rough road and it would have only been a matter of time. I eventually went to 6 gutter mounts because a heavy rack and spare tire is a lot of weight to keep under control, let alone anything else you put in there. You used to be able to buy Con Ferr gutter mounts, haven't checked in awhile though.
As for material, I think 1"x1" .120 would be max and .090 is even better if the rack is designed well.