In the paper the rigorous solution of vectorial integro-differential equation of dif-fraction on three dimensional dielectric bodies with complex permittivity was derived by means of Galerkin method. The solution was applied to investigate the diffraction of electromagnetic waves on doubly periodic metallic gratings, to investigate the properties of eigenwaves of nanowaveguides and photonic crystals. The possibility of creation of grating with low coefficients of reflection and transmission at plasmonic resonance was shown. The validity of application of method of approximate boundary conditions for thin dielectric layer considering finite value of metals' permittivity in optical frequency for calculation of structures, which contain nanoscale metallic layers, was proved.