A caged compound CeOs2Al10, crystallizing in the orthorhombic YbFe2Al10-type structure, undergoes a mysterious phase transition at T_0=29 K. We report the results of electron diffraction, magnetization, and magnetoresistance for single crystals. Superlattice reflections characterized by a wave vector q = (0, -2/3, 2/3) observed at 15 K indicate a structural modification in the ordered state. Activation-type behavior of the electrical resistivity along the three principal axes below 50 K suggests gap opening in the conduction band. The magnetic susceptibility chi = M/B is highly anisotropic, chi_a>chi_c>chi_b, all of which sharply decrease on cooling below T_0. Furthermore, a metamagnetic anomaly in the magnetization and a step in the magnetoresistance occur at B=6-8 T only when the magnetic field is applied parallel to the orthorhombic c axis. However, T_0 hardly changes under magnetic fields up to 14 T, irrespective of the field direction. By using these data, we present a B-T phase diagram and discuss several scenarios for the mysterious transition.