Categorical symmetry and non-invertible anomaly in symmetry-breaking and topological phase transitions