Plaster of Paris is an odourless powder consisting mostly or entirely of calcium sulfate hemihydrate (CaSO4•1/2H2O) and is generally non-toxic.
Although Plaster of Paris is non-toxic, however, ingestion of a sufficient quantity could lead to mechanical obstruction of the gut, especially the pyloric region.
Where there is a danger is in the dust: Prolonged and repeated exposure to airborne free respirable crystalline silica can result in lung disease (i.e., silicosis)
and/or lung cancer. This is especially true of industrial grade building quality material that may contain crystalline silica. Under normal 'domestic' conditions, you are very unlikely to suffer any harm using it.
This is the MATERIAL SAFETY DATA SHEET for Plaster of Paris: