In the book series it is shown that there are many different species (eg, Nephilim, Warlocks, Fey, etc.) but all of these species were created from the offspring of Angels, Demons or Humans mixing. Seeing as the Nephilim and Fey can reproduce they could be considered a species, whereas Warlocks are infertile and can only be created from Demon and human parents. Angels and Demons could also be possible transformations as neither of them are hybrids and they are know to the DNA (Ichor)
I think that Warlocks could be considered hybrids, and therefore they wouldnt be able to be a transformation, but the Nephilim and Fey are able to reproduce and have the characteristics of a species, even though they were made from two different species and could possible be considered a hybrid.
What do you think?
(Also a Nephilim or Fey transformation in the Non Cannon Transformations would be so cool, lets make it happen)
Here is a link to the shadowhunter fandom where it goes into more detail about the species