Transformation formulas "transform" the complex or quaternion plane used for calculating the fractal. So they can be used to apply any kind of mapping: For example they can map the fractal onto a sphere. Any kind of distortion can be performed using transformations.
As you can see, transformations do not introduce new "shapes" into or add more details to the fractal: They only can distort the fractal or parts of the fractal in an arbitrary manner.
In order to fully understand the use of transformation formulas you need to know some basics of fractal iteration:
Let's describe fractal type Escapetime: A fractal gets drawn into a window, which is a rectangular area. For fractal type "Escapetime" each "screen-"pixel in that window corresponds to a specific complex number, so that the window basically is a small rectangular area cut out from the complex number plane. In order to calculate the fractal the complex number corresponding to a pixel gets passed to the calculation routine, which finally returns a color for that pixel.
Managing transformations is done in the mapping tab of the parameter window of Escapetime or Quaternion fractals. This tab looks as follows:
Sometimes transformations try to distort the input area but cannot determine a reasonable complex value, for example if they perform a spherical mapping. For these and other special cases the transformation can set the pixel to a solid color and a fixed opacity value. You can adjust these parameters by selecting the solid color and adjusting the opacity value of the solid color right beneath.
If you select a transformation you can see that the lower part of the window will change: It contains the parameters of the transformation as defined by the author of that formula. To read more about formula parameters and the user interface provided to them read User Interface.