One of the first advantages to mention is the simplicity. With only a single cog and chain ring there is much less to go wrong with the drive train making them cheaper to run and easier to maintain. This makes them a popular choice for riding during the winter months when typical British weather means there is lots of mud and muck to contend with. It also means an end to chain suck.
Getting rid of derailleur, shifters and sprockets means you can make a bike that is really light and without the extra sprockets your chain line will run in a perfectly straight line from sprocket to chain ring. When you are in the right gear on the right terrain a single speed is smooth and easy to pedal as well as being mechanically more efficient than a geared bike in the same ratio as no energy is being wasted else where in the drive train.
Single speeding can also do wonders for your pedalling technique, riding style and fitness. With only one gear you will have to conserve energy and ride efficiently to maintain your flow. If you lose speed approaching a climb or brake hard and late you can't use your gears to bail you out and it takes longer to build speed back up.
There are some single speed mountain bikes available and geared bikes that are suitable for single speed conversion. Not all bikes are suitable. It is not possible to create a single speed rear suspension bike as the movement of rear suspension slackens the chain line without a rear derailleur to provide tension and hold the chain in place.
On a single speed specific frame the rear dropouts will be horizontal rather than vertical. A vertical drop out (where the wheel slides upwards when installing) doesn't provide any means of regulating chain tension when there is no reach mech in use. Conversion is possible with a single speed chain-tensioning device.
You'll need a new rear wheel built on a single speed specific hub before you move onto the intricacies of gear selection. With only one gear you need to get it right! It needs to be low enough to manage most of the hills you ride with enough resistance to allow smooth pedalling through single track. If you ride in many different area or want to race you'll need a selection of sprockets.