I have been trying to learn a little bit about setting up a nice wiki, but the project has not kept my interest as much as I had hoped. The main thrust of this project has been figuring out how to get the Navbox template working on a wiki at my office, so most of my research has centered on that.
The Navbox template is a fairly slick way of grouping articles in the midst of a fairly flat structure. I would propose that anyone wanting to make nice looking wikis needs to learn about templates or at least understand them enough to use them well. They are used extensively for navigation, which is important for a hierarchy that continues to grow.
A good place to start with getting templates on a wiki is the Navbox template talk page. At the top of that page it has a description of what you will need to get going, including a link to HTML tidy. It even points to a deprecated version in case you can't get the current one to work. I have found that wiki development is like HTML development: easiest way to learn is to learn from examples. Wikis have pages for copying templates from one wiki to another.
Finally, a few notes about installation. Installation is more-or-less straightforward, although you need a number of pieces. It is all freely available for download. Installing on Fedora is fairly easy. You can even get going with yum install mediawiki.