Unfortunately, I gone silent after 3 exciting posts on Enterprise Infrastructure Series, there were many events which caused this silence..
This post will cover most parts in theory because I could not get time to prepare sample examples, but most probably thought process should help you understand what it takes to monitor existing infrastructure and automate provisioning for the same...
Equipped with Chef's cook books and nagios's simple configuration templates, you can almost have automated provisioning and plugin automated monitoring right at your finger tips. Nagios is one of the most simple tools to setup. Nagios has inheritable configuration. Where you define simple base configuration such as just monitoring for ping or ssh, and later on you can create complex configuration which can be as complex as monitoring web server and java jmx processes and many more things.
For provisioning, we extensively used chef databags - so for creating a new configuration for a virtual machine, it can be as simple as entering virtual machine details as databag parameters in json format and which in turn can be processed by chef receipe on virtual machine host and can create virtual machine from base image - then automatically joining that to chef server and then applying appropriate roles.
Since all this will require me post internal, not generalized source code, I thought I would skip that part, but would be very happy if you want me to help you out with setting up these things.
Also this means that I will have to create an elaborate example on this configuration - I wish I could do that but since long I have moved out of my previous fabulous organization thoughtworks ( http://thoughtworks.com )
But I think, Iwill have time in future and will start putting up some examples around this as well.