Our company over the years, has purchased dozens of applications that generally have SQL Server back ends. My job is to report off those databases (or their OLAP counterparts). Some databases are logically structured, reasonably well documented (with ER diagrams, schemata and so on); more than half are not. The project management involved in the introduction of new applications here NEVER considers the need to actually get the data back out. "The report writers will work that out for themselves" is the attitude. So we waste hours, days, weeks deciphering undocumented databases. Ultimately we're wasting company money. My question is this: Is this situation a failure of our local project management, or is it normal to expect no doco? Is it unthinkable to include a clause in a contract that obliges a vendor to document their database and keep it up to date? If they don't want to document, fine. They can sell their crappy product to someone else. What are your expectations when a vendor wants to sell you their product? I'm starting to think that any vendor who says "Sorry, we don't do schema" has something to hide, and it's not intellectual property. It's more likely to be the lack thereof. No right answer, so the most upvoted after a week will get selected.
I think what @dataminor is getting at, is how widespread is the problem of vendors selling software that appears to tick all the boxes when it comes to data input and day to day use, but fails miserably when it comes to getting data out of the database (because of lack of documentation). I think it's more widespread than it should be, but I don't blame the vendors necessarily. I blame the philosophy of organisations engaging "project managers" who might know the business but don't have a clue about databases, to investigate and roll out new software. I've found that if the BI people are able to "get in on the ground floor" they can get traction on what is reasonable to expect the vendor to deliver. Vendors waiting expectantly for the signature on a contract are far more interested in your welfare than those who have already cashed the cheque.
From what I've seen over the years from vendors starting with Microsoft and ending with the smallest 3rd party imaginable, is that most, if not all, databases are undocumented or inadequately documented. I've spent tons of time figuring out how to query SCOM databases because the structures supplied by Microsoft are completely inadequate. It's not a question of your project managers being bad at their jobs. They're just dealing with the fact that most databases are not documented appropriately. Then you get into databases like the one in Microsoft Dynamics CRM which doesn't have documentation OR even referential integrity. No RI makes it practically impossible to discover which data comes from where. In short, there is no answer except suffering until there is some sort of paradigm switch in how vendors behave. Personally, I'm not holding my breath waiting for that one.