Exploratory and evolutionary data schema design

When I was at the SD West conference last March in Santa Clara, I listened to Scott Ambler's talk about databases and the need to make them evolve in the same manner we write code: by refactoring them. Most programmers know about Agile methodologies and how they apply to our code, but not a whole lot of people know how to apply those methods to database design.

Your data schema will change. This is as certain as death and taxes (although the income tax is really only about 100 years old!) You design your project and your data schema, you implement and deploy your solution and... boom!, you need to change something. When your database design needs to change and you have code in production, or your tables are being accessed by many other applications, you can't just rename columns or delete tables. Scott, in his book Refactoring Databases: Evolutionary Database Design, talks about how to implement changes to your schemas without affecting current production code. 

Here's some more information on the subject.

During his talk at SD West, Scott mentioned that data problems in the United States is costing us about 600 billion dollars! I truly believe that with proper evolutionary database design practices in place, we can bring that value down quite a bit, perhaps making some money in the process. After all, $600 billion seems like quite a reasonable market to me! 

Unfortunately, we currently lack tools to version our database schemas. The only one I've come across so far is from Ruby on Rails and is called Migrations, which is really useful. You basically edit a file for each new revision of your schema that tells Migrations what to do to get your schema to the next version number. Then, should you want to go from version 2 to 8, Migrations will do that for you automatically.

But I need something more powerful. I work with two other developers and we might each have a different version of a database design on our desktops in order to isolate, or sandbox, our development. Once we are ready, we apply our changes to the integration server. This means that four different computers will have different schema versions. If I integrate first, and then Joe integrates, what happens? I added a column A to a table and he removed the table! We have a conflict. What if he added a column B and I added a column C to the same table? We need to merge.

Wait, that sounds a lot like source control! Yep, and Migrations doesn't do that. I want to create a tool that handles database schemas the way source control handles source code, with commits, branches and merges in a way that allows programmers to experiment with their designs. Exploratory and evolutionary data schema design; we'll see where this takes me!

The MVC pattern and me

I started working on a project late last year and I find it absolutely amazing how I've managed to grow as a software developer since then. The techniques I used in the beginning and the latest code I wrote are different, but in a good way, like... less amateurish. I knew I was evolving when I started hearing a little voice in my head that starting telling me that I wasn't coding properly, saying for example that my controllers in my MVC pattern shouldn't be linked in a parent-child relationship.

Huh? I guess... I guess it makes sense! I used to do that a lot; I would create the main application controller and all the other ones would have a pointer to it in case they wanted to tell the software that something had happened or in case they wanted to know who was the currently logged on user. It had never occurred to me that this was wrong until I had views within views within views and controllers within controllers within controllers. I was getting very confused and lost within my own code. Design patterns are supposed to help me, not make things more complicated!

   The MVC design pattern (image taken from wikipedia)

The tiny little programmer that lives under my software developer hat (reference to Ratatouille, here) started instructing me to use events instead of calling parent controller methods. Events! I had completely forgotten about them! This is what web programming does to you, by the way: you forget how to build a desktop application properly. And then last week, I realized I could've simply used a singleton class to hold information such as the currently logged on user instead of calling properties from parent controllers. Now why didn't I think of that before?

(Note to self: read this blog post before you design your next program!)

But there is something else I've come to realize. Whenever I use the MVC (Model-View-Controller) design pattern, I don't use it in quite the same way. Depending on the situation or task at hand, I'll code it differently. Sometimes I'll have each view create their own controller, through which they modify and get information from the models. Some other times, the controllers give the model objects to the views and tell them to display them however they want to display them. The controller doesn't care what the view does and kinda trusts it not to mess with the model object being passed.

I'm unfortunately pretty chaotic when I program and this blog will help me put things in perspective. I need to go back over previous designs and decide what went wrong and what went right and writing about that will force me to think about the issue much more clearly that just giving it a though while I'm falling asleep at night. Believe me, that doesn't work... "I guess I could use this control.... ZZzzzzzz." Yeah... um, no.

My way (or you might wanna choose the highway)

Anyway, so how am I supposed to use this design pattern? I see it everywhere, from Ruby on Rails to Cocoa applications. Even ASP.Net now has its own implementation as a new built in feature. Yet everything I code seems to be more complicated than the usual little MVC diagrams that I find everywhere on the Internet, like the one above. Alright, let's see...

First of all, I will never again access information about a model directly from the controller. Instead, the controller will always give me access to the model object. It is logical after all: the view knows how to display the model and the controller shouldn't get in the way.

Secondly, we need to make sure that the model is completely isolated from the rest of the application. This means that if the information contained in the model object needs to be transformed in any way, the controller will be in charge of doing that. The controller also knows about the models in general and can make decisions concerning passing information back to the models.

This is getting complicated. It looks like models are well isolated, like views require models and controllers, and it looks like controllers require models. Does a controller need to know about a view? No. Views know what kind of controller they need and simply ask for it. The controller, in turn, will give the view the models it wants. Phew! I got it out of my system! Finally something I can base myself on when I design my applications! It's worth repeating:

• Views know what kind of controller they need and simply ask for it.
  
• Controllers give the views the model objects that they want and act as a conduit for information going to and from (sometimes) the models.
  

Now, how can we prevent the views from updating the models? It depends. If I'm the only programmer on a project and I know for sure that there will never be more than one or two developers working on it, and that the project will never be used as a library or incorporated in another project, then I don't really have to prevent the views from updating the models. Just knowing that I shouldn't do it is enough.

In other cases, however, it would be best for the controllers to provide the views with an interface to a model object instead of the model object itself. The interface would only give access to methods that can be used to get data out of the model. Then only the controllers and the models themselves would know how to update a specific model.

What if I ever feel like coding controllers that have child controllers again? How do I prevent myself from doing such an insanity again? This usually happens when I have multiple views embedded within each other, all interacting with the same model object, but in different way. I enjoy the way I can choose different views on the fly for different reasons and each view might require their own controllers because of varying business logic. Well I can simply allow all my views to have access to the same controller, eliminating the need for more than one. This is not always the best scenario, however, since I do not relish the thought of having one giant, massive controller.

Another way is create a helper class that will hold the data that can be needed by multiple controllers. A singleton would be perfect for that sort of thing. And finally, the same model object can be shared by multiple controllers. You believe in sharing, don't you?

Reusable code

In the end, it seems that following what I just preached about will yield code that is better designed, with better defined interactions and that is more reusable. Models will be isolated and will definitely be reusable with the aid of pre-made controllers; it's just a matter of snapping different views on top of them. This way, a desktop application can become a web application or service and vice-versa without to much trouble. This is actually what is going to happen with my project and I know that all my design flaws will jump in at my face when I get started on that.

If nothing else, the time I spend designing a good MVC architecture for my applications will help me structure my ideas much more clearly and that is a good thing. I don't think I'll be perfect anytime soon but its at least something that I'll have gained. And I'm all about improving myself!

This blog and it's meaning

Language seems to be an inadequate tool when I try to describe with words the whys and the reasoning behind the way I feel. I have tried multiple times in the past by writing a journal, then starting up blogs were my multiple dimensional
thoughts had to fit in two dimensions. This is yet another attempt at putting down on paper, digitally speaking, what and who I really am, and to try to make sense of myself.

The title of this blog is almost a happy coincidence: The Lone Star Developer. I do happen to live in a small town in South Texas and I happen to be a software developer by trade. But I also feel lonely in the sense that I don't fit in. I was born and raised in a big city with its sounds, its lights and all sorts of different and interesting people. I used to go to Djambe Jams and go rock climbing regularly. While going through college, learning my trade, there was no thought in my mind that I would one day develop shrink wrap software for the coolest corporation. By the time I got my diploma, I said pew! to corporations and decided I wanted to work for cool companies like Pixar. 

It didn't turn out that way.

Instead, I met and married a wonderful person and after a while decided to move to South Texas, near Corpus Christi, the biggest city in the region. I found a job as an in-house developer, maintaining web pages on our company's Intranet, which helps our employees manufacture pumps and valves for the oil fields. We are a paperless company, which means that every one of our business systems has to be programmed and put on the Intranet by us. There is always something to do, and that's good. 

I guess some people at Pixar build web pages, but I sure didn't envision such a future for myself! Imagine, I was dreaming of a job in a big city in California working for the coolest company in the world and instead I landed in South Texas, working for a pump and valve manufacturer. This, my friends, is a culture shock.

So no, I don't fit in and I want to address the disadvantages and challenges of my situation: no mentors, low amount of cultural events, dealing with old technology and a infinitely small peer community. How can I develop my skills and keep myself challenged enough to keep growing? So far, I have reached the "good enough" stage in my professional life and it is not where I want to stall. I want to program using Bayesian filters instead of if statements but the level of knowledge that I need in order to do that seems exponentially difficult to reach.

From Joel Spolsky, on his popular blog: "A very senior Microsoft developer who moved to Google told me that Google works and thinks at a higher level of abstraction than Microsoft.

Google uses Bayesian filtering the way Microsoft uses the statement," he said. That's true. Google also uses full-text-search-of-the-entire-Internet the way Microsoft uses little tables that list what error IDs correspond to which help text. Look at how Google does spell checking: it's not based on dictionaries; it's based on word usage statistics of the entire Internet, which is why Google knows how to correct my name, misspelled, and Microsoft Word doesn't.

I decided recently to be pro-active and to take my future into my own hands. This blog is my way of reaching out and to talk about things that are important to me. I will explore subjects I'd like to master and post them here for everyone to see and comment on. Hopefully, by connecting to the world and exposing myself this way, the world will connect back to me and make me feel like I'm not in a small town anymore.