Overengineering an Empire

Published By: Jason Kolb on January 29, 2007 - 5:10pm
Original Blog Entry Located Here
Filed In: IT Management

I used to work with a great guy named Ryan Clenney who was a wiz with Excel, but he worked in operations instead of IT. A project would come up that would require some sort of data integration from multiple data sources. IT would talk about creating a data warehouse and begin mapping fields one by one, and Ryan would run off and whip up a macro-driven Excel spreadsheet. By the time IT was done getting the data mapping and the project scope ready, Ryan would show us an Excel spreadsheet that accomplished the same thing we were planning to. He had done the entire ETL (extract, transform, and load) in an Excel macro and produced a working prototype before we in IT had even gotten past planning.

(By the way Ryan, if you see this please drop me a line, I lost your contact info!)

IT has a tendency to overengineer everything. Primarily because, if it goes down it’s our asses on the line (both figuratively and in the neverending-conference-call-till-2am sense). I think everyone in the back of their minds also wants to build something that will change the world, something monumental that other geeks will marvel at. So we get grandiose visions of service oriented architectures and enterprise mashups dancing in our heads and we lose sight of the original business goal. By the time it’s done we’ve built a Ferrari engine to power a matchbox car.

But people like Ryan just need a few key components and a little guidance from IT and they can build one-off solutions like a champ. The solutions may not work TOGETHER, they probably won’t scale very well, and problems will certainly come up if you try to run your entire business on them, but they work like a charm to fix individual problems. It's like IT builds a tower to pick an apple, these guys bring a ladder. As long as IT actually helps these people, they can solve a LOT of problems that normally get thrown at IT.

Unfortunately, IT usually does not help. We gave Ryan access to ODBC data sources and systems, and helped him out whenever he ran into a wall. However, that’s not how it usually works. IT is very territorial and protective, and does not like other people intruding on their territory. In the age of SOX, part of this paranoia is because of valid security concerns, but much if it is simply ego-driven and counterproductive. If IT denies people like Ryan their datasources and access to the systems they need, then everything is forced to run at IT’s pace, and IT dictates the pace of productivity.

I’ve seen IT personnel become downright confrontational when it comes to providing access to systems. There are literally arguments about who gets access, what they have access to, and for how long. Projects are stopped in their tracks for weeks while it is hashed out. IT believes that it knows how to run the business just as well as the executive departments do, and feel that it is their job to educate them on the proper way to do things.

I was reading a response from James Dellow to Rod Boothby's response to my response to a post he wrote :P That's when I think I finally got a clear understanding of the whole users-as-programmers thing. There ARE certain times when you need to whip up a quick one-off solution to solve an urgent business problem. Ideally, the systems that you build your infrastructure on are robust enough to either address the problem quickly or SUPPORT A QUICK ONE-OFF SOLUTION.

Now, I am not technically familiar with products like Coghead and Teqlo, but I would have to assume that they provide hooks to Web and REST services as external data sources. IF that is true, and IF they provide a front-end to layer additional data on top of that and create an integrated data layer on top of all of those data sources, customizable by Macros, then I now definitely see their market. It is the Ryans of the world, the guys who want to quickly solve an individual business problem without waiting for IT. When IT catches up with their engineered and robust solution that’s great, but we can’t realistically expect the business to stop in its tracks while we overengineer something for an immediate need. There is no IT in TEAM, or something like that.

I think I will have to take a deeper look at these apps to see if they do meet that criteria. If so I can certainly see some potential uses for them.


Sponsored White Paper
Recent Blog Entries