Web Frameworks – Open Source or Roll Your Own?

A while back I wrote about my experience coding and maintaining an in-house web framework at a previous job. It was a full-stack web framework. We had libraries for front-end Javascript up to server-side database connections. And the entire stack was tightly coupled. But while the framework was serviceable, it was almost always behind modern trends in web development. I always felt like we were playing catch-up. And as a developer I wanted to widen my horizons and try out more things. So more than once I had discussions with higher management about using open source web frameworks in some projects.
 
Unfortunately, our in-house web framework already had a long history and most of the devs in our company were used to it. The company had tried using a different Java-based framework stack before. It was back in the days when things like Struts, Spring, Hibernate, etc were beginning to ramp up. It kind of ended in disaster – that project ended up taking a lot of effort, had a lot of technical problems, and so on. I believe this gave the company leadership the impression that investing in other frameworks are not worth the risk and effort. It’s a form of Not Invented Here syndrome.
 
I admit there are some advantages to having your own in-house web framework. After all, all the popular web frameworks today started out in-house in some company and later they decided to release as open source. And many companies do fine using in-house frameworks. Using an in-house framework means full control over the behavior. And you can tailor the functionality and coding style to your internal processes. In fact it can be a good value add to your company if your in-house framework did something better or unique compared to open source ones.
 
But there are also significant advantages to using open source frameworks. Maybe I should have used some of these during discussions when I was there:
 
  • With an open source framework, we wouldn’t have to maintain the core functionality of the framework ourselves. We wouldn’t have to maintain the full stack ourselves. We would only need to maintain any customisations and extensions we write for our own needs. For our in-house web framework, every so often we’d spend some effort to come out with a new version with incremental improvements. With an open source framework, we could instead redirect that effort to higher-value work.
  • There’s more learning material available online, and they don’t have to be maintained in-house.
  • There’s a wider base of experience to draw from. For our in-house framework if you encountered a problem requiring deep framework knowledge, there were only a handful of high-level experts in the company. For an open-source web framework, that expertise is widely available on the internet through sites such as Stack Overflow.
  • Having knowledge/experience in established frameworks means your company can get more contracts. You can take on projects that use those frameworks. You don’t have to spend proposal space trying to convince clients that your in-house framework is great. With open source frameworks you can reuse marketing copy by someone else!
  • Working on well-known, established frameworks is better for your developers career-wise. It gives them more knowledge that could be transferable to other jobs. While this isn’t a benefit to the company per se, it will make the company more attractive to developers. It even allows the flexibility of hiring developers experience with that framework.

There are some disadvantages too of course:

  • As mentioned above, there is time and effort involved in learning a new open source framework. This effort is mostly for the experienced developers – for new hires they will need to learn something regardless
  • Having an in-house team developing your own framework means you have a core set of developers experienced in the full stack. Reliance on open source frameworks means most of your developers won’t be familiar with the low level details.

In the end, there’s no guarantee that using an open source framework will be painless or be better than developing one in-house. So I can understand the decision to stick with what you know. But for me as a developer, I feel that it’s more rewarding to be exposed to different frameworks.

In fact I wrote this post because recently someone asked me what my “go-to web framework” was and I said I didn’t have one. I’d rather be flexible enough to learn and work with any existing framework. In our industry where change happens quickly and can catch you by surprise, I think that flexibility is a much more valuable asset to have.

MarkLogic NoSQL

I recently attended a few training sessions for MarkLogic held at an office in a nearby business center. Now, I’ll forgive you for not knowing what MarkLogic is, as even I hadn’t heard of it before six months ago. MarkLogic is (apparently) the leading Enterprise NoSQL provider.
 
NoSQL is big and sexy right now because of the supposed advantages in handling big data, and large web companies like Google and Facebook use a lot of NoSQL in the backend. Most of the popular/well-known NoSQL solutions are open-source/free ones: MongoDB, Cassandra, CouchDB, and so on. But these aren’t actually very popular on the enterprise side, hence “Enterprise NoSQL” isn’t a very common phrase.
 
One of the reasons NoSQL isn’t currently very popular for enterprise projects is that the popular open-source solutions such as MongoDB don’t guarantee ACID transactions. In fact, MongoDB has the concept of “eventual consistency” for their distributed servers, which implies that they don’t have real-time consistency.
 
MarkLogic does guarantee ACID transactions, along with government-grade security. Both of these are things that enterprise clients love. So that’s the market they’re in. Their highest-profile project to-date was the backend datastore for Healthcare.gov (also known as Obamacare/ACA). That project involved consolidating data with different structures from multiple sources and needed to scale up to a ridiculously high capacity, so it seemed tailor-fit for a big data level NoSQL solution.
 
During the second-to-last day of the training, Jason Hunter, CTO of Marklogic for Asia-Pacific, dropped in to answer some questions. He talked a bit about how MarkLogic started and how they got him on board, and a bit about their competition (some disparagement towards MongoDB and Oracle) and about Healthcare.gov.
 
He also had a really good sales pitch about why NoSQL is just a better approach compared to RDBMS. (Although it would have been more fun if he had given this talk with like an Oracle salesperson there to debate with him.)
 
One of his points was that RDBMS restrictions such as limits on column size are outdated. They were sensible in the old days when disk space was at a premium, but these days you don’t need to limit how many characters you store in a last name field. MarkLogic stores all records as documents with no file size limit (AFAIK) to avoid such an issue. From experience developing web forms, there’s the occasional client or system user who doesn’t understand why we need to have character limits on fields. We also had this system where they wanted to be able to type unlimited-length rich text content (we stored it as HTML in the backend) in memoboxes. These were fields where they had to write stuff like notes and assessments and most of their data was pages of text. I feel like that sort of thing would have been a great use for NoSQL.
 
Another point he raised was that RDBMS systems were very bad at allowing the user to search like three columns at random. You needed to know in advance which three columns to index. For document stores like MarkLogic, typically the entire document is indexed so that problem is avoided. Of course, that means that functions like “sorting by last name” need a bit more setup. You need to build a range index in MarkLogic to sort by a specific field. So it’s kind of a trade-off either way.
 
Now to be fair, Oracle does support indexed full-text search over any number of columns via Oracle Text. But it’s not the default behavior and definitely not straightforward. I used to work with Oracle Text a lot in some of my older projects, and the amount of time it took to index any nontrivial amount of data often gave us a headache.
 
I should do a test sometime to determine how well MarkLogic’s indexing performs. The story goes that MarkLogic started out as a document search application before they changed gears to become a database. They even approached VC’s with the intent of competing with Google. They’ve had a lot of time to get good at this, so I have high expectations.
 
The MarkLogic server itself is an interesting piece of engineering. It’s basically a document store, a search engine and an application server all rolled into one package. Upon installation you get some administrative web applications for configuration purposes. The admin interface seems robust and thorough. Contrast that with Oracle where you often find yourself needing to tinker around with configuration files and such
 
You can run web applications on the MarkLogic server itself. The supported languages are XQuery and server-side JavaScript. Odd choices I know. I suspect due to historical reasons they started out with XQuery, but the SJS side has the same capabilities (or so we’re told). If you’re not a fan of either option, you can also expose the server’s functionality via a REST interface. They also provide existing Java and Node.JS APIs on top of that REST interface. All of this means you can deploy any kind of webapp in front of MarkLogic server.
 
The world is moving towards bigger data stores, so it’s not unreasonable to think that NoSQL is on the way up and will be big players in the future. So I think the training was worth it (even if I did have to stay in Ortigas for a while). It’s early still. MarkLogic might still turn out to be as painful to work with as Oracle was. But at the very least it’s interesting to try a different approach to enterprise data storage. Looking forward to see what kind of applications we can build with this tech.

Google Code Jam and Competitive Programming

Back in 2004, I signed up for the Google Code Jam for the first time. Unfortunately I didn’t make it past the qualifying round.

I was a bit luckier in 2008 and 2010, making it to round 2 both times. In fact in 2008 as I recall I was one of only two participants from the Philippines who made it to round 2, which allowed me to jokingly brag about being the #2 programmer in the country.

I registered again a few times after that, and tried again last year (2016), where I only made it to round 1.

I’m no stranger to programming competitions, although the GCJ was the only one I’ve ever really participated in. Back in college, one of my friends usually joined such competitions, and I had a good time going over some of the problems with him.

Programming competitions don’t usually reflect the kind of programming done in most real-world jobs (although it might be interesting to use competitive programming problems in whiteboard interviews and see what happens). They focus on pushing the boundaries of the programmers talents in problem solving. You also need to be fairly familiar with algorithmic approaches to searching, sorting, and so on. Having a decent background in math also helps a lot.

The Google Code Jam in particular focuses on scaling up your code to handle very large amounts of data – well past the integer limit sometimes. Each question usually has a small and a large input. For the large inputs, you only have a short amount of time (I think it was 5 minutes?) to submit the output. Presumably you wrote and tested your code before starting the timer, so the short time is really only for your code’s execution. You can afford to be inefficient or maybe even brute force your way through the small input, but if you want to solve the large inputs on time you have to be able to optimize your approach.

When I first signed up, I dreamed of winning the entire thing (and of course being able to brag about being the best programmer in the world). But realistically, a lot of the people joining these competitions are “pros” more or less, so the chance of a “casual” like me winning are not that high. They are “pros” not in the sense that they do it for a living, but more of they do a lot of these competitions and regularly practice for them. Websites like TopCoder provide such competitions regularly for those who are interested. Also, I feel like I am very old and rusty, having worked too long in web development stuff where my focus is usually on a different kind of problem.

Still, I think it’s a good idea for programmers of all skill levels to give it a try, just to see where you are at. Who knows, you might even make it all the way! The next Google Code Jam starts soon – the qualification round is on Apr 7. I signed up, although I’m unsure if I will have the time this year even if I advance, due to scheduling problems. But a man can still dream, right? Who knows, maybe I’ll see some of you there too in the final round!

Fake Devs and Whiteboard Interviews

Recently, a developer needed to undergo a tech interview at US immigration:

This may surprise some people I’ve worked with, but I didn’t have formal computer science training in school. I’m not actually a computer science major. Yet I’ve worked as a software developer for more than a decade now. Literally zero times have I needed to write a sorting function or balance a BST.

I have a rudimentary understanding of some sorting algorithms (mostly just bubble sort and selection sort), and I have some idea of how to balance a BST. And if I needed to implement any of those I could probably whip something up after a few tries. (Something buggy and/or inefficient probably.) If I had to interview at a company that used one of these whiteboard interviews, there’s a good chance I wouldn’t pass.

For maybe 90% (or even as high as 99%) of modern-day programming work, it simply isn’t needed to know these things by heart. Those skills are more necessary if you’re doing low level things like writing an OS or a graphics driver or such. For everyday web development or Java or Rails or whatever, it’s an indulgence. And even for those instances that you need them, well, the internet is right there.

I’m not a fan of the idea that not knowing data structures and algorithms means you aren’t a real software developer. I have a couple of developer friends who jokingly refer to themselves as “fake devs”. It’s supposed to mean someone who doesn’t have any formal computer science background and has at best a passing familiarity with data structures, algorithms, and other computer science concepts. As one of them stated “I wouldn’t know how to make a map without using a map.” (I thought about writing a post about that, but it’s surprisingly easy – just look it up.). One of them even asked me to give them a quick crash course in data structures and algorithms because he was applying in a large and well-known software company.

For me, this term is silly. If you’re doing software development work, then you’re a software developer.

I’m not saying knowledge of these things isn’t useful. If you want to do more complicated work it is certainly useful to have these tools in your toolbox. Knowing them also makes you more flexible and expands your thinking. And certainly one should be the type of developer who is willing to seek out that knowledge if it turns out you need to use it.

But the problems with the whiteboard interviewing process isn’t that they require knowledge of these things; it’s that they require it under pressure, off the top of your head, without access to the internet. This doesn’t reflect how programming work happens in the real world, where you often have a team, an IDE, and even the entire internet to help you.

Unfortunately, it’s often far too costly for companies to evaluate you under real-world conditions, so many resort to this common interviewing process. After all, it is difficult to evaluate the performance of even developers you have been working with for months, what more someone who just whipped up a short function on a whiteboard in twenty minutes? We tried some variations on it at our last company, but for me it is a problem yet unsolved. Maybe someday…

 

Running Python 2.x and 3.x on Windows

I’ve been hesitant to try Python 3.x because it’s not backward compatible with Python 2.x which I’ve been using for scripting since forever. But recently I found out that since Python 3.3, they’ve included a launcher in the Windows version that supports having both versions installed.

You can use the launcher to specify the Python version to use at the command line (it defaults to whichever version was installed first):

Even better, the launcher will recognize a header in your .py files that can specify which version of python to use:

#!/usr/bin/env python3

If the launcher sees this header, it will automatically launch the appropriate python version. Handy!

Weekend Project: Twitter Trivia Bot

I had been meaning to try writing a Twitter bot for a while now. I figured a trivia bot would be pretty easy to implement, so I spent some time a couple of weekends to rig one together.

It’s (mostly) working now, the bot is active as triviastorm on Twitter, with a supporting webapp deployed on http://trivia.roytang.net/. The bot tweets out a trivia question once every hour. It will then award points to the first five people who gave the correct answer. The bot will only recognize answers given as a direct reply to the tweet with the question, and only those submitted within the one hour period.

Some technical details:

My scripting language of choice for the past few years has been Python 2.7. I’m using Tweepy to interact with the Twitter API, PyMySQL to connect to the database, and Flask to run the webapp. I haven’t used Flask in some time, but it’s still very straightforward. I actually had a harder time configuring the webapp with mod_wsgi on my host.

The main problem with a trivia system is that you need a large and high-quality set of questions. Right now the bot is using a small trivia set –around a thousand questions I got from a variety of sources. If I want to leave this bot running for a while, I’m going to need a much larger trivia set. However, reviewing and collating the questions is a nontrivial task. Hopefully I can add new questions every so often.

Feel free to follow the bot and help test it out. I’d be grateful!

Integrating Open Source Libraries

There are a few things that one should consider when using and integrating an open source library into your application:

  1. What are the licensing terms for the library? There are some liberal licenses that mostly let you do anything you want. The MIT license is an example of a very permissive license. Other licenses may provide a number of restrictions. Can you integrate with closed-source software? Can you distribute binaries without the source? Do you need to put some kind of attribution somewhere in your software? Another thing to look our for are the so-called viral licenses. Viral licenses specify that if you integrate their code into your system, then the terms of that license apply to your own software as well. These can be very dangerous from the standpoint of a company developing a commercial product. The most well-known example of such copyleft licenses is the GPL. Integrating GPL code into your system will often mean your software needs to be open source as well. While more open source is good, it may not be in the best interest of your company, so tread lightly.
  2. Which version of the library will you use? Many open source libraries will release updates on a regular basis. There may be some compatibility issues depending on the version you use. For example the popular open source CMS WordPress releases regular updates. If you are integrating with WordPress, you need to decide which version you are supporting. You will also need a plan for upgrading to future versions and ensuring compatibility. (See the next item.)
  3. How will you handle bugs and updates in the version you’ve chosen? It’s happened that our system was having problems due to a bug in an open source Javascript rich text control library. After investigating, we found that a fix was available but was only applied in a future version. Upgrading to that future version would require that we test the entire system for compatibility problems. That was out of the question. I ended up having to apply a fix to our local version only. This meant that we now had a different version from the “official” released version. This may lead to further compatibility problems in the future. You should also be aware of licensing issues here too. The license may require that you submit any modifications to the community as open source contributions.
  4. Is there an active help and support system for the software? The demands of enterprise software development often need timely support. There was a time we had to use a certain open source document management system. We later discovered that it was very hard to get help on any issues encountered, either from official channels or the community. This led to many difficulties and issues in that project. You might also want to check popular programming forums such as StackOverflow. Check how many questions about that library usually remain unanswered. If the number is low, that’s a good indicator that you might have trouble looking for help later on.
  5. What dependencies does the library have? You might want to make sure the dependencies don’t have any problems with the previous items above.

The great thing about modern software development is that we have a large number of open source libraries available to use and draw inspiration from, but that doesn’t mean you can just willy-nilly pick up just any library. The above questions provide a good starting point for evaluating open source libraries for your use.

Working with Client-Server Programs

Back when I was starting out as a software developer, webapps weren’t really a thing. Not as much as they are now anyway. My company provided training to new hires, but I didn’t get any web development training at the time, even though they already had a few web development projects in play at the time.

Instead my initial training involved mostly development of so-called client-server software. This was software that was installed and run on the client machine but they would connect to a remote database server. Up until the early 2000s most enterprisey-type systems used these kinds of program.

I was trained in using mostly two tools:  Borland Delphi and Oracle Forms and Reports. These were the sort of tools that would have been billed as reducing the need for specialized programmers or developers. They featured drag and drop user interfaces to design forms or report layouts. The Oracle tools featured robust database integration that let you drag and drop to associate form fields to database tables and fields. Supposedly even people with minimal training would be able to do the work of application programmers.

In practice of course, the only people patient enough to work with these tools in-depth were the programmers themselves. And client requirements always eclipsed the capabilities of these tools, such that expert programmers were still needed to push the tools to their limits and beyond.

Tools like Oracle Forms and Reports had a lot of problems with modern software development practices. For one thing, the “source” files weren’t actually text files. They were mostly binary format files with some PL/SQL interspersed, and could only be opened in the proprietary IDE that only Oracle provided.

Binary formats meant that while they could reap some of the benefits of source control (namely version history and such), actually figuring out the differences between revisions involved opening up both versions in the IDE and comparing each item/script/setting in the two files. Doing things like global find and replace were also a pain in the ass, especially in systems that had hundreds of forms. It was so bad I remember spending some time trying to reverse engineer the binary format so that I could attempt to make some of this work easier. No dice.

Another issue was that Oracle’s IDEs were notoriously buggy. Their Reports IDE in particular often had me incensed. If you dragged the wrong thing to the other wrong thing, you would crash the IDE entirely. In fact, we had a list of things to avoid doing that would crash the entire IDE entirely. The built-in text editors were so bad, I often had a entirely separate text editor open in a separate window so I could copy-paste PL/SQL code there for any complicated edits. (Back then we were for some reason fond of Crimson Editor although it wasn’t as full-featured as UltraEdit or as reliable with updates as Notepad++) This was actually also sometimes a problem as some versions of the Reports IDE also had a crash on paste.

Most of my first two years of work involved maintenance of Oracle Forms-based systems (with some Delphi work thrown in occasionally). I didn’t get an introduction to web development until late into the second year of my career. I was so impressed with web development I made a mock-up of one of the really complicated screens from one of our Oracle Forms projects and it seemed so pretty. I secretly hoped we could convince our clients to port them over to a web application. (1) No dice again; and (2) I probably would have regretted such a thing.

I know I would have regretted it because some number of years later, we got a project to migrate an existing client-server system to a web application. The original application was written in Powerbuilder. I figured it was a fairly straightforward reverse-engineer and implement as a webapp, but nooooo. One of the higher-ups decided that to save on costs, we should look into attempting to automate the porting process somehow. We were to write translation software that would take the Powerbuilder source and convert it into the appropriate web application screens.

This was ridiculous for a good number of reasons, but as a tech lead, I had to look into it and had to prove it was in fact not viable (Spoiler alert: it was not). The first concern was that Powerbuilder also had a binary format. This was solved when I found that someone had already written a tool to export Powerbuilder binaries into some kind of text format. It was a bit crude, but it was a way forward.

The second concern is that client-server form behaviors do not map well to web applications. My favorite example of this is editable grids. These are some kind of excel-like grids that were commonplace in client-side systems. And when users upgrade their legacy systems to web apps, they inevitably expect that their editable grids should work the exact same way on the web as they did on the client-side forms.

Back then Javascript-backed editable grids were still in their infancy, and each one we tried had their own sets of problems and limitations which would lead to some number of client complaints. Some clients even wanted editable grids that could accept copy-pastes from excel! Yeah, we had to develop Google Sheets! I personally had to roll my own editable grid framework from scratch at least once too. I ranted so often against client’s editable grid requirements that a few developers often quoted something I said in one of our internal chat rooms:

    <jaeger> you say “editable grid” i hear “problems”

Anyway yeah, porting client-side behavior to a web framework would not have been straightforward. At least not to the way our internal web development framework liked to do things. In order to have any kind of reasonable mapping, I would have needed to build some kind of Javascript rich-client framework (or see which ones of the emerging ones we could use). That also meant rewriting a lot of the backend code to support a bunch of new operations. It was a lot more work than the expected output of “a series of steps to easily or automatically migrate PowerBuilder source code to web applications.”

When I raised that the idea was not viable, I was asked to give a more detailed explanation. This involved taking one of the more complex Powerbuilder source files, putting it into an Excel, and line-by-line explaining how and why that line could or could not be translated into a web application. As an alternative option to an automated porting program they wanted a guide or process so simple that even junior developers with minimal experience could follow the guide and port the programs quickly. I think they eventually went with this approach. (Which was closer to our normal reverse-engineer-and-reimplement methodology.)

These days web applications are the norm and client-server programs are a thing of the past. They were relics of a different age. The browser is our universal client now, on the desktop at least. On mobile, “apps” are effectively client-server programs, though they have a different set of advantages and limitations. Maybe in the future mobile apps will converge as well, into a unified browser client – although probably mobile browsers need to be more feature-rich before this convergence can take place. My historical disdain for client-server programs carries over to mobile apps though – I don’t like to use too many of them, although that is a story for another blog post.

Handling unexpected errors in web applications

So after so many months of development you deployed your webapp to production and it’s up and running and everything is fine and you celebrate and your work is done right?

Not really.

Two days later you get an urgent support call in the middle of the night. (Your clients are halfway across the world.) They’re asking why the website is inaccessible. You check via your browser and sure enough there’s an error 500. You have to ssh into the server. Half-asleep, you have to quickly comb through the log files to figure out if there’s something you can do about it right now or you have to sadly tell the client that the team has to handle it in the morning.

Nobody expects software errors. And nobody can predict how they will happen. (If we could do that, there wouldn’t be any errors!) But in general you should stay ahead of future headaches during development time by having robust error handling and reporting principles in place for your development team to follow.

There are two classes of errors:

  • Validation errors – checks on user inputs. These errors are expected, and the most important thing here is to present clear error messages that tell the user what was wrong and how they can correct it.
  • Everything else is an unexpected error, which we can breakdown further according to how the development team would react:
    • “That shouldn’t happen!” – the broadest category, usually caused by human errors in coding/configuration or hardware problems. Examples include: database failure, typos in configuration files, different data formats used by different developers, unexpected system crashes and so on.
    • “Hmm, oh yeah, we didn’t think of that.” – i.e., design flaws or unhandled cases. These are errors caused by program flow or usage that was not anticipated in the program’s design but are considered valid use.
    • “That can’t possibly happen!”, also popularly known as “We have no clue how that happened.” Most unexpected errors start out under this category then after some detective work (some developers use the term “debugging”), it transfers to one of the above categories.

Unexpected errors may also be either recoverable (the user can either attempt to repeat the operation to continue or use other functions in the system) or catastrophic (the entire system or subsystem is completely unusable.)

Ideally, what happens when an unexpected error occurs? There’s a number of things that probably need to be considered:

  1. The user needs to be aware that it’s not an error on his part and that the problem needs to be reported to support or the system administrator.
  2. The user needs to be presented with enough information to give to the support team to allow them to determine the cause and fix the error.
  3. The program needs to capture what information it can to help the support team find and fix the error.

For client-side errors (typically JavaScript problems, but can also be problems in HTML/CSS rendering), all of the above are difficult, since the mechanisms by which you present #1 and #2 and by which you can do #3 may be compromised by the error. Typically for any client-side errors, you are almost entirely reliant on the user’s description of the error and the steps that led to it – not the most reliable source of information, but you make do with what you have. If you are unable to reproduce the error during debugging, you are in for a whole lot of speculation and trial and error.

For server-side errors, it’s a bit easier than client-side errors. The first two items are typically handled by having a catch-all error page in the application that gives an error code and message for unexpected errors.

For the last item, it generally means logging, either to a file or a database-backed audit trail (the file is more reliable). Generally we log the error/exception encountered, including the stack trace, and some log messages on each request/page access. A majority of the time, having the stack trace is sufficient. The access logs allow us to back-trace previous steps which might have contributed to the problem. Any special conditions encountered in the code should also be logged – this helps to identify errors that can only happen under some combination of special conditions.

You don’t want to log too much information (what to log might need to be a separate post all on its own), but just enough to help the support team determine the cause of the problem.

Sometimes, the error is specific to the data being used by the users, or happens only in the environment they are using. And the support team investigating the problem will often not have access to that data or that environment, so your logging and the error reports have to be good enough to allow them to deduce the conditions that caused the error.

Error handling for deployed systems is already hard – so make sure not to make it harder on the future support team by having standard error handling and logging mechanisms.

Problems in Large Software Dev Teams

Hopefully by now most developers and project managers are well aware of the mythical man-month and Brooks’ Law:

Adding manpower to a late software project makes it later

The idea is that communications overhead scales up quickly as you add more people to a project. Oftentimes it is counter-intuitively not worthwhile to keep adding more people to try to catch up. Some implications of larger team/project size may not be immediately obvious. Some problems scale up faster as team/project size grows:

  1. Lower productivity due to increased overhead as mentioned above.
    • Meetings will tend to involve more people and take longer
    • There will be a lot more emails
    • Project management effort scales up quickly too
    • More people need to be allocated to maintaining builds and servers
    • More time needs to be spent on task prioritization, bug triage, etc
    • More people asking WTF happened to their code (LOL)
    • Any decision making that requires consensus building takes longer
    • It becomes more difficult to find the right person to ask things
  2. Simply due to the number of people, there are more things that could go wrong
    • Developers breaking the build happens more often
    • People going on sick days will happen more often
    • Server performance becomes much more important since any delay or downtime affects more people
    • Schedule delays or others unexpected problems will be more likely
  3. Maintainability becomes more important
    • Technical debt becomes more burdensome and poor code is more likely to come back and bite you in the ass in the future
    • The need for good coding and development standards increases
    • Higher likelihood of code duplication (“I didn’t know that Developer R already wrote a function that does X!”)
    • More important for code to be well-decoupled, to reduce the likelihood of one developer breaking a lot of things
  4. Source control gets harder to use, with so many people making so many changes.
    • The team needs to develop standards for commit messages and linking commits to bug reports. to make it easier to track and monitor changes
    • Source control commit comments need to be a lot more helpful or descriptive.
    • More commits happening in the same amount of time, the more you need to be constantly updating from the repository.
    • Merging is more likely to become difficult and complicated (may be made easier by modern source control systems)
    • More important to use more, smaller files instead of fewer large files (less likely to produce conflicts)
    • Need better coding/programming standards. Otherwise you have the problem of changes/commits being difficult to track for example if one developer uses different autoformatting standard (his commits will have many small reformats)
  5. Having consistent rules for naming, UI,  and other things becomes more important
    • The more developers you have, the more likely that they will have different ways of thinking. There are far more likely conflicts among a team of 8-10 developers than between 2-3 developers.
    • It becomes more important to have a standard or plan for where different kinds of files should be placed. Otherwise you run into problems like different developers using different folders for their css or different package naming conventions, etc.
    • Consistency and standards more difficult to enforce (since there are more devs)
    • Need to keep things consistent on all levels: databases, code, UI, and so on.
  6. Documentation becomes more important
    • Tribal knowledge is often spread out among multiple developers
    • Undocumented things are less likely to be passed on to new developers
    • Developers unaware of undocumented things are more likely to have difficulties or to break things
    • Becomes a lot more difficult to absorb new developers into the team in times of urgentness
    • Documentation more likely to quickly become out of date due to rapid pace of changes

Anything you want to add?