Wearing dice on my head since 2008 Programmer, engineer, scientist, critic, gamer, dreamer, and kid-at-heart. Randomly amazed.

Software Development

My history in text editors

| Categories: Tech Life Software Development | Tags:

Text editors (and by extension IDEs) are a programmer’s best friend. I thought I’d look back at a number of text editors I’ve used over the years. (I grew up with Windows, so I won’t list vim/emacs/nano here, even though I’m at least a bit proficient with vim by now. That is, I know how to exit vim.) Notepad – of course, the default editor in Windows. The one we turn to when all else fails.

According to Malcolm Gladwell’s book Outliers, you need 10,000 hours of continuous sustained practice to become an expert. There are 168 hours in a week. If you never sleep and you eat as you practice, you can become an expert in 60 weeks. (Around 14 months) If you sleep 8 hours a day, you only have 112 hours in a week. If you eat as you practice, you can become an expert in 90 weeks.

Version Issues

| Categories: Software Development | Tags:

SCM (Software Configuration Management) doesn’t just refer to version control for the software you’re building. It also means controlling the versions of software you depend on. This includes operating system and programming runtimes. Sometimes even minor version differences can cause issues in running your software. I have two example stories to share: One of our clients asked us for help with an upgrade their production servers from CentOS 6.4 to 6.

Systemic change is difficult. I’m talking about software projects/systems, but there are a lot of parallels with societal systems too, like governments or states. I’ve been in large projects with hundreds of thousands of LOC where a lot of the code was painful to read and full of code smells and so on. It happens over time as projects get maintained by different developers and teams or different enhancements or changes are made.

Ten years ago this month, I started studying Django by trying to build my own blog application. I found the code lying around while I was going through some backups lately. It’s way out of date, it uses an early version of django. I thought of bringing it up to speed, but that didn’t seem practical. Instead, for archival purposes, I cleaned it up a bit and uploaded the code to a github repo.

Malcolm Gladwell, in an article from 1996 discussing the Challenger disaster, tells us: This kind of disaster is what the Yale University sociologist Charles Perrow has famously called a “normal accident.” By “normal” Perrow does not mean that it is frequent; he means that it is the kind of accident one can expect in the normal functioning of a technologically complex operation. Modern systems, Perrow argues, are made up of thousands of parts, all of which interrelate in ways that are impossible to anticipate.

Working hours and overtime

| Categories: Gaming Software Development | Tags:

Rockstar was in the gaming news recently because they mentioned that some of them had worked 100-hour weeks on their massive sequel to Red Dead Redemption coming out soon (no idea if I’ll play this). The idea of 100 hour weeks seemed insane to me, and it got me thinking: I’ve done some serious overtime before, have I ever gotten close to that amount of work in a week? Luckily, I didn’t have to speculate too much, because I had data (I love data).

Mentoring in Software Development

| Categories: Software Development | Tags:

Mentoring is one of those tasks that’s to be expected of anyone in a senior software development role. This usually involves reviewing other people’s code, helping them with tough technical issues, and even giving career advice. I’m not sure how good I am when it comes to mentoring other software developers. When I first became technical lead on projects, I got some evaluations from junior developers saying I can be “intimidating”.

StackOverflow

| Categories: Software Development | Tags:

As a programmer, I’ve always been a big fan of StackOverflow. I asked my first question there and also wrote my first answer in September 2008, which was the month the site launched, so I was pretty much there from the beginning. The site was a huge boon to programmers when it first came out, because the internet as a venue for asking questions and answers back then was a horrible fragmented landscape of small forums and mailing lists and sites like Experts Exchange, all of which were terribly designed.

The Setup 2018 Edition

| Categories: Gaming Software Development Tech Life | Tags:

While browsing through my old blog posts, I found one about my setup from 2010. I figured it was a good time to do an update. I like doing posts like these because it provides an easy reference for me to look back and see what I was working with at a certain point in time. What Hardware Do I Use? Desktop. I bought a new desktop rig back in late 2015, here are the specs:

My Coding Approach

| Categories: Software Development | Tags:

I was thinking about my typical approach to coding. When writing a new feature, I tend to implement in the direction of where the data flows, starting from the user interface then to the backend and back to the frontend and wherever else that goes. I will build incrementally, using debugging tools or console printouts to ensure that each step is working correctly. As an example, here’s how I did a recent web-based function:

Random thoughts while walking at night: The structure of government can be a bit analogous to the structure of a software development project. The Constitution is like the requirements for a project. It’s kind of high-level and (I believe) shouldn’t be too detailed. Supposedly the requirements are written by the client. For a country like the Philippines the client is “we the sovereign Filipino people”. Slight tangent: I used to know this guy who was one of those rabid “we need to amend the constitution” types and he asked me to review a “mathematical model to track the budget as a function of tax collection and monetary policy” that he wanted to include in a proposed new constitution.

A while back I wrote about my experiencecoding and maintaining an in-house web framework at a previous job. It was a full-stack web framework. We had libraries for front-end Javascript up to server-side database connections. And the entire stack was tightlycoupled. But while the framework was serviceable, it was almost always behind modern trends in web development. I always felt like we were playing catch-up. And as a developer I wanted to widen my horizons and try out more things.

MarkLogic NoSQL

| Categories: Software Development | Tags:

I recently attended a few training sessions for MarkLogicheld at an office in a nearby business center. Now, I'll forgive you for not knowing what MarkLogic is, as even I hadn't heard of it before six months ago. MarkLogic is (apparently) the leading Enterprise NoSQL provider. NoSQL is big and sexy right now because of the supposed advantages in handling big data, and large web companies like Google and Facebook use a lot of NoSQL in the backend.

Back in 2004, I signed up for the Google Code Jam for the first time. Unfortunately I didn’t make it past the qualifying round. I was a bit luckier in 2008 and 2010, making it to round 2 both times. In fact in 2008 as I recall I was one of only two participants from the Philippines who made it to round 2, which allowed me to jokingly brag about being the #2 programmer in the country.

Recently, a developer needed to undergo a tech interview at US immigration:[]1 This may surprise some people I’ve worked with, but I didn’t have formal computer science training in school. I’m not actually a computer science major. Yet I’ve worked as a software developer for more than a decade now. Literally zero times have I needed to write a sorting function or balance a BST. I have a rudimentary understanding of some sorting algorithms (mostly just bubble sort and selection sort), and I have some idea of how to balance a BST.

I’ve been hesitant to try Python 3.x because it’s not backward compatible with Python 2.x which I’ve been using for scripting since forever. But recently I found out that since Python 3.3, they’ve included a launcher in the Windows version that supports having both versions installed. You can use the launcher to specify the Python version to use at the command line (it defaults to whichever version was installed first):

I had been meaning to try writing a Twitter bot for a while now. I figured a trivia bot would be pretty easy to implement, so I spent some time a couple of weekends to rig one together. It’s (mostly) working now, the bot is active as triviastorm on Twitter, with a supporting webapp deployed on http://trivia.roytang.net/. The bot tweets out a trivia question once every hour. It will then award points to the first five people who gave the correct answer.

Integrating Open Source Libraries

| Categories: Software Development | Tags:

There are a few things that one should consider when using and integrating an open source library into your application: What are the licensing terms for the library? There are some liberal licenses that mostly let you do anything you want. The MIT license is an example of a very permissive license. Other licenses may provide a number of restrictions. Can you integrate with closed-source software? Can you distribute binaries without the source?

Back when I was starting out as a software developer, webapps weren’t really a thing. Not as much as they are now anyway. My company provided training to new hires, but I didn’t get any web development training at the time, even though they already had a few web development projects in play at the time. Instead my initial training involved mostly development of so-called client-server software. This was software that was installed and run on the client machine but they would connect to a remote database server.

So after so many months of development you deployed your webapp to production and it’s up and running and everything is fine and you celebrate and your work is done right? Not really. Two days later you get an urgent support call in the middle of the night. (Your clients are halfway across the world.) They’re asking why the website is inaccessible. You check via your browser and sure enough there’s an error 500.

Hopefully by now most developers and project managers are well aware of the mythical man-month and Brooks’ Law: Adding manpower to a late software project makes it later The idea is that communications overhead scales up quickly as you add more people to a project. Oftentimes it is counter-intuitively not worthwhile to keep adding more people to try to catch up. Some implications of larger team/project size may not be immediately obvious.

Just a list I’ve been maintaining for a while: (Disclaimer: This list in no way implies that developers who don’t exhibit all of these attributes are terrible human beings who don’t deserve to live. But working with developers who exhibit many of these traits will probably result in a better experience over the course of your developer career.) Laziness, Impatience and Hubris – from the well-known (notorious?) Larry Wall quote Communicates well; is able to explain and communicate his ideas clearly, especially to nontechnical people; able to write good documentation Understands the concerns with scheduling and project management and communicates clearly with the team to avoid problems.

So the other day I was reworking a Python script that I had been using for years on my home PC to manage and categorize some downloaded files for me. This time I wanted to add some smarter behavior to make it more able to figure out when to group files into folders without constantly needing manual intervention from me. To do this, I needed to persist some data between runs – so that the script remembers how it categorized previous files and is able to group similar files together.

Because of the nature of the web and the fact that you should never trust user input, all the validation in a web application should be done on the server side. You can additionally provide validation on the client side (via JavaScript), but this is only a concession towards a better user experience and should not be used as a substitute for server-side validation. One would think that anyone with a basic understanding of how HTTP works would understand the above easily and any failure to practice it should be considered amateur hour.

Cleaning up your Code

| Categories: Software Development | Tags:

In one of my most recent projects, a large system that had gone through a relatively long and unstable period of many, many changes due to sales demonstrations, different clients and whatnot, one of the “fun buffer tasks” I always kept around for devs was code cleanup. Because of the unstable nature of the project, there was always a lot of duplication, unused/unnecessary/obsolete classes/functions/files and so on. Unnecessarily large CSS files where most of the selectors were no longer really needed or JS libraries that weren’t actually used.

Related: Learning new skills While many people working as programmers/software developers are happy enough specializing in a single programming language or platform, I generally consider it a better idea to have a wider toolset and the ability to easily pick up new programming languages as needed. The benefits should be obvious: when you have a wide variety of tools under your belt and are able to quickly learn to use a new tool, the number of work options you have increases greatly.

In any reasonably large software project, the system will be so large that no one developer will have a good grasp of the details of every function in the codebase. The tendency is for developers to specialize – that is, developers tend to focus only on certain parts of the codebase and become more familiar with that part, while not having much knowledge about the other parts. This tendency is self-reinforcing – once it becomes known that the developer is an “expert” in the given module, there is a tendency that he will be assigned the most difficult and urgent tasks or fixes related to that module, further cementing his expertise.

I was in a meeting once with my boss (literally the CEO, a Malaysian) and some representatives of another company (Americans) where we were discussing the technical details of a possible future partnership. At one point, one of the Americans said to my boss that he was pleasantly surprised that I was openly speaking up independently of my boss and willing to correct him on some points when he didn’t quite get the technical details right.

In JavaScript, referencing variables that are declared outside of a function’s scope can be tricky. If you have code like this: var btn = document.getElementById("BTN"); var test = 1; btn.onclick = function() { alert(test); } test = 2; The click handler above retains a reference to the test variable even though it falls out of scope as soon as the script block finishes execution. When you actually click the button, the alert will show the last value of the variable when the block finished execution (2) instead of the value at the time the function was initialized (1).

This is a story of something I consider to be one of my worst mistakes in software product development. Some years ago I was asked whether it was feasible to write software that would be integrated with Software X that allowed us to export that software’s output into a format that was compatible with Standard Y. I took a look and after a while came back with “Well sure. We could use Programming Language M that has an API that lets us integrate into Software X so we can export the output data.

Unclear error messages

| Categories: Software Development | Tags:

“Button for non-service floor does not light up.” For more than a decade I regularly went to an office building where the elevators verbally spouted this nonsense message whenever you tried to go to a floor that the current elevator car did not service. For context, the elevators in the building were zoned programmatically – this means that they only service a particular subset of the floors that are provided on the elevator panel itself.

  There was this project we had where there was a strange bug. The developer working on it found that the problem only appears when the record ID was 12. When it was 11 or less, everything was fine. When it was 13 or more, everything was also fine. After some investigation, it was found that there was some code that executed with a condition of “if record id == 12”, which was already a WTF.

The software development process is already difficult mainly because a lot of it so imprecise. Requirements are often only vague wishes that the client has, with no regard to the sheer number of instructions needed to implement those requirements. Throughout the entire process it’s important to use feedback loops to determine whether development is on the right path. And like all feedback loops, their effectiveness often hinges on how quickly we are able to turn around and give and incorporate feedback into future iterations

It Should Be Easy, Right?

| Categories: Software Development | Tags:

In Tagalog: “Madali lang naman diba?” Probably one of the most annoying things a programmer can hear, especially from a client or a manager who has no appreciation of how complex software development is. It’s presumptuous at best and actively damaging to schedule and morale at worst. We already know estimation is hard, there is no need to make it more complicated by automatically assuming the best-case scenario (or in many cases, an impossible scenario)

Favor Composition Over Inheritance

| Categories: Software Development | Tags:

“Composition over inheritance” is an object-oriented programming principle that I’m sad to say many devs I’ve encountered aren’t too familiar with. Composition provides greater flexibility, modularity, and extensibility in large software systems as compared to inheritance, especially for statically typed languages like Java that don’t support multiple inheritance The most common examples of the problems caused by too much inheritance involved generic object such as the game objects example in the wikipedia page linked above.

Coding Frameworks

| Categories: Software Development | Tags:

For the better part of my software development career so far, I’ve had the doubtful pleasure of being one of the devs using and maintaining our in-house web development framework. Framework coding is a bit different from the actual application development. At the core it’s a simple idea: you have a whole bunch of code that helps do programming tasks that you expect will often be necessary in a certain set of projects, so you write that code with the intent of reusing it across multiple projects.

A Programmer's Hubris

| Categories: Software Development | Tags:

“We will encourage you to develop the three great virtues of a programmer: laziness, impatience, and hubris.” — Larry Wall, Programming Perl (1st edition) Hubris is a fancier word for an excessive sense of pride. Why is this to be considered a great virtue for a programmer? Programming is at least partly an act of creation, which means there is an element of craftsmanship involved. A craftsman imbued with hubris is able to take pride in his work – he is driven to create work that is the best quality he can provide.

Type Interfaces Matter

| Categories: Software Development | Tags:

… especially for strongly-typed languages. In one of the bigger Java projects that I took over, I was often annoyed to find some devs had written method signatures like public void doTheThing(HashMapparams) Which is silly – not because of the naming, that’s obviously not a real-world function name. The silly part is requiring a particular implementation (HashMap) instead of the generic interface (Map). It unnecessarily restricts your API and makes it less flexible.

Overhead scale up rapidly as project team size increases. Every time you add a new person to the team, he comes with a lot of overhead such as the need to learn the project details, responsibilities of other team members, who to consult when there’s trouble, custom project procedures, and so on It’s a reinforcing cycle too. As overhead increases, the team imposes more processes and restrictions to make sure everyone is doing the right thing and there are no screw-ups.

Overtime

| Categories: Software Development | Tags:

Overtime in software development projects seems to be a given. Sure, there are projects and companies that don’t need it, but those feel like the exception rather than the norm Overtime in software development is a natural consequence of schedule pressure and the fact that estimation is hard, which is why it’s understandably common, but that doesn’t mean we shouldn’t try to avoid it More than once I’ve been in a situation where the team stays overnight to try to get a build or release ready for the next day only to run out of time and have to delay the deployment anyway.

Will AI replace all our jobs?

| Categories: Software Development | Tags:

A while back, news came out that an AI was able to perform a diagnosis after the human doctors were stumped. Having just finished binge-watching the entire run of Star Trek Voyager, I immediately thought that was the sort of AI advancement that could lead to such things as Voyager’s Emergency Medical Hologram, an AI which could replace a human doctor. Of course, it is still very unlikely for this to happen anytime soon, medicine is a very complex field and new things are still being discovered all the time.

Without a doubt, the number one problem in software development projects is schedule pressure, that is, the pressure to meet unreasonable deadlines and targets Almost all other problems can be overcome if there were no schedule pressure: weaker developers could be mentored to become better, more productive, and commit less faults less faults overall will be committed anyway if there were no schedule pressure poor requirements could be threshed out in more detail difficult clients can be argued, worn-down, and eventually reasoned with problematic team members can be counseled, or replaced with new blood and so on But of course, the reality is that we live in a world with deadlines and targets, many of them set by people who have no idea about the complexities of software development.

The Secrets to Perfect Estimates

| Categories: Software Development | Tags:

Estimation is hard. Estimation has long been the bane of many software developers and software development projects. But there are two secret ways to be able to produce perfect estimates for software development work all the time! One is dependent on talent, and the other is dependent on technology Psychic precognition, i.e. be able to predict the future Have a time machine, so you can go back in time and tell yourself how long the work would have taken Such precognition is necessary to have perfect estimates because of all the unknowns present at the start of a software project.

I like to say that software development is a challenging career because no two projects are ever the same and there are always new challenges to face and new concepts to learn, but the truth of the matter is a bit more complex. Writing software is about breaking down large problems into a series of very small technical problems for which we already have solutions. Examples of small enough technical problems include list sorting, comparison, arithmetic operations, path traversal, string concatenation, returning a string as an HTTP response, rendering text to the screen, retrieving submitted parameters from an HTTP request, and so on.

Living With Development Standards

| Categories: Software Development | Tags:

I was originally going to write a post about the problems development teams face as they get larger, but the section on development standards was long enough by itself so here we are. Having some sort of development standard in a project development team becomes a lot more important as project size goes up (for obvious reasons). There are different kinds of standards to consider, but generally I break them down into design standards and coding standards.

Pros: It is a very rewarding career financially. Software development often ranks in the top 10 highest-earning careers in most countries There is a lot of scope – you could be developing web applications, mobile applications, embedded applications, client-side, server-side, data analysis, artificial intelligence, games, etc It is very difficult to be bored. You can always automate away the boring stuff. Different projects always present different challenges. The field is evolving rapidly so there are always new things to learn.

Evaluation of programmer performance is notoriously hard. You want to be accurate and at the same time fair such that all programmers on your staff are evaluated in the same matter. However, there are no good, objective, universally accepted standard metrics. It follows from the fact that there are no good, objective, universally accepted standard metrics for program size. Typically each programmer in a team will not be doing the same task or even the same type of task, so in order to produce fair evaluations you will need some standard metric of program size to normalize any evaluation.

Learning to SQL

| Categories: Software Development | Tags:

****Since I wasn’t a graduate of computer science, there were many concepts of software development I really only got exposed to when I started working. One of those was the concept of a relational database, and hence SQL. The company I worked at gave all new hires a training regimen that started with about a week of SQL. Despite not knowing anything about it beforehand, I took to it like a mouse takes to cheese.

Certifications

| Categories: Software Development | Tags:

I never really put much stock in certifications. I felt that they were no guarantee of knowledge or expertise, and that many people who did have knowledge or expertise wouldn’t necessarily have a certification to say so. Add to that it often seem overpriced to even apply for the certifications, so I didn’t have a high opinion of them. That being said, I have had the opportunity to take professional certification exams twice in my life (both luckily paid for by my employer at that time).