Category Archives: Software Development

The Setup 2018 Edition

While browsing through my old blog posts, I found one about my setup from 2010. I figured it was a good time to do an update. I like doing posts like these because it provides an easy reference for me to look back and see what I was working with at a certain point in time.

What Hardware Do I Use?

Desktop. I bought a new desktop rig back in late 2015, here are the specs:

Case: Silverstone Precision 10SST-PS10B
Fan: 120mm internal aux fan
PSU: Cooler Master B600 V2 600W
CPU: Intel Core i5-4460
RAM: 2x DDR3 Kingston HyperX Fury 8gb 1600
Graphics: Asus GTX970 STRIX OC 4gb
SSD: Samsung 850Evo 250gb
HDD: Seagate 2tb
OS: Windows 8.1 OEM which I immediately upgraded to Windows 10 (upgrade was free at that time)

This cost significantly more than any desktop I had assembled before, but it’s worth it. Still going strong up to now. Can easily handle most programming tasks I throw at it, including the notoriously resource-hungry Android development process and of course Google Chrome. Pretty good for gaming as well, although I’m not really the “4k 120fps” kind of gamer.

I have two old monitors (a 19″ and a 21″) hooked up to this desktop (haven’t bothered buying a new monitor… it’s on my list).

Late last year I also impulse-bought a Razer Blackwidow Ultimate 2016, I’m mostly happy with it (it’s my first “real” mechanical keyboard”) Recently one of the keys has been acting up, being a bit unresponsive but it got better after I removed the keycaps and cleaned up a bit under there.

Moved the tower case under the desk to get a bit more desk space. Will see if this works

As you can see above, I also have a much wider desk now than the dinky old computer table in the old post.

Laptop. My only laptop currently is a work-issued Acer Aspire ES1-433-38DR. It’s nice and handy and workable, but it’s not even a year old and I’ve already had some issues with the laptop suddenly dying when it’s not plugged in. For this reason (and a few other reasons like being able to publish iOS apps), I’ve been considering getting a Macbook Pro. (The only thing holding me back is the ridiculous price!)


Gaming consoles. Gaming is one of my indulgences, so there’s a few:

  • I still have a PS3, but it’s no longer the original model from the old post. We had to buy a new one after some time because the original conked out. The PS3 is rarely used nowadays however. I still keep it around because I actually still have some backlog that I like to fantasize about finishing one day. *Shakes fist at Ni No Kuni*
  • I also have a WiiU that I purchased around 2012 or 2013 I think. My WiiU library isn’t huge, but it let me play 3 different Zelda games! I often think about getting a Switch, but I want to finish what remains of my WiiU library first. (Only Twilight Princess and Earthbound remain AFAIK)
  • I have a PS4, purchased in October 2014. My PS4 library is larger than my WiiU one, but these days I typically only play one game on either console at a time.
  • I also bought a Steam Link to stream gaming from my desktop. I use my Logitech F710 wireless controller with it. It’s great when I have platformers on Steam that I want to play.
  • Handhelds: I have both a PS Vita and a 3DS. Okay, actually I have two 3DSes, the other one is a 3DSXL. I rarely do handheld gaming anymore though, unless there’s a long ride or something. I do still tend to carry one of these whenever I go out with a bag. Just in case.


TV. Mid 2017 one of our 32-inch Samsung TVs in the house conked out with a bad display so we decided to buy a new one. We decided on a 43″ LG Smart TV. Super happy with this, the display is much better than the previous TV. IDK how it works, I think they have some sort of software optimization that makes things look better. It’s great for Netflix, Youtube, Gaming, etc.

Tablet. I have an 9.7″ iPad Pro that I bought late 2016. I also bought an Apple Pencil for it, my main intention was to have a tool for sketching. Pretty happy with it, even though it looks pretty battle-scarred by now as I’m not the most careful user.

Cellphone. Wow, I was still using the Nokia 5800 in that last post. I’ve gone through quite a few Android smartphones since then, mostly Samsung Galaxy models, with the most recent one being a secondhand S5. I always have problems with battery life (probably due to being on data all the time) so last December I decided to buy myself an ASUS Zenfone Max 4. It’s not as high-spec as the Samsung Galaxy line, but this has 10000mAH battery which means I can go all day without having to scrounge around for a charger. I haven’t used my Powerbank in a while!

Others. I have a 10,0050mAH Asus Zenpower powerbank that was way more useful before I got the Zenfone. (I just now realized they’re related brands). I also have a Fitbit Charge 2 for some reason. I find it more useful for tracking my walking than the mobile apps I used to use.

What Software Do I Use?

OS. Windows 10 is mostly serviceable. I’d love to have a Linux setup as well, but it’s difficult to justify that as a primary OS when Windows has games! Instead I have a Linux Mint VM that I run on my desktop when I need to do Linux-y stuff.

Text Editor. These days any programming I do usually involves some combination of Python, JavaScript/Node or PHP, so the text editor is my weapon of choice. I tried Atom for a while, but it was a tad too slow for my tastes. I also tried Sublime Text, which is great and has everything I need but for some reason keeps nagging me to pay for it. Recently I installed VS Code, which has been pretty good so far so for now it’s my main editor.

Other Programming Stuff. For times that I need to dig into any Java stuff, I also still have an installation of Eclipse Mars on my machine. (I don’t miss Eclipse). I also have Unity3d installed on my desktop, as I still like to pretend I have time to work on one of my gamedev projects. For SSH, I used to use Putty, but lately I discovered the great “Bash on Ubuntu on Windows”. The name is a handful, but it makes doing Linux shell stuff like SSH and curl and whatever a lot easier.

Web Browser. Most of my work is web-related and I often use cloud resources, so the web browser is important. I recently switched from Google Chrome to Mozilla Firefox as my main browser, since the latest Firefox promised a bunch of speed improvements. TBH, I can’t quite tell the difference.

Office Suite. As mentioned above, I often use cloud resources, so it’s mostly GMail, Google Calendar, Google Drive and Google Docs/Sheets for my needs. I also have a local installation of LibreOffice because sometimes I have to open Excel files that are hundreds of megabytes large and Google Sheets refuses to import it. For cloud storage, my preference is Dropbox.

Note-taking. I’ve grown fond of Google Keep. It’s lightweight and handy enough for my usage, and the Android widget means I can use it to capture random thoughts and ideas throughout the day. On the iPad I also use NotesPlus if I happend to be in a meeting where I feel like writing notes down with the Apple Pencil.

Communications. Usually some combination of Slack + Skype for work-related stuff, although some clients prefer WeChat or WhatsApp so I have those too. For personal comms, it’s mostly Viber and FB Messenger and sometimes Discord.

Sketching. On my iPad I use and am a big fan of the Procreate app for digital sketches, which I think I even paid for a while back.

Odds and Ends. I have a bunch of other software utilities I often recommend. I use VLC Media Player for watching stuff on my desktop (doesn’t everybody?). I use Space Sniffer to check disk usage. Audacity for the rare instance that I have to do some kind of audio editing.

All the photos in this post are available as a Flickr dataset:

My Coding Approach

I was thinking about my typical approach to coding. When writing a new feature, I tend to implement in the direction of where the data flows, starting from the user interface then to the backend and back to the frontend and wherever else that goes. I will build incrementally, using debugging tools or console printouts to ensure that each step is working correctly.

As an example, here’s how I did a recent web-based function:

  1. The user needs to see a new button on the screen, so I add the HTML for that button and just have the click handler be an alert confirming the button has been clicked.
  2. The button needs to open a modal dialog. I write the HTML for the modal dialog, then add the code to the button to launch the modal.
  3. On load, the modal dialog needs to fetch some data from the server. I write the Javascript to make the ajax call on launch of the modal.
  4. On the backend, I map the URL route used by the above ajax call to a controller method. The controller method simply outputs a debug message to the console. I refresh the page on the front end, click the button to launch the modal and verify on my server’s console that the controller method has been called.
  5. I modify the controller method to call a method on the relevant model. I write the stub method on the model with a dummy return value, which the controller method then outputs to the console. I verify the entire flow again.
  6. I write the retrieval logic in the controller method. I verify the entire flow again, using server console output to verify the correct data is being returned to the controller.
  7. I have the controller write the returned data to the response as JSON. I go back to the frontend and modify the success handler of the ajax call to log the response to the Javascript console. At this point, I also update the error handler to do whatever is appropriate. I verify the entire flow again, this time confirming that the correct data is displayed in the browser console.
  8. The user needs to see the data from the response, so I write the Javascript and HTML to render the response inside the modal dialog. I verify the entire flow again, this time confirming that the correct data is shown in the modal dialog.

I’m not sure if this is the best approach, but it works for me. It feels incremental, systematic, methodical, and easy to see where things go wrong (which they inevitably do). As kind of an upside, I also strangely feel compelled to not stop until I have working complete flow. Of course the downside is that if I have to stop at any point, I now have a nonworking UI component, but at least that’s a clear indicator of where I need to pick up next time.

I realize that a lot of modern software engineering advocates recommend some kind of test-driven approach. I imagine with such an approach start with the smallest iota of functionality, write a test for it, then write the actual code, then work outwards from there, building more complexity on top. So for the example above, maybe I should have started with the retrieval logic on the backend then work my way outwards from there.

I do that sometimes but I find that as a full-stack engineer my natural tendency is work my way across a new feature in this manner – from user interface to backend and back. Combined with the difficulty of writing code to test UI/HTML/JavaScript, this means I rarely get into the mindset of test-driven development. It’s not something I ever got used to. Maybe someday a switch will click and I will get it and my mind will be blown.

My approach does have some benefits though, in that I’m defining the interfaces first before diving into the guts of the logic. In that sense it’s kind of “test-driven” except all my testing is manual.

I do think having such a systematic, incremental method of implementation is a good skill for the junior programmer to learn though. My experience is that many younger programmers (especially those fresh out of college) tend to write huge chunks of code/logic/functionality then get surprised when the very first thing breaks down.

Recently when sitting down with a junior to help them walk through some problem I often show them how I debug problems this way. I trace the entire flow from start to end, adding debug messages as I go, to help identify at which step the process goes awry. I also try to describe my proposed approach for complex new functions this way. I’ll prototype a small part of it from frontend to backend, writing debug messages all the way, and it conveniently leaves the junior with a template for writing the rest of the functionality. (Although I am as of yet unable to attest to the efficacy of this approach)

I’ve been writing code for a long time, and while I wasn’t self-aware enough back when I started out to document how I was writing software, I like to think I’ve improved a lot since then. And for sure there’s still a lot to learn. Maybe some years from now I’ll look back on this blog post and chuckle about how naive I was and write a new blog post detailing an even better approach.

Software Development and Government

Random thoughts while walking at night: The structure of government can be a bit analogous to the structure of a software development project.

The Constitution is like the requirements for a project. It’s kind of high-level and (I believe) shouldn’t be too detailed. Supposedly the requirements are written by the client. For a country like the Philippines the client is “we the sovereign Filipino people”.

Slight tangent: I used to know this guy who was one of those rabid “we need to amend the constitution” types and he asked me to review a “mathematical model to track the budget as a function of tax collection and monetary policy” that he wanted to include in a proposed new constitution. I told him I didn’t believe such detailed rules shouldn’t be in the constitution – that would unnecessarily tie us down to a specific model that we may or may not regret later on. I ended up declining to help him with his strange hobby.

The Legislative branch are the analysts. They make the detailed specs/laws to define how we’re going to satisfy the requirements/constitution. In a real project, the client usually signs off on the specs. In government, we supposedly sign off when we vote for the legislators.

The Executive branch are the developers. They’re responsible for implementing the specs/laws in order to fulfill the requirements/constitution. They can decide the technical approach to use, as long as they satisfy the specs and the requirements.

The Judiciary is the QA, they review the work of the developers/executive and make sure they’re following the specs/laws and adhering to the requirements/constitution. Sometimes they even have to review the specs/laws themselves to make sure they aren’t contradicting the requirements/constitution.

Sometimes a project is so large and you need to form smaller subteams to specialize in different areas or modules. That’s called Federalism. Or maybe local government.

Sometimes one person handles specs, development and QA. That’s called dictatorship.

Sometimes people want to rewrite or amend the requirements/constitution. That’s called agile development. Or maybe scope creep.

Sometimes the bureaucracy needs to be reorganized to be more efficient. That’s called refactoring.

The big difference between government and software projects is that in software projects, team members are selected based on meritocracy. In theory at least, you choose the best people for the job. In government, who gets the job is determined by who is best at bamboozling the largest number of people to vote them in. Maybe someday we’ll get a meritocracy in government too, but until then I’m probably sticking to software development.

Web Frameworks – Open Source or Roll Your Own?

A while back I wrote about my experience coding and maintaining an in-house web framework at a previous job. It was a full-stack web framework. We had libraries for front-end Javascript up to server-side database connections. And the entire stack was tightly coupled. But while the framework was serviceable, it was almost always behind modern trends in web development. I always felt like we were playing catch-up. And as a developer I wanted to widen my horizons and try out more things. So more than once I had discussions with higher management about using open source web frameworks in some projects.
Unfortunately, our in-house web framework already had a long history and most of the devs in our company were used to it. The company had tried using a different Java-based framework stack before. It was back in the days when things like Struts, Spring, Hibernate, etc were beginning to ramp up. It kind of ended in disaster – that project ended up taking a lot of effort, had a lot of technical problems, and so on. I believe this gave the company leadership the impression that investing in other frameworks are not worth the risk and effort. It’s a form of Not Invented Here syndrome.
I admit there are some advantages to having your own in-house web framework. After all, all the popular web frameworks today started out in-house in some company and later they decided to release as open source. And many companies do fine using in-house frameworks. Using an in-house framework means full control over the behavior. And you can tailor the functionality and coding style to your internal processes. In fact it can be a good value add to your company if your in-house framework did something better or unique compared to open source ones.
But there are also significant advantages to using open source frameworks. Maybe I should have used some of these during discussions when I was there:
  • With an open source framework, we wouldn’t have to maintain the core functionality of the framework ourselves. We wouldn’t have to maintain the full stack ourselves. We would only need to maintain any customisations and extensions we write for our own needs. For our in-house web framework, every so often we’d spend some effort to come out with a new version with incremental improvements. With an open source framework, we could instead redirect that effort to higher-value work.
  • There’s more learning material available online, and they don’t have to be maintained in-house.
  • There’s a wider base of experience to draw from. For our in-house framework if you encountered a problem requiring deep framework knowledge, there were only a handful of high-level experts in the company. For an open-source web framework, that expertise is widely available on the internet through sites such as Stack Overflow.
  • Having knowledge/experience in established frameworks means your company can get more contracts. You can take on projects that use those frameworks. You don’t have to spend proposal space trying to convince clients that your in-house framework is great. With open source frameworks you can reuse marketing copy by someone else!
  • Working on well-known, established frameworks is better for your developers career-wise. It gives them more knowledge that could be transferable to other jobs. While this isn’t a benefit to the company per se, it will make the company more attractive to developers. It even allows the flexibility of hiring developers experience with that framework.

There are some disadvantages too of course:

  • As mentioned above, there is time and effort involved in learning a new open source framework. This effort is mostly for the experienced developers – for new hires they will need to learn something regardless
  • Having an in-house team developing your own framework means you have a core set of developers experienced in the full stack. Reliance on open source frameworks means most of your developers won’t be familiar with the low level details.

In the end, there’s no guarantee that using an open source framework will be painless or be better than developing one in-house. So I can understand the decision to stick with what you know. But for me as a developer, I feel that it’s more rewarding to be exposed to different frameworks.

In fact I wrote this post because recently someone asked me what my “go-to web framework” was and I said I didn’t have one. I’d rather be flexible enough to learn and work with any existing framework. In our industry where change happens quickly and can catch you by surprise, I think that flexibility is a much more valuable asset to have.

MarkLogic NoSQL

I recently attended a few training sessions for MarkLogic held at an office in a nearby business center. Now, I’ll forgive you for not knowing what MarkLogic is, as even I hadn’t heard of it before six months ago. MarkLogic is (apparently) the leading Enterprise NoSQL provider.
NoSQL is big and sexy right now because of the supposed advantages in handling big data, and large web companies like Google and Facebook use a lot of NoSQL in the backend. Most of the popular/well-known NoSQL solutions are open-source/free ones: MongoDB, Cassandra, CouchDB, and so on. But these aren’t actually very popular on the enterprise side, hence “Enterprise NoSQL” isn’t a very common phrase.
One of the reasons NoSQL isn’t currently very popular for enterprise projects is that the popular open-source solutions such as MongoDB don’t guarantee ACID transactions. In fact, MongoDB has the concept of “eventual consistency” for their distributed servers, which implies that they don’t have real-time consistency.
MarkLogic does guarantee ACID transactions, along with government-grade security. Both of these are things that enterprise clients love. So that’s the market they’re in. Their highest-profile project to-date was the backend datastore for (also known as Obamacare/ACA). That project involved consolidating data with different structures from multiple sources and needed to scale up to a ridiculously high capacity, so it seemed tailor-fit for a big data level NoSQL solution.
During the second-to-last day of the training, Jason Hunter, CTO of Marklogic for Asia-Pacific, dropped in to answer some questions. He talked a bit about how MarkLogic started and how they got him on board, and a bit about their competition (some disparagement towards MongoDB and Oracle) and about
He also had a really good sales pitch about why NoSQL is just a better approach compared to RDBMS. (Although it would have been more fun if he had given this talk with like an Oracle salesperson there to debate with him.)
One of his points was that RDBMS restrictions such as limits on column size are outdated. They were sensible in the old days when disk space was at a premium, but these days you don’t need to limit how many characters you store in a last name field. MarkLogic stores all records as documents with no file size limit (AFAIK) to avoid such an issue. From experience developing web forms, there’s the occasional client or system user who doesn’t understand why we need to have character limits on fields. We also had this system where they wanted to be able to type unlimited-length rich text content (we stored it as HTML in the backend) in memoboxes. These were fields where they had to write stuff like notes and assessments and most of their data was pages of text. I feel like that sort of thing would have been a great use for NoSQL.
Another point he raised was that RDBMS systems were very bad at allowing the user to search like three columns at random. You needed to know in advance which three columns to index. For document stores like MarkLogic, typically the entire document is indexed so that problem is avoided. Of course, that means that functions like “sorting by last name” need a bit more setup. You need to build a range index in MarkLogic to sort by a specific field. So it’s kind of a trade-off either way.
Now to be fair, Oracle does support indexed full-text search over any number of columns via Oracle Text. But it’s not the default behavior and definitely not straightforward. I used to work with Oracle Text a lot in some of my older projects, and the amount of time it took to index any nontrivial amount of data often gave us a headache.
I should do a test sometime to determine how well MarkLogic’s indexing performs. The story goes that MarkLogic started out as a document search application before they changed gears to become a database. They even approached VC’s with the intent of competing with Google. They’ve had a lot of time to get good at this, so I have high expectations.
The MarkLogic server itself is an interesting piece of engineering. It’s basically a document store, a search engine and an application server all rolled into one package. Upon installation you get some administrative web applications for configuration purposes. The admin interface seems robust and thorough. Contrast that with Oracle where you often find yourself needing to tinker around with configuration files and such
You can run web applications on the MarkLogic server itself. The supported languages are XQuery and server-side JavaScript. Odd choices I know. I suspect due to historical reasons they started out with XQuery, but the SJS side has the same capabilities (or so we’re told). If you’re not a fan of either option, you can also expose the server’s functionality via a REST interface. They also provide existing Java and Node.JS APIs on top of that REST interface. All of this means you can deploy any kind of webapp in front of MarkLogic server.
The world is moving towards bigger data stores, so it’s not unreasonable to think that NoSQL is on the way up and will be big players in the future. So I think the training was worth it (even if I did have to stay in Ortigas for a while). It’s early still. MarkLogic might still turn out to be as painful to work with as Oracle was. But at the very least it’s interesting to try a different approach to enterprise data storage. Looking forward to see what kind of applications we can build with this tech.

Google Code Jam and Competitive Programming

Back in 2004, I signed up for the Google Code Jam for the first time. Unfortunately I didn’t make it past the qualifying round.

I was a bit luckier in 2008 and 2010, making it to round 2 both times. In fact in 2008 as I recall I was one of only two participants from the Philippines who made it to round 2, which allowed me to jokingly brag about being the #2 programmer in the country.

I registered again a few times after that, and tried again last year (2016), where I only made it to round 1.

I’m no stranger to programming competitions, although the GCJ was the only one I’ve ever really participated in. Back in college, one of my friends usually joined such competitions, and I had a good time going over some of the problems with him.

Programming competitions don’t usually reflect the kind of programming done in most real-world jobs (although it might be interesting to use competitive programming problems in whiteboard interviews and see what happens). They focus on pushing the boundaries of the programmers talents in problem solving. You also need to be fairly familiar with algorithmic approaches to searching, sorting, and so on. Having a decent background in math also helps a lot.

The Google Code Jam in particular focuses on scaling up your code to handle very large amounts of data – well past the integer limit sometimes. Each question usually has a small and a large input. For the large inputs, you only have a short amount of time (I think it was 5 minutes?) to submit the output. Presumably you wrote and tested your code before starting the timer, so the short time is really only for your code’s execution. You can afford to be inefficient or maybe even brute force your way through the small input, but if you want to solve the large inputs on time you have to be able to optimize your approach.

When I first signed up, I dreamed of winning the entire thing (and of course being able to brag about being the best programmer in the world). But realistically, a lot of the people joining these competitions are “pros” more or less, so the chance of a “casual” like me winning are not that high. They are “pros” not in the sense that they do it for a living, but more of they do a lot of these competitions and regularly practice for them. Websites like TopCoder provide such competitions regularly for those who are interested. Also, I feel like I am very old and rusty, having worked too long in web development stuff where my focus is usually on a different kind of problem.

Still, I think it’s a good idea for programmers of all skill levels to give it a try, just to see where you are at. Who knows, you might even make it all the way! The next Google Code Jam starts soon – the qualification round is on Apr 7. I signed up, although I’m unsure if I will have the time this year even if I advance, due to scheduling problems. But a man can still dream, right? Who knows, maybe I’ll see some of you there too in the final round!

Fake Devs and Whiteboard Interviews

Recently, a developer needed to undergo a tech interview at US immigration:

This may surprise some people I’ve worked with, but I didn’t have formal computer science training in school. I’m not actually a computer science major. Yet I’ve worked as a software developer for more than a decade now. Literally zero times have I needed to write a sorting function or balance a BST.

I have a rudimentary understanding of some sorting algorithms (mostly just bubble sort and selection sort), and I have some idea of how to balance a BST. And if I needed to implement any of those I could probably whip something up after a few tries. (Something buggy and/or inefficient probably.) If I had to interview at a company that used one of these whiteboard interviews, there’s a good chance I wouldn’t pass.

For maybe 90% (or even as high as 99%) of modern-day programming work, it simply isn’t needed to know these things by heart. Those skills are more necessary if you’re doing low level things like writing an OS or a graphics driver or such. For everyday web development or Java or Rails or whatever, it’s an indulgence. And even for those instances that you need them, well, the internet is right there.

I’m not a fan of the idea that not knowing data structures and algorithms means you aren’t a real software developer. I have a couple of developer friends who jokingly refer to themselves as “fake devs”. It’s supposed to mean someone who doesn’t have any formal computer science background and has at best a passing familiarity with data structures, algorithms, and other computer science concepts. As one of them stated “I wouldn’t know how to make a map without using a map.” (I thought about writing a post about that, but it’s surprisingly easy – just look it up.). One of them even asked me to give them a quick crash course in data structures and algorithms because he was applying in a large and well-known software company.

For me, this term is silly. If you’re doing software development work, then you’re a software developer.

I’m not saying knowledge of these things isn’t useful. If you want to do more complicated work it is certainly useful to have these tools in your toolbox. Knowing them also makes you more flexible and expands your thinking. And certainly one should be the type of developer who is willing to seek out that knowledge if it turns out you need to use it.

But the problems with the whiteboard interviewing process isn’t that they require knowledge of these things; it’s that they require it under pressure, off the top of your head, without access to the internet. This doesn’t reflect how programming work happens in the real world, where you often have a team, an IDE, and even the entire internet to help you.

Unfortunately, it’s often far too costly for companies to evaluate you under real-world conditions, so many resort to this common interviewing process. After all, it is difficult to evaluate the performance of even developers you have been working with for months, what more someone who just whipped up a short function on a whiteboard in twenty minutes? We tried some variations on it at our last company, but for me it is a problem yet unsolved. Maybe someday…


Running Python 2.x and 3.x on Windows

I’ve been hesitant to try Python 3.x because it’s not backward compatible with Python 2.x which I’ve been using for scripting since forever. But recently I found out that since Python 3.3, they’ve included a launcher in the Windows version that supports having both versions installed.

You can use the launcher to specify the Python version to use at the command line (it defaults to whichever version was installed first):

Even better, the launcher will recognize a header in your .py files that can specify which version of python to use:

#!/usr/bin/env python3

If the launcher sees this header, it will automatically launch the appropriate python version. Handy!

Weekend Project: Twitter Trivia Bot

I had been meaning to try writing a Twitter bot for a while now. I figured a trivia bot would be pretty easy to implement, so I spent some time a couple of weekends to rig one together.

It’s (mostly) working now, the bot is active as triviastorm on Twitter, with a supporting webapp deployed on The bot tweets out a trivia question once every hour. It will then award points to the first five people who gave the correct answer. The bot will only recognize answers given as a direct reply to the tweet with the question, and only those submitted within the one hour period.

Some technical details:

My scripting language of choice for the past few years has been Python 2.7. I’m using Tweepy to interact with the Twitter API, PyMySQL to connect to the database, and Flask to run the webapp. I haven’t used Flask in some time, but it’s still very straightforward. I actually had a harder time configuring the webapp with mod_wsgi on my host.

The main problem with a trivia system is that you need a large and high-quality set of questions. Right now the bot is using a small trivia set –around a thousand questions I got from a variety of sources. If I want to leave this bot running for a while, I’m going to need a much larger trivia set. However, reviewing and collating the questions is a nontrivial task. Hopefully I can add new questions every so often.

Feel free to follow the bot and help test it out. I’d be grateful!

Integrating Open Source Libraries

There are a few things that one should consider when using and integrating an open source library into your application:

  1. What are the licensing terms for the library? There are some liberal licenses that mostly let you do anything you want. The MIT license is an example of a very permissive license. Other licenses may provide a number of restrictions. Can you integrate with closed-source software? Can you distribute binaries without the source? Do you need to put some kind of attribution somewhere in your software? Another thing to look our for are the so-called viral licenses. Viral licenses specify that if you integrate their code into your system, then the terms of that license apply to your own software as well. These can be very dangerous from the standpoint of a company developing a commercial product. The most well-known example of such copyleft licenses is the GPL. Integrating GPL code into your system will often mean your software needs to be open source as well. While more open source is good, it may not be in the best interest of your company, so tread lightly.
  2. Which version of the library will you use? Many open source libraries will release updates on a regular basis. There may be some compatibility issues depending on the version you use. For example the popular open source CMS WordPress releases regular updates. If you are integrating with WordPress, you need to decide which version you are supporting. You will also need a plan for upgrading to future versions and ensuring compatibility. (See the next item.)
  3. How will you handle bugs and updates in the version you’ve chosen? It’s happened that our system was having problems due to a bug in an open source Javascript rich text control library. After investigating, we found that a fix was available but was only applied in a future version. Upgrading to that future version would require that we test the entire system for compatibility problems. That was out of the question. I ended up having to apply a fix to our local version only. This meant that we now had a different version from the “official” released version. This may lead to further compatibility problems in the future. You should also be aware of licensing issues here too. The license may require that you submit any modifications to the community as open source contributions.
  4. Is there an active help and support system for the software? The demands of enterprise software development often need timely support. There was a time we had to use a certain open source document management system. We later discovered that it was very hard to get help on any issues encountered, either from official channels or the community. This led to many difficulties and issues in that project. You might also want to check popular programming forums such as StackOverflow. Check how many questions about that library usually remain unanswered. If the number is low, that’s a good indicator that you might have trouble looking for help later on.
  5. What dependencies does the library have? You might want to make sure the dependencies don’t have any problems with the previous items above.

The great thing about modern software development is that we have a large number of open source libraries available to use and draw inspiration from, but that doesn’t mean you can just willy-nilly pick up just any library. The above questions provide a good starting point for evaluating open source libraries for your use.