When the program works as expected, there is often no difference in the quality of the logging. However, as soon as the program fails, or you get the wrong result, you can almost immediately tell the good programmers from the bad.

Example 1: “Let’s make a debug version”

For example, a tester came to me with a call case that didn’t work. We looked at the logs, and saw that the problem seemed to be in a neighboring module. A call to the other module to get a list of values simply returned null. When we enabled logging in the neighboring module and re-ran the test case, there was no more information available. No clues to why null was returned – did we supply the wrong arguments, did some external system fail, was there an error in the neighboring module, or what?

When asking the developer responsible for the code, the answer we got was: “Oh, then we have to make a debug version to see what happens”. Fail! It must be possible to get some sense of where the problem is just from the logs. If this had been in a production system, adding a debug version would have been a lot of work. The code needs to include enough information in the log statements so you at least have some idea of why it fails.

Example 2: Show Me How We Got Here

One of our products at work finds the cheapest route for the delivery of SMS:s (text messages) to mobile phones. Depending on the current location of the phone, and the operator the destination subscriber belongs to, there are many possible routes to choose from, each with a given cost and other characteristics.  In addition, there can be exceptions that forbid some routes, and promote other routes. There are typically many thousands of routes defined. The system finds the cheapest route in each case, given the constraints, and delivers the SMS.

Now, suppose a certain SMS gets delivered using route A, but we believe it should have used route B. Why was route A chosen? If there is no logging information (other than “route A was chosen”), we are left with hundreds of possible routes, their costs, the exceptions, and a complex algorithm. Good luck figuring out why A was chosen.

In our implementation, the log includes all potential routes, in cost order. As routes are eliminated due to different restrictions, the eliminated routes are listed in the log, together with the reason for elimination. With all the information on the input to the algorithm, as well as the steps taken, listed in the log, it is fairly easy to see why a certain route was picked.

Why Not?

So, why don’t all programmers write debuggable code? I can think of three reasons:

1. You have to be humble enough to realize that there will be cases when your code doesn’t work as expected. I believe many programmers have a hard time with this.
2. If you test your own code thoroughly, you will make sure it works (or fails gracefully) in a lot of different scenarios. For each scenario, it is natural to add logging. If you don’t test those cases, you are less likely to add logging there.
3. Many programmers don’t trouble-shoot their own code in production systems often enough. If you have a problem in a live system, and the logs tell you nothing about why there is a problem, you have a strong incentive to add logging that will actually help you the next time you are in a similar situation.

Is Your Code Debuggable?

There are of course cases where even good logging messages don’t give you the full story on why something fails. You may still have to make that debug version. But your regular logging should at the very least give you some hints on what the problem might be.

So, how well have you prepared? When your code fails, will the logs tell you what’s going on?

“We have Set competitions at the school where I teach, both as individual time trials and as head-to-head matches. When many students at my school became capable of finding sets very quickly, I developed what I called C.P. Set (Complementary Pairs) to slow them down a bit and reduce the number of simultaneous calls in the head-to-head matches.  The idea is to find two pairs of cards that both require the same 3rd card to complete a Set. The completing card does not need to be on the board; only the four cards forming the complementary pairs are picked up.”

The picture below shows two pairs that have the same third card. The two cards to the left (pair 1) form a set with the top card. Likewise, the two cards to the right (pair 2) also form a set with the top card. Therefore, pair 1 and pair 2 are complementary.

At Michael’s school, they started out playing Complementary Pairs Set with 12 cards on the table, but soon realized that they could also play with only 9 cards on the table. So if the four cards in the picture above (pair 1 and pair 2) were present anywhere among the 9 cards on the table, they could be removed as complementary pairs, and be replaced by four new ones. Note that the third card (the top card in the picture) doesn’t have to be present on the table.

Michael had read my first post on Set probabilities, and was wondering what the chances are of not finding any complementary pairs when playing with 9 cards on the table. He asked if I could find the answer by modifying my simulator. With 9 cards on the table, there are 36 different possible pairs. For each pair, there is exactly one card that can form a set with it. So the question is: how often does it happen that all the 36 pairs require a different third card (of the possible 81 cards) to form a set?

It turns out that it is very rare that there are no complementary pairs present among 9 cards. When I simulated playing one million games, the odds of there not being any complementary pairs were around 12,000 to 1. Put another way, it only happened around 80 times out of a million. The odds were constant per round of play, which is not surprising, given that the third card doesn’t have to be present on the table. The Java program I used is available at BitBucket.

After Michael came up with the idea of complementary pairs, he found out that the same variation of Set had already been proposed, but under the name of Super Set. However, I think complementary pairs is a more descriptive name, so I’ll stick to that.

If you haven’t yet tried Complementary Pairs Set, give it a try. And if you can’t find any complementary pairs, look harder – the odds of there not being any are 12000:1. Happy playing!

office Interactions

The picture Jason paints of working at an office makes you wonder why we work at an office at all. Constant interruptions from bosses, colleagues and meetings make it hard to get anything substantial done. Why not have everybody work remotely all the time instead? For me, the main reason is that software development requires a lot of collaboration. Brain-storming, discussing, and refining ideas with other developers usually yield solutions much better than what you can come up with on your own. Furthermore, these kinds of interactions are much more effective in person, with a whiteboard handy (pretty much all discussions benefit from a few quick drawings). Remote collaboration tools try their best to recreate the face to face dynamics, but it is never the same as being in the same room with someone.

Jason recommends switching from active communication (face to face or phone call) to passive communication (e-mail or chat), to avoid interrupting people. This is good advice for avoiding interruptions. But after the initial request, the sooner you switch to talking, the better. Written communication works well for simple questions and answers. However, talking to someone is at least an order of magnitude more efficient when there is even a little bit of uncertainty or ambiguity . For example, if I get an e-mail with the question “Do we support feature XXX?”, most of the time the answer is not a simple yes or no. Perhaps it depends on which other features the customer has. Or we may have just developed the more capable feature YYY that they could use instead. Writing all this down takes time and effort – talking (in person or on the phone) lets you have the back and forth immediately, instead of spreading it out in writing.

Good Interruptions

While I totally agree with Jason that programmers need long stretches of uninterrupted time in order to be productive, there are cases when interruptions should be tolerated. If someone is working on a problem (a tester, a support engineer, or another developer), and has a question I know the answer to off the top of my head, I’m fine with being interrupted. Sure, my productivity takes a hit, but we should optimize for the company’s productivity, not only my productivity. If one (or several) other people in the company would be stalled, it’s better to ask me and get the answer in one minute, than to dig around for an answer in the code or in the documentation.

Another case where interruptions are acceptable is when there is a trouble report. In my team at work, we have virtually no trouble report back-log. We try to finish off trouble reports as soon as we get them. Getting back to the customer right away has several benefits: the problem is fresh in their minds, it’s easy to get additional information if needed, and the customer sees that we care about fixing the problem quickly.  The quick turn-around time means that no back-log of unanswered trouble reports builds up. It is also easy to stay on top of what the outstanding problems are at all times, since there are so few of them.

Meetings – Inefficient use of time

When it comes to meetings, I agree with Jason: the fewer the better. Having a meeting is often very poor use of time. When I worked at Ericsson, we had a weekly status meeting for the whole R&D department on Tuesdays at 10 o’clock. So at 9.30, you don’t feel like starting anything new, since you know you are going to be interrupted soon. Knowing that you will have to attend a meeting soon causes you to get less done. An unplanned interruption is much better – at least you don’t know about it beforehand. Another problem with those status meetings was that pretty much all information given could just as easily have been sent out in an e-mail.

Another reason meetings are inefficient is that often the people there aren’t needed for the whole meeting, so some of their time is wasted. At Symsoft, I have very few meetings. Instead of calling a meeting, I’ll walk over to the people involved (all developers are a maximum of 5 offices away) and discuss the issue. No meeting, and only the people directly involved are interrupted. Way better than the overhead of a meeting.

It’s not that hard

The work place Jason describes in his talk is pretty bleak. But it doesn’t have to be that way. I have never worked for a manager that came around frequently for “status updates”, probably because they have all been programmers too, and know not to interrupt. I hardly have any meetings at all at Symsoft, and it was the same when I worked at Tilgin. Creating a good environment for software developers isn’t that hard. I do get interrupted, but almost always for the right reasons. I become less productive, but we become more productive. And I still have plenty of quiet, uninterrupted time for developing new features and fixing bugs. How about you?

The product is software. I like the programs I work on to be the main business of the company. This rules out working in an IT-department, since their job only supports the real business (whatever it is) indirectly. I also like to work on the central parts of the system – the more important it is, the better. If my parts stop working, it should immediately become an emergency issue. Finally, I don’t want to merely configure, adapt and glue together software from other companies – I want to write significant chunks of functionality myself.

Great colleagues. It is very stimulating to work with smart programmers who are passionate about software development. Time and again I see how discussing a problem or a design with a colleague leads to a solution that is better than either of us would have come up with by ourselves. Not only does it lead to better code, the process itself is also very enjoyable.

How do you know if someone is a good programmer? A very good sign is if they keep learning and improving their skills, for example by reading books and blogs, taking courses, and going to conferences. It is not a necessary condition though; I’ve worked with plenty of really great developers that don’t. Finally, good developers tend to attract other good developers, because of the reasons above. The fact that a company has many great developers makes it easier to recruit more.

Challenging problems. Programmers like solving problems with code. There should be at least  some aspect of the product that requires clever solutions, be it requirements on low latency, many concurrent requests, or limited hardware resources. However, a lot of productions software consists of regular code without any particularly difficult parts. So you should not expect to only work on “hard problems” and shun everything else. Besides, it is a big challenge to organize even the boring code in a way that makes it easy to understand and maintain.

Cool technology. This is mainly about using interesting programming languages (for example Clojure, Erlang or Go), but also includes frameworks and applications (for example Hadoop or Cassandra). This is one area where a company may have a problem. If their application is written in a certain language (say C++), it won’t change. So if you want to change to using some new language, you pretty much have to change jobs. For example, if you want to work with Erlang in Stockholm, you could try Klarna or Campanja.

Users. One of the joys of coding is making something that is useful to others. Making something that nobody uses is boring. Having users (the more the better) focuses the development effort and gives valuable feedback. The only exception would be a start-up, but then the overriding priority must be to get users as soon as possible.

Good salary. Companies that have a lot of good developers know the value of great people. Since the variation between great and average programmers is big, it makes economic sense to pay for quality developers – the variation in productivity is much greater than the variation in salary. On the flip side, companies that don’t pay their programmers well tend to be the companies that view programmers as interchangeable “resources”. Those are the companies you want to avoid for other reasons as well, not just for the low salary.

Good tools. This is almost self-evident. Having a fast computer and several monitors speeds up development – who’s against that? (OK, pointy-haired bosses that only see the cost, not the benefit, would be against).

40 hours a week. If you constantly have to work overtime to ship, something is wrong in the organization. Besides, working long hours don’t equate with being productive.

Minimal bureaucracy. For the development process, this is more or less fixed with agile development methodologies, which seem almost universally adopted. General administrative overload is mostly a problem at larger companies in my experience.

Working from home. It’s handy to be able to work from home sometimes, but it is not high on my list. I like being at the office and interacting with people. I’ve worked with a remote office using video conferencing, chat and e-mail, but it didn’t come close to the kind of productivity you get from being co-located.

Short commute. Obviously hard to influence, but not spending hours each day in traffic is really great.

Comments

If you are a consultant, some things on the list change. I’ve always liked working for product companies, mostly because I prefer to really get to know a system deeply and seeing it evolve over time. So I don’t have any first-hand knowledge of working as a consultant, but my take is this. As a consultant, it is much easier to get exposed to cool new technology, since you have a chance to work with many different clients. However, even if you have great colleagues, you probably won’t work with them day to day, since you may be with different customers.

So this is the list, in order of importance, of what I look for in a company. In real life there are always some compromises, but the higher up something is on the list, the less willing I am to compromise on it. What are your priorities?

The main theme of part 1 was the divide and conquer paradigm. In the second part the main themes were greedy algorithms, dynamic programming and NP-Complete problems. The lectures were excellent, with clear and easy to follow algorithm development and proofs. At six weeks, it was one week longer than part 1, and I found it quite a bit harder than part 1. Here’s more on each part.

Greedy Algorithms

In the greedy paradigm, you iteratively make the best possible choice, and hope that in the end you have an optimal solution. Usually the algorithms are easy to come up with, and easy to analyze, but it is equally easy to come up with an incorrect algorithm. The greedy algorithms covered were:

• Job scheduling
• Minimum spanning tree 
• Clustering
• Huffman coding

For job scheduling, a number of jobs, each with a weight and duration, are to be ordered so that the weighted sum of the completion times is as small as possible. The first programming assignment dealt with job scheduling. 10,000 jobs were to be scheduled in two different ways, and the answers were the sums of the completion times. Straight-forward, and the easiest of the programming assignments.

A minimum spanning tree (MST) connects all the nodes of a graph together as cheaply as possible. Two algorithms were covered: Prim’s, where you start with an arbitrary node, and expand with the cheapest available edge, and Kruskal’s, where you sort the edges in cost order, and keep using the cheapest ones (while avoiding cycles) until the minimum spanning tree is complete. For Kruskal’s algorithm, the Union Find data structure was introduced. It is needed to be able to quickly determine if a cycle would be created. It’s analogous to how Dijkstra’s shortest path algorithm uses a heap. I hadn’t seen the Union Find data structure before, but it is quite neat – both simple and easy to implement.

In clustering, the objective is to collect “related” nodes of a graph together in groups. This is a special case of Kruskal’s algorithm, but you stop before the clusters are merged into one big group. There were programming assignments to calculate the cost of the MST of a 500 nodes graph, and to calculate the maximum spacing of a 4-clustering of another graph. Both were straight-forward.

Huffman coding is a way of assigning binary code words to symbols of an alphabet, considering the probability of each symbol occurring, so that the resulting encoding is as short as possible. I learnt about Huffman coding when I went to university, and I still remember the idea, so I skipped this part.

Dynamic Programming

This was the most interesting part of the course for me. I had heard about dynamic programming before, but I had never really looked into it. If you can express the optimal solution in terms of the solution to smaller sub-problems, then dynamic programming may be suitable. The algorithms covered were:

• Independent set of maximum weight
• Knapsack problem
• Sequence alignment
• Optimal binary search trees
• Bellman-Ford single-source shortest path
• Floyd-Warshall all pairs shortest path
• Johnson’s all pairs shortest path

The independent set problem was used to illustrate how dynamic programming works. Given a path of nodes with non-negative weights, pick a subset of non-adjacent nodes with the maximum total weight. For the simple 4-nodes example with weights 1, 4, 5 and 4 (with maximum weight 4 + 4 = 8), we saw how both a greedy approach and a divide and conquer approach failed.

Then the optimal solution is expressed in terms of two smaller paths (without the last one or two nodes respectively). A recursive solution would lead to an exponential number of sub-problems, but by caching the result of already solved sub-problems (“memoization”), the time-complexity becomes O(n). Finally the solution is reformulated as a bottom up solution, where a table is created where the solutions to the sub-problems is filled in. I thought this was a great introduction to dynamic programming – well-reasoned, clear and easy to follow.

Next was the classic knapsack problem (I knew about it, but had never attempted to solve it). Given n items, each with a value and a size, and a knapsack of size W, find a subset of items that can fit in the knapsack that maximizes the total value of the items. Programming assignment 3 deals with the knapsack problem. In the first part, there are 100 items, and the knapsack capacity is 10,000. This can be solved with the bottom up approach of filling in a 2-dimensional array with the solutions to the sub-problems. In the second part, there are 500 items, and the knapsack capacity is 2,000,000. In this case the table approach would take up too much memory. Instead it can be solved recursively with caching of intermediate results.

The next application of dynamic programming was the really cool algorithm for sequence alignment. Given two strings (for example DNA), how can we make them align so that we minimize the cost, where there is a cost for each inserted gap and mismatched letter. For example, the strings AGGGCT and AGGCA can be made to align by inserting a gap in the second string so it becomes AGG_CA. The cost in this case is one gap, and one mismatch (the T and A at the end). For long strings (thousands of letters), a brute force solution is staggeringly time consuming. Fortunately, with dynamic programming, it is possible to express the solution in terms of the solution to sub-problems, and the running time is O(mn), where m and n are the lengths of the two strings. Very cool!

For optimal binary search trees, given n items, each with a given probability of being selected, construct the binary search tree that will minimize the average search time for the items. This is similar to Huffman codes in that they both output a binary tree, and they both try to minimize the average depth. However, in Huffman codes, the items are only leaves in the tree, whereas here the items are both nodes and leaves. I didn’t find this algorithm that interesting, since if you have a balanced search tree (albeit not optimal), the performance will still be quite good.

Next were three different algorithms for finding shortest paths in directed graphs. The Bellman-Ford algorithm finds the shortest path from a source node to all other nodes in the graph, and is developed by using the dynamic programming idea of building the solution from the solution to sub-problems. In this case, the sub-problems consist of the shortest path using only a fixed number of edges. The algorithm works for negative edge weights, unlike Dijkstra’s algorithm (covered in part 1).

Floyd-Warshall’s algorithm computes all pairs shortest paths in a directed graph. The key trick in its development is to let the sub-problem consist of the shortest path limited to using a certain sub-set of the edges. In this way, the dynamic programming solution is found. It is built using a 3-dimentional array, where each axis has the number of nodes. In programming assignment 4, you had to find the shortest of all the shortest paths in three given 1000-node graphs. If there were any negative cycles in the graph, this had to be indicated. This assignment was a little bit trickier, since you have to make sure you don’t run out of memory (the 3-dimensional array contains a billion entries).

The final graph algorithm was Johnson’s algorithm for finding all pairs shortest paths. It uses a clever technique to re-weight the edges to make sure there are no negative edges, and then uses Dijkstra’s algorithm to find the shortest paths.

NP-Complete Problems and Ways to Deal With Them

We covered NP-complete problems when I went to university, so this wasn’t new to me, but it was good with a refresher. For a lot of problems, like the travelling salesman problem, there are no known algorithms that run in polynomial time. So we are stuck with solutions with exponential running time, meaning that only problems with very small input sizes can be solved optimally. However, all is not lost. Various ways to deal with NP-complete problems were covered.

The following topics, algorithms and techniques were covered:

• P, NP, and what they mean
• Reductions between problems
• NP-complete problems
• The P vs. NP problem
• Vertex cover
• Travelling salesman
• Approximate solutions to the knapsack problem
• Local search
• Papadimitriou’s 2SAT algorithm

The first part of this section of the course defined the concepts of P, NP, NP-hard and NP-complete problems. Next, the reductions between NP-complete problems was covered, and the question of whether P = NP. After the theory had been established, various ways of dealing with NP-complete problems were presented.

It is still possible to solve the NP-complete problems for small inputs using brute force. It may also be possible to find special cases that are of interest and can be solved more quickly. Yet another way is to find algorithms that are better than brute force search, but still not polynomial. An example of this was an algorithm for the travelling salesman (more on this below). Otherwise we may have to turn to non-optimal solutions. An example algorithm was developed for the general knapsack problem, where it is possible to come within a given fraction of the optimal solution (the closer to the optimal solution, the longer the running time).

One meta-algorithm covered was local search. The idea is to start with a randomly generated solution that is most likely not optimal. Then we iteratively try to improve on that solution by making local changes until we can’t improve anymore. Then we start with a new random solution and repeat the process. There is obviously the risk of finding local maxima, while still not finding a global maximum.

For programming assignment 5, we had to find the shortest path for a travelling salesman problem with 25 cities to visit. This was the hardest of the programming problems. The algorithm that was developed in the lecture videos used dynamic programming to improve the running time from O(n!) to O(n^2×2^n), so it is not polynomial. The algorithm, while reasonable easy to understand conceptually, takes quite a bit of work to implement correctly. However, the biggest problem with the solution wasn’t running time, but memory usage. I started out using the Java class BitSet to keep track of which cities to include. But I ran out of memory and changed to integers instead. I also changed from pre-calculating all integers with x number of bits set, and instead calculated them on the fly. With these memory optimizations, I was finally able to run the algorithm, but it took quite a bit of work.

The last algorithm covered in the class was a clever randomized local search algorithm for the 2SAT problem. In the 2SAT problem (which I had never heard of before, even though it is famous), the task is to find boolean values for n boolean variables used in clauses of the form (x or y). A number of these clauses are basically and:ed together, and the task is to determine if there is an assignment of true or false to all the variables that makes the whole expression true (there may not be such an assignment). The algorithm covered was Papadimitriou’s algorithm, which uses the idea of a random walk to try to find an assignment of the variables that makes the expression true. A clever idea, and a nice analysis of the properties of the algorithm.

The final programming problem was to solve the 2SAT problem. One way would have been to use Papadimitiou’s algorithm, but I chose to use the program I developed in part 1 of the course for finding strongly connected components (SCC) instead. By (again very cleverly) transforming the 2SAT expression to a graph, it was possible to check if it is solvable by checking if a given boolean variable and its negation are both present in the same SCC. If so, the 2SAT expression can not be solved. This programming problem was harder than the first ones, but not as hard as the travelling salesman one.

Conclusion

I really enjoyed this course. It started in the beginning of December, had a break over Christmas, and continued in January. Since I am always very busy around Christmas and New Year’s, I wasn’t able to start the course until the second week of January. This meant that I had to work quite hard to complete it in the time left. But even if that had not been the case, the course was still a lot harder than part 1 I thought. Especially the quizzes, but also the programming assignments. I skipped part 2 of programming assignment 2 (clustering in a bigger graph), but completed all the other programming assignments, all the quizzes, and the final exam. Since I started late, my score on the first three weeks’ assignments was reduced by 50%. My final score was 67% (80% without reductions), and the cut-off for getting a statement of accomplishment was 70%, so no certificate for me this time.

Regardless of the score, I learnt a lot by taking this course. It took a lot of time and effort, by I have definitely improved my algorithm knowledge and skill. I can really recommend both parts of the course to anyone wanting to learn more about algorithms and problem solving.

Here’s an excerpt:

About 55,000 tourists visit Liechtenstein every year. This blog was viewed about 170,000 times in 2012. If it were Liechtenstein, it would take about 3 years for that many people to see it. Your blog had more visits than a small country in Europe!

Click here to see the complete report.

First I talked about challenges with large-scale software development. Then I listed several development practices used to cope with these challenges. I went on to present ways to become a better programmer, and ended with some fun facts from work.

Characteristics of Production Software

Programs are BIG. The main characteristic of software used in live systems is the size. For example, our main repository at work contains 1.8 million lines of Java (I used the tool cloc to find the number of lines). The sheer size of the programs complicates software development, and a lot of practices have been developed to deal with it.

Software is never done. Software that is used keeps growing and evolving. The customers find more and more uses for it, and want more and more features. Software that is not used is discontinued, but successful software is developed continuously for many years. Several developers are almost always involved. For example, I checked the subversion log of one of the main classes in our SMS application. There were 150 check-ins over 7 years, by 8 different developers (2 of which don’t work at Symsoft anymore).

Complexity from aggregation. Most features in production software are quite simple, but because there are so many of them, you get subtle (or not so subtle) interactions between them causing bugs. The complexity of the system comes from the aggregation of many simple parts, not from any complex parts.

Reading code. A consequence of the characteristics mentioned above is that reading code is a very important skill. Before a program can be modified, you need to understand what it does, and how it does it. Only then can new functionality be added so it fits in with the existing structure, and without breaking anything.  Reading and understanding a program can be a major effort, and one sign of a well-designed program is that it is relatively straight-forward to modify it.

How to Manage

Many techniques have been developed to make developing and maintaining large programs easier. Here are the most important.

Modularize. This is an obvious first step, and (I believe) universally used. The software is split into subsystems, layers or modules so that smaller chunks of functionality can be dealt with at a time.

Iterate. Developing software bit by bit is as helpful for small 30-lines scripts as it is for systems with millions of lines of code. I like the following quote:

“A complex system that works is invariably found to have evolved from a simple system that worked.” – John Gall

Self-documenting code. The naming of the classes, methods and variables is incredibly important. The names, when chosen well, let you understand what the program does just by reading them. They also greatly reduce the need for comments in the code.

No duplication. Code-duplication causes a lot of problems when you come back to modify the program (happens all the time) – you may forget to make the intended change in all the duplicates. So instead of copy-pasting, combine the logic into one method. It makes the code more compact, and easier to modify in the future. There is an excellent article by Martin Fowler, Avoiding Repetition, on this subject. Unfortunately, I have seen a lot of duplicated code in production software over the years.

Unit testing. Unit testing (JUnit etc.) is useful both in small-scale and large-scale development. It’s an easy way to ensure your smallest program parts work as expected, and you get repeatable tests that can be run again and again. Making sure your code can be unit-tested also automatically improves the structure of the code – it becomes less monolithic, more de-coupled.

Version control. Using version control (like git or subversion) is a no-brainer, and as far as I can tell pretty much always used in professional software development. Version control systems are used both to keep track of different working versions of the software, and for knowing exactly what code is included in each release.

Version control is also useful on an individual level, for example for the programming assignments at a university course. Any program I spend more than 10 minutes writing, I stick in a local git repository. That way, I can always go back to previous (working) versions of my program.

Write for people first, computer second. The code you write will be read many times in the future (by you, or another developer). The computer doesn’t care how the code is written, so make it as easy as possible to understand for the next person that has to read it. A corollary to this is: don’t be too clever. It’s better to be clear than to be clever. The following quote puts it another way:

“It’s OK to figure out murder mysteries, but you shouldn’t need to figure out code. You should be able to read it.” – Steve McConnell

Plan for failure – logging and error handling. What do you do when the program doesn’t work as expected? You need some way of seeing what is going on. Usually this is in the form of tracing or logging. This will inevitably happen, so you might as well build in support for tracing and logging from the start (and make it possible to activate and deactivate while the program is running). It’s a similar story with error handling; put in place a unified way of handling errors in the program, because errors will happen.

Issue tracking. For any real system, it is necessary to have a way to keep track of bug reports - e-mails don’t cut it. At work we use Jira, but there are many products with similar capabilities. Often, the same system is also used to keep track of new features to be implemented.

Becoming a better programmer

There is almost no limit to what you can do to improve as a programmer. Here are some basic tips, appropriate if you are taking a software development course at university, but want to progress beyond learning the content of the course.

Program! The best way to learn to program is to actually program. Reading a book or listening to a lecture can make it seem like you have learnt and understood the concepts, but it is not until you actually start writing your own programs that you really learn. This is why (in my opinion) the best programming courses have lots of programming assignments – that’s when you are forced to put the theory into practice.

But you don’t have to stop at just doing the assignments. Solve some other problems with programs as well. If you don’t know what to solve, you can look for good problems at Programming Praxis, Ruby Quiz (the problems don’t all have to be solved in Ruby), or Project Euler (quite mathematical).

Learn a scripting language. Being able to write small scripts to automate tasks on the command line or to filter out information from a log file containing 100,000 lines is quite useful. Make sure to learn how to use regular expressions, and a language like Ruby, Python or Perl. All those languages are also used for “serious” applications, but even just using them for quick shell scripts is worthwhile.

Learn an IDE and a text editor well. For Java development it’s really worthwhile to learn to use an Integrated Development Environment (IDE), such as IntelliJ IDEA or Eclipse, really well (as I have written about before). In addition, it is always useful to know a good text editor, for example Emacs or Vim. In both cases (IDE and editor), learn as many keyboard shortcuts as you can. The goal is to be able to go from thought to program as effectively as possible.

Books. The number one book that every programmer should read is Code Complete by Steve McConnell. It’s always on top of every list of the best programming books, and for good reason (I’ve reviewed it on Amazon). It has a lot to say about how to write code (in a language-neutral way). The next book to read is The Pragmatic Programmer by Andrew Hunt and David Thomas. It contains a lot of tips on how to develop software efficiently.

Hacker News and Proggit. There are always interesting articles related to programming at both Hacker News and reddit/r/programming. Just remember to not spend too much time there – program instead. But for inspiration these sites are great.

Fun Facts

I also added a few fun facts about how we develop software at Symsoft. All of these are not necessarily true everywhere, but I wanted to add in a few bits of trivia to give a better sense of what it can be like to work.

HashMap and ArrayList. The two Java data structures we use the most in our code.

English. We’re in Sweden, but everything is in English – the code, documentation, bug reports etc. No big surprise, I just wanted to make sure everybody knows.

People interactions. Even if your job description is coding, there is quite a bit of interaction with other people. Mostly discussing designs and bugs with colleagues, but also contact with product management and sales, and with customers. There is no such thing as a solitary coder.

Time for bug-fixing. We schedule around 30% of the developers’ time for bug-fixing. This includes the time for investigating reported problems, fixing the bugs and testing the solution. It is worth noting that many reported problems aren’t actual bugs. Sometimes the system is not configured properly, sometimes the problem is in a surrounding system, and sometimes the bug report is actually a request for a new feature.

No UML. We don’t use UML, but there are lots of whiteboards around. We use them all the time when discussing a solution to a problem, and when describing how the system works.

The talk took about 45 minutes to deliver. It was interesting to prepare and fun to give. What is your opinion and experience on what it is like to work as a professional software developer? Let me know in the comments.

Each bug can teach you something

Feedback is one key to developing better products. It’s one of the primary tenets of agile development. Unit testing and iterative development are both techniques to provide feedback faster. With unit testing you get feedback on whether the code works, and with each delivered release you can hear what the customer thinks of the new features. A bug report is another form of feedback on your code.

There can be many different causes of a bug. Some possibilities are: a simple coding mistake (like a nested if-statement where you end up in the wrong branch), or a faulty assumption on your part (maybe the incoming messages don’t always have certain fields present, so you got a null pointer exception), or there is a missing requirement (you should have answered the message in a different way if a given parameter is present), or the customer is using the software in an unanticipated (but correct) way, leading to bugs.

In each of these cases, you can learn something about how to code, about your product, or about the domain it operates in.

For example, when I developed VoIP products at Tilgin, there was a case where we received a faulty message that caused our software to loop indefinitely. The messages contained elements encoded using tag-length-value (TLV) parameters, where the length value was the total length of the element. This way you can skip unused or unknown elements by jumping forward “length” number of bytes. In this case, the length value was zero, so after the skip we pointed to the same element we pointed to before the skip, causing an infinite loop.

Before this bug, my code carefully checked length values for too big values that would cause a read past the end of the message buffer. However, up until then, it had never occurred to me that a length of zero could be just as bad.

Your Own Code Becomes Easier To Debug

When you spend time trouble-shooting problems and fixing bugs, it doesn’t take long until you want to make your own code as easy as possible to debug. It is frustrating not having all available information presented.

One extremely common problem is exceptions that don’t include dynamic information. For example, suppose there is code that expects a value to be in the range 0 – 20. How many times have you seen an exception that just says ”Illegal value”? That doesn’t tell you much if you are trying to find a bug. If for instance 21 was received, it should say something like ”Illegal value: 21, not in range 0 – 20″.

It helps to include the allowed range, and it definitely helps to include the current value. The current value could be 21, or -128, or 65535. All of those could give you a clue as to what caused it, which you don’t get from a plain ”Illegal value”.

Even Steve McConnell breaks this rule in some places in the excellent book Code Complete. For example, in chapter 15 there is an example where an unexpected type of character is detected, but the error message doesn’t include the character in question.

Every time you find and fix a bug, you need to ask yourself: is there anything in my code I should do differently in order to eliminate bugs like this in the future? Is there anything I should be doing to make trouble-shooting this kind of bug easier in the future? This is very fertile ground for improvements.

Both You And the Customer will be Happy

As I mentioned in Why I Love Coding, one of the joys of programming is making something that is useful to other people. You get the same kind of kick out of fixing a bug, but on a different time scale. Delivering new features usually takes a while, but a bug-fix can be done in an hour. Each fixed bug makes you feel you are accomplishing something, and that’s a great feeling.

It’s a bit of a paradox that fixing a bug will make the customer happy. If there wasn’t a bug in the first place, there wouldn’t be a need to fix it, so why should they be happy? However, my experience is that they are happy to receive a bug-fix, especially if it is solved quickly. Everybody knows that there will always be bugs. What matters is that somebody is ready to fix them quickly when they are discovered.

Solving Problems is Fun

Many programmers enjoy solving problems, like mathematical puzzles, programming challenges, Sudoku or crosswords. Even reading murder mysteries feed into this: you look at the clues and try figure out how it happened.

Debugging and fixing bugs is the same. Each bug is a new mystery to figure out. Often your first reaction when seeing a new bug report is: that’s impossible? How could that happen? That’s when you start looking for clues. What do the logs say? Any error reports from the system? What else happened in the system at this time? Was anything changed recently – new software, configuration changes, traffic disturbances? Let the figuring-out begin!

These are four reasons I like debugging and bug-fixing so much. What is your experience?

It’s quite an interesting book, and it makes some great points about the future of testing. However, despite the phrase “Help me test like Google” on the cover, it is not as useful as I had hoped when it comes to improving your own testing.The book starts off by describing the key roles at Google: SWE (Software Engineer), SET (Software Engineer in Test) and TE (Test Engineer). Briefly, the SWE builds features for Google’s products, the SET develops testing infrastructure and larger-scale automatic tests, and the TE tests the products from a user’s perspective. After the introductory chapter, there is a chapter each on the SET and TE roles, and there is also a chapter on the TEM (Test Engineer Manager) role. The final chapter is about the future of testing at Google (and in general).

Software Engineer in Test (SET)

As the different roles are explained in the respective chapters, there is also quite a bit of detail on how the testing is done at Google. The most interesting part in the chapter on the SET role is the part about the infrastructure. There is (of course) extensive support for running tests automatically. There is common infrastructure for compilation, execution, analysis, storage and results reporting of tests. Tests are categorized as small, medium, large or enormous. Small tests are basically unit tests where everything external is mocked out, and they are expected to execute in less than 100 ms.

Medium tests involve external subsystems, and can use database access, but generally run on one machine (use no network services), and are expected to run in under a second. Large and enormous tests run a complete application, including all external systems needed. They can be nondeterministic because of the complexity, and they are expected to complete in 15 minutes and 1 hour respectively. A good way to summarize them is that small tests lead to code quality, and medium, large and enormous tests lead to product quality. The common test execution environment for running the tests has been developed over time, and has several nice features. It will automatically kill tests that take too long to run (thus the time limits mentioned above).

It has several features to facilitate running many different test concurrently on a machine – it’s possible to request an unused port to bind to (instead of a hardcoded port number that could clash with another test), writing to the file system can be done to a temporary location unique to the current test, and private database instance can be created and used to avoid cross talk from other tests. Further, their continuous integration system uses dependency analysis to run only tests affected by a certain change, thus being able to pinpoint exactly which change broke a certain test. This system has been developed by Google for many years, and has become quite capable and tailored to their way of working.

Test Engineer (TE)

The most interesting part in the TE chapter is the description of the process used for developing the test plan for a product. The test plan’s purpose is to map out what needs to be tested for the product, and when it is done it should be clear what test cases are needed. It can be a challenge to find the right level of detail for a test plan, but it seems like they have found a good balance at Google.

The Google process for coming up with the test plan is called ACC, which stands for Attribute, Component and Capability. Attributes are the qualities of the product, the key selling points that will get the people to use the product. The examples given for Chrome include fast, secure and stable. There won’t typically be that many attributes.

Next, the Components are the major subsystems of the product, around 10 seems to be a reasonable number to include. Finally there are the Capabilities, which are the actions the system can perform for the user. Whereas there are relatively few attributes and components, there can be quite a number of capabilities. The capabilities lie at the intersection of attributes and components. It is natural to create a matrix with attributes along one axis, and components along the other axis. Then each capability will fit in at the given coordinates. A key property for a capability is that it is testable, and each capability will lead to one or more test cases to verify its functionality. Thus the matrix is an aid in enumerating all the test cases that are needed.

The matrix allows you to look at what capabilities affect a certain module. If you look along the other dimension, you will see all capabilities supporting a certain attribute. The matrix is also useful in risk analysis, and when tracking testing progress.

In the same chapter, there is also a good story about a 10-minute test plan. James Whittaker did an experiment where he forced people to come up with a test plan for a product in 10 minutes. The idea was to boil it down to the absolute essentials, without any fluff, but still being useful. Because of the time constraint, most people just made lists or grids – no paragraphs of text. In his opinion (and I agree), this is the most useful level – it is quick to come up with and doesn’t need a lot of busy-work filling out sections in a document template, and still it’s a useful basis for coming up with test cases. The common theme in all cases was that people based the plan on capabilities that needed testing.

Tools

There are other interesting testing tools described in the book too. One such tool developed at Google is BITE – Browser Integrated Test Environment. When testing a browser-based app, like Google Maps, and something went wrong, there was a lot of information to extract and put into the bug report. For example, what actions lead up to the bug, what version of the software was running, how the bug manifest itself etc. The BITE browser extension keeps track of all the actions the tester made in the application, and supports filing a bug report by automatically including all the relevant information. It also has support for easily marking in a screen shot where the bug appeared.

Another interesting tool is Bots. It involves automatic tests where many different versions of Chrome fetch the top 500 webpages on the web. The resulting HTML is compared and detailed “diff”-reports are produced.

Tips

There was also a sprinkling of interesting ideas (that can definitely be of use in any test organization) throughout the book. Here are the ones that stuck in my head: When asking people to estimate a value for something (for example the frequency of a certain failure scenario), use an even number of values (e.g. rarely, seldom, occasionally, often). That way you can’t just pick the middle value – you’re forced to think about it more carefully.

Another example in the same area. If you want people’s opinion of how likely a certain failure scenario is, you could just ask them about it. But another technique is to assign a value yourself, and then ask what they think. Then you have given them something to argue against. Often, people have an easier time to say what something isn’t, then what it is.

There is also a quote from Larry Page that is referred to several times in the book (for example regarding the relatively few testers at Google) “Scarcity brings clarity”, and (later on), Scarcity is the precursor of optimization. Worth thinking about.

As well as describing how the testing is done, and which tools are used, there are also a number of interviews with various people in the test organization. The chapter on TEM (Test Engineer Manager) in particular consists almost entirely of interviews, 8 in total. Most interviews in the book were interesting to read, but many of them weren’t that useful in terms of tips or ideas to use in your own testing.

The Future of Testing

For me, the best chapter in the book was chapter 5, “Improving How Google Tests Software”. It is the last and shortest chapter, only 7 pages. In it, James Whittaker shares some profound insights about testing at Google, and testing in general. One of the flaws he sees with testing is that testers are… testers. They are not part of the product development team. Instead, they exist in their own organization, and this separation of testers and developers gives the impression that testing is not part of the product; it’s somebody else’s responsibility. Further, the focus of testing is often the testing activities and artifacts (the test cases, the bug reports etc.), not the product being tested. But customers don’t care about testing per se, they care about products.

Finally, a lot of the testing mindset we have today developed in a different era. When you released a product, that was it. There was no easy way to upgrade it, and users had to live with whatever bugs slipped through. However, these days so much of the software can be fixed and upgraded without a lot of fuss. In this environment, it makes less sense to have testers act as users and try to discover what bugs they might run into. Instead, you can release the software, and see what bugs the actual users encounter. Then you make sure these bugs are fixed and that the new release is pushed out quickly.

So his opinion is that testing should be the responsibility of all the developers working on the product. It should be their responsibility to test the product and to develop the appropriate tools (with some exceptions, for instance security testing). Whether you agree or disagree with this, it is definitely food for thought!

Conclusion

Initially, when I had just finished reading the book, I felt a little disappointed. It was interesting to read, but there didn’t seem to be that much to take away from it and apply to your own testing. Pretty much all of the techniques and tools are tailored for Google and their needs, which is just as it should be. But that means that they may not be applicable to your own situation.

However, as I am going through it again while writing this review, I realize that there are quite a few good ideas in it – they just have to be adapted to your specific situation. So while not directly applicable, the ideas in the book serve as inspirations for how testing can be organized and executed.

5. People Interaction

Programming seemed like quite a solitary job – a feature needs to be done, so you sit down at your computer and code it up. The truth is there is quite a lot of interaction with other people. You discuss your designs with your colleagues, you are in meetings reviewing new features, and you talk to the testers testing your code.

It really helps to be tactful and diplomatic during these interactions; if this doesn’t come naturally, learn how. One of the best books on the subject is How To Win Friends and Influence Peopleby Dale Carnegie. If you haven’t read it, I really recommend that you do.

4. Writing Matters

It helps a lot to be able to write clearly in order to get your points across. To some extent, coding and writing are quite similar. In both cases you need to express your ideas clearly and unambiguously in a structured way. There are of course lots of e-mails that need to be written, but there is also documentation of the features you have developed, describing bugs in bug reports so that it is clear what the problem is, as well as writing good explanations for bugs you have fixed. There wasn’t much emphasis on writing at university, but being able to write well definitely is an asset at work.

3. Software is Never Done

Before I started working, I thought that you developed a feature, and then you were done with it. However, in reality you quite often come back to features. Maybe it was not exactly what the customer wanted, or you are adding more functionality to it, or similar funtionality so you want to combine it, or you are fixing a bug in it. One way or another, you often return to code you wrote before.

I didn’t really understand either that new features are almost always inserted into existing code. At university, we always developed programs from scratch, but that is almost never the case in the real world. Sure, you create new functionality, but it always has to fit in to what is already there. Therefore, a large part of creating a new feature is understanding the existing code in order for the new feature to fit in. This is something we never practiced at school.

2. Few Clever Algorithms

At university I did an M.Sc. in Computer Science and Engineering. I studied communication systems, which included signal processing, error correcting codes, queuing theory and so on. We also had core computer science courses like algorithms and data structures, and I loved all of it. I thought it was really cool to be taught all these clever algorithms and data structures, and I expected to see them in use at work.

My first job was as a software developer at Ericsson in Montreal, working with the mobile switching center that handles calls in a cellular network. There was a lot of code controlling call set-up, hand-offs, roaming etc, but I was pretty disappointed to see that it was all done with quite basic data structures and algorithms. The most interesting part I found was the code keeping track of roaming subscribers currently in the system. It consisted of one thousand binary trees, where the last three digits of the subscriber number determined which tree a given subscriber belonged to. To find a subscriber, you picked the tree based on the last three digits of the number, then traversed the tree to find the subscriber. Apart from that, it was pretty much only linked lists or simpler.

1. Complexity from Aggregation

So, given that there were no clever algorithms in use, and that the whole application was only using pretty basic data structures, it seemed at first that there wouldn’t be many challenges working on the system. Wrong! I quickly realized that the system is hugely complicated, not with complicated features, but in having lots and lots of simple features aggregated together. This is something I have seen in all systems I have worked with. Most features are dead simple, but because there are so many of them, you get subtle (or not so subtle) interactions between them causing bugs. The complexity of the system comes from the aggregation of many simple parts, not from any complex parts.