Proprietary Services Make Your Software Worse

This idea has been bubbling around in my head for a while now, and I finally had some impetus to write it up. I'm sure this is a conclusion that other people have come to as well, but for a while I was struggling to find the right words. The general idea here is that certain licensing practices for services like databases, PKI, VPNs, or authentication services that you may need to integrate with at work have licensing models that make it more expensive to have proper testing that accurately mimics a production environment. This in turn makes testing harder, leads to "untestable" code, and in turn makes your software worse.

To find out the details, please keep reading.

A Word About Biases

I am basically a card-carrying member of the FSF, it does not take much for me to hate on proprietary software in general. I have made peace with this, but just know that what you're reading. The goal here is mainly to provide arguments for high-quality software engineers to be able to resist predatory licenses in their work. When your boss shows up and asks "Hey, why don't we use this proprietary service instead?" you show them this article. If your boss doesn't ask you, well, that might be a red flag.

What Is Good Software?

In order to know that our software is somehow "worse" we need to actually define what makes good software in the first place, and then how a departure from that makes software "worse". Many people online have simply asserted that X is a hallmark of good software and Y is not, without giving much explanation. Others have asserted extremely complex definitions of what makes good software. I personally subscribe to a very simplistic definition of good software: The fundamental property of good software is its testability.

Everything else, all the other things that we associate with good software, comes from this one property. The idea that "testable software is good software" ought to be an axiom of software engineering in the most literal sense (an axiom comes from a similar Greek word, which means 'that which commends itself as evident') - testable software has that epistemological advantage of making it easy to verify that your software does what you want it to do.

Some examples of how testability leads to all the other properties we commonly associate with good software:

  • "Good software is typically thoroughly tested" - testability makes software easy to test.
  • Yes, I am one of those crazy people who believes in 100% test coverage. I don't care if "100% coverage is impossible", I believe that everyone should try to reach 100% coverage. If your tools don't let you get to 100% coverage, then you're using the wrong tools, and preventing testing is definitely a reason to stop using a tool. More on this later.
  • "Good software has few bugs" - testability implies that its easy to test different code-paths in software and therefore bugs are found early and there are fewer of them hiding in rarely executed bits of code.
  • "Good software is easy to maintain" - maintainability typically means that we can make changes to software easily in response to changing dependencies. If your software is testable and tested, then minor code changes can be quickly tested to make sure that they do not break anything and then rapidly deployed to production. That makes for very maintainable software.
  • "Good software is readable and/or self documenting" - you know what makes for excellent documentation of code? Working tests that people can play with to understand bits and pieces of a program at every level. Tests are self-documenting. They document the live, current working state of your code, and if they pass, then they're in sync with the software. Personally, I find examples of how to use a library to be the best form of documentation - tests inherently provide this.
  • "Good software is generally well-modularized into self-contained pieces that do one thing and do it well" - yup, this is basically testable software. Anything written to be thoroughly tested will have this property. Writing software with a focus on testability effectively forces us to break it up into manageable pieces which can then be tested independently.

So What Brough This Rant On?

Why now? Why did I bother writing a bunch of philosophy about what makes good code? There are tons of self-declared experts who subject us to their own definitions, why me and why now?

Well, I was recently confronted with two events:

  1. Someone explained to me that the reason Oracle is so popular despite being evil is that "no one ever got fired for picking Oracle". This was news to me - for my whole, admittedly short career - Oracle DB and other products have been an absolute abomination onto mankind. Everything from a billing system at a company where I interned, to PeopleSoft used to manage students at the University of Connecticut - if it depended on Oracle software in any way it was just awful.
  2. My fiancée, who works as an accountant, presented me with a problem at work: She needed to RDP into a system in order to use a certain bit of software. The software was some kind of document/versioning management system that allowed multiple accountants to work together on a particular set of documents. It was effectively version control except with a GUI and for Microsoft Office files. It functioned exactly like older version control systems which needed you to check files out, and while you had a file checked out, no one else could touch it. I, of course, could not fix slow RDP connections back to her office, but instead I started asking unproductive questions that prevented her from getting any work done, even more effectively than a slow connection. Specifically: Why bother RDP-ing into a central server with all the other accountants? Versioning software should allow multiple users to work together, and syncing documents is certainly faster than trying to stream video of a screen across the internet, particularly, if the screen has said documents on it. She explained that the reason that they needed to do this is because they had only one license for this particular bit of software, and many licenses were expensive, so everyone had to remote in to use just one installation of it.

And so it dawned on me... my problems at work, and her problems at work, were connected.

Get To The Point...

So why are we here? The basic idea I am trying to present is that many (if not all) proprietary "enterprise-grade" services, like databases, VPNs, public-key infrastructure services (PKIs), authentication/authorization systems like Active Directory, and so on, have licensing terms that strongly disincentivize (if not outright prevent) a user from spinning up an arbitrary number of service instances. These licensing schemes range from pricing pegged to "number of seats", to "up to X amount of RAM", to the number of machines the software is installed on, and my personal favorite "how globally distributed the deployment is". These proprietary, licensed services that your software has to integrate with, almost certainly prevent you from spinning up virtualized testing instances of the given service because the idea that you could run as many Oracle database instances as like in VMs without paying for them is an affront to Oracle's sales department.

The problem with all of this, is that we established above that good software is synonymous with testable (and tested) software. Spinning things up, wrecking them completely, destroying them, and spinning then back up again, is absolutely essential to proper testing. How many times have we all encountered a problem in software engineering, looked deeper into it, asked ourselves or a more senior engineer "why isn't this tested" and found out that "its not testable"? If we dig deeper into why something is "not testable", we typically find bits like this:

  • Well we only have one Active Directory deployment, so the testing environment has a completely different authentication scheme than prod.
  • We test in production because that's where all the services are accessible through the proper TLS/SSL/PKI/Whatever. We can't get the right authentication scheme working in test.
  • We have a testing database, but only one, everyone gets a schema of their own which they can manually manipulate for a specific test, but testing everything in an automated way isn't really possible.
  • We cannot simulate a properly distributed environment in test because we need to pay for more licenses and corporate is only giving us one test license.

I'm pretty sure, at this juncture, that everyone has heard or been in a situation kind of like this, at which point they threw up their hands and said "fine, whatever, everything cannot be tested automatically anyway" and moved on with their lives.

Well, my assertion is that this is where "untestable software" comes from, and untestable software is worse than testable software.

What Does The Alternative Look Like?

Now, I will gladly admit that I love to pick on Oracle. They're pretty awful, between the licensing terms, the strippers-and-steak salespeople, the contracts that pull you farther and farther into their ecosystem, but are there any concrete examples of how this problem proliferates through software development? The best example I can think of is more of the alternative, i.e. what can we do with Free and Open-Source Software?

My personal favorite example is PostgreSQL. I periodically write random web-apps with Flask, and I typically use PostgreSQL as my database of choice. The best part of doing that is that I can trivially get to 100% test coverage if my service is a relatively simple web-app integrating with PostgreSQL. How? well, I use Vagrant to spin up a VM, I use Ansible to provision a PostgreSQL instance and install my web-app, and then I just run my integration tests. Note that there is no mocking involved, I am not mocking anything, I am using a virtualized PostgreSQL instance that is basically guaranteed to behave as close to prod as is possible. I spin up a database, populate it with relevant data, run my tests, wipe the whole thing out, rinse and repeat as many times as you like.

In a distributed setting? Not a problem! Vagrant can spin up multiple VMs, you can configure networking to behave like production, and then you can run your database or whatever it is you need in a virtualized setting. Need PKI? Not an issue, use EasyRSA to just create a PKI for one particular battery of tests, make the certificates expire in a few minutes, then destroy everything once you're done!

That's what the alternative looks like. That's how you get to 100% test coverage, and this is how you write software that you know will work. Virtualization is an unbelievable gift for those of us who want to emulate production environments to make our software bulletproof, we should use it. If there is a FOSS tool to do what you need to do, then there is no excuse to not get to 100% coverage, if there isn't such a tool, consider making one.

There are huge implications here, not only for testing, but for several other key areas of software engineering. For example: because PostgreSQL is free and open source, its free to learn, and to tinker for work or for school. You don't have to worry about licensing, you can just get up and go.

Why Are You Picking On Oracle?

Astute observers will point out that a lot of this stuff is possible with OracleDB, for example:

That's true, Oracle has figured out that you need to be able to do some of that stuff in order to have a claim to helping developers make good software. However, this comes with two, virtually insurmountable problems:

  1. Oracle lives and dies not on the database itself, but on the add-ons, plug-ins, and other additions that it makes contracts for. Many of those are not available for virtualized testing like OracleDB is. So yeah, you can virtualize the OracleDB and test with it, but not all the other crap that's even less reliable.
  2. Because Oracle software is not free and open source, we cannot inspect it to simply write our own ansible modules (amongst other things). We cannot modify their APIs to add endpoints that help with provisioning and testing. We are stuck waiting for Oracle to develop these things.
  3. For comparison, pgAdmin, doesn't have an ansible module for provisioning it. No problem! I inspected the software, literally cut and paste certain setup functions, and made my own ansible module. Now I can provision pgAdmin with ansible without issue. I didn't have to wait for anyone.

What If I Use Something Other Than Oracle?

Its true, Oracle is really awful, and there are certainly other software companies that have less restrictive licenses, for their proprietary service software. Are they just as bad? Well no, but I would like to take a moment to demonstrate how those still hurt your users and you as a software developer.

The example in question is CockroachDB. They make a pretty nice, distributed, SQL database. The general idea with their license terms is that if you are running your own small cluster, you don't have to pay for it, its open source, etc, but if you want certain "enterprise" features like Distributed Backup/Restore, Geo-partitioning, Role Based Access Control, Enterprise directory authentication, and so on, well then you pay them for the "Enterprise" edition.

This is still bad.

For starters, that "Role Based Access Control", and "Enterprise Directory Authentication" are the real killers. That's exactly the kind of thing that causes testing to be different from prod, but there is an even deeper problem here. If you pay for those enterprise features, you probably need them in production. If you need them in production, you need to test them. Depending on the license terms (which are unfortunately unclear from CockroachDB's website), you can end up in a situation where you can run the full version in production, but not in the test environment. So if you need Role Based Access Control, great, you pay for it, and you get it, but you cannot have it on arbitrary dev machines, you only have it in prod, and hence, you end up, once again, in a situation where you are heavily incentivized (through money) to pay only for the prod instances, your test instances are the free ones (or are heavily incentivized to be used less depending on license terms), and you end up with "untestable code" as soon as it needs to touch one of those paid features.

In essense, it is not hard to imagine licensing terms that actually give you software that's less amenable to a testing environment when you pay for it than when you do not. At least when you go for the free, non-enterprise version, 100% of the features you use can be easily run in a virtual environment.

Conclusion

Proprietary licensing structures inherently motivate your testing environment to be more different from prod than it needs to be. They are therefore an impediment to the testability of code, and I consider the testability (and testing) of code to be the primary pre-requisite of good software. Therefore, dependencies on such licensed software services makes your code worse. The problem is endemic to proprietary licensed software because the virtualized testing requires an the ability to spin up arbitrary instances of whatever it is that you're using in a way that mimics production environments as closely as possible. This runs head-first into the problem that the vendor faces: "if people can spin up unlimited prod-like instances of our thing, then how can we make them pay for it"? Through the examples above, we can see that even without onerous things like DRM, proprietary licenses still hurt their users.

The kicker is that there are "generalizations" of this problem everywhere in industry, not just in software engineering. To bring everything full circle, this is the same problem that my fiancée's company was facing, except without the testing component. They were losing tons of productive time just so that they wouldn't have to pay for the licenses and forcing everyone to use a really complicated workflow to get reasonably basic things done. This is a kind of lose-lose, where the company making the software cannot enforce their preferred license terms (i.e. they wanna charge more money for more users), and their client loses significant productivity. This likely causes a death spiral as the makers of the software have to charge more money for single installations because of how they're used through RDP, clients bend over backwards to avoid more license terms, and so on.

I don't think this is an unsolvable situation. I can definitely see arguments from the other side like "well, its not like you're offering any better approach to the developers of the thing to get paid". Something would obviously need to be worked out to benefit everyone, however, as things stand right now, dependencies on services with such license terms are bad for testability and lead to worse code.

  • Last updated:
  • Contributors