Evaluation Criteria and Template for Free Software

Criteria for Evaluating Free

When considering free software and services it is wise to heed the caution in the old saying "There's no such thing as a free lunch". Too often we confuse FREE with simplicity, low risk, and no cost. Although it might seem contradictory at first, FREE can be costly, and a free mistake can be very costly. Whether considering a "paid" solution or a free one, doing your homework beforehand is job one!

Functionality

The first question the software decision maker needs to answer is does the product do what he/she needs it to do. Not only does this involve the capabilities of the product itself, but as per Wheeler (2010) “you should consider how it integrates and is compatible with existing components…If there are relevant standards…does this program support them?...You should also consider what hardware, operating systems and related programs it requires.”


Other features to consider regarding functionality include:

Ease of Use / Complexity
Free does not necessarily mean simple and often it is not, either from a system support or an end user perspective. Are user interfaces and functionality intuitive? Will the software run on a proprietary server or a client device? Does the system need to be maintained and supported by technologists in order for users to access it, or is it downloadable and production ready? What is the risk to the user or organization if the software doesn’t perform as expected? What is the risk to the user if the data base is corrupted?

Customization
If the product doesn’t do everything required of it (and rarely does it), can it be tailored to meet the organization’s/user’s needs? Can the product be customized with tailoring/configuration options? Can the product be customized via programming changes? Is the programming language popular and widely used? Does the user have skills in-house to modify the product if required? If not, are there fee-for-service providers available? Is it capable of interfacing with other stand alone programs that can fill the gap? (See below for more ways the product may need to integrate with other system elements.)

Integration and Interoperability
Will the product work with the other products and platforms that you need it to function with? For example can you integrate it with your email system, user data bases, or learning management system? Does it need to be compatible with mobile devises? Finally, does the user require other software interfaces to make it accessible to the end user?

Transportability
Transportability refers to the ability to move the program and related data/outputs from one location or environment to another. If the program operates on a proprietary server, will there be a need to migrate either the program itself or production outputs to another device or environment? Will users have a need to access program outputs if the host device is no longer available to them? If yes, what options are available? Are there freemium host service providers available to either the organization or end user? Are there fee-for-service host service providers available? Can data be easily migrated, either into the system initially, or after the fact if required?

Scalability
Can the software grow as users requirements grow? Is there a limit to the number of users who can access the system? Is there a limit to the number of users who can access the system free of charge? What about the data? Is there a limit to the size of the database that the program can effectively handle?

Security
Another factor to consider is security. Does the program provide the required level of access security to users at both the software and data levels? Does the program allow for different permission levels and options to easily be set? Are there different publishing / view options available?

Community

Although some open source products may offer a formal or fee-for-support option, the community of users accessed through online community forums, is generally the users' first line of support for both user and system support with free and open source products.

A large and active community implies better support, greater acceptance of the product and improves the likelihood that the product will endure and improve.

Consider the following criteria when evaluating the community:

• What is the volume of posts?
• What is the average response time for replies to post questions?
• What is the tone of posts responses? Are they friendly and helpful?
• Is there an FAQ (frequently asked questions) section?
• Are there tutorials?
• What type of support is available for the product?

Support
Support is often perceived as one of the biggest issues in open source software deployments (Hein 2003). It is critical to consider the different types of support available. Whether from a volunteer participant community or via fee-for-service support there are generally two categories of support for software products:

• User support: The support provided to end users on how to use the product.
• System Support: This is the support provided to developers for solving of problems with the software itself. System support and maintenance can be major or minor, and provide fixes, release updates or advice for: typographical errors, bugs and broken code, and design flaws, as well as fixes for problems caused by problems in the code.

Documentation

Documentation is an important component of the support available on any product. There are commonly three categories of documentation that are required for a product. As per van den Berg (2005):

• “User documentation contains all documents that describe how to use the system. For certain applications there can be different level in the user documentation, corresponding with different user levels and rights. For example, many applications that have an administrator role have a separate piece of documentation for administrators. Additionally, there can be various user-contributed tutorials and How-To’s, be it on the project’s website or elsewhere.
Telephone, email, forums, manuals”
• Developer documentation contains “separate documents on how to add or change the code, as well as documentation within the source code, by way of comments. The comments usually explain what a section of code does, how to use and change it and why it works like it does.”
• Maintainers documentation should be available for larger server-based applications and includes “the install and upgrade instruct ructions…with the required infrastructure and the steps for installing the software properly explained”
(p. 15)

Service Platforms

Another aspect to consider regarding support is the varying levels of service offerings required and if they are available. Does an online forum meet support requirements or might the user require telephone helpdesk support, personal response email support, live chat room support etc. Are there premium fee-for-service options available? Where is the support centre located? What time zone is it in? What language do the support providers speak? Does it matter?

Reliability

A critical issue to consider is the reliability of the program being installed. Unreliable software can result in catastrophic consequences at both the individual device level as well as at the enterprise network level. There are a number of factors, however, which can be assessed to help mitigate the risks:

Longevity
As summed up by van den Berg (2005) “The longevity of a product is a measure of how long it has been around. It says something about a project’s stability and chance of survival”. (p. 13)

Release Activity
A good mark of how involved the community is with the product is to track the product’s release activity. How many new releases are there a year? Do they include new functionality or merely broken code fixes? In What is Web 2.0: Design Patterns and Business Models for the Next Generation of Software (2005), Tim O’Reilly describes how "in the open source movement users are treated as co-developers…the open source dictum, ‘release early and release often’ in fact has morphed into an even more radical position, ‘the perpetual beta,’ in which the product is developed in the open with new features slipstreamed in on a monthly weekly, or even daily basis’". A static product is a good indicator that the community has lost interest.

Reputation / Referrals / Market Share
It is important to learn who else is using the product and what they think about it. A lot of information about a programs strengths and weaknesses can be obtained with a little research. A Google search can reveal what reviewers are saying about the product? Is there up-to-date coverage of the product? What are the best in class products and what are some alternative solutions? What is the product’s market share? Is it increasing or decreasing? If people are abandoning the product maybe they know something others don’t. Conversely, a widely popular product is likely to meet the needs of a wide variety of users. Does the product have a reputation for being reliable or buggy? Who else who has used the product? What do they think? How does their situation compare in terms of both functional requirements and level of expertise?

Real Cost of Deployment

Although upfront purchase costs may be zero, software procurement costs are a relatively small component of the overall cost of a project. Are hardware and network capacity sufficient to support the program’s implementation? What training requirements exist for technology support personnel, developers and end users? Are there any costs associated with migration from a pre-existing system to the new one? Are there any costs associated with developing content / data to effectively use the product? Does the user need to purchase hosting services or additional consulting or service support? ( For More: See Real Cost of Free Software and Free for Profit on The Open Source)

Licensing Issues

A common mistake is to believe that just because something is free that means that users are free to do whatever they please with it. Equally common is the belief that having paid for something, it belongs to the user. Licensees ARE NOT owners. It is imperative to pay attention to those pesky little things called licensing agreements and the equally pesky pop up boxes that (most) everyone agrees to without reading.



With both fee-for-product licensing agreements and “free” software licensing agreements, terms and conditions of appropriate use, reuse, and redistribution are placed on the user. It is beyond the scope of this document to provide an in-depth tutorial on license agreements, but license agreements will generally fall into a number of broad categories:

• Traditional copyright licenses in which all rights are retained by the original developer. Use and distribution of the resource as well as use in derivative works is forbidden without permission from the original creator. Permission may be granted free of charge or for a fee at the discretion of the owner.
• Open source “copyleft” licenses that prevent the inclusion of source code or the product in proprietary or commercial products.
• Open source licenses that permit use of the product in proprietary or commercial products.
• Free Software licenses that allow use of the product by a user but do not provide access to source code or allow for product redistribution. Remember “FREE” here DOES NOT MEAN that it is NOT proprietary.

Other factors to be considered within the license agreement include warranty information, restrictions on migration, copying, numbers of allowed users and what if any access the use of the product grants the vendor/supplier to your computer.

According to Wheeler (2010) in How to Evaluate Open Source Software / Free Software (OSS/FS) Programs “Users of proprietary software must set up organizations to track proprietary programs, and ensure that extra copies are not made. Otherwise, they risk being sued for piracy. Since it’s usually quite easy to copy programs, this is not an easy task to do. In some cases, users of proprietary programs must bear the risk and costs of later license audits. Increasingly to ensure that customers do not have unauthorized copies, vendors are relying on license audits to ensure compliance” (Section 4.13.2 License Audits)

Failure to understand the permissions and restrictions of the license agreement can result in the ultimate in peskiness: an expensive lawsuit.

Next up History of Open and Free Culture