We Agilists talk a lot about the importance of Continuous Integration (CI) as a practice and we talk about it like it assumed; however, when adopting CI in what was a waterfall, iterative or undisciplined environment there will be a lot of hurdles to adoption that have to be planned as part of the CI roll-out. Here are just a few to consider:
Is your code structured in such a way so that different sub-systems can be compiled and tested without the need to compile and test other sub-systems. If is isn’t you will need to refactor your code to remove the dependencies, create proxy classes, interfaces, and interface simulators for the dependent sub-systems.
Is your code organized in the source code repository in such a way that you can check-out entire trees/directories and compile test? If not, you will need to spend time (task stories) to reorganize your code. Be sure to look for opportunities to separate business classes from helper, utility, and other common code.
Are you using a source code repository that will even support the role of a Committer? If not, you will have to spend time migrating to such a source code control (SCC) tool. Why a Committer instead of everyone being able to commit code to the baseline? Simple, Code Committers are responsible for what actually is committed to the baseline. They perform such tasks as code-reviews, architectural compliance, and making sure the code conforms to style and quality standards. Think of it as a nice CMMI kind of practice that improves your product’s quality. You are performing peer reviews, right? What about overall code reviews?
Scalable web-based applications has got a lot of air-play on social networks like Twitter lately, mostly because Twitter has been overcome by scalability issues and the service unreliable. Having an unreliable internet service is embarrassing and unacceptable. There is no better way to kill your brand image than to have your viral social media strategy fall flat on its face in front of the “instant on” world.
Having said that, I’ve had a lot of requests for guidance on how to build scalable web-based applications that can withstand getting Slashdotted. Building a scalable web-based application can happen incrementally so you don’t have to apply all the principles below all at once. There is an appropriate implementation roadmap that is appropriate for different types of web-based applications. For example, if your site is proving an web-service API (SOAP service) for other web-services or client applications, you should spend time looking at capacity planning for those services with less emphasis on your own public facing web-site. Anyway… here is my brain dump. There is more where this came from.
Application architecture
Judicious use of the singleton pattern
Judicious Use of the Concurrency pattern
Client side form validation
Use of AJAX for web-services requests
Caching of semi-dynamic data (pre-rendering semi-static pages)
Use of MVC pattern
Object persistence is separated from the object
Use of XML and meta-data instead of traditional row/column SQL commands for each data element
Place SOAP interfaces on a separate cluster of servers (aka application servers)
System architecture
Use a load-balancer and multiple Web servers
Separate your web-application into a minimum of four tiers: UI, Application Services, Object Persistence Layer, Database Services
Use a separate NIC on each server and switch for each tier of the application (e.g. web-page server to application server, application server to object persistence server, and object persistence server to database server)
Use a load balancer between each tier of of the application (e.g. between web-page servers and application servers, between application servers and object persistence servers, etc.)
Use an enterprise service bus object persistence service to make sure object concurrency issues are handled across multiple databases
Use clusters of virtualized servers running across multiple physical servers
Add performance monitoring services on each virtualized server to check memory, network, harddrive and application utilization
Conduct performance tests to determine if any one object needs more server resources, and move those objects onto their own virtualized and physical servers
Network architecture
Use lots of partitioning of data across NICS and Physical Switches (aka switch processors)
Use firewalls in front of each switch in the top three tiers (UI, Application, and Object Persistence Layers)
Place network probes between each firewall and load-balancer to monitor utilization and intrusion detection
Place each application stack, a complete web application, application server and database server, in at least two data centers in two different geographic locations using two different internet data service providers
Create a private point-to-point network between data centers for transaction load balancing using two different internet date service providers
I know was a lot of technical mumbo-jumbo. Frankly, I haven’t met that many web designers, web-developers, or even a lot of software developers that understand everything I’ve listed. In order to implement the list, it will take more than one skillset from several technology professionals. (Business owners can contact me privately about assessing the maturity of your development staff. You might be surprised.)
If you don’t understand it, I am happy to elaborate privately. Understand that this is what I do when I’m not playing drums in a Jazz combo.
