Developers working with three tiered software architectures -- divided into presentation, business logic and data tiers -- cannot efficiently handle temporary application data that must be sharable between servers to provide high application availability and seamless scalability. But technologies, many of them open source, have emerged that deliver an important piece of infrastructure to manage this work-in-progress data.

To understand this, it is valuable to step back and think in terms of the classes of data we need to manage in tiered applications and the best mechanism to manage each class based on properties such as rate of change, life cycle and need for long-term persistence.

Developers will concede that some types of data don’t need to be stored for long, so in most cases such data does not belong in a relational database. This class of data is essentially work-in-progress data for incomplete business workflows – data that is no longer needed when a workflow is complete, but essential for reliably reaching the end of a workflow.

Examples of such work-in-progress data include user session data regarding the stage of a workflow, user authentication and authorization information, content that a user creates or edits, or customer-service chat messages and related conversation logs. Other cases involve in-memory temporary data-sets used by batch processes for intermediate computations as part of a larger work effort. These might occur in trading systems or applications that perform complex pricing, cost allocation or matching algorithms.

Click to see: Diagram of managing work-in-progress apps data

Consider this analogy. We all remember taking multiple-choice tests that used computer-graded forms in which you had to fill in little circles. The paper was a great way to capture the final answers but was not useful in helping us arrive at the answers, especially in math or science, where interim calculations were needed. For these calculations, we often used a scratch pad.

The scratch pad helps us organize data in our brains, perhaps by drawing a diagram, or by using special symbols and tools, or by allowing us to break a problem down into smaller pieces. Working on problems using a scratch pad also affords us the ability to stop work midstream and later pick it up where we left off. We can even efficiently retrace our steps and check for inaccuracies.