We've been thinking hard about how to classify data beyond the typical "On the Move"/"At Rest" definitions. We have divided the domains into areas of control independent of the physical storage mechanisms, which in the cloud may be abstract and nebulous concepts.
I stopped the other day to think about something that someone in the office said about a customer project after a good 3 months of work. It made me sit up and wonder what constitutes obsolete software these days. Before the cloud we could go years before upgrading to something new. I remember in the early days of .NET they were still applications as far back as 2-3 years ago that were still running on .NET 1.1 warts and all. The comment was a really innocuous one but made me think. We've been working on a Service Fabric application for a customer to process and generate events based on a set of rules. A very common pattern and with the Service Fabric we make light work of it. However, Microsoft released the Azure Event Grid which is completely serverless. You give it some rules and it goes off and works out how to route messages and invoke which services based on inputs. It doesn't need to worry scale or load because it's designed to scale on demand. Whenever I used to do the cloud introduction workshops for startups years ago I used to discuss the idea of cadence of software delivery in the cloud. It seems to be on steroids at the moment! Every pain point that you feel Microsoft or other cloud vendors seem to be pre-empting and delivering something new to fix it before you're aware of it. As software developers we're used to spending a lot of our time building plumbing or infrastructure code to bridge services and route user requests; it's the same for data scientists spending 80% of their time cleaning data. It just seems these days that by the time we build a system we've already accumulated a bunch of technical debt. I've been working with the cloud for a long time now and I'm very aware that customer drivers for the cloud are pushing the pace of change beyond the limits of anything we've seen. Many customers are waiting for the next best thing to plug something that they see as essential otherwise they can't deploy. This is something we're getting very used to now, not waiting for software over years! It's worth stopping and thinking every once in a while. I did a talk and broke the back of the new Machine Learning services in Azure a few weeks ago and now I have something in production. We use a combination of Agile and Crisp DM and a secret sauce as part of our methodology; the new Workbench tool shrinks the time we need to spend on Data Understanding and Data Preparation phases. It's things like this that make us more productive but they happen so quickly that we might miss the change and carry on in our own crappy way with the best of intentions. The moral of this post is to keep your eyes open for all of the crazy new tech you're seeing. Often it will waste time for you as the beta cycle is shrunk (delivery cadence!) but sometimes you'll have a great time which works really for you. Keep your eyes on the cloud; if you blink you'll miss the innovation. Happy trails!