Modern Data

I recently had the opportunity to spend some time speaking with Maxim Lukiyanov, Principal Program Manager at Microsoft’s AzureML team. I asked him what exciting things he’d been working on and he gave me an in-depth overview of the new Notebook VMs in the Azure ML Workspace service. I previously

To enable the new breed of analysts and data scientists to deliver high quality data analytics, IT teams need to augment conventional data quality processes with modern measures aimed at improving the usability of data As data grows, more data means more data problems, data governance and quality

Right, so I promised our awesome marketing lead Lucy that I would write a post every day for a month so here is the first one! Right at the start of my Spark journey there was a bunch of things available between 0.4 and 0.7 and they were all disconnected and unpackaged from one another. Most of th

So yesterday was the first day of Spark summit Europe 2017 in the beautiful city of Dublin in my (heart's & heritage) home country Ireland. Day 1 was the demo and workshop day with the rest of team Elastacloud learning about optimisation of Spark applications (I mean really useful but a little bit z

I gave a talk last week on MMLSpark, Microsoft's latest innovation to scale and operationalise Data Science. Two posts ago you heard about Machine Learning Modelling and Experimentation which couples fantastically with MMLSpark through a client called Workbench which contains a myriad of features th

Here at Elastacloud we have been using Azure Data Factory (ADF) since it initial release, I’ve recently been catching up on some of the new features that have been released over the past few months. Some of which are pretty neat given some of the issues we’ve encountered in a number of projects ove

At Elastacloud we pride ourselves on pushing the boundaries of what is capable in the data domain. One example of this is our continuing work with Power BI Custom Visuals. We always contribute our efforts for free to make data insight democratised and ubiquitous. This week we spiked out an early v

Azure Stream Analytics (ASA) allows real-time analytic computations on streaming data. Stream Analytics allows one to set both inputs and outputs for the job and the bulk of the logic/operation is done within a T-SQL Query , which makes up the Streaming job. The ‘Job Topology’ outlines this in

A few days ago, Microsoft announced Azure Event Grid in preview, a new service to route events from publishers to subscribers, with a lot of integration with the rest of the Azure services. Tom Kerkhove has done a great write-up of the basics over at https://www.codit.eu/blog/2017/08/21/exploring-a

A customer currently generates XML files from operating devices and wish to gain insight into the data they are generating. They require that the data is transformed and landed in an Azure SQL Database and PowerBI visualizations. At present this data resides within the customers own network, so to

At Elastacloud, we have an expectation of building leading edge data stacks for large enterprise customers. We choose the best technologies that fit our ethos of elasticity, focus, agility and economy in the cloud. A key emergent trend in our customer is the notion of a wide data stack. 1. Data ac

Empowering data analytics is one of the mainstays of modern Cloud Computing. Approaches that may themselves be tried and tested are applied at internet scale with the co-ordinated, distributed computation of hundreds or thousands of machines over terabytes or petabytes of data, leading to new cha

I managed to meet up with Andrew Morgan at the Spark Summit who I met a few years ago when he was a regular at Data Science London and working on a time series predictive model called Trend Calculus which looked pretty fantastic. He has moved onto a new project which allows you to predict the news

Yesterday was the start of the actual conference after a great workshop day! Team Elastacloud were once again up bright and early (perhaps regretting the night before and lack of sleep), which was tricky after the amazing Spark meetup in the evening. Our hotel is just a hop, skip & jump from the Con

I'm at the Spark Summit in Dublin and have been in a session with Team Elastacloud today on Spark optimisation. It has been a pretty good session. Many of the things I knew already; several things on partitions, optimisations of queries and parallelism. All good stuff. I didn't have the chance to lo

Last week I gave a talk to Spark London about the relatively new Microsoft Machine Learning Framework for Apache Spark, MMLSpark. For those of you that have known me for a while you'll know that I'm like a scratched record about Apache Spark and have dedicated a huge amount of time and effort to und

Last week I gave a talk to a packed Azure User Group on operationlizing machine learning. It's a subject I've spent a lot of time thinking about for the last couple of years. At Elastacloud I've been looking at solutionizing for customers on this for some time now so it was fairly timely that this w

A customer has a dataset that they wish to be enriched with hourly weather data, but weather depends on where you are in the world. We want to find the weather for a given time in a specific location. With a source dataset that includes geographical coordinates and an Azure Function with a Time

Working with some customers it is often desirable for them to maintain an on-premise, operational system but have that system stream it's data to the cloud for new types of work such as analysis or business intelligence. In this instance streams of transactional information is pretty straight forwa

We've been building a system for higher level management of storage assets in a cloud context for some time. The Elastacloud Data Catalogue (EDC) manages the state, history and the utilisation of Cloud Storage. Unlike other offering the EDC takes a data oriented view of storage, rather than a direct

Following on from @David's earlier post on Excel to Cloud migrations, a different possibility is also worth considering. David's approach is valid and leverages BI technologies to bootstrap a more powerful cloud enabled path. This is like turbocharging the existing design as Excel is seen naturally

It is very common for companies to use Excel Spreadsheets both to store data and derive insights from that data. We encounter many companies for whom Excel is a useful tool, but find the tool is unable to keep pace with the data they are generating. There is a drive to produce more powerful analyti