• Channels

  • Contact

  • Main Site

  • More

    Use tab to navigate through the menu items.
    To see this working, head to your live site.
    • Categories
    • All Posts
    • My Posts

    Software Engineering

    Everything Engineering coming out of Elastacloud right now.

    Software Engineering
    Darren Fuller

    Filtering DataFrames with other DataFrames in Spark

    Something that comes up quite frequently when people start using Spark is "How can I filter my DataFrame using the contents of another DataFrame?". People with SQL experience will immediately look to trying to replicate the following. SELECT * FROM table_a a WHERE EXISTS (SELECT * FROM t
    Views 
    0 comments0
    0
    Recent Activity:
    Jan 24, 2019
    Darren Fuller

    Read the documentation for better performance

    Documentation is not something people often spend time reading, or if they do then its to quickly find the one thing their after and then get out as quickly as possible, very similar to how I do my Christmas shopping. Sometimes it's worth spending time reading the documentation though as there can b
    Views 
    0 comments0
    0
    Recent Activity:
    Dec 06, 2018
    Darren Fuller

    Kotlin and Azure Functions - Part 1

    I recently posted a tweet in which I briefly mentioned that I'd been writing a simple Azure Function using Kotlin . I thought I'd put together something a bit more in-depth, and which did something a bit more than say hello back based on a name in the query string (although I do cover this below,
    Views 
    0 comments0
    0
    Recent Activity:
    Nov 04, 2018
    Darren Fuller

    Comparing results in Azure Data Studio

    I've posted about Azure Data Studio before (then called SQL Ops Studio) but wanted to just bring it up again, specifically where it's helped me in a specific use case. We have a scenario where a model created by one of our Data Scientists is predicting values for the next few days, what I wanted t
    Views 
    0 comments0
    0
    Recent Activity:
    Oct 26, 2018
    Darren Fuller

    Saving a DataTable to Parquet with Parquet.NET 3

    A while ago I wrote a post about extracting data from SQL and into Parquet . This was aimed mainly at on-premise systems where Data Gateway or others are not an option but you still want to get your data into a format which can be used by tools such as Azure Databricks . Since that post Parquet
    Views 
    0 comments0
    0
    Recent Activity:
    Sep 17, 2018
    Darren Fuller

    You've gotta comprehend

    One of the things I often find myself doing is collecting data from external sources to get at feeds of data such as weather, pricing, scheduling etc... Often these sources have been around for a while and the world of APIs has moved on, meaning we often find data in massively nested structures, XML
    Views 
    0 comments0
    0
    Recent Activity:
    Aug 17, 2018
    merv

    What's the Time Please

    Frequently we get requirements for UTC times in Azure SQL databases to be converted into various world time zones. Whilst scripting within the table builds and adding columns, or relying on sever settings, can produce this outcome there is a more flexible approach using the AT TIME ZONE (T-SQL) s
    Views 
    0 comments0
    1
    Recent Activity:
    Jun 26, 2018
    Darren Fuller

    Save the date

    There's still one thing we encounter all to frequently when working with, manipulating or simply extracting data. We can book tickets over our mobile phones, check the news, integrate AI into our websites and determine if one message should be reviewed before another based on sentiment analysis. But
    Views 
    0 comments0
    0
    Recent Activity:
    May 26, 2018
    Darren Fuller

    Unit testing and Python

    You might have read a post from me once-or-twice where I've advocated using Python for quickly prototyping solutions, exploring data etc... But regardless of whether you're using it for building a quick prototype or create a larger application it is, as with all languages, best practice to write s
    Views 
    0 comments0
    2
    Recent Activity:
    Apr 19, 2018
    Darren Fuller

    An eye on the bank balance

    We don't all have a bottomless pit of money and I think most of us have learnt by now that it certainly doesn't grow on trees, so in our home lives we keep an eye on how much we're spending otherwise we get ourselves into trouble. The same is true when building services in the cloud. Even large, mul
    Views 
    0 comments0
    1
    Recent Activity:
    Mar 02, 2018
    David

    Operationalising Data Science

    Operationalising Data Science is something of a difficult question for us here at Elastacloud, given the wealth of options available (AzureML, Spark, python web service etc). Recently as part of Renewables we’ve been updating our model to the latest and greatest and wanted to move away from using
    Views 
    0 comments0
    0
    Recent Activity:
    Feb 20, 2018
    merv

    Azure DW - Rolling Dates From A Static Calendar?

    One of the requirements we most often come across when working on solutions involving Azure Analysis Services is for rolling date time values so that the end user can filter their data based on calendar-based sequences (e.g. rolling day, week, month or year sequences). I have found that one of th
    Views 
    0 comments0
    0
    Recent Activity:
    Feb 07, 2018
    Darren Fuller

    Extracting data from SQL into Parquet

    Lately I've needed to extract some data from a SQL database (and others) and move it into Azure to be worked on using other tools. We're big users of Parquet which is available across a range of tools such as Azure SQL Data Warehouse , Azure Data Lake Analytics and of course, Spark . The first c
    Views 
    0 comments0
    0
    Recent Activity:
    Feb 05, 2018
    Darren Fuller

    Think secure

    I saw this the other day and had to grab a quick picture of it, but I will add that I left both the lock and the locker contents alone after that. It did get me thinking about how we view security in the software industry which, given recent events, is more visible now than it has been in the past.
    Views 
    0 comments0
    0
    Recent Activity:
    Jan 23, 2018
    Divyesh B Chudasama

    SQL Error SQL71006 – Data Factory

    When working with Database Projects you will sometimes want to use a post-deploy script to perform any number of operations on the database every time the database project is deployed to a target database. In doing so you may encounter an error such as the following: ‘Error SQL71006: Only one sta
    Views 
    0 comments0
    0
    Recent Activity:
    Jan 09, 2018
    Darren Fuller

    ARM and ARM Functions

    Recently I needed to deploy an Azure Data Lake Store - Gen 2 instance and thought I'd take the opportunity to use some custom ARM template functions . These aren't something you often see in the example templates but can be really useful if there's a complex expression which you find yourself wri
    Views 
    0 comments0
    0
    Recent Activity:
    Jan 07, 2019
    richardp

    Azure Durable Functions Fan In/Out Example

    Durable Functions is an Azure Functions SDK extension which enables the Azure Function server-less environment to run "stateful" applications. The following example below briefly demonstrates the Fan In/Out pattern that the Durable Functions SDK enables with ease. In this simple example, a me
    Views 
    0 comments0
    1
    Recent Activity:
    Nov 15, 2018
    Richard Conway

    Don't turn your back on Big Compute! (part 1)

    We started Elastacloud eight years ago by focussing on High Performance Computing on Microsoft Azure. HPC is normally in the purview of Researchers, Engineers and Risk Calculators in finance but it is good for so much more. Here over the next few posts I'll break down the problem domain and why I di
    Views 
    0 comments0
    2
    Recent Activity:
    Oct 28, 2018
    Darren Fuller

    When 1am is not 1am!

    Dates, you either hate them or you're not paying close enough attention. Recently I've needed to break a day down into buckets of time for a project, for most days of the year this is fine and the 24 hours of the day (and yes, I know, but I can ignore that part for now) nicely breaks down into equal
    Views 
    0 comments0
    0
    Recent Activity:
    Oct 10, 2018
    Darren Fuller

    Azure Active Directory authentication with Azure Functions

    So you've been given access to an API which is an HTTP triggered Azure Function which is secured using Azure Active Directory, it needs to be called programmatically so you've been given a few things to do this including An Azure tenant id An application id and secret The applications id URI A funct
    Views 
    0 comments0
    0
    Recent Activity:
    Aug 17, 2018
    Darren Fuller

    DotNetCore and Azure Storage

    I was busy scanning through various news feeds and came across a link to a post talking about how to use Azure Storage in a DotNetCore application. Curious as always at seeing how others do things I followed the link and was immediately hit with one of my pet peeves, there at the bottom of the scr
    Views 
    0 comments0
    0
    Recent Activity:
    Jul 28, 2018
    Divyesh B Chudasama

    Azure Data Factory v2.0 - What's New?

    Microsoft have now released a v2 of Data Factory. Though this is still in preview, it has the handy ‘Author and Deploy’ tool; this includes the copy activity wizard to assist creating a copy data pipeline. Most of this is the same as v1, however there are changes that have been introduced in this
    Views 
    0 comments0
    0
    Recent Activity:
    Jun 07, 2018
    Darren Fuller

    Integrating Machine Learning into your .NET applications

    There were a lot of great announcements from Build this year, from Visual Studio Live Share to Azure Sphere and everything in between. One thing that I was particularly excited by was the announcement of the preview of ML .NET , a cross-platform, open source machine learning platform for .NET, al
    #thetramway 
    Views 
    0 comments0
    1
    Recent Activity:
    May 15, 2018
    Admin

    React TypeScript Tab Component

    Whilst looking for an example of some Tab components online for a react project I came across exactly what I needed on a github repo by Diego Castillo. However, I was using a Typescript project and, as I couldn't find anything else specific to Typescript I decided to alter Diego's code to make it w
    Views 
    0 comments0
    0
    Recent Activity:
    Apr 12, 2018
    Darren Fuller

    Import Time_Savers

    Pandas is a great library, I probably don't need to spend much or any time telling people that, but it's a great library for a number of reasons and not just for data science work. I've been needing to extract data from some pretty odd structures lately, so to get up and running and try out some id
    Views 
    0 comments0
    0
    Recent Activity:
    Feb 22, 2018
    Darren Fuller

    Modern documentation

    Documentation, it's often a curse word among developers, the Word document you have to put together to satisfy some process or another which you know almost no-one will ever read, that the next person will be unlikely to update (because it's Word, it's too long or both), and will probably be out-of-
    Views 
    0 comments0
    1
    Recent Activity:
    Feb 15, 2018
    Divyesh B Chudasama

    Invalid provider type specified – Service Account Auth

    A recent project required me to use the Google Analytics Core Reporting API for data ingestion. The API call was being made in an Azure function, which worked completely fine locally but failed during the service account authentication process when hosted in Azure with an ‘Invalid provider type
    Views 
    0 comments0
    0
    Recent Activity:
    Feb 06, 2018
    Darren Fuller

    Quick visualizations with SQL Operations Studio

    Over the last couple of months I've been getting a lot more acquainted with Microsoft's latest tool for working with databases, SQL Operations Studio . It's a free tool for working with SQL Server, Azure SQL DB and Azure SQL Datawarehouse which runs on Windows, Mac and Linux and is based on Visual
    Views 
    0 comments0
    0
    Recent Activity:
    Feb 01, 2018
    Darren Fuller

    Using R scripts in Azure Functions

    Recently we've been working on how we could deploy models written in R into a production system, where they could be called as part of a data pipeline or on-demand. From a data engineering point-of-view one of the best tools in your toolbox when you want to execute relatively small pieces of code
    Views 
    0 comments0
    0
    Recent Activity:
    Jan 19, 2018
    Divyesh B Chudasama

    Azure Resource Manager - Referencing with Linked Templates

    Azure Resource Manager Templates (more commonly known as ARM Templates) serve as a very handy tool for Cloud Engineers. ARM Templates allow you to automate resource deployment in Azure in a “cookie-cutter” type of approach i.e. they are just scripts instructing Azure what resources to deploy and
    Views 
    0 comments0
    0
    Recent Activity:
    Nov 24, 2017
    • Twitter Social Icon
    • LinkedIn Social Icon
    • Facebook Social Icon

    Visit the Elastacloud website