Thursday 4 June 2020

More Star Wars 3D models

I only made R2D2 as a learning excercise, and I'm aware of a lot of mistakes I made with him and I know I could make him better if I tried again...

But, keeping with the star wars theme, I have made a few more models.

I made this tie fighter because I also figured it was be easy.  It is missing some detail, but I'm still happy with the result.

After making the tiefighter, I could not resist the idea of making an x-wing.  I was worried this was a step to far.  But again, happy with the result.


Finally, my latest model is a BB8.  This was one of my first own modeling projects without using blueprints for precision modeling.  Again, he's missing a lot of the detail from the movie shots.  But I'm learning...









Extracting Addresses from FreeText

The Task

A recent task I was set required the extraction of Addresses from Freetext.  Not an easy task, but I was supprised how successful my final result was...  So supprised, that I thought I would share it.

This was a SQL Server based task.  SQL Server's native pattern matching is not sofisticatd enough for the task, so I opted to use Regular Expressions.  I could have used an SSIS package to process to data, but I instead opted to use CLR and a Table-Valued Function to ruturn the found addresses.

I'm not an expert with Regex, I generally have to relearn it everytime I use it, but I often get the pattern and results I want.

Finding addresses

The addresses I needed to find are all UK based, meaning I can use the postal code as a starting point for address extraction.  The post code is the most distinctive identifiable componet of an address and was a manditory requirment of the addresses I was to extract.  My approch was to identify valid formated post codes, and then calculate the remainer of the address from there.

Based on this, I was able to formulate a regular expression pattern to perform the address extraxtion which can be split into the following components:
  • Flat Number (optional)
  • AddressLines (hidden)
    • Door Number (optional)
    • AddressLines (optional)
  • Post Code (mandatory)
I used a hidden/un-named group to combine the door number and address lines regex.  This ensures that if a door number is found, it will control the length of the address lines.

Post Code Regex

Based on the information from wikipedia regarding uk post code formatting (wikipedia-Postcodes_in_the_United_Kingdom.)  The format is comprised of an outward code and an inward code seperated by a space.  I'm aware the space can often be omitted.

The outward code contains an area code of one or two alphabetical characters followed by a district code which is a single digit, then optionally a single alphanumeric character. 

The inward code is made up from the sector code which is a single digit followed by the unit code which is two alphabetical characters.

To locate a valid UK post code I ended up with the following regex:
(?<PostCode>(?<Area>[A-Z]{1,2})(?<District>\d\w?)\s*(?<Sector>\d)(?<Unit>[A-Z]{2}))
 

Address Lines Regex

To find the address lines, I'm looking for between 1 and 4 blocks of text seperated by comma which preceed the Post Code.

The regex I used for this was: 
(?<AddressLines>([A-Z-''\s]{2,}[,\.\s]){1,4})
 

Door Number Regex

To find the door number, I used an assumption that it must start with a digit, but could contain a leter (example 2b or 7a.) I also allowed the door number to be either space or comma seperated from the address.

To find the door number, I used regex:
(?<DoorNumber>\d+[,\s]+|\d\w[,\s]+)
 

Flat Number Regex

For the data I was working with, flat numbers always followed the word flat.  For that reason I was able to look explisitly for the literal word Flat (case-insensitive) and attempt to find a flat number in the following.  As with door number, I permitted a flat number to be suffixed with a letter.

The regex I used to find the Flat Numbers was:
(?<Flat>Flat\s?(?<FlatNo>\d\w*)\s?,?\s?)
 

Full Regex

So, all of the above regex components combined (with an unnamed group to combine Door Number and Address Lines) resulted in the following regex:
(?<Flat>Flat\s?(?<FlatNo>\d\w*)\s?,?\s?)?((?<DoorNumber>\d+[,\s]+|\d\w[,\s]+)?(?<AddressLines>([A-Z-''\s]{2,}[,\.\s]){1,4}))?\s?(?<PostCode>(?<Area>[A-Z]{1,2})(?<District>\d\w?)\s*(?<Sector>\d)(?<Unit>[A-Z]{2}))

I was able to perform the regex using a clr function and return a table containing the match result along with the components of the address.

Results

When given the task, I was not expecting to do well beyond post codes.  The post code matching was almost perfect except for invalid or typo's in the freetext.  However, I got some very impressive results.  Any address with a door or flat number extracted perfectly.  However, those without where hit and miss, sometimes I grabed to much text preceding the post code and sometime not enough.

Conclusion

I'm not a regex expert, I'm sure it could be improved or optimised.  I'm also sure I may have missed some posible symbols which are valid addresses or other posible formatting of flat and door umbers.  However, for the task and dataset I was working with, the extraction was far better than expected.

I do have a few ideas how it could be improved, however, I have meet my requirement and the project has completed.  So for now, this is it.

Friday 22 May 2020

R2D2 Animated and in THREE.JS

My efforts have payed off, I have been successful in exporting my R2D2 from Blender to THREE.JS and Animating him.  I have also learnt a lot more about the THREE.JS animation system.

Monday 18 May 2020

I made R2D2 (in Blender..)

I'm not creative or artistic.  I have don't see myself making interior designs or 3D rendered works of art.  But I am increasingly using 3D models in my visualisation work (and playing with AR.)  So I'm highly dependant upton the 3D models I can get for free or purchase cheaply.

This is a problem for me...  As I have a very small budget and many of the things I've wantted are way out of my budget.  And so, I needed to learn how to make some of the models I need myself.

So far, I have made a Donut, an Anvil, a Chair, a Coke Can, Titanic and R2D2.  I can see that I will always be an ammeter at 3D modeling, but I'm happy with my results.

I had no real need for R2D2, but I felt he was an interesting subject to learn and practice my skils I'd learnd from YouTube (most specificly Blender Guru.)  

I made him twice, the first time was based on reference photos from Google Images, this was ok, but I cut a lot of corners...  The second attempt was based on images and blueprints from https://astromech.net/.  He's not entirly acurate, and Not fully finished, but he will do and I have learnt alot.

R2D2 3D Model

My reason for modeling R2D2 was to improve my skills.  I had attempted a few modeling projects for objects I wantted, but they failed (attempt 2 coming soon). I ultimatly wanted to animate R2D2, but that's not currently going very well either.

For now, I'm just happy with the model as an early result in my 3D modeling skills.

Clone of a SkinnedMesh (loaded Model) in THREE.JS with animations

One of my biggest annoyances with using loaded models in THREE.JS was the failure to clone them, and more importantly, animate the clone.  Without the ability to clone a skinned mesh or loaded model and then independently animate it, I was forced to keep loading the model from file.  This was very slow and resulted in load loading times.

I tried everything...  I discovered a clone method on SkeletonUtils which would supposedly work. but I just got errors or a mesh that would not animate.  I even found functions written and published by other people, but these also did not seem to work.  I did my best to optimise the load from file to speed things up, but it was useless.

So, I created my own function to perform the clone operation...  Being an ammeter with only a simple understanding of the THREE.JS object model, I set about writing the function to clone any Object3D.  I had observed that many of the models I was working with featured a hierarchy of objects of types Group, SkinnedMesh or Bone (others currently unsupported, but probably a simple change.)  I then used comparison techniques and trial and error until things started to work.

I then created a function specific to the SkinnedMesh type, because I needed to copy some important properties only relevant to this type.  Finally, I created a function to copy any userData.

The functions are a work-in-progress and might need a tweak to work with other models, but it dramatically helped my project.

Here is the three functions I ended up with...

    public cloneObject3D(objTHREE.Object3DparentTHREE.Object3D): THREE.Object3D {
        let retTHREE.Object3D = null;
        switch(obj.type) {
            case "Group"ret = new THREE.Group(); break;
            case "SkinnedMesh"ret = this.cloneSkinnedMesh(obj as THREE.SkinnedMeshparent); break;
            case "Bone"ret = (obj as THREE.Bone).clone(); break;
            defaultconsole.log('Unknown Clone Type: ' + obj.type); ret = new THREE.Object3D();
        }
        obj.children.forEach((c,i=> {
            ret.add(this.cloneObject3D(c,ret));
        });
        if (obj["ID"]) { ret["ID"] = obj["ID"]; }
        ret.name = obj.name;
        ret.position.set(obj.position.x,obj.position.y,obj.position.z);
        ret.rotation.set(obj.rotation.x,obj.rotation.y,obj.rotation.z);
        ret.scale.set(obj.scale.xobj.scale.y,obj.scale.z);
        ret.quaternion.set(obj.quaternion.x,obj.quaternion.y,obj.quaternion.z,obj.quaternion.w);
        ret.matrixWorldNeedsUpdate = obj.matrixWorldNeedsUpdate;
        ret.userData = this.cloneAny(obj.userData);
        return ret;
    }

    public cloneSkinnedMesh(objTHREE.SkinnedMeshparentTHREE.Object3D): THREE.SkinnedMesh {
        let geo = obj.geometry.clone();
        let mat = obj .material;
        mat = (obj.material as THREE.MeshPhongMaterial).clone();
        let ret = new THREE.SkinnedMesh(geo,mat); 
        let bones = new Array<THREE.Bone>();
        
        obj.skeleton.bones.forEach((b=> { 
            let nb = parent.getObjectByName(b.nameas THREE.Bone;
            bones.push(nb); 
        });

        ret.skeleton = new THREE.Skeleton(bones);

        obj.skeleton.boneMatrices.forEach((bm,bmi=> {
            ret.skeleton.boneMatrices[bmi] = bm;
        });
        obj.skeleton.boneInverses.forEach((bi,bii=> {
            ret.skeleton.boneInverses[bii] = bi;
        });
        obj.bindMatrix.elements.forEach((e,ei=> {
            ret.bindMatrix.elements[ei] = e;
        });
        obj.bindMatrixInverse.elements.forEach((e,ei=> {
            ret.bindMatrixInverse.elements[ei] = e;
        });
                
        return ret;
    }

    public cloneAny(objany): any {
        let ret = {};
        Object.getOwnPropertyNames(obj).forEach((pn=> {
            ret[pn] = obj[pn];
        });
        return ret;
    }

Friday 24 April 2020

Projecting a point onto a sphere

Maths is not a strong skill of mine, but I'm not afraid of it either.  A mathamatical poblem I recently needed to solve was to plot a point in 3d space on a sphere.

I needed to project points using the latitude and longitude coordinate system on both a plane and a sphere.  The plane was simple, but the sphere was more tricky.
Point on Sphere

The formula I settled on is bellow.  I had to add 90° to the latitude otherwise I found it would only plot for one hemesphere.
y = radius * Math.cos(latitude+90°);
x = radius * Math.cos(longitude) * Math.sin(latitude + 90°);
z = radius * Math.sin(longitude) * Math.sin(latitude + 90°);

Monday 20 April 2020

Visualising Earth in PowerBI Custom Visual

Early on in my 3D visualisation experimentation was a desire to recreate the PowerMap experiance.  I eventually managed to find a source of land and country boundaries, which I was able to project into a 3D scene and then plot points based on latitude and longitude.
Country data as map
After many hours searching, experimenting and crying, I found a formula for projecting the points onto a sphere so I could create the globe.
Country data as sphere
But my map data was not perfect, and I wantted a more realistic globe image.  So I downloaded some image data from NASA at https://visibleearth.nasa.gov/.  There Blue Marble collection of images was perfect for my needs.
3D Earth Visualisation
Daytime Globe View

Nighttime Map Visualisation
Night Map View

Using my limited experiance and skills of 3D and PowerBI visual development, I was able to create a PowerBI Cutom Visual with a much more realisitic look and feel.  Points can be plotted by latitude and longitude, or a country can be highlighted by name, ISO2 or ISO3.


3D Earth Visualisation

My projection code (which is the key to making this work..) is as follows:

    private projectAsMap(longnumberlatnumberrnumber): THREE.Vector3 {         let z = long / 180.0;         let y = lat / 180.0;         return new THREE.Vector3(r, -y, -z);     }     private projectAsGlobe(longnumberlatnumberrnumber): THREE.Vector3 {         let s = -long * (Math.PI / 180);         let t = (lat + 90) * (Math.PI / 180);         let x = r * Math.cos(s) * Math.sin(t);         let z = r * Math.sin(s) * Math.sin(t);         let y = r * Math.cos(t);         return new THREE.Vector3(xyz);     }

Rabbits in PowerBI Custom Visual

My inspiration..

My inspiration for this Visual and Blog post is the Aquarium PowerBI Custom visual by ENLIGHTEN DESIGNS.  This visual won the peoples choice award in Microsofts Power BI Best Visual contest.  The contest was used to encurage development and inspiration of new visuals and help promote the PowerBI product.  The Aquarium was simple and fun.

I had a long struggle learning key concepts of 3D programming with WebGL and THREE.JS and furthermore getting this to work inside of PowerBI.  But the results were worth the pain, sweat and tears.

So why Rabbits in PowerBI?.. 

For me, I wantted to making something fun, just like the Aquarium visual or Rockets bouncing around the screen.  So I first considered sheep, playing on the idea that counting sheep puts you to sleep.  However, when looking for a free or low budget sheep 3d model, I was disapointed with my findings.  But, I quickly switched to the idea of rabbits.  After all, they might be more appealing than sheep and can multiply.
Rabbit 3D Model

So, I found a low cost 3d model of a rabbit complete with animations.  The model was amazing, I can't credit the artist/modeler enough.  But the animations where in seperate files and I did not know how to either combine them or load the multiple files as one model.  Later, this turned out to be a blessing.  There are file size limits which I don't know how to get arround.  My attempts to combine the files resulted in large files which I then could not include in PowerBI.  So multiple smaller files worked better once I learnt how to using them togeather.

The visual and code was simple.  I encodd the model, textures and animations as base64 encodings of there original formats (I used FBX for model and animations.)  On Visual constructor, I load decode the base64 strings to load the model, textures and animations into memory.  
3D Rabbits Visualisation
I use a timer to randomly select a rabbit to animate.  The animations avalible can be controlled in the settings.  The rabbits also support tooltips and selection.

Upon Visual update, I recalculate the color and size of all rabbits alread in the scene.  Then I add any additional needed rabbits or remove any extra rabbits.  The rabbits can also be colored to help visually identify categorisation.  Or, a colored spot can be placed below the rabbit.
3D Rabbits Visualisation Colors
It's not a serius visualisation, but was a fun learning project for me.  If I get a chance to work on this visual again, then I would like to make the rabbits also walk around the screen and perhaps add a more visually pleasing grass or field for the rabbits to play.

My First PowerBI Custom Visual

My First PowerBI Custom Visual was an attempt to port my THREE.JS browser based visual.  The port was simple, but I had to cut some of the magic.

My original browser based visual was 7 visuals in one which could transform, filter, recolor, etc.  I was now going to take advantage of PowerBI suppliying the data and performing calculations, grouping, etc.

Originally, the visual represented every record as a partical which would move and recolor to formulat the visuals.  However, with PowerBI performing the groupings and calculations, I don't have visibilty of every record, just the resulting data set.

My first goal was to port a simple implimentation of my "clock" visual.  Instead of particals, I rewrote the visual to use a BoxBufferGeometry.  I also opted to rotate a parent group, where-as previously, I moved the camera.
My First 3D PowerBI Visualisation
Finishing touches included support for selection, highlighting and tooltips.  Lastly, a scale line which moves with the mouse and the ability to control angle of view and rotation.
My First 3D PowerBI Visualisation (Tooltip and Scale)

I hope to reveal some of the code and techniques I used to create this visual in future blog posts.  Please contact me if you interested in knowing more.

Creating a PowerBI Custom Visual with THREE.JS

So, how do we do it...  I'll walk you through a basic skeliton custom visual in PowerBI using THREE.JS.  We are just going to create a spinning box in a PowerBI Visual, future blog posts will cover manipulating a scene based on data and interactivity.
Example 3D Visualisation in PowerBI

First, if your not setup for PowerBI Visual development, you need the tooling installed.  You will need to install Node and PowerBI-Visual-Tools.  Lastly, I also recomend visual studio code.  More information can be found here: https://docs.microsoft.com/en-us/power-bi/developer/visuals/custom-visual-develop-tutorial

The first thing we need is a PowerBI Visual project.  So run:
PBIVIZ new MyFirstThreeJSViz
This will create a new folder and load all the necessary and default code, config and librarys for custom visual development.

Next, we need to install THREE.JS..  So inside the new MyFirstThreeJSViz run the following:
npm i three
That's the visual created and THREE.JS library installed, now we just need to code something.  I usually use VS Code as an IDE, but feel free to use whatever tool you prefer.  Bellow if a screenshot of the shell following the above instructions
Shell example of steps so far

I'll try and walk you through the code block by block, but if you get lost, the full code is at the bottom.

First, we need to import the library into our code.  The primary visual file we are interested in is visual.ts which can be found in the src folder.  Add the bellow import at the top of the file:
import THREE.js
Next, we should define some objects which we will need in our visual.  In THREE.JS, you need four basic objects to sucessfully render a scene. You need a Scene object, this will contain everything which makes up the visual.  Next, a Camera.  The camera controls the viewport into the visual and can be moved to create an animated tour.  Next, a Light.  without a light in our scene, we would not be able to see anything.  Finally, a renderer, this is the object which controls the rendering draw calls and produces the frame.
private scene: THREE.Scene = new THREE.Scene();
private camera: THREE.PerspectiveCamera = new THREE.PerspectiveCamera(75, window.innerWidth / window.innerHeight, 0.1, 800);
private light: THREE.PointLight = new THREE.PointLight();
private renderer: THREE.WebGLRenderer = new THREE.WebGLRenderer({ antialias: true });
On top of these four basic objects, I often also create the following because I use them very frequantly.   A Clock, raycaster, font, modelloadmanager.  Also, it's not uncommon for me to use more than one light.  For this project, we only need the Clock.
private clock: THREE.Clock = new THREE.Clock();
I'm creating the objects inline with there definition, but they need some additional setup.  We need to add to our constructor the logic to connect everything up and start the animation loop.  We do this inside the "if (document) {".  First, lets setup our necessary objects (lights, camera, Action!!)
// Lights
this.light.position.set(0,0.8,2);
this.scene.add(this.light);
// Camera
this.camera.position.set(0,0.5,2);
this.camera.lookAt(0,0,0);
this.scene.add(this.camera);
// Renderer
this.renderer.setSize(window.innerWidth, window.innerHeight);
this.renderer.setClearColor(new THREE.Color(0xFFFFFF));
this.target.appendChild(this.renderer.domElement);
Next, we need to handel resize events.  Add an event listener for resize (the handler function is below..)
// Events
window.addEventListener("resize",this.onResize.bind(this));
Our visual/world would be very empty without anything to see/draw.  Lets add a Red cube to the scene.
// Draw Something
let boxGeometry = new THREE.BoxBufferGeometry(1,1,1);
let boxMaterial = new THREE.MeshPhongMaterial({ color: new THREE.Color(0xff0000) });
let box = new THREE.Mesh(boxGeometry,boxMaterial);
box.name = 'theBox';
this.scene.add(box);
The only thing remaining that we need in our consturator is to start the animation loop.
this.animate();
We are almost there, we just need to code the onResize event handler function and the animate function.  First, the onResize function.  Not much needed to know about this, it's very rare I have to tweak it.  You can just use it as is.
public onResize() {
    this.renderer.setSize(window.innerWidth, window.innerHeight);
    this.camera.aspect = window.innerWidth / window.innerHeight;
    this.camera.updateProjectionMatrix();
}
For the animate function, we first setup some timeing variables which can be useful for animation/tween timing.  Then, we rotate the box using the animation delta time from the THREE.JS Clock.  Next is most important, we render the scene.  Finally, we request the next animation frame.
public animate() {
    let t = performance.now(); // Timestamp
    let d = this.clock.getDelta(); // Animation Delta Time
 
    // Spin the box
    this.scene.getObjectByName('theBox').rotateX(0.1 * d);
    this.scene.getObjectByName('theBox').rotateY(0.5 * d);
 
    // Render Scene using Camera
    this.renderer.render( this.scene, this.camera );
 
    // Request the next Animation Frame
    requestAnimationFrame( this.animate.bind(this) );
}
That's it, you should now be able to Start the PowerBI Visual using PBIVIZ Start, connec to PowerBI and add the developer visual.  In a future post, I will show how to add 3D objects to the scene based upon data.

The Full Code of visual.ts is bellow:

/* *  Power BI Visual CLI * *  Copyright (c) Microsoft Corporation *  All rights reserved. *  MIT License * *  Permission is hereby granted, free of charge, to any person obtaining a copy *  of this software and associated documentation files (the ""Software""), to deal *  in the Software without restriction, including without limitation the rights *  to use, copy, modify, merge, publish, distribute, sublicense, and/or sell *  copies of the Software, and to permit persons to whom the Software is *  furnished to do so, subject to the following conditions: * *  The above copyright notice and this permission notice shall be included in *  all copies or substantial portions of the Software. * *  THE SOFTWARE IS PROVIDED *AS IS*, WITHOUT WARRANTY OF ANY KIND, EXPRESS OR *  IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, *  FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE *  AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER *  LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, *  OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN *  THE SOFTWARE. */ "use strict";
import "core-js/stable"; import "./../style/visual.less"; import powerbi from "powerbi-visuals-api"; import VisualConstructorOptions = powerbi.extensibility.visual.VisualConstructorOptions; import VisualUpdateOptions = powerbi.extensibility.visual.VisualUpdateOptions; import IVisual = powerbi.extensibility.visual.IVisual; import EnumerateVisualObjectInstancesOptions = powerbi.EnumerateVisualObjectInstancesOptions; import VisualObjectInstance = powerbi.VisualObjectInstance; import DataView = powerbi.DataView; import VisualObjectInstanceEnumerationObject = powerbi.VisualObjectInstanceEnumerationObject;
import * as THREE from "three";
import { VisualSettings } from "./settings"; export class Visual implements IVisual {     private targetHTMLElement;     private settingsVisualSettings;          private sceneTHREE.Scene = new THREE.Scene();     private cameraTHREE.PerspectiveCamera = new THREE.PerspectiveCamera(75window.innerWidth / window.innerHeight0.1800);     private lightTHREE.PointLight = new THREE.PointLight();     private rendererTHREE.WebGLRenderer = new THREE.WebGLRenderer({ antialias: true });     private clockTHREE.Clock = new THREE.Clock();
    constructor(optionsVisualConstructorOptions) {         console.log('Visual constructor'options);         this.target = options.element;         if (document) {             // Lights             this.light.position.set(0,0.8,2);             this.scene.add(this.light);             // Camera             this.camera.position.set(0,0.5,2);             this.camera.lookAt(0,0,0);             this.scene.add(this.camera);             // Renderer             this.renderer.setSize(window.innerWidthwindow.innerHeight);             this.renderer.setClearColor(new THREE.Color(0xFFFFFF));             this.target.appendChild(this.renderer.domElement);
            // Events             window.addEventListener("resize",this.onResize.bind(this));
            // Draw Something             let boxGeometry = new THREE.BoxBufferGeometry(1,1,1);             let boxMaterial = new THREE.MeshPhongMaterial({ color: new THREE.Color(0xff0000) });             let box = new THREE.Mesh(boxGeometry,boxMaterial);             box.name = 'theBox';             this.scene.add(box);
            this.animate();         }     }
    public onResize() {         this.renderer.setSize(window.innerWidthwindow.innerHeight);         this.camera.aspect = window.innerWidth / window.innerHeight;         this.camera.updateProjectionMatrix();     }
    public animate() {         let t = performance.now(); // Timestamp         let d = this.clock.getDelta(); // Animation Delta Time                  // Spin the box         this.scene.getObjectByName('theBox').rotateX(0.1 * d);         this.scene.getObjectByName('theBox').rotateY(0.5 * d);
        // Render Scene using Camera         this.renderer.renderthis.scenethis.camera );
        // Request the next Animation Frame         requestAnimationFramethis.animate.bind(this) );     }
    public update(optionsVisualUpdateOptions) {         this.settings = Visual.parseSettings(options && options.dataViews && options.dataViews[0]);         console.log('Visual update'options);     }
    private static parseSettings(dataViewDataView): VisualSettings {         return <VisualSettings>VisualSettings.parse(dataView);     }
    /**      * This function gets called for each of the objects defined in the capabilities files and allows you to select which of the      * objects and properties you want to expose to the users in the property pane.      *      */     public enumerateObjectInstances(optionsEnumerateVisualObjectInstancesOptions): VisualObjectInstance[] | VisualObjectInstanceEnumerationObject {         return VisualSettings.enumerateObjectInstances(this.settings || VisualSettings.getDefault(), options);     } }


Why Create a 3D PowerBI Custom Visual using THREE.JS

The idea of creating my own custom visuals probably started with me first hearing about the Microsoft contest for PowerBI custom visuals.  Not that I thought for a moment I might win, but that it would be a useful skill and perhaps even fun learning experiance.  Sadly, for a year or two, I got nowhere.  Work and personal life kept me from putting much time into it, and I also found it very difficult to find documentation or same code which actuarly made sense to me.

Some guidence that I once read said that it's easier to develop a custom visual by first making the code and visual work outside of PowerBI, then porting it in.  So that's where I started.  I of cause used D3, but the visuals I created where primitive and boaring.  I also then failed to get them into PowerBI anyway.

I remember being at SQLBits visiting the DELL stand to get my book stamped and I noticed something which gave me an idea.  Cardboard.. more specificly Google Cardboard.  Being an IT geek, I was interested in VR and AR, but I'd not really experimented with it.  But seeing how easy and accesable the equipment could be, I stared thinking about using it as a 3D Visual representation of Data.  Basicly using the power of 3D to create an emerse data visualisation.

The idea would not go away, and I quickly learnt how to develop 3D webbased visuals using WebGL and the THREE.JS library.  I did experiment with VR and AR, but I was disapoited with the results.  Mostly the lack or complexity of interactivity.  If I put my phone in the cardboard/vr-headset, then how do I interact with the visual?..

Here are some examples of the visuals I was able to create using THREE.JS.
Clock Visual

Flag Visual

Globe Visual

Map Visual

Spiral Visual


All that aside, I got a lot of attention with the 3D visualisations I was able to create in a browser.  However, I was still unable to get this into PowerBI.  Another year passed, and SQL Bits had a session on Creating your own PowerBI Visualisations.  A must attend session for me.  But the session was busy, and I ended-up standing outside the dome trying to take in as much as posible.  I got the basics and it looked so simple.

I was very quickly able to create basic custom visuals what just showed tables and primitive charts, but every attempt I made to load the THREE.JS library failed.  Typescript was new to me and I was getting lost in typings and finding any documentation limited and confusing.  I tried reaching out for help online, but nothing worked for me.

I must have revisited my code about once every 6 months or so attempting to port my visual into PowerBI using the latest framework, tooling and documentation.  There had been many improvments, but I always ended up stuck with some wired error message which I could not make sense of and google seemed unable to help me either.

Last year, I had a breakthrough.  I think I'd been getting confused with old documentation and my lack of knowledge about typescript.  It could not be any simpler...  Create Visual, Install THREE.JS, Write Code.  No need for typings, chaning build configuration or anything.  It just worked.  I was both exstatic that I could finally attempt to port my 3 year old visual into PowerBI, but also frustrated that it was so simple, and I'd lost so much time going round in circules.

My next posts will show how to create a PowerBI Custom Visual with THREE.JS and some walkthroughs of the visuals I have created.

Saturday 18 April 2020

Going UTC - Working with Daylight savings in SQL (BST/GMT)

Early in my IT carrer, all servers and data was calculated and stored in UK Time with Daylight Savings.  As my IT experiance advanced, I increasingly needed to work with multi-timezone data and/or systems being access from differant timezones.

As a SQL Server DBA, it became increasinly important to know what time an event happend.  All my server estate was configured to UK with Daylight Savings time.  Every year I have 1 hour of missing from March and an hour repeated in October.  Something needed to change.

I started encuraging the use of UTC time and DateTime-Offset data types, but these are not Daylight Savings aware and systems needed to dislay and calculate based on UK Time.

So, I created a collection of SQL CLR functions that could be used to calculate and convert to and from UK Time with Daylight savings to UTC time.

The rules for Daylight Savings time are as follows:
In the UK the clocks go forward 1 hour at 1am on the last Sunday in March, and back 1 hour at 2am on the last Sunday in October.
The period when the clocks are 1 hour ahead is called British Summer Time (BST). There’s more daylight in the evenings and less in the mornings (sometimes called Daylight Saving Time).
When the clocks go back, the UK is on Greenwich Mean Time (GMT).
The code:

    [Microsoft.SqlServer.Server.SqlFunction(
        DataAccess = DataAccessKind.None, 
        IsDeterministic = true, 
        SystemDataAccess = SystemDataAccessKind.None)]
    public static SqlDateTime BSTStart(SqlInt32 year)
    {
        DateTime dt = new DateTime(year.Value, 3, 31);
        while (dt.DayOfWeek != DayOfWeek.Sunday)
        {
            dt = dt.AddDays(-1);
        }
        return new SqlDateTime(dt.AddHours(1));
    }

    [Microsoft.SqlServer.Server.SqlFunction(
        DataAccess = DataAccessKind.None, 
        IsDeterministic = true, 
        SystemDataAccess = SystemDataAccessKind.None)]
    public static SqlDateTime BSTEnd(SqlInt32 year)
    {
        DateTime dt = new DateTime(year.Value, 10, 31);
        while (dt.DayOfWeek != DayOfWeek.Sunday)
        {
            dt = dt.AddDays(-1);
        }
        return new SqlDateTime(dt.AddHours(2));
    }

    [Microsoft.SqlServer.Server.SqlFunction(
        DataAccess = DataAccessKind.None, 
        IsDeterministic = true, 
        SystemDataAccess = SystemDataAccessKind.None)]
    public static SqlBoolean IsBST(SqlDateTime dt)
    {
        return (dt >= BSTStart(dt.Value.Year) && dt <= BSTEnd(dt.Value.Year));
    }

    [Microsoft.SqlServer.Server.SqlFunction(
        DataAccess = DataAccessKind.None, 
        IsDeterministic = false, 
        SystemDataAccess = SystemDataAccessKind.None)]
    public static SqlDateTime GETUKTIME()
    {
        DateTime dt = DateTime.UtcNow;
        if (dt < BSTStart(dt.Year) || dt > BSTEnd(dt.Year)) return new SqlDateTime(dt);
        return new SqlDateTime(dt.AddHours(1));
    }

    [Microsoft.SqlServer.Server.SqlFunction(
        DataAccess = DataAccessKind.None, 
        IsDeterministic = true, 
        SystemDataAccess = SystemDataAccessKind.None)]
    public static SqlDateTime ToUKTime(SqlDateTime dt)
    {
        if (dt < BSTStart(dt.Value.Year) || dt > BSTEnd(dt.Value.Year)) return new SqlDateTime(dt.Value);
        return new SqlDateTime(dt.Value.AddHours(1));
    }

Friday 17 April 2020

Hello World from 2020

I find my self lost in life, both personally and in my carrer, the last few years has been challenging and has almost left me defeated. I'm nolonger a SQL Server DBA which was an important love of mine.  I had two years of working in Azure enviroments which was both a lot of fun, but also a little frustrating at times.  I had very little interation with my beloved SQL Server (except alittle ADW.)  However, my current role has taken me back to SQL Server as a SQL Developer.

In need of a purpose, I've chosen to published some of the blog post I wrote years ago and to also create some new ones.  I hope to publish some more Blog posts shortly which I've already started to draft.  Comming very soon will be some blog post regarding creating 3D PowerBI Custom Visuals using THREE.JS.