Data first, dashboard second
We start at the source systems, not at the surface. In one running platform the data comes from SQL Server and PostgreSQL, from Excel files, from SharePoint and OneDrive, over SFTP and out of unstructured PDF reports. What sits where, how often does it change, which records are duplicates, which fields are free text. That is the unglamorous half, and the one that decides the outcome.
Getting data out of systems that will not give it up
DBF and DBC from Visual FoxPro, Access, undocumented SQL databases grown over years. That is what we built HydraData for, our own data pump: it reads the legacy data, checks and transforms it internally with DuckDB, and writes the result into a SQL database. Power BI then sits on that database, not on the intermediate step. HydraData is published under the MIT licence.
Metrics everyone agrees on
Before a report exists, we settle with your departments how a metric is calculated and when a transaction counts. Those definitions live in the model, not in individual reports, otherwise they drift apart again.
When it gets large: Fabric with a medallion architecture
For a platform of that size we build in Microsoft Fabric along the medallion pattern: raw data, cleaned layer, reporting layer, cleanly separated. Processing runs mostly in PySpark notebooks. We currently operate one such pipeline holding around 75 million records from the sources listed above, with a short delay rather than an overnight run.
Reporting with Power BI
Reports, roles and permissions, scheduled refresh. Count the licences in: Power BI is charged per report consumer, not per project. We establish early how many people actually need the figures – that shifts the economics more than the choice of technology.
Not every report needs a platform
The target system depends on what you can and want to operate. In another project the opposite was the right answer: sources were a SQL Server and an InfluxDB, extracted with our data pump, reported in Power BI – metrics as measures in the model, transformations in Power Query and M, with no platform underneath. A platform costs money every month and needs someone to look after it. We will tell you when it pays off, and until then we say no.
Who runs it afterwards
Refresh, gateway, permissions and new reports need an owner. Either we take that on, or we set it up so your own people can. Both are fine; leaving it unsaid is not.
Four stages to a figure
Each stage narrows what can still be decided on the next one, which is why the order is not arbitrary.
Sources
First it is settled which systems contribute, in what form they release it and how often the data changes. What is missing here is missing from the report later, and adding it costs more than recording it at the start.
In productionIn one running pipeline six source systems feed the reporting, among them reports that only arrive as PDF and are read by Azure AI Document Intelligence.
Data quality
This is where it is decided which record is valid, which duplicate disappears and what happens to free-text fields. Without that, the next stage calculates cleanly on a crooked set of data.
In productionIn the Fabric pipeline PySpark notebooks handle the checking and reshaping; for legacy data from DBF or Access our data pump HydraData does it internally with DuckDB.
Model
This means the semantic model with its relationships and calculation rules, not an AI model. What revenue means and which transactions count towards it is defined in this one place and applies from then on to every report built on it.
In productionIn the Fabric project it sits on the reporting layer of the medallion structure holding around 75 million records; in the lean counter-example measures in the Power BI model carry the same job.
Operation
At the end stands the question of what the pipeline costs per month and who looks after it. Cadence belongs here too, because every additional run draws capacity.
In productionBecause notebooks consume far fewer capacity units than dataflows in our pipelines, an F2 carries what would otherwise be ordered as an F64; in the counter-example a SQL Server and an InfluxDB were enough with no platform underneath.
The first step is usually smaller than expected: often it is enough to make the numbers in the legacy system visible at all. The legacy system stays untouched, and you decide with figures rather than gut feeling whether a replacement is worth it.
First the conversation, then the analysis, then the quote.
Tell us what you run and what gets in the way. If it fits, we look at your system more closely. Whether that analysis runs as a separate service or feeds into a quote, we tell you before we start. After that you get an assessment: effort, timeframe, price range and the risks we see.
Discuss your project