I'm a developer and pretty new to the data pipeline side of things and could use some help. I've been tasked with setting up a reporting/dashboard solution given a few constraints. IT's a pretty standard ETL-type set up from my understanding of it. A quick lowdown on the situation is:
- Most of the data that will need to be reported on is available to be queried from a REST API.
- This data will need to be cleaned up at some stage.
- The data should be available to Power BI users in the company (mainly tech and management), and elsewhere.
Ideally, the Power BI users would like a no/low code environment where they can generate and format the data to form specific reports. There are going to be situations where these formatted reports will need to be used to programmatically power other things: automated notifications, datapoints to be written back to other systems, things like that. The hope is that the business can create these reports where they're comfortable and able to do it quickly, and then we can use these reports as a single source of truth for the rest of these data-based operations.
I was hoping I could pull data from these Power BI data stores via REST, but from what I could Google around this does not seem currently possible. It seems like uplifting the transformation stage to something like Data Factory would make sense, since it seems like business could do what they want there and it comes with some other handy features.
I guess ultimately my questions are:
- Has anyone solved a similar need before?
- Is it possible to read data directly from the data flows in Data Factory?
- If so, does doing so make sense?
I'm definitely open to any suggestions or feedback, really appreciate any help.