Projects worked on for Engie Electrabel(March 2018 - October 2018)
In 2018 I worked most of the year as one of the lead python developers at ETS group. This involved working on various parts of their many operational projects mostly targeted at data mining of windmill and hydro energy production in order to optimise revenue from renewables.
Various windmill and data suppliers had their own way of sharing energy production data. Some used nice JSON api's. Some used a plain old csv file. Some had just a flat text file. Some used xml documents. Basically a whole plethora of different ways to exchange data is used.
The octopus project gathers all this data with various worker classes and then stores all the data in a structured PostgreSQL database. From there on the B2B web application Back to Earth is able to show this data and allow editing of contracts with suppliers etc.
Wrote various workers within this existing micro framework using TDD to fetch data of various suppliers.
Back to Earth administration pages
Build on flask, flask-admin and a lot of customisation done in blueprints called BlueSnake internally developed by the Admian team. Basically they wrapped flask and flask-admin with their own custom engine that would allow for efficient creation of web apps, allowing code reuse of a lot of boiler plate code normally to be rewritten when you start with a python flask app from scratch. IMHO it was nice but documentation did lack a lot and you had to either know things, sift through a lot of well written but sometimes difficult to read code in order to find your way around.
Nowadays in 2020 I highly recommend the Starlette framework in Python: it has the same goals of allowing faster development of new python webapps. And it's faster in performance due to async/reactor pattern (like nodejs). It's also really well documented and easy to start working with. So new people on the team would be able to pick it up more quickly than a custom in-house framework.
Various improvements were made like a nice graphviz integration to get a visual overview of the contracts using cool looking dynamic generated node graphs.
Smaller project involving around importing and exporting data from a system called pipedrive.
They had an API but it did not expose all data. They wanted to move to a more custom/in house system.
And gave me the assignment to extract all data using some python scripts. In a week or 2 I had a working screen scraping export script that logged in using chrome browser extension to have a proper session and allowed to get all the data they wanted exported into csv files.
This page in under construction, and will be completed soon...
If maintenance happens, or weather conditions are not optimal enough or demand is lower than the supply available with other resources it still happens wind mills have to be shut down (curtailment).
To handle the automatisation of these events and have a general overview of such type of events, a project was started around this to make it more user friendly. The main complexity with this was getting data out of a big database called tourbillion and the events would overlap and had to be normalised.
Pandas was used a lot internally. It's a huge python data analysis library with a lot of cool features (highly recommended if you ever need to do datamining). However my knowledge of the pandas framework was sometimes insufficient, so I had to learn a lot around this new framework in order to then actually use it's potential properly. A lot of features I could write from scratch easily enough but using pandas would allow to do it with better runtime performance.
I'm grateful for having the experience working at ETS group. The people were nice and I learned to improve my python pandas and Vue.js skills a lot while working there on their various projects. Their git and atlassian bugtracking integration with branching for each feature was pretty sweet. Deployment and CI/CD was also done well. Code reviews were really tight and pre-commit was used together with tight autopip8 rules.
My next customer became VIAA