The year is coming to an end, and this is a good juncture to look back at 2023. This is a short rewind in 2023 from a programming and coding perspective.
I had the opportunity to incorporate a couple of new frameworks in my work this year.
Read below on my what I have experienced, and my takeaways.
1. Ploomber
This is a project management tool for a data science workflow.
It is similar to other frameworks such as Apache Airflow. It allows the project to be streamlined as a DAG (direct acyclic graph).
According to the developers, it is more suited than Airflow for ML projects. It is more targeted towards the development or explorative stage. What it does not aim to do is data orchestration and running regular production tasks.
Advantages of Ploomber:
- low technical burden (just a package that can be installed in conda)
- can include unit tests, debugging
- keeps track of any new changes in project files
- easily rerun parts of the projects, where scripts have been edited
- can incorporate various languages such as SQL and, R within the pipeline
Disadvantages of Ploomber:
- probably not needed for scripting and quick prototyping
Takeaway: Data projects are quickly evolving, and we need to be able to pivot flexibly at times. It is useful to have an equally flexible pipeline in place to take care of making sure upstream and downstream scripts & outputs are kept up-to-date.
2. WSL2 and Bash scripting
This was more of a lesson for me in managing environments in a virtual machine.
WSL2 = Windows Subsystem for Linux2. It is basically adding a virtual Linux-based command line system that you can access from within Windows itself.
I am currently running Ubuntu with Bash on WSL2. Currently, I find it sufficient enough to do everything data science related from the command line. Git works nicely within WSL2 as well. And I can integrate everything via my favourite IDE (integrated development environment), Visual Studio Code.
Pros:
- can run pretty much anything, as if within a native Linux environment
- don’t need to fiddle with installing similar commands into the Windows command line (cmd)
- WSL2 is a one-liner setup on the Windows command line now, with newer versions of Windows
Cons:
- sometimes the virtualisation gets a bit abstract
- need to be mindful to connect/execute within the right command-line (WSL2 vs windows native)
Takeaway: With big data/cloud computing, we’ll need to deal with remote machines at times. It can get quite abstract but beats running everything on a local machine at times. Getting comfortable with navigating Unix systems is handy.
3. Handlebars
This is a framework that was a bit of a tryout for me. It is based in the web development space, and not so much typical analytics programming.
What is this? It’s a template compiler. Basically, it compiles into HTML and Javascript. Apparently, some code and speed optimisations happen under the hood as well.
I had quite a bit of fun with using ‘moustaches‘ in my code too!
If you don’t know what they are, they are these {{curly braces}}. They are pretty versatile and can do a little bit more than your usual declared variable. For example, one can either load whole templates of website sections or declare helper functions with moustaches.
Don’t have too much by ways of pros & cons on this one, I’ll probably need a bit more time to explore it.
With frontend web development, there is an abundance of frameworks (Can be a bit overwhelming!). It feels like I’m spoilt for choice to decide between a variety of frameworks just for Javascript alone (ie. Vue, Angular, React, etc).
Takeaway: Programming is just a tool and a means to an end. I see the frontend as a means to communicate and put ideas out there. I am pretty sure it will come in handy at some point!
Closing
I am really thankful to be able to geek out on these frameworks. It is like getting a new tool, testing it out, and actually building something from it.
Perhaps, at this point, just looking at a few frameworks might not have amounted to something tangible in the real world. I’m sure it will all add up at a later time.
“The best way to predict the future is to create it” -Peter Drucker