In the world of growing data, you always need to have top-of-the-line and efficient fetching integrations. At Foundation Direct, we need integrations to fetch data from Google Ads, GA4, Bing Ads, and much more. These integrations fetch and process a lot of data, and speed is an important measurement in measuring the success of these integrations. In this blogpost I will discuss how we improved our integrations by transferring some of these high volume integrations from Python to Go.
Let’s dive in!
A Little About Python
Python is a great language, and has extensive libraries that can help with almost anything. The language reads easy as well, and it’s quick to develop with it.
Although Python has many strengths, there is one major weakness: it is slow. One of the primary reasons for this slow behavior is due to Python being interpreted. Python code is executed line by line by the Python interpreter. This line by line behavior adds overhead, as the interpreter needs to translate the high-level code into machine code, which can slow down execution.
A Little About Go
Go is an open-source programming language developed by Google in 2007 and officially released in 2009. Go was designed for simplicity, efficiency, and strong support for concurrent programming, Go is known for its fast compilation times and strong performance. It combines the ease of use typically found in high-level languages (like Python) with the efficiency of systems languages (like C++, C). Go offers a strong standard library, a special `go` reserved keyword for lightweight concurrency, and offers an easy-to-understand syntax. Lastly, Go is a compiled language, unlike Python’s interpreted quality.
The Transition
Upon researching Go, we decided to dive deeper into it and do a POC on our Google Ads API fetch. With Python, we were clocking this fetch at around 2 hours and 10 mins and it utilized the Google Ads API streaming endpoint as well as concurrent futures Python standard library. Our goal was to get this fetch under 1 hour and 45 minutes, so we began learning Go!
Go is a very fun language to learn, and Go offers great tutorials to get programmers of any skill level started.
After a few days, we were able to start implementing the Go code into the Google Ads API integration. After we implemented it, we began testing and we clocked a full Google Ads API integration run at… 34 minutes, wow!
Notes on Go
After implementing our other integrations using Go over the last few weeks I have garnered some notes on Go.
First, it’s very fast, and it handles concurrency incredibly!
Second, it’s type-safe, which proved to make things difficult initially because we have different fetches that have varying types. Once we were able to get one working, the rest were very easy to get working.
Third, it was relatively manageable to pick it up after having exposure to a few other languages. It reads easily as well, and has great VS Code support.
Conclusion
This was a brief blogpost to discuss our transition to Go for fetching integrations. I briefly covered the differences between compiled Go and interpreted Python, as well as the speed increase we had on one of our API fetching jobs. I highly recommend investigating Go, and if it fits your stack you will save a lot of execution time utilizing Go.
Lastly, as a side note, we were able to easily integrate this into the Kestra orchestration tool utilizing Kestra Task Runners in GCP Batch. We simply set up a Go Dockerfile image and execute that image as a container with the Batch API (which uses Compute Engine). So this transition to Go only really took effort in writing the Go code, and then it was simply a just ‘replace Python call’ in Kestra with a ‘Go call’. If you have not checked out Kestra, I highly recommend checking it out, it’s an incredible tool!
Thank you, and Happy Coding!
