Application

Going From Unreliable System To A Highly Available System – with Airflow

The Client

McQueen Autocorp is a leading US-based used car selling company with a buyer network spread across the United States. McQueen Autocorp offers a nationwide modern car selling service that facilitates sellers with an instant offer for their car(s) using price prediction algorithms and real-time bids from an extensive network of licensed buyers.

Disclaimer: The name McQueen Autocorp is a placeholder as there is an NDA signed between both parties.

Challenge

The custom application used by our client was operating via classic manual scheduling.

Since the team had to maintain the system manually, it resulted in the pausing of jobs. Then, these jobs had to be manually checked and run. Resultantly, some jobs often got skipped and wouldn’t run on time. It led to frequent failure in the initialization of the application. The system was not reliable.

Scheduled jobs needed manual intervention for monitoring and running, which was tedious. Since data pipelines were complex, it would take hours to trace and rectify the bugs and get the application running again.

Their top priority was finding a solution to eliminate manual intervention, provide traceability of the bugs, and orchestrate workflows and jobs to solve the issue.

Tick the box
Sohail Qureshi
the authorSohail Qureshi

Leave a Reply

Discover more from Advert-Media

Subscribe now to keep reading and get access to the full archive.

Continue reading