The Works

In Simple, a data engineer has the task of taking data and organizing it into one place for your company to use for future references. Fortunately your job is not to independently retrieve and write data but rather to program a system which does it.

Think of it like this, a Data engineer is like a plumber, they create the piping that transfers data, and repair the pipes and update them to modern standards

How its done

• Collecting - Dinky goes out and "grabs" raw data from different places,like a website's sales list or a factory's sensor readings.
• Cleaning - Raw data is usually messy or broken. Dinky writes code to fix errors and make sure all the information looks the same so it's actually useful.
• Organizing - Dinky puts the clean data into a digital filing cabinet or warehouse. This makes sure everything is in the right drawer so that when someone needs to find a specific sales number or sensor reading, they can get it in seconds instead of digging through a pile.
• Automating - Instead of doing all this work by hand every morning, Dinky builds a "data robot" (a pipeline). This robot runs the collecting, cleaning, and organizing steps automatically on a schedule, so the data is always fresh without Dinky having to lift a finger.
• Fixing – When something goes wrong, like a website changing its layout or a sensor breaking—the "data robot" might get stuck. Dinky jumps in to investigate the error, updates the code to handle the new problem, and gets the data flowing smoothly again.

My Plan

The path to data enginner is pretty tedious but straightforward.

1. You start as a Data Analyst, learn simple coding techniques and how to do the job you need.
2. Continue on to Junior Software Engineer, focus on more intricate coding and how to actually "engineer" the pipeline.
3. Finally end as a Database Administrator, A seasoned veteran in the field of data engineering.