What's new in Apache Airflow 2.9?
6,634
Published 2024-04-09
Here are the significant updates:
➡️ Custom names for mapped tasks in the UI
💻 No more [1], [2], etc. Clear task names for easy search!
➡️ New @task.bash decorator
💻 Easier to read and to write than the BashOperator
➡️ New API endpoint for Dataset creation and updating
💻 Create cross-airflow dependencies, and trigger DAGs from external tools
➡️ Add TaskFail to Gantt chart
💻 Easily observe how long it took for a task to fail at every attempt
➡️ Task log grouping to the UI
💻 Group your logs to debug your tasks efficiently. No more +1000 line of logs
➡️ New logical operators for Datasets
💻 Use | and & to create complex dataset dependencies
➡️ Pause/Disable DAGs after X failures automatically
💻 No more +100 DAG Runs in failure over the weekends
➡️ Add the on_skipped_callback
💻 Do something when a task is skipped
➡️ Introduce the DatasetOrTimeSchedule
💻 Trigger your DAGs based on time-based schedules or datasets
➡️ Add datasets to the Graph view
💻 Visualize what a DAG is connected to, directly in the Graph view
➡️ New button to create Dataset events
💻 In one click, create a Dataset event to test your DAGs
➡️ Add Dataset conditions to the "next run" modal
💻 Don't guess which dataset is missing to trigger your DAG anymore
➡️ New "Clear only failed tasks"
💻 Pick your DAG run and clear only failed tasks!
➡️ New Task Duration page!
💻 Adios the unreadable Landing Times/Task duration pages. Track your Task Instances and look at the median durations to spot any bottlenecks
➡️ The TOP bar with Graph, Calendar, Code, etc is GONE!
💻 The Grid view is the new view by default
And more!
🏆 BECOME A PRO: www.udemy.com/course/the-complete-hands-on-course-…
👍 Smash the like button to become an Airflow Super Hero!
❤️ Subscribe to my channel to become a master of Airflow
All Comments (7)
-
Which Udemy training has this video in more detail?I want to create a dataset by importing data from postgresql. So when there is an update in the interconnected tables, I will check if there will be an update in the other tables.
-
I have many cases where I depend on many datasets but i must ensure that the dag run is attempted only once per day. Can you do that with the DatasetOrTimeSchedule schedule? something like DatasetAndTimeSchedule?
-
Hello Marc, QQ: Does the "Clear only failed tasks" option rerun the failed task and all of its upstream tasks as well?
-
Where do you need to store the dataset that's going to be used in the DAG?
-
There is a typo in your video description. Version - 2.8, not 2.9
-
Another new feature of 2.9 is the ability to give a name to an expanded task using map_index_template
