The baseoperator module seems to be a better choice to keep
If you were relying on
It is still they will be skipped again by the newly introduced NotPreviouslySkippedDep.
custom-auth backend based on Previously num_runs was used to let the scheduler terminate after a certain amount of loops.
wait_for_downstream=True flag. Previously, a sensor is retried when it times out until the number of retries are exhausted. and the user wants to treat any files as new. case. (#5403), [AIRFLOW-2737] Restore original license header to airflow.api.auth.backend.kerberos_auth, [AIRFLOW-3635] Fix incorrect logic in delete_dag (introduced in PR#4406) (#4445), [AIRFLOW-3599] Removed Dagbag from delete dag (#4406), [AIRFLOW-4737] Increase and document celery queue name limit (#5383), [AIRFLOW-4505] Correct Tag ALL for PY3 (#5275), [AIRFLOW-4743] Add environment variables support to SSHOperator (#5385), [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency (#5363), [AIRFLOW-3370] Add stdout output options to Elasticsearch task log handler (#5048), [AIRFLOW-4396] Provide a link to external Elasticsearch logs in UI. To continue using the default smtp email backend, change the email_backend line in your config file from: To continue using S3 logging, update your config file so: [AIRFLOW-463] Link Airflow icon to landing page, [AIRFLOW-149] Task Dependency Engine + Why Isnt My Task Running View, [AIRFLOW-361] Add default failure handler for the Qubole Operator, [AIRFLOW-353] Fix dag run status update failure, [AIRFLOW-447] Store source URIs in Python 3 compatible list, [AIRFLOW-443] Make module names unique when importing, [AIRFLOW-444] Add Google authentication backend, [AIRFLOW-446][AIRFLOW-445] Adds missing dataproc submit options, [AIRFLOW-431] Add CLI for CRUD operations on pools, [AIRFLOW-329] Update Dag Overview Page with Better Status Columns, [AIRFLOW-360] Fix style warnings in models.py.
See https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html for more info.
Now you dont have to create a plugin to configure a e.g.
AIP-21.
All versions of the Astro CLI support all versions of Astro Runtime. You can easily generate config using make() of airflow.providers.google.cloud.operators.dataproc.ClusterGenerator. Previously, this rule was enforced
[AIRFLOW-539] Updated BQ hook and BQ operator to support Standard SQL. The sync-perm CLI command will no longer sync DAG specific permissions by default as they are now being handled during environment variables rather than checking for the presence of a file.
Each logger is a named bucket to which messages can be written for processing.
The methods throw an explanatory exception * for auth backends (#8072), Fix Example in config_templates for Secrets Backend (#8074), Add backticks in IMAGES.rst command description (#8075), Change version_added for store_dag_code config (#8076), [AIRFLOW-XXXX] Remove the defunct limitation of Dag Serialization (#7716), [AIRFLOW-XXXX] Add prerequisite tasks for all GCP operators guide (#6049), [AIRFLOW-XXXX] Simplify AWS/Azure/Databricks operators listing (#6047), [AIRFLOW-XXXX] Add external reference to all GCP operator guide (#6048), [AIRFLOW-XXXX] Simplify GCP operators listing, [AIRFLOW-XXXX] Simplify Qubole operators listing, [AIRFLOW-XXXX] Add autogenerated TOC (#6038), [AIRFLOW-XXXX] Create Using the CLI page (#5823), [AIRFLOW-XXXX] Group references in one section (#5776), [AIRFLOW-XXXX] Add S3 Logging section (#6039), [AIRFLOW-XXXX] Move Azure Logging section above operators (#6040), [AIRFLOW-XXXX] Update temp link to a fixed link (#7715), [AIRFLOW-XXXX] Add Updating.md section for 1.10.9 (#7385), [AIRFLOW-XXXX] Remove duplication in BaseOperator docstring (#7321), [AIRFLOW-XXXX] Update tests info in CONTRIBUTING.rst (#7466), [AIRFLOW-XXXX] Small BREEZE.rst update (#7487), [AIRFLOW-XXXX] Add instructions for logging to localstack S3 (#7461), [AIRFLOW-XXXX] Remove travis config warnings (#7467), [AIRFLOW-XXXX] Add communication chapter to contributing (#7204), [AIRFLOW-XXXX] Add known issue - example_dags/__init__.py (#7444), [AIRFLOW-XXXX] Fix breeze build-docs (#7445), [AIRFLOW-XXXX] Less verbose docker builds, [AIRFLOW-XXXX] Speed up mypy runs (#7421), [AIRFLOW-XXXX] Fix location of kubernetes tests (#7373), [AIRFLOW-XXXX] Remove quotes from domains in Google Oauth (#4226), [AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs (#7318), [AIRFLOW-XXXX] Fix typo in the word committer (#7392), [AIRFLOW-XXXX] Remove duplicated paragraph in docs (#7662), Fix reference to KubernetesPodOperator (#8100), [AIRFLOW-6751] Pin Werkzeug (dependency of a number of our dependencies) to < 1.0.0 (#7377), When task is marked failed by user or task fails due to system failures - on failure call back will be called as part of clean up, [AIRFLOW-4026] Add filter by DAG tags (#6489), [AIRFLOW-6613] Center dag on graph view load (#7238), [AIRFLOW-5843] Add conf option to Add DAG Run view (#7281), [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates (#7038), [AIRFLOW-6438] Filter DAGs returned by blocked (#7019), [AIRFLOW-6666] Resolve js-yaml advisory (#7283), [AIRFLOW-6632] Bump dagre-d3 to resolve lodash CVE advisory (#7280), [AIRFLOW-6667] Resolve serialize-javascript advisory (#7282), [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skipable (#7134), [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner (#7090), [AIRFLOW-6319] Add support for AWS Athena workgroups (#6871), [AIRFLOW-6677] Remove deprecation warning from SQLAlchmey (#7289), [AIRFLOW-6428] Fix import path for airflow.utils.dates.days_ago in Example DAGs (#7007), [AIRFLOW-6595] Use TaskNotFound exception instead of AirflowException (#7210), [AIRFLOW-6620] Mock celery in worker cli test (#7243), [AIRFLOW-6608] Change logging level for Bash & PyOperator Env exports, [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker (#6633), [AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978), [AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909), [AIRFLOW-6316] Use exampleinclude directives in tutorial.rst (#6868), [AIRFLOW-6519] Make TI logs constants in Webserver configurable (#7113), [AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886), [AIRFLOW-6261] flower_basic_auth eligible to _cmd (#6825), [AIRFLOW-6238] Filter dags returned by dag_stats, [AIRFLOW-5616] Switch PrestoHook from pyhive to presto-python-client, [AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg (#7236), [AIRFLOW-6557] Add test for newly added fields in BaseOperator (#7162), [AIRFLOW-6584] Pin cassandra driver (#7194), [AIRFLOW-6537] Fix backticks in RST files (#7140), [AIRFLOW-4428] Error if exec_date before default_args.start_date in trigger_dag (#6948), [AIRFLOW-6330] Show cli help when param blank or typo (#6883), [AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting (#7097), [AIRFLOW-6436] Cleanup for Airflow configs doc generator code (#7036), [AIRFLOW-6436] Add x_frame_enabled config in config.yml (#7024), [AIRFLOW-6436] Create & Automate docs on Airflow Configs (#7015), [AIRFLOW-6527] Make send_task_to_executor timeout configurable (#7143), [AIRFLOW-6272] Switch from npm to yarnpkg for managing front-end dependencies (#6844), [AIRFLOW-6350] Security - spark submit operator logging+exceptions should mask passwords, [AIRFLOW-6358] Log details of failed task (#6908), [AIRFLOW-5149] Skip SLA checks config (#6923), [AIRFLOW-6057] Update template_fields of the PythonSensor (#6656), [AIRFLOW-4445] Mushroom cloud errors too verbose (#6952), [AIRFLOW-6394] Simplify github PR template (#6955), [AIRFLOW-5385] spark hook does not work on spark 2.3/2.4 (#6976), [AIRFLOW-6345] Ensure arguments to ProxyFix are integers (#6901), [AIRFLOW-6576] Fix scheduler crash caused by deleted task with sla misses (#7187), [AIRFLOW-6686] Fix syntax error constructing list of process ids (#7298), [AIRFLOW-6683] REST API respects store_serialized_dag setting (#7296), [AIRFLOW-6553] Add upstream_failed in instance state filter to WebUI (#7159), [AIRFLOW-6357] Highlight nodes in Graph UI if task id contains dots (#6904), [AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter (#7329), [AIRFLOW-6627] Email with incorrect DAG not delivered (#7250), [AIRFLOW-6637] Fix Airflow test command in 1.10.x, [AIRFLOW-6636] Avoid exceptions when printing task instance, [AIRFLOW-6522] Clear task log file before starting to fix duplication in S3TaskHandler (#7120), [AIRFLOW-5501] Make default in_cluster value in KubernetesPodOperator respect config (#6124), [AIRFLOW-6514] Use RUNNING_DEPS to check run from UI (#6367), [AIRFLOW-6381] Remove styling based on DAG id from DAGs page (#6985), [AIRFLOW-6434] Add return statement back to DockerOperator.execute (#7013), [AIRFLOW-2516] Fix mysql deadlocks (#6988), [AIRFLOW-6528] Disable flake8 W503 line break before binary operator (#7124), [AIRFLOW-6517] Make merge_dicts function recursive (#7111), [AIRFLOW-5621] Failure callback is not triggered when marked Failed on UI (#7025), [AIRFLOW-6353] Security - ui - add click jacking defense (#6995), [AIRFLOW-6348] Security - cli.py is currently printing logs with password (#6915), [AIRFLOW-6323] Remove non-ascii letters from default config (#6878), [AIRFLOW-6506] Fix do_xcom_push defaulting to True in KubernetesPodOperator (#7122), [AIRFLOW-6516] BugFix: airflow.cfg does not exist in Volume Mounts (#7109), [AIRFLOW-6427] Fix broken example_qubole_operator dag (#7005), [AIRFLOW-6385] BugFix: SlackAPIPostOperator fails when blocks not set (#7022), [AIRFLOW-6347] BugFix: Cant get task logs when serialization is enabled (#7092), [AIRFLOW-XXXX] Fix downgrade of db migration 0e2a74e0fc9f (#6859), [AIRFLOW-6366] Fix migrations for MS SQL Server (#6920), [AIRFLOW-5406] Allow spark without kubernetes (#6921), [AIRFLOW-6229] SparkSubmitOperator polls forever if status JSON cant (#6918), [AIRFLOW-6352] Security - ui - add login timeout (#6912), [AIRFLOW-6397] Ensure sub_process attribute exists before trying to kill it (#6958), [AIRFLOW-6400] Fix pytest not working on Windows (#6964), [AIRFLOW-6418] Remove SystemTest.skip decorator (#6991), [AIRFLOW-6425] Serialization: Add missing DAG parameters to JSON Schema (#7002), [AIRFLOW-6467] Use self.dag i/o creating a new one (#7067), [AIRFLOW-6490] Improve time delta comparison in local task job tests (#7083), [AIRFLOW-5814] Implementing Presto hook tests (#6491), [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516), [AIRFLOW-XXXX] Move airflow-config-yaml pre-commit before pylint (#7108), [AIRFLOW-XXXX] Improve clarity of confirm message (#7110), [AIRFLOW-6705] One less chatty message at breeze initialisation (#7326), [AIRFLOW-6705] Less chatty integration/backend checks (#7325), [AIRFLOW-6662] Switch to init docker flag for signal propagation (#7278), [AIRFLOW-6661] Fail after 50 failing tests (#7277), [AIRFLOW-6607] Get rid of old local scripts for Breeze (#7225), [AIRFLOW-6589] BAT tests run in pre-commit on bash script changes (#7203), [AIRFLOW-6592] Doc build is moved to test phase (#7208), [AIRFLOW-6641] Better diagnostics for kubernetes flaky tests (#7261), [AIRFLOW-6642] Make local task job test less flaky (#7262), [AIRFLOW-6643] Fix flakiness of kerberos tests, [AIRFLOW-6638] Remove flakiness test from test_serialized_db remove, [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors (#7323), [AIRFLOW-6702] Dumping kind logs to file.io.
(#23183), Fix dag_id extraction for dag level access checks in web ui (#23015), Fix timezone display for logs on UI (#23075), Change trigger dropdown left position (#23013), Dont add planned tasks for legacy DAG runs (#23007), Add dangling rows check for TaskInstance references (#22924), Validate the input params in connection CLI command (#22688), Fix trigger event payload is not persisted in db (#22944), Drop airflow moved tables in command db reset (#22990), Add max width to task group tooltips (#22978), Add template support for external_task_ids.
But since the field is string, its technically been permissible to store any string value.
_operator suffix has been removed from operators. Properly handle BigQuery booleans in BigQuery hook. If you previously had a plugins/my_plugin.py and you used it like this in a DAG: The name under airflow.operators.
with different values. # settings.py and cli.py. The connection module has new deprecated methods: Previously, users could create a connection object in two ways.
(#22347), Fix postgres part of pipeline example of tutorial (#21586), Extend documentation for states of DAGs & tasks and update trigger rules docs (#21382), DB upgrade is required when updating Airflow (#22061), Remove misleading MSSQL information from the docs (#21998), Add the new Airflow Trove Classifier to setup.cfg (#22241), Rename to_delete to to_cancel in TriggerRunner (#20658), Update Flask-AppBuilder to 3.4.5 (#22596). AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030,
Now, additional arguments passed to BaseOperator cause an exception. the amount of queued tasks or use a new pool. Previously, only one backend was used to authorize use of the REST API. Optional.
AWS Batch Operator renamed property queue to job_queue to prevent conflict with the internal queue from CeleryExecutor - AIRFLOW-2542. The following forms of execution_date_fn are all supported: As recommended by Flask, the
This does not affect existing code, but we highly recommend you to restructure the operators dep logic in order to support the new feature. names used across all providers.
Most users would not specify this argument because the bucket begins empty
There were previously two ways of specifying the Airflow home directory
For example to get help about the celery group command,
empty string. To minimize the upgrade time for a Deployment, contact Astronomer support.
release may contain changes that will require changes to your configuration, DAG Files or other integration Built-in operator classes that use this dep class (including sensors and all subclasses) already have this attribute and are not affected. The Used Slots column in Pools Web UI view session_lifetime_days and force_log_out_after options. default_pool is initialized with 128 slots and user can change the
Example of Updating usage of this sensor: If you are upgrading from Airflow 1.10.x and are not using CLUSTER_CONFIG,
The following configurations have been moved from [scheduler] to the new [metrics] section.
Another reason is extensibility: if you store the API host as a simple string
set to us-east-1 during installation.
Confirm that your local upgrade was successful by scrolling to the bottom of any page. Users created and stored in the old users table will not be migrated automatically.
FAB has built-in authentication support and Role-Based Access Control (RBAC), which provides configurable roles and permissions for individual users.
[AIRFLOW-1472] Fix SLA misses triggering on skipped tasks.
If you set the dag_default_view config option or the default_view argument to DAG() to tree you will need to update your deployment. These two flags are close siblings
to use this version.
(breaking change). Then
This will now return an empty string ('''). type for all kinds of Google Cloud Operators. To change that default, read this forum post. From Airflow 3.0, the extra field in airflow connections must be a JSON-encoded Python dict. connection used has no project id defined. For example, to upgrade to Airflow 2.1, your Dockerfile would include the following line: If you're developing locally, make sure to save your changes before proceeding. it prints all config options while in Airflow 2.0, its a command group.
In the new behavior, the trigger_rule of downstream tasks is respected. changes the previous response receiving NULL or '0'.
just specific known keys for greater flexibility.
Add two methods to bigquery hooks base cursor: run_table_upsert, which adds a table or updates an existing table; and run_grant_dataset_view_access, which grants view access to a given dataset for a given table.
untangle cyclic imports between DAG, BaseOperator, SerializedDAG, SerializedBaseOperator which was
If a .py file in the DAGs folder is a zip compressed file, parsing it will fail with an exception. is less forgiving in this area.
do from tempfile import TemporaryDirectory. Now users instead of import from airflow.utils.files import TemporaryDirectory should available.
(#18431), Speed up webserver boot time by delaying provider initialization (#19709), Configurable logging of XCOM value in PythonOperator (#19378), Add hook_params in BaseSqlOperator (#18718), Add missing end_date to hash components (#19281), More friendly output of the airflow plugins command + add timetables (#19298), Add sensor default timeout config (#19119), Update taskinstance REST API schema to include dag_run_id field (#19105), Adding feature in bash operator to append the user defined env variable to system env variable (#18944), Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161), Use inherited trigger_tasks method (#23016), In DAG dependency detector, use class type instead of class name (#21706), Fix tasks being wrongly skipped by schedule_after_task_execution (#23181), Allow extra to be nullable in connection payload as per schema(REST API). In previous versions, the LatestOnlyOperator forcefully skipped all (direct and indirect) downstream tasks on its own. you use operators or hooks which integrate with Google services (including Google Cloud - GCP). (#5196), [AIRFLOW-4447] Display task duration as human friendly format in UI (#5218), [AIRFLOW-4377] Remove needless object conversion in DAG.owner() (#5144), [AIRFLOW-4766] Add autoscaling option for DataprocClusterCreateOperator (#5425), [AIRFLOW-4795] Upgrade alembic to latest release.
The UID to run the first process of the Worker PODs when using has been changed to 50000 The authenticated user has full access.
You can learn about the commands by running airflow --help.
(#21694), Disable default_pool delete on web ui (#21658), Extends typing-extensions to be installed with python 3.8+ #21566 (#21567), Fix logging JDBC SQL error when task fails (#21540), Filter out default configs when overrides exist.
For more detail on changes between Software versions, see Astronomer Software Release Notes. Hooks can define custom connection fields for their connection type by implementing method get_connection_form_widgets.
If you are using DAGs Details API endpoint, use max_active_tasks instead of concurrency.
Since BigQuery is the part of the GCP it was possible to simplify the code by handling the exceptions
In the Software UI, open your Deployment and click Open Airflow.
This This was necessary in order to take advantage of a bugfix concerning refreshing of Kubernetes API tokens with EKS, which enabled the removal of some workaround code. For example: Now if you resolve a Param without a default and dont pass a value, you will get an TypeError. pool queries in MySQL).
Sensors are now accessible via airflow.sensors and no longer via airflow.operators.sensors.
Previously, there were defined in various places, example as ID_PREFIX class variables for (AIRFLOW-1323), post_execute() hooks now take two arguments, context and result [AIRFLOW-2112] Fix svg width for Recent Tasks on UI.
(#13984), An initial rework of the Concepts docs (#15444), Improve docstrings for various modules (#15047), Add documentation on database connection URI (#14124), Add Helm Chart logo to docs index (#14762), Create a new documentation package for Helm Chart (#14643), Add docs about supported logging levels (#14507), Update docs about tableau and salesforce provider (#14495), Replace deprecated doc links to the correct one (#14429), Refactor redundant doc url logic to use utility (#14080), docs: NOTICE: Updated 2016-2019 to 2016-now (#14248), Skip DAG perm sync during parsing if possible (#15464), Add picture and examples for Edge Labels (#15310), Add example DAG & how-to guide for sqlite (#13196), Add links to new modules for deprecated modules (#15316), Add note in Updating.md about FAB data model change (#14478), Fix logging.exception redundancy (#14823), Bump stylelint to remove vulnerable sub-dependency (#15784), Add resolution to force dependencies to use patched version of lodash (#15777), Get rid of Airflow 1.10 in Breeze (#15712), Run helm chart tests in parallel (#15706), Bump ssri from 6.0.1 to 6.0.2 in /airflow/www (#15437), Remove the limit on Gunicorn dependency (#15611), Better dependency already registered warning message for tasks #14613 (#14860), Use Pip 21.
If you were relying on
It is still they will be skipped again by the newly introduced NotPreviouslySkippedDep.
custom-auth backend based on Previously num_runs was used to let the scheduler terminate after a certain amount of loops.
wait_for_downstream=True flag. Previously, a sensor is retried when it times out until the number of retries are exhausted. and the user wants to treat any files as new. case. (#5403), [AIRFLOW-2737] Restore original license header to airflow.api.auth.backend.kerberos_auth, [AIRFLOW-3635] Fix incorrect logic in delete_dag (introduced in PR#4406) (#4445), [AIRFLOW-3599] Removed Dagbag from delete dag (#4406), [AIRFLOW-4737] Increase and document celery queue name limit (#5383), [AIRFLOW-4505] Correct Tag ALL for PY3 (#5275), [AIRFLOW-4743] Add environment variables support to SSHOperator (#5385), [AIRFLOW-4725] Fix setup.py PEP440 & Sphinx-PyPI-upload dependency (#5363), [AIRFLOW-3370] Add stdout output options to Elasticsearch task log handler (#5048), [AIRFLOW-4396] Provide a link to external Elasticsearch logs in UI. To continue using the default smtp email backend, change the email_backend line in your config file from: To continue using S3 logging, update your config file so: [AIRFLOW-463] Link Airflow icon to landing page, [AIRFLOW-149] Task Dependency Engine + Why Isnt My Task Running View, [AIRFLOW-361] Add default failure handler for the Qubole Operator, [AIRFLOW-353] Fix dag run status update failure, [AIRFLOW-447] Store source URIs in Python 3 compatible list, [AIRFLOW-443] Make module names unique when importing, [AIRFLOW-444] Add Google authentication backend, [AIRFLOW-446][AIRFLOW-445] Adds missing dataproc submit options, [AIRFLOW-431] Add CLI for CRUD operations on pools, [AIRFLOW-329] Update Dag Overview Page with Better Status Columns, [AIRFLOW-360] Fix style warnings in models.py.
See https://airflow.apache.org/docs/apache-airflow/stable/howto/custom-operator.html for more info.
Now you dont have to create a plugin to configure a e.g.
AIP-21.
All versions of the Astro CLI support all versions of Astro Runtime. You can easily generate config using make() of airflow.providers.google.cloud.operators.dataproc.ClusterGenerator. Previously, this rule was enforced
[AIRFLOW-539] Updated BQ hook and BQ operator to support Standard SQL. The sync-perm CLI command will no longer sync DAG specific permissions by default as they are now being handled during environment variables rather than checking for the presence of a file.
Each logger is a named bucket to which messages can be written for processing.
The methods throw an explanatory exception * for auth backends (#8072), Fix Example in config_templates for Secrets Backend (#8074), Add backticks in IMAGES.rst command description (#8075), Change version_added for store_dag_code config (#8076), [AIRFLOW-XXXX] Remove the defunct limitation of Dag Serialization (#7716), [AIRFLOW-XXXX] Add prerequisite tasks for all GCP operators guide (#6049), [AIRFLOW-XXXX] Simplify AWS/Azure/Databricks operators listing (#6047), [AIRFLOW-XXXX] Add external reference to all GCP operator guide (#6048), [AIRFLOW-XXXX] Simplify GCP operators listing, [AIRFLOW-XXXX] Simplify Qubole operators listing, [AIRFLOW-XXXX] Add autogenerated TOC (#6038), [AIRFLOW-XXXX] Create Using the CLI page (#5823), [AIRFLOW-XXXX] Group references in one section (#5776), [AIRFLOW-XXXX] Add S3 Logging section (#6039), [AIRFLOW-XXXX] Move Azure Logging section above operators (#6040), [AIRFLOW-XXXX] Update temp link to a fixed link (#7715), [AIRFLOW-XXXX] Add Updating.md section for 1.10.9 (#7385), [AIRFLOW-XXXX] Remove duplication in BaseOperator docstring (#7321), [AIRFLOW-XXXX] Update tests info in CONTRIBUTING.rst (#7466), [AIRFLOW-XXXX] Small BREEZE.rst update (#7487), [AIRFLOW-XXXX] Add instructions for logging to localstack S3 (#7461), [AIRFLOW-XXXX] Remove travis config warnings (#7467), [AIRFLOW-XXXX] Add communication chapter to contributing (#7204), [AIRFLOW-XXXX] Add known issue - example_dags/__init__.py (#7444), [AIRFLOW-XXXX] Fix breeze build-docs (#7445), [AIRFLOW-XXXX] Less verbose docker builds, [AIRFLOW-XXXX] Speed up mypy runs (#7421), [AIRFLOW-XXXX] Fix location of kubernetes tests (#7373), [AIRFLOW-XXXX] Remove quotes from domains in Google Oauth (#4226), [AIRFLOW-XXXX] Add explicit info about JIRAs for code-related PRs (#7318), [AIRFLOW-XXXX] Fix typo in the word committer (#7392), [AIRFLOW-XXXX] Remove duplicated paragraph in docs (#7662), Fix reference to KubernetesPodOperator (#8100), [AIRFLOW-6751] Pin Werkzeug (dependency of a number of our dependencies) to < 1.0.0 (#7377), When task is marked failed by user or task fails due to system failures - on failure call back will be called as part of clean up, [AIRFLOW-4026] Add filter by DAG tags (#6489), [AIRFLOW-6613] Center dag on graph view load (#7238), [AIRFLOW-5843] Add conf option to Add DAG Run view (#7281), [AIRFLOW-4495] Allow externally triggered dags to run for future exec dates (#7038), [AIRFLOW-6438] Filter DAGs returned by blocked (#7019), [AIRFLOW-6666] Resolve js-yaml advisory (#7283), [AIRFLOW-6632] Bump dagre-d3 to resolve lodash CVE advisory (#7280), [AIRFLOW-6667] Resolve serialize-javascript advisory (#7282), [AIRFLOW-6451] self._print_stat() in dag_processing.py should be skipable (#7134), [AIRFLOW-6495] Load DAG only once when running a task using StandardTaskRunner (#7090), [AIRFLOW-6319] Add support for AWS Athena workgroups (#6871), [AIRFLOW-6677] Remove deprecation warning from SQLAlchmey (#7289), [AIRFLOW-6428] Fix import path for airflow.utils.dates.days_ago in Example DAGs (#7007), [AIRFLOW-6595] Use TaskNotFound exception instead of AirflowException (#7210), [AIRFLOW-6620] Mock celery in worker cli test (#7243), [AIRFLOW-6608] Change logging level for Bash & PyOperator Env exports, [AIRFLOW-2279] Clear tasks across DAGs if marked by ExternalTaskMarker (#6633), [AIRFLOW-6359] Make Spark status_poll_interval explicit (#6978), [AIRFLOW-6359] spark_submit_hook.py status polling interval config (#6909), [AIRFLOW-6316] Use exampleinclude directives in tutorial.rst (#6868), [AIRFLOW-6519] Make TI logs constants in Webserver configurable (#7113), [AIRFLOW-6327] http_hook: Accept json= parameter for payload (#6886), [AIRFLOW-6261] flower_basic_auth eligible to _cmd (#6825), [AIRFLOW-6238] Filter dags returned by dag_stats, [AIRFLOW-5616] Switch PrestoHook from pyhive to presto-python-client, [AIRFLOW-6611] Add proxy_fix configs to default_airflow.cfg (#7236), [AIRFLOW-6557] Add test for newly added fields in BaseOperator (#7162), [AIRFLOW-6584] Pin cassandra driver (#7194), [AIRFLOW-6537] Fix backticks in RST files (#7140), [AIRFLOW-4428] Error if exec_date before default_args.start_date in trigger_dag (#6948), [AIRFLOW-6330] Show cli help when param blank or typo (#6883), [AIRFLOW-6504] Allow specifying configmap for Airflow Local Setting (#7097), [AIRFLOW-6436] Cleanup for Airflow configs doc generator code (#7036), [AIRFLOW-6436] Add x_frame_enabled config in config.yml (#7024), [AIRFLOW-6436] Create & Automate docs on Airflow Configs (#7015), [AIRFLOW-6527] Make send_task_to_executor timeout configurable (#7143), [AIRFLOW-6272] Switch from npm to yarnpkg for managing front-end dependencies (#6844), [AIRFLOW-6350] Security - spark submit operator logging+exceptions should mask passwords, [AIRFLOW-6358] Log details of failed task (#6908), [AIRFLOW-5149] Skip SLA checks config (#6923), [AIRFLOW-6057] Update template_fields of the PythonSensor (#6656), [AIRFLOW-4445] Mushroom cloud errors too verbose (#6952), [AIRFLOW-6394] Simplify github PR template (#6955), [AIRFLOW-5385] spark hook does not work on spark 2.3/2.4 (#6976), [AIRFLOW-6345] Ensure arguments to ProxyFix are integers (#6901), [AIRFLOW-6576] Fix scheduler crash caused by deleted task with sla misses (#7187), [AIRFLOW-6686] Fix syntax error constructing list of process ids (#7298), [AIRFLOW-6683] REST API respects store_serialized_dag setting (#7296), [AIRFLOW-6553] Add upstream_failed in instance state filter to WebUI (#7159), [AIRFLOW-6357] Highlight nodes in Graph UI if task id contains dots (#6904), [AIRFLOW-3349] Use None instead of False as value for encoding in StreamLogWriter (#7329), [AIRFLOW-6627] Email with incorrect DAG not delivered (#7250), [AIRFLOW-6637] Fix Airflow test command in 1.10.x, [AIRFLOW-6636] Avoid exceptions when printing task instance, [AIRFLOW-6522] Clear task log file before starting to fix duplication in S3TaskHandler (#7120), [AIRFLOW-5501] Make default in_cluster value in KubernetesPodOperator respect config (#6124), [AIRFLOW-6514] Use RUNNING_DEPS to check run from UI (#6367), [AIRFLOW-6381] Remove styling based on DAG id from DAGs page (#6985), [AIRFLOW-6434] Add return statement back to DockerOperator.execute (#7013), [AIRFLOW-2516] Fix mysql deadlocks (#6988), [AIRFLOW-6528] Disable flake8 W503 line break before binary operator (#7124), [AIRFLOW-6517] Make merge_dicts function recursive (#7111), [AIRFLOW-5621] Failure callback is not triggered when marked Failed on UI (#7025), [AIRFLOW-6353] Security - ui - add click jacking defense (#6995), [AIRFLOW-6348] Security - cli.py is currently printing logs with password (#6915), [AIRFLOW-6323] Remove non-ascii letters from default config (#6878), [AIRFLOW-6506] Fix do_xcom_push defaulting to True in KubernetesPodOperator (#7122), [AIRFLOW-6516] BugFix: airflow.cfg does not exist in Volume Mounts (#7109), [AIRFLOW-6427] Fix broken example_qubole_operator dag (#7005), [AIRFLOW-6385] BugFix: SlackAPIPostOperator fails when blocks not set (#7022), [AIRFLOW-6347] BugFix: Cant get task logs when serialization is enabled (#7092), [AIRFLOW-XXXX] Fix downgrade of db migration 0e2a74e0fc9f (#6859), [AIRFLOW-6366] Fix migrations for MS SQL Server (#6920), [AIRFLOW-5406] Allow spark without kubernetes (#6921), [AIRFLOW-6229] SparkSubmitOperator polls forever if status JSON cant (#6918), [AIRFLOW-6352] Security - ui - add login timeout (#6912), [AIRFLOW-6397] Ensure sub_process attribute exists before trying to kill it (#6958), [AIRFLOW-6400] Fix pytest not working on Windows (#6964), [AIRFLOW-6418] Remove SystemTest.skip decorator (#6991), [AIRFLOW-6425] Serialization: Add missing DAG parameters to JSON Schema (#7002), [AIRFLOW-6467] Use self.dag i/o creating a new one (#7067), [AIRFLOW-6490] Improve time delta comparison in local task job tests (#7083), [AIRFLOW-5814] Implementing Presto hook tests (#6491), [AIRFLOW-5704] Improve Kind Kubernetes scripts for local testing (#6516), [AIRFLOW-XXXX] Move airflow-config-yaml pre-commit before pylint (#7108), [AIRFLOW-XXXX] Improve clarity of confirm message (#7110), [AIRFLOW-6705] One less chatty message at breeze initialisation (#7326), [AIRFLOW-6705] Less chatty integration/backend checks (#7325), [AIRFLOW-6662] Switch to init docker flag for signal propagation (#7278), [AIRFLOW-6661] Fail after 50 failing tests (#7277), [AIRFLOW-6607] Get rid of old local scripts for Breeze (#7225), [AIRFLOW-6589] BAT tests run in pre-commit on bash script changes (#7203), [AIRFLOW-6592] Doc build is moved to test phase (#7208), [AIRFLOW-6641] Better diagnostics for kubernetes flaky tests (#7261), [AIRFLOW-6642] Make local task job test less flaky (#7262), [AIRFLOW-6643] Fix flakiness of kerberos tests, [AIRFLOW-6638] Remove flakiness test from test_serialized_db remove, [AIRFLOW-6701] Rat is downloaded from stable backup/mirrors (#7323), [AIRFLOW-6702] Dumping kind logs to file.io.
(#23183), Fix dag_id extraction for dag level access checks in web ui (#23015), Fix timezone display for logs on UI (#23075), Change trigger dropdown left position (#23013), Dont add planned tasks for legacy DAG runs (#23007), Add dangling rows check for TaskInstance references (#22924), Validate the input params in connection CLI command (#22688), Fix trigger event payload is not persisted in db (#22944), Drop airflow moved tables in command db reset (#22990), Add max width to task group tooltips (#22978), Add template support for external_task_ids.
But since the field is string, its technically been permissible to store any string value.
_operator suffix has been removed from operators. Properly handle BigQuery booleans in BigQuery hook. If you previously had a plugins/my_plugin.py and you used it like this in a DAG: The name under airflow.operators.
with different values. # settings.py and cli.py. The connection module has new deprecated methods: Previously, users could create a connection object in two ways.
(#22347), Fix postgres part of pipeline example of tutorial (#21586), Extend documentation for states of DAGs & tasks and update trigger rules docs (#21382), DB upgrade is required when updating Airflow (#22061), Remove misleading MSSQL information from the docs (#21998), Add the new Airflow Trove Classifier to setup.cfg (#22241), Rename to_delete to to_cancel in TriggerRunner (#20658), Update Flask-AppBuilder to 3.4.5 (#22596). AIP-39: Add (customizable) Timetable class to Airflow for richer scheduling behaviour (#15397, #16030,
Now, additional arguments passed to BaseOperator cause an exception. the amount of queued tasks or use a new pool. Previously, only one backend was used to authorize use of the REST API. Optional. AWS Batch Operator renamed property queue to job_queue to prevent conflict with the internal queue from CeleryExecutor - AIRFLOW-2542. The following forms of execution_date_fn are all supported: As recommended by Flask, the
This does not affect existing code, but we highly recommend you to restructure the operators dep logic in order to support the new feature. names used across all providers.
Most users would not specify this argument because the bucket begins empty
There were previously two ways of specifying the Airflow home directory
For example to get help about the celery group command,
empty string. To minimize the upgrade time for a Deployment, contact Astronomer support.
release may contain changes that will require changes to your configuration, DAG Files or other integration Built-in operator classes that use this dep class (including sensors and all subclasses) already have this attribute and are not affected. The Used Slots column in Pools Web UI view session_lifetime_days and force_log_out_after options. default_pool is initialized with 128 slots and user can change the
Example of Updating usage of this sensor: If you are upgrading from Airflow 1.10.x and are not using CLUSTER_CONFIG,
The following configurations have been moved from [scheduler] to the new [metrics] section.
Another reason is extensibility: if you store the API host as a simple string
set to us-east-1 during installation.
Confirm that your local upgrade was successful by scrolling to the bottom of any page. Users created and stored in the old users table will not be migrated automatically.
FAB has built-in authentication support and Role-Based Access Control (RBAC), which provides configurable roles and permissions for individual users.
[AIRFLOW-1472] Fix SLA misses triggering on skipped tasks.
If you set the dag_default_view config option or the default_view argument to DAG() to tree you will need to update your deployment. These two flags are close siblings
to use this version.
(breaking change). Then
This will now return an empty string ('''). type for all kinds of Google Cloud Operators. To change that default, read this forum post. From Airflow 3.0, the extra field in airflow connections must be a JSON-encoded Python dict. connection used has no project id defined. For example, to upgrade to Airflow 2.1, your Dockerfile would include the following line: If you're developing locally, make sure to save your changes before proceeding. it prints all config options while in Airflow 2.0, its a command group.
In the new behavior, the trigger_rule of downstream tasks is respected. changes the previous response receiving NULL or '0'.
just specific known keys for greater flexibility.
Add two methods to bigquery hooks base cursor: run_table_upsert, which adds a table or updates an existing table; and run_grant_dataset_view_access, which grants view access to a given dataset for a given table.
untangle cyclic imports between DAG, BaseOperator, SerializedDAG, SerializedBaseOperator which was
If a .py file in the DAGs folder is a zip compressed file, parsing it will fail with an exception. is less forgiving in this area.
do from tempfile import TemporaryDirectory. Now users instead of import from airflow.utils.files import TemporaryDirectory should available.
(#18431), Speed up webserver boot time by delaying provider initialization (#19709), Configurable logging of XCOM value in PythonOperator (#19378), Add hook_params in BaseSqlOperator (#18718), Add missing end_date to hash components (#19281), More friendly output of the airflow plugins command + add timetables (#19298), Add sensor default timeout config (#19119), Update taskinstance REST API schema to include dag_run_id field (#19105), Adding feature in bash operator to append the user defined env variable to system env variable (#18944), Duplicate Connection: Added logic to query if a connection id exists before creating one (#18161), Use inherited trigger_tasks method (#23016), In DAG dependency detector, use class type instead of class name (#21706), Fix tasks being wrongly skipped by schedule_after_task_execution (#23181), Allow extra to be nullable in connection payload as per schema(REST API). In previous versions, the LatestOnlyOperator forcefully skipped all (direct and indirect) downstream tasks on its own. you use operators or hooks which integrate with Google services (including Google Cloud - GCP). (#5196), [AIRFLOW-4447] Display task duration as human friendly format in UI (#5218), [AIRFLOW-4377] Remove needless object conversion in DAG.owner() (#5144), [AIRFLOW-4766] Add autoscaling option for DataprocClusterCreateOperator (#5425), [AIRFLOW-4795] Upgrade alembic to latest release.
The UID to run the first process of the Worker PODs when using has been changed to 50000 The authenticated user has full access.
You can learn about the commands by running airflow --help.
(#21694), Disable default_pool delete on web ui (#21658), Extends typing-extensions to be installed with python 3.8+ #21566 (#21567), Fix logging JDBC SQL error when task fails (#21540), Filter out default configs when overrides exist.
For more detail on changes between Software versions, see Astronomer Software Release Notes. Hooks can define custom connection fields for their connection type by implementing method get_connection_form_widgets.
If you are using DAGs Details API endpoint, use max_active_tasks instead of concurrency.
Since BigQuery is the part of the GCP it was possible to simplify the code by handling the exceptions
In the Software UI, open your Deployment and click Open Airflow.
This This was necessary in order to take advantage of a bugfix concerning refreshing of Kubernetes API tokens with EKS, which enabled the removal of some workaround code. For example: Now if you resolve a Param without a default and dont pass a value, you will get an TypeError. pool queries in MySQL).
Sensors are now accessible via airflow.sensors and no longer via airflow.operators.sensors.
Previously, there were defined in various places, example as ID_PREFIX class variables for (AIRFLOW-1323), post_execute() hooks now take two arguments, context and result [AIRFLOW-2112] Fix svg width for Recent Tasks on UI.
(#13984), An initial rework of the Concepts docs (#15444), Improve docstrings for various modules (#15047), Add documentation on database connection URI (#14124), Add Helm Chart logo to docs index (#14762), Create a new documentation package for Helm Chart (#14643), Add docs about supported logging levels (#14507), Update docs about tableau and salesforce provider (#14495), Replace deprecated doc links to the correct one (#14429), Refactor redundant doc url logic to use utility (#14080), docs: NOTICE: Updated 2016-2019 to 2016-now (#14248), Skip DAG perm sync during parsing if possible (#15464), Add picture and examples for Edge Labels (#15310), Add example DAG & how-to guide for sqlite (#13196), Add links to new modules for deprecated modules (#15316), Add note in Updating.md about FAB data model change (#14478), Fix logging.exception redundancy (#14823), Bump stylelint to remove vulnerable sub-dependency (#15784), Add resolution to force dependencies to use patched version of lodash (#15777), Get rid of Airflow 1.10 in Breeze (#15712), Run helm chart tests in parallel (#15706), Bump ssri from 6.0.1 to 6.0.2 in /airflow/www (#15437), Remove the limit on Gunicorn dependency (#15611), Better dependency already registered warning message for tasks #14613 (#14860), Use Pip 21.