ADE gives us the possibility to do actual development with quite a small team. It never "gets it the way" and also seems to minimize time spend on maintenance. ADE has been really reliable and does make it easier to follow best practices by using well made templates and with good documentation and community tips.

ADE is easy to to start using, but at the same time offers so many possibilities for customization. It's quite plausible to only use GUI and be happy with the default options, but by reading the documentation and ADEs generated SQL and then using the different load options like RUN IDs and TIMESLOT you can tune up the performance of your data ops. Also ADE is not a "black box" - you can from the logs and code previews check exactly what your loads are doing, you will be able to pinpoint any problems.

Run ID manipulation is only possible with External API. It's often needed if adding new loads from older entities since run ids are saved only for about a month. Also when entity and package counts rise, it can be hard to keep track of intertwined relationships between the entities, packages and load schedules. For example you introduce a new load from a new source to an pre-existing link entity and suddenly the subsequent loads are ran in multiple DAGs (schedules).

I don't understand how we could have achieved this kind of speed in our multi-team without Agile Data Engine. We have moved the most complex and critical tailor made python/aws/lambda applications to ADE and the 50% of time -maintenance has gone to near zero % of my time.

With ADE implementation project and leveraging ADE-team's suggested practises, you'll get the data platform architecture you have always wanted. The core capabilities and features, like trunk based development, deployment management and load orchestration frees teams capacity from jacking the code to implementing solutions faster to production.

The notification framework in this context is quite unique and it needs attention and cloud skills right in the beginning. The notifier lambda (in AWS) function is the most critical single component in the architecture and would have massive impact if failed. The SPOF factor is quite high for this. But when the solution is in place and working, you can forget it mostly.