AWS Lambda - Why does upgrading the python version cause the packages dependencies to brake?

If you are upgrading your AWS python Lambda’s from 3.7 to 3.8 or 3.9, you may find your previously packaged runtime dependencies no longer work.

A common thing developers package is the requests library, so you would see an an error like:

cannot import module requests from packages.requests
or 
Unable to import module 'lambda_function': No module named lambda_function

This appears to be in part caused by a change in the underlying python runtime used for Lambdas.

The AWS python package documentation has been updated to reflect this change but its very subtle and can be easily overlooked. The documentation uses a package (no s) directory for installing the dependency, not for including the dependency.

Here’s the documentation section in question:

You need to ensure that the external dependencies are in the root, not the packages folder.
Your directory structure including the requests dependencies, would now look like the following:

lambda_handler.py
/requests

Addendum: Importing Indirect Dependencies

Its worth noting that some people have relied on the requests package via boto as this popular Stack Overflow post would indicate. This is not a good practice:

• you have less control over the version you use
• the dependency can decide to remove the package you are importing indirectly

References:
python requests Documentation
AWS Lambda python deployment package
AWS Lambda python runtimes