HOME BLOG

Archive for the ‘Python’ Category

How to view the list of registered tasks in Celery

Posted on: August 27th, 2022 by Olu No Comments

Hi folks,

In this post I talk about a nice tip that could come in handy while working with Celery. It is how to get all registered Celery tasks. You can use code like:

from celery import current_app
all_task_names = current_app.tasks.keys()
all_tasks = current_app.tasks.values()
foo_task = current_app.tasks['tasks.foo']

all_task_classes = [type(task) for task in current_app.tasks.itervalues()]

The task registry is only populated as the modules containing tasks are imported. If you have not imported all modules you can do like the celery worker does, and import all configured task module sources:

current_app.loader.import_default_modules()

That’s all for now. Till next time, happy software development.

 

References

How to get all tasks and periodic tasks in Celery [duplicate]. https://stackoverflow.com/a/12652113

How to install Python 3.6 on MacOS 12 using pyenv

Posted on: August 17th, 2022 by Olu No Comments

Hi folks,

Here’s a quick note on how to install Python 3.6 on MacOS 12 using pyenv.

I needed to do this while workingon a legacy app. Note that Python 3.6 has reached its end-of-life so you should normally be using a more recent version of the langauge as a developer.

Here is the error I was getting:

python-build: use readline from homebrew
Installing Python-3.6.0...
python-build: use tcl-tk from homebrew
python-build: use readline from homebrew
python-build: use zlib from xcode sdk

BUILD FAILED (OS X 12.0.1 using python-build 20180424)

Inspect or clean up the working tree at /var/folders/q9/s5s1hzrd6m1_0by3tx5j6sh40000gn/T/python-build.20220815235540.47954
Results logged to /var/folders/q9/s5s1hzrd6m1_0by3tx5j6sh40000gn/T/python-build.20220815235540.47954.log

Last 10 log lines:
clang -Wno-unused-result -Wsign-compare -Wunreachable-code -DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include   -I/Library/Developer/CommandLineTools/SDKs/MacOSX.sdk/usr/include   -std=c99 -Wextra -Wno-unused-result -Wno-unused-parameter -Wno-missing-field-initializers   -I. -I./Include -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/olu/.pyenv/versions/3.6.0/openssl/include -I/Users/olu/.pyenv/versions/3.6.0/include  -I/usr/local/opt/readline/include -I/usr/local/opt/readline/include -I/Users/olu/.pyenv/versions/3.6.0/openssl/include -I/Users/olu/.pyenv/versions/3.6.0/include   -DPy_BUILD_CORE  -c ./Modules/pwdmodule.c -o Modules/pwdmodule.o
./Modules/posixmodule.c:8146:15: error: implicit declaration of function 'sendfile' is invalid in C99 [-Werror,-Wimplicit-function-declaration]
        ret = sendfile(in, out, offset, &sbytes, &sf, flags);
              ^
./Modules/posixmodule.c:10340:5: warning: code will never be executed [-Wunreachable-code]
    Py_FatalError("abort() called from Python code didn't abort!");
    ^~~~~~~~~~~~~
1 warning and 1 error generated.
make: *** [Modules/posixmodule.o] Error 1
make: *** Waiting for unfinished jobs....

After a lot of research, a solution was found no StackOverflow as follows:

CFLAGS="-I$(brew --prefix openssl)/include -I$(brew --prefix bzip2)/include -I$(brew --prefix readline)/include -I$(xcrun --show-sdk-path)/usr/include" 

LDFLAGS="-L$(brew --prefix openssl)/lib -L$(brew --prefix readline)/lib -L$(brew --prefix zlib)/lib -L$(brew --prefix bzip2)/lib"

pyenv install --patch 3.6.13 < <(curl -sSL https://github.com/python/cpython/commit/8ea6353.patch\?full_index\=1)

Hopefully this helps others. Till next time, happy software development.

 

References

1. Problems installing python 3.6 with pyenv on Mac OS Big Sur. https://stackoverflow.com/a/68227540.

How to mock Python functions with Pytest

Posted on: June 8th, 2022 by Olu No Comments

Hi folks,

In this post I talk about a nice way to mock functions when developing Python applications and using Pytest.

There are many ways of doing mocking in Python. But in this post I will cover how to do it using a nice library called pytest-mock.

To use pytest-mock, you need to install it using a command

 

pip install pytest-mock

 

Say you want to mock a function called is_eligible inside a module called application. You just need to use the mocker fixture which is provided by pytest-mock.

So your function can look as follows.

 

def test_some_function(mocker):
    mocker.patch('application.is_eligible', return_value=True)
    assert some_func_using_is_eligible() == True

 

That’s all for now. Till next time, happy software development.

 

References

Mocking functions in Python with Pytest Part I. https://www.freblogg.com/pytest-functions-mocking-1

How to add Search feature to your Dango app with Haystack and Solr

Posted on: August 4th, 2021 by Olu No Comments

Hi folks,

In this post I talk briefly about how you can add search functionality to your Django app. Why add search platforms like Solr, Elasticsearch,e tc. to your app? Because it makes it easier for users to find relevant information, especially when it may be resource-intensive to perform those searches with your database alone. There’s an interesting article on this topic: Solr and RDBMS: Designing your application for the best of both. https://lucidworks.com/post/solr-and-rdbms-the-basics-of-designing-your-application-for-the-best-of-both/. There are several search tools you can use including Solr, Sphinx, etc. let’s assume you want to use Solr.

There’s a modular library you can use called Haystack (https://django-haystack.readthedocs.io/en/master/toc.html). Haystack supports several search tools, including Solr, Elasticsearch, etc.

If you want to use Haystack with Solr, you’ll need to have Solr running first. You can install Solr on different platforms. Haystack config details are foundĀ  at https://django-haystack.readthedocs.io/en/master/tutorial.html#installation.

Details about installing Solr for use with Haystack can be found here https://django-haystack.readthedocs.io/en/master/installing_search_engines.html.

When you install and start Solr, you would usually want to create something called a core. You perform indexing in that core and perform your searches against the core. Multiple cores can be created on a single Solr instance. When referencing Solr in an application, you would usually use a url like http://127.0.0.1/solr/<yourcore>. Haystack provides management command for interacting building search index and updating search index. You can install Solr using a docker container. See https://hub.docker.com/r/geerlingguy/solr/.

That’s all for now. Till next time, happy software development.

How to view stack trace on app crash when running Django management commands

Posted on: July 19th, 2021 by Olu No Comments

Hi folks,

In this post I describe how to view stack trace on app crash when running Django management commands. When using older versions of Django, if you run a management command and it crashes, you may notice that only the error message is displayed. It would be helpful to see a stack trace as well to facilitate debugging. Thankfully, it’s easy to enable stack trace display.

To view stack trace on error when running Django management command, use the –traceback option. E.g.

 

python manage.py some-command --traceback

 

That’s all for now. Happy software development.

What is Pandas and Why Should You Care About It?

Posted on: July 4th, 2021 by Olu No Comments

Hi folks,

In this post I’ll talk about Pandas.

First, what is Pandas?

Pandas is a software library written in Python programming language. It is used for data manipulation and analysis. It gives you data structures for manipulating numerical tables and time series.

When should you consider learning and using Pandas?

    • If you have to stitch data together from different sources
    • If you need to read data from one or more sources and enrich it
    • If you need to perform statistics on data e.g. find means, counts, etc.
    • If you need to perform data transformations.

Pandas provides several powerful constructs that allow you manipulate data easily.

You can read data into Pandas from many sources, e.g. csv files and MS Excel files. You can also read data into Pandas dataframes from arbitrary sources if the data is available as a list or dictionary.

You can filter and group data, you can also transform data among other things.

When you combine Pandas with a plottingĀ  library like Matplotlib you can create great graphs and charts to visualize data easily.

You can learn more about Pandas by visiting the Pandas website https://pandas.pydata.org/. For some general information no it, visit its Wikipedia page https://en.wikipedia.org/wiki/Pandas_(software).

That’s all for now. Happy software development.

Black – the code formatter

Posted on: May 25th, 2021 by Olu No Comments

Hi folks,

In this post I’ll talk about a very interesting tool called Black. Once upon a time, I was tasked with writing Python software with a few other developers. We came from different backgrounds and thus had very different styles of writing code. There was a broad mix of code writing styles and believe me, the code was a thing to behold. But we decided to just play along and let everyone be and allow “diversity” of code styles. It wasn’t very pretty but we lived with it.

Some months later I found myself reading through the codebase of a Python project and discovered a very interesting tool called Black. The software is described by its authors as The Uncompromising Code Formatter. Once I discovered what it does I realized that this is a tool that would have solved the problem I described earlier. Black formats Python code so that it looks neat consistent. That way, developers don’t have to bicker over poor code format. This also allows developers to focus more on content and less on formatting.

I highly recommend using Black as a code formatter if working on Python in a team setting. You can install Black from PyPi using pip. Click here to go to the project’s Github home page.

That’s all for now. Happy software development.

How to request sensitive details in Python without displaying the input on the screen

Posted on: September 26th, 2020 by Olu No Comments

Hi folks,

Here’s a short code snippet that demonstrates how to request sensitive information from a Python application without displaying the details on screen as the user enters it. The tool to use is a library called getpass as well as the subprocess library. Here it is.

 

import subprocess

info_process = subprocess.run([
'python', '-c', 
"print(import('getpass').getpass('Enter the details here'))"
], stdout=subprocess.PIPE)

details = info_process.stdout.decode('utf-8').strip()

That’s all for now. Happy coding

How to do mocking in setUp in Python

Posted on: August 29th, 2020 by Olu No Comments

Hi folks,

In this post I will talk about how to mock functions in Python across all methods of your test class.

To do this, all you have to do is perform your mocking in the setUp method of your test class.

A way to do this is to create a patch object using code like:

mock_item_patch = mock.patch('some.module.path.item')

 

Next, call start() on the patch object to create your mock using code like

mock_item = mock_item_patch.start()

 

Once you do this, you can set return_value or side_effect as needed. E.g.

mock_item.foo.return_value = 'bar'

 

Don’t forget to add a call to stop the patch on clean-up using code

self.addCleanup(mock_item_patch.stop)

 

Below is code showing all these in action.

 

from unittest import mock, TestCase


class YourTestClass(TestCase):
    def setUp(self):
        mock_item_patch = mock.patch('some.module.path.item')
        mock_item = mock_item_patch.start()
        mock_item.foo.return_value = 'bar'
        self.addCleanup(mock_item_patch.stop)


    def test_something(self):
        ...

 

That’s all for now. Happy software development.

How to perform acions when a Python program finishes whether successfully or not

Posted on: April 25th, 2020 by Olu No Comments

Hi folks,

In this post I talk about a good way to perform actions when your Python program completes. For example you may want to clean up temporary files created by program or close database connections or other resources resources when your program terminates. You may want to do this regardless of whether the program finishes successfully or crashes due to an error. An excellent way to perform such clean-up tasks automatically is to use the atexit module. Here is an example below of how to use it in a class below.

 

import atexit


class Foo:

    def __init__(self):
        atexit.register(self.goodbye)
        x = 5

    def goodbye(self):
        print('You are leaving the program')



f = Foo()

 

If you run the following code, you will see the message ‘You are leaving the program’ just before the program exits.

Here is another example where we intentionally cause the program to crash.

import atexit


class Bar:

    def __init__(self):
        atexit.register(self.goodbye)
        x = 5 / 0 # error!

    def goodbye(self):
        print('You are leaving the program')



f = Bar()

 

If you run this program, you will see the error, but nevertheless still see the message ‘You are leaving the program’ printed before the program exits.

You can read more about atexit in the Python documentation.

That’s it for now. Happy software development.