How can I mock requests and the response?

I am trying to use Pythons mock package to mock Pythons requests module. What are the basic calls to get me working in below scenario?

In my views.py, I have a function that makes variety of requests.get() calls with different response each time

def myview(request): res1 = requests.get('aurl') res2 = request.get('burl') res3 = request.get('curl') 

In my test class I want to do something like this but cannot figure out exact method calls

Step 1:

# Mock the requests module # when mockedRequests.get('aurl') is called then return 'a response' # when mockedRequests.get('burl') is called then return 'b response' # when mockedRequests.get('curl') is called then return 'c response' 

Step 2:

Call my view

Step 3:

verify response contains 'a response', 'b response' , 'c response'

How can I complete Step 1 (mocking the requests module)?

17 Answers

This is how you can do it (you can run this file as-is):

import requests import unittest from unittest import mock # This is the class we want to test class MyGreatClass: def fetch_json(self, url): response = requests.get(url) return response.json() # This method will be used by the mock to replace requests.get def mocked_requests_get(*args, **kwargs): class MockResponse: def __init__(self, json_data, status_code): self.json_data = json_data self.status_code = status_code def json(self): return self.json_data if args[0] == ' return MockResponse({"key1": "value1"}, 200) elif args[0] == ' return MockResponse({"key2": "value2"}, 200) return MockResponse(None, 404) # Our test case class class MyGreatClassTestCase(unittest.TestCase): # We patch 'requests.get' with our own method. The mock object is passed in to our test case method. @mock.patch('requests.get', side_effect=mocked_requests_get) def test_fetch(self, mock_get): # Assert requests.get calls mgc = MyGreatClass() json_data = mgc.fetch_json(') self.assertEqual(json_data, {"key1": "value1"}) json_data = mgc.fetch_json(') self.assertEqual(json_data, {"key2": "value2"}) json_data = mgc.fetch_json(') self.assertIsNone(json_data) # We can even assert that our mocked method was called with the right parameters self.assertIn(mock.call('), mock_get.call_args_list) self.assertIn(mock.call('), mock_get.call_args_list) self.assertEqual(len(mock_get.call_args_list), 3) if __name__ == '__main__': unittest.main() 

Important Note: If your MyGreatClass class lives in a different package, say my.great.package, you have to mock my.great.package.requests.get instead of just 'request.get'. In that case your test case would look like this:

import unittest from unittest import mock from my.great.package import MyGreatClass # This method will be used by the mock to replace requests.get def mocked_requests_get(*args, **kwargs): # Same as above class MyGreatClassTestCase(unittest.TestCase): # Now we must patch 'my.great.package.requests.get' @mock.patch('my.great.package.requests.get', side_effect=mocked_requests_get) def test_fetch(self, mock_get): # Same as above if __name__ == '__main__': unittest.main() 

Enjoy!

Try using the responses library. Here is an example from their documentation:

import responses import requests @responses.activate def test_simple(): responses.add(responses.GET, ' json={'error': 'not found'}, status=404) resp = requests.get(') assert resp.json() == {"error": "not found"} assert len(responses.calls) == 1 assert responses.calls[0].request.url == ' assert responses.calls[0].response.text == '{"error": "not found"}' 

It provides quite a nice convenience over setting up all the mocking yourself.

There's also HTTPretty:

It's not specific to requests library, more powerful in some ways though I found it doesn't lend itself so well to inspecting the requests that it intercepted, which responses does quite easily

There's also httmock.

Here is what worked for me:

import mock @mock.patch('requests.get', mock.Mock(side_effect = lambda k:{'aurl': 'a response', 'burl' : 'b response'}.get(k, 'unhandled request %s'%k))) 

I used requests-mock for writing tests for separate module:

# module.py import requests class A(): def get_response(self, url): response = requests.get(url) return response.text 

And the tests:

# tests.py import requests_mock import unittest from module import A class TestAPI(unittest.TestCase): @requests_mock.mock() def test_get_response(self, m): a = A() m.get(' text='a response') self.assertEqual(a.get_response('), 'a response') m.get(' text='b response') self.assertEqual(a.get_response('), 'b response') m.get(' text='c response') self.assertEqual(a.get_response('), 'c response') if __name__ == '__main__': unittest.main() 

this is how you mock requests.post, change it to your http method

@patch.object(requests, 'post') def your_test_method(self, mockpost): mockresponse = Mock() mockpost.return_value = mockresponse mockresponse.text = 'mock return' #call your target method now 

Here is a solution with requests Response class. It is cleaner IMHO.

import json from unittest.mock import patch from requests.models import Response def mocked_requests_get(*args, **kwargs): response_content = None request_url = kwargs.get('url', None) if request_url == 'aurl': response_content = json.dumps('a response') elif request_url == 'burl': response_content = json.dumps('b response') elif request_url == 'curl': response_content = json.dumps('c response') response = Response() response.status_code = 200 response._content = str.encode(response_content) return response @mock.patch('requests.get', side_effect=mocked_requests_get) def test_fetch(self, mock_get): response = requests.get(url='aurl') assert ... 

I started out with Johannes Farhenkrug's answer here and it worked great for me. I needed to mock the requests library because my goal is to isolate my application and not test any 3rd party resources.

Then I read up some more about python's Mock library and I realized that I can replace the MockResponse class, which you might call a 'Test Double' or a 'Fake', with a python Mock class.

The advantage of doing so is access to things like assert_called_with, call_args and so on. No extra libraries are needed. Additional benefits such as 'readability' or 'its more pythonic' are subjective, so they may or may not play a role for you.

Here is my version, updated with using python's Mock instead of a test double:

import json import requests from unittest import mock # defube stubs AUTH_TOKEN = '{"prop": "value"}' LIST_OF_WIDGETS = '{"widgets": ["widget1", "widget2"]}' PURCHASED_WIDGETS = '{"widgets": ["purchased_widget"]}' # exception class when an unknown URL is mocked class MockNotSupported(Exception): pass # factory method that cranks out the Mocks def mock_requests_factory(response_stub: str, status_code: int = 200): return mock.Mock(**{ 'json.return_value': json.loads(response_stub), 'text.return_value': response_stub, 'status_code': status_code, 'ok': status_code == 200 }) # side effect mock function def mock_requests_post(*args, **kwargs): if args[0].endswith('/api/v1/get_auth_token'): return mock_requests_factory(AUTH_TOKEN) elif args[0].endswith('/api/v1/get_widgets'): return mock_requests_factory(LIST_OF_WIDGETS) elif args[0].endswith('/api/v1/purchased_widgets'): return mock_requests_factory(PURCHASED_WIDGETS) raise MockNotSupported # patch requests.post and run tests with mock.patch('requests.post') as requests_post_mock: requests_post_mock.side_effect = mock_requests_post response = requests.post(') assert response.ok is True assert response.status_code == 200 assert 'widgets' in response.json() # now I can also do this requests_post_mock.assert_called_with(') 

Repl.it links:

If you want to mock a fake response, another way to do it is to simply instantiate an instance of the base HttpResponse class, like so:

from django.http.response import HttpResponseBase self.fake_response = HttpResponseBase() 

One possible way to work around requests is using the library betamax, it records all requests and after that if you make a request in the same url with the same parameters the betamax will use the recorded request, I have been using it to test web crawler and it save me a lot time.

import os import requests from betamax import Betamax from betamax_serializers import pretty_json WORKERS_DIR = os.path.dirname(os.path.abspath(__file__)) CASSETTES_DIR = os.path.join(WORKERS_DIR, u'resources', u'cassettes') MATCH_REQUESTS_ON = [u'method', u'uri', u'path', u'query'] Betamax.register_serializer(pretty_json.PrettyJSONSerializer) with Betamax.configure() as config: config.cassette_library_dir = CASSETTES_DIR config.default_cassette_options[u'serialize_with'] = u'prettyjson' config.default_cassette_options[u'match_requests_on'] = MATCH_REQUESTS_ON config.default_cassette_options[u'preserve_exact_body_bytes'] = True class WorkerCertidaoTRT2: session = requests.session() def make_request(self, input_json): with Betamax(self.session) as vcr: vcr.use_cassette(u'google') response = session.get(') 

This worked for me, although I haven't done much complicated testing yet.

import json from requests import Response class MockResponse(Response): def __init__(self, url=' headers={'Content-Type':'text/html; charset=UTF-8'}, status_code=200, reason = 'Success', _content = 'Some html goes here', json_ = None, encoding='UTF-8' ): self.url = url self.headers = headers if json_ and headers['Content-Type'] == 'application/json': self._content = json.dumps(json_).encode(encoding) else: self._content = _content.encode(encoding) self.status_code = status_code self.reason = reason self.encoding = encoding 

Then you can create responses :

mock_response = MockResponse( headers={'Content-Type' :'application/json'}, status_code=401, json_={'success': False}, reason='Unauthorized' ) mock_response.raise_for_status() 

gives

requests.exceptions.HTTPError: 401 Client Error: Unauthorized for url: 

Can you use requests-mock instead?

Suppose your myview function instead takes a requests.Session object, makes requests with it, and does something to the output:

# mypackage.py def myview(session): res1 = session.get("") res2 = session.get("") res3 = session.get("") return f"{res1.text}, {res2.text}, {res3.text}" 

# test_myview.py from mypackage import myview import requests def test_myview(requests_mock): # set up requests a_req = requests_mock.get("", text="a response") b_req = requests_mock.get("", text="b response") c_req = requests_mock.get("", text="c response") # test myview behaviour session = requests.Session() assert myview(session) == "a response, b response, c response" # check that requests weren't called repeatedly assert a_req.called_once assert b_req.called_once assert c_req.called_once assert requests_mock.call_count == 3 

You can also use requests_mock with frameworks other than Pytest - the documentation is great.

I will add this information since I had a hard time figuring how to mock an async api call.

Here is what I did to mock an async call.

Here is the function I wanted to test

async def get_user_info(headers, payload): return await httpx.AsyncClient().post(URI, json=payload, headers=headers) 

You still need the MockResponse class

class MockResponse: def __init__(self, json_data, status_code): self.json_data = json_data self.status_code = status_code def json(self): return self.json_data 

You add the MockResponseAsync class

class MockResponseAsync: def __init__(self, json_data, status_code): self.response = MockResponse(json_data, status_code) async def getResponse(self): return self.response 

Here is the test. The important thing here is I create the response before since init function can't be async and the call to getResponse is async so it all checked out.

@pytest.mark.asyncio @patch('httpx.AsyncClient') async def test_get_user_info_valid(self, mock_post): """test_get_user_info_valid""" # Given token_bd = "abc" username = "bob" payload = { 'USERNAME': username, 'DBNAME': 'TEST' } headers = { 'Authorization': 'Bearer ' + token_bd, 'Content-Type': 'application/json' } async_response = MockResponseAsync("", 200) mock_post.return_value.post.return_value = async_response.getResponse() # When await api_bd.get_user_info(headers, payload) # Then mock_post.return_value.post.assert_called_once_with( URI, json=payload, headers=headers) 

If you have a better way of doing that tell me but I think it's pretty clean like that.

The simplest way so far:

from unittest import TestCase from unittest.mock import Mock, patch from .utils import method_foo class TestFoo(TestCase): @patch.object(utils_requests, "post") # change to desired method here def test_foo(self, mock_requests_post): # EXPLANATION: mocked 'post' method above will return some built-in mock, # and its method 'json' will return mock 'mock_data', # which got argument 'return_value' with our data to be returned mock_data = Mock(return_value=[{"id": 1}, {"id": 2}]) mock_requests_post.return_value.json = mock_data method_foo() # TODO: asserts here """ Example of method that you can test in utils.py """ def method_foo(): response = requests.post("") records = response.json() for record in records: print(record.get("id")) # do other stuff here 

For those, who don't want to install additional libs for pytest, there is an example. I will duplicate it here with some extension, based on examples above:

import datetime import requests class MockResponse: def __init__(self, json_data, status_code): self.json_data = json_data self.status_code = status_code self.elapsed = datetime.timedelta(seconds=1) # mock json() method always returns a specific testing dictionary def json(self): return self.json_data def test_get_json(monkeypatch): # Any arguments may be passed and mock_get() will always return our # mocked object, which only has the .json() method. def mock_get(*args, **kwargs): return MockResponse({'mock_key': 'mock_value'}, 418) # apply the monkeypatch for requests.get to mock_get monkeypatch.setattr(requests, 'get', mock_get) # app.get_json, which contains requests.get, uses the monkeypatch response = requests.get(') response_json = response.json() assert response_json['mock_key'] == 'mock_value' assert response.status_code == 418 assert response.elapsed.total_seconds() == 1 ============================= test session starts ============================== collecting ... collected 1 item test_so.py::test_get_json PASSED [100%] ============================== 1 passed in 0.07s =============================== 

Just a helpful hint to those that are still struggling, converting from urllib or urllib2/urllib3 to requests AND trying to mock a response- I was getting a slightly confusing error when implementing my mock:

with requests.get(path, auth=HTTPBasicAuth('user', 'pass'), verify=False) as url:

AttributeError: __enter__

Well, of course, if I knew anything about how with works (I didn't), I'd know it was a vestigial, unnecessary context (from PEP 343). Unnecessary when using the requests library because it does basically the same thing for you under the hood. Just remove the with and use bare requests.get(...) and Bob's your uncle.

For pytest users there is a convinient fixture from

For example to mock GET to you can:

def test_me(response_mock): with response_mock('GET -> 200 :Nice'): response = send_request() assert result.ok assert result.content == b'Nice' 

I will demonstrate how to detach your programming logic from the actual external library by swapping the real request with a fake one that returns the same data. In your view if external api call then this process is best

import pytest from unittest.mock import patch from django.test import RequestFactory @patch("path(projectname.appname.filename).requests.post") def test_mock_response(self, mock_get, rf: RequestFactory): mock_get.return_value.ok = Mock(ok=True) mock_get.return_value.status_code = 400 mock_get.return_value.json.return_value = {you can define here dummy response} request = rf.post("test/", data=self.payload) response = view_name_view(request) expected_response = { "success": False, "status": "unsuccessful", } assert response.data == expected_response assert response.status_code == 400